Submitting Forms with Selenium in Python

When scraping, we often run into sites that need us to submit information. Forms play several essential roles in the web ecosystem, serving as crucial elements that facilitate user interaction, data input, and communication with web applications.

Selenium stands out as a powerful and versatile tool, providing a seamless solution for automating browser actions. With Selenium, we can find any element on a page and we have the power to interact with it just like a regular human user would. Not only can we find these elements quickly, but we can fill and submit them at lightning speed in comparison to a regular user.

In this article, we will delve into the fundamental concepts, practical examples, and best practices for harnessing the full potential of Selenium to navigate and interact with web forms.

• TLDR
• Interacting With Form Elements
• Methods of Submitting Forms
• Uploading Files with Selenium
• Troubleshooting
• Conclusion
• More Web Scraping Guides

TLDR: How To Submit Form In Selenium​

To submit a form in Selenium, these steps should be followed:

1. Locate the Form: Identify the HTML form on the webpage using Selenium's element locating methods.

2. Fill in Data: Use Selenium to input data into the form fields, mimicking user interactions.

3. Submit the Form: Locate the submit button or trigger the form submission action using Selenium.

4. Handle Confirmation: If there's a confirmation pop-up or post-submission action, use Selenium to handle it.

The example below shows a basic breakdown of how to fill a form in Selenium:

from selenium import webdriver
from selenium.webdriver.common.by import By
#launch a browser instance
driver = webdriver.Chrome()
#navigate to the page
driver.get("https://quotes.toscrape.com/login")
#find our elements
username_box = driver.find_element(By.ID, "username")
password_box = driver.find_element(By.ID, "password")
login_button = driver.find_element(By.XPATH, "/html/body/div/form/input[2]")
#send keys to the input boxes
username_box.send_keys("my_username")
password_box.send_keys("my-super-secret-password")
#click the submit button
login_button.click()
#close the browser
driver.quit()

• First, we find our elements with the find_element() method
• Once we have our input elements, we fill them with the send_keys() method
• Finally, after filling the form, we click the "Login" button to submit our information

Interacting with Form Elements In Selenium​

To interact with form elements, we first have to identify them. In Selenium, we use two methods to do this.

1. find_element finds a single element on the page and returns it to us.
2. find_elements() finds all elements of a certain criteria and returns them in the form of a list.

Interacting With Text Input Boxes​

As we have done already, we can select values by saving them as variables. Anytime you use find_element() or find_elements(), you get a web element that you can try all sorts of various methods to interact with. When filling an input box, we use the send_keys() method.

In the TLDR section above, we used find_element() to find each individual input field. Let's revise the example to find both elements and return them in a list.

from selenium.webdriver.common.by import By
#launch a browser instance
driver = webdriver.Chrome()
#navigate to the page
driver.get("https://quotes.toscrape.com/login")
#find our elements
elements = driver.find_elements(By.CLASS_NAME, "form-control")
#send keys to the input boxes
for element in elements:
    element.send_keys("just some random text")
login_button = driver.find_element(By.XPATH, "/html/body/div/form/input[2]")
login_button.click()
#close the browser
driver.quit()

In the example above, we:

• Launched a Chrome browser instance using the WebDriver.
• Navigated to the Quotes to Scrape login URL.
• Found all elements on the page with the class name "form-control" and store them in the elements list.
• Iterated through each element in the elements list and type "just some random text" into the input boxes.
• Located the login button using its XPath ("/html/body/div/form/input[2]").
• Clicked the login button to simulate form submission.
• And finally closed the Chrome browser instance using driver.quit().

Deleting Values in Input Boxes​

Deleting values in input boxes is often necessary for several reasons in web scraping, particularly to clear existing search queries or filter criteria, enabling the scraper to input new parameters and dynamically adapt to changing requirements, ultimately retrieving the desired data accurately.

If we wish to delete values from an input box, we can use the clear() method. This method removes all input from our web element. Take a look at the example below:

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
#launch a browser instance
driver = webdriver.Chrome()
#navigate to the page
driver.get("https://quotes.toscrape.com/login")
#find our elements
username_box = driver.find_element(By.ID, "username")
password_box = driver.find_element(By.ID, "password")
#send keys to the input boxes
username_box.send_keys("my_username")
password_box.send_keys("my-super-secret-password")
#sleep so we can view the filled input
sleep(1)
#clear the input
username_box.clear()
#sleep so we can view the cleared input
sleep(1)
#close the browser
driver.quit()

In the example above, we:

• Find the input boxes with find_element()
• Fill input boxes with send_keys()
• sleep() for one second so we can see the input boxes get filled
• Delete input from username_box with clear()
• sleep() for another second so we can see that the input has been cleared from username_box

Interacting With Checkboxes​

Checkboxes are commonly used in forms and settings to allow users to select or deselect options.

When selecting checkboxes, we simply click() them just like you would when browsing the web like normal. The click() method gives us the power to do anything we would normally do with our mouse, and to think the way we would think when browsing the web.

The example below finds all elements, of a certain type and selects them.

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
#launch a browser instance
driver = webdriver.Chrome()
#navigate to the page
driver.get("https://designsystem.digital.gov/components/checkbox/")
#find our elements
elements = driver.find_elements(By.CLASS_NAME, "usa-checkbox__label")
for element in elements:
    if not element.is_selected():
        element.click()
        sleep(1)
    else:
        None
#close the browser
driver.quit()

In the code example above:

• Created an instance of the Chrome browser using the webdriver.Chrome() constructor.
• Opened the specified URL ("https://designsystem.digital.gov/components/checkbox/") in the Chrome browser.
• Located all elements on the page with the class name "usa-checkbox__label" and store them in the elements list.
• Iterated through each element in the elements list.
  • Checked if the checkbox is not already selected using element.is_selected().
  • If not selected, simulated a click on the checkbox using element.click() and introduce a 1-second delay using sleep(1) for visualization.
  • If the checkbox is already selected, did nothing (else: None).
• Quit the Chrome browser instance using driver.quit().

Interacting With Radio Buttons​

Radio buttons are commonly used in forms to allow users to choose a single option from a set.

When interacting with radio buttons, we also use the click() method. Not all the elements returned are clickable, so we use a try statement to attempt clicking them and an except statement to do nothing with them if they're not clickable.

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
#launch a browser instance
driver = webdriver.Chrome()
#navigate to the page
driver.get("https://designsystem.digital.gov/components/radio-buttons/")
#find our elements
elements = driver.find_elements(By.CLASS_NAME, "usa-radio__label")
for element in elements:
    if not element.is_selected():
        try:
            element.click()
            sleep(1)
        except:
            None
    else:
        None
#close the browser
driver.quit()

In the example above, we followed a similar route with the previous example.

• Located all elements on the page with the class name "usa-radio__label" and stored them in the elements list.
• Iterated through each element in the elements list.
  • Checked if the radio button was not already selected using element.is_selected().
  • If not selected, attempted to simulate a click on the radio button using element.click() and introduced a 1-second delay using sleep(1) for visualization. If an exception occurred during the click attempt, did nothing (except: None).
  • If the radio button was already selected, did nothing (else: None).
• Quit the Chrome browser instance using driver.quit().

Interacting With Dropdowns​

Dropdowns, also known as select elements, are commonly used in forms to allow users to choose from a list of predefined options.

Once again, as before, we use the click() method to begin interacting with the element. Once we've clicked the button, we get a list of options. We use find_element() to find the selection we'd like to click, and then we use click() to select the option.

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
#launch a browser instance
driver = webdriver.Chrome()
#navigate to the page
driver.get("https://designsystem.digital.gov/components/time-picker/")
#find our dropdown element
element = driver.find_element(By.ID, "appointment-time")
#click the dropdown
element.click()
#find our choice
choice = driver.find_element(By.ID, "appointment-time--list--option-2")
sleep(1)
#click the choice
choice.click()
sleep(1)
#close the browser
driver.quit()

Let's review the code block step by step:

• After we opened the specified URL, we located the dropdown element with the ID "appointment-time."
• Clicked on the dropdown to reveal the available choices.
• Located the desired choice element within the dropdown with the ID "appointment-time--list--option-2."
• Introduced a 1-second delay using sleep(1) for visualization purposes.
• Clicked on the chosen option.
• Introduced another 1-second delay using sleep(1).
• Closed the Chrome browser instance using driver.quit().

Methods Of Submitting Forms In Selenium​

Just like filling a form manually, there are multiple ways to submit forms in Selenium. Each method usually has its own best usecases and sometimes it comes down to personal preference.

We can submit forms in the following ways:

1. Method 1: Clicking the Submit Button
2. Method 2: Using the Submit Method on an Element
3. Method 3: Sending Enter Key to a Form Element
4. Method 4: JavaScript Execution

The best choice usually depends on the page that you're working with. Depending on what you're doing it may be more efficient click() the submit button, press the Enter key, use the submit() method, or to use JavaScript Execution to submit the information.

Most importantly, you need to think, "What would a human do on this page?", the answer to that question is most often the method that you should use. This is why we're using Selenium to begin with, we want our program to behave like a human.

Method 1: Clicking the Submit Button​

As you've done throughout this tutorial, to use the submit button, simply use find_element() to find the button you're looking for and use click() to click on it.

submit_button = driver.find_element(By.ID, "submit_button_id")
submit_button.click()

When to use this method?

• Use this method when the form includes a clearly identifiable submit button, and interacting with the button triggers the form submission.
• Appropriate for standard HTML forms where the submit button is a standard input element with the type "submit."
• Suitable for cases where simulating user interactions is the primary objective.

Method 2: Using the Submit Method on an Element​

Selenium also allows you to call the submit() method on any form element, not just the submit button. This method submits the form that contains the element.

form_element = driver.find_element(By.NAME, "some_form_element")
form_element.submit()

When to use this method?

• Use this method when you have a direct reference to the form element, and calling the submit() method on that element is a straightforward way to trigger submission.
• Helpful in scenarios where submitting the form aligns with accessibility practices, as the submit() method can trigger the default form submission behavior.

Sending Enter Key to a Form Element​

You can also submit information by sending the Enter key to the submit button. It works the same way it would when using the Enter key in your browser.

from selenium.webdriver.common.keys import Keys
input_field = driver.find_element(By.ID, "input_field_id")
input_field.send_keys(Keys.ENTER)

When to use this method?

• Use this method when the form does not have a visible submit button, and pressing the Enter key is a valid way to submit the form.
• Appropriate for cases where the user typically submits the form by pressing Enter after typing in a text input or textarea.

JavaScript Execution​

Selenium also gives us the execute_script() method which allows us to insert JavaScript code directly into our Python script.

from selenium import webdriver
from time import sleep
#launch a browser instance
driver = webdriver.Chrome()
#navigate to the page
driver.get("https://quotes.toscrape.com/login")
#find our element
driver.execute_script("document.querySelector('.btn').click();")
sleep(1)
#close the browser
driver.quit()

In the example above, we do the following:

• Launch Chrome and navigate to the site
• Pass native JavaScript, document.querySelector('.btn').click(); into our execute_script() method to execute JavaScript arbitrarily on our webpage

When to use this method?

• Use JavaScript execution when form submission involves complex logic, validations, or custom event handling that may not be effectively triggered by standard Selenium methods.
• Consider using JavaScript execution as a last resort when other methods fail or when dealing with intricate web applications where standard methods may not suffice.

Uploading Files In Forms With Selenium​

As you've probably needed to do before as a regular user, we regularly need to upload files to the web. When uploading a file, we pass the absolute path to the file into the send_keys() method. Different operating systems have different path layouts, but Python's os module gives us a really easy way to work around that.

First, inside of the same folder as your scraper, create a new file, example.txt. and just add some random text to it. Then save the file and add the following code to a Python script:

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
import os 
#launch a browser instance
driver = webdriver.Chrome()
#navigate to the site
driver.get("https://designsystem.digital.gov/components/file-input/")
#find the file upload element
file_upload = driver.find_element(By.ID, "file-input-single")
#pass the absolute path to the file into the send_keys() method
file_upload.send_keys(os.path.abspath("./example_file.txt"))
#sleep so we can view the file upload
sleep(2)
#close the browser
driver.quit()

In the example above, we do the following:

• Start the browser and navigate to the site with webdriver.Chrome() and driver.get()
• Find the upload element with driver.find_element(By.ID, "file-input-single")
• Once we've found our element, we use file_upload.send_keys(os.path.abspath("./example_file.txt"))

The OS module gives us simple and universal way to find the absolute path to example.txt no matter which operating system we''re using!

Dealing With Form CAPTCHAs In Selenium​

To solve CAPTCHAS in Selenium, you can either do it manually or use a CAPTCHA solving service. The code example below uses 2captcha. In order to use 2captcha, you need to sign up on their site and pay for service. You can install the 2captcha-python module with the following command:

pip install 2captcha-python

Next, we need to identify the sitekey of our captcha. You can do this with your developer console and inspecting the page. You can view an example of this below.

Once you've found your sitekey, we'll make a Python script that uses both our API_KEY and site_key as variables.

Replace site_key and API_KEY in the code below:

from selenium import webdriver
from selenium.webdriver.common.by import By
from twocaptcha import TwoCaptcha
#save API_KEY
API_KEY = "YOUR-SUPER-SECRET-API-KEY"
#save site_key
site_key = "YOUR-CAPTCHA-SITEKEY"
#create a TwoCaptcha instance
solver = TwoCaptcha(API_KEY)
#open Chrome
driver = webdriver.Chrome()
#website url
website = "https://recaptcha-demo.appspot.com/recaptcha-v2-checkbox.php"
#navigate to site
driver.get(website)
#save a sscreenshot before solving
driver.save_screenshot("before_solve.png")
#get the response
response = solver.recaptcha(sitekey=site_key, url=website)
#save the response code
code = response["code"]
#find the response element on the page
response_element = driver.find_element(By.ID, "g-recaptcha-response")
#use JS to add the response code to the page
driver.execute_script(f'arguments[0].value="{code}";', response_element)
#find the submit button
submit = driver.find_element(By.CSS_SELECTOR, 'button[type="submit"]')
#click the submit button
submit.click()
#take a screenshot after solving
driver.save_screenshot("after_solve.png")
driver.quit()

In the code above, we:

• Save our site_key and API_KEY as variables
• Create a TwoCaptcha instance with solver = TwoCaptcha(API_KEY)
• Save our website url and navigate to it with driver.get(website)
• Take a screenshot before solving with driver.save_screenshot("before_solve.png")
• Get the response object with response = solver.recaptcha(sitekey=site_key, url=website)
• Save response["code"] as a variable
• Find our response element with response_element = driver.find_element(By.ID, "g-recaptcha-response")
• Insert our code into the response element with driver.execute_script(f'arguments[0].value="{code}";', response_element)
• Find the submit button with submit = driver.find_element(By.CSS_SELECTOR, 'button[type="submit"]')
• Submit the form with submit.click()

Before solving the code, we take the following screenshot:

After we've solved, we get the following screenshot:

Troubleshooting​

When automating forms with Selenium, developers face a myriad of issues they run into that will cause problems because at the end of the day, we're trying to simulate a human, but we're using a pre-defined script that only does what we tell it to do.

Issue 1: Element Not Found​

If we search for an element in the current page and it isn't there, Selenium will throw an element not found error. The proper way to address this is by waiting for the element to pop-up on the site so it can be found. We also need to make sure that our locators are correct.

If you pass an incorrect locator to find_element() or find_elements(), Selenium will look for the wrong element, so it won't find the element that you're actually searching for.

To properly address this, we can need to ensure that our locators are correct and then use a proper waiting strategy to find the element.

The code below uses an implicit wait while we interact with elements:

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
#define our url
url = "https://espn.com"
#open Chrome
driver = webdriver.Chrome()
#set implicit wait to 10 seconds
driver.implicitly_wait(10)
#navigate to the url
driver.get(url)
#find the search button
search_button = driver.find_element(By.ID, "global-search-trigger")
#click it so the search bar opens
search_button.click()
#find the search bar after waiting implicitly for it to open
search_bar = driver.find_element(By.ID, "global-search-input")
#send input to the search bar
search_bar.send_keys("detroit")
#pause so we can view the screen
sleep(5)
driver.quit()

In the code above, we do the following:

• Define our url
• Open a Chrome instance with webdriver.Chrome()
• Set an implicit wait of 10 seconds for the session with driver.implicitly_wait(10)
• Find the search button with driver.find_element()
• click() the search
• Use driver.find_element() to find the search bar while implicitly waiting 10 seconds for it to appear
• Enter "detroit" into the search bar with search_bar.send_keys("detroit")
• Pause for 5 seconds using the sleep() function so we can view the screen

Issue #2: Element Not Interactable​

Sometimes, Selenium can find an element but we can't interact with it. To address this, you need to know how your website behaves and what actions allow us to interact with the element.

For instance, if we are clicking a submit button, but the submit button isn't clickable until we've filled all the required fields, we need to make sure that all the fields are filled and that the submit button is clickable.

In the previous code example, if we try to click() any of our search results, we get an error because the element is not interactable. In order to make it interactable, we need to hover our mouse over it. We do this with ActionChains.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from time import sleep
#define our url
url = "https://espn.com"
#open Chrome
driver = webdriver.Chrome()
#set implicit wait to 10 seconds
driver.implicitly_wait(10)
#create an ActionChains object
action = ActionChains(driver)
#navigate to the url
driver.get(url)
#find the search button
search_button = driver.find_element(By.ID, "global-search-trigger")
#click it so the search bar opens
search_button.click()
#find the search bar after waiting implicitly for it to open
search_bar = driver.find_element(By.ID, "global-search-input")
#send input to the search bar
search_bar.send_keys("detroit")
#find the search results and return them as a list
results = driver.find_elements(By.CLASS_NAME, "search_results__label")
#iterate through the results
for result in results:
    #if we've found the result we want
    if result.text == "Detroit Lions":
        #save the result
        target = result
        #exit the loop
        break
#move the cursor to the target and click it
action.move_to_element(target)\
    .click()\
    .perform()
#pause so we can view the screen
sleep(5)
driver.quit()

The example above is almost the same as the previous one, but with a few key differences:

• After sending input to search_bar, we use find_elements() to find all search result elements
• We iterate through the results with a for loop until we find out target result
• Once we've found our target, we move to it and click on it with action.move_to_element(target).click().perform()

When dealing with ActionChains, things can get very unreadable very quickly. We can use \ to continue our code on a new line.

Issue #3: Incorrect Element Interactions​

Occasionally, when we click() a button or send_keys() to an element, it simply won't work correctly! When investigating issues like this, best practice is to use JavaScript execution in order to interact with the element.

JavaScript has builtin DOM (Document Object Model) support for interacting with and manipulating webpages. If JavaScript can't interact with your element, it is most likely a problem with the site and not your scraper.

Let's pretend our previous method with ActionChains failed. We can use execute_script() to execute native JavaScript and take advatage of JavaScript's first class DOM support.

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
#define our url
url = "https://espn.com"
#open Chrome
driver = webdriver.Chrome()
#set implicit wait to 10 seconds
driver.implicitly_wait(10)
#navigate to the url
driver.get(url)
#find the search button
search_button = driver.find_element(By.ID, "global-search-trigger")
#click it so the search bar opens
search_button.click()
#find the search bar after waiting implicitly for it to open
search_bar = driver.find_element(By.ID, "global-search-input")
#send input to the search bar
search_bar.send_keys("detroit")
#find the search results and return them as a list
results = driver.find_elements(By.CLASS_NAME, "search_results__label")
#iterate through the results
for result in results:
    #if we've found the result we want
    if result.text == "Detroit Lions":
        #save the result
        target = result
        #exit the loop
        break
#click the element using execute_script()
driver.execute_script("arguments[0].click()", target)
#pause so we can view the screen
sleep(5)
driver.quit()

Issue #4: Form Not Submitting​

Sometimes we can fill a form out in its entirety, but it still won't submit. This actually happens quite often when filling forms manually as well. In this case, first double check your scraper to make sure that your fields are getting filled correctly.

In the event your fields are getting filled correctly, try different methods of submission that we went over earlier in this article: Keys.ENTER, driver.execute_script() ...etc.

Let's use try and except to attempt different ways of clicking our result:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
from time import sleep
#define our url
url = "https://espn.com"
#open Chrome
driver = webdriver.Chrome()
#set implicit wait to 10 seconds
driver.implicitly_wait(10)
#create an ActionChains object
action = ActionChains(driver)
#navigate to the url
driver.get(url)
#find the search button
search_button = driver.find_element(By.ID, "global-search-trigger")
#click it so the search bar opens
search_button.click()
#find the search bar after waiting implicitly for it to open
search_bar = driver.find_element(By.ID, "global-search-input")
#send input to the search bar
search_bar.send_keys("detroit")
#find the search results and return them as a list
results = driver.find_elements(By.CLASS_NAME, "search_results__label")
#iterate through the results
for result in results:
    #if we've found the result we want
    if result.text == "Detroit Lions":
        #save the result
        target = result
        #exit the loop
        break
try:
    target.send_keys()
except:
    print("failed to send keys..scrolling to item!")
    #move the cursor to the target and click it
    action.move_to_element(target)\
        .click()\
        .perform()
#pause so we can view the screen
sleep(5)
driver.quit()

Once again our code example is basically the same, here are the differences:

• First, we try to interact with target by using the enter key
• If we fail when using the enter key, we handle the exception with an except statement that then moves to the target and clicks on it

Issue #5: Handling Alerts and Pop-ups​

We see pop-ups all the time on the web... especially when submitting forms. Selenium has a builtin alert methods such as dismiss() and accept() but these methods typically don't work.

More often than not, you will need to know the proper locators on the pop-ups or modals because they are often custom made and the developer wanted people to have to manually click to dismiss them.

It is best practice to locate your accept, dismiss, or close button using proper waiting techniques and then click() the proper button to handle the modal.

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
#open Chrome
driver = webdriver.Chrome()
#set implicit wait
driver.implicitly_wait(10)
#navigate to the url
driver.get("https://www.hubspot.com")
#find the "accept" button
modal = driver.find_element(By.ID, "hs-eu-confirmation-button")
#click the button
modal.click()
#pause so we can see whats happening
sleep(2)
driver.quit()

In the code above, we:

• Open Chrome with webdriver.Chrome()
• Set an implicit wait of 10 seconds
• Navigate to the website with driver.get()
• Find the "accept" button with driver.find_element()
• click() the accept button

Issue #6: Dynamic Content Issues​

As has been mentioned previously in this guide, quite often, we need to wait for items to properly appear. Content that changes on the screen is called dynamic content. When dealing with dynamic content we need to properly use either implicit or explicit waits for objects to appear.

• When we use an implicit wait, we set a default timeout for any object to appear on the screen, as long as the object appears within the waiting period, our code can interact with it
• When using an explicit wait, we typically want to use it in combination with ExpectedConditions so that webdriver can wait explicitly until the element appears on screen

The example below finds and closes the same pop-up we closed earlier, but this time we do it with an explicit wait.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from time import sleep
#open Chrome
driver = webdriver.Chrome()
#navigate to the url
driver.get("https://www.hubspot.com")
#find the "accept" button
modal = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "hs-eu-confirmation-button")))
#click the button
modal.click()
#pause so we can see whats happening
sleep(2)
driver.quit()

The differences you should notice from implicit wait code we've used throughout this article:

• WebDriverWait(driver, 10) sets an explicit wait of 10 seconds
• until(EC.presence_of_element_located((By.ID, "hs-eu-confirmation-button"))) tells Selenium to wait until our specific element has been located

Issue #7: Iframe Handling​

While <iframe> elements aren't as commonly used as they once were do to security reasons, sometimes they still get embedded into sites. To handle an iframe, we first use find_element() to find our iframe element.

Once we've located it, we can use driver.switch_to.frame(name_of_iframe) to interact with that specific element. To find elements from within the iframe, we can simply use find_element() or find_elements() just like we would when interacting with the native webpage.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
#open Chrome
driver = webdriver.Chrome()
#set implicit wait
driver.implicitly_wait(10)
#navigate to the site
driver.get("https://iframetester.com")
#find the search bar
url_search = driver.find_element(By.ID, "url-search")
#enter a url into the search bar to create an iframe
url_search.send_keys("https://quotes.toscrape.com")
#submit with the enter key
url_search.send_keys(Keys.ENTER)
#find the iframe
iframe = driver.find_element(By.ID, "iframe-window")
#switch to the frame
driver.switch_to.frame(iframe)
#find the first link
link = driver.find_element(By.TAG_NAME, "a")
#print the text of the link
print(link.text)
driver.quit()

The code above executes just like any other Selenium script that we've used previously in this article but with one main difference:

• driver.switch_to.frame(iframe) switches the focus of driver to the iframe

Issue #8: Browser Compatibility Issues​

These issues are actually far more uncommon than they used to be. If you believe you might be having compatibility issues, it is best to make sure that you have the latest browser driver installed. If Chromedriver is up to date, and you're still running into issues with this, try Geckodriver and Firefox. If you started out on Gecko with Firefox (and it is up to date) try with Chromedriver and Chrome.

Conclusion​

You now know how to properly submit form information with Selenium. Feel free to explore your new superpower in the open web an see what you can do! You can automate actual form filling, or you could even build your own chatbot that runs on the send_keys() and submit methods that we discussed in this article!

If you'd like to learn more about Selenium, take a look at the Selenium Documentation, they small code examples for Selenium in many different programming languages such as Python, Java, C#, Ruby, JavaScript, and Kotlin.

More Web Scraping Guides​

Want to know more about scraping in general but not sure where to start? Take a look at some of the ScrapeOps Selenium Web Scraping Playbook guides!

The playbook contains a plethora of useful material like these below:

• The Python Selenium Guide - Web Scraping With Selenium
• Using Proxies With Python Selenium
• Selenium Undetected Chromedriver - Bypass Anti-Bots With Ease