Python Selenium Guide - Waiting For Page or Element To Load

One of the key challenges in Selenium automation is managing the timing differences between script execution and the varying speeds at which web pages load.

It is considered best practice to always use one of Selenium's builtin waiting functions to wait for objects to load.

In this Selenium guide, we focus on the nuanced strategies of waiting for pages or elements to load and covering:

Why Do We Care About Page Load in Selenium?

When accessing specific data on a webpage, you will often need to wait for specific elements to load. In order to wait efficiently, there are many different methods of waiting and each one has its own use cases.

• Filling forms: Quite often, in order to interact with a page, you might need to fill a form and submit information. After submitting your information, the page itself might change or pop-ups may appear that you need to deal with.

• Pop-ups & modals: Knowing how to properly wait will improve how you handle pop-ups and modals and gives you a sure fire way to deal with them in a fraction of a second during runtime.

• Waiting for a specific element: Sometimes you don't need to wait for an entire page to load and all you need is a specific element. When waiting properly, you can handle whichever element you need to and your code will get on to the next thing it needs to do.

• Resource mangement: Using proper waits also can also help with resource management. When your scraper isn't waiting for unneccessary content to load, it's not wasting the time and energy of loading them.

• Avoid detection: Knowing how to wait properly can also help your scraper avoid detection. When using the right wait conditions and waiting for the right content to appear, your scraper can act much more like a human.

Installing Selenium

First, we need to have Selenium and a Webdriver installed on our machine. While Selenium supports many different browsers, we'll be using Chromedriver for this tutorial.

You can check your version of Chrome with the following command:

google-chrome --version

The output you receive from running the command should look like this:

Google Chrome 119.0.6045.123

Once you know your version number, you can head over here to find the driver matching your version.

Once you've got your browser and Webdriver installed, you're all set to install Selenium.

pip install selenium

You can test your installation by adding the following code to a Python script and then running the script.

from selenium import webdriver
from time import sleep
#create an instance of chrome
driver = webdriver.Chrome()
#navigate to the page
driver.get("https://quotes.toscrape.com")
#sleep for a few seconds so we can look at the page
sleep(10)
driver.quit()

This code is quite simple, it does the following:

• Opens an instance of Chrome with webdriver.Chrome()
• Navigates to https://quotes.toscrape.com
• Waits for 10 seconds with sleep(10) so you can view the page
• Closes the browser with driver.quit() when we're don looking at the page

If the sample script ran through, congratulations! You're now ready to use Selenium.

Common Ways to Wait for a Page or Element to Load in Selenium

The most common practices when waiting for a page to load are:

Each of these methods is appropriate in different circumstances and it is up to you to decide when to use each one.

This section will dive deeper into each of these methods and show simple code examples of how to use them.

Implicit Waits

Implicit waits constantly check the DOM (Document Object Model) for an arbitrary amount of time to see if an element exists on a page.

Implicit waits are great for setting a default timeout to wait for content to load. If the element doesn't appear within the set time, an exception will be thrown.

In the code below, we use the implicitly_wait() method:

from selenium import webdriver
from selenium.webdriver.common.by import By
#save our url as a variable
url = "https://www.amazon.com"
#create a webdriver instance
driver = webdriver.Chrome()
#set implicit wait to 10 seconds
driver.implicitly_wait(10)
#navigate to the url
driver.get(url)
#css selector of the login modal
css_selector = ".nav-signin-tooltip-footer"
#find the login modal using its css selector
login_modal = driver.find_element(By.CSS_SELECTOR, css_selector)
#print the text of the modal
print(login_modal.text)
driver.quit()

In the code above, we do the following:

• Save our url as a variable
• Open a browser instance with webdriver.Chrome()
• Set our implicit wait to 10 seconds with driver.implicitly_wait(10)
• Save the CSS Selector of our login modal as a variable
• Find the element with driver.find_element(By.CSS_SELECTOR, css_selector)
• Print the text of login_modal to the terminal

Explicit Waits:

Explicit waits are achieved by using the WebDriverWait and ExpectedConditions methods together.

This combination provides a powerful and effective solution for when waiting not only for custom elements to load, but to wait for these elements to meet certain conditions.

We can wait for an element to be located with visibility_of_element_located().

• If we want an element to be clickable, we can use the condition element_to_be_clickable().
• When we need an element to display a specific string of text, we can use the text_to_be_present_in_element() method.

Explicit waits allow for us to synchronize with page elements precisely and this gives us more efficiency with time and resources.

Let's perform the same task we did with Implicit, but this time, we'll use an Explicit Wait:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
#save our url as a variable
url = "https://www.amazon.com"
#create a webdriver instance
driver = webdriver.Chrome()
#navigate to the url
driver.get(url)
#css selector of the login modal
css_selector = ".nav-signin-tooltip-footer"

#find the login modal using its css selector
try:
    login_modal = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.CSS_SELECTOR, css_selector))
            )
    
#print the text of the modal
finally:
    print(login_modal.text)
    driver.quit()

In the code above, we do the following:

• Save our url as a variable
• Open a browser instance with webdriver.Chrome()
• Save the CSS Selector of our login modal as a variable
• Create a try clause to find our login modal
• Inside the try clause, we set our Explicit to 10 seconds with WebDriverWait(driver, 10)
• Use .until(EC.presence_of_element_located() to return our login modal once it has been located
• End our try statement with the finally keyword
• Print the text of the modal to the terminal

Implicit waits, set globally for the entire WebDriver instance, are well-suited for scenarios where a standardized wait time can be applied universally to handle varying load times of elements.

On the other hand, explicit waits, applied using WebDriverWait and ExpectedConditions, and allow for precise control over specific elements or conditions, making them ideal for scenarios where different elements may have different load times, or when waiting for dynamic content that changes asynchronously.

Fluent Waits:

Fluent Waits provide us with a more flexible way to perform an Explicit Wait. With Fluent Waits, we can set a maximum time to wait for a condition, as well as how frequently we would like to check for the condition.

Selenium Python does not contain a Fluent Wait class, but we can achieve the same functionality by passing kwargs (keyword arguments) into the methods we would use to for Explicit Waits.

Let's do the same thing once again, but this time, let's do it with a Fluent Wait:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
#save our url as a variable
url = "https://www.amazon.com"
#create a webdriver instance
driver = webdriver.Chrome()
#navigate to the url
driver.get(url)
#css selector of the login modal
css_selector = ".nav-signin-tooltip-footer"
#set our wait conditions
wait = WebDriverWait(driver, timeout=10, poll_frequency=1)
#find the login modal using its css selector
login_modal = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, css_selector)))    
#print the text of the modal
print(login_modal.text)
driver.quit()

In the code above, we do the following:

• Save our url as a variable
• Open a browser instance with webdriver.Chrome()
• Navigate to the url with driver.get()
• Save our CSS Selector as a variable
• Set our wait conditions with wait = WebDriverWait(driver, timeout=10, poll_frequency=1)
• timeout=10 tells Selenium that we want to wait up to 10 seconds for this element to appear on the screen
• poll_frequency=1 tells Selenium that we want to poll the DOM every second to see if our element has appeared yet

Time.sleep():

In Python, we would can time.sleep() if we simply want to pause the execution of a script for a specified amount of time.

In production, it is not recommended to use this method unless you actually need to pause the script for one reason or another.

Reasons to use sleep() are quite limited and really should only be used when you either need to view the page, or manually interact with it for some reason (solving a captcha).

Next, let's try to accomplish the same functionality with sleep():

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
#create an instance of chrome
driver = webdriver.Chrome()
#navigate to the page
driver.get("https://www.amazon.com")
#sleep for a few seconds so we can look at the page
sleep(3)
#save the css selector as a variable
css_selector = ".nav-signin-tooltip-footer"
#find our login modal
login_modal = driver.find_element(By.CSS_SELECTOR, css_selector)
#print the text of the modal
print(login_modal.text)
driver.quit()

In the code above, we do the following:

• Open a browser instance with webdriver.Chrome()
• Navigate to our url with driver.get()
• Pause the execution of our code for 3 seconds with sleep(3)
• Find our login modal with driver.find_element()
• Print the modal text to the terminal

Page Load Timeout:

Sometimes, we need to set a time limit for a page to load. This can be useful when crawling multiple pages.

With set_page_load_timeout() we can tell Selenium to wait an explicit amount of time for a page to load.

If the page doesn't load before the timeout, an exception will be thrown. When you know how to handle exceptions in Python, you can tell Selenium to give up and move onto the next portion of your code.

Let's now wait for the page with a set_page_load_timeout():

from selenium import webdriver
from selenium.webdriver.common.by import By
#save our url as a variable
url = "https://www.amazon.com"
#craete a webdriver instance
driver = webdriver.Chrome()
#set page load timeout to 10 seconds
driver.set_page_load_timeout(10)
#navigate to the url
driver.get(url)
#css selector of the login modal
css_selector = ".nav-signin-tooltip-footer"
#find the login modal using its css selector
login_modal = driver.find_element(By.CSS_SELECTOR, css_selector)
#print the text of the modal
print(login_modal.text)

In this iteration of our login modal example, we do the following:

• Open a browser instance with webdriver.Chrome()
• Save our url as a variable
• Set the default timeout for loading the page to 10 seconds with driver.set_page_load_timeout(10)
• Navigate to our url with driver.get()
• Wait up to 10 seconds for the page to load
• Find our login modal with driver.find_element(By.CSS_SELECTOR, css_selector)
• Print the element text to the terminal

Custom Wait Conditions:

When dealing with objects that that need to meet a certain criteria, it can be very beneficial to have ExpectedCondition builtin to your custom code.

Let's create a custom function that looks for a CSS Selector and returns it.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

#create a special function that looks for the modal specifically
def wait_for_modal(driver, url, css_selector):
  wait = WebDriverWait(driver, 10)
  driver.get(url)
  element = wait.until(EC.presence_of_element_located((
    By.CSS_SELECTOR, css_selector
  )))
  return element
#create a browser instance
driver = webdriver.Chrome()
#save our url as a variable
url = "https://www.amazon.com"
#save our selector as a variable
css_selector = ".nav-signin-tooltip-footer"
#call the custom function to find the modal
login_modal = wait_for_modal(driver, url, css_selector)
print(login_modal.text)

The code above (as you've probably noticed) is very similar to our examples from before. Let's breakdown how this code works:

• First we create a function, wait_for_modal() which takes our driver, url, and css_selector as arguments
• This function:
  • Sets WebDriverWait to 10 seconds
  • Navigates to our url argument
  • Waits for our ExpectedCondition until it finds our element
  • Returns the element variable
• Once we've defined wait_for_modal(), we open an instance of Chrome with webdriver.Chrome()
• Save our url and CSS selector as variables
• Pass our driver, url, and css_selector into the function to return the value
• Print the returned value, login_modal to the terminal

JavaScript/Ajax Waits:

When waiting for JavaScript to load dynamic content, we can use set_script_timeout() in order to wait for JavaScript elements (such as the login modal) to load on the page.

The code below accomplishes our same goal (printing the modal text to the terminal) by using set_script_timeout() to wait for our JavaScript content to load.

from selenium import webdriver
from selenium.webdriver.common.by import By
#open chrome
driver = webdriver.Chrome()
#set the script timeout to 3 seconds
driver.set_script_timeout(3)
#save our url as a variable
url = "https://www.amazon.com"
#navigate to the url
driver.get(url)
#save our selector as a variable
css_selector = ".nav-signin-tooltip-footer"
#find the modal
login_modal = driver.find_element(By.CSS_SELECTOR, css_selector)
#print the text of the modal
print(login_modal.text)

In the example above, we do the following:

• Open a browser with webdriver.Chrome()
• Set our script timeout to 10 seconds with driver.set_script_timeout(10) (this tells Selenium to wait 10 seconds for all JavaScript content to load)
• Save our url as a variable
• Save our CSS Selector as a variable
• Find our modal with driver.find_element(By.CSS_SELECTOR, css_selector)
• Print the modal text to the terminal

How to Choose the Appropriate Waiting Strategy

When waiting for elements to load on the page, it's very important to use the right strategy and to always try to choose the right tool for the job.

The following methods: implicitly_wait(), set_page_load_timeout() and set_script_timeout() are all good options when you are waiting for multiple dynamic elements to load on a webpage.

One might choose an Explicit or Fluent Wait when they're waiting for specific elements to load on the page, and then interact with them. These strategies are perfect for filling forms, clicking a submit button, and waiting for new content to appear on the screen.

Custom Wait conditions are ideal when waiting on the page involves complex logic that would be messy or difficult to code using just one of the strategies above. Custom Waits are a powerful tool to meet the needs of special usecases.

The 3 Wait Common Usecases In Selenium

The three most common times you'll have to perform Waits in Selenium are:

1. Waiting for a page to load
2. Waiting for an element to load on a page
3. Accepting cookie consent

In the following sections, we'll perform all three of these actions.

Wait For Page To Load

First, we'll simply navigate to a page and wait for the page to load.

from selenium import webdriver
#save our url as a variable
url = "https://www.mcdonalds.com"
#craete a webdriver instance
driver = webdriver.Chrome()
#set page load timeout to 10 seconds
driver.set_page_load_timeout(10)
#navigate to the url
driver.get(url)
#tell us the page finished loading
print("Page finished loading!")

Wait For Page Element to Load

Quite often, you'll need to wait for a specific element to load on the page when scraping.

Let's wait for a cookie modal to load and print its contents to the terminal:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
#save the url as a variable
url = "https://www.mcdonalds.com"
#open a browser instance
driver = webdriver.Chrome()
#Define how to wait
wait = WebDriverWait(driver, timeout=20)
#navigate to the url
driver.get(url)
#wait for the x button to appear
x_button = wait.until(EC.presence_of_element_located(
    (By.ID, "onetrust-policy-text")))
#print the text of the x button
print(x_button.text)
driver.quit()

Accepting A Cookie Consent & Scraping Data

Let's use Selenium to wait for a cookie consent modal to appear and then close the modal. McDonald's uses cookies by default, so by viewing their site, you are already giving consent.

While you they give you the option to edit this consent, we're not interested in editing our consent, we just need to close the pop-up to accept the cookies.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
#save the url as a variable
url = "https://www.mcdonalds.com"
#save the css selector of the cookie button as a variable
cookie_selector = ".onetrust-close-btn-ui"
#open a browser instance
driver = webdriver.Chrome()
#navigate to the url
driver.get(url)
#Define how to wait
wait = WebDriverWait(driver, timeout=20)
#wait for the x button to appear
x_button = wait.until(EC.presence_of_element_located(
    (By.CSS_SELECTOR, cookie_selector)))
#click the x button
x_button.click()
driver.quit()

In the example above, we:

• Save our url and CSS Selector as variables
• Open a browser with webdriver.Chrome()
• Navigate to the site
• Define our wait conditions with WebDriverWait()
• Wait until the x_button is present and return it as a variable
• click() the button

Best Practices for Waiting in Selenium Automation

When waiting in Selenium, there are many different ways to do it: Implicitly, Explicitly, Fluently... Most importantly, one can avoid the shortcomings of hard-coded delays when using them.

When handling dynamic elements, you can use many (or sometimes any) of the strategies we've gone through in this article. Always make sure to set your timeouts wait long enough for your content to load.

Conclusion

Congratulations! You've made it to the end of the article. You now understand the basics of a Selenium script and you now can make informed decisions about how to wait for pages and elements to load in Selenium.

Interested in learning more about scraping in Python or Selenium? Take a look at the links below:

• ScrapeOps Python Playbook
• Selenium Documentation

More Cool Articles

Want to learn more but don't know where to start? Choose one of these articles:

• Selenium vs Scrapy
• Python Pyppeteer Guide
• Python Requests Web Scraping Guide