Playwright Guide: Waiting For Page or Element To Load

Before diving into web scraping tasks using Playwright, it's essential to understand the time required for a web browser to fully load and display a website. Without this understanding, you may face issues such as premature execution of your web scraping script, resulting in incomplete screenshots and other problems.

This guide will explore techniques for ensuring the browser has fully loaded the page, including fonts, styles, and images, and ensuring that specific DOM elements have appeared or particular API calls have been fetched before proceeding with further web scraping or automation tasks.

• Why Do We Care About Page Load in Playwright?
• How To Wait For Page To Load With Playwright
• Methods For Waiting for a Page or Element to Load in Playwright
• Common Situations for Waiting for a Page or Element to Load in Playwright
• Combining Waiting Strategies
• Best Practices for Waiting in Playwright
• Conclusion
• More Web Scraping Tutorials

Need help scraping the web?

Then check out ScrapeOps, the complete toolkit for web scraping.

Proxy Manager

Scraper Monitoring

Job Scheduling

Why Do We Care About Page Load in Playwright?​

Many websites exhibit dynamic behavior, continuously loading new content asynchronously, with elements appearing and disappearing in the process. Incomplete loading may lead to misinterpretation of data, causing inaccuracies in the extracted information.

Automated scripts may execute prematurely or cause errors due to elements that are not fully loaded yet or have been changed dynamically.

The following detailed points elaborate on why we care about page load in Playwright:

1. Filling Forms: Efficiently waiting for forms to load is critical for accurate input and submission. Playwright provides strategies to synchronize with form elements, ensuring a seamless automation process.

2. Pop-ups & Modals: Waiting for the appearance of pop-ups and modals is essential for interacting with these dynamic elements. Playwright offers specialized methods to handle these scenarios effectively.

3. Waiting for a Specific Element: In scenarios where specific elements are pivotal to the automation process, Playwright provides methods to precisely wait for their full loading, preventing premature interactions.

4. Resource Management: Efficient resource management is crucial for optimizing page load times. Playwright equips users with tools to manage resources effectively, ensuring a streamlined automation experience.

5. Avoiding Detection: To navigate web scraping without detection, Playwright provides methods to wait intelligently, minimizing the risk of being flagged by anti-bot mechanisms.

How To Wait For Page To Load With Playwright​

There are several methods available to wait for a page to load, each serving a specific purpose. Let's delve into the various options:

We'll delve into the specifics of each waiting method shortly. For now, let's see a simple example demonstrating a 15-second delay using page.waitForTimeout().

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: true });

  const page = await browser.newPage();
  await page.goto("https://www.reddit.com/");
  await page.waitForTimeout(15000);
  await page.screenshot({ path: 'reddit.png' });
  
  await browser.close(); 
})()

In this example, page.waitForTimeout(15000) pauses the execution for 15 seconds, providing time for the page to load.

Although static delays are generally not recommended for waiting for page loads, they can be useful in specific scenarios. As we explore other waiting methods, we'll find more dynamic and reliable ways to ensure the page is fully loaded before proceeding with further actions.

Methods For Waiting for a Page or Element to Load in Playwright​

Now we will explore all the methods that Playwright provides to wait for page load, in detail:

goto Method Options​

The page.goto(url, options) method stands out as the most valuable waiting strategy. While primarily employed for navigating to a web page, it proves versatile by accommodating various options to pause for specified durations or await specific events before progressing to the subsequent actions.

Two pivotal options frequently employed in the context of page or element loading are waitUtil and timeout.

• waitUntil: The waitUntil option in the page.goto(url, options) method can be configured with four WaitForOptions types:

  • load,
  • domcontentloaded,
  • networkidle, and
  • commit.

  More than one waitUnil options can be employed by passing them as an array.

  • domcontentloaded:
    • This option instructs Playwright to wait until the DOMContentLoaded event has fired.
    • This event occurs when the initial HTML document has been completely loaded and parsed.
    • It indicates that the DOM tree is available to the browser, excluding external resources like stylesheets and images. Let's see an example:

   const { chromium } = require('playwright');

   (async () => {
     const browser = await chromium.launch({ headless: true });

     const page = await browser.newPage();
     await page.goto("https://finance.yahoo.com", {  waitUntil: "domcontentloaded" });
     await page.screenshot({ path: `yahoo-domcontentloaded.png` });

     await browser.close(); 
   })()

Observing the page, it's evident that certain elements, such as advertisement banners and specific styles such as graphs, are not rendered correctly.

This discrepancy is expected, as the domcontentloaded event guarantees only the proper loading and parsing of HTML, leaving out assurance for the complete rendering of external resources like images, fonts, and certain styles.

• load (Default):
  • If no waitUntil option is provided, the default behavior is to wait until the load event is fired.
  • The load event signifies that the entire page, including the DOM tree, CSS styles, fonts, and images, has finished loading.
  • This is a more comprehensive wait condition, encompassing all resources associated with the page.
  Let's modify our earlier example to utilize the load event instead of domcontentloaded and observe if this results in the correct rendering of advertisement banners:

   const page = await browser.newPage();
   await page.goto("https://finance.yahoo.com", {  waitUntil: "load" });
   await page.screenshot({ path: `yahoo-load.png` });

Success! It's evident that employing the load event provides greater assurance that the page has been fully rendered, encompassing images, styles, and fonts. This completeness was not entirely guaranteed when relying solely on domcontentloaded.

• networkidle: 'networkidle' - DISCOURAGED Wait until there are no network connections for at least 500 ms. Playwright doesn't recommend using this method, instead rely on web assertions to assess readiness instead.

• timeout: This option specifies the maximum navigation time in milliseconds. If the navigation events (like load, domcontentloaded, etc.) are not completed within this time, the page.goto() method will throw an error. It sets a time limit for the entire page navigation process.

   await page.goto("https://finance.yahoo.com", {  waitUntil: "load", timeout: 60000 });
   // Set timeout to 1 minute

The timeout option in the page.goto(url, options) method and the page.waitForTimeout() function serve different purposes. The waitForTimeout() method is not directly related to page navigation or waiting for specific events on the page. It simply pauses the execution of the script for the specified duration.

TimeoutError in Playwright​

TimeoutError is a significant issue when using Playwright. For instance, if your internet connection is slow while executing a script, the default timeout of 30 seconds may elapse before the load event fires.

To handle this, you can set a longer navigation timeout, such as 100000 milliseconds. This ensures the script waits for the load event before taking a screenshot and closing the browser.

Here’s how you can implement this workaround:

await page.setDefaultNavigationTimeout(10000000);

By adjusting the default navigation timeout, the script will wait for the page to load fully, avoiding premature timeout errors, and then proceed to take a screenshot and close the browser.

Moreover, you can also use a try-catch block to implement retries with progressively increasing timeout values. This approach helps manage varying network conditions more effectively:

const attemptNavigation = async (page, url, timeout) => {
  try {
    await page.goto(url, { timeout });
    // Take a screenshot or perform other actions
    await page.screenshot({ path: 'screenshot.png' });
 } catch (error) {
    if (error.name === 'TimeoutError') {
      console.log(`TimeoutError encountered. Retrying with increased timeout...`);
      timeout += 30000; // Increase timeout by 30 seconds
      await attemptNavigation(page, url, timeout);
 } else {
      throw error;
 }
 }
};

(async () => {
  const browser = await playwright.chromium.launch();
  const page = await browser.newPage();
  const url = 'https://example.com';

  await attemptNavigation(page, url, 30000); // Start with 30 seconds timeout

  await browser.close();
})();

Locators And Auto-Waiting​

Locators are the central piece of Playwright's auto-waiting and retry-ability features. In essence, locators provide a way to find elements on the page at any given moment. A locator can be created using the page.locator() method. For instance, you can select an element using its CSS selector with locators like this:

await page.locator('css=button.primary);

But how do locators enable auto-waiting? Let's understand this with an example. If you want to click on the button selected using the locator, you can simply call the click() method on it.

Playwright will perform auto-waiting for the button to be present in the DOM, visible, and enabled before performing the click action:

await page.locator('css=button.primary').click();

In this example, Playwright’s auto-waiting mechanism ensures that the click operation is performed only when the button is in a state that allows it to be clicked. This includes waiting for the button to:

• Be attached to the DOM: The element must be present in the document.
• Be visible: The element must not be hidden by CSS styles.
• Be enabled: The element must not be disabled.

These conditions are automatically checked by Playwright before performing actions like click() or fill(), ensuring more reliable and stable scripts. This built-in auto-waiting mechanism reduces the need for manual waits and retries, making your tests more concise and easier to maintain.

Additionally, Playwright locators provide more advanced features, such as chaining and filtering, to target elements more precisely. For example, you can chain locators to narrow down your selection:

await page.locator('div.container').locator('button.primary').click();

In this case, Playwright will first locate the div element with the class .container and then find the <button> element with the class .primary within that div.

Understanding and effectively using locators and their auto-waiting capabilities is crucial for writing robust Playwright scripts that interact with web elements reliably.

Wait for Selector​

Up to this point, our discussion has revolved around page loading concerning the capture of viewport or full-sized screenshots. However, there are instances when your focus is solely on a specific element, such as a crypto candlestick chart or a Power BI dashboard.

In these cases, rather than waiting for the entire page to load along with all associated events and API calls, a more efficient approach is to wait for that particular element to load and render. This not only saves time but also ensures the proper rendering of the targeted element.

Playwright facilitates this process through its locator.waitFor() method. Notably, this method includes a {visible: true} flag, instructing Playwright to wait until the element is present in the DOM tree and does not possess CSS properties like { display: none } or { visibility: hidden }.

Let's see an example where we await the loading of a crypto graph, capturing its screenshot using the waitFor() method:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();
  await page.goto("https://www.tradingview.com/markets/cryptocurrencies/");

  const element = page.locator("css=.chart-jDMZqyge");
  element.waitFor({ visible: true });

  await element.screenshot({ path: `crypto-graph.png` });
  
  await browser.close(); 
})()

Note that I have set { headless: false } so you can observe on your screen that our script takes a screenshot as soon as the crypto-graph appears, then quickly closes the browser.

However, in production code, you should always prefer { headless: true } to conserve resources. Running in headless mode allows for faster execution and lower resource consumption, making it ideal for automated scripts running in production environments.

Wait for Page Navigation​

Page navigation involves events such as clicking on links, submitting forms, or any action that triggers a change in the page's URL. In Playwright, await page.waitForURL('**/action.php'); is a valuable method designed to handle scenarios where a script needs to wait for the completion of page navigation before proceeding with further actions.

Additionally, this method also shares the same timeout and waitUntil options like page.goto(), offering same functionality but specifically for the new page being navigated to.

Use Cases:​

• Form Submissions: When automating form submissions, waiting for navigation ensures that subsequent interactions are performed on the fully loaded page, preventing premature actions.

• Link Clicks: After triggering a click on a link, waiting for navigation becomes crucial to guarantee that the new page has fully loaded before executing additional steps.

• Single Page Applications (SPAs): In SPAs where page content dynamically changes without a full page reload, await page.waitForURL('**/action.php'); synchronizes script execution with the application's state.

Here's an example wherein we navigate to the login page, input the username and password, and subsequently await the navigation and loading of the next page using the await page.waitForURL('**/logged-in-successfully'); method:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();
  await page.goto("https://practicetestautomation.com/practice-test-login/");
  await page.locator('css=#username').fill('student');
  await page.locator('css=#password').fill('Password123');

  await Promise.all([
    page.click('#submit'),
    page.waitForURL('**/logged-in-successfully/', { 
      waitUntil: "load" 
 }),
 ]);

  await page.screenshot({ path: `login.png` });
  await browser.close();
})()

The use of Promise.all() concurrently handles both statements, avoiding race conditions and ensuring a smooth synchronization with the navigation events.

Wait for Timeout​

To introduce a pause in script execution, allowing sufficient time for proper page loading, the page.waitForTimeout() method was previously employed. However, its use is DISCOURAGED in recent versions of Playwright.

The alternative is to use setTimeout(), which serves the same purpose. Here is an example, where the setTimeout() function ensures a delay of 15 seconds before capturing a screenshot on the Twitter page.

await page.goto("https://example.com");
await new Promise(resolve => setTimeout(resolve, 15000));
// Further Actions

Wait for Function​

Playwright's page.waitForFunction() is designed to pause script execution until a specified function completes its evaluation within the page's context.

This functionality proves valuable in situations where custom script evaluation is necessary for waiting during page loading, providing a more tailored approach than relying solely on Playwright's built-in methods and events.

Let's explore an example where we wait for a specific DOM element to become visible on the screen before capturing its screenshot:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();
  await page.goto("https://www.tradingview.com/markets/cryptocurrencies/");
  
  await page.waitForFunction(() => {
    const element = document.querySelector('.chart-jDMZqyge');
    return element && element.offsetHeight > 0 && element.offsetWidth > 0;
 });
  
  await page.screenshot({ path: `crypto-market.png` });
  
  await browser.close(); 
})()

Wait for XPath​

In Playwright, page.waitFor() and page.locator('xpath=//') are methods designed to wait for the presence of an XPath expression on the page before proceeding with further actions. It allows you to wait for the presence of a specific XPath expression, be it an element or text content.

Here is an example:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();
  await page.goto("https://example.com")
  
  await page.locator('xpath=//html/body/div/p[2]/a').click();
  await page.waitForURL('https://www.iana.org/help/example-domains');
  
  const text = await page.locator('xpath=//html/body/div/article/main/div/p[1]').textContent();
  console.log(text);

  await browser.close();
})();

// As described in RFC 2606 and RFC 6761, a
// number of domains such as example.com and example.org are maintained
// for documentation purposes. These domains may be used as illustrative
// examples in documents without prior coordination with us. They are not
// available for registration or transfer.

Common Situations for Waiting for a Page or Element to Load in Playwright​

Now that we've covered the most common waiting methods, let's delve into some typical scenarios where applying a combination of these methods and options can yield optimal solutions to address challenges related to page and element loading:

1. Waiting for a Specific Element to be Visible on the Page​

In certain scenarios, capturing screenshots of specific DOM elements rather than the entire page is crucial:

1. Product Thumbnails in E-commerce: In an e-commerce site, you might want to capture screenshots of individual product thumbnails for quality assurance or visual documentation.

2. Form Submissions: After submitting a form, you may want to take a screenshot of a specific confirmation message or result element to verify the success of the form submission.

3. Dashboard Widgets: In a dashboard application, you might want to capture screenshots of individual widgets or components to monitor their appearance and updates independently.

4. User Profile Sections: In a social media platform, you might want to capture screenshots of specific user profile sections, such as profile pictures or bio information.

5. Graphs and Charts: When dealing with data visualization, capturing screenshots of specific graphs or charts allows for detailed inspection and monitoring of data trends.

The locator.waitFor() method with { visible: true } flag proves instrumental in waiting for the presence and visibility of a specific element before proceeding with taking a screenshot.

2. Waiting for Page to be Ready for Button Clicks and Form Submissions​

In scenarios where you need to interact with buttons or submit forms on a web page, it's crucial to ensure that the page has completed its navigation and is ready for the subsequent actions. page.waitForURL() with { waitUntil: load } proves invaluable in such cases, providing a means to pause the script until the page has fully loaded after a button click or form submission.

3. Waiting for Page Load Before Taking Screenshot​

When you inspect the network tab in your browser's DevTools, you'll notice two essential DOM events occurring during the page load: load and DOMContentLoaded, each timestamped.

1. The first event, DOMContentLoaded, occurs after the initial HTML has been loaded and parsed.

   await page.goto(url, {waitUntil: "domcontentloaded"});

2. The subsequent event, named load, transpires when additional elements like styles, fonts, and images have been fetched and integrated into the webpage.

   await page.goto(url, {waitUntil: "load"});

These events serve as metrics, indicating when the page has finished loading, providing insights into the time required for a specific page to complete its loading process.

When Load is not Enough​

Sometimes, simply waiting for the page load event is insufficient, especially if you need to ensure that certain network requests or specific elements are fully loaded before proceeding. While Playwright provides a networkidle option, its use is generally discouraged. Instead, you can employ more reliable methods to wait for essential elements or network activities to complete.

1. Wait for a Specific URL Using waitForURL: You can use the waitForURL method to wait for a particular URL to be fetched, indicating that most of the page content has loaded. This is particularly useful when you know a specific network request signifies the completion of significant loading processes.

   // Example: Wait for a specific URL
   await page.waitForURL('**/special-endpoint');

2. Wait for a Specific DOM Element Using waitFor: Another approach is to wait for a specific DOM element to become visible on the page, signaling that the page has mostly been rendered. The waitFor() method is ideal for this purpose.

   // Example: Wait for a specific DOM element
   const element = await page.locator('#special-element');
   element.waitFor({ visible: true });

3. Wait for a Specific Network Requests Using waitForResponse: Monitor specific network requests to ensure they are completed before proceeding. This can be particularly useful for ensuring that important API calls have finished.

   await page.waitForResponse(response => 
       response.url().includes('special-api-endpoint') && response.status() === 200
   );

4. Web Assertions: Use assertions to check the state of the page and ensure that it has reached the desired condition. This method is more reliable as it directly verifies the elements and their properties.

   await expect(page).toHaveText('#text-element', 'Expected Text');

By using these alternatives, you can create more reliable and efficient scripts that accurately determine when a page is ready for interaction.

4. Waiting for an API Call to Populate the Page Content​

In scenarios where you expect a website to be fully loaded only after a specific API request or response, Playwright's page.waitForRequest() and page.waitForResponse() methods can be employed.

These methods accept a URL, such as an API endpoint, or a predicate function. The predicate function allows you to evaluate specific expressions. For example, you can use this function to verify whether the desired data has been successfully received through the API request.

This approach is particularly useful when waiting for dynamic content to be populated on the page as a result of asynchronous API calls. By incorporating these Playwright methods, you can synchronize your script with the completion of API requests, ensuring that the page is fully loaded and ready for capturing screenshots.

Here are examples illustrating the use of these methods:

• waitForRequest(url): Wait for a particular request to take place.

   // Start waiting for request before clicking. Note no await.
   const requestPromise = page.waitForRequest('https://example.com/resource');
   await page.getByText('trigger request').click();
   const request = await requestPromise;

   // Alternative way with a predicate. Note no await.
   const requestPromise = page.waitForRequest(request =>
     request.url() === 'https://example.com' && request.method() === 'GET',
   );
   await page.getByText('trigger request').click();
   const request = await requestPromise;

• waitForResponse(url, callback): Awaits the response to be fetched by the browser following the initiation of a request.

   // Start waiting for response before clicking. Note no await.
   const responsePromise = page.waitForResponse('https://example.com/resource');
   await page.getByText('trigger response').click();
   const response = await responsePromise;

   // Alternative way with a predicate. Note no await.
   const responsePromise = page.waitForResponse(response =>
     response.url() === 'https://example.com' && response.status() === 200
    && response.request().method() === 'GET'
   );
   await page.getByText('trigger response').click();
   const response = await responsePromise;

Combining Waiting Strategies​

In the realm of web automation, achieving robust and reliable scripts often demands a thoughtful combination of waiting strategies. This is crucial to address diverse scenarios and ensure precise synchronization with the dynamic behaviors of web pages.

A strategic approach involves adapting to variable loading times, optimizing wait durations, and effectively handling timeouts and exceptions.

Best Practices for Waiting in Playwright​

When it comes to waiting in Playwright, there are several best practices to keep in mind:

1. Optimize Wait Times: Striking the appropriate balance in wait times is crucial to optimize automation efficiency. A useful strategy involves leveraging the browser's inspect tab to investigate the time duration required for page loading on a specific site. Additionally, the networkidle strategy, as discussed earlier, proves valuable for further optimizing waiting times during the automation process.

2. Avoid excessive waiting to improve efficiency: Waiting too long can slow down your automation and waste resources. To avoid this, you can use the page.setDefaultTimeout() method to set a maximum timeout for all wait methods.

   setDefaultNavigationTimeout() takes priority over setDefaultTimeout().

3. Handling exceptions: All the waiting methods we covered operate asynchronously and may encounter failures due to network issues or server-side errors. It is advisable to encapsulate these methods within try...catch blocks to handle exceptions. Here's a code example demonstrating robust error handling:

   try {
     await page.goto('https://example.com')
   } catch (error) {
     console.error('Navigation Unsuccessful!', error.message);
   }

4. Migrating from Puppeteer: This guide describes migration to Playwright from Puppeteer. The APIs have similarities, but Playwright offers much more possibilities for web testing and cross-browser automation.

Conclusion​

In the realm of web scraping and automation, the initial and crucial step involves waiting for page loading. Playwright stands out with its variety of methods and options, serving as an effective initial solution and waiting strategy before proceeding with subsequent automation tasks, such as navigating through pages and extracting data.

One of Playwright's standout features is its auto-waiting mechanism, which simplifies the process by automatically waiting for elements to be ready before performing actions. This, combined with the power of locators, ensures that your scripts are both robust and reliable.

However, it's important to recognize the inherent complexity of websites. Therefore, before diving into any web scraping or automation task, conducting a thorough investigation of the target website using the browser's inspect network activity tab in your DevTools is imperative.

While Playwright provides ample methods for most websites, there may be instances where creating a custom waiting strategy or function becomes necessary for optimal synchronization before executing automated tasks.

More Web Scraping Tutorials​

• Puppeteer Guide: Waiting For Page or Element To Load
• Playwright Guide: How To Take Screenshots
• NodeJS Playwright: Logging Into Websites

21: https://pptr.dev/# Playwright Guide: Waiting For Page or Element To Load

• Playwright Guide: Waiting For Page or Element To Load
  • Introduction
  • Why Do We Care About Page Load in Playwright?
  • How To Wait For Page To Load With Playwright
  • Methods For Waiting for a Page or Element to Load in Playwright
    • goto Method Options
    • TimeoutError in Playwright
    • Locators And Auto-Waiting
    • Wait for Selector
    • Wait for Page Navigation
      • Use Cases:
    • Wait for Timeout
    • Wait for Function
    • Wait for XPath
  • Common Situations for Waiting for a Page or Element to Load in Playwright
    • 1. Waiting for a Specific Element to be Visible on the Page
    • 2. Waiting for Page to be Ready for Button Clicks and Form Submissions
    • 3. Waiting for Page Load Before Taking Screenshot
      • When Load is not Enough
    • 4. Waiting for an API Call to Populate the Page Content
  • Combining Waiting Strategies
  • Best Practices for Waiting in Playwright
  • Conclusion
  • More Web Scraping Tutorials

Introduction​

Before diving into web scraping tasks using Playwright, it's essential to understand the time required for a web browser to fully load and display a website. Without this understanding, you may face issues such as premature execution of your web scraping script, resulting in incomplete screenshots and other problems. Relying on a fixed waiting period (e.g., two minutes) for the browser to resolve all HTTP(s) requests (including HTML, CSS, JS, fonts, and images) is both inefficient and site-dependent. Instead, Playwright offers more efficient waiting strategies, particularly auto-waiting, to address this issue.

This guide will explore techniques for ensuring the browser has fully loaded the page, including fonts, styles, and images, and ensuring that specific DOM elements have appeared or particular API calls have been fetched before proceeding with further web scraping or automation tasks.

Why Do We Care About Page Load in Playwright?​

Many websites exhibit dynamic behavior, continuously loading new content asynchronously, with elements appearing and disappearing in the process. Incomplete loading may lead to misinterpretation of data, causing inaccuracies in the extracted information. Automated scripts may execute prematurely or cause errors due to elements that are not fully loaded yet or have been changed dynamically. The following detailed points elaborate on why we care about page load in Playwright:

1. Filling Forms: Efficiently waiting for forms to load is critical for accurate input and submission. Playwright provides strategies to synchronize with form elements, ensuring a seamless automation process.

2. Pop-ups & Modals: Waiting for the appearance of pop-ups and modals is essential for interacting with these dynamic elements. Playwright offers specialized methods to handle these scenarios effectively.

3. Waiting for a Specific Element: In scenarios where specific elements are pivotal to the automation process, Playwright provides methods to precisely wait for their full loading, preventing premature interactions.

4. Resource Management: Efficient resource management is crucial for optimizing page load times. Playwright equips users with tools to manage resources effectively, ensuring a streamlined automation experience.

5. Avoiding Detection: To navigate web scraping without detection, Playwright provides methods to wait intelligently, minimizing the risk of being flagged by anti-bot mechanisms.

How To Wait For Page To Load With Playwright​

There are several methods available to wait for a page to load, each serving a specific purpose. Let's delve into the various options:

We'll delve into the specifics of each waiting method shortly. For now, let's see a simple example demonstrating a 15-second delay using page.waitForTimeout().

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: true });

  const page = await browser.newPage();
  await page.goto("https://www.reddit.com/");
  await page.waitForTimeout(15000);
  await page.screenshot({ path: 'reddit.png' });
  
  await browser.close(); 
})()

In this example, page.waitForTimeout(15000) pauses the execution for 15 seconds, providing time for the page to load.

Although static delays are generally not recommended for waiting for page loads, they can be useful in specific scenarios. As we explore other waiting methods, we'll find more dynamic and reliable ways to ensure the page is fully loaded before proceeding with further actions.

Methods For Waiting for a Page or Element to Load in Playwright​

Now we will explore all the methods that Playwright provides to wait for page load, in detail:

goto Method Options​

The page.goto(url, options) method stands out as the most valuable waiting strategy. While primarily employed for navigating to a web page, it proves versatile by accommodating various options to pause for specified durations or await specific events before progressing to the subsequent actions. Two pivotal options frequently employed in the context of page or element loading are waitUtil and timeout.

• waitUntil: The waitUntil option in the page.goto(url, options) method can be configured with four WaitForOptions types: load, domcontentloaded, networkidle, and commit. More than one waitUnil options can be employed by passing them as an array.

  • domcontentloaded: This option instructs Playwright to wait until the DOMContentLoaded event has fired. This event occurs when the initial HTML document has been completely loaded and parsed. It indicates that the DOM tree is available to the browser, excluding external resources like stylesheets and images. Let's see an example:

   const { chromium } = require('playwright');

   (async () => {
     const browser = await chromium.launch({ headless: true });

     const page = await browser.newPage();
     await page.goto("https://finance.yahoo.com", {  waitUntil: "domcontentloaded" });
     await page.screenshot({ path: `yahoo-domcontentloaded.png` });

     await browser.close(); 
   })()

Observing the page, it's evident that certain elements, such as advertisement banners and specific styles such as graphs, are not rendered correctly. This discrepancy is expected, as the domcontentloaded event guarantees only the proper loading and parsing of HTML, leaving out assurance for the complete rendering of external resources like images, fonts and certain styles.

• load (Default): If no waitUntil option is provided, the default behavior is to wait until the load event is fired. The load event signifies that the entire page, including the DOM tree, CSS styles, fonts, and images, has finished loading. This is a more comprehensive wait condition, encompassing all resources associated with the page. Let's modify our earlier example to utilize the load event instead of domcontentloaded and observe if this results in the correct rendering of advertisement banners:

   const page = await browser.newPage();
   await page.goto("https://finance.yahoo.com", {  waitUntil: "load" });
   await page.screenshot({ path: `yahoo-load.png` });

Success! Employing the load event provides greater assurance that the page has been fully rendered, encompassing images, styles, and fonts. This completeness was not entirely guaranteed when relying solely on domcontentloaded.

• networkidle: 'networkidle' - DISCOURAGED wait until there are no network connections for at least 500 ms. Playwright doesn't recommend using this method, instead rely on web assertions to assess readiness instead.

• timeout: This option specifies the maximum navigation time in milliseconds. If the navigation events (like load, domcontentloaded, etc.) are not completed within this time, the page.goto() method will throw an error. It sets a time limit for the entire page navigation process.

   await page.goto("https://finance.yahoo.com", {  waitUntil: "load", timeout: 60000 });
   // Set timeout to 1 minute

The timeout option in the page.goto(url, options) method and the page.waitForTimeout() function serve different purposes. The waitForTimeout() method is not directly related to page navigation or waiting for specific events on the page. It simply pauses the execution of the script for the specified duration.

TimeoutError in Playwright​

TimeoutError is a significant issue when using Playwright. For instance, if your internet connection is slow while executing a script, the default timeout of 30 seconds may elapse before the load event fires. To handle this, you can set a longer navigation timeout, such as 100000 milliseconds. This ensures the script waits for the load event before taking a screenshot and closing the browser.

Here’s how you can implement this workaround:

await page.setDefaultNavigationTimeout(10000000);

By adjusting the default navigation timeout, the script will wait for the page to load fully, avoiding premature timeout errors, and then proceed to take a screenshot and close the browser.

Moreover, you can also use a try-catch block to implement retries with progressively increasing timeout values. This approach helps manage varying network conditions more effectively:

const attemptNavigation = async (page, url, timeout) => {
  try {
    await page.goto(url, { timeout });
    // Take screenshot or perform other actions
    await page.screenshot({ path: 'screenshot.png' });
 } catch (error) {
    if (error.name === 'TimeoutError') {
      console.log(`TimeoutError encountered. Retrying with increased timeout...`);
      timeout += 30000; // Increase timeout by 30 seconds
      await attemptNavigation(page, url, timeout);
 } else {
      throw error;
 }
 }
};

(async () => {
  const browser = await playwright.chromium.launch();
  const page = await browser.newPage();
  const url = 'https://example.com';

  await attemptNavigation(page, url, 30000); // Start with 30 seconds timeout

  await browser.close();
})();

Locators And Auto-Waiting​

Locators are the central piece of Playwright's auto-waiting and retry-ability features. In essence, locators provide a way to find elements on the page at any given moment. A locator can be created using the page.locator() method. For instance, you can select an element using its CSS selector with locators like this:

await page.locator('css=button.primary);

But how do locators enable auto-waiting? Let's understand this with an example. If you want to click on the button selected using the locator, you can simply call the click() method on it. Playwright will perform auto-waiting for the button to be present in the DOM, visible, and enabled before performing the click action:

await page.locator('css=button.primary').click();

In this example, Playwright’s auto-waiting mechanism ensures that the click operation is performed only when the button is in a state that allows it to be clicked. This includes waiting for the button to:

• Be attached to the DOM: The element must be present in the document.
• Be visible: The element must not be hidden by CSS styles.
• Be enabled: The element must not be disabled.

These conditions are automatically checked by Playwright before performing actions like click() or fill(), ensuring more reliable and stable scripts. This built-in auto-waiting mechanism reduces the need for manual waits and retries, making your tests more concise and easier to maintain.

Additionally, Playwright locators provide more advanced features, such as chaining and filtering, to target elements more precisely. For example, you can chain locators to narrow down your selection:

await page.locator('div.container').locator('button.primary').click();

In this case, Playwright will first locate the div element with the class .container and then find the <button> element with the class .primary within that div.

Understanding and effectively using locators and their auto-waiting capabilities is crucial for writing robust Playwright scripts that interact with web elements reliably.

Wait for Selector​

Up to this point, our discussion has revolved around page loading concerning the capture of viewport or full-sized screenshots. However, there are instances when your focus is solely on a specific element, such as a crypto candlestick chart or a Power BI dashboard. In these cases, rather than waiting for the entire page to load along with all associated events and API calls, a more efficient approach is to wait for that particular element to load and render. This not only saves time but also ensures the proper rendering of the targeted element.

Playwright facilitates this process through its locator.waitFor() method. Notably, this method includes a {visible: true} flag, instructing Playwright to wait until the element is present in the DOM tree and does not possess CSS properties like { display: none } or { visibility: hidden }.

Let's see an example where we await the loading of a crypto graph, capturing its screenshot using the waitFor() method:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();
  await page.goto("https://www.tradingview.com/markets/cryptocurrencies/");

  const element = page.locator("css=.chart-jDMZqyge");
  element.waitFor({ visible: true });

  await element.screenshot({ path: `crypto-graph.png` });
  
  await browser.close(); 
})()

Note that I have set { headless: false } so you can observe on your screen that our script takes a screenshot as soon as the crypto-graph appears, then quickly closes the browser. However, in production code, you should always prefer { headless: true } to conserve resources. Running in headless mode allows for faster execution and lower resource consumption, making it ideal for automated scripts running in production environments.

Wait for Page Navigation​

Page navigation involves events such as clicking on links, submitting forms, or any action that triggers a change in the page's URL. In Playwright, await page.waitForURL('**/action.php'); is a valuable method designed to handle scenarios where a script needs to wait for the completion of page navigation before proceeding with further actions. Additionally, this method also shares the same timeout and waitUntil options like page.goto(), offering same functionality but specifically for the new page being navigated to.

Use Cases:​

• Form Submissions: When automating form submissions, waiting for navigation ensures that subsequent interactions are performed on the fully loaded page, preventing premature actions.

• Link Clicks: After triggering a click on a link, waiting for navigation becomes crucial to guarantee that the new page has fully loaded before executing additional steps.

• Single Page Applications (SPAs): In SPAs where page content dynamically changes without a full page reload, await page.waitForURL('**/action.php'); synchronizes script execution with the application's state.

Here's an example wherein we navigate to the login page, input the username and password, and subsequently await the navigation and loading of the next page using the await page.waitForURL('**/logged-in-successfully'); method:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();
  await page.goto("https://practicetestautomation.com/practice-test-login/");
  await page.locator('css=#username').fill('student');
  await page.locator('css=#password').fill('Password123');

  await Promise.all([
    page.click('#submit'),
    page.waitForURL('**/logged-in-successfully/', { 
      waitUntil: "load" 
 }),
 ]);

  await page.screenshot({ path: `login.png` });
  await browser.close();
})()

The use of Promise.all() concurrently handles both statements, avoiding race conditions and ensuring a smooth synchronization with the navigation events.

Wait for Timeout​

To introduce a pause in script execution, allowing sufficient time for proper page loading, the page.waitForTimeout() method was previously employed. However, its use is DISCOURAGED in recent versions of Playwright. The alternative is to use setTimeout(), which serves the same purpose. Here is an example, where the setTimeout() function ensures a delay of 15 seconds before capturing a screenshot on the Twitter page.

await page.goto("https://example.com");
await new Promise(resolve => setTimeout(resolve, 15000));
// Further Actions

Wait for Function​

Playwright's page.waitForFunction() is designed to pause script execution until a specified function completes its evaluation within the page's context. This functionality proves valuable in situations where custom script evaluation is necessary for waiting during page loading, providing a more tailored approach than relying solely on Playwright's built-in methods and events. Let's explore an example where we wait for a specific DOM element to become visible on the screen before capturing its screenshot:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();
  await page.goto("https://www.tradingview.com/markets/cryptocurrencies/");
  
  await page.waitForFunction(() => {
    const element = document.querySelector('.chart-jDMZqyge');
    return element && element.offsetHeight > 0 && element.offsetWidth > 0;
 });
  
  await page.screenshot({ path: `crypto-market.png` });
  
  await browser.close(); 
})()

Wait for XPath​

In Playwright, page.waitFor() and page.locator('xpath=//') are methods designed to wait for the presence of an XPath expression on the page before proceeding with further actions. It allows you to wait for the presence of a specific XPath expression, be it an element or text content. Here is an example:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();
  await page.goto("https://example.com")
  
  await page.locator('xpath=//html/body/div/p[2]/a').click();
  await page.waitForURL('https://www.iana.org/help/example-domains');
  
  const text = await page.locator('xpath=//html/body/div/article/main/div/p[1]').textContent();
  console.log(text);

  await browser.close();
})();

// As described in RFC 2606 and RFC 6761, a
// number of domains such as example.com and example.org are maintained
// for documentation purposes. These domains may be used as illustrative
// examples in documents without prior coordination with us. They are not
// available for registration or transfer.

Common Situations for Waiting for a Page or Element to Load in Playwright​

Now that we've covered the most common waiting methods, let's delve into some typical scenarios where applying a combination of these methods and options can yield optimal solutions to address challenges related to page and element loading:

1. Waiting for a Specific Element to be Visible on the Page​

In certain scenarios, capturing screenshots of specific DOM elements rather than the entire page is crucial:

1. Product Thumbnails in E-commerce: In an e-commerce site, you might want to capture screenshots of individual product thumbnails for quality assurance or visual documentation.

2. Form Submissions: After submitting a form, you may want to take a screenshot of a specific confirmation message or result element to verify the success of the form submission.

3. Dashboard Widgets: In a dashboard application, you might want to capture screenshots of individual widgets or components to monitor their appearance and updates independently.

4. User Profile Sections: In a social media platform, you might want to capture screenshots of specific user profile sections, such as profile pictures or bio information.

5. Graphs and Charts: When dealing with data visualization, capturing screenshots of specific graphs or charts allows for detailed inspection and monitoring of data trends.

The locator.waitFor() method with { visible: true } flag proves instrumental in waiting for the presence and visibility of a specific element before proceeding with taking a screenshot.

2. Waiting for Page to be Ready for Button Clicks and Form Submissions​

In scenarios where you need to interact with buttons or submit forms on a web page, it's crucial to ensure that the page has completed its navigation and is ready for the subsequent actions. page.waitForURL() with { waitUntil: load } proves invaluable in such cases, providing a means to pause the script until the page has fully loaded after a button click or form submission.

3. Waiting for Page Load Before Taking Screenshot​

When you inspect the network tab in your browser's DevTools, you'll notice two essential DOM events occurring during the page load: load and DOMContentLoaded, each timestamped.

1. The first event, DOMContentLoaded, occurs after the initial HTML has been loaded and parsed.

   await page.goto(url, {waitUntil: "domcontentloaded"});

2. The subsequent event, named load, transpires when additional elements like styles, fonts, and images have been fetched and integrated into the webpage.

   await page.goto(url, {waitUntil: "load"});

These events serve as metrics, indicating when the page has finished loading, providing insights into the time required for a specific page to complete its loading process.

When Load is not Enough​

Sometimes, simply waiting for the page load event is insufficient, especially if you need to ensure that certain network requests or specific elements are fully loaded before proceeding. While Playwright provides a networkidle option, its use is generally discouraged. Instead, you can employ more reliable methods to wait for essential elements or network activities to complete.

1. Wait for a Specific URL Using waitForURL: You can use the waitForURL method to wait for a particular URL to be fetched, indicating that most of the page content has loaded. This is particularly useful when you know a specific network request signifies the completion of significant loading processes.

   // Example: Wait for a specific URL
   await page.waitForURL('**/special-endpoint');

2. Wait for a Specific DOM Element Using waitFor: Another approach is to wait for a specific DOM element to become visible on the page, signaling that the page has mostly rendered. The waitFor() method is ideal for this purpose.

   // Example: Wait for a specific DOM element
   const element = await page.locator('#special-element');
   element.waitFor({ visible: true });

3. Wait for a Specific Network Requests Using waitForResponse: Monitor specific network requests to ensure they are completed before proceeding. This can be particularly useful for ensuring that important API calls have finished.

   await page.waitForResponse(response => 
       response.url().includes('special-api-endpoint') && response.status() === 200
   );

4. Web Assertions: Use assertions to check the state of the page and ensure that it has reached the desired condition. This method is more reliable as it directly verifies the elements and their properties.

   await expect(page).toHaveText('#text-element', 'Expected Text');

By using these alternatives, you can create more reliable and efficient scripts that accurately determine when a page is ready for interaction.

4. Waiting for an API Call to Populate the Page Content​

In scenarios where you expect a website to be fully loaded only after a specific API request or response, Playwright's page.waitForRequest() and page.waitForResponse() methods can be employed.

These methods accept a URL, such as an API endpoint, or a predicate function. The predicate function allow you to evaluate specific expressions. For example, you can use this function to verify whether the desired data has been successfully received through the API request.

This approach is particularly useful when waiting for dynamic content to be populated on the page as a result of asynchronous API calls. By incorporating these Playwright methods, you can synchronize your script with the completion of API requests, ensuring that the page is fully loaded and ready for capturing screenshot.

Here are examples illustrating the use of these methods:

• waitForRequest(url): Wait for a particular request to take place.

   // Start waiting for request before clicking. Note no await.
   const requestPromise = page.waitForRequest('https://example.com/resource');
   await page.getByText('trigger request').click();
   const request = await requestPromise;

   // Alternative way with a predicate. Note no await.
   const requestPromise = page.waitForRequest(request =>
     request.url() === 'https://example.com' && request.method() === 'GET',
   );
   await page.getByText('trigger request').click();
   const request = await requestPromise;

• waitForResponse(url, callback): Awaits the response to be fetched by the browser following the initiation of a request.

   // Start waiting for response before clicking. Note no await.
   const responsePromise = page.waitForResponse('https://example.com/resource');
   await page.getByText('trigger response').click();
   const response = await responsePromise;

   // Alternative way with a predicate. Note no await.
   const responsePromise = page.waitForResponse(response =>
     response.url() === 'https://example.com' && response.status() === 200
    && response.request().method() === 'GET'
   );
   await page.getByText('trigger response').click();
   const response = await responsePromise;

Combining Waiting Strategies​

In the realm of web automation, achieving robust and reliable scripts often demands a thoughtful combination of waiting strategies. This is crucial to address diverse scenarios and ensure precise synchronization with the dynamic behaviors of webpages. A strategic approach involves adapting to variable loading times, optimizing wait durations, and effectively handling timeouts and exceptions.

Best Practices for Waiting in Playwright​

When it comes to waiting in Playwright, there are several best practices to keep in mind:

1. Optimize Wait Times: Striking the appropriate balance in wait times is crucial to optimize automation efficiency. A useful strategy involves leveraging the browser's inspect tab to investigate the time duration required for page loading on a specific site. Additionally, the networkidle strategy, as discussed earlier, proves valuable for further optimizing waiting times during the automation process.

2. Avoid excessive waiting to improve efficiency: Waiting too long can slow down your automation and waste resources. To avoid this, you can use the page.setDefaultTimeout() method to set a maximum timeout for all wait methods.

   setDefaultNavigationTimeout() takes priority over setDefaultTimeout().

3. Handling exceptions: All the waiting methods we covered operate asynchronously and may encounter failures due to network issues or server-side errors. It is advisable to encapsulate these methods within try...catch blocks to handle exceptions. Here's a code example demonstrating robust error handling:

   try {
     await page.goto('https://example.com')
   } catch (error) {
     console.error('Navigation Unsuccessful!', error.message);
   }

4. Migrating from Puppeteer: This guide describes migration to Playwright from Puppeteer. The APIs have similarities, but Playwright offers much more possibilities for web testing and cross-browser automation.

Conclusion​

In the realm of web scraping and automation, the initial and crucial step involves waiting for page loading. Playwright stands out with its variety of methods and options, serving as an effective initial solution and waiting strategy before proceeding with subsequent automation tasks, such as navigating through pages and extracting data. One of Playwright's standout features is its auto-waiting mechanism, which simplifies the process by automatically waiting for elements to be ready before performing actions. This, combined with the power of locators, ensures that your scripts are both robust and reliable.

However, it's important to recognize the inherent complexity of websites. Therefore, before diving into any web scraping or automation task, conducting a thorough investigation of the target website using the browser's inspect network activity tab in your DevTools is imperative.

While Playwright provides ample methods for most websites, there may be instances where creating a custom waiting strategy or function becomes necessary for optimal synchronization before executing automated tasks.

More Web Scraping Tutorials​

• Puppeteer Guide: Waiting For Page or Element To Load
• Playwright Guide: How To Take Screenshots
• NodeJS Playwright: Logging Into Websites