Skip to main content

How to Block Images and Resources

Selenium Guide: How to Block Images and Resources

In web automation and testing with Selenium, controlling web resources is crucial. Blocking unnecessary resources, such as images, can enhance test performance by speeding up page loading and minimizing data usage.

There are several methods to block resources in Selenium and in this guide, we'll divide into these methods and going to walk through:

Need help scraping the web?

Then check out ScrapeOps, the complete toolkit for web scraping.

TL:DR - How to Block Images and Resources​

In Selenium, optimizing performance often involves minimizing resource load. To block images and resources in Chrome using Selenium:

  1. Create a ChromeOptions Instance: Start by initializing a ChromeOptions object. This class allows you to set various configurations for the Chrome browser.

    from selenium import webdriver
    options = webdriver.ChromeOptions()

  2. Disable Images: Add an argument to the ChromeOptions instance to disable images. This is done by setting the prefs property with the appropriate configurations to prevent image loading.

    prefs = {"profile.managed_default_content_settings.images": 2}
    options.add_experimental_option("prefs", prefs)

In this code, the prefs dictionary is set to disable image loading. The key profile.managed_default_content_settings.images is set to 2, which instructs Chrome to block images.

  1. Initialize the WebDriver with Options: Finally, create a WebDriver instance for Chrome, passing in the configured options.

    driver = webdriver.Chrome(options=options)

The final code will look like this:

from selenium import webdriver
options = webdriver.ChromeOptions()
prefs = {"profile.managed_default_content_settings.images": 2}
options.add_experimental_option("prefs", prefs)
driver = webdriver.Chrome(options=options)

This WebDriver instance will launch Chrome with the specified configurations, effectively blocking images and saving resources during your automated browsing sessions.

The above example demonstrates a practical approach to reducing resource usage in Selenium tests, particularly useful when running tests that do not require image rendering.

This can lead to faster execution times and lower data consumption.

Blocking Images and Resources in Selenium​

Configuring Selenium to block images and resources is an essential skill for optimizing web automation and testing. This process involves a series of steps and understanding these steps is essential for effectively implementing resource blocking.

First, we want to ensure Selenium is installed in your Python environment. You can use the command below:

pip install selenium

Next, you'll need to download the appropriate WebDriver for your browser.

  • For Chrome, it's ChromeOptions and make sure that your version of chromedriver matches version of Chrome you're using.
  • For Firefox users, you will need to use Geckodriver.

The next step is to configure the browser settings within Selenium to block resources. We'll need to use browser-specific options, such as ChromeOptions or FirefoxOptions, to customize the browser behavior. These options control various aspects of the browser, including which resources to load.

Within these options, set the preferences to block images and potentially other resources like CSS and JavaScript.

This is typically done by altering specific settings that control resource loading. For example, in Chrome, you can add preferences to the ChromeOptions to disable images:

from selenium import webdriver
options = webdriver.ChromeOptions()
prefs = {"profile.managed_default_content_settings.images": 2,
    "profile.default_content_setting_values.javascript": 2}
options.add_experimental_option("prefs", prefs)

In this code snippet, we're creating a ChromeOptions object and setting preferences to control the loading of images and JavaScript. The key "profile.managed_default_content_settings.images": 2 is used to block images. The number 2 here represents the setting to disable image loading in Chrome.

Similarly, "profile.default_content_setting_values.javascript": 2 is used to disable JavaScript execution, with 2 again indicating the 'disable' setting. These numeric values are part of internal settings of Chrome, where different numbers represent different behaviors such as allow, block, or prompt for user input.

By setting these preferences to 2, we instruct Chrome, when driven by Selenium, to block these types of resources.

If the expected blocking does not occur, revisiting the WebDriver setup and configuration is necessary. Compatibility between the WebDriver version and the browser, along with accurate setting of preferences, plays a significant role.

Example code for Chrome​

To demonstrate blocking resources in Chrome using Selenium, we'll use a practical example.

The following code illustrates how to configure Selenium to block images and other resources, such as JavaScript, for the Chrome browser.

from selenium import webdriver

# Initialize ChromeOptions
options = webdriver.ChromeOptions()

# Set preferences to block images and JavaScript
prefs = {
    "profile.managed_default_content_settings.images": 2,
    "profile.default_content_setting_values.javascript": 2
}
options.add_experimental_option("prefs", prefs)

# Create a WebDriver instance with the configured options
driver = webdriver.Chrome(options=options)

# Navigate to the website
driver.get("https://www.nytimes.com")

The code initializes Chrome with custom settings using Selenium ChromeOptions. These settings include a preference dictionary that instructs Chrome to block both images and JavaScript.

You will see the website like this:

NYT Blocked

By setting specific keys in this dictionary, such as "profile.managed_default_content_settings.images" and "profile.default_content_setting_values.javascript", to 2, we disable the loading of these resources. Once these preferences are added to the ChromeOptions and the WebDriver is initialized with these options, Chrome will open with these restrictions in place.

The driver.get("https://www.nytimes.com") command then navigates to the New York Times website, where the effect of these settings can be observed that the page loads without images and JavaScript, demonstrating a streamlined and resource-efficient browsing session.

Example code for Firefox​

To block resources in Firefox using Selenium, a slightly different approach compared to Chrome is used. Below is an example code for setting up resource blocking in Firefox.

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

# Initialize FirefoxOptions
options = Options()

# Disable images
options.set_preference("permissions.default.image", 2)

# Disable JavaScript (optional)
options.set_preference("javascript.enabled", False)

# Create a WebDriver instance with the configured options
driver = webdriver.Firefox(options=options)

# Navigate to the website
driver.get("https://www.nytimes.com")

In this example, the Firefox-specific Options class is used to customize browser settings. The set_preference method sets preferences to control the loading of images and JavaScript. Specifically, "permissions.default.image": 2 disables image loading, and "javascript.enabled": False turns off JavaScript execution.

These preferences are applied when the Firefox WebDriver is initialized with the options object. Upon navigating to a website, Firefox will load the page without images and, optionally, without executing JavaScript.

The key differences from Chrome include the use of the Firefox Options class and the distinct preference settings unique to the Firefox browser. This approach is essential for Firefox users seeking to optimize resource usage in their Selenium automation tasks.

Explaining Selenium Options​

Selenium provides various options and capabilities across different browsers that allow users to customize and control resource loading. Understanding these options is key to effectively managing web resources during automated testing or web scraping.

  1. Browser-Specific Options:

    • Both ChromeOptions and FirefoxOptions are classes in Selenium used to customize respective browser settings. They offer a range of options to control aspects like headless browsing, window size, and resource loading preferences.
    • Example settings include disabling images, JavaScript, and other resource-intensive elements, which are particularly useful for speeding up tests and reducing bandwidth usage.

  2. General Capabilities:

    • In addition to browser-specific options, Selenium also supports capabilities that are common across different browsers. These are set using the DesiredCapabilities class. This feature is used to define general preferences and properties of browser sessions, such as handling cookies, setting proxy configurations, and controlling timeouts.
    • Chrome, in particular, provides experimental options that can be used to further customize browser behavior. These options might include advanced resource-blocking configurations not available through standard preferences.

The settings within browser-specific options of Selenium play a critical role in controlling how resources are loaded during automated sessions. Here's an overview of how these settings can be utilized:

  • Disabling Images:

    • In Chrome, setting the preference "profile.managed_default_content_settings.images": 2 effectively blocks all images from being loaded. Firefox achieves a similar outcome using the preference "permissions.default.image": 2.
    • Both settings ensure that images are not loaded, which can significantly speed up page loading times during testing.

  • Controlling JavaScript Execution:

    • The approach to disabling JavaScript differs between browsers. In Chrome, this usually requires using experimental options or integrating a browser extension, as there isn't a direct preference setting available for disabling JavaScript.
    • On the other hand, Firefox provides a straightforward preference for this, "javascript.enabled": False, which when set, turns off JavaScript execution in the browser.

  • Blocking Other Resources:

    • These browsers also allow for blocking other resource types like stylesheets, fonts, or specific URLs. This is achieved through similar methods of setting preferences or options, tailored to the type of resource you intend to block.
    • By configuring these settings, you can control what content is loaded during your Selenium sessions, which can be crucial for certain types of testing or data scraping activities.

Browser-Specific Settings​

In Selenium automation, each browser comes with its own set of unique preferences and capabilities, particularly when it comes to blocking resources. Understanding these variations is crucial for tailoring your automation scripts to different browsers.

For instance, Chrome and Firefox, as previously discussed, have their specific ways of handling resource blocking through preferences in ChromeOptions and FirefoxOptions. Chrome relies on a set of preferences and experimental options, while Firefox often uses more direct settings available in its Options class.

However,if you want to use other browsers like Safari, Edge, or Opera, you need to consider their capabilities and limitations.

  • Safari, for example, has limited options for customization compared to Chrome and Firefox.

  • Edge, which is now Chromium-based, shares many capabilities with Chrome, allowing for similar configurations. However, always check for any Edge-specific nuances or limitations in its WebDriver implementation.

  • Opera also has its set of unique preferences and options, often similar to Chrome due to its Chromium base, but with its specific features and settings.

Adapting your resource-blocking strategy to various browsers requires careful consideration and testing. Researching browser-specific documentation is essential to understand the available settings.

Regular testing ensures that your configurations work as intended. Given the frequent updates to browsers and WebDriver versions, staying current is crucial. In cases where built-in options are limited, using a proxy server can be an effective solution.

Lastly, leveraging the collective knowledge of the Selenium community through forums and discussion boards can provide valuable insights and problem-solving strategies.

Understanding Resources in a Web Page:​

When navigating a web page, it's important to understand the various types of resources that the page loads. These resources are essential components that make up the content and functionality of the website.

  • Images: Perhaps the most noticeable resources on a web page, images can range from background graphics to icons and photographs. They are typically embedded using the <img> tag in HTML or through CSS for backgrounds. Images can significantly impact page loading times, especially if they are high-resolution or numerous.

  • Stylesheets (CSS): Cascading Style Sheets (CSS) are used to define the look and feel of a webpage. They control everything from layout to font styles and colors. CSS files are often external resources linked within the HTML, though they can also be inlined directly into the HTML code.

  • JavaScript: JavaScript is a powerful scripting language used to create interactive and dynamic web pages. It can control webpage behavior, respond to user actions, and even fetch additional data from servers without reloading the page. JavaScript files are usually external resources but can also be embedded directly in the HTML.

  • HTML Content: The HTML code itself is a resource. It defines the structure and content of the web page, including text, links, and references to other resources like images, CSS, and JavaScript.

  • Fonts: Many modern websites use custom fonts, loaded from external sources, to maintain brand consistency and aesthetic appeal. These font files can be significant in size and impact loading times.

  • Multimedia: This includes video and audio content embedded in web pages. Like images, multimedia elements can be large and affect performance.

  • Plugins and Third-Party Content: Many sites include content and functionality from third-party sources, such as social media feeds, maps, analytics scripts, and advertisements. These are often loaded from external servers and can vary widely in size and impact performance.

  • API Calls and AJAX: Websites often make background requests to APIs for data. These AJAX calls are a key part of how modern web applications function, allowing for the dynamic loading of content.

How To Block Images​

Blocking images in web automation can significantly enhance performance by reducing load times and bandwidth usage. Here's a code example demonstrating how to block images in both Chrome and Firefox using Selenium.

Blocking Images in Chrome:

from selenium import webdriver

# Initialize ChromeOptions
chrome_options = webdriver.ChromeOptions()

# Set preferences to block images
prefs = {"profile.managed_default_content_settings.images": 2}
chrome_options.add_experimental_option("prefs", prefs)

# Create a WebDriver instance with the configured options
chrome_driver = webdriver.Chrome(options=chrome_options)

# Navigate to a website
chrome_driver.get("https://www.nytimes.com")

Blocking Images in Firefox:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

# Initialize FirefoxOptions
firefox_options = Options()

# Set preference to block images
firefox_options.set_preference("permissions.default.image", 2)

# Create a WebDriver instance with the configured options
firefox_driver = webdriver.Firefox(options=firefox_options)

# Navigate to a website
firefox_driver.get("https://www.nytimes.com")

In both cases, when the WebDriver navigates to a website, it will load the page without displaying any images. This approach is particularly useful for tests where image rendering is not necessary, thereby saving data and reducing page load times.

How To Block CSS Loading​

Blocking CSS loading in Selenium is a bit more complex than blocking images, as there's no direct built-in preference for this in most browsers. However, you can achieve this by intercepting and blocking requests for CSS files.

Here’s how you can block CSS loading in Chrome and Firefox using Selenium along with browser-specific capabilities.

Blocking CSS in Chrome:

To block CSS in Chrome using Selenium, we can use the Chrome DevTools Protocol (CDP) to intercept and block requests for CSS files.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service

# Initialize ChromeOptions
chrome_options = Options()

# Enable browser logging
chrome_options.set_capability('goog:loggingPrefs', {'browser': 'ALL'})

service = Service('path_to/chromedriver.exe')
driver = webdriver.Chrome(service=service, options=chrome_options)

# Navigate to a website
driver.get("https://www.nytimes.com")

# Intercept and block CSS requests using Chrome DevTools Protocol commands
driver.execute_cdp_cmd("Network.setBlockedURLs", {"urls": ["*.css"]})
driver.execute_cdp_cmd("Network.enable", {})

# Refresh the page to apply CSS blocking
driver.refresh()

Blocking CSS in Firefox:

One approach for blocking CSS in Firefox is to manipulate the Firefox profile to disable CSS.

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.service import Service

# Initialize FirefoxOptions
firefox_options = Options()

# Disable CSS
firefox_options.set_preference("permissions.default.stylesheet", 2)

# Create a WebDriver instance with the configured options
service = Service('path/to/geckodriver')
firefox_driver = webdriver.Firefox(service=service, options=firefox_options)

# Navigate to a website
firefox_driver.get("https://www.nytimes.com")

In Firefox, the set_preference method is used to disable stylesheet loading. However, this method might not be as effective as in Chrome, as it depends on how Firefox interprets the "permissions.default.stylesheet" preference in different versions.

These code examples demonstrate approaches to blocking CSS in Chrome and Firefox using Selenium. It's important to note that these methods may have limitations and could affect the functionality of web pages, as CSS is crucial for layout and styling.

How To Block Media Loading​

In Chrome, you can use the Chrome DevTools Protocol (CDP) commands to block media requests:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service

# Initialize ChromeOptions
chrome_options = webdriver.ChromeOptions()

service = Service('path/to/chromedriver')
driver = webdriver.Chrome(service=service, options=chrome_options)

# Navigate to a website
driver.get("https://www.nytimes.com")

# Intercept and block media requests
driver.execute_cdp_cmd("Network.setBlockedURLs", {"urls": ["*.mp3", "*.mp4"]})
driver.execute_cdp_cmd("Network.enable", {})

# Refresh the page to apply media blocking
driver.refresh()

In this snippet, media files such as MP3 and MP4 are blocked using the CDP commands Network.setBlockedURLs and Network.enable.

Blocking media in Firefox using Selenium can be challenging since there isn't a direct method like DevTools Protocol in Chrome. However, you can attempt to block media by using Firefox preferences.

Here's an example of how you might approach this:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.service import Service

# Initialize FirefoxOptions
firefox_options = Options()

# Attempt to disable various media types
firefox_options.set_preference("media.autoplay.default", 1) # 1 for block, 0 for allow
firefox_options.set_preference("media.autoplay.enabled.user-gestures-needed", False)
firefox_options.set_preference("media.autoplay.blocking_policy", 2)

# Create a WebDriver instance with the configured options
service = Service('path/to/geckodriver')
firefox_driver = webdriver.Firefox(service=service, options=firefox_options)

# Navigate to a website
firefox_driver.get("https://www.nytimes.com")

In this code snippet:

  • The media.autoplay.default preference is set to 1 to block media autoplay, which is often the main concern with media loading.
  • The media.autoplay.enabled.user-gestures-needed is set to False to prevent media from playing without user interaction.
  • The media.autoplay.blocking_policy is set to 2, which represents the strictest blocking policy in Firefox for media autoplay.

How To Block Fonts Loading​

In Chrome, you can still use Chrome DevTools Protocol (CDP) commands to block font requests as well:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service

# Initialize ChromeOptions
chrome_options = webdriver.ChromeOptions()

service = Service('path/to/chromedriver')
driver = webdriver.Chrome(service=service, options=chrome_options)

# Navigate to a website
driver.get("https://www.nytimes.com")

# Intercept and block font requests
driver.execute_cdp_cmd("Network.setBlockedURLs", {"urls": ["*.woff", "*.woff2", "*.ttf"]})
driver.execute_cdp_cmd("Network.enable", {})

# Refresh the page to apply font blocking
driver.refresh()

In this code, font file types such as WOFF, WOFF2, and TTF are targeted and blocked using the CDP commands.

As with media blocking, directly blocking fonts in Firefox using Selenium is less straightforward. You might attempt to use Firefox preferences to disable fonts, but this approach can be limited and may not work consistently across different versions of Firefox.

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.service import Service

# Initialize FirefoxOptions
firefox_options = Options()

# Attempt to disable fonts
firefox_options.set_preference("browser.display.use_document_fonts", 0)

# Create a WebDriver instance with the configured options
service = Service('path/to/geckodriver')
firefox_driver = webdriver.Firefox(service=service, options=firefox_options)

# Navigate to a website
firefox_driver.get("https://www.nytimes.com")

In this Firefox example, the browser.display.use_document_fonts preference is set to 0, which attempts to prevent the browser from using downloadable fonts. However, this setting may not completely block all custom fonts, and its effectiveness can vary.

Blocking fonts, especially in Firefox, is a more complex task and may not always yield consistent results. It's important to test thoroughly to ensure that the blocking behaves as expected and to be aware that this approach might affect the layout and readability of web pages.

How To Block Scripts Running​

In Chrome, this is how we can block scripts running:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service

# Initialize ChromeOptions
chrome_options = webdriver.ChromeOptions()

service = Service('path/to/chromedriver')
driver = webdriver.Chrome(service=service, options=chrome_options)

# Navigate to a website
driver.get("https://www.nytimes.com")

# Block all JavaScript execution
driver.execute_cdp_cmd("Page.setJavaScriptEnabled", {"enabled": False})

# Refresh the page to apply script blocking
driver.refresh()

For Firefox, we can disable JavaScript entirely using Firefox preferences:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.service import Service

# Initialize FirefoxOptions
firefox_options = Options()

# Disable JavaScript
firefox_options.set_preference("javascript.enabled", False)

# Create a WebDriver instance with the configured options
service = Service('path/to/geckodriver')
firefox_driver = webdriver.Firefox(service=service, options=firefox_options)

# Navigate to a website
firefox_driver.get("https://www.nytimes.com")

How To Block XHR & Fetch Requests​

Blocking XMLHttpRequests (XHR) and Fetch requests can be essential for testing scenarios where you need to simulate conditions without certain network calls or to speed up page loading by preventing specific data-fetching operations.

Here's how to block XHR and Fetch requests in Chrome:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service

# Initialize ChromeOptions
chrome_options = webdriver.ChromeOptions()

service = Service('path_to/chromedriver.exe')
driver = webdriver.Chrome(service=service, options=chrome_options)

# Navigate to a website
driver.get("https://www.nytimes.com")

# Block XHR and Fetch requests
driver.execute_cdp_cmd("Network.setRequestInterception", {
    "patterns": [{"urlPattern": "*", "resourceType": "XHR"},
                 {"urlPattern": "*", "resourceType": "Fetch"}]
})

# Define a function to handle the interception
def intercept_request(request):
    if request['resourceType'] in ['XHR', 'Fetch']:
        driver.execute_cdp_cmd("Network.continueInterceptedRequest", {
            "interceptionId": request['interceptionId'],
            "errorReason": "Failed"
        })

# Set up request interception
driver.execute_cdp_cmd("Network.setRequestInterception", {"patterns": [{"urlPattern": "*"}]})
driver.requests_interceptor = intercept_request

# Refresh the page to apply blocking
driver.refresh()

Blocking XHR and Fetch requests in Selenium is straightforward in Chrome using CDP commands but can be challenging in Firefox due to the lack of similar built-in capabilities.

These techniques are particularly useful in testing scenarios where you need to control network traffic to simulate different conditions or to test the resilience of web applications against failed or absent data-fetching operations.

Dynamic Resource Blocking and AJAX Requests​

When dealing with web applications, one of the significant challenges in Selenium automation is handling dynamic content and asynchronous (AJAX) requests. These elements can introduce complexity, especially when implementing resource blocking strategies.

Dynamic web applications often load and change content asynchronously, without full page reloads. This behavior is typically driven by JavaScript and AJAX requests fetching data from the server. Here are some key points to consider:

  • Timing Issues: Since AJAX requests and dynamic content loading can happen at any time, they may occur after your resource blocking settings are applied. This can result in unexpected behavior where some resources are not blocked as intended.

  • Identifying Dynamic Requests: It can be challenging to identify which requests to block, as URLs or resources might not be known beforehand. Dynamic requests often depend on user interactions or other runtime conditions.

  • Handling Asynchronous Nature: The asynchronous nature of these requests means that the usual linear flow of a script might not align with when these resources are requested or loaded.

To handle dynamic content and AJAX requests effectively while blocking resources, we can consider the following strategies:

  • Dynamic Resource Blocking: Adjust your blocking strategy to handle resources that are loaded dynamically. This might involve setting up a mechanism to continuously apply blocking rules or to reapply them upon detection of certain events or conditions.

  • Using Selenium Waits: You can use built-in wait functions of Selenium, like WebDriverWait and ExpectedConditions, to wait for certain elements to load before proceeding. This is particularly useful for AJAX-loaded content.

  • Proxy Servers: Consider using a proxy server for more complex blocking scenarios. A proxy can inspect and modify traffic on the fly, giving you more control over dynamic resources.

  • Monitoring: Implement continuous monitoring of network activity within your Selenium script. Tools like DevTools Protocol can be used to intercept and analyze network requests dynamically, allowing you to block or modify requests as they occur.

  • Custom Scripts and Extensions: In some cases, injecting custom JavaScript into the browser session with Selenium to control or monitor resource loading and AJAX calls can be effective

Handling dynamic content and AJAX requests in Selenium requires a more adaptive and responsive approach compared to static resource blocking. By combining Selenium capabilities with additional tools and strategies, you can effectively manage dynamic resources, ensuring that your automation scripts are both robust and efficient.

Handling Errors​

While implementing resource blocking in web automation with Selenium, you may encounter various issues and errors. These can range from incorrect resource blocking to challenges with browser compatibility.

Understanding these common problems and knowing how to address them is crucial for smooth automation processes.

Here are some typical issues and their solutions:

  1. Incomplete Page Load: This can occur if the preferences or options are not correctly set up or if the browser version is not compatible with the specified WebDriver version. To solve this, you'll need to ensure that the WebDriver is up to date and compatible with your browser. Double-check the syntax and parameters used for setting preferences or options.

  2. Timeouts or Synchronization Issues: Asynchronous loading of resources can lead to timeouts or synchronization issues where the script proceeds before the necessary elements are available. To prevent it, you can use wait function of Selenium to ensure that elements are loaded before interaction. Adjust timeout settings as needed.

  3. Scripts or Stylesheets Still Loading: Some resources, particularly scripts and stylesheets, might load before your blocking rules are applied, or they might be loaded dynamically. You should implement waits or retry mechanisms to delay actions until the page is fully loaded. For dynamic content, reapply blocking rules upon detection of new load events.

  4. Unexpected Page Behavior : Blocking certain resources like JavaScript or CSS can significantly alter the functionality and layout of a webpage. You should thoroughly test your automation scripts across different pages to understand the impact of blocking specific resources.

Best Practices​

When incorporating resource blocking in Selenium scripts, it's crucial to apply best practices that boost performance and ensure the reliability of your automation tasks. Selenium, being a powerful tool for web browser automation, has its methods and strategies for resource blocking.

Here are essential tips and best practices:

  • Get familiar with the Selenium WebDriver API, especially the capabilities and options that allow you to control browser behavior, such as disabling image loading or other resource types.

  • Use browser-specific preferences and capabilities in Selenium to block or modify specific types of requests, such as images, stylesheets, or scripts. This can lead to more efficient script performance, which is particularly beneficial in tasks like data scraping.

  • Selenium scripts are synchronous by default, but Python asyncio library can be used for asynchronous operations. This is useful for handling the timing and synchronization of script execution and resource loading.

  • For dynamically loaded content, employ explicit waits in Selenium, like WebDriverWait combined with expected conditions, such as element_to_be_clickable or visibility_of_element_located, to ensure that elements are fully loaded before proceeding.

  • Be cautious with resource blocking to avoid scenarios where essential page content fails to load. This might lead to missing crucial information necessary for your automation objectives.

  • Thoroughly test your Selenium scripts with resource blocking implemented. This ensures that key page elements are not unintentionally blocked. Use debugging tools and techniques in Python to monitor script execution and network activity.

  • Keep up to date with the latest releases of Selenium and the web drivers for the browsers you are automating. New updates can introduce changes in handling resources or new functionalities for more refined control.

  • Document your resource-blocking logic. Well-commented and organized code is easier to maintain, particularly for complex scripts or when collaborating within a team.

Case Study: Blocking Images and Resources on Wikipedia​

In this case study, we'll examine the performance impact of blocking images and other resources while scraping a Wikipedia page using Selenium in Python. Wikipedia, known for its comprehensive articles, often includes a variety of images and supplementary resources.

These elements, while enriching the content, can significantly impact the loading time and data usage during web scraping or automated browsing.

The goal here is to demonstrate how effective resource blocking can be in enhancing performance, especially in web scraping tasks.

By comparing a standard Selenium setup for scraping against a setup with resource blocking enabled, we can quantify the benefits in terms of faster execution times and reduced resource load.

Setup Without Blocking​

To understand the impact of resource blocking, we first establish a baseline by setting up a standard Selenium script for scraping the Wikipedia main page without any form of resource blocking.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service

# Specify the path to chromedriver
service = Service('path_to/chromedriver')

# Initialize the Chrome WebDriver
driver = webdriver.Chrome(service=service)

# Navigate to the Wikipedia main page
driver.get("https://en.wikipedia.org/wiki/Main_Page")

# Example of scraping: Extract the 'Did you know' section text
dyk_section = driver.find_element(By.CSS_SELECTOR, "div#mp-dyk")
dyk_text = dyk_section.text

# Print the scraped content
print("Did You Know Section: ")
print(dyk_text)

# Close the WebDriver session
driver.quit()

This setup will load all elements of the Wikipedia page, including images, scripts, and stylesheets. It serves as a baseline to compare against a similar setup with resource blocking, allowing us to measure the performance differences in terms of page load times and overall resource usage.

Website without blocking any resources will look like this:

No blocking

Setup With Blocking​

In contrast to the standard setup, we now modify the Selenium script to include resource blocking, specifically targeting images and other heavy resources. This approach is aimed at improving performance by reducing page load times and minimizing the amount of data downloaded.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service

# Initialize ChromeOptions
chrome_options = Options()

# Block images and other heavy resources such as CSS and JavaScript
prefs = {
    "profile.default_content_setting_values.images": 2,
    "profile.managed_default_content_settings.stylesheets": 2,
    "profile.managed_default_content_settings.javascript": 2
}
chrome_options.add_experimental_option("prefs", prefs)

# Specify the path to chromedriver
service = Service('path_to/chromedriver')

# Initialize the Chrome WebDriver with the configured options
driver = webdriver.Chrome(service=service, options=chrome_options)

# Navigate to the Wikipedia main page
driver.get("https://en.wikipedia.org/wiki/Main_Page")

# Example of scraping: Extract the 'Did you know' section text
dyk_section = driver.find_element(By.CSS_SELECTOR, "div#mp-dyk")
dyk_text = dyk_section.text

# Output the scraped content
print("Did You Know Section (With Resource Blocking):")
print(dyk_text)

# Close the WebDriver session
driver.quit()

Website with resources blocked will look like this:

With blocking

  • In this version of the script, ChromeOptions is used to set preferences that block the loading of images, CSS, and JavaScript.

  • These preferences are passed to the Chrome WebDriver upon initialization and script then proceeds as before to navigate to Wikipedia and scrape content.

  • By blocking unnecessary resources, the browser spends less time fetching and rendering these elements, leading to faster overall execution of the script.

  • The amount of data downloaded is significantly reduced, which is particularly beneficial in scenarios with limited bandwidth or in large-scale scraping operations.

Why to Block Images and Resources​

Resource blocking in web automation is a technique that involves selectively preventing certain web elements from loading during automated browser sessions. Implementing resource blocking is particularly useful in web scraping and automated testing, as it offers several advantages:

  • Speed Enhancement: By blocking unnecessary resources, the amount of data that needs to be downloaded and processed by the browser is reduced. This leads to faster page load times, which is especially beneficial in web scraping where efficiency is key.

  • Reduced Bandwidth Usage: In scenarios where bandwidth is a concern, such as in large-scale data scraping or testing in environments with limited network resources, blocking images and heavy resources can significantly decrease data usage.

  • Focus on Relevant Content: In many web automation tasks, the primary interest lies in the textual content or specific HTML elements of a webpage. Resource blocking allows the automation process to focus on these elements without the overhead of loading and rendering irrelevant resources.

Furthermore, Selenium offers the capability to configure browsers to ignore certain types of web content, thereby streamlining the automation process:

  • Using WebDriver Options: In browsers like Chrome, preferences can be set in the ChromeOptions object to disable the loading of images, CSS, and JavaScript. This is achieved by modifying the browser settings or capabilities.

  • Headless Mode: Running browsers in headless mode, where graphical rendering is unnecessary, is another effective way to conserve resources. This mode focuses on the DOM (Document Object Model) structure rather than the visual rendering, making it ideal for scraping and automated testing of web applications.

  • Custom Preferences: For more specific needs, custom browser preferences and capabilities can be set to tailor the browsing environment to the requirements of the automation task.

While resource blocking has its advantages, it's important to recognize scenarios where it might not be appropriate:

  • Scraping for Visual Content: If the objective of the scraping task is to collect images or visual layouts, then blocking these resources would be counterproductive.

  • Testing User Interface (UI) and User Experience (UX): In automated UI/UX testing, the visual elements and interactive scripts are crucial. Blocking these resources would not give an accurate representation of the user's experience.

  • Dynamic Content Dependent on JavaScript: For web pages where the content is dynamically loaded via JavaScript, disabling script loading might result in incomplete or missing content.

Benefits of Blocking Images and Resources​

Blocking images and resources in web automation offers several benefits, enhancing the efficiency and effectiveness of tasks like web scraping, automated testing, and even general browsing.

Understanding these advantages can help tailor automation strategies to specific needs, particularly in scenarios where performance and resource management are crucial.

  • Faster Page Loading: Web pages often contain numerous resources that can slow down loading times. By blocking non-essential resources like images, stylesheets, or JavaScript files, the amount of data the browser needs to fetch and render is reduced. This leads to quicker page loads, which is particularly beneficial in web scraping where speed and efficiency are key.

  • Bandwidth Usage: Images and multimedia content are major consumers of bandwidth. In scenarios where bandwidth is limited or costly blocking these resources can significantly lower data usage, keeping operational costs down and improving performance.

  • Avoid Detection: Some websites employ tracking scripts and other mechanisms to detect and limit scraping activities. By blocking these scripts and similar resources, it's possible to reduce the possibility of being detected, thereby avoiding rate limits or bans.

  • Testing and Development: For developers and testers, simulating different network conditions or understanding how a website behaves without certain resources is invaluable. Blocking resources can help in testing the robustness and performance of web applications under varied conditions.

  • Privacy and Security: Blocking external resources, especially from third parties, can increase privacy by preventing the loading of tracking and analytics scripts. Additionally, it minimizes exposure to security risks associated with external content, which might include malicious scripts.

  • Content Filtering: In certain contexts, there may be a need to filter out specific types of content. Blocking resources can serve this purpose, for instance, by preventing the display of images or execution of scripts from certain domains.

  • Limited Device Resources: Devices with constrained CPU and memory resources can benefit from resource blocking. It helps in delivering a smoother browsing experience, reducing the computational load and memory usage typically required to process and display heavy web content.

Ideal Use Cases for Blocking Images and Resources​

Blocking images and other resource-intensive elements can be highly beneficial in various web automation scenarios. Here are some specific examples where this approach proves to be a suitable and effective choice:

  • Enhanced Performance in Resource-Limited Environments:

    • In environments where computing resources are limited, such as on low-power devices or certain cloud instances, efficiently managing system resources is key. By blocking non-essential resources like images and heavy scripts, Selenium tasks can run more smoothly, consuming less CPU and memory. This is not only beneficial for the performance of the tasks themselves but also for the overall system performance.

    • In web scraping, especially when dealing with large volumes of data or multiple concurrent scraping operations, minimizing resource usage is crucial. Blocking unnecessary resources reduces the load on the system, enabling more efficient scraping processes. This is particularly important when scraping is performed on servers or services where resource usage directly impacts costs.

  • Automated Testing Optimization:

    • In continuous integration and deployment environments reducing the duration of testing cycles is essential. By blocking resources that are not critical to the functionality being tested, these cycles can be significantly accelerated. This results in quicker feedback loops, allowing teams to identify and address issues more rapidly.

    • In agile and DevOps setups, where development and operations are tightly integrated, efficiency and speed are paramount. Optimizing automated tests by blocking unnecessary resources contributes to a more streamlined development process. It allows teams to focus on critical aspects of the application under test, ensuring that resources are allocated to the most essential parts of the development and testing pipeline.

For more information check Web Scraping Without Getting Blocked

Conclusion​

In summary, blocking images and resources in Selenium automation is a powerful technique that significantly enhances performance, especially in web scraping and automated testing.

It leads to faster execution, reduced bandwidth usage, and more efficient resource management, making it an essential strategy in resource-limited environments and fast-paced development cycles. This approach, when used appropriately, can substantially optimize web automation tasks.

More Selenium Guides​

If you would like to learn more about Web Scraping, then be sure to check out: The Web Scraping Playbook

Check our other guides: