Selenium Guide: How To Find Elements by CSS Selectors
CSS Selectors provide a powerful and flexible way to pinpoint elements, enabling precise interactions in your browser automation scripts.
In this comprehensive guide, we will demystify the art of locating HTML elements using CSS Selectors with Selenium.
- TLDR
- CSS Selectors Demystified
- Finding CSS Selectors
- Finding One Element
- Finding All Elements
- Explicit Wait
- Using Basic CSS Selectors
- Using Attribute CSS Selectors
- Combining CSS Selectors
- Debugging and Troubleshooting
- Best Practices
- Conclusion
- More Web Scraping Guides
TLDR: How to Select Elements by CSS with Selenium
When selecting elements with CSS in Selenium, we use Selenium's single find_element()
method or multiple find_elements()
method. Take a look below to see this in action.
from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
#open Chrome
driver = webdriver.Chrome()
#navigate to the site
driver.get("https://quotes.toscrape.com")
#find the h1 element
h1 = driver.find_element(By.CSS_SELECTOR, "h1")
#print the h1 element
print(f"H1 element: {h1.text}")
#find all quotes (span elements with the class: 'text')
quotes = driver.find_elements(By.CSS_SELECTOR, "span[class='text']")
for quote in quotes:
print(f"Quote: {quote.text}")
#find all tags (anything with the class 'tag')
tags = driver.find_elements(By.CSS_SELECTOR, ".tag")
for tag in tags:
print(f"Tag: {tag.text}")
#find the login link (first a element with href: '/login')
login_link = driver.find_element(By.CSS_SELECTOR, "a[href='/login']")
login_link.click()
#find the username box (element with ID: 'username')
username = driver.find_element(By.CSS_SELECTOR, "#username")
username.send_keys("some text")
#find the password box (input elements with the type: 'password')
password = driver.find_element(By.CSS_SELECTOR, "input[type='password']")
password.send_keys("my super secret password")
#sleep 5 seconds so the user can see the filled input boxes
sleep(5)
driver.quit()
Method | Description |
---|---|
find_element(By.CSS_SELECTOR, "my_selector") | Finds the first element that matches. If no element matches the selector, a NoSuchElementException will be raised. |
find_elements(By.CSS_SELECTOR, "my_selector") | Finds all elements that match the CSS selector and return them in a list, returns an empty list if no element is found. |
CSS Selectors Demystified
Cascading Style Sheets (CSS) is a layout for styling webpages written in HTML. When styling with CSS, developers use certain selectors to target specific items on a webpage.
For instance, if we make a custom CSS class and name it our-custom-class, our CSS may look like this:
.our-custom-class {
background: black; /*set the background to black*/
padding-right: 24px; /*24 pixels of padding on the sides*/
color: white; /*set the font color to white*/
height: 100vh; /*vh means viewport height---set the height to 100% of the viewport */
display: flex; /*use a flex display for elements within this element*/
justify-content: center; /*put content in the center*/
align-items: center; /*align nested items to the center of this element */
}
Now let's add an HTML page that links to our CSS page:
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>Our Demo Page</title>
<link rel="stylesheet" type="text/css" href="demo.css">
</head>
<body>
<h1>Some Words</h1>
<p>These are some smaller words.</p>
</body>
</html>
If we open the page in our browser, it will look like the image below:
Now let's add our class to our body element:
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>Our Demo Page</title>
<link rel="stylesheet" type="text/css" href="demo.css">
</head>
<body class="our-custom-class">
<h1>Some Words</h1>
<p>These are some smaller words.</p>
</body>
</html>
Now that we've styled the element, it looks like this:
When searching for CSS selectors on an existing page, we can search and select items with the following criteria:
- Tag Name
- ID
- Class
- Attribute
- Descendant
- Child
- Adjacent Sibling
- General Sibling
- Pseudo-class
- Pseudo-element
Selector Type | CSS Syntax | Description |
---|---|---|
Tag Name | tag | Selects elements by their tag. |
ID | #id | Selects an element by its ID. |
Class | .class | Selects elements by their class. |
Attribute | [attribute=value] | Selects elements with a specific attribute. |
Descendant | ancestor descendant | Selects an element within another element. |
Child | parent > child | Selects direct children of an element. |
Adjacent Sibling | previous + next | Selects an element directly after another. |
General Sibling | sibling ~ sibling | Selects all siblings of a specified element. |
Pseudo-class | element:pseudo-class | Selects elements in a certain state. |
Pseudo-element | element::pseudo-element (Rare in Selenium, as Selenium focuses on the DOM structure, not rendering). | Selects specific parts of an element. |
Finding CSS Selectors
To find CSS Selectors on a page, first we must inspect the page through the developer console. Simply right-click the element and choose to copy the CSS selector:
Finding One Element Using CSS Selector
If we'd like to find just one element, we can use Selenium's find_element()
method to find and return a single element. The code below finds a single element and prints its text to the terminal.
from selenium import webdriver
from selenium.webdriver.common.by import By
#open an instance of Chrome
driver = webdriver.Chrome()
#navigate to the page
driver.get("https://quotes.toscrape.com")
#selector for the h1 element
selector = "h1"
#find element with this selector
element = driver.find_element(By.CSS_SELECTOR, selector)
#print the text of the element
print(element.text)
#close the browser
driver.quit()
In the code above, we:
- Open Chrome with
webdriver.Chrome()
- Navigate to the page with
driver.get()
- Create a selector:
"h1"
- Find the element with
driver.find_element()
- Print the text of the element to the terminal
- Close the browser with
driver.quit()
Finding All Elements Using CSS Selector
Now that we have a basic understanding of CSS selectors, let's apply this new knowledge. The code example below finds all tag
elements on the page.
from selenium import webdriver
from selenium.webdriver.common.by import By
#open an instance of Chrome
driver = webdriver.Chrome()
#navigate to the page
driver.get("https://quotes.toscrape.com")
#look for custom elements named "tag"
selector = ".tag"
#find all elements with this selector
elements = driver.find_elements(By.CSS_SELECTOR, selector)
#print the text of each element
for element in elements:
print(element.text)
#close the browser
driver.quit()
In the code above, we:
- Open an instance of Chrome with
webdriver.Chrome()
- Navigate to the page with
driver.get()
- Create a custom selector,
".tag"
- Find all
.tag
elements withdriver.find_elements()
and return them in a list - Print the text of each list to the terminal
- CLose the browser with
driver.quit()
Explicit Wait: Using WebDriverWait with CSS Selectors
When interacting with dynamic elements, it is imperative that we use proper waiting strategies. Sometimes elements will not be interactable until we're hovering over them, sometimes elements won't be visble until we perform a certain action.
If we don't properly wait for these things to happen, Selenium will throw an exception and our scraper will crash.
Wait Until an Element is Present
In this section, we'll wait until an element is present and then find the element. In the example below, we find the menu button on the screen and then click on it.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
#open Chrome
driver = webdriver.Chrome()
#navigate to the site
driver.get("https://espn.com")
#id selector for the menu button
selector = "#global-nav-mobile-trigger"
#create an ActionChains object
actions = ActionChains(driver)
#wait until the menu button appears
open_menu = WebDriverWait(driver, 5).until(
EC.presence_of_element_located((
By.CSS_SELECTOR, selector))
)
#perform a chain of actions
actions\
.move_to_element(open_menu)\
.click()\
.perform()
#close the browser
driver.quit()
In the example above, we:
- Open Chrome
- Navigate to the site
- Use
WebDriverWait
to wait for the menu button to appear - Create an
ActionChain
that clicks on the menu button - Close the browser
Wait Until an Element is Visible
This code builds on our previous example and makes sure to find the menu after we've clicked on it and it is now visible.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
#open Chrome
driver = webdriver.Chrome()
#navigate to the site
driver.get("https://espn.com")
#id selector for the menu button
selector = "#global-nav-mobile-trigger"
#create an ActionChains object
actions = ActionChains(driver)
#wait until the menu button appears
open_menu = WebDriverWait(driver, 5).until(
EC.presence_of_element_located((
By.CSS_SELECTOR, selector))
)
#perform a chain of actions
actions\
.move_to_element(open_menu)\
.click()\
.perform()
#id selector for the menu that appears
menu_selector = "#global-nav-mobile"
#wait until the menu appears
menu = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((
By.CSS_SELECTOR, menu_selector
))
)
#print the contents of the menu
print(menu.text)
#close the browser
driver.quit()
In the code example above, we:
- Create an instance of Chrome
- Navigate to the site
- Create a selector for the menu button with
selector = "#global-nav-mobile-trigger"
- Wait for the element to appear with
WebDriverWait
- Perform a chain of actions to
move_to_element(open_menu).click().perform()
- Use another instance of
WebDriverWait
for the actual menu to appear - Print the contents of the menu to the terminal
- Close the browser
Wait Until an Element is Clickable
In the previous example, if we were to click any of the menu items, we could get an exception because the element is not interactable. Most often, when these types of errors happen, it is because we need to hold the cursor over whichever element we're trying to select.
The following example builds on top of the previous two, and we'll use ActionChains
to accomplish our task. First we'll find the element we want to click, then we'll move to the element, and click on it.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
#open Chrome
driver = webdriver.Chrome()
#navigate to the site
driver.get("https://espn.com")
#id selector for the menu button
selector = "#global-nav-mobile-trigger"
#create an ActionChains object
actions = ActionChains(driver)
#wait until the menu button appears
open_menu = WebDriverWait(driver, 5).until(
EC.presence_of_element_located((
By.CSS_SELECTOR, selector))
)
#perform a chain of actions
actions\
.move_to_element(open_menu)\
.click()\
.perform()
#id selector for the menu that appears
menu_selector = "#global-nav-mobile"
#wait until the menu appears
menu = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((
By.CSS_SELECTOR, menu_selector
))
)
#css selector for the NFL button
nfl_selector = "#global-nav-mobile > ul > li.active > ul > li:nth-child(2) > a > span.link-text"
#find the NFL button and wait until present
nfl_button = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((
By.CSS_SELECTOR, nfl_selector
))
)
#perform a chain of actions to click the element
actions\
.move_to_element(nfl_button)\
.click()\
.pause(4)\
.perform()
#close the browser
driver.quit()
This example is pretty much the same as the two previous with a few exceptions:
- We save the CSS selector of the "NFL" button
- We then find the button with
find_element()
- We perform another
ActionChain
that:- Moves to the button with
move_to_element()
clicks()
on the elementpause()
for 4 seconds so we can view the screen changing after we click
- Moves to the button with
Using Basic CSS Selectors
This section is devoted to showing off the flexibility of CSS selectors. There are code examples to find all sorts of elements on a page using different CSS criteria.
Find By ID
The ID attribute provides a unique identifier for an HTML element, making it a reliable and direct way to locate specific elements on a webpage.
As we have done previous examples, we can find an element by its ID using the #
character.
from selenium import webdriver
from selenium.webdriver.common.by import By
#open Chrome
driver = webdriver.Chrome()
#navigate to the site
driver.get("https://quotes.toscrape.com/login")
#find the username by ID
username = driver.find_element(By.CSS_SELECTOR, "#username")
#type stuff in the box
username.send_keys("some text")
#close the browser
driver.quit()
In the example above, we:
- Open Chrome and navigate to the site
- Find the username element using its CSS selector
- Type in the box with
send_keys()
- Close the browser
Find By Class Name
The class name serves as an identifier for the element, allowing developers to apply consistent styling or behavior to multiple elements across a webpage.
This example performs the same task as the previous one, but we find the username element by its class name instead.
from selenium import webdriver
from selenium.webdriver.common.by import By
#open Chrome
driver = webdriver.Chrome()
#navigate to the site
driver.get("https://quotes.toscrape.com/login")
#find the username by class name
username = driver.find_element(By.CSS_SELECTOR, ".form-control")
#type stuff in the box
username.send_keys("some text")
#close the browser
driver.quit()
The only difference between this example and the previous one:
#username
gets replaced by.form-control
Find By Tag Name
HTML elements are defined by tags, and each tag has a specific name that denotes the type of element it represents.
Now let's find the same element, but we'll find it using its tag name instead.
from selenium import webdriver
from selenium.webdriver.common.by import By
#open Chrome
driver = webdriver.Chrome()
#navigate to the site
driver.get("https://quotes.toscrape.com/login")
#find the username by tag
username = driver.find_element(By.CSS_SELECTOR, "div > input")
#type stuff in the box
username.send_keys("some text")
#close the browser
driver.quit()
Using Attribute CSS Selectors
When using CSS selectors, we can also use the attributes of our element as criteria.
Attribute Presence
The example below finds all elements that contain an ID and then prints their ID to the console.
from selenium import webdriver
from selenium.webdriver.common.by import By
#open Chrome
driver = webdriver.Chrome()
#navigate to the site
driver.get("https://quotes.toscrape.com/login")
#find all elements that contain an ID
new_list = driver.find_elements(By.CSS_SELECTOR, "[id]")
#print the id of each element
for item in new_list:
print(item.id)
#close the browser
driver.quit()
Attribute Value
This example uses the previous example, but instead, we only search for elements where the ID is username.
from selenium import webdriver
from selenium.webdriver.common.by import By
#open Chrome
driver = webdriver.Chrome()
#navigate to the site
driver.get("https://quotes.toscrape.com/login")
#find all elements that contain the ID, username
new_list = driver.find_elements(By.CSS_SELECTOR, "[id='username']")
#print the id of each element
for item in new_list:
print(item.id)
#close the browser
driver.quit()
This approach allows for great flexibility in finding many items by changing a few simple variables or keywords. The only difference between this example and our previous one:
"[id]"
gets replaced with"[id='username']"
Attribute Contains
We can also look for attributes that contain certain information by adding the *
character.
from selenium import webdriver
from selenium.webdriver.common.by import By
#open Chrome
driver = webdriver.Chrome()
#navigate to the site
driver.get("https://quotes.toscrape.com/login")
#find all elements that contain the ID, username
new_list = driver.find_elements(By.CSS_SELECTOR, "[id*='user']")
#print the id of each element
for item in new_list:
print(item.id)
#close the browser
driver.quit()
In this example, we once again change on thing:
"[id='user']"
becomes"[id*='user']"
to search for elements that contain the string "user" in the ID