Skip to main content

Python Selenium - Managing Cookies

Selenium Guide: Managing Cookies

Managing cookies is an important aspect of web automation, and Selenium provides methods to interact with cookies in various ways.

In this comprehensive guide, we will delve into the world of Selenium and explore practical strategies for handling cookies. We'll be covering:

Need help scraping the web?

Then check out ScrapeOps, the complete toolkit for web scraping.

Handling Cookies With Selenium​

Cookies are used to store information about your browser. They are used to remember your browser when you visit a website, track you as you navigate a website, and to analyze the usage of a website.

In Selenium, we have a whole set of methods devoted to managing cookies:

  • driver.add_cookie() adds a cookie to our stored cookies
  • driver.get_cookie(name_of_cookie) returns an individual cookie
  • driver.get_cookies() returns all cookies that are currently being stored in the browser
  • driver.delete_cookie(name_of_cookie) will delete a specific cookie from the browser
  • driver.delete_all_cookies() (as you probably guessed) clears all cookies from the browser

The code below shows how all these methods can be used:

from selenium import webdriver

#open an instance of chrome
driver = webdriver.Chrome()

#navigate to the website
driver.get("https://quotes.toscrape.com")

#get cookies and save as a variable
cookies = driver.get_cookies()

#print the current list of cookies
print("Cookies:", cookies)

#add a cookie
driver.add_cookie({"name": "our new cookie!", "value": "some random text!"})

#update our cookie variable
cookies = driver.get_cookies()

#print our cookies
print("Cookies after adding:", cookies)

#iterate through our list of cookies
for cookie in cookies:
    #if this line executes, we found our cookie
    if cookie["name"] == "our new cookie!":

        print("We have a match!")
        cookie["value"] = "we changed our text"

        #delete the old cookie
        driver.delete_cookie("our new cookie!")

        #replace it with the updated cookie
        driver.add_cookie(cookie)

#update the cookies variable
cookies = driver.get_cookies()

#print the cookies
print("Cookies after modifying:", cookies
)
#add another cookie
driver.add_cookie({"name": "another_cookie", "value": "we added a second cookie!"})

#update our list
cookies = driver.get_cookies()

#print the cookies
print("There should be two cookies here:", cookies)

#delete all cookies
driver.delete_all_cookies()

#update our list
cookies = driver.get_cookies()

#print the list
print("Cookies after deleting all:", cookies)
driver.quit()

Understanding Cookies​

Let's delve into the fundamentals of cookies to unravel their purpose, functionality, and the implications they hold for both users and developers.

What are Cookies and What do They Do?​

In the section above, we learned that all cookies have a name and a value. There are a number of other fields that make up a cookie. Let's take a look at some of the information that makes up a cookie:

  • domain is the website you were at when the cookie was created
  • name is the name of a cookie
  • value is the value that the cookie is storing
  • httpOnly is a boolean. If httpOnly is set to true, client-side scripts cannot access this data
  • path is the portion of the domain that can access the cookie. If the path is /, this means that it is on the root path (this cookie can be accessed from the entire domain)
  • sameSite is also used to restrict access to a cookie. sameSite can be set to None, Lax, or Strict
  • secure is another boolean. If secure is set to True, this cookie can only be sent as an encrypted request to the server using https

The cookies we created earlier used the attributes listed above. While cookies can contain more information, there are a couple more attributes we need to pay attention to:

  • session is another boolean that tells the site whether or not our cookie is used to store a session
  • expirationDate or expiry is the UNIX timestamp when a cookie expires
  • hostOnly is another boolean value. If hostOnly is set to False, subdomains of this site can also access the cookie.

Types of Cookies​

Cookies serve various purposes and can be categorized based on their functions, lifespan, and origin. Here are some common types of cookies:

Session Cookies are stored and used to manage a user session. When you close a browser, Session Cookies are removed when you close the browser. These cookies are used specifically to store information about your browsing session such as pages you visited.

Persistent Cookies are stored on a device even after the browser is closed. These cookies typically store things like login information and user preferences on a site.

How are Cookies Stored and Accessed by the Browser?​

As we did earlier, we can retrieve cookies using Selenium. You can also view cookies stored from right inside your browser.

While not necessarily relevant to scraping, this is important for anyone who cares about their data when browsing the web.

To view cookies in Chrome follow the steps below:

  • click the icon to the left of your url

Python Selenium Guide - Site Info

  • After clicking this button, select Cookies and Site Data

Python Selenium Guide - Cookies and Site Data pop up

  • Select Manage cookies and site data

Python Selenium Guide - Manage Cookies and Site Data

  • Once you've selected to manage your cookies, you get a pop-up with a list of sites that can access your data

Python Selenium Guide - List of sites that can access your data

  • To delete a cookie, simply click the trashcan icon

Why Manage Cookies in Selenium?​

There are many reasons to manage our cookies in Selenium. You can use them for managing credentials and for managing how your automated browser is tracked. Any type of stored browser information typically gets stored in cookies.

With proper cookie management, we can keep logged into a site or we can clear all information from a site and log out much more quickly and effectively. We can also remove cookies to make it more difficult to track our browser.

While we use Selenium primarily for scraping, there are many developers who use it to automate testing on their sites.

When using Selenium for automated site tests, the ability to quickly manage cookies makes the login and logout process much more efficient (just like it does for us when scraping). Imagine writing unit tests for a social media site and having to hard code user authentication every time you need to do something different!

Cookie management is a critical skill when dealing with the following scenarios:

  • User Authentication: logging into a website
  • Session Handling: remembering specifics about a browsing session
  • Site with persistent login: sites that "remember" you when you revisit the site
  • Testing site permissions: when testing access points on a site, it is imperative that only permitted users get to access sensitive data. Selenium allows developers to automate tests around this and ensure that their sites are functioning properly.
  • Managing dynamic content: when viewing a site with dynamic content, the content generated by the site can often be generated by information stored in the browser, such as a store's website saving your location to display your nearest store

Selenium Query Commands for Cookies​

In Selenium, you can use various commands and methods to interact with cookies. Here's a list of common Selenium WebDriver commands for managing cookies in Python:

Selenium Query CommandsOutput
driver.get_cookies();Return The List of all Cookies
driver.get_cookie(name_of_cookie)Return specific cookie according to name
driver.add_cookie({"name": name_of_cookie, "value": value_of_cookie})Create and add the cookie
driver.delete_cookie(name_of_cookie)Delete specific cookie
driver.delete_all_cookies()Delete all cookies

Methods for Handling With Cookies​

Here are some common methods and operations for handling cookies in Selenium:

Retrieving cookies​

The process of retrieving cookies involves obtaining information about the cookies currently stored for a specific domain in the browser. This can include fetching all cookies or getting details about a particular cookie by its name.

from selenium import webdriver

#open chrome
driver = webdriver.Chrome()

#navigate to the url
driver.get("https://quotes.toscrape.com")

#create a variable to hold our cookies
cookies = driver.get_cookies()

#print the cookies list
print("Cookies:", cookies)

#close the browser
driver.quit()

In the code above, we:

  • Open Chrome and navigate to the page with webdriver.Chrome() and driver.get()
  • Create a variable, cookies, using driver.get_cookies()
  • Print cookies to the terminal so we can see them

Adding cookies​

Adding cookies is the action of setting new cookies in the browser for a specific domain.

from selenium import webdriver

#open chrome
driver = webdriver.Chrome()

#navigate to the url
driver.get("https://quotes.toscrape.com")

#add a new cookie
driver.add_cookie({"name": "new_cookie", "value": "added_cookie"})

#create a variable to hold our cookies
cookies = driver.get_cookies()

#print the cookies list
print("Cookies:", cookies)

#close the browser
driver.quit()

In this code, we:

  • Open a browser and navigate to the url with webdriver.Chrome() and driver.get()
  • Add a cookie with the name of new_cookie and the value of added_cookie
  • Get our list of cookies with driver.get_cookies()
  • Print the list so we can see the new cookie we just added

Deleting cookies​

The deletion of cookies involves removing either a specific cookie or all cookies associated with a particular domain from the browser. This action helps manage the state and session information stored in cookies.

from selenium import webdriver

#open chrome
driver = webdriver.Chrome()

#navigate to the url
driver.get("https://quotes.toscrape.com")

#add a new cookie
driver.add_cookie({"name": "new_cookie", "value": "added_cookie"})

#create a variable to hold our cookies
cookies = driver.get_cookies()

#print the cookies list
print("Cookies after adding:", cookies)

#delete the new cookie
driver.delete_cookie("new_cookie")

#update the cookies variable
cookies = driver.get_cookies()
print("Cookies after deleting:", cookies)

#close the browser
driver.quit()

In this example, we:

  • Open a browser and navigate to the url with webdriver.Chrome() and driver.get()
  • Add a cookie with driver.add_cookie({"name": "new_cookie", "value": "added_cookie"})
  • Create a cookies variable and print our cookies list to the terminal
  • Delete the cookie with driver.delete_cookie("new_cookie")
  • Print the newly emptied cookies to the terminal

Modifying cookies (by changing key-value pairs)​

Modifying cookies encompasses altering the values or properties of existing cookies, such as changing the key-value pairs.

This process involves retrieving a specific cookie, creating a new cookie with the desired modifications, and then adding the modified cookie back to the browser.

from selenium import webdriver

#open chrome
driver = webdriver.Chrome()

#navigate to the url
driver.get("https://quotes.toscrape.com")

#add a new cookie
driver.add_cookie({"name": "new_cookie", "value": "added_cookie"})

#create a variable to hold our cookies
cookies = driver.get_cookies()

#print the cookies list
print("Cookies after adding:", cookies)

#iterate through our cookies
for cookie in cookies:
    #find the cookie we just created
    if cookie["name"] == "new_cookie":

        #change the value of the cookie in the list
        cookie["value"] = "modified_cookie"

        #delete the original cookie
        driver.delete_cookie(cookie["name"])

        #replace it with the modified one
        driver.add_cookie(cookie)

#update the cookies variable
cookies = driver.get_cookies()
print("Cookies after deleting:", cookies)

#close the browser
driver.quit()

In the code above, we:

  • Open an instance of Chrome and navigate to the page with webdriver.Chrome() and driver.get()
  • Add a cookie with driver.add_cookie()
  • Print the cookies list
  • Iterate through our cookies list and find the cookie with the name we're searching for
  • Once we've found our cookie in the list, we modify the value
  • Remove the original cookie with driver.delete_cookie() and replace it with the modified cookie with driver.add_cookie()
  • Print the cookies to the terminal so we can view our modified cookie

Anytime we wish to edit cookie attributes, we do this by changing the key-value pairs. Our cookies are stored in a JSON object or as we would say in Python, a dictionary.

In the example above, when we changed the value of the cookie,could have changed other attributes (keys) and their values.

  • To change the expiry date, instead of: cookie["value"] = "modified_cookie, we could use cookie["expirationDate"] = some_new_timestamp

  • If we wanted to change the domain name and path, we could do cookie["domain"] = some_new_domain

  • To change the path, we would use cookie["path"] = "/our/new/path/

  • To use the secure flag, we would use cookie["secure"] = True

  • To set httpOnly, we would use cookie["httpOnly"] = True

Working with Multiple Cookies In Selenium​

As we've gone over in the sections above, we can use driver.get_all_cookies() to retrieve all of our cookies. This method gets each cookie in the current browser and returns all the cookies in the form of a list.

If we wish to filter cookies based on certain data, we iterate through our list and we looking for a certain attribute to target. Take a look at the snippet below that we used in one of the previous sections of this article:

for cookie in cookies:
    #find the cookie we just created
    if cookie["name"] == "new_cookie":

        #change the value of the cookie in the list
        cookie["value"] = "modified_cookie"

        #delete the original cookie
        driver.delete_cookie(cookie["name"])

        #replace it with the modified one
        driver.add_cookie(cookie)

We iterate through the list and use if cookie["name"] == "new_cookie" to find our target. If we wanted all the cookies from quotes.toscrape.com, we could do this:

for cookie in cookies:
    #find the cookie we just created
    if cookie["domain"] == "quotes.toscrape.com":

        #change the value of the cookie in the list
        cookie["httpOnly"] = True

        #delete the original cookie
        driver.delete_cookie(cookie["name"])

        #replace it with the modified one
        driver.add_cookie(cookie)

We've now modified this snippet to take all cookies from quotes.toscrape.com and switch httpOnly to True.

When using cookies for login and authentication, it is important to save them. If you wish to use them longterm, you may even want to save them to a file. Once our cookies are loaded into the brower, we can usually just refresh the page and we'll be logged in automatically.

While you need to push different buttons to get cookies using your normal browser. Selenium's builtin cookie methods give you a universal way to create, read, update and delete your cookies no matter which browser Selenium is controlling. To remove all cookies, we can simply use the delete_all_cookies() method and this will clear the browser cache.

When managing cookies across multiple domains, we can simply filter through our cookie list and use key-value pairs to target a specific domain. If we want only cookies from a certain site, we can use get_cookies() to retrieve all our cookies and iterate through it with a for loop to target our cookies by domain name.

Common challenges when handling cookies are typically related to expiration or adding cookies from a previous into our current one.

In order to properly handle any exceptions thrown, it is best to use try to attempt to add a cookie into the browser and a catch statement to handle any exceptions that may get thrown.

This combination of try and catch allows our script to continue execution should any errors occur. If you are experiencing any performance issues, it is always a good idea clear your cookies.

If you wish to login to certain sites effectivley and efficiently, you may want to save your cookies to a file, this way you can come back and read them any time you need to.

In the example below, we're going to take an in-depth look at how we can use cookies for authentication.

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep

#open chrome
driver = webdriver.Chrome()

#navigate to the url
driver.get("https://www.linkedin.com")

#save username and password
username = "YOUR-USERNAME@SOME_EMAIL.com"
password = "YOUR-SUPER-SECRET-PASSWORD"

#find the username element
username_id = driver.find_element(By.ID, "session_key")

#find the password element
password_id = driver.find_element(By.ID, "session_password")

#fill in the username
username_id.send_keys(username)

#fill in the password
password_id.send_keys(password)

#xpath of the signin button
signin_button_xpath = '//*[@id="main-content"]/section[1]/div/div/form/div[2]/button'

#find the signin button by xpath
signin_button = driver.find_element(By.XPATH, signin_button_xpath)

#click the signin button
signin_button.click()

#create a variable to hold our cookies
cookies = driver.get_cookies()

#close the browser
driver.close()

#open a new browser
driver = webdriver.Chrome()

#go back to the site
driver.get("https://linkedin.com")

#add cookies from the previous section
for cookie in cookies:
    print(cookie)
    driver.add_cookie(cookie)

#sleep so user can see we're not logged in
sleep(5)

#refresh the page
driver.refresh()

#sleep so user can see we're logged in
sleep(5)

#close the browser
driver.quit()

In the example above, we:

  • Open Chrome and navigate to the page with webdriver.Chrome() and driver.get()
  • Next we find the username and password objects using their ID
  • We fill these objects with the username and password variables using the send_keys() method
  • Next we use XPATH to find and click() the login button
  • After signing in, we retrieve all of our cookies with driver.get_cookies() and save them as a variable
  • Next, we close the browser with driver.close()
  • We then open up a new browser and head back to the page
  • We add our cookies from the previous session to the current session with driver.add_cookie()
  • We sleep() for 5 seconds just so you can see that we are not logged in yet
  • We then use driver.refresh() and you can see that we're logged in

Now let's use cookie management to handle a SPA (Single Page Application). Believe it or not, Github is a single page application. Let's use cookies to interact with Github!

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep

driver = webdriver.Chrome()

#navigate to the url
driver.get("https://www.github.com/login")

#save username input box as variable
username_input = driver.find_element(By.NAME, "login")

#save password input box as variable
password_input = driver.find_element(By.NAME, "password")

#send email to username_input
username_input.send_keys("YOUR-EMAIL@example.com")

#send password to password_input
password_input.send_keys("YOUR-PASSWORD-HERE")

#find the signin button
signin_button = driver.find_element(By.NAME, "commit")

#click the signin button
signin_button.click()

#create a bool to know if we're logged in
logged_in = False

#loop until the user says we're logged in
while not logged_in:
    print("Are you logged in? y/n")
    resp = input()
    if resp == "y":
        logged_in = True

#create a variable to hold our cookies
cookies = driver.get_cookies()

#close the browser
driver.close()

#open a new browser
driver = webdriver.Chrome()

#go back to the site
driver.get("https://github.com")

#add cookies from the previous section
for cookie in cookies:
    print(cookie)
    try:
        driver.add_cookie(cookie)
    except:
        print("failed to add cookie")

#sleep so user can see we're not logged in
sleep(5)

#refresh the page
driver.refresh()

#sleep so user can see we're logged in
sleep(5)

#close the browser
driver.quit()

Github uses 2 Factor Authentication, when running the script above, make sure to login using your 2FA before you input y to tell your script that you've logged in.

This script stops and pauses when you need to enter your 2FA. We input y to let the script know that you've finished entering your 2FA and that the script is ready to continue its execution.

In the example above, we:

  • Open an instance of Chrome and navigate to the site
  • We use Selenium's driver.find_element() in order to find our username_input and password_input by NAME and save them as variables
  • We then use the send_keys() method to fill the required input fields
  • After filling the inputs, we find the signin_button using find_element(By.NAME, "commit")
  • We click signin_button with signin_button.click()
  • After we click() to sign in, we create a loop that exits once we've told the bot that we're logged in
  • Once logged in, we save our cookies as a variable with the get_cookies() method
  • After we've saved our cookies, we navigate back to the page and try to add them all back in, we use except to handle any exceptions
  • After adding the cookies back in, we refresh the page, and we're back into Github at lightning fast speed
  • Our page loads so quickly because it's a Single Page Appliction and everything is handled dynamically based on the the cookies stored inside the browser

Conclusion​

You've reached the end of this article. You should have a solid grasp on adding, deleting, and even modifying cookies. You now know how to save them as variables and use them in future sessions.

You've even logged into LinkedIn and Github using cookies! Now that you've harnessed the power of cookies in Selenium, go use your knowledge to scrape something cool!

More Selenium Articles​

Want to learn more, but you're unsure where to start? Read one of the articles below!