The robots.txt file for YouTube contains an extensive set of directives that restrict access for most automated crawling. The file includes numerous rules such as Disallow: /channel, Disallow: /playlist, and Disallow: /watch, effectively barring access to key areas of the site for regular web scrapers. These rules are applicable to all generic user agents, with exceptions made for whitelisted bots like Googlebot and Bingbot.

A select few paths are explicitly allowed, including Allow: /s/$, Allow: /s/img, and Allow: /m/$, but these directives don’t grant access to significant sections of the site. References to sitemaps are also present in the file. Practical implication for typical web scrapers from this configuration is that access is highly restricted, while access for search engines and certain other bots is maintained for indexing purposes. Overall, the robots.txt configuration indicates a restrictive approach towards generic web scraping, permitting limited access to a few select sections.

The terms of service for YouTube include explicit statements about automated access and data extraction. The terms state:

“access the Service using any automated means (such as robots, botnets or scrapers) except (a) in the case of public search engines, in accordance with YouTube’s robots.txt file; or (b) with YouTube’s prior written permission;”

This broadly restricts scraping and other automated activity across the entire “Service,” which covers both public and logged-in areas, unless you fall under the public search engine exception or have prior written permission. The terms also prohibit using content beyond what is “expressly authorized by the Service” or permitted by written permission, which restricts bulk downloading or reuse outside provided features. Enforceability can vary based on whether a user has explicitly agreed to the terms (for example, via account creation), but YouTube frames these restrictions as universally applicable to all use of the Service.

YouTube provides official APIs (for example, the YouTube Data API) as an authorized channel for programmatic access. The terms also address attempts to bypass technical or access controls:

“circumvent, disable, fraudulently engage with, or otherwise interfere with any part of the Service (or attempt to do any of these things), including security-related features or features that (a) prevent or restrict the copying or other use of Content or (b) limit the use of the Service or Content;”

and outline consequences for violations:

“YouTube reserves the right to suspend or terminate your Google account or your access to all or part of the Service…”

This means bypassing barriers like logins, rate limits, or CAPTCHAs would violate the terms, and consequences can include access restriction or account termination, with potential legal exposure under the indemnity and other legal terms. Practically, scraping is forbidden unless you qualify under the public search engine exception, use the embeddable player or official API as authorized, or obtain prior written permission—making it only permissible under specific conditions.