The robots.txt file of Twitter has clear instructions prohibiting any form of crawling or scraping by entities not specifically whitelisted. All user agents other than a few specific ones such as 'googlebot' and 'bingbot' are disallowed from accessing any part of the website (Disallow: /). This makes it clear that Twitter does not allow any form of web scraping for general or public user agents beyond these specific, trusted web crawlers.

The next thing to notice is the specificity in the parts that are allowed for the trusted web crawlers. URLs with specific patterns like Allow: /i/streams/profile/* for 'googlebot' show that only Google is allowed to crawl specific directories of Twitter's website based on the pattern mentioned in the Allow: rule. There is also Disallow: /search/realtime directive for 'googlebot', which means that real-time search results pages are off-limits even for the trusted 'googlebot'. From a web scraping perspective, these rules indicate that Twitter is very strict about who is allowed to crawl and/or scrape its website.

Twitter's terms of service specify that data collection is essentially not authorised without prior permission. The guidelines specify that "you may not do, or attempt to do... scrape the Services or scrape content from the services". This statement is clearly designated towards prohibiting any form of web scraping or automated data collection activities without Twitter’s explicit consent.

Even though web scraping is generally prohibited, there do exist certain provisions for accessing Twitter data. Twitter provides API access, however, it's clearly mentioned that "If you provide an API that enables third parties to interact with or access our services, you agree to comply with our API rules and you agree to terms and conditions of Twitter API". These terms place the onus on any entity interacting with their data through APIs, to follow Twitter's regulations vigilantly. Any infringement of these rules could lead to penalties including account termination.