What file tells web crawlers which parts of a website they may access?

The file that tells web crawlers which parts of a website they may access is robots.txt.

Robots.txt is the familiar filename for the Robots Exclusion Protocol, a set of instructions placed at a site’s root, such as example.com/robots.txt. It can identify particular crawlers with a User-agent line and specify permitted or disallowed paths with directives such as Allow and Disallow. Search-engine crawlers commonly check this file before requesting other pages.

The standard was proposed by Martijn Koster in 1994, when uncontrolled crawlers could place heavy loads on web servers. It became a widely followed convention and was formally standardised as IETF RFC 9309 in 2022. A robots.txt file can also point crawlers toward XML sitemaps, though a sitemap has a different purpose: it lists URLs rather than primarily restricting crawling.

Robots.txt is advisory, not a security barrier. Cooperative crawlers follow it, but malicious software can ignore it. It also does not guarantee that a blocked URL will disappear from search results; preventing indexing generally requires separate controls such as noindex directives.