A software program that crawls the web, searching and indexing web pages, is called a spider. “Web crawler” and “spider” are standard synonyms for the same type of automated program.
Search engines use spiders to visit web pages, follow hyperlinks and collect information about what they find. The downloaded content is processed and added to a search index, allowing users to retrieve relevant pages quickly when they enter a query. A spider does not usually understand a page exactly as a human reader does; it analyses signals such as text, links, metadata and page structure.
A crawler normally begins with a collection of starting addresses called seed URLs. As it examines each page, it discovers additional links and adds them to a crawl frontier. Search engines must decide which pages to visit, how often to revisit them and how quickly to make requests.
The term “spider” comes from the way the program appears to travel across a web of interconnected links. Site owners can provide instructions in a robots.txt file, although those instructions are requests rather than a complete technical barrier. Crawlers are also used for archiving, link checking and data collection.