Crawler

A web crawler is a program, often called a bot or robot, which systematically browses the {{glossary("World Wide Web","Web")}} to collect data from webpages. Typically search engines (e.g., Google, Bing, etc.) use crawlers to build indexes.

See also

• Related glossary terms:
  • {{Glossary("Search engine")}}
• Web crawler on Wikipedia