search engine spider

Software that gathers information about websites for a search engine. Also called a "web spider", "web crawler" or "bot" (short for "robot" - software that automatically does repetitive tasks).

The spider browses the World Wide Web at regular intervals. It starts at heavily used sites and follows links to other sites. The spider "crawls" from site to site, and page to page within a site, "reading" the content as it goes.