Spiders work in the robots.txt protocol and they are used to crawl or spider the web looking for pages to include in the search engines index.