A crawler is another name for a spider, which is a robot that crawls websites making note of links and other information, it then will sue this information to index the site on a search engine.