This is a type of text file that is represented by .txt extension and it informs spiders of the pages that it cannot index. When a spider enters a site to crawl it will first ask for the robots.txt file, this will then show the spider where on that website it can't go. However if this is only the case if the robot follows the Robots Exclusion Program.