Robots.txt is a convention that stops web spiders and other robots from gaining access to all or certain parts of a web site. The specifications for this are found in the top level directory contained in the web site.