This is the term for what spiders do, they will spider, or surf and index documents for a search engine.