What is robots.txt? When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which web pages of your site should be indexed and which web pages should be ignored.