A robots.txt file is a text file that stops web crawler software, such as Googlebot, from crawling certain pages of your site. The file is essentially a list of commands, such Allow and Disallow, that tell web crawlers which URLs they can or cannot retrieve.

So, if a URL is disallowed in your robots.txt, that URL and its contents will not appear in Google Search results.

Robots.txt Tester Tool

You need a robots.txt file only if your site includes content that you don’t want Google or other search engines to index. To let Google index your entire site, you don’t need a robots.txt file (not even an empty one).

To test which URLs Google can and cannot access on your website, go to Webmaster Tools > Crawl > robots.txt Tester.

Google announced that they had updated the Robots.txt tester tool within Google Webmaster Tools.

This tool will provide you three important data such as:

1. Highlighting the lines in your robots.txt file that is blocking a specific page.

2. You can test changes to the robots.txt tool and see if there is any error before making it available to search engines.

3. You will be also able to view the older versions of your robots.txt so that you can check any mistake that you have made in the past.

It is recommended that you check this tool, even if you’re sure that your robots.txt file is fine. Some of these issues can be subtle and easy to miss. While you’re at it, also double-check how the important pages of your site render with Googlebot, and if you’re accidentally blocking any JS or CSS files from crawling.