Search Engine Optimization-Tell Google which pages you don’t want crawled
For non-sensitive information, block unwanted crawling by using robots.txt A robots.txt file tells search engines whether they can access and therefore crawl parts of your site. This file, which must be named robots.txt, is placed in the root directory of...