Robots.txt tip from Bing: Include all relevant directives if you have a Bingbot section
Pay close attention to the directives you give to Bingbot and other search engines in your robots.txt files. Frédéric Dubut, a senior program manager at Microsoft working on Bing Search, said on Twitter Wednesday that when you create a specific section in your robots.txt file for its Bingbot crawler, you should make sure to list all the default directives in that section. Specify directives for Bingbot. “If you create a section for Bingbot specifically, all the default directives will be ignored (except Crawl-Delay),” he said. “You MUST copy-paste the directives you want Bingbot…
Read More