Correct robots file for wordpress
First things that I do on my new websites is set correct robots.txt file. What does this file? It’s like directive for search engines robots what to see and what to ignore. Note, that this file is not 100% guarantee, because search engines spiders can ignore some rules, but usually this helps
All code below must be inserted in robots.txt file in utf-8 format. I attached copy to post. You must insert this file in root folder of your site. Also, many seo plugins for wordpress allows to create this file from admin page (for example, Seo By Yoast can do this)
User-agent: * Disallow: /cgi-bin/ Disallow: */trackback Disallow: */comment- Disallow: *?replytocom= Disallow: */feed Disallow: /?s= Disallow: /xmlrpc.php Disallow: /archives/date/ Disallow: /archives/tag/ Disallow: /archives/author/ Disallow: /page/ Disallow: /tag/ Allow: /wp-content/uploads/ Host: yoursite.com User-agent: Googlebot-Image Allow: /wp-content/uploads/ User-agent: ia_archiver Disallow: /
In this file you must change yoursite.com to your domain
Some explanation. This file disallow comment pages, feed pages, tags pages, archives, trackbacks pages and system pages from spiders. Category pages are still allowed, but I recommend to add description to each category.