Correct robots file for wordpress

First things that I do on my new websites is set correct robots.txt file. What does this file? It’s like directive for search engines robots what to see and what to ignore. Note, that this file is not 100% guarantee, because search engines spiders can ignore some rules, but usually this helps

All code below must be inserted in robots.txt file in utf-8 format. I attached copy to post. You must insert this file in root folder of your site. Also, many seo plugins for wordpress allows to create this file from admin page (for example, Seo By Yoast can do this)

In this file you must change yoursite.com to your domain

Some explanation. This file disallow comment pages, feed pages, tags pages, archives, trackbacks pages and system pages from spiders. Category pages are still allowed, but I recommend to add description to each category.

Correct robots file for wordpress
5 (100%) 1 vote

Want to get unique wordpress tips?

We won't rent or sell or spam your email. We promise!

Leave a Reply

1 Comment on "Correct robots file for wordpress"

Notify of
avatar
Sort by:   newest | oldest | most voted
Sinbrew
Guest

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: */trackback
Disallow: */*/trackback
Disallow: /feed/
Disallow: */*/feed/*/
Disallow: */feed
Disallow: /*?*
Disallow: /?s=
Sitemap: http://yoursite.com/sitemap.xml

wpDiscuz