Correct robots file for wordpress

First things that I do on my new websites is set correct robots.txt file. What does this file? It’s like directive for search engines robots what to see and what to ignore. Note, that this file is not 100% guarantee, because search engines spiders can ignore some rules, but usually this helps

All code below must be inserted in robots.txt file in utf-8 format. I attached copy to post. You must insert this file in root folder of your site. Also, many seo plugins for wordpress allows to create this file from admin page (for example, Seo By Yoast can do this)

In this file you must change yoursite.com to your domain

Some explanation. This file disallow comment pages, feed pages, tags pages, archives, trackbacks pages and system pages from spiders. Category pages are still allowed, but I recommend to add description to each category.

Subscribe
Notify of
guest
2 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Sinbrew
4 years ago

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: */trackback
Disallow: */*/trackback
Disallow: /feed/
Disallow: */*/feed/*/
Disallow: */feed
Disallow: /*?*
Disallow: /?s=
Sitemap: http://yoursite.com/sitemap.xml

Daniel
4 months ago
Reply to  Sinbrew

The perfect robot, you’re welcome 😉 * Change the parts in Spanish, since my website is in Spanish … User-agent: * Disallow: /search/resultados/ Disallow: /register/ Disallow: /resetting/ Disallow: /login Disallow: /profile/ Disallow: /to/ Disallow: /saveMail Disallow: /customLinks.js Disallow: /cta/test/ Disallow: /app/Resources/scripts/cookies.js Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php #robots your web # http://www.robotstxt.org/ # http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449 #es necesario personalizar algunas opciones o puede dar problemas User-agent: BotRightHere Disallow: / User-agent: WebZip Disallow: / User-agent: larbin Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: Copernic Disallow: / User-agent: psbot Disallow: / User-agent: Python-urllib Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: CherryPicker Disallow: / User-agent: EmailCollector Disallow:… Read more »

2
0
Would love your thoughts, please comment.x
()
x