Robots.txt

Status
Not open for further replies.

paddym

New Member
Hi Guys

I recently setup a wordpress blog. The issue I am having is webmaster tools is displaying a error "Some important page is blocked by robots.txt."

I have checked the robots.txt file but the same issue appears. I have also ran the file through a checker and it displays no issues.

Am I making a basic error :ambivalence:

I would be grateful if anyone could throw a eye on this for me :rapture:.
Robots.txt is also attached.

User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /go/ Disallow: /comments/feed/ Disallow: /feed/ Disallow: /trackback/ Disallow: /index.php Disallow: /xmlrpc.php User-agent: Googlebot-Image Allow: /wp-content/uploads/ User-agent: Adsbot-Google Allow: / User-agent: Googlebot-Mobile Allow: /
 
K

Kieran

Guest
You are disabling your comments feed. Is there a reason for this?
 

paddym

New Member
The original robots file was giving errors in webmaster tools. I downloaded a robot.txt file from yoast.comI have reverted back to a basic version of the file that works fine on one of my other sites.
I am hoping this will rectify the problem. Could it be a plugin causing the issue?


User-agent: *

Disallow:

Sitemap: http://mysite.me/sitemap.xml
 

mneylon

Administrator
Staff member
It looks like you were blocking access to a lot of things that you shouldn't have been
Blocking access to index.php? I can't see how that would be a good idea - though if you can explain it to me :)
Also blocking access to all rss feeds? I can see that causing you more problems than solutions
 

Softology

New Member
Disallow: /comments/feed/
Disallow: /feed/
Disallow: /trackback/
Disallow: /index.php

These are the reasons. If you have a reason to block the above, Ignore the message otherwise remove these and you are good to go.
 
Status
Not open for further replies.
Top