Thread: Robots.txt

Try uploading a txt file to a directory other than your root.
Can you see it? Make sure your .txt file is ASCII and that your server supports the txt as a Mime type.

Guys I think you are misunderstanding me here..when the robots file is up I can navigate to it no problem. I have no issue with that.
The issue is that I have taken the file down (i'm not required to have one and had no problems before i ever created one) but Google is still telling me it finds one and has a problem with it even though it does not exist on web server.

First you told Google you have the file in place so rules are created inside the SE of how and what.

Than you decided to remove it, but SE didn't update its rules.
My question is - why remove it in the first place.

I will say, it's something else wrong there and you can not see it.
In 8 years, this is the first I've heard of this issue as it never happened to me.

I know, i'm completely stumped aswell.

I removed it as it started giving errors in webmaster tools even though it didn't contain any disallows..

I don't think that was the problem (disallow).
There is a chance your file had some weird characteres... which are not visible in browser.

From the looks of it those errors are backdated and Google just hasn't caught up with itself just yet, very normal.

Originally Posted by distressed

I'm hoping some of you guys can help with this one... I posted a robots file a few weeks ago, wasn't blocking anything, hadn't even gotten round to editing it but within a week had 50+ crawl errors in google webmaster tools. Deleted the file but errors are still there, my homepage was last crawled over a month ago and my indexing has gone right down. It's so bad at the moment the site isn't even returned in a domain name search.

I have also posted a Sitemap but am recieving an error as follows:
Network unreachable: robots.txt unreachable
We were unable to crawl your Sitemap because we found a robots.txt file at the root of your site but were unable to download it. Please ensure that it is accessible or remove it completely.

Have searched high & low the damn robots file is definitely gone. Can anyone throw some light on this?
thanks a mil.
www.healthlink.ie

If they dont get a valid response (200, 404) for your robots.txt then they'll remove your site. This is after the Belgian newspapaer sued them.

THe crawl errors aren't caused by robots.txt BTW - more likely XML sitemap, or 404s. But it's quite normal to have crawl errors.

First thing though is to sort out why they cant get a valid response for robots.txt.