Thread: Robots.txt

I'm hoping some of you guys can help with this one... I posted a robots file a few weeks ago, wasn't blocking anything, hadn't even gotten round to editing it but within a week had 50+ crawl errors in google webmaster tools. Deleted the file but errors are still there, my homepage was last crawled over a month ago and my indexing has gone right down. It's so bad at the moment the site isn't even returned in a domain name search.

I have also posted a Sitemap but am recieving an error as follows:
Network unreachable: robots.txt unreachable
We were unable to crawl your Sitemap because we found a robots.txt file at the root of your site but were unable to download it. Please ensure that it is accessible or remove it completely.

Have searched high & low the damn robots file is definitely gone. Can anyone throw some light on this?
thanks a mil.

You can try to resubmit the sitemap.xml file.

just put in a robots.txt file there, which is blank.

I've resubmitted it 8 or 9 times at this stage, under different names etc. Also recreated the robots file but no joy so removed it again.

Is the URL of the site the one in the post, because the domain does NOT have a robots.txt file avaialble.
Make sure you named it right (no spaces, etc.)

I know it's not there, I deleted it when I first started getting errors -that's the problem, why does Google still think it exists??!

If you want to share your google webmaster username and password, I am willing to have a look for you.

Because robots.txt file should be there.
If you can see it in your browser, so should SE.

Can you give us an example of the errors Google displays?

What was the content of robots.txt ?

User-Agent: *
Allow: /

unreachable = could have many causes

Server down, too slow or too may redirects.

Yahoo and MSN are not as fussy as Google. If you create the robots.txt file, upload it on the server and still can not navigate to it, there must be something wrong with your server setup.