Irish SEO,  Marketing & Webmaster Discussion

 

Robots.txt

This is a discussion on Robots.txt within the Search Engine Optimisation forums, part of the Online Marketing category; I'm hoping some of you guys can help with this one... I posted a robots file a few weeks ago, ...


Go Back   Irish SEO, Marketing & Webmaster Discussion > Online Marketing > Search Engine Optimisation

Register Forum Rules FAQDonate Calendar Search Today's Posts Mark Forums Read

  #1 (permalink)  
Old 14-07-2008, 01:37 PM
Frontpage User
 
Join Date: Jul 2008
Posts: 6
Nominated 0 Times in 0 Posts
TOTW/F/M Award(s): 0
distressed will become famous soon enough
Default Robots.txt

I'm hoping some of you guys can help with this one... I posted a robots file a few weeks ago, wasn't blocking anything, hadn't even gotten round to editing it but within a week had 50+ crawl errors in google webmaster tools. Deleted the file but errors are still there, my homepage was last crawled over a month ago and my indexing has gone right down. It's so bad at the moment the site isn't even returned in a domain name search.

I have also posted a Sitemap but am recieving an error as follows:
Network unreachable: robots.txt unreachable
We were unable to crawl your Sitemap because we found a robots.txt file at the root of your site but were unable to download it. Please ensure that it is accessible or remove it completely.

Have searched high & low the damn robots file is definitely gone. Can anyone throw some light on this?
thanks a mil.

Last edited by distressed; 23-07-2008 at 10:30 AM..
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
  #2 (permalink)  
Old 14-07-2008, 01:42 PM
paul's Avatar
ninja SEO
 
Join Date: Dec 2006
Location: .de
Posts: 1,251
Nominated 0 Times in 0 Posts
TOTW/F/M Award(s): 0
paul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud of
Default

You can try to resubmit the sitemap.xml file.

just put in a robots.txt file there, which is blank.
__________________
my sites :
irish poker / irish jobs / seo faq / advertise jobs free / green card / skiing

Last edited by paul; 14-07-2008 at 01:45 PM..
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
  #3 (permalink)  
Old 14-07-2008, 01:52 PM
Frontpage User
 
Join Date: Jul 2008
Posts: 6
Nominated 0 Times in 0 Posts
TOTW/F/M Award(s): 0
distressed will become famous soon enough
Default

I've resubmitted it 8 or 9 times at this stage, under different names etc. Also recreated the robots file but no joy so removed it again.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
  #4 (permalink)  
Old 14-07-2008, 01:58 PM
louie's Avatar
Senior Member
 
Join Date: Jan 2006
Location: Dublin, Ireland
Posts: 2,204
Nominated 6 Times in 4 Posts
Nominated TOTW/F/M Award(s): 1
louie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud of
Send a message via Yahoo to louie Send a message via Skype™ to louie
Default

Is the URL of the site the one in the post, because the domain does NOT have a robots.txt file avaialble.
Make sure you named it right (no spaces, etc.)
__________________
:. Web Design & Development Web Design Ireland
:. Search Engines Optimization Search Engines Optimization
:. Car Parts & Accessories Car Parts
:. Cars Ireland Cars Ireland
:. I Have 2 Find It Directory SEF Directory
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
  #5 (permalink)  
Old 14-07-2008, 02:05 PM
Frontpage User
 
Join Date: Jul 2008
Posts: 6
Nominated 0 Times in 0 Posts
TOTW/F/M Award(s): 0
distressed will become famous soon enough
Default

I know it's not there, I deleted it when I first started getting errors -that's the problem, why does Google still think it exists??!
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
  #6 (permalink)  
Old 14-07-2008, 02:10 PM
paul's Avatar
ninja SEO
 
Join Date: Dec 2006
Location: .de
Posts: 1,251
Nominated 0 Times in 0 Posts
TOTW/F/M Award(s): 0
paul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud of
Default

If you want to share your google webmaster username and password, I am willing to have a look for you.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
  #7 (permalink)  
Old 14-07-2008, 02:15 PM
louie's Avatar
Senior Member
 
Join Date: Jan 2006
Location: Dublin, Ireland
Posts: 2,204
Nominated 6 Times in 4 Posts
Nominated TOTW/F/M Award(s): 1
louie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud of
Send a message via Yahoo to louie Send a message via Skype™ to louie
Default

Because robots.txt file should be there.
If you can see it in your browser, so should SE.

Can you give us an example of the errors Google displays?
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
  #8 (permalink)  
Old 14-07-2008, 02:42 PM
paul's Avatar
ninja SEO
 
Join Date: Dec 2006
Location: .de
Posts: 1,251
Nominated 0 Times in 0 Posts
TOTW/F/M Award(s): 0
paul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud of
Default

What was the content of robots.txt ?
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
  #9 (permalink)  
Old 14-07-2008, 02:43 PM
Frontpage User
 
Join Date: Jul 2008
Posts: 6
Nominated 0 Times in 0 Posts
TOTW/F/M Award(s): 0
distressed will become famous soon enough
Default

User-Agent: *
Allow: /
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
  #10 (permalink)  
Old 14-07-2008, 02:48 PM
louie's Avatar
Senior Member
 
Join Date: Jan 2006
Location: Dublin, Ireland
Posts: 2,204
Nominated 6 Times in 4 Posts
Nominated TOTW/F/M Award(s): 1
louie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud of
Send a message via Yahoo to louie Send a message via Skype™ to louie
Default

unreachable = could have many causes

Server down, too slow or too may redirects.

Yahoo and MSN are not as fussy as Google. If you create the robots.txt file, upload it on the server and still can not navigate to it, there must be something wrong with your server setup.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
Reply

Tags
robotstxt

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads

Thread Thread Starter Forum Replies Last Post
Crawlers, robots and excessive bandwidth bazzjazz Server / Technical Administration Tips and Queries 9 30-04-2008 03:36 PM
Googlebot blocked from robots.txt + sitemap.xml Cormac Search Engine Optimisation 8 17-04-2008 08:48 PM
Bad Robots Cormac Webmaster Discussion 1 17-10-2006 03:31 PM


Sponsored links

Pepperjam Network
Paid On Results www.zanox.com


All times are GMT +1. The time now is 02:29 PM.


Powered by: vBulletin Version 3.8.2, Copyright ©2000 - 2009, Jelsoft Enterprises Limited.
Hosted in Ireland by Blacknight - Test your ISP |Irish Hosting Directory| Armchair.ie|Logo by Eden Web Design|Avatars by Afterglow |Latest Blog Entries | VPS HostingAd Management by RedTyger

Search Engine Friendly URLs by vBSEO 3.3.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51