Irish SEO, Marketing & Webmaster Discussion
 
 

Go Back   Irish SEO, Marketing & Webmaster Discussion > Online Marketing > Search Engine Optimisation
Robots.txt Robots.txt
Register Forum Rules FAQDonate Members List Calendar Search Today's Posts Mark Forums Read


Reply
Page 1 of 2 1 2 >
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 14-07-2008, 01:37 PM
Frontpage User
  
Join Date: Jul 2008
Posts: 6
distressed will become famous soon enough
Default Robots.txt
I'm hoping some of you guys can help with this one... I posted a robots file a few weeks ago, wasn't blocking anything, hadn't even gotten round to editing it but within a week had 50+ crawl errors in google webmaster tools. Deleted the file but errors are still there, my homepage was last crawled over a month ago and my indexing has gone right down. It's so bad at the moment the site isn't even returned in a domain name search.

I have also posted a Sitemap but am recieving an error as follows:
Network unreachable: robots.txt unreachable
We were unable to crawl your Sitemap because we found a robots.txt file at the root of your site but were unable to download it. Please ensure that it is accessible or remove it completely.

Have searched high & low the damn robots file is definitely gone. Can anyone throw some light on this?
thanks a mil.
Last edited by distressed; 23-07-2008 at 10:30 AM.
Reply With Quote
  #2 (permalink)  
Old 14-07-2008, 01:42 PM
paul's Avatar
Web Whore
Recent Blog: Google Developer Day 2008 (Müenchen)
  
Join Date: Dec 2006
Location: .de
Posts: 967
paul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud of
Default
You can try to resubmit the sitemap.xml file.

just put in a robots.txt file there, which is blank.
__________________
my sites :
irish poker / irish jobs / seo faq / advertise jobs free / green card / skiing
Last edited by paul; 14-07-2008 at 01:45 PM.
Reply With Quote
  #3 (permalink)  
Old 14-07-2008, 01:52 PM
Frontpage User
  
Join Date: Jul 2008
Posts: 6
distressed will become famous soon enough
Default
I've resubmitted it 8 or 9 times at this stage, under different names etc. Also recreated the robots file but no joy so removed it again.
Reply With Quote
  #4 (permalink)  
Old 14-07-2008, 01:58 PM
louie's Avatar
Senior Member
Recent Blog: Irish Webmaster Meetup #2
  
Join Date: Jan 2006
Location: Dublin, Ireland
Posts: 1,923
louie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud of
Send a message via Yahoo to louie Send a message via Skype™ to louie
Default
Is the URL of the site the one in the post, because the domain does NOT have a robots.txt file avaialble.
Make sure you named it right (no spaces, etc.)
__________________
:. Web Design & Development Web Design Ireland
:. Search Engines Optimization Search Engines Optimization
:. Directory Submission Directory Submission
:. News & Press Release Ireland GiveItSocks.com
:. Used Cars Ireland, Car Parts & Car Audio Cars For Sale, Car Parts & Accessories
:. I Have 2 Find It Directory SEF Directory
Reply With Quote
  #5 (permalink)  
Old 14-07-2008, 02:05 PM
Frontpage User
  
Join Date: Jul 2008
Posts: 6
distressed will become famous soon enough
Default
I know it's not there, I deleted it when I first started getting errors -that's the problem, why does Google still think it exists??!
Reply With Quote
  #6 (permalink)  
Old 14-07-2008, 02:10 PM
paul's Avatar
Web Whore
Recent Blog: Google Developer Day 2008 (Müenchen)
  
Join Date: Dec 2006
Location: .de
Posts: 967
paul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud of
Default
If you want to share your google webmaster username and password, I am willing to have a look for you.
__________________
my sites :
irish poker / irish jobs / seo faq / advertise jobs free / green card / skiing
Reply With Quote
  #7 (permalink)  
Old 14-07-2008, 02:15 PM
louie's Avatar
Senior Member
Recent Blog: Irish Webmaster Meetup #2
  
Join Date: Jan 2006
Location: Dublin, Ireland
Posts: 1,923
louie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud of
Send a message via Yahoo to louie Send a message via Skype™ to louie
Default
Because robots.txt file should be there.
If you can see it in your browser, so should SE.

Can you give us an example of the errors Google displays?
__________________
:. Web Design & Development Web Design Ireland
:. Search Engines Optimization Search Engines Optimization
:. Directory Submission Directory Submission
:. News & Press Release Ireland GiveItSocks.com
:. Used Cars Ireland, Car Parts & Car Audio Cars For Sale, Car Parts & Accessories
:. I Have 2 Find It Directory SEF Directory
Reply With Quote
  #8 (permalink)  
Old 14-07-2008, 02:42 PM
paul's Avatar
Web Whore
Recent Blog: Google Developer Day 2008 (Müenchen)
  
Join Date: Dec 2006
Location: .de
Posts: 967
paul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud ofpaul has much to be proud of
Default
What was the content of robots.txt ?
__________________
my sites :
irish poker / irish jobs / seo faq / advertise jobs free / green card / skiing
Reply With Quote
  #9 (permalink)  
Old 14-07-2008, 02:43 PM
Frontpage User
  
Join Date: Jul 2008
Posts: 6
distressed will become famous soon enough
Default
User-Agent: *
Allow: /
Reply With Quote
  #10 (permalink)  
Old 14-07-2008, 02:48 PM
louie's Avatar
Senior Member
Recent Blog: Irish Webmaster Meetup #2
  
Join Date: Jan 2006
Location: Dublin, Ireland
Posts: 1,923
louie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud of
Send a message via Yahoo to louie Send a message via Skype™ to louie
Default
unreachable = could have many causes

Server down, too slow or too may redirects.

Yahoo and MSN are not as fussy as Google. If you create the robots.txt file, upload it on the server and still can not navigate to it, there must be something wrong with your server setup.
__________________
:. Web Design & Development Web Design Ireland
:. Search Engines Optimization Search Engines Optimization
:. Directory Submission Directory Submission
:. News & Press Release Ireland GiveItSocks.com
:. Used Cars Ireland, Car Parts & Car Audio Cars For Sale, Car Parts & Accessories
:. I Have 2 Find It Directory SEF Directory
Reply With Quote
Reply
Page 1 of 2 1 2 >

« Search Engine Friendly Redirect Checker | Analytics - Installing UTM tracking code? »
Thread Tools
Display Modes
Linear Mode Linear Mode

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Crawlers, robots and excessive bandwidth bazzjazz Server / Technical Administration Tips and Queries 9 30-04-2008 03:36 PM
Googlebot blocked from robots.txt + sitemap.xml Cormac Search Engine Optimisation 8 17-04-2008 08:48 PM
Bad Robots Cormac Webmaster Discussion 1 17-10-2006 03:31 PM


All times are GMT +1. The time now is 09:06 PM.

Contact Us - Irish Webmaster Discussion - Archive - Top

Powered by: vBulletin Version 3.7.2, Copyright ©2000 - 2008, Jelsoft Enterprises Limited.
Hosted in Ireland by Blacknight - Test your ISP |Irish Hosting Directory| Armchair.ie|Logo by Eden Web Design|Avatars by Afterglow |Latest Blog Entries | VPS HostingAd Management by RedTyger

Search Engine Friendly URLs by vBSEO 3.2.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56