Irish SEO, Marketing & Webmaster Discussion

 
ThinkGeek - Cool Stuff for Geeks and Technophiles

Robots.txt

This is a discussion on Robots.txt within the Search Engine Optimisation forums, part of the Online Marketing category; Try uploading a txt file to a directory other than your root. Can you see it? Make sure your .txt ...


Go Back   Irish SEO, Marketing & Webmaster Discussion > Online Marketing > Search Engine Optimisation

Robots.txt Robots.txt

Register Forum Rules FAQDonate Members List Calendar Search Today's Posts Mark Forums Read


Notices
Welcome to the Irish Webmaster Forum, Ireland's leading discussion forum for anyone involved in the web!

You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features (including private member only sections and content). Registration is fast, simple and absolutely free so please, join our growing community today!

If you have any problems with the registration process or your account login, please let us know.

Reply
Page 2 of 2 < 1 2

 

LinkBack Thread Tools Display Modes
  #11 (permalink)  
Old 14-07-2008, 03:24 PM
Cormac's Avatar
Cormac Moylan
Recent Blog: Comment on Obama is a Hypocrite because? by Cormac
  
Join Date: Jan 2006
Location: Baile Ath Cliath / Corcaigh
Posts: 1,219
Nominated 0 Times in 0 Posts
TOTW/F/M Award(s): 0
Cormac is a splendid one to beholdCormac is a splendid one to beholdCormac is a splendid one to beholdCormac is a splendid one to beholdCormac is a splendid one to beholdCormac is a splendid one to beholdCormac is a splendid one to beholdCormac is a splendid one to behold
Send a message via AIM to Cormac Send a message via MSN to Cormac Send a message via Yahoo to Cormac Send a message via Skype™ to Cormac
Default
Try uploading a txt file to a directory other than your root.
Can you see it? Make sure your .txt file is ASCII and that your server supports the txt as a Mime type.
__________________
my blog | Apple Mac Blog | Indie Music Blog | *
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
  #12 (permalink)  
Old 14-07-2008, 03:48 PM
Frontpage User
  
Join Date: Jul 2008
Posts: 6
Nominated 0 Times in 0 Posts
TOTW/F/M Award(s): 0
distressed will become famous soon enough
Default
Guys I think you are misunderstanding me here..when the robots file is up I can navigate to it no problem. I have no issue with that.
The issue is that I have taken the file down (i'm not required to have one and had no problems before i ever created one) but Google is still telling me it finds one and has a problem with it even though it does not exist on web server.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
  #13 (permalink)  
Old 14-07-2008, 04:09 PM
louie's Avatar
Senior Member
Recent Blog: Search Engine Optimization Techniques
  
Join Date: Jan 2006
Location: Dublin, Ireland
Posts: 2,010
Nominated 5 Times in 3 Posts
TOTW/F/M Award(s): 0
louie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud of
Send a message via Yahoo to louie Send a message via Skype™ to louie
Default
First you told Google you have the file in place so rules are created inside the SE of how and what.

Than you decided to remove it, but SE didn't update its rules.
My question is - why remove it in the first place.

I will say, it's something else wrong there and you can not see it.
In 8 years, this is the first I've heard of this issue as it never happened to me.
__________________
:. Web Design & Development Web Design Ireland
:. Search Engines Optimization Search Engines Optimization
:. Directory Submission Directory Submission
:. News & Press Release Ireland GiveItSocks.com
:. Used Cars Ireland, Car Parts & Car Audio Cars For Sale, Car Parts & Accessories
:. I Have 2 Find It Directory SEF Directory
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
  #14 (permalink)  
Old 14-07-2008, 04:24 PM
Frontpage User
  
Join Date: Jul 2008
Posts: 6
Nominated 0 Times in 0 Posts
TOTW/F/M Award(s): 0
distressed will become famous soon enough
Default
I know, i'm completely stumped aswell.

I removed it as it started giving errors in webmaster tools even though it didn't contain any disallows..
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
  #15 (permalink)  
Old 14-07-2008, 04:29 PM
louie's Avatar
Senior Member
Recent Blog: Search Engine Optimization Techniques
  
Join Date: Jan 2006
Location: Dublin, Ireland
Posts: 2,010
Nominated 5 Times in 3 Posts
TOTW/F/M Award(s): 0
louie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud oflouie has much to be proud of
Send a message via Yahoo to louie Send a message via Skype™ to louie
Default
I don't think that was the problem (disallow).
There is a chance your file had some weird characteres... which are not visible in browser.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
  #16 (permalink)  
Old 14-07-2008, 04:55 PM
Hardcore Geek
Recent Blog: Next Webmaster Meetup In Cork - October 31
  
Join Date: Aug 2006
Location: Dublin
Posts: 1,030
Nominated 0 Times in 0 Posts
TOTW/F/M Award(s): 0
Gavin has a spectacular aura about
Default
From the looks of it those errors are backdated and Google just hasn't caught up with itself just yet, very normal.
__________________
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
  #17 (permalink)  
Old 15-07-2008, 12:32 PM
RedCardinal's Avatar
Richard Hearne
Recent Blog: Best Online Publication - The Irish Times
  
Join Date: Feb 2006
Posts: 941
Nominated 0 Times in 0 Posts
TOTW/F/M Award(s): 0
RedCardinal is a splendid one to beholdRedCardinal is a splendid one to beholdRedCardinal is a splendid one to beholdRedCardinal is a splendid one to beholdRedCardinal is a splendid one to beholdRedCardinal is a splendid one to beholdRedCardinal is a splendid one to behold
Default
Quote:
Originally Posted by distressed View Post
I'm hoping some of you guys can help with this one... I posted a robots file a few weeks ago, wasn't blocking anything, hadn't even gotten round to editing it but within a week had 50+ crawl errors in google webmaster tools. Deleted the file but errors are still there, my homepage was last crawled over a month ago and my indexing has gone right down. It's so bad at the moment the site isn't even returned in a domain name search.

I have also posted a Sitemap but am recieving an error as follows:
Network unreachable: robots.txt unreachable
We were unable to crawl your Sitemap because we found a robots.txt file at the root of your site but were unable to download it. Please ensure that it is accessible or remove it completely.

Have searched high & low the damn robots file is definitely gone. Can anyone throw some light on this?
thanks a mil.
www.healthlink.ie
If they dont get a valid response (200, 404) for your robots.txt then they'll remove your site. This is after the Belgian newspapaer sued them.

THe crawl errors aren't caused by robots.txt BTW - more likely XML sitemap, or 404s. But it's quite normal to have crawl errors.

First thing though is to sort out why they cant get a valid response for robots.txt.
__________________
Search Engine Optimisation - Red Cardinal Internet Marketing
Internet Consultant Ireland | Search Engine Optimisation Services
Catering Company Dublin - My sister's handmade canape company!
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
Reply
Page 2 of 2 < 1 2

Tags
robotstxt

« Search Engine Friendly Redirect Checker | Analytics - Installing UTM tracking code? »
Thread Tools
Display Modes
Linear Mode Linear Mode

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On
Forum Jump

Similar Threads

Thread Thread Starter Forum Replies Last Post
Crawlers, robots and excessive bandwidth bazzjazz Server / Technical Administration Tips and Queries 9 30-04-2008 03:36 PM
Googlebot blocked from robots.txt + sitemap.xml Cormac Search Engine Optimisation 8 17-04-2008 08:48 PM
Bad Robots Cormac Webmaster Discussion 1 17-10-2006 03:31 PM


All times are GMT +1. The time now is 04:22 AM.

Contact Us - Irish Webmaster Discussion - Archive - Top

Powered by: vBulletin Version 3.7.3, Copyright ©2000 - 2008, Jelsoft Enterprises Limited.
Hosted in Ireland by Blacknight - Test your ISP |Irish Hosting Directory| Armchair.ie|Logo by Eden Web Design|Avatars by Afterglow |Latest Blog Entries | VPS HostingAd Management by RedTyger

Search Engine Friendly URLs by vBSEO 3.2.0