Googlebot blocked from robots.txt + sitemap.xml
This is a discussion on Googlebot blocked from robots.txt + sitemap.xml within the Search Engine Optimisation forums, part of the Online Marketing category; I'm running Wordpress MU for the ammado blog. All is going well but I'm having issues with some search visibility ...
| |||||||
| Register | Forum Rules | FAQ | Donate | Members List | Calendar | Search | Today's Posts | Mark Forums Read |
17-04-2008, 10:49 AM
| |||||
| |||||
 Googlebot blocked from robots.txt + sitemap.xml I'm running Wordpress MU for the ammado blog. All is going well but I'm having issues with some search visibility issues. I have a robots.txt file in the blog root of the MU site. (using this plugin for MU robot.txt files) http://en.blog.ammado.com/robots.txt I also have a sitemap.xml file in the same location http://en.blog.ammado.com/sitemap.xml I have added en.blog.ammado as a Google Sitemap account and I tried to add the sitemap.xml file to my account. Google is throwing up a "Googlebot is blocked from http://en.blog.ammado.com/" error message. I'm not sure where GB is being blocked though. Any ideas on this one? I have also added a robots.txt file to blog.ammado.com just in case - http://blog.ammado.com/robots.txt         |
|
17-04-2008, 04:04 PM
| |||||
| |||||
| Cormac it might be something to do with your .htaccess file |
TOTW/F/M Award(s): 1|
17-04-2008, 04:53 PM
| |||||
| |||||
| Hi, I'm pretty sure the .htaccess file which I am using is the bog standard one for WPMU: Code: RewriteEngine On
RewriteBase /
#uploaded files
RewriteRule ^(.*/)?files/$ index.php [L]
RewriteRule ^(.*/)?files/(.*) wp-content/blogs.php?file=$2 [L]
RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule . - [L]
RewriteRule ^([_0-9a-zA-Z-]+/)?(wp-.*) $2 [L]
RewriteRule ^([_0-9a-zA-Z-]+/)?(.*\.php)$ $2 [L]
RewriteRule . index.php [L]
|
|
17-04-2008, 05:48 PM
| |||||
| |||||
| robots and sitemaps Just viewed the robots.txt it just gives the address of the sitemap. I am not sure here but if you provide a robots without specifying who or what is allowed will google assume its not allowed and bugger off , Try to allow all and see what happens User-agent: * Disallow: |
|
17-04-2008, 06:01 PM
| |||||
| |||||
| Quote:
|
|
17-04-2008, 08:25 PM
| |||||
| |||||
| The problem is the robots meta tag. <meta name='robots' content='noindex,nofollow' /> That will stop GB. Remove it and you're dandy. |
|
17-04-2008, 08:42 PM
| |||||
| |||||
| Nice one. Didn't spot that at all. I must have selected 'do not allow my Wordpress installation to appear in search engines' when I was testing out stuff locally. When I uploaded the test environment, I forgot to change this setting. Yikes! |
|
17-04-2008, 08:48 PM
| |||||
| |||||
| Yup, that was probably it. Happens all the time  When working on a site, we add this feature until the site is "ready for indexing". Sometimes we forget too Anyway, you should be all good now. |
 |
| Tags |
| blocked, googlebot, robotstxt, sitemapxml |
| Thread Tools | |
| Display Modes | |
Linear Mode
|
|
| ||||||||
| |  |  | ||||||
All times are GMT +1. The time now is 09:14 PM.
