Hi
How will bing interpret robots.txt
when it returns a HTTP header status code 5xx ?
and when it returns a 404 or 4xx in general ?
I can't find a document that tells me
I knozw that Google will not crawl the site any further when the status of robots txt is returning a 500 message
And Goolge will interpret a 4xx like a permission to crawl everything
How is that in Bing ?
Is there any official documentation ?
Thanks
How will bing interpret robots.txt
when it returns a HTTP header status code 5xx ?
and when it returns a 404 or 4xx in general ?
I can't find a document that tells me
I knozw that Google will not crawl the site any further when the status of robots txt is returning a 500 message
And Goolge will interpret a 4xx like a permission to crawl everything
How is that in Bing ?
Is there any official documentation ?
Thanks