robots.txt file should be farely simple.
you tell the robots that follows the rules what directory to spider and which one not.
Also not a very good ideea as you give the secret directories out for the ones that aren't really just visitors.
in the robots.txt file you choose what spiders should index your website and what not.
An example:
User-agent: *Disallow: /secure/Disallow: /images/that tells all robots not to spider the /secure and /images folders.Everything else is ok.a better way of protecting a directory is to have it based on login and valid session. |