I would like to prevent the spidering of a download directory. Since not all spiders respect robots.txt file and people can discover a site structure by viewing the robots.txt file, I would like to use htaccess to deny spiders but allow users without the need for a password to have access.
Here is an example, let's say I have a pdf file in a download directory. I would like to protect that file name from being discovered by a spider.
I have searched this forum and nothing seems to be what I require. But I probably missed it because I'm new to mod rewrite and htaccess.
Please let me know if this is possible. And some sample code would be greatly appreciated
thanks in advance for your help
mw