Welcome! Log In Create A New Profile


Ban folders from being indexed

Posted by Bg-Foxx 
Ban folders from being indexed
May 01, 2007 06:33AM
How can I ban certain folders from being indexed? I tried several times to enter these folders in the "URLs must not include:" text box, but they are still getting indexed. Should I use the entire URL or just the /folder/?

Thanks in advance.
Re: Ban folders from being indexed
May 01, 2007 11:30AM
Don't include the slashes and use only the foldername!
Try indexing over the commandline with -n flag!


Re: Ban folders from being indexed
May 02, 2007 10:31AM
If you have access to the sites to be indexed, there is another easy possibility:
Create a file named robots.txt and place it into the root-folder of your site to be indexed.
Content of robots.txt:
User-agent: *
Disallow: /admin/
Disallow: /backup/
Disallow: /folder_xyz/
# End of file

Now all well educated crawlers like Sphider or Google will not index these folders.
For more details see http://www.robotstxt.org

Sorry, only registered users may post in this forum.

Click here to login