Welcome! Log In Create A New Profile

Advanced

Ban folders from being indexed

Posted by Bg-Foxx 
Ban folders from being indexed
May 01, 2007 03:33AM
How can I ban certain folders from being indexed? I tried several times to enter these folders in the "URLs must not include:" text box, but they are still getting indexed. Should I use the entire URL or just the /folder/?

Thanks in advance.
Re: Ban folders from being indexed
May 01, 2007 08:30AM
Don't include the slashes and use only the foldername!
Try indexing over the commandline with -n flag!

A.

http://www.made-easy.cc
Tec
Re: Ban folders from being indexed
May 02, 2007 07:31AM
If you have access to the sites to be indexed, there is another easy possibility:
Create a file named robots.txt and place it into the root-folder of your site to be indexed.
Content of robots.txt:
#
#
User-agent: *
Disallow: /admin/
Disallow: /backup/
Disallow: /folder_xyz/
#
# End of file

Now all well educated crawlers like Sphider or Google will not index these folders.
For more details see http://www.robotstxt.org

Tec
Sorry, only registered users may post in this forum.

Click here to login