Welcome! Log In Create A New Profile

Advanced

Avoid indexing directories

Posted by sogliola 
Avoid indexing directories
March 28, 2007 08:54PM
Hi to all!

I've just installed Sphider on a website where directory browsing si enabled :-(.
There's a way to tell Sphider to ignore directories during indexing phase?

Thanks for your help.
Mauro.
Anonymous User
Re: Avoid indexing directories
March 28, 2007 09:23PM
Did you consider 1 level, 2 levels, 3 levels, etc?
Re: Avoid indexing directories
March 29, 2007 09:42AM
I index my site using full depth, but I want to index only linked pages.
Anonymous User
Re: Avoid indexing directories
March 30, 2007 12:05AM
Ando, can you solve this?
Re: Avoid indexing directories
March 31, 2007 08:15PM
But how does Sphider find your directories if there are no links to them?
Re: Avoid indexing directories
April 04, 2007 04:50PM
I don't know... for now I solved the problem filtering out all the pages that ends with '/' character.
Re: Avoid indexing directories
April 04, 2007 09:52PM
how about using Robot.txt???

User-agent: *
Disallow: /folder_you_dont_want_to_index/
Anonymous User
Re: Avoid indexing directories
April 09, 2007 12:49AM
Ando asked

"But how does Sphider find your directories if there are no links to them?"

Answer: Browsing directories (Server must be configured to allow it). I did with 1000 pages.

Sogliola asked

"There's a way to tell Sphider to ignore directories during indexing phase?"


Answer

Use "URLs must not include"
Sorry, only registered users may post in this forum.

Click here to login