Avoid indexing directories

Hi to all!

I've just installed Sphider on a website where directory browsing si enabled :-(.
There's a way to tell Sphider to ignore directories during indexing phase?

Thanks for your help.
Mauro.

Did you consider 1 level, 2 levels, 3 levels, etc?

I index my site using full depth, but I want to index only linked pages.

Ando, can you solve this?

But how does Sphider find your directories if there are no links to them?

I don't know... for now I solved the problem filtering out all the pages that ends with '/' character.

how about using Robot.txt???

User-agent: *
Disallow: /folder_you_dont_want_to_index/

Ando asked

"But how does Sphider find your directories if there are no links to them?"

Answer: Browsing directories (Server must be configured to allow it). I did with 1000 pages.

Sogliola asked

"There's a way to tell Sphider to ignore directories during indexing phase?"


Answer

Use "URLs must not include"