Welcome! Log In Create A New Profile

Advanced

file checking forbidden in robots.txt

Posted by dorothy 
file checking forbidden in robots.txt
May 07, 2007 12:15AM
I can't seem to get sphider to index my site, I get an error that says: file checking forbidden in robots.txt

I added in Sphider2 for a user agent with appropriate disallow directories in robots.txt, sphider did not index at all without it being in the robots file...

I am stumped, any help would be greatly appreciated. Thanks!
Re: file checking forbidden in robots.txt
May 07, 2007 07:21AM
Stupid thing to do.
use disallow: *
and exclude what you need to exclude, do not talk to sphider in specific!

A.
Sorry, only registered users may post in this forum.

Click here to login