indexing despite noindex tag

Hi,

I have a rather strange question.

A friend is running a website. He does not want to get google and others to index his page, so he put noindex tags on all of his pages on his site.

Now he asked me if I could include a local search script for his site. I came across sphider and I think that's the best he can ask for his site.

Of course, when indexing, I realized I had a problem.

Is it prossible to bypass this somehow, as I am actually indexing "from the inside", so I should be able to index everything without removing the noindex tags?

I thought maybe the robots.txt file helps, but I am not sure how to interpret that. As I can see, I could maybe create a rule there for sphider as a User-agent, but I did not figure out how to do that.

Can someone help? Very appreciated, thanks!!!

I think the solution is to strip all files of the "noindex" header, and then
place the robots.txt file on the document root, so that it allows
Sphider to do everything while disallowing all other search engines.

Put this in the beginning of your robots.txt:

User-agent: Sphider
Disallow: 
User-agent: *
Disallow: /

This will let Sphider index the site while disallowing all other bots.

-------------------------------------
MyBrute - settle your differences in the Arena!