Welcome! Log In Create A New Profile

Advanced

indexing despite noindex tag

Posted by phonky 
indexing despite noindex tag
July 09, 2009 04:34PM
Hi,

I have a rather strange question.

A friend is running a website. He does not want to get google and others to index his page, so he put noindex tags on all of his pages on his site.

Now he asked me if I could include a local search script for his site. I came across sphider and I think that's the best he can ask for his site.

Of course, when indexing, I realized I had a problem.

Is it prossible to bypass this somehow, as I am actually indexing "from the inside", so I should be able to index everything without removing the noindex tags?

I thought maybe the robots.txt file helps, but I am not sure how to interpret that. As I can see, I could maybe create a rule there for sphider as a User-agent, but I did not figure out how to do that.

Can someone help? Very appreciated, thanks!!!
Re: indexing despite noindex tag
July 09, 2009 06:29PM
I think the solution is to strip all files of the "noindex" header, and then
place the robots.txt file on the document root, so that it allows
Sphider to do everything while disallowing all other search engines.
Re: indexing despite noindex tag
July 10, 2009 08:55PM
Put this in the beginning of your robots.txt:

User-agent: Sphider
Disallow: 
User-agent: *
Disallow: /

This will let Sphider index the site while disallowing all other bots.

-------------------------------------
MyBrute - settle your differences in the Arena!
Sorry, only registered users may post in this forum.

Click here to login