Welcome! Log In Create A New Profile


follow but not index

Posted by fneumeier 
follow but not index
May 17, 2007 10:43PM

is there an easy way to let the spider follow links but not index specific pages where it finds these links on?


www.domain.com/linkspages/page1.html contains links to pages www.domain.com/linkspages/subpages/page2.html, page2.html etc.

I don't want www.domain.com/linkspages/page1.html being indexed but the spider shall follow the links on this page and index the other pages. Requiring "subpages" being part of the URL won't do it as the spider would never find the links as www.domain.com/linkspages/page1.html doesn't contain this term ...

Any ideas?

Re: follow but not index
May 18, 2007 08:29AM
would <rel="noindex"> do it? is sphider using this meta information in a page?
Re: follow but not index
May 18, 2007 01:32PM
yes it is using it
Sorry, only registered users may post in this forum.

Click here to login