Welcome! Log In Create A New Profile

Advanced

No Index a Folder

Posted by cmarangon 
No Index a Folder
January 21, 2010 01:12PM
Hello!

Reading the FAQ I got some tips about robots.txt.

However, I have a forum with more than 21,000 posts, and Sphider is indexing it, taking a long time.
So I am indexing it by parts. I rerun Sphider after 1 hour:-)
Server load goes to about 10!

How can I avoid Sphider index a folder?
By other side I also wish to have a Sphider searching only in the forum, not searching the rest of the site.
I am, able to use two DB for that!
How can I make sphider index only a directory?

Here is my robost.txt:
(www.areaseg.com/robots.txt)
-------
User-agent: *
Disallow: /lista
Disallow: /cgi-bin
Disallow: /cv
Disallow: /mural
-------------

The 21,000 files are in the directory
"/mural/msg/"

But the Sphider are indexing it.

Should I add "/mural/msg" to the robots.txt?



Edited 2 time(s). Last edit at 01/21/2010 01:24PM by cmarangon.
Re: No Index a Folder
January 21, 2010 01:42PM
Read the Docs, http://www.sphider.eu/docs.php#mustinc
winking smiley
Re: No Index a Folder
January 21, 2010 04:06PM
The problem is that Sphides seems do not obey robots.txt protocol.

Any tip?
Re: No Index a Folder
January 21, 2010 04:54PM
It does obey robots.txt.

If you make changes in robots.txt or any other option that changes it's indexing behaviour content indexed previous to those changes remains in the database.
To eliminate that content you have to erase (truncate) your database and do a reindex.
Re: No Index a Folder
February 14, 2010 10:12PM
cmarangon Wrote:
-------------------------------------------------------
> The problem is that Sphides seems do not obey
> robots.txt protocol.
>
> Any tip?

v1.3.5 ignores the robots.txt files.

I went back to v1.3.4 ...
Sorry, only registered users may post in this forum.

Click here to login