Question on indexing
March 02, 2015 02:17AM
To whom it may concern,

My customer's site is www.pdmpantiqueprints.com/. I installed Sphider and the installation went fine. I'm indexing with the Full option, nothing in the Must Include or Must Not Include fields, and it's taking some time. The site probably has about 10k pages or slightly more. We currently have Google Custom Search but it constantly misses pages so we thought we'd look into an alternative. I start the indexing process, it shows that it's indexed about 1,700 pages, and then it seems to stop. So I go back to the Admin page and Continue Indexing. I'm on the 3rd repetition now I think. Is this normal? Also, in the future to index new pages, how is that best done? My customer probably adds about 50 pages/week, and he does keep a log file of the new URL's.

Josh
Tec
Re: Question on indexing
March 04, 2015 07:18PM
Indexing with a PHP script based search engine like Sphider is very problematically on 'Shared Hosting' servers. Indexing huge amount of links might become interrupted, because the granted time slice ended before index procedure is finished.

Tec
Re: Question on indexing
March 05, 2015 11:36PM
Thanks for getting back to me! Is the time slice controlled by the hosting company or by Sphider? If the latter, can it be edited? What do you think of Perl based search engines? I was looking at one called Perlfect, if Sphider doesn't work out.
Tec
Re: Question on indexing
March 06, 2015 03:16PM
<<< Is the time slice controlled by the hosting company or by Sphider?>>>
It is your hosting company, controlling the server.

Tec
Re: Question on indexing
March 08, 2015 02:47AM
Thanks again. Once I put "redir" in the "must not include" field that helped a lot, after 4 or so repetitions the whole site has been indexed. The first time around it probably took 20 repetitions. Testing it out now, so far it's great! Now I'm optimistic this will still work. We're thinking once a week we'll just reindex the whole site.
Sorry, only registered users may post in this forum.

Click here to login