Welcome! Log In Create A New Profile

Advanced

Memory Error -> auto-add File to don't index?!

Posted by mahakala 
Memory Error -> auto-add File to don't index?!
April 19, 2007 09:34AM
hi there,

i like sphider - really a nice engine.

i need to index several very large sites ~25.000++ links with .pdfs and .docs, too.

while runnig it by ssh-shell i can't see what file causes an error, because the log doesn't write it down.

now i'm indexing and after some time it crashes with an Memory error (e.g. Allowed memory size of 52428800 bytes exhausted (tried to allocate 901 bytes)).

i deactivated pdf/doc and so on but still the same error.

anybody a hint for me how to solve it?

great would be a small piece of code what remembers the last file and if it crashes with it, it will be automatically added to the "URLs must not include" index.

best regards,
thomas

[url=http://spirituelle-hilfe.com]^[/url][url=http://forum.spirituelle-hilfe.com]^[/url]
Anonymous User
Re: Memory Error -> auto-add File to don't index?!
April 19, 2007 12:25PM
It is a php memory problem.

In "php.ini"

use a larger number, depending on your system. Ex.
memory_limit = 256M
Re: Memory Error -> auto-add File to don't index?!
April 20, 2007 02:31PM
hi there,

thanks for the info but it isn't "my" server, so i can't change this value

maybe one solution would be to write the current indexed site or file in a temp-database (sql or file) and if sphider (run by ssh) hangs with it, it's added to the ignore list when sphider restarts.
badly i have no idea how to program that. maybe someone has a hint?

regards,
thomas

[url=http://spirituelle-hilfe.com]^[/url][url=http://forum.spirituelle-hilfe.com]^[/url]



Edited 2 time(s). Last edit at 04/20/2007 03:26PM by mahakala.
Re: Memory Error -> auto-add File to don't index?!
April 20, 2007 11:09PM
bump

[url=http://spirituelle-hilfe.com]^[/url][url=http://forum.spirituelle-hilfe.com]^[/url]
Re: Memory Error -> auto-add File to don't index?!
April 23, 2007 09:44AM
bump

[url=http://spirituelle-hilfe.com]^[/url][url=http://forum.spirituelle-hilfe.com]^[/url]
Anonymous User
Re: Memory Error -> auto-add File to don't index?!
April 23, 2007 09:07PM
If any of the sites you want to index has +25 000 url's and you keep errors without the pdf's and the doc's (usually the ones that "consume" more memory),maybe you arrived to the limits of today's Sphider.

Some writers have been saying that Sphider is good for up to 20 000 url's (Ando Says 100 000; In my experience, there is no limit as far as you index each site by parts with less than 10 000 url's each - Still, I am not quite sure of this).

Maybe you want to try indexing just 3 levels deep.

If you can get the programming solutions you are asking for, please share it.

Thanks.
Re: Memory Error -> auto-add File to don't index?!
April 25, 2007 04:52PM
hi there,

i have currently in database: 67 sites, 72351 links, 11 categories and 1637549 keywords, Keyword-link realations: 23832099, Cached texts total: 589,961.82 kb, Sites size total: 1,816,115.40 kb

i don't think this is the problem.

my wish is to exclude the last file indexed (and caused a problem) when running by ssh.
so it should be very simple for the one who knows how to programm it, so skip the last file in database and resume with the next one.
but i don't know where exactly this kind of code should be insert to.

regards,
thomas

[url=http://spirituelle-hilfe.com]^[/url][url=http://forum.spirituelle-hilfe.com]^[/url]
Re: Memory Error -> auto-add File to don't index?!
May 01, 2007 08:47AM
Ok guys, this was a problem I had when I first downloaded the script!
If your error is somewhere near line 478.... Then this is caused by a MySQL select statement which overloads php cache memory! You could possibly split down this part by limiting the Query which is executed and execute several smaller ones! For me this worked fine!

A.
Re: Memory Error -> auto-add File to don't index?!
May 16, 2007 03:28PM
hi a,

how have you done this exactly?

t.

[url=http://spirituelle-hilfe.com]^[/url][url=http://forum.spirituelle-hilfe.com]^[/url]
Sorry, only registered users may post in this forum.

Click here to login