Welcome! Log In Create A New Profile

Advanced

Sphider crashes when too many search results?

Posted by TMTM 
Sphider crashes when too many search results?
November 11, 2014 10:50AM
Hi,
I have a website with several pages, circa 70 short texts and about 40 long articles. Everything works smoothly unless I try to search for the frequent words like medicine or anatomy (it's a medical website). Sphider crashes and I can see only the header of the page.

Anyone has the same problem? Is there a solution for this issue?

Maybe I should limit the numer of search results in config? Is it this option - Bound number of search results ?

Or maybe there's an option to mark selected words as "too general" so they are excluded from being indexed?

Thanks!



Edited 1 time(s). Last edit at 11/11/2014 02:32PM by TMTM.
Tec
Re: Sphider crashes when too many search results?
November 11, 2014 04:33PM
Regarding the amount of keywords and/or search results, the Sphider scripts do not contain any limitations. To index about 100 pages is no problem for Sphider. Even sites containing thousands of pages were already and successfully indexed. So, if your installation crashes, there must be another reason. In order to follow your issue, please describe more precise what happens.

<<< Sphider crashes and I can see only the header of the page >>>
In search results, the header(?) of the page is never presented. First row in each search result is always the title tag (parsed from the head part of HTML).
- If Sphider crashes, what is shown on your monitor?
- Is only one 'crashed' result presented in result listing, or several of them?
- Does Sphider crash for all queries, or only for queries producing a huge amount of results?
- Which amount?
- What time does it take after you entered the query, until the (crashed) results are presented?
- While indexing, did the crawler present any error messages, or warnings?
- How many links and keywords are stored in your database (presented as status row at the bottom of 'Sites' view in admin backend).
- How many pages does your site contain? Is the amount of links presented in admin backend, the correct count of links?
- Are you running Sphider on a 'Shared Hosting' server?

Tec



Edited 1 time(s). Last edit at 11/11/2014 04:36PM by Tec.
Re: Sphider crashes when too many search results?
November 11, 2014 05:03PM
- In search results, the header(?) of the page is never presented. First row in each search result is always the title tag (parsed from the head part of HTML).
I integrated Sphider with my website with this tutorial: http://valleynumber21.blogspot.com/2010/08/how-to-integrate-sphider-search-to.html so the page header is presented. But nothing else – no sphider search results, no footer.

- If Sphider crashes, what is shown on your monitor?
Just the page header and sphider search box. No other Sphider content.

- Is only one 'crashed' result presented in result listing, or several of them?
Nothing is presented. There are several words that causes this issue – few most common words.

- Does Sphider crash for all queries, or only for queries producing a huge amount of results?
Only huge amount. Other queries runs smoothly.

- Which amount?
Can’t tell exactly, but more then 100. For queries below this number everything is ok.

- What time does it take after you entered the query, until the (crashed) results are presented?
About 20-30 seconds and then nothing is shown . For less common words I get results within a second or two.

- While indexing, did the crawler present any error messages, or warnings?
No, it indexes without any problem.

- How many links and keywords are stored in your database (presented as status row at the bottom of 'Sites' view in admin backend).
64 links and 9267 keywords.

- How many pages does your site contain? Is the amount of links presented in admin backend, the correct count of links?
It shows correct amount. However I use rewrite rule, but I blocked php pages in URL must not include (for example articles?page=) so they do not duplicate with rewrited urls (articles/title…). But the issue was even before I did that.

- Are you running Sphider on a 'Shared Hosting' server?
On localhost (MAMP).



Edited 1 time(s). Last edit at 11/11/2014 06:36PM by TMTM.
Tec
Re: Sphider crashes when too many search results?
November 12, 2014 10:54AM
It looks like a 'time out' created by your server. Because a script, which is running for 20 – 30 seconds, might be killed by the server.
So you will have to find out, why it takes so long to present the results. The search algorithm is not as complicated that it should run for more than 20 seconds to grab in a database containing about 10.000 keywords, and afterwards to build up a result listing with 100 + x results. In an 'localhost' environment 0.2 seconds should be quite enough. Don't know your server, but eventually your MySQL database might be the bottleneck.

In order to find out, open the script
…/include/searchfuncs.php
in your editor and find the row
            $query1 = "SELECT distinct link_id, weight, domain from ".$mysql_table_prefix."link_keyword$wordmd5, ".$mysql_table_prefix."keywords where ".$mysql_table_prefix."link_keyword$wordmd5.keyword_id= ".$mysql_table_prefix."keywords.keyword_id and keyword='$searchword' $domain_qry order by weight desc";
Delete this row and replace it with
        $limit = 10;    // defines max. amount of results 
        $query1 = "SELECT distinct link_id, weight, domain from ".$mysql_table_prefix."link_keyword$wordmd5, ".$mysql_table_prefix."keywords where ".$mysql_table_prefix."link_keyword$wordmd5.keyword_id= ".$mysql_table_prefix."keywords.keyword_id and keyword='$searchword' $domain_qry order by weight desc limit $limit";

Tec
Re: Sphider crashes when too many search results?
November 13, 2014 02:33AM
Thanks for helping TEC smiling smiley

Unfortunately the efect is the same. The result page loads sloooowly and then no results.
Tec
Re: Sphider crashes when too many search results?
November 13, 2014 09:10AM
<<< Unfortunately the efect is the same. >>>
Well, this is what I was expecting. Because it clearly shows that you do have a problem with the MySQL implementation on your server.
Why?
The hack I gave to you, only limits the max. amount of results for the PHP scripts of your Sphider-plus installation. But the SQL query
$query1 = "SELECT distinct link_id, weight, domain from . . . .
needs to grab the complete database to extract the most relevant 10 (as example) results.
In other words: The Sphider scripts do not cause your issue, because it remains even for just 10 'medicine' results. But the SQL query is not able to find (and sort) the hundreds of results for a common word like 'medicine' within the 'time-out' limit of approximately 20 seconds (which is defined as max. execution time for any script on your server).

Suggestion1: By means of a tool like 'phpMyAdmin' try to repair your existing database.
Suggestion2: By means of phpMyAdmin delete all tables inside of your db and afterwards run
…/admin/install.php
to create a new set of tables for your Sphider installation.
If the above does not solve your issue:
Suggestion3: Rename your existing MAMP server (so the server remains available as backup) and perform a completely new installation of your server. Hopefully the MySQL part will become more effective.
Suggestion4: Switch over to XAMPP, which is working very well on Windows systems.

Tec
Re: Sphider crashes when too many search results?
November 13, 2014 10:12AM
TEC, thanks for clues and suggestions smiling smiley Now I know where the problem is. I'll try that.
Re: Sphider crashes when too many search results?
December 17, 2014 10:33PM
That would be a server side problem. If you are using shared purchased hosting, you may want to upgrade to a larger package. you may also want to take a look at your .htaccess file. You can set the memory limits in that file too.

Are you on ClickRaider?
Sorry, only registered users may post in this forum.

Click here to login