Show all posts by user
Post your modifications and addons here.
Re: Purging of pages that no longer exist? - 5 years ago
First select the row in question $row = mysql_fetch_row($result); $link_id = $row[0]; $visible = $row[1]; row 1 var contains the count and is pushed into $visible if ($visible > 0) { $visible --; If the count is still greater than 0 then we will deincrement the count by 1 i.e. $visible --; once deincremented then we will update $row[1] with -1 mysql_query("update Now ifby rr1024 - Sphider Support
2 more spider issues - 5 years ago
It's done this a couple times now finds a link to a zip file and hangs waiting for data or something but it just stops. Links found: 127. New links: 111 931. Retrieving: at 14:53:42 Above the the point it seem to hang and wait for data?? Another small issue Sphider is now spidering: 1. Retrieving: at 12:50:18. already in database Completed at 12:50:22. It looks like they are uby rr1024 - Sphider Support
Re: Index base page with greater words admin panel - 5 years ago
The following shows the report I had the word quantity set to 150 and then visited some of the pages and most had way more than 150 words. I hope spider is not using the min word length which is set to 3 on my system. It's ok to ignor words less than three for search purposes but not for counting the words on the page. Sphider is now spidering: Disallowed files and directories in robots.tby rr1024 - Sphider Support
Restricted Sites do not index option mod request - 5 years ago
Is there anything in sphider that allows us to prevent a site from being indexed when we tell the spider it can leave a site it would be nice to have a do not index do not follow to in the data base. I know there is the advance filter options in the advanced but that is mainly for some site filtering. for instance a site maybe linked to google.com that means I have add to the filter must nby rr1024 - Sphider Support
Re: Index base page with greater words admin panel - 5 years ago
Oh....good...prevents keyword spamming...good thinkingby rr1024 - Sphider Support
Re: More Memory Issues - 5 years ago
Thanks REC.... Using both mods In top of page for spider.php added ini_set("memory_limit","-1" Then replaced $result = mysql_query("select link from ".$mysql_table_prefix."temp where id='$sessid'" With $result = mysql_query("select link from ".$mysql_table_prefix."temp where id='$sessid' limit 0,25" and spider managed to inby rr1024 - Sphider Support
Re: More Memory Issues - 5 years ago
Well correction it seems I have two problems now, the above fix allowed me to start indexing again, before it wouldn't even start it would go write into a memory error. Now when I index a large site like wikipedia it indexes for about 10 minutes and then tosses this error. Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 896529 bytes) in /home/public_html/tby rr1024 - Sphider Support
Re: More Memory Issues - 5 years ago
Problem seem to be fixed with this modby rr1024 - Sphider Support
Re: Allowed memory size of 33554432 bytes exhausted - 5 years ago
Great mod fixed my problem tooby rr1024 - Sphider Support
Re: More Memory Issues - 5 years ago
Maybe unset($vars); after each page is indexed would be the way to go...that may resolve the issue with page mem problemsby rr1024 - Sphider Support
Re: More Memory Issues - 5 years ago
Memory Usage = 551624 Continuing suspended indexing. Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 88 bytes) in /public_html/testing/bot/admin/spider.php on line 451 Using ECHO "Memory Usage = ". memory_get_usage()."<br />"; Mine is a shared server so I think that each function in spider.php unset($vars); to prevent hittingby rr1024 - Sphider Support
More Memory Issues - 5 years ago
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 93 bytes) in /public_html/testing/bot/admin/spider.php on line 449by rr1024 - Sphider Support
Index base page with greater words admin panel - 5 years ago
In the admin panel I set the words to 300 I started indexing a site and most pages were not indexed because it didn't meet the 300 word min. 38. Retrieving: at 19:10:41. Size of page: 21.08kb. Starting indexing at 19:10:43. Page contains less than 300 words Links found: 65. New links: 0 The issue was I went to the page and it clearly did have more than 300 words even if excluding linksby rr1024 - Sphider Support
Re: V1.3.3 Re-indexing problem - 5 years ago
"Once the page is removed from the index, re-index will lose more pages, never to index them again!" Maybe it would be better to "not-remove from index" but to set a flag to not display until the spider returns to check page again. This would be a option in the admin panel page failure = 5 so there are 5 chances before the page is removed from index.by rr1024 - Sphider Support
Re: Question about sphider and new domains? - 5 years ago
Then how to we capture that from the admin panel and tell it to promote the new link to a site to index... because once you have indexed a site in the admin panel the output page shows the link but if you don't write it down or memorize it then the only way is to manually search the db but then again you don't know what your looking for. seems like a good mod to have a page that finds all the lby rr1024 - Sphider Support
Re: View page in admin panel - 5 years ago
admin.php?f=21&site_id=1 doing the browse site urls and I wanted to see how each page is ranked and which keywords and the keyword ID's in the db are used to rank said page.by rr1024 - Sphider Support
View page in admin panel - 5 years ago
I was looking at pages in the admin panel and was wondering if there was a way to view the page rank for give key words?by rr1024 - Sphider Support
Sphider and using templates - 5 years ago
Have you guys consider using a template engine instead of mixing the html and php. In search.php you use and include but it's still kinda messy and the admin panel is a total mess....sorry to say that but Sphider is a great script so don't get me wrong...I really like it I've been using this which is an update from the orignal fasttemplate.php3 but it has many new nice features and workby rr1024 - Sphider Mods
Index of subdomains - 5 years ago
I was playing around and found that when you add a site from the admin panel and then tell it to index to level 2 from the www.mydomain.com and the site has a sub domain link such as sub.mydomain.com it didn't follow the link. Is there a setting for this or do I have to add it as a seperate domain?by rr1024 - Sphider Support
Re: Avoid no keyword submit...anyone please help - 5 years ago
my bad...sorry didn't test it FIND if (isset($_GET['query'])) $query = $_GET['query']; REPLACE WITH if (isset($_GET['query'])) $query = TRIM( $_GET['query'] ); Here you may consider triming all your $_GET vars....up to you of course Then do IF ( !EMPTY( $query ) ) { $search_results = get_search_results($query, $start, $category, $type, $results, $domain ); require("$templaby rr1024 - Sphider Support
Re: how to backup/restore huge database? - 5 years ago
This big dump works great...I use it all the time for large sql files for really big ones I just turn it on at night go to bed and it's usually done but that was a couple 20G db's It's very handy but you must not leave it on your server! I always delete it when doneby rr1024 - Sphider Support
Re: Avoid no keyword submit...anyone please help - 5 years ago
Something that may work also and be a little cleaner IF ( !EMPTY( TRIM( $query ) ) ) { $search_results = get_search_results($query, $start, $category, $type, $results, $domain ); require("$template_dir/$template/search_results.html" } remove the periodsby rr1024 - Sphider Support