Welcome! Log In Create A New Profile


Allow spiders or crawlers to crawl your entire database

Posted by ClickRaider 
Allow spiders or crawlers to crawl your entire database
March 29, 2010 03:18AM
Fairly simple code. Create a new php file , I called mine data1.php, data2.php and so on up until data10.php. Insert the following duplicated code into each file. Create link so robots can index and follow to the data php files. I put mine in my site logo by adding a hyperlink to the image. That link takes them to a page that then has links to all data files which you will need to create too. Example here: http://www.clickraider.com/robots.html or go to http://www.clickraider.com and click on the logo. It is kind of like a hidden site map. Replace winking smiley with ).

I set them up so results can be random each time the site is crawled and I created all 10 to make the chances of randomizing higher.

I set the results limit to 1000 so that the spider will not exhaust itself and stop crawling.

This code will allow engines to spider your keywords domains and descriptions. I was going to put it in the header, but some engines have a limit of how many keywords you can use in META TAGS.

Heres the code. Remember to replace the XXXXXXX with your sites info.


$include_dir = "./include";
include ("$include_dir/commonfuncs.php"winking smiley;

$include_dir = "./include";
$template_dir = "./templates";
$settings_dir = "./settings";
$language_dir = "./languages";

require_once("$settings_dir/database.php"winking smiley;
require_once("$language_dir/en-language.php"winking smiley;
require_once("$include_dir/searchfuncs.php"winking smiley;
require_once("$include_dir/categoryfuncs.php"winking smiley;

include "$settings_dir/conf.php";

include "$template_dir/$template/header.html";?>
include "$language_dir/$language-language.php";

$query = "SELECT url, title, description FROM XXXXXXXXX ORDER BY RAND() LIMIT 0, 1000";//Replace XXXXX with your database table name for links.
$result = mysql_query($query);

while($row = mysql_fetch_array($result, MYSQL_ASSOC)){

echo "<a href=\"http://www.XXXXXXXXX.com/search.php?query={$row['title']}&search=1\">{$row['url']}</a><br>" .
"{$row['title']} <br>" .
"ClickRaider Results: {$row['description']} <br><br>";


include "$template_dir/$template/footer.html";
Sorry, only registered users may post in this forum.

Click here to login