Export url list

This script exports your current domains from Sphider database to a file called url.txt
This file will contain all URLs, each with its individual spider depth and, if available, the category.
url.txt will be placed in folder .../admin/ so it is compatible with the script "Import URL list" which I published here:
[www.sphider.eu]

Open .../admin/admin.php
Search for
function showsites($message) {

Now include as additional part of the selectable items:

<li> <a href='admin.php?f=42'><br>Export the current URL list</a><br></li>


In .../admin/admin.php search for

switch ($f) {

Include as case42 the following code:

case 42:
print "<b><br><center>Export URL list</center></b><br><br>\n";

// Create filename and open file
$filename = "url.txt";
// If you prefer differennt filenames for each export
// uncomment the following 2 rows
//$date = date("d.m.Y-H.i.s";
//$filename = "url_$date.txt";
if (!$handle = fopen($filename, "w") {
print "Unable to open $filename";
exit;
}

print ("Starting to export to file: $filename <br><br>";

// Get url and spider_depth from database
$result = mysql_query("select * from ".$mysql_table_prefix."sites order by url";
echo mysql_error();
$rows = mysql_num_rows($result);
for ($i=0; $i<$rows; $i++)
{
$site_id = (mysql_result($result, $i, "site_id");
$url = (mysql_result($result, $i, "url");
$spider_depth = (mysql_result($result, $i, "spider_depth");

echo ("<br>&nbsp;&nbsp;&nbsp;&nbsp;$url<br>";

// Search for possible category_id
$res_id = mysql_query("select * from ".$mysql_table_prefix."site_category where site_id ='$site_id'";
echo mysql_error();
$cat = mysql_fetch_array($res_id);
$cat_id = $cat['category_id'];

// If exist, get name of category
if ($cat_id != '0') {
$res_cat = mysql_query("select * from ".$mysql_table_prefix."categories where category_id ='$cat_id'";
echo mysql_error();
$cat = mysql_fetch_array($res_cat);
$category = $cat['category'];
}
// Now write all data to file
if (!fwrite($handle, "$url,$spider_depth,$category\n") {
print "Unable to write to $filename";
exit;
}
}

// Close file
fclose($handle);

print "<br><br><hr><center><b><br>Export finished !</b></center><p />";

break;


Happy coding

Tec

i understand that this export the sites that have been added to Sphider, what do i have to change when i want the complete list of links in the database written to the file.

Is this possible with a few enhancements to this mods.

Thanks in advance.

Peter

antpeks

You think that Tec's MOD doen'st give the complete list of links?
or
You need help to apply Tec's MOD?

i have implemented tec, solution, and it exports only the sites i have added, due to indexing i have over 50.000 links in the database and mayber 5000 sites. What i need is a solution like TEc's one where i can export the whole LINK list on the sites list.

Peter

At this time I haven't means to test a solution.
(I was convinced that there was an export of all url's.

Peter:

As you correctly noted, the script only exports the URLs you before entered into Sphiders admin section.
In order to export additionally all links from your database, the easiest and fastest way will be using something like phpMyAdmin.
[www.phpmyadmin.net]
But be aware that the table $mysql_table_prefix"links" is not static. It is modified every time you index/reindex. So I decided to export only (all) the URLs as known from admin section.

Tec



Edited 1 time(s). Last edit at 08/12/2007 02:24AM by Tec.

thanks for the answer Tec. I will use PHPmyadmin.

Peter