Welcome! Log In Create A New Profile

Advanced

Only Index New Pages?

Posted by belgin fish 
Forum List Message List New Topic
belgin fish
Only Index New Pages?
November 29, 2009 09:14PM
Hi, I'm looking for a mod that would only index new pages of a website, and not reindex the whole site again. Is this possible? How could i do it?

Thanks
Reply Quote
Willy
Re: Only Index New Pages?
November 29, 2009 11:50PM
Use the follow sitemap modification.
You can find it here, [www.sphider.eu]
You will need a sitemap.xml for this to work.
Only urls with a lastmod date newer than the last index date and new pages will be indexed, the others will not.



Edited 2 time(s). Last edit at 11/29/2009 11:56PM by Willy.
Reply Quote
belgin fish
Re: Only Index New Pages?
November 30, 2009 12:24AM
thank you, i appreciate it!
Reply Quote
belgin fish
Re: Only Index New Pages?
December 03, 2009 05:04AM
Ok, well I've tried to get it working but it's not sad smiley it just keeps indexing every page.
sad smiley



Edited 1 time(s). Last edit at 12/03/2009 05:07AM by belgin fish.
Reply Quote
Willy
Re: Only Index New Pages?
December 03, 2009 11:15AM
Did you create a correct sitemap.xml?
Reply Quote
belgin fish
Re: Only Index New Pages?
December 03, 2009 02:44PM
But like, if I make the sitemap will it only index links from the sitemap.xml?
Reply Quote
Willy
Re: Only Index New Pages?
December 03, 2009 03:24PM
Yes, that's the idea.
It has to be a sitemap in .xml format, with a lastmod date for each url. Because it is that lastmod date which tells sphider if it needs to reindex that url or not.
Reply Quote
belgin fish
Re: Only Index New Pages?
December 03, 2009 10:57PM
ok, what tool do you suggest i use to have an auto updated sitemap?
Reply Quote
Willy
Re: Only Index New Pages?
December 04, 2009 12:07AM
I don´t know of a specific tool, there have to be several.
Here is a modification for Sphider, [www.sphider.eu] but I have no experience with it.
And there is [gsitecrawler.com] or [enarion.net]



Edited 1 time(s). Last edit at 12/04/2009 12:11AM by Willy.
Reply Quote
belgin fish
Re: Only Index New Pages?
December 04, 2009 01:53AM
Ok, I Tried using the one for sphider, but i don't get how it's going to work with the only index new files since it creates sitemaps not in the root directory and not with the name "sitemap.xml"
Reply Quote
Willy
Re: Only Index New Pages?
December 04, 2009 10:30AM
You can change the path to where Sphider is supposed to find the sitemap created altering 
$input_file = "$url2/$sitemap_name"; // create path to sitemap
Reply Quote
belgin fish
Re: Only Index New Pages?
December 04, 2009 02:42PM
but if it creates a different sitemap for each url what should i do?
Reply Quote
Willy
Re: Only Index New Pages?
December 04, 2009 05:08PM
I don´t thnk it creates a different sitemap for each url, maybe it does for each site indexed.
Also try the other two tools I linked to.
Reply Quote
belgin fish
Re: Only Index New Pages?
December 04, 2009 11:26PM
Ya i ment it makes a new one for each site. And I'll try the other options.
Reply Quote
Willy
Re: Only Index New Pages?
December 05, 2009 01:04AM
The follow sitemap function will only work for sites with a sitemap.xml in their root directory.



Edited 1 time(s). Last edit at 12/05/2009 10:00AM by Willy.
Reply Quote
Newer Topic Older Topic
Sorry, only registered users may post in this forum.

Click here to login