Sphider not indexing some pages [ SOLVED ]

Re: Sphider not indexing some pages [ SOLVED ]

naresh — Sun, 19 Jun 2011 15:45:26 +0300

Keith,

No, what I was saying is: In sphider you give it a URL in the admin panel to index. If you give this URL without the www then it is not indexing any links on that page with the www. Your issue seems to be a different one. Your links are relative paths which don't begin with http.

But while searching for my issue I found a post on this forum which seemed to have addressed your issue. If I remember correctly what it said was that Sphider indexes the relative links when they begin as "/abc/def/" but does not index if they begin as "abc/def/" ( notice the missing / in the beginning ). I tried to quickly find that post but I could not. Search for it with keywords like "sphider not indexing" or something like that. Good luck.

Naresh.

Re: Sphider not indexing some pages [ SOLVED ]

ketl70 — Sat, 18 Jun 2011 19:20:19 +0300

My I ask a simple question?

I'm new to PHP and all its requirements and cannot get Sphider to follow any of my links. example: href= "/arts/notes.html

Are you saying that all the links must have the full URL to the page you want indexed. ie; http:// mysite.com/arts/notes.html
or [www.mysite.com]

Loaded Sphider with no issues just will not index. Any and all suggestions will be appreciated.

Thanks

Sphider not indexing some pages [ SOLVED ]

naresh — Sat, 18 Jun 2011 08:15:20 +0300

Sorry, I posted too soon. I just figured it out. When I indexed my site without 'www' Sphider was unable to index all the links which had the www. For example when I indexed [example.com] then spider missed the links
< a href = "[www.example.com/page1www.example.com];

I didn't know how to delete the post so I just edited it. :)

Great job guys. I was trying to use Google custom search which makes no sense because after I add content to my site, it seems I just have to wait for Google to index it at its own sweet time. I just installed Sphider and the installation was so easy. The search results are great too and will fit right into my tiny little website :).