indexing files in protected directories

Hi,

I would like to add search function to my website. The problem is, some areas of the website have restricted access (via .htaccess file, using apache authetication). Is there a way to get Sphider to index these pages too?


thanks

I have this exact problem. Short of modifying the Sphider code, I don't think this is possible. I haven't decide yet if I am going to attempt to modify the source code.

For the time being I have decided to temporarily replace user-based authentication with host based authentication (just so that the server and myself can access the protected directory) - effectively taking it offline whilst I run the indexing. Once the indexing has been done, I can return it to user-based authentication.

It would be nice if Apache could combine user-auth with host-auth so that you could require someone to login unless they came from a particular host (the one running the spider in this case).

On my system I have a number of different directories, which can be accessed by different people - so I will probably have to install a copy of Sphider into each directory, each with its own database. This way only people who have access the directory can search it.

Not the simplest of tasks, however it is a lot simpler than it would be without Sphider.

I never got around to implementing Sphider until recently. I know this thread is a little old, but I thought it worth spending a couple of minutes describing my work-around in case anyone else has a similar problem.

The solution I went for was to edit the apache conf such that it also used host based authentication and allow unconditional access from the server where Sphider was located. I put this in my Apache config, but it will probably work in an .htaccess file.

Basically, I put the following directives after the http-auth directives:
Order deny,allow
Deny from all
Allow from <ip address of server>
Satisfy Any

I am on a dedicated server, obviously if you were on a shared server this wouldn't be a good solution. In such cases the work around given by webnetix would be better, [www.sphider.eu]

The solution by webnetix does not work for a .htaccess protected web site.

After adding this code to the end of conf.php and changing the username and password to working values, the code prints at the top of every page in the admin and executing an index still results in Unreachable: http 401.

Unless I am missing something, this solution is useless.

The solution provided above is a suitable workaround.



Edited 1 time(s). Last edit at 04/21/2009 12:42AM by leogab.

I have tried webnetix's solution and it works fine.

This worked beautifully for me. I've been going nuts trying to figure out why I couldn't index my local host. I just commented out the Auth stuff and away she goes. Thanks