sphider return internal server 500 for some sites for ex:

Hi,

Why does the following server return error 500 for the following url?
[www.diplomatie.be]

When i visit with browser it shows the content.

guess its had to do with the server configuration of that site, but how can it be solved?

Best regards,
Ugur

Sometimes webmasters are strange people. Obviously the webmaster of that URL is a little bit restrictive against all kind of crawlers. For example he does not accept a user agent string like 'Sphider'. Just change the according Admin setting for example to
Mozilla/5.0
Afterwards also a crawler like Sphider will index that site. Hopefully nobody tells him . .

A good example that not always the own server does have a connection problem whenever 'error 500' messages appear. Often discussed as the reason for this error message.

Tec