SP 1.5 - not obeying robots.txt in certain instances? directory structure: /public_html/ ../templates/ /templates/ is .htaccess protected and in robots.txt i have: User-agent: * Disallow: /templates/ but Sphider+ is trying to access the directory (i'm being prompted for authentication) other than this particulap problem, SP seems to be following robots.txt fine. http://www.sphider.eu/forum/read.php?3,5211,5211#msg-5211 Wed, 19 Jun 2013 12:25:42 +0300 Phorum 5.2.10 http://www.sphider.eu/forum/read.php?3,5211,5211#msg-5211 SP 1.5 - not obeying robots.txt in certain instances? http://www.sphider.eu/forum/read.php?3,5211,5211#msg-5211
/public_html/
../templates/

/templates/ is .htaccess protected and in robots.txt i have:

User-agent: *
Disallow: /templates/

but Sphider+ is trying to access the directory (i'm being prompted for authentication)

other than this particulap problem, SP seems to be following robots.txt fine.]]> sphider.Man Sphider Mods Tue, 21 Oct 2008 20:12:44 +0300