Forum List
Message List
New Topic
sphider.Man
SP 1.5 - not obeying robots.txt in certain instances?
October 21, 2008 08:12PM
Registered: 4 years ago
Posts: 4
directory structure:
/public_html/
../templates/
/templates/ is .htaccess protected and in robots.txt i have:
User-agent: *
Disallow: /templates/
but Sphider+ is trying to access the directory (i'm being prompted for authentication)
other than this particulap problem, SP seems to be following robots.txt fine.
Reply
Quote
Newer Topic
Older Topic
Print View
RSS
Sorry, only registered users may post in this forum.
Click here to login