1.3.5 does not obey robots.txt

Re: 1.3.5 does not obey robots.txt

Tuzi — Wed, 20 Feb 2013 05:46:26 +0200

Before the release of seven with Mount accompany went from 1.03 now quite busy recently,D3 Gold so it is no time to continue to release. The purpose of this set is still the same, that is, want to give you a scientific fitted ideas, so as to achieve full anti, blood, steal relatively satisfied with the panel, looking pickup blood cells and other related additional properties and from what part of Value. Of course, these additional properties is not to say this one is the best,buy d3 gold but also according to the specific circumstances of AH to decide, just an idea.

Re: 1.3.5 does not obey robots.txt

docbear — Sun, 27 Jan 2013 09:08:35 +0200

Thank you very much! Works very well now.

I owe you a pint.

(tu)

Re: 1.3.5 does not obey robots.txt

halfacre — Fri, 02 Dec 2011 02:41:09 +0200

Kudos on that patch, Matt. I've been pounding my head against this for days thinking I was doing something wrong. You're my hero.

Re: 1.3.5 does not obey robots.txt

berbatov — Fri, 21 Jan 2011 01:20:22 +0200

Hello Everyone Thank you :):)

Re: 1.3.5 does not obey robots.txt

rossv — Thu, 13 Jan 2011 10:04:49 +0200

Thanks heaps for your patch, that solved a big hassle I was having..!

Ross..

Re: 1.3.5 does not obey robots.txt

ClickRaider — Sun, 29 Aug 2010 07:40:56 +0300

Some sites have different rules set in the META TAGS for robots regardless of robot.txt file, some are not set for sphider unless they say "index, follow".

Re: 1.3.5 does not obey robots.txt

Malcolm — Fri, 06 Aug 2010 05:14:04 +0300

Thankd man, perfect!

Re: 1.3.5 does not obey robots.txt

matt — Thu, 04 Mar 2010 20:34:27 +0200

Hi again all, there's yet another problem in the check_robot_txt function within spiderfuncs.php.

around line 217, it has

return null;

this tells it to return null and exit the function when disallow rule returns nothing. Well, now this is a huge problem because most robots.txt start with a general "disallow none" rule and then further restrict directories.

i.e,

User-agent: *
Disallow:
Disallow:  /cgi-bin/ 
Disallow: /private/

and on and on, the way this is written it exits the function when it see the first Disallow rule. So change the return null line (line 217) to this;

continue;

this tells the script to simply continue to check the next line within the loop.

Now it finally parses and applies rules from robots.txt correctly.

Now when you index you should see the disallowed rules at the top of the output/logs, i.e.

Disallowed files and directories in robots.txt:
http: //example.com/cgi-bin/
http: //example.com/private/

etc, etc. I added a space because the forum software keeps wanting to linkify the text.

I have updated the patch at [mdj.us]

Re: 1.3.5 does not obey robots.txt

Willy — Thu, 25 Feb 2010 19:40:51 +0200

Thanks Matt,

I tiny but significant change...(tu)

Re: 1.3.5 does not obey robots.txt

t-p — Mon, 22 Feb 2010 23:39:28 +0200

Thanks matt for taking time to help, really appreciate it.

Re: 1.3.5 does not obey robots.txt

matt — Mon, 22 Feb 2010 19:34:18 +0200

Hey guys,

I discovered the same thing, looks like when the author updated the eregi functions to preg_match, he forgot to use the case modifier.

Basically, if you have User-agent instead of user-agent, it won't count it.

Anyways, I have created a patch for it here; [mdj.us]

If you don't know what the hell that is, then just open spiderfuncs.php and look for preg_match and make sure to add the "i" modifier in the regular expression if it's missing. There are three instances to change.

Re: 1.3.5 does not obey robots.txt

Convergence — Thu, 11 Feb 2010 03:13:05 +0200

t-p Wrote:
-------------------------------------------------------
> Hi,
>
> I updated my 1.3.4 by replacing th changed files.
>
>
> I noticed that the 1.3.5 does not completely
> follow (it follows some and indexed some!) my
> robots.txt, while the 1.3.4 did follow fully.
>
> Can anybody guide me please to solve this problem?
> Do I have to have php5+ to use 1.3.5? Thanks.

t-p,

I just, today, installed 1.3.5 and also noticed it does not obey robots.txt. - I would be interested if anyone has a fix for this, as well. I have two existing installs using 1.3.4 (but they show 1.3.3 in admin...)

BTW, I'm using php5 on my servers and that is not the problem...

1.3.5 does not obey robots.txt

t-p — Thu, 07 Jan 2010 00:11:46 +0200

Hi,

I updated my 1.3.4 by replacing th changed files.

I noticed that the 1.3.5 does not completely follow (it follows some and indexed some!) my robots.txt, while the 1.3.4 did follow fully.

Can anybody guide me please to solve this problem? Do I have to have php5+ to use 1.3.5? Thanks.