Welcome! Log In Create A New Profile


facebook indexing

Posted by 9716181663 
facebook indexing
April 25, 2012 06:07AM
Dear Sir/Mam

Now i am facing another problem. when i am trying to index facebook.com it shows me error like :

Disallowed files and directories in robots.txt:

1. Link http://facebook.com/: file checking forbidden in robots.txt file

Completed at 00:56:54.

when i look into robots.txt file on my server that is empty. Please Reply as soon as possible.

Re: facebook indexing
April 25, 2012 06:37AM
This is due to the robots.txt file on the Facebook server.

As you can not change that file you have to live with those restrictions, I guess.
Re: facebook indexing
April 25, 2012 06:48AM
Thanks Friend You are right.
Re: facebook indexing
April 25, 2012 07:36AM
Alternatly you may use a mod called "Temporary ignore robots.txt", which has been published nearly 5 years ago for the original Sphider.
Don't try to search for it by means of the above search form. The PHORUM search engine is most crummy.
Here is the direct link:

If you don't like to mod your Sphider installation, Sphider-plus contains this mod as state of the art.

Re: facebook indexing
May 17, 2012 03:54PM
Might not work without tweaks:


# Notice: if you would like to crawl Facebook you can
# contact us here: http://www.facebook.com/apps/site_scraping_tos.php
# to apply for white listing. Our general terms are available
# at http://www.facebook.com/apps/site_scraping_tos_terms.php
Sorry, only registered users may post in this forum.

Click here to login