Welcome! Log In Create A New Profile

Advanced

facebook indexing

Posted by 9716181663 
facebook indexing
April 25, 2012 09:07AM
Dear Sir/Mam

Now i am facing another problem. when i am trying to index facebook.com it shows me error like :

Disallowed files and directories in robots.txt:
http://facebook.com/

1. Link http://facebook.com/: file checking forbidden in robots.txt file

Completed at 00:56:54.


when i look into robots.txt file on my server that is empty. Please Reply as soon as possible.

Thanks
Rahul
errahulsinghsolanki@gmail.com
Re: facebook indexing
April 25, 2012 09:37AM
This is due to the robots.txt file on the Facebook server.

As you can not change that file you have to live with those restrictions, I guess.
Re: facebook indexing
April 25, 2012 09:48AM
Thanks Friend You are right.
Tec
Re: facebook indexing
April 25, 2012 10:36AM
Alternatly you may use a mod called "Temporary ignore robots.txt", which has been published nearly 5 years ago for the original Sphider.
Don't try to search for it by means of the above search form. The PHORUM search engine is most crummy.
Here is the direct link:
http://www.sphider.eu/forum/read.php?3,1081

If you don't like to mod your Sphider installation, Sphider-plus contains this mod as state of the art.

Tec
Re: facebook indexing
May 17, 2012 06:54PM
Might not work without tweaks:

https://www.facebook.com/robots.txt

# Notice: if you would like to crawl Facebook you can
# contact us here: http://www.facebook.com/apps/site_scraping_tos.php
# to apply for white listing. Our general terms are available
# at http://www.facebook.com/apps/site_scraping_tos_terms.php
Sorry, only registered users may post in this forum.

Click here to login