Google Answers Logo
View Question
 
Q: Hiding from web crawlers ( No Answer,   3 Comments )
Question  
Subject: Hiding from web crawlers
Category: Computers > Internet
Asked by: susank-ga
List Price: $5.00
Posted: 07 Mar 2004 13:15 PST
Expires: 11 Mar 2004 06:56 PST
Question ID: 314324
I have a password protected site. Can the webcrawlers - for instance
those that do the indexing for google - get onto the site and index
what is 'behind' the password protected area? Specifically, can they
index materials that are put there for downloading?
Answer  
There is no answer at this time.

Comments  
Subject: Re: Hiding from web crawlers
From: robertskelton-ga on 07 Mar 2004 18:40 PST
 
If a human cannot access the files without a password, then a web
crawler cannot either.
Subject: Re: Hiding from web crawlers
From: jakkwylde-ga on 07 Mar 2004 19:59 PST
 
That is not necessarily the case, but more often than not it is.  A
lot depends on the security mechanism implemented by the developer.

To make sure that your pages are not followed by the google crawler,
I'd be sure to include a robots.txt files, along with appropriate
metatags in your source for the password protected areas.

To read more about robots.txt, and the metatags, check out the following links:

description of robots.txt
====================================================
http://www.robotstxt.org/wc/exclusion.html#robotstxt

more on robots.txt
====================================================
http://www.searchengineworld.com/robots/robots_tutorial.htm

description of relevant metatags
====================================================
http://www.robotstxt.org/wc/exclusion.html#meta

standards for robots/crawlers exclustion
====================================================
http://www.robotstxt.org/wc/norobots.html
Subject: Re: Hiding from web crawlers
From: susank-ga on 11 Mar 2004 06:55 PST
 
Thanks all! I think that does it.

Important Disclaimer: Answers and comments provided on Google Answers are general information, and are not intended to substitute for informed professional medical, psychiatric, psychological, tax, legal, investment, accounting, or other professional advice. Google does not endorse, and expressly disclaims liability for any product, manufacturer, distributor, service or service provider mentioned or any opinion expressed in answers or comments. Please read carefully the Google Answers Terms of Service.

If you feel that you have found inappropriate content, please let us know by emailing us at answers-support@google.com with the question ID listed above. Thank you.
Search Google Answers for
Google Answers  


Google Home - Answers FAQ - Terms of Service - Privacy Policy