![]() |
|
![]() | ||
|
Subject:
Hiding from web crawlers
Category: Computers > Internet Asked by: susank-ga List Price: $5.00 |
Posted:
07 Mar 2004 13:15 PST
Expires: 11 Mar 2004 06:56 PST Question ID: 314324 |
I have a password protected site. Can the webcrawlers - for instance those that do the indexing for google - get onto the site and index what is 'behind' the password protected area? Specifically, can they index materials that are put there for downloading? |
![]() | ||
|
There is no answer at this time. |
![]() | ||
|
Subject:
Re: Hiding from web crawlers
From: robertskelton-ga on 07 Mar 2004 18:40 PST |
If a human cannot access the files without a password, then a web crawler cannot either. |
Subject:
Re: Hiding from web crawlers
From: jakkwylde-ga on 07 Mar 2004 19:59 PST |
That is not necessarily the case, but more often than not it is. A lot depends on the security mechanism implemented by the developer. To make sure that your pages are not followed by the google crawler, I'd be sure to include a robots.txt files, along with appropriate metatags in your source for the password protected areas. To read more about robots.txt, and the metatags, check out the following links: description of robots.txt ==================================================== http://www.robotstxt.org/wc/exclusion.html#robotstxt more on robots.txt ==================================================== http://www.searchengineworld.com/robots/robots_tutorial.htm description of relevant metatags ==================================================== http://www.robotstxt.org/wc/exclusion.html#meta standards for robots/crawlers exclustion ==================================================== http://www.robotstxt.org/wc/norobots.html |
Subject:
Re: Hiding from web crawlers
From: susank-ga on 11 Mar 2004 06:55 PST |
Thanks all! I think that does it. |
If you feel that you have found inappropriate content, please let us know by emailing us at answers-support@google.com with the question ID listed above. Thank you. |
Search Google Answers for |
Google Home - Answers FAQ - Terms of Service - Privacy Policy |