Google Answers Logo
View Question
Q: How do I stop screen scraping? ( No Answer,   1 Comment )
Subject: How do I stop screen scraping?
Category: Computers > Security
Asked by: cgreenawalt87-ga
List Price: $3.00
Posted: 08 Nov 2006 11:11 PST
Expires: 01 Dec 2006 12:22 PST
Question ID: 781104
Not exactly sure why this is happening, but a couple of folks have
started to use bots which crawl my site. No, it's not search engine
bots- they show up in the server logs as user agents NetSpider,
WebReaper, and a few others. I looked at ways of blocking crawlers,
and I found a way to modify my server's .htaccess file to disallow
these agents from visiting. But this seems very easy to work
around...what I'd really like is a way to gradually throttle traffic.
Thus, as a client requests unreasonable numbers of pages, we decide to
limit them to one page per 5 seconds or something like that. Any ideas
on ways to do that using an Apache server? I've got full access to the
server, so I can install modules, etc, if that helps. Many thanks.
There is no answer at this time.

Subject: Re: How do I stop screen scraping?
From: czh-ga on 08 Nov 2006 15:00 PST
How do I stop screen scraping?

Important Disclaimer: Answers and comments provided on Google Answers are general information, and are not intended to substitute for informed professional medical, psychiatric, psychological, tax, legal, investment, accounting, or other professional advice. Google does not endorse, and expressly disclaims liability for any product, manufacturer, distributor, service or service provider mentioned or any opinion expressed in answers or comments. Please read carefully the Google Answers Terms of Service.

If you feel that you have found inappropriate content, please let us know by emailing us at with the question ID listed above. Thank you.
Search Google Answers for
Google Answers  

Google Home - Answers FAQ - Terms of Service - Privacy Policy