Google Answers Logo
View Question
Q: Size of the dark or invisible web - bobbie7-ga ( Answered 5 out of 5 stars,   1 Comment )
Subject: Size of the dark or invisible web - bobbie7-ga
Category: Miscellaneous
Asked by: dinkytwist-ga
List Price: $40.00
Posted: 21 Jun 2006 00:14 PDT
Expires: 21 Jul 2006 00:14 PDT
Question ID: 739886
How large is the invisible or dark web compared to the surface web?
Subject: Re: Size of the dark or invisible web - bobbie7-ga
Answered By: bobbie7-ga on 21 Jun 2006 05:35 PDT
Rated:5 out of 5 stars
Hello again Dinkytwist,

Thank you for asking me to work on this research project.

The Invisible Web, sometimes referred to as the Deep Web, consists of
information stored in searchable databases as well as other
non-standard file types on the web. This material is generally missed
by search engine spiders because they cannot or will not index this

?BrightPlanet has estimated the size of the Invisible Web to be 500
times the size of the accessible web, which would make it
approximately 550 billion web pages.?

?The Invisible Web, sometimes referred to as Deep Web, is estimated to
be more than 500 times larger than what traditional search engines can
Nielsen BuzzMetrics

A highly informative study regarding the size of invisible web.
Graphs, pie charts and detailed statistics are provided. 
Download here:

However these numbers may be overestimated as the following reports point out.

Exploring the Academic Invisible Web by Dr. Dirk Lewandowski
Download here:

The Deep Web: Resource Discovery in the Library of Texas
By Kathleen R. Murray and William E. Moen
February 2004

?A quick search for information about the deep Web is certain to
retrieve the 2001 white paper reporting the results of the DeepPlanet
study that quantified and characterized the content in the surface
versus the deep Web (Bergman, 2001). The study estimates that the deep
Web is 500 times larger than the surface Web and contains higher
quality resources. The number of deep websites is estimated at 200,000
and characterized by rapid growth. Additionally, a full 95% of deep
Web content is publicly accessible, requiring no subscriptions or

?While there is general agreement that the deep Web is home to a
wealth of high quality and primary source data, there is not agreement
as to its size relative to the surface Web. Sherman (2001) asserts
that the DeepPlanet study is flawed in its measurement techniques and
that the deep Web is actually 2 - 50 times larger than the surface

?Regardless of measurement approach, there is no doubt that the deep
Web is much larger than the surface Web trolled by the major search
engines and is believed to be growing rapidly (Bergman, 2001). ?

Download here:

Search Engine Sizes by Danny Sullivan, Editor-In-Chief
January 28, 2005


Below you will find the most recent data regarding the size of the deep web 

Deep Web Research 2006

The January 2006 Issue of LLRX has a feature article written by Marcus
P. Zillman titled Deep Web Research 2006. The guide extensively
documents resources that include articles, books, websites,
presentations, search engines, and technology applications that
facilitate the challenging task of accessing information, published in
many formats, that encompass the hundreds of millions of pages
comprising the "deep web."

Surface web 8 billion pages

Deep web 900 billion pages

 ?The Deep Web covers somewhere in the vicinity of 900 billion pages
of information located through the World Wide Web in various files and
formats that the current search engines on the Internet either cannot
find or have difficulty accessing. The current search engines find
about 8 billion pages at the time of this writing.?

Deep Web Research Research 2006 by Marcus P. Zillman
Published January 15, 2006

About Marcus P. Zillman

?Marcus P. Zillman, M.S., A.M.H.A., is Executive Director of the
Virtual Private Library and Founder/Creator of BotSpotŪ. He is the
author of nine different Internet MiniGuides 2006, Internet Sources
Manual and eCurrent Awareness Resources 2006 Report. His Subject
Tracer? Information Blogs (45 and constantly growing) are freely
available from the Virtual Private Library, which include the latest
resources on Deep Web Research and Bot Research.?

LLRX January 2006 Issue Deep Web Research Featured Article

Search terms used: 
Size of the dark OR invisible web

I hope the information provided is helpful!

Best regards,

Request for Answer Clarification by dinkytwist-ga on 05 Jul 2006 23:32 PDT
Thanks Bobbie7 but it didn't really provide an answer.  Rather than
trying to establish the size of the "Invisible Web" I'd like to ask it
a different way perhaps using Google as an example to establish a
reference to the known/surface or visible web:
1. How many million (or billion) web pages would Google say they have indexed?
1. What percentage of the www would google say this represents?

If this constitutes another question or much more work I would be
happy to pay more or provide a good tip.  Please advise.  Cheers.

Clarification of Answer by bobbie7-ga on 06 Jul 2006 07:52 PDT
While not recent, the following might be helpful.

"July 12, 1999 ? A new study of coverage of the indexable Web by
search engines states that, as of February 1999, only 42 percent of
the Web is indexed by the combined search engines. The study, by Steve
Lawrence and C. Lee Giles of NEC Research Institute, appeared in the
July 8, 1999, issue of Nature (pp. 107-109). Lawrence and Giles made
headlines a year ago with their study of overlap among search engines
that showed that each Web search engine indexed a fairly discrete
corner of the Web, with little overlap among them. In that study,
Lawrence and Giles reported that the combined coverage by all Web
search engines was about 60 percent of the Web."

I would suggest you post a new question without my name in the title
so that all the researchers can get a chance to answer your question.


Clarification of Answer by bobbie7-ga on 06 Jul 2006 07:58 PDT
"Our research indicates that only 30 percent to 40 percent of the Web
is indexed and that people's questions [to search engines] go
unanswered half the time," Gurry said.

IDG News Service
July 28, 2003,aid,111789,00.asp
dinkytwist-ga rated this answer:5 out of 5 stars and gave an additional tip of: $15.00
Again Google Answers has delivered especially the researcher bobbie7-ga

Subject: Re: Size of the dark or invisible web - bobbie7-ga
From: bobbie7-ga on 26 Jun 2006 21:20 PDT
Thank you very much for the five stars and generous tip!

Important Disclaimer: Answers and comments provided on Google Answers are general information, and are not intended to substitute for informed professional medical, psychiatric, psychological, tax, legal, investment, accounting, or other professional advice. Google does not endorse, and expressly disclaims liability for any product, manufacturer, distributor, service or service provider mentioned or any opinion expressed in answers or comments. Please read carefully the Google Answers Terms of Service.

If you feel that you have found inappropriate content, please let us know by emailing us at with the question ID listed above. Thank you.
Search Google Answers for
Google Answers  

Google Home - Answers FAQ - Terms of Service - Privacy Policy