Google Answers Logo
View Question
Q: Information growth ( Answered 5 out of 5 stars,   1 Comment )
Subject: Information growth
Category: Computers > Internet
Asked by: will_fawcett-ga
List Price: $8.00
Posted: 23 Mar 2004 10:21 PST
Expires: 22 Apr 2004 11:21 PDT
Question ID: 319657
At what rate does information on the internet grow? What are
predictions for the future? Berkley university have done some research
into this, so they might be a good place to start.

Clarification of Question by will_fawcett-ga on 24 Mar 2004 08:58 PST
Who is answering this question at the moment?
Subject: Re: Information growth
Answered By: omnivorous-ga on 24 Mar 2004 14:33 PST
Rated:5 out of 5 stars
Will_Fawcett --

A very interesting question which has generated enormous debate. 
After a period of explosive growth between 1995-2000, there are those
who argue that growth of "the Internet," defined as PUBLIC web page
sites, has slowed to zero.  In the O'Neill/Lavoie/Bennett's article in
D-Lib Magazine in April, 2003, they actually argue that the Internet
may have shrunk between 2001-2002:
D-Lib Magazine
"Trends in the Evolution of the Public Web" (April, 2003)

The three authors work for the Online Computer Library Center, which
began to size the Internet in 1997.  They estimated that by June, 2002
there were 3,080,000 websites in the "public" web or 35% of the total
of 1.4 billion pages.  They use estimates from Shapiro and Varian in
their book "Information Rules" A Strategic Guide to the Network
Economy" to estimated that it's the equivalent of about 1.5 million
books -- or a fraction of a good university library.
"Information Rules" (Shapiro & Varian)

Indeed Berkeley has some of the best information, particularly the
study "How Much Information?" which did a 2000 and 2003 study,
indicates that the web is growing dramatically -- in contrast to the
D-Lib article.  The interesting aspect of the Berkeley study is that
it also attempts to measure the TOTAL amount of information available
-- including paper:
Berkeley School of Information Management Systems

In their 2000 study, the estimates were:
Public web: 14-28 terabytes
Total web including "deep web": 25-50 terabytes 
Average web site: 441 pages
Average page size: 10K-20K bytes

In the 2003 study, the estimates are far higher for both the public
web and private web, probably because better statistical sampling
techniques were used AND continued growth of information available via
the web:
Public web: 167 terabytes
Total web including "deep web": 66,800-91,850 terabytes

Here are links to the 2 studies:
UCal Berkeley
"How Much Information?" (2000)

UCal Berkeley
"How Much Information?" (2003)

BrightPlanet did the core work on the "deep web" and its size, which
put the remote portion of the web at 400-550 times larger than the
public web.  The "deep web" includes all of the pages that are not
searchable, including public databases with a CGI interface; private
Internet pages run by companies; proprietary databases like
Lexis/Nexis or Thomson Gale databases; and pages that block search
robots for various reasons:
"Deep Web White Paper," (Bergman, July 2001)

It's pretty clear that the growth -- and growth by category -- is
pretty dramatic.  Only new domain registrations show a slowdown,
mentioned in the D-Lib article.  But intranets within corporations are
growing rapidly, with many corporations going to electronic
documentation.  Yet we may never have good estimates of growth because
early measures missed so many areas.

And search engines are struggling to penetrate these recesses of the
Internet in order to retain their value as the heartbeat (or perhaps
the "brains" is more accurate) of the Internet.  Eighteen months ago
Google estimated that it reached about half of the 4 billion pages on
the Internet.  A recent Google press release now claims about 4.28
billion web pages (with reach into images and message boards expanding
the total index to 6 billion items):
Google, Inc.
"Google Achieves Search Milestone," (Feb. 17, 2004)

Google search strategy:
"size of the Internet"
"limits of the Internet"

Best regards,


Request for Answer Clarification by will_fawcett-ga on 25 Mar 2004 06:54 PST
how many MB is a terabyte?

Clarification of Answer by omnivorous-ga on 25 Mar 2004 07:02 PST
Will_Fawcett --

A terabyte is 1 million megabytes.  And a megabyte, of course, is 1 million bytes.

It's probably easier to start using exponential figures because the
masses of data are growing so rapidly:
UC Berkeley
"Data Powers of Ten" (2000)

Best regards,

will_fawcett-ga rated this answer:5 out of 5 stars
Great answer, v helpful

Subject: Re: Information growth -- How many Bytes is a Tera byte
From: pgmer6809-ga on 21 Mar 2005 15:16 PST
The answer given is 10^9 and this is correct if you follow the standard strictly.
However in the computer world there is some debate.
Disk makers use 1 megabyte as 10^6 bytes. Memory makers use a kilobyte
as 2^10 = 1024 bytes and a megabyte as 1024 KB or 2^20 bytes.  A
gigabyte of memory would then be 1024 Megabytes, or 2^30 bytes, and a
terabyte = 1024 Megabytes or 2^40 bytes.
See this link for a comparaison of sizes and terminology.

Important Disclaimer: Answers and comments provided on Google Answers are general information, and are not intended to substitute for informed professional medical, psychiatric, psychological, tax, legal, investment, accounting, or other professional advice. Google does not endorse, and expressly disclaims liability for any product, manufacturer, distributor, service or service provider mentioned or any opinion expressed in answers or comments. Please read carefully the Google Answers Terms of Service.

If you feel that you have found inappropriate content, please let us know by emailing us at with the question ID listed above. Thank you.
Search Google Answers for
Google Answers  

Google Home - Answers FAQ - Terms of Service - Privacy Policy