Google Answers Logo
View Question
 
Q: Database Technology ( Answered,   0 Comments )
Question  
Subject: Database Technology
Category: Computers > Internet
Asked by: ace24-ga
List Price: $4.00
Posted: 26 Mar 2003 15:26 PST
Expires: 25 Apr 2003 16:26 PDT
Question ID: 181396
What technology is used by the search engines such as 'Google' to
query their vast database in virtually no time?  I'm interested in
knowing the types of software and hardware used.
Answer  
Subject: Re: Database Technology
Answered By: larre-ga on 26 Mar 2003 16:51 PST
 
Thanks for asking!

Google's search algorithms are proprietary, i.e. they don't share 'em,
however, Google does provide a few glimpses into its hardware and
search technology.


Our Search: Google Technology
*****************************

"Google runs on a unique combination of advanced hardware and
software. The speed you experience can be attributed in part to the
efficiency of our search algorithm and partly to the thousands of low
cost PC's we've networked together to create a superfast search
engine.

The heart of our software is PageRank™, a system for ranking web pages
developed by our founders Larry Page and Sergey Brin at Stanford
University. And while we have dozens of engineers working to improve
every aspect of Google on a daily basis, PageRank continues to provide
the basis for all of our web search tools."

Google Technology
://www.google.com/technology/


Google Technology Overview
************************** 

"Google has developed an advanced search technology that involves a
series of simultaneous calculations typically occurring in under half
a second-without any human intervention. At the heart of this
technology is PageRank™ technology and hypertext-matching analysis
developed by Larry Page and Sergey Brin. Google's search architecture
also is scalable, which enables us to continue to index the Internet
as it expands."

Google Press Center: Technology Overview
://www.google.co.uk/ukpress/overview_tech.html

The Life of a Google Query (diagram)
://www.google.co.uk/ukpress/query.html


Application Programming Interface
*********************************

Google's database power is available to researchers and developers in
the form of API's, or Application Programming Interfaces. "Simply put,
the service lets developers automatically query 2 billion documents
from its database on a limited basis. Then they can publish results as
they choose, as long as it's for noncommercial purposes."

C|NET - Developers Dig Into Google Interface


+ Technology Behind Google
**************************

"Google's hardware is a massive "farm" of more than 10,000 servers,
capable of not only indexing more than 3 billion web documents but
handling thousands of queries per second with sub-second response
times. It's an awesome engineering feat in its own right."

This page includes links a virtual Insider's Tour of Google. 

Search Engine Watch -- Search Day August 12, 2002
Technology Behind Google, by Chris Sherman
http://news.com.com/2100-1023-884546.html


About the Google Dance
**********************

An outsiders look at some of Google's hardware and software
technology. "During the month, Google sends out robots (spiders) to
crawl the web and archive every website it finds. These archived
websites will be presented in the next update of the Google index
(database). The update process is initiated every 30 days or so,
although on some occasions there have been more than 30 days between
updates."

http://www.google-dance.com/HTML-about.html


Google Dance Technology
***********************

"Never heard of the Google Dance? It's an interesting event that
happens (roughly) once a month and causes quite a stir in some web
circles with frustrated, ecstatic, and confused people coming together
at WebmasterWorld to talk about the Google Dance."

Technology and Culture from the Trenches
http://www.kuro5hin.org/print/2003/1/27/221829/873

**********************************************************************

Google Search Terms:

google technology
"google's technology"
"google dance"
"google hardware" OR "google software"


The Fine Print
**************

Google Answers Researchers are independent contractors to Google
Answers. We are not directly employed by Google, nor do we speak
officially for them. We -are- proud of our indirect affiliation with
Google, however. Many of us are Googlephiles and Google-watchers. The
information I've gathered and detailed represents a knowledgeable
search of available web materials to best answer your query.

It's been a pleasure to address your question. If you have any
questions about the material or links provided, please, feel free to
ask.

--larre

Request for Answer Clarification by ace24-ga on 26 Mar 2003 20:16 PST
Thanks Larre,

You have answered part of the question, however I was also interested
in knowing what kind of database (not necessarily by Google) is used
to store this vast information and how several simultaneous queries
are handled at the same time.  If possible can you also please let me
know any commercially (off the shelf) available products to cater to
these needs (ofcourse not to the same scale as Google or other search
engines).

Regards,
ace24-ga

Clarification of Answer by larre-ga on 26 Mar 2003 21:56 PST
Certainly!

Let's start with Open Source. "mnoGoSearch is a full-featured web
search engine software for intranet and internet servers. mnoGoSearch
for UNIX is a free software covered by the GNU General Public License
and mnoGoSearch for Windows is a commercial search software version."

Because mnoGoSearch is Open Source, you can "open the hood" to see
exactly how the code is written and how it works. The software uses an
SQL database.

mnoGoSearch
http://search.mnogo.ru/


Publicly Available Database Software/Nonprofit Organizations
********************************************************************

Software created by Nonprofit entities, such as universities or
government research labs, available free of charge under certain
conditions, or by fee, as determined by the individual package
license.

Association for Computing Machinery 
Special Interest Group on Management of Data
http://www.acm.org/sigmod/databaseSoftware/nonprofit/


Publicly Available Database Software/Commercial Organizations
*********************************************************************

Software created by commercial entities (and their research
departments) available free of charge under certain conditions, or for
a fee, under individual licensing agreements.

Association for Computing Machinery 
Special Interest Group on Management of Data
http://www.acm.org/sigmod/databaseSoftware/commercial/


MySQL Database
**************

MySQL is the world's most popular Open Source database. 

MySQL.com - Products
http://www.mysql.com/products/index.html


Guide to Database Software
**************************

About.com offers reviews of and links to web database software:

About.com - Web Design - Database Software
http://webdesign.about.com/cs/databasesoftware/


ChaCha Search from U.C. Berkeley 
********************************

Cha-Cha uses Ray Larson's Cheshire II search engine, SleepyCat
Software, Inc. database software, The Apache Group's web server and
servlet engine. All listed items are linked from the site home page.
Technical data is also available.

ChaCha
http://cha-cha.berkeley.edu/

Chacha - A System for Organizing Internet Search Results
http://cha-cha.berkeley.edu/papers/usits99/index.html


Oracle
******

Oracle is a commercial database used for many heavy-duty applications.
It's considered scalable, fast and reliable.

Oracle.com
http://www.oracle.com/

Google Search Appliance
***********************

Last, but not least, Google licenses their own technology as a Search
Appliance, an all in one search server/software application.

Google Search Appliance
://www.google.com/appliance/index.html


**********************************************************************

Additional search terms:

"search engine" "database software"


I hope this helps. 

Best regards,

--larre
Comments  
There are no comments at this time.

Important Disclaimer: Answers and comments provided on Google Answers are general information, and are not intended to substitute for informed professional medical, psychiatric, psychological, tax, legal, investment, accounting, or other professional advice. Google does not endorse, and expressly disclaims liability for any product, manufacturer, distributor, service or service provider mentioned or any opinion expressed in answers or comments. Please read carefully the Google Answers Terms of Service.

If you feel that you have found inappropriate content, please let us know by emailing us at answers-support@google.com with the question ID listed above. Thank you.
Search Google Answers for
Google Answers  


Google Home - Answers FAQ - Terms of Service - Privacy Policy