I have not been able to find any good ressources on the web regarding
the *detailed* functioning of search engines such as Google or
Inktomi.

I would like to have *detailed* information on the entire process of:

- the crawling of information on the web
- the indexing of the information collected by the crawler
- the query of information contained in the index

This information must go further and deeper than the usual FAQ stuff
which can be found all around the web.

---

Request for Question Clarification by aceresearcher-ga on 20 May 2003 14:57 PDT

levon,

Most Search Engines, especially Google, are quite secretive about
their inner workings -- including forbidding employees to speak of
their methods to non-employees. Google Answers Researchers are
individual contractors, are not actual Google Employees, and are not
privy to Google's inner secrets, nor to such information that is not
posted publicly on the Internet.

HOWEVER:
I would be able to provide you with links to some seriously in-depth
information on how the Google and Inktomi Search Engine algorithms
work. Would you consider this an acceptable Answer?

Also, when you say "the query of information contained in the index",
can you describe exactly what you mean? Every Google Search is a query
on Google's index.

Thanks,

aceresearcher

---

Clarification of Question by levon-ga on 21 May 2003 05:33 PDT

Hi Aceresearcher,

Thanks for your comment. I am aware that Google and other search
engines do not publicize their inner workings. Thus, I would be happy
with in-depth information on how search engines work in general. As
stated in my original question, I would like to know as much as
possible about:

- CRAWLING: How is information on the web collected by the spider? How
much of the HTML data on a page is collected by the spider? Are there
any algorithms or complex rules applied at this stage already?

- INDEXING: How is the spider-collected data indexed? What kind of
algorithms are used for this task?

- QUERY: How is the index queried? How is relevance calculated? What
kind of algorithms are used for this task?

Best regards,
Levon

How Google was created? 

http://www-diglib.stanford.edu/diglib/pub/projectdir/google.html

Articles on Search Engine technology 
http://www.searchenginewatch.com/resources/article.php/2156601

How search engine work?
http://www.searchenginewatch.com/webmasters/article.php/2168031

How do they rank their pages?
http://www.searchenginewatch.com/webmasters/article.php/2167961

Get membership and Information on the detailed workings of the major
search engines.
http://www.searchenginewatch.com/benefits/article.php?source=work

How search engine work?
http://www.searchengines.com/search_engines_101.html
More details: http://www.searchengines.com/editor1.html

A good article -- but it's all theoretical:
http://www-db.stanford.edu/%7Ebackrub/google.html