I am creating a Web site containing 500,000 static HTML pages on
Windows 2000 Server using ASP.NET.

I want to include a search facility that will return the most relevant
results to the top of the list. The ranking algorithm will need to
examine metadata and hypertext and content (and, possibly links, and
anything else it can think of) to make a useful ranking decision.

Because there are so many pages, I need a sophisticated ranking
algorithm.

My question is: what is the optimum search tool for this environment?
I am having trouble understanding the exact capabilities of various
software products.

My budget is in the $10,000 to $25,000 range.

Please do not simply respond with URLs of product descriptions. I am
looking for some comparative reviews of ranking algorithms, or a
detailed discussion of the types of algorithms and the products that
implement them and their applicability to my site, or something very
specific that allows an informed product choice.

---

Clarification of Question by sa-ga on 13 May 2002 21:41 PDT

Thanks, Interceptor.

The Google Search Appliance may well be ideal for my needs, but the
information givcen about its relevance ranking algorithms at
://www.google.com/appliance/faq.html#pagerank is not sufficient
for me to make a decision.

And, perhaps because I am in New Zealand, Google does not respond to
my request for a demo model.

Hence my seeking some sort of comparative review or more in depth
discussion.

Hi.

I believe I found a few sites that can be of help.

I'm not an expert in website search solutions, but I believe that
you'll hardly find more information about such solutions that at
SearchTools.com (http://www.searchtools.com). They are professionals
and they provide free extensive info on the matter. They also provide
consultation (for a fee).

Considering your budget I believe you should consider asking for their
professional advice. You might save more money in the end. This kind
of professionals are pretty rare as there's little insider
information.

About the Google service, I believe that the "Search Appliance" is not
for you. It works on Linux and the two products that they provide are:

1. 20000$ for a unit that can handle up to 150000 pages.
2. 250000$ for a unit that can handle millions of pages.

I think however that the other search services that Google provides
might be good for you.

Other resources:

Google Directory - Computers > Software > Internet > Servers > Search
http://directory.google.com/Top/Computers/Software/Internet/Servers/Search/

Search Engine Watch - Search Engine Software For Your Web Site
http://searchenginewatch.com/resources/software.html

Hotwired.Lycos.com - Webmonkey - Adding Search to Your Site by Avi
Rappoport (president at SearchTools.com)
http://hotwired.lycos.com/webmonkey/00/09/index2a.html?tw=e-business

Webmasterbase.com - Secrets to Site Search Success - by Paul Bruemmer
http://www.webmasterbase.com/article/608/16

Google search terms:

search tools for websites intranets
://www.google.com/search?num=30&hl=en&safe=off&q=search+tools+for+websites+intranets

"search engine" software
://www.google.com/search?q=%22search+engine%22+software&btnG=Google+Search&num=30&hl=en&safe=off

about website "search solutions"
://www.google.com/search?num=30&hl=en&safe=off&q=about+website+%22search+solutions%22

"search solutions"
://www.google.com/search?num=30&hl=en&safe=off&q=%22search+solutions%22


That's it. I hope this information helps. Feel free to ask for a
clarification. I want you to be happy with the Google Answers service.

Regards

ldcdc-ga

---

Request for Answer Clarification by sa-ga on 14 May 2002 14:35 PDT

ldcdc, this doesn't really answer my question. Specifically, I need either:

1. some comparative reviews of ranking algorithms, or
2. a detailed discussion of the types of algorithms and the products that
implement them and their applicability to my site, or
3. something very specific that allows an informed product choice. 

I am familiar with the sites noted and have already explorered them without success.

---

Clarification of Answer by ldcdc-ga on 14 May 2002 19:30 PDT

Hi.

Here are some links of use (I think):

NetworkComputing.com - Search Engines: The Hunt Is On - By Avi
Rappoport
http://www.nwc.com/1120/1120f1.html

Searchenginewatch.com - Membership Benefits (why you should decide to
pay)
http://searchenginewatch.com/about/subscribe.html?source=rank#benefits

PageRank Explained - by Chris Ridings -
www.Search.engine-submission.co.uk
http://www.goodlookingcooking.co.uk/PageRank.pdf

Special Search Engines - Search engines for your site (it's a
subtopic)
http://www.leidenuniv.nl/ub/biv/specials.htm

I must tell you that insider information about how search engines (of
any kind) work is virtually impossible to find, as it is always a
highly guarded company secret. And any information not released by
companies themselves would be either false or illegal.

I have a website and I know how hard it is to find information about
how a specific search engine calculates the ranking. And this kind of
info is much more demanded by the market that the one that you need.
And where's a big need there should be many services. But that not
true for SE information.

Information from the users is what could be found, but only "big"
websites decide to go for a "professional" product. And "big" sites
are rare. And the information is pretty valuable. And big companies
don't release much "real" info anyway.

I stayed up all night (I live in Europe) searching for insider
information but I found what you can see: very-very little.

Now, if I were you, I think I would go for a free service from Google
(at least for a while to see how searching influences website's value)
or I would ask for professional help right from the start.

From my searches I can tell you that I've not encountered another site
like SearchTools.com. And many sites recommend them. That's why I
think they are the ones to ask the type of questions that you have.
They have the experience and thus the information that you need.

I'm sorry, but that's all I found.

Regards,

ldcdc-ga

---

Clarification of Answer by ldcdc-ga on 16 May 2002 07:49 PDT

Hi.

Here are two articles I though you might want to read.

They are from Searchenginewatch.com.

http://searchenginewatch.com/searchday/02/sd0515-stools1.html
http://searchenginewatch.com/searchday/02/sd0516-stools2.html

And here's a quote (from one of the articles) that proves some of the
things I said earlier:

"Avi Rappoport, Principal Consultant for Search Tools Consulting, is
the leading authority on site, Intranet and topical portal search
engines."

It is said by himself, but if the guys at Searchenginewatch.com
wouldn't agree, they wouldn't let him say it in their newsletter.

Good luck!

And again I'm sorry for the lack of "decisive" info.

ldcdc-ga

Thanks, ldcdc. I followed the trail and eventually ended up at
http://www.dtsearch.com/, which meets my needs pretty well. And I'm
sorry you had to stay up all night!

Hello sa-ga, 
 
Our search returned the following results: 
 
Search your network with Google!
(://www.google.com/services/index.html)

Here is a quote off the web page:

"The Google Search Appliance puts the power of Google's renowned
search engine behind your network. A fully integrated hardware and
software design enables users to plug in and start searching in under
an hour. And once your crawl is up and running, the Google Search
Appliance finds more of the documents your users are looking for,
returning results with the same high relevance users find on
Google.com."

The cost is within your budget and you know that it works :-)

The results were obtained from this Google search: Right on the Google
website.

I hope you found the answer you liked here...but if you require help
on any of the information please feel free to post a clarification :-)
 
Thank you for your interest in Google Answers.

Regards, 
 
Interceptor-ga

Hi, 

Try the following links for more detailed descriptions of how Google
works:

The Anatomy of a Large-Scale Hypertextual Web Search Engine
Sergey Brin and Lawrence Page 
http://www7.scu.edu.au/programme/fullpapers/1921/com1921.htm

The PageRank Citation Ranking: Bringing Order to the Web (1998)
http://citeseer.nj.nec.com/page98pagerank.html

The following is also useful
Finding Authorities and Hubs From Link Structures on the World Wide
Web
http://citeseer.nj.nec.com/463036.html

Generally the site at citeseer.nj.nec.com is an amazing source of
technical papers and has a huge amount of material on web searching -
the papers are all interlinked try starting with the two above and
following the links to related papers.

     Cheers,
                  Mike

Thanks, Mike.

That is a very interesting site, and helps me to get a clearer view of
how PageRank works.