Why do some seemingly viable search engines fail to find a unique
keyword in my web page? I use this to "test" different search engines
for their effectiveness in locating my page. Yahoo, Google and Lycos
all find this keyword within a few weeks following my page updates.
MSN and the built-in search button on my MS Internet Explorer cannot
find it, even after a year or so of exposure in the same domain.

Hi there,

Our search returned the following results:

Here is a useful description of how search engines work, and what
makes search engines different:
http://searchenginewatch.com/webmasters/work.html

This article describes how search engines decide to list results the
way that they do (a good resource for those trying to increase their
position in these lists):
http://searchenginewatch.com/webmasters/rank.html

You were concerned about why your website is not showing up in MSN and
search through MS Internet Explorer. Firstly, IE uses the MSN search
engine so there is no difference between how the two of them work.
Here is a short description of how MSN search works:

"Microsoft's MSN Search service is a LookSmart-powered directory of
web sites, with secondary results that come from Inktomi. Direct Hit
data is also made available."
http://searchenginewatch.com/links/major.html

If you're viewing the link contents of this response while going
along, you will have noticed that MSN returns results based on the
Directory search engine LookSmart. There is more information about how
to submit a successful submission to them here:
http://searchenginewatch.com/webmasters/directories.html
The important thing to note from the quote above is that MSN functions
on a DIRECTORY search method, rather than the CRAWLER method used for
all the other engines you mentioned (Yahoo,Google,Lycos). The first
link in this response has more information on how directory-based
search engines work.

You mentioned that MSN fails to recognize a unique keyword you have
put on your website while all the other engines picked it up. This
makes sense because since all the other sites are "crawlers", they are
periodically crawling the web and indexing web content. MSN, on the
other hand, does not do this; instead they base their search results
on submissions by web masters. More information on this is available
here:
http://searchenginewatch.com/webmasters/index.html


In summary:
1. MSN runs on the LookSmart directory service
2. In order to be succussfully listed on directory services, a
different procedure is required.
3. Directory services in general donot monitor the content of your
page, so the description submitted to the given company of your site
is very important in determining where your site shows up.

If you need any help with this content, feel free to post a
clarification.

Hope this helps!
answerguru-ga

The answer explains my observation. It is curious, however, that the
MSN/MSIE approach apparently does not truly "search" the web itself
for keywords or content.

It is worth noting that what superficially appear to be different
search engines from the front end may be using the same databases in
the back end.  Lycos, AOL and Yahoo (web pages), for example, all use
Google, so you shouldn't be surprised that they all find it.  Another
large database, which is used by other engines is Inktomi.

Being nitpicky here, but it is the DMOZ directory that Lycos, AOL,
Netscape, et al use (which is the same directory that Google uses, but
Google adds its own patented filters.)  They may have similar results
because they are drawing from the same directory.  Google, however,
determines site placement through an algorhythm based on links from
popular sites in the same category.