since Google does not index everything on the web is there a search I
can do that may take hours or days but would return more restuls? 
Mostly used on more obscure topics which may have notes on unpopular
or unknown websites.

---

Request for Question Clarification by justaskscott-ga on 29 Sep 2005 18:50 PDT

Would this product be sufficient?

"Copernic Agent"
Copernic
http://www.copernic.com/en/products/agent/index.html

If so, I would be happy to post it as an answer.

I don't think you'll find a site that indexes everything. One can find
information on Google or Yahoo even on the most obscure topics
(believe me on that, as someone who deals resularly with a very
obscure topic), and perhaps you have to enhance your search abilities,
ask for sites discussing this topic on Google Answers, or realise that
there is some information that is still unavailable online: for
example, some historical documents, trascripts, copyright protected
material, nwespaper archives, etc.

I was thinking of running my on bot to crawl the web.  Are there bot's
out there you can download?  For example I want to search for specific
text string.  I have done this on Google and I know the string exists
yet Google comes up blank because it does not index this string of the
site.

I would recommend against a bot (from experiance).

I once coded a bot myself, but took about 2 days non-stop to index 500
000 pages, so unless your running a fast server with very high
bandwidth, bots are probably not your best bet.

If you do decide to proceed with a bot, you will probably need to know
some programming (C/C++, Java, I once found source code for C#), get
the source code and build it yourself to have real control over what
the bot searches for (a bit of background in regexes is also good).

Vyasa