I need to know what type of hardware/software (with approx. costs)
would be best to go through the entire internet looking for companies
that are in the same type of business as myself so I can have new
people to sell to. Through my research I have already found 2,200
companies and about 600,000 web pages. I believe in the end there will
be about 17,000 companies and about 5,000,000 webpages. I want to put
together a program that will have about 100-200 keywords (which will
change from time to time) and phrases to go through all the web pages
understanding that a certain percentage of these "keywords" must be
present in order to make it through a filter and deliver a rich
data-source of potential customers. If I am on target, I would then
have 2 or 3 people who go through what I believe would initially be
about 15,000 web pages and maybe a few hundred per day thereafter that
made it through the filter to determine which one's are in fact
relevant sites. This program would run every day and the customers I
would be going after are in every country on earth that the U.S. is
allowed to do business in. In the end I would expect that all I am
interested in would be the best 600/700 customers worldwide.
Additionally I need to know the qualifications of the person that
would put this all together for me. My thoughts are to have somebody
that puts this all together, and then have someone that has a lot of
talent but not necessarily everything the person that put it together
had and the second person "hopefully less expensive" would run it on an
ongoing basis. At first blush I would think the 2nd person would need
to watch the 1st person so they could understand the basics of how it
was all put together.

Thank You,
Joel

Thanks for asking!

Let me address these questions in almost the opposite order they're
originally stated. I'll discuss software first, then the
qualifications needed to create the software, third, the hardware,
because the software should determine the hardware.

Software

Your description of the task is so complete, it's very clear that
you're seeking an Intelligence Agent, or Internet Bot. Your Agent
would be programmed to search for web pages meeting certain criteria.
That criteria can very likely be more sophisticated than a simple set
of keywords. The Agent would apply an algorithm (a set of programming
steps) to decide whether to include or exclude any given page, and be
able to roughly rank its relevance. It's very likely that you'd want
the Agent to also take a snapshot of pages located, and place them in
a cache (storage area) in case the content changes periodically. More
specific features of the Agent should be determined by the Agent
builder/programmer in accordance with the exact specifications the two
of you decide upon.

If you'd like a general introduction to what Agents are, and how they
function, I'd recommend this tutorial by Michael Weiss. The tutorial
is most specific to the telecommunications industry, so bear in mind
that some of the conclusion statements will tend to reflect that
emphasis. Beyond that cautionary note, it's a relatively untechnical
document that will explain Agents, their uses, and help you gain a
basis for understanding some of the technicalities of the programming
you are contemplating.

A gentle introduction to agents and their applications, Michael Weiss
- MITEL Corp.
http://www.magma.ca/~mrw/agents/


Software Development

I'll outline the three typical methods for obtaining a customized
Intelligence Agent.

1) The kit-built Agent is an Agent constructed using an Agent
Development Tool. Benefits of this method include the lowest price,
and a lesser degree of technical expertise required by the programmer
who configures it for you. On the downside, these Agents are likely to
need more extensive modification in order to perform as you would
like. A kit-based solution would likely require a greater degree of
human results filtering. Kit pricing starts at $95, and several of the
software packages state they can be successfully configured by a
programmer familiar with web programming and script installation.
Typical salary in your area for a web programmer is approximately
$45,000 per year. Contract programming at this level commands an
hourly rate between $85 - $150 per hour.

I'm including a link to a directory of Agent Kits. Even if you reject
this option, it would be a good idea to give the list of kits a quick
once-over to familiarize yourself with some of the names. Your
programming candidate or consultant should be able to discuss the
features and abilities of a variety of the kits available.

http://www.agentland.com/Resources/8How_to_develop_an_agent/Development_tools/

2) Agent programming consultants build custom Agents on contract to
your specifications. As independent developers, they already have or
have access to the tools, resources, and personnel needed for Agent
programming. They work with you to define the project and deliver a
turn-key solution for your use. As an appendix, I've included a
listing of several firms who are active in this field, along with
contact information. In selecting a consultant, you should ask for
demonstration of Agents previously developed by the firm. Experienced,
professional level firms typically charge a fixed price based upon an
hourly rate in the range of $125 - $250 per hour.

3) An In-house programmer with specialization in artificial
intelligence and Agent programming can work with you develop a
completely customized software application. Here's a short list of
desirable qualifications. It can also be used as a cross-check if you
decide to use a consultant. You might also consider using the services
of an IT employment agency as a middleman to pre-screen qualified
candidates. Such an agency can also help you decide on an appropriate
salary level. In Oklahoma, average salary range for Software
Developers and Programming Engineers is $58,000 - $78,000 annually.

Software Engineer - Agent Programming

Programming languages: C++, Java, Perl, object and markup languages,
among others.
Internet Protocols: HTTP, FTP, JDBC and TCP/IP
Education at the post-baccalaureate level (or equivalent research
experience) in Computer Science
2-5 years work experience in Agent programming or a related field,
such as search engine programming, or artificial intelligence.


Hardware

When having the choice, I've always believed in selecting hardware
that is ideally suited to run the software you've chosen. Therefore, I
can't give you specifics, but I feel safe in stating that you'll
require a web server. A web server is a most often a PC, with
mid-to-high level features. You'll want the guidance of your Agent
specialist in selecting a system that has the required degree of power
and speed for the specific application. The price will likely be in
the range of $2,500 - $5,000 depending upon the required features. A
web server should have built-in backup capability, either using a zip
drive, or employing redundant hard drives, such as an external
Firewire drive. Software, including the operating system, will be an
additional expense. I'm partial to Apache web servers, running a
stable version of Linux. This is the most typical web server
configuration, assuring that  you can readily locate a well-qualified
server administrator. You'll also require software in the form of Perl
modules, Java, and other server applications. Your programmer or
consultant can best assist you with custom configuration.

Web servers are also available for lease by many web hosting
companies. In this type of arrangement, you have administrative
control over a web server located in a secure facility, typically on
or very near an Internet backbone (the major data thoroughfares of the
Web). Pricing starts at about $200/month.

Common sense factors for consideration in selecting a web server,
either for lease or purchase, may be found at:

http://dcb.sun.com/practices/howtos/webserver_part1.jsp


Server Administrator

A Server Administrator will be the final member of your Agent team.
This individual will be responsible for the day-to-day tasks of the
Agent, and server administration, including backups and server logs,
which will document the Agent's journeys. A server administrator would
have 2 or more years experience working with the hardware and server
operating system you've selected, along with experience and/or
certification in the operation and maintenance of the specific brand
of network connection hardware used. You'd probably want to bring this
individual on-board midway through the programming stage for
on-the-job training in the tasks necessary for ongoing Agent
operations and configuration.

--------------------

I've used SalaryExpert.com as my reference for salaries quoted.

Salary Calculation Tool:
http://www.salaryexpert.com/index.cfm?FuseAction=FreeSalaryTools.Dsp_FreeSalaryTools

--------------------

Appendix A - Agent Developers

Bot Technology - Bot and Intelligence Agent Designers
Post Office Box 220
Marco Island, Florida 34146-0220
941-434-3850
E-mail team@bottechnology.com
Website: http://www.bottechnology.com/
Contact/Info Form: http://www.bottechnology.com/email/index.html

Artificial Life - Smart Bot Application Builders
Artificial Life USA, Inc.
919 3rd Avenue, Suite 2741, New York, NY 10022
212-829-4386
E-mail: info@artificial-life.com
Website: http://www.artificial-life.com/

NQL - Intelligent Software Solutions based on Network Query Language
E-Botz, Inc.
400 E. 13th Street
Vancouver, WA 98660 
866-423-6987 (toll-free) or
360-693-5000
E-mail: sales@nqli.com
Website: http://www.nqli.com/

--------------------

Search strategies:
Intelligence Agents
bot developers
"agent developers"

I hope this information is useful to your project planning. If you'd
like me to clarify any portion of this information, please feel free
to ask.

~larre-ga

---

Clarification of Answer by larre-ga on 10 Jun 2002 05:30 PDT

I managed to leave out the salary figures for a Network Administrator.
The range I found was $44,000 to $60,000 annually, however, I wasn't
able to locate a regional breakdown. Judging by the other salary
ranges, I would guess that your area would tend to be towards the
lower end of the range.

I consider the software developer salary range to be a tad low, mostly
due to averaging. The skillset and educational requirements you're
seeking from this particular field of  software development is more
specialized than average, and therefore would probably command
additional compensation.

Originally, I did not address the issue of 'filtering' through the
data gathered by the Agent. It's likely the programming would include
an intelligent filtering of the results. Any Agent could probably be
configured to process the same data repeatedly, applying a stricter
standard with each pass. Depending upon the solution you select, this
progressive filtering could be a part of the programming algorithm, or
a new configuration of the Agent program, applied manually. Many
custom intelligence Agents have the ability to learn from their
mistakes and apply an ever-increasing degree of refinement to future
results.

Also, I'm working on the assumption that the humans filtering the
results produced by the Agent do not require specialized computer
training. I would think this task could even be handled by temporary
employees with clerical or administrative skills. In the typical 
modern office environment, it's taken for granted that most employees
are capable of simple computer-related tasks, if given specific
directions.

If I've left any additional points uncovered, I'll be happy to
clarify.

~larre-ga