Thanks for asking!
Let me address these questions in almost the opposite order they're
originally stated. I'll discuss software first, then the
qualifications needed to create the software, third, the hardware,
because the software should determine the hardware.
Your description of the task is so complete, it's very clear that
you're seeking an Intelligence Agent, or Internet Bot. Your Agent
would be programmed to search for web pages meeting certain criteria.
That criteria can very likely be more sophisticated than a simple set
of keywords. The Agent would apply an algorithm (a set of programming
steps) to decide whether to include or exclude any given page, and be
able to roughly rank its relevance. It's very likely that you'd want
the Agent to also take a snapshot of pages located, and place them in
a cache (storage area) in case the content changes periodically. More
specific features of the Agent should be determined by the Agent
builder/programmer in accordance with the exact specifications the two
of you decide upon.
If you'd like a general introduction to what Agents are, and how they
function, I'd recommend this tutorial by Michael Weiss. The tutorial
is most specific to the telecommunications industry, so bear in mind
that some of the conclusion statements will tend to reflect that
emphasis. Beyond that cautionary note, it's a relatively untechnical
document that will explain Agents, their uses, and help you gain a
basis for understanding some of the technicalities of the programming
you are contemplating.
A gentle introduction to agents and their applications, Michael Weiss
- MITEL Corp.
I'll outline the three typical methods for obtaining a customized
1) The kit-built Agent is an Agent constructed using an Agent
Development Tool. Benefits of this method include the lowest price,
and a lesser degree of technical expertise required by the programmer
who configures it for you. On the downside, these Agents are likely to
need more extensive modification in order to perform as you would
like. A kit-based solution would likely require a greater degree of
human results filtering. Kit pricing starts at $95, and several of the
software packages state they can be successfully configured by a
programmer familiar with web programming and script installation.
Typical salary in your area for a web programmer is approximately
$45,000 per year. Contract programming at this level commands an
hourly rate between $85 - $150 per hour.
I'm including a link to a directory of Agent Kits. Even if you reject
this option, it would be a good idea to give the list of kits a quick
once-over to familiarize yourself with some of the names. Your
programming candidate or consultant should be able to discuss the
features and abilities of a variety of the kits available.
2) Agent programming consultants build custom Agents on contract to
your specifications. As independent developers, they already have or
have access to the tools, resources, and personnel needed for Agent
programming. They work with you to define the project and deliver a
turn-key solution for your use. As an appendix, I've included a
listing of several firms who are active in this field, along with
contact information. In selecting a consultant, you should ask for
demonstration of Agents previously developed by the firm. Experienced,
professional level firms typically charge a fixed price based upon an
hourly rate in the range of $125 - $250 per hour.
3) An In-house programmer with specialization in artificial
intelligence and Agent programming can work with you develop a
completely customized software application. Here's a short list of
desirable qualifications. It can also be used as a cross-check if you
decide to use a consultant. You might also consider using the services
of an IT employment agency as a middleman to pre-screen qualified
candidates. Such an agency can also help you decide on an appropriate
salary level. In Oklahoma, average salary range for Software
Developers and Programming Engineers is $58,000 - $78,000 annually.
Software Engineer - Agent Programming
Programming languages: C++, Java, Perl, object and markup languages,
Internet Protocols: HTTP, FTP, JDBC and TCP/IP
Education at the post-baccalaureate level (or equivalent research
experience) in Computer Science
2-5 years work experience in Agent programming or a related field,
such as search engine programming, or artificial intelligence.
When having the choice, I've always believed in selecting hardware
that is ideally suited to run the software you've chosen. Therefore, I
can't give you specifics, but I feel safe in stating that you'll
require a web server. A web server is a most often a PC, with
mid-to-high level features. You'll want the guidance of your Agent
specialist in selecting a system that has the required degree of power
and speed for the specific application. The price will likely be in
the range of $2,500 - $5,000 depending upon the required features. A
web server should have built-in backup capability, either using a zip
drive, or employing redundant hard drives, such as an external
Firewire drive. Software, including the operating system, will be an
additional expense. I'm partial to Apache web servers, running a
stable version of Linux. This is the most typical web server
configuration, assuring that you can readily locate a well-qualified
server administrator. You'll also require software in the form of Perl
modules, Java, and other server applications. Your programmer or
consultant can best assist you with custom configuration.
Web servers are also available for lease by many web hosting
companies. In this type of arrangement, you have administrative
control over a web server located in a secure facility, typically on
or very near an Internet backbone (the major data thoroughfares of the
Web). Pricing starts at about $200/month.
Common sense factors for consideration in selecting a web server,
either for lease or purchase, may be found at:
A Server Administrator will be the final member of your Agent team.
This individual will be responsible for the day-to-day tasks of the
Agent, and server administration, including backups and server logs,
which will document the Agent's journeys. A server administrator would
have 2 or more years experience working with the hardware and server
operating system you've selected, along with experience and/or
certification in the operation and maintenance of the specific brand
of network connection hardware used. You'd probably want to bring this
individual on-board midway through the programming stage for
on-the-job training in the tasks necessary for ongoing Agent
operations and configuration.
I've used SalaryExpert.com as my reference for salaries quoted.
Salary Calculation Tool:
Appendix A - Agent Developers
Bot Technology - Bot and Intelligence Agent Designers
Post Office Box 220
Marco Island, Florida 34146-0220
Contact/Info Form: http://www.bottechnology.com/email/index.html
Artificial Life - Smart Bot Application Builders
Artificial Life USA, Inc.
919 3rd Avenue, Suite 2741, New York, NY 10022
NQL - Intelligent Software Solutions based on Network Query Language
400 E. 13th Street
Vancouver, WA 98660
866-423-6987 (toll-free) or
I hope this information is useful to your project planning. If you'd
like me to clarify any portion of this information, please feel free