I have an idea for building a CI package. I am trying to determine if
the type of tool/package that I need is available. I have no idea what
the costs of building one of these tools would be, and I am aware that
the amount of money to develop one would vary drastically. Okay, don't
laugh but I would like for someone to try and guess what the costs of
building such a tool would be. Also what type of person/people would I
have to hire. How long would it take to prototype, and then build. My
chief goal is to build this for only our company's use, however if the
type of tool I am thinking of does not exist, I would be interested in
building a tool that I could sell/license in the open market. With
this thought in mind there would now be 2 costs, one to build for
internal use and one for sale in the open market. In terms of bringing
this tool to market, my main goal would be to have a guess of
developmental costs though additonal costs would be welcome. What if I
didn't want to market it to consumers/businesses? How would I go about
trying to sell the tool to another company and (I know this whole
question must sound silly, with all the unknowns)have there been many
sales of software package's that weren't 1st sold and used with some
type of history? Though I can't go into detail, the major areas that
this tool would focus on are knowing who your competition is, where
they market (globally), their prices, company info, etc. My guess is
that this tool would go throughout the web and bring back around five
million web pages for the 1st filter, then down to around ten or
fifteen thousand for the second filter. I think I would have to
employee 2 or 3 people that go through those pages and determine what
is and what is not relevant. In time I would like to add additonal
capabilities such as some type of clipping service or other resources
of information. This tool would continually be searching for new
contacts, and checking for changes in websites for pricing changes
etc. It would find the physical address of the business and catalog
it. It would track how often users came to my site through various
links and provide reports so that I could make marketing decisons that
matched those sites that brought the most traffic. It would
differentiate between customers that I had actually done business with
and those I hadn't. Eventually I would think that this tool if used
only within our company would be bringing back twenty million web
pages through the 1st filter. I am going to list this question twice
so that I can get a couple of people to dig deep on this question in
additon to any comments that anyone is willing to offer.

Thank You,
Joel

Hi Joel,

First off I wanted to thank you for being such a faithful GA customer
:)

Regarding your question, I'm going to attempt to answer it fully from
my own experience. I am a software engineer turned project manager by
trade, and much of what I do involved helping companies improve their
business processes. I also have several years of experience with
developing custom software applications.

Now on to your question:

Regarding the existence of a "CI Package" type of program, I have
spent quite a considerable amount of time searching, but have come up
empty. My viewpoint is that nothing is likely to come up because of
the target customer such a product attempts to reach. If you really
think about it, the information needs of most companies could be well
fulfilled with a manual web search or even by leafing through some
government agency documents (should the products/services they supply
fall under that umbrella). Going further, a company could buy a CD-ROM
containing a list of businesses sorted by category. Although many
companies could benefit from this, you wouldn't believe how many don't
check their competition out thoroughly. Obviously the costs of
alternatives are far more appealing that designing a software solution
that provides only a slight improvement over MUCH cheaper solutions.

I understand that this is just an idea at this point, and you are very
right when you say that there are a lot of unknowns. The biggest
unknown, in my opinion, is that you don't have anything fully defined
as what as what EXACTLY the program is meant to do, how it will do it,
etc. I would highly recommend that you hire some people who can assist
you in helping you find out what it is that you actually want to build
- this is usually done through some sort of Project Feasibility
Analysis. If the results of this analysis provide some evidence that
this project is worth pursuing, then the entire software package
should be fully documented to clarify the "WHAT" before taking a step
towards the "HOW" of the project. If you think about it, will you ever
really know how to do something without fully understanding what it is
you're trying to do?

As far as costs go, you can really become quite accurate after the
project has been fully documented. You will realize once you sit down
with someone and weed this whole thing out that there will be a lot
more involved than your original idea. In fact, its a known fact that
actually defining the project requirements usually requires 1/3 of the
entire project's resources (ie. time and money) or more.

Let me say a couple of things about internal software vs. marketable
software that you mentioned before giving you some numbers. They are
COMPLETELY different! While creating a tool specifically for one
company, you essentially need the approval of one group of people. If
you in turn try to turn around and sell that same product, you may
find a few other companies that need the exact same thing...but if you
come to expect this revenue from other companies from the beginning,
you'd be digging yourself a nice little financial grave! Case and
point, Microsoft Office. Why is this product being used by a mass
majority of home and business users across the world? Certainly not
because a program that was developed for one company was suddenly
marketed to the rest of the world! Microsoft spends millions and
millions of dollars researching the practices and needs of all sorts
of companies and people. At the end of the day, what it takes to sell
such a successful product is the ability to cater to EVERYONE. The
average MS Word user is said to only use 10% of the functionality
available in the program. Does that mean that Microsoft developed an
overly complex product? NO, becuase everyone's 10% extends over
different features...this is the problem you're facing when trying to
market an in-house product. You've only done the 10% that your
business needs, but have you though about the other 90% that other
companies would want? The requirement for this boils down to EXPENSIVE
MARKET RESEARCH!

You've scratched the surface of a smart option in my mind...defining
and building a product specifically for your own company (with no
explicit intent to market it elsewhere) and simply making it
"available for license" to those companies who would like to purchase
it. This keeps your development costs down and provides the option for
additional income. Just make sure that whatever person/firm that
develops it actually gives you the right to license it (sometimes they
keep that right for themselves).

Now, to get to the cost of all this...obviously this could be way off
since I don't have an EXACT understanding to estimate this accurately.
My completely wild guess for developing a system similar to the one
you've described for your own company only would be somewhere in the
neighbourhood of $50-75K....now making that same product marketable on
a global scale would run approximately 50 times that cost. As I said,
this is based on my own experience, and quite honestly I don't know
anyone who could nail down a price even within 50% variance without a
lucky guess for such a question :)

As far as hiring individuals to develop the project, I would suggest
sticking with the people that are going to help you define the project
itself (they are likely to have a better idea of what to do when it
comes down to actually developing and testing the product). To answer
this question specifically, you need to ask yourself (or consult your
project definition document) the platform and/or programming languages
that the development will occur in, where the program will be deployed
at the end, etc. It seems that you want a program that has
web-integrated functionality (for updates and so forth), so will need
to take that into account. Your project documentation should guide you
towards what types and how many people your project will require to be
completed in X days. Functionality restrictions often guide you
towards a specific skill set that is required. Obviously if you are
planning on marketing the product you will need to consider the
business side of things (eg. marketing, sales, R&D, etc.).


I know this may be a lot to swallow, so feel free to post a
clarification if you are having problems with anything in particular.

Hope this helped!

answerguru-ga

---

Request for Answer Clarification by joel1357-ga on 12 Jun 2002 23:22 PDT

webadept-ga I want to lay out exactly what I want to do but because
everything that I say here is open for everyone else to read I am
severly restricted from getting the exact answer I need. I wish there
was a way that I could just tell someone what it is that I want
without everyone else in the world being able to read it. You seem to
be on the right track from my perspective. My brothers and I have a
company with several hundred employees, many of which are very
experienced and knowledgable about computers. The problem is that of
all those employees not one of them has done something like I want to
do. I have sat down with a couple of guys from our MIS department and
basically they are telling me that the hardware/software will cost me
about $ 35k plus the cost of expertise to put it all together. There
thoughts were this person would command something in the realm of $
125-200 hour and that the prototype would take approx. 100-160 hours.
Then after that was completed the actual tool would be put together.
Additionally they said that I would need someone to run this and I
should bring that person on within a month of bringing in the
developer. Obviously I am not a technical person but my people have
told me that I will need 2 or 3 servers, and from here on out I have
no idea what they are talking about but..something like each one would
have about 35 gigs of ram and 500 gigs of hard drive...expandable up
to a terabyte. Fine i'm ready to go..I'm hearing your
suggestions..they sound reasonable..actually I'm kind of lost. I know
what I want, I just desperately need the right person to make it
happen. Before I left the office yesterday pretty frustrated one of
the computer guys said that I would just have to be patient because in
their estimation there are probably only a couple of hundred people in
the U.S. that could actually put together what I want.

Thanks,
Joel

---

Clarification of Answer by answerguru-ga on 13 Jun 2002 05:59 PDT

Hi Joel,

I feel your frustration in this matter....its evident that both
webadept-ga and myself are eager to help you out in this matter, but
the rules tie our hands from doing so.

I agree with webadept when he says that the technical knowledge is not
as scarce as you think...it just requires some knowledge of
web-integrated programming.

By the way, just wanted to let you know that clarifications are meant
to address the person who answered the question...was there anything
that you wanted to ask me about? If not it may be better to post a
comment when addressing someone who has just commented on your
question. Just a friendly notice :)

answerguru-ga

Hi Joel, 

Can't keep my mouth shut on this one, since its so close to home. I do
this type of work on a regular basis, and since I am refrained from
handing you my card and talking to you face to face, the next best
thing is to give you some of my experience and try to help you out.

What you are talking about is done with companies, but the changes in
"needed information" are drastic enough to keep a "mass market"
package out of the range of feasibility. Your company is going to have
a set of needs. Your goal is not really to download 500,000 web pages,
but to find the 100-500 pages that are relevant, and to check on a
regular basis if others show up on the internet.

I'm going to address your in-house package only here, for several
reasons. One, is to save you time and money.

Languages good for this type of thing are C, C++(these are used by
Google) and Perl. A well written C++ program is going to be much
faster than a Perl program could hope to be, but development time is
going to be much faster with Perl, (several months faster). Many of
the tasks needed are already handled by Perl in what are called Mods.
These Mods are functions that are done over and over by programmers,
and someone or a group of someone's have put together a package to
take care of the redundant need. To see a list of Mods currently for
Perl, go to this website.

http://search.cpan.org/Catalog/

Wander around in there and you'll see descriptions of several tasks
you are thinking of doing, already developed in various ways.


The OS system isn't really important, though a Unix or Linux would
probably be much faster and easier to maintain. This puts your
hardware costs at about 2000.00 to start (that's top end) and probably
5000-8000 down the road when the package gets developed.

You are looking at three steps to this development for your in-house
package.


The first step is really two steps or two parts. The first is to find
a guy who writes programs for the Internet in Perl, and can show he
knows how to write bots. This guys is going to be interested in
databases, sorting data and some AI work.

Next is to sit down by yourself first, and then with him, and come up
with a description of the end product. Make up a fictional report or
10, that shows what you want to see, and what the information is going
to mean to you.

Next you need to decide when you feel the information is going to
spoil, knowing that as soon as a report is printed, its out of date.
Things happen, company's change, websites change. So you need to
figure out when "you" feel the information needs to be refreshed. This
could be a week a month, a day, depending on what you feel you need. I
caution that the sooner it spoils the more cost is going to be
involved later. I would advise something around a month to start, but
your mileage may vary.

The next step is rather hard but will control costs. Tell the guy you
hired to start gathering that information, with the goal in mind of
having an exact layout of what a system that will do this for your
company will take to create. I would give him 3-4 months for this. I
would expect to see those reports your created in the last step to
start showing up in 2 weeks. He'll at first need to do them by hand,
adjusting searches and creating "tools" to take out the drudgery of
the task. In three months though he's going to have a very clear idea
of what it takes to do this for you.

Last step, for in-house, is to hire another programmer, and an
assistant for the guy your hired before. The new programmer will take
the direction from your current one and start building a real program.
What your current guy is probably going to have is a bunch of little
utils that he can run, but anyone else is going to scratch their heads
over. The new programmer is going to take those ideas and start the
development of a user interface, and a real backend.

The new assistant is going to start learning how to do the tasks with
the utils, and help test the new front end. This gives the current
programmer someone to teach and explain things too, keeping the idea
fresh and clean in his head, so he can assist the new programmer in
keeping on track.

In four months you should have a version 0.1 to work with in your
company. A user, given 3 days training should be able to print out the
reports for you. Your second programmer is probably not required now.
I would hire him on a project basis anyway. The assistant and your
original programmer are probably still needed for troubleshooting and
fixing bugs, but you should have at this time a working in-house model
of what you want.

As I said, your mileage may vary, but this model has worked for me for
several years, and continues to work well today.

Some things to expect. 


1) Your needs are going to change several times during this process.
When you discover that other types of information are available, you
are going to want to add them. Your current description tells me that
right off the bat.

2) At least 5 things are going to go wrong. Count on it, and work with
it, that's just life and business, and I'm sure you are filmier with
that rule or you wouldn't be where you are today.

I hope this helps you with your task and if we can be of any
assistance to you, don't hesitate to ask :-)

webadept-ga

Hello,

Just adding a few links of interest to the mix.

http://www.ktic.com/TOPIC7/14_TECH.HTM
http://www.bidigital.com/ci/Ratings
http://mtu.www.media.mit.edu/people/mtu/agents.html
http://researchindex.dlib.vt.edu/cs
http://www.wkap.nl/prod/j/1387-2532
http://lair3.slis.indiana.edu/~jm/content/lair/project/aim/journals_articles.htm

ba-bye
V

Hi Joel, 

First of all I want to say up front that I'm at an extream
disadvantage with the estimates that your MIS staff is giving you.
They have heard the direct description of what you want and I have
only my best guess. Obviously you trust your staff, or they wouldn't
be there, and they are probably very good at what they do. I, on the
other hand, don't even know what they do :-)

What prompted me to write back so soon was the comment that only a
couple hundred people in the U.S. could get this going for you. I
disagree there completely. What you have been able to describe so far
is pretty much a  common programming task in the Perl world. I would
say that any Perl programmer whose been at it for 5 or 6 years, and
has worked on the Internet during that time, has a good working
knowledge of how to gather and filter information.

Using tools like Java or Visual Basic... that could be a problem.
There are things you can do in Perl in 5 or 6 lines of code that would
take a Java programmer 50-100, and Visual Basic won't do them at all.
Most of those things deal with getting and parsing web-pages. So this
could be the discrepancy in the time lines and costing factors.

For instance, a while back a Temp Agency wanted to keep track of how
many ads various other companies were placing on Monster.com. What
they were for, and if they put in a salary value, what it was. They
also added on later that they wanted to know how many email addresses
the company used in placing these adds, and how often they changed.

That project is exactly 405 lines of code, written in one week to
final stage. It runs on a 500.00 computer and prints out a report once
a day and once a week. They don't even need to ask it to, it just does
it every morning before they come in.

They added two local newspaper's web want adds to the system for about
the same amount of code, and a few more weeks in time.

That's a very specific task, and much smaller than what you are
talking about, but its shows you that this kind of thing can be done
and very cheaply for the benefit it gave them.

Moving that same program up to cover a world-wide area, each major
city, a list of lets say 1000 companies to watch, all over the world,
boosts up the code, but not as much as you might think. We're still
getting and parsing, and searching. The tasks start breaking down to
similar sets. The computer goes up, but again, not as much as your MIS
is suggesting (for this anyways) We aren't talking 3-4 computers,
maybe two.

I guess what it boils down to is, if all you have is a hammer then
getting a screw into some drywall cleanly, is an all day job, and
really expensive, and only a few people in the US are going to know
how to do it. :-)

webadept-ga