I've a disagreeable client.  He's the CIO of a public company.  I'm
his software vendor.

He gets very upset when his users find bugs in the software I provide
him.  At a meeting last week, I told him that I take every bug
seriously, and will work to fix all bugs that are discovered.  I also
told him that I cannot guarantee bug-free code, as such a thing is
impossible.

He became livid at my answer.  Bug-free systems are indeed possible,
he claims.  In fact, the Software Engineering Institute at Carnegie
Mellon in Pittsburgh has a program called the Capability Maturity
Model that ensures against all errors in any code produced by the
highest levels of the process.  In his words, "zero-defect
environments exist where quality is critical: they're the environment
where the bomb finds the right window."

I know this to be a misconception on his part, and told him so.  He
was insistent, claming that a local (and very large) financial
institution has adopted the SEI's practices, and as a result produces
systems with no bugs.

His background is as CIO, and not as a software developer.  I can
certainly understand how, given a lack of developpment experience, a
CIO might think that bug-free code is possible.  But this guy got his
job because he was famous as a CIO.  He was even a professor within a
nationally-ranked business school (top 5) of a large university.  His
job: teaching would-be CIOs.  I would hope he would have some insight
on software products and processes.

On to my question:

I can find many references to authorities, from Steven McConnell to
Kurt Goedel, that bug-free code is not possible.  Or, more accurately,
that we can never be assured that our code is free of bugs.

What I can't find is very convincing references to the possibility of
bug-free code.  Some companies profess it, some documents say we
should strive for bug-free code (a dangerous sentiment in itself).
But I can't find anything that doesn't have a cheesy web site or links
to third-rate magazines that claims zero defects as a true possibility.

What I'm looking for is some high authority on maintaining bug-free
code.  Someone who truly believes that a particular, real-life useful
system can be free of all bugs, and that zero defects can be
guaranteed.  Someone that others take very seriously: perhaps he's
written up in many prominent magazines, or holds a position at a
university or at a very large US company.

Your job as a researcher of this question: take my personal conviction
zero defects cannot be guaranteed.  Show me how wrong I am.

---

Request for Question Clarification by hammer-ga on 15 Nov 2002 09:25 PST

Hello Doug,

From reading the comments, it must be clear by now that a number of
very good researchers have attempted to play Devil's Advocate for you,
but are not able to find support for the claim that writing
zero-defect software is viable within the time, budget and
requirements constraints of real-world software development.

I'm a developer myself. I've fought this fight, and I sympathize.

Given that (as I'm sure you suspected) the information you request
does not appear to exist in a way that really meets the requirements
of your question, is there something that we *can* provide which will
help you manage this problem and which you will accept as an answer?
- Hammer

Doug,

Do you think that your CIO would accept the word of Steve Cross, the
Director and CEO of Carnegie Mellon's Software Engineering Institute
(SEI) ( http://www.sei.cmu.edu ), which created the Capability
Maturity Model?

"Stephen E. Cross is the Director and Chief Executive Officer (CEO) of
the Software Engineering Institute (SEI), a United States Department
of Defense sponsored Federally Funded research and Development Center
(FFRDC) situated as a college level unit at Carnegie Mellon
University. He was appointed to this position on November 1, 1996. He
joined the university in 1994 as a member of the research faculty. He
also holds an appointment in Carnegie Mellon's School of Computer
Science as a Principal Research Scientist in the Robotics Institute.
Dr. Cross is the past chairman of the Defense Advanced Research
Projects Agency (DARPA) Information Science and Technology panel
(ISAT) and continues to serve on the executive committee. He is also a
member of the Naval Post Graduate School's (NPS) Software Engineering
Advisory Board. He has published more than 50 papers on technology and
the applications of advanced information processing technology. He
received his Ph.D. from the University of Illinois, his M.S.E.E. from
the Air Force Institute of Technology, and his B.S.E.E. from the
University of Cincinnati. He is a graduate of the U.S. Air Force
(USAF) Test Pilot School (Flight Test Engineer Program), the Air War
College, and the National Defense University. He retired from the USAF
in 1994. Dr. Cross was profiled in a January 4, 2002 article in the
on-line version of Business Week entitled "Gurus of Tomorrow's
Technology."
http://www.sei.cmu.edu/staff/sc/

---------------------------------------------

From : aceresearcher
To : sc@sei.cmu.edu
Subject : Capability Maturity Model
Date : Fri, 15 Nov 2002 16:17:51 -0500

Mr. Cross,
 
I am in need of written verification from someone who would be
considered an authority on the Capability Maturity Model (could be a
co-author, co-designer, CMM support person, etc) that the Model does
NOT in fact say that it is possible to create complex software that is
completely bug-free.
 
(This is to respond to someone who is insisting that the Capability
Maturity Model DOES say that it is possible to create complex software
that is completely bug-free.)
 
If you would be willing to dash of 1 or 2 sentences to that effect, or
to pass this e-mail on to someone who could do the dashing and reply
to me, I would be MOST appreciative.
 
THANKS VERY MUCH!

---------------------------------------------
From : Steve Cross <sc@sei.cmu.edu>
To : aceresearcher
CC : Bill Peterson <wcp@sei.cmu.edu>
Subject : Re: Capability Maturity Model
Date : Sun, 17 Nov 2002 05:04:41 -0500

"You could refer the person who asked you the question to our web site
(www.sei.cmu.edu) for appropriate statements.  The rest of the message
is my personal opinion.

A model does not guarantee anything. It can only provide guidance as
to what is possible. The CMMI, as the SW-CMM before it, provides an
approach for an organization to define its engineering and engineering
management work processes and to improve them over time.  What needs
to be improved should be guided by the business goals of the
organization (this is more explicit via than the CMMI which replaces
the SW-CMM). If that business goal is defect free software, the CMMI
will help the organization move to that goal, but will not guarantee
it will be achieved.

We have introduced the PSP and TSP to help the individual engineering
and teams of engineers to produce near defect free software. PSP and
TSP provide individual and team experience in using CMM-based
processes in the design and development of software.  PSP and TSP are
also described on our web site.  Having said all this, it is our
strongest position that software development organizations are better
off adopting and using the CMMI than not and that individual engineers
and teams should be PSP trained and use TSP in the individual work.

Thanks for contacting me.

Steve

P.S.  You and the person contacting you may want to engage with your
local Software Process Improvement Network (SPIN).  The list of SPINs
are also listed on our web site.  Many of these groups have monthly
meetings and serve as defacto CMM user groups.

P.S.  cc'd is Bill Peterson, Program Director of the SEI process
group. If you have any further questions, please contact him."

---------------------------------------------

Note Mr. Cross' remarks:

"A model does not guarantee anything."

"If that business goal is defect free software, the CMMI will help the
organization move to that goal, but will not guarantee it will be
achieved."

"We have introduced the PSP and TSP to help the individual engineering
and teams of engineers to produce ***near defect free*** software."

---------------------------------------------

"CMMI Adoption", from SEI's website:
http://www.sei.cmu.edu/cmmi/adoption/adoption.html

Software Engineering Institute (SEI)
Carnegie Mellon University
Pittsburgh, PA 15213-3890
(412) 268-5800
http://www.sei.cmu.edu/cgi-bin/contactus.cgi/sei-home.html?owner=webmaster


Search Strategy

"Carnegie Mellon" "Software Engineering Institute" "Capability
Maturity Model"
://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=%22Carnegie+Mellon%22+%22Software+Engineering+Institute%22+%22Capability+Maturity+Model%22&btnG=Google+Search


Before Rating my Answer, if you have questions or need additional
information, please post a Request for Clarification, and I will be
glad to see what I can do for you.
 
I hope this Answer provides you with exactly the information you
needed!
 
Good Luck!!!
 
aceresearcher

I think this is about as good as it's going to get.

Of course, this doesn't really provide me with the information I'm
looking for, which is how this CIO became convinced that bug free
software is truly possible.  It's certainly unfair to insist upon such
a clarification!

By the way, this CIO is also on the faculty of Carnegie Mellon.

Perplexion continues.

I have to say, that I'm far more taken with your tale of woe, than I
am with the prospect of finding what you wish. In my mind, it would
seem to be oxymoronic to respect and look up to someone who held the
belief that bug free software was possible, after they had undergone a
thorough grounding in the topic. Good luck in convincing your client
of Goedel's validity!

The strongest case in support of defect-free software is the NASA
software that controls the launching of the space shuttle. In the last
three iterations of the software (each of which is roughly 420,000
lines of code) there has been exactly one error.

They attribute this success to their rigid development process. This
process can be summerized into four principles.

1. The product is only as good as the plan for the product. 
2. The best teamwork is a healthy rivalry. 
3. The database is the software base. 
4. Don't just fix the mistakes -- fix whatever permitted the mistake
in the first place.

This shuttle software is perhaps the best example of pure defect-free
software, however, it is achieved at a cost most commercial entities
(and customers) are unwilling to pay.

Here is a detailed article on the NASA software development process:
http://www.fastcompany.com/online/06/writestuff.html


For further reference, here is a listing of links on the topic of
software quality control:
http://www.laatuk.com/books/quality_sources.html

Hope this has helped.

I don't think you need to tell him that bug-free code is impossible,
just tell him that it tends to be very expensive.  I don't know what
business your customer is in, but unless the bugs in your code involve
a relatively good chance of loss of life, most likely you don't want
to use the methods NASA does.

You're not going to find a mathematical proof that bug-free code is
impossible, because it's not--you just test every possible way the
code can be used until it all works perfectly.  The problem is that
any UI code tends to involve far too many possibilities for this to be
done in practice.

Doug --

In order to provide a defect-free code base, the environment needs be
tightly defined.  In the world of open-architecture systems, this is
an impossibility, as the environment changes with every new piece of
code, every new scenario.

I had the honor during the mid-1990s to work with one of the best
quality organizations in the world, Motorola.  Their Six Sigma efforts
are well-known for the dramatic reductions in defects.  Yet they
despaired of every attaining Six Sigma results in software for the
reasons mentioned above.

For Motorola, exhaustive testing of known environments alone would
have taken generations for PC-compatible software, as there are in
excess of 10,000 programs that are Windows-compatible (not to mention
the variations of Microsoft operating systems).  To prove that it's
not possible is impossible, just as it was impossible for me to prove
to a Google Answers customer that this tale has no basis in fact:
https://answers.google.com/answers/main?cmd=threadview&id=88400

If you can get agreement on the scale of a test plan to rule OUT bugs,
perhaps you can get your customer to realize that it will consume the
manpower of the state for a decade.

Best regards,

Omnivorous-GA

The issue of programmers producing "error free" code is somewhat
analogous to a manufacturer having zero-defects coming off the
production line.

It's a bad analogy, but an analogy nonetheless.

A production line is a repeatable process.  The manufacturer knows
that when a new process is brought online, the number of unforeseen
complications will be large in the beginning, and decline over time if
suitable feedback/quality assurance is done.  The "theory" of zero
defects is closely related to the ideas of Six Sigma quality control. 
Such an approach only applies to repeatable processes that are
amenable to improvement.

Strangely enough there are many processes in which for practical
purposes, allowing for a fraction of the production to be defective
makes good economic sense.  The manufacturer may or may not sell
product knowingly with untested defects, depending on the marketplace
for that product and its competition.

This is _not_ the practice of software developers writing custom
software for a single client.  Basically the software vendor in this
circumstance is like a manufacturer constantly introducing new
solutions and processes, with little or no "repeatability" in the
specific tasks.

The "learning from experience" that software developers do is quite a
few levels of abstraction removed from what line managers, shop
stewards, and machinists do to improve the defect rate in an operating
process.  At least that is the case with developers who keep pace with
the state of the art.

Since your discussions with the client touched on SEI and the CMM
material, it might be instructive to point out for your own
edification the relation of the upper levels of the CMM rankings and
repeatable processes.  You can then ask whether this has any bearing
on the sort of work you do for your client.

My feeling is that your client is knowingly jerking your chain. 
Corporate America, unfortunately, is top heavy with folks whose appeal
to decision makers is not so much that they know how to get things
done as that they know how to avoid embarassing their sponsors.  We
pat ourselves on the back for being such an open and innovative
society, but the reality is that big companies are invariably
hide-bound.  They are unable to provide rewards to risk takers over
the long run, so internal IT departments crawl to a halt with
requirements planning, specifications reviews, prioritization
meetings, and a host of other mechanisms to avoid taking decisions.

That's why outside software suppliers like you have work.  And that's
not such a bad thing, is it?

best wishes, mathtalk-ga

A point of view to consider:
In order to provide bug free software, you must have both "Bug Free
Hardware" and a "Bug Free Operating System". This is a combination
I've not come across yet. So I would say it is impossible for any
software vendor who relies on these to produce bug free software.

I think you may try working a different angle.  Let this CIO convince
you he believes what he says, i.e. bug-free software is possible.
Take into account the folowing facts:

Facts of computing systems:
-----------------------------
Software requires several components and runs within a changing
environment.  These consist of:
Operating System
Hardware (including software on chips
Environmental influence (i.e. memory is electronic and magnetic static
can change everything).
Connectivity (along any journey, some data is lost and re-requested or
otherwise effected by the external environment)

CIO Assumption: Of the thousands of people and machines that went into
creating all of the systems within his company, not one person made a
single mistake and all components are free of flaws, defects and
environmental pressures OR The software you were commissioned to
develop must take into account the flaws in the systems under his
control.

Facts about people:
Humans design computers systems
Humans are known to be flawed
Humans are dramatically effected by internal and external
environmental, emotional and electro-magnetic pressures.
Humans are unpredictable and have varying and diverse, although
limited, capacities

CIO assumtions: All of the human factors under his control are free of
flaws, defects adn environmental pressures OR The software you were
commisioned to develop, needs to take into account mistakes (flaws) in
the human systems under his control.

Therefore, this CIO has already (in essence) admitted that flaws may
exist within his environment.  What I might do is propose a change in
your arrangement.

Proposal:
If the software you provides contains a single mistake, error or flaw
you will provide it free fo charge given that the systems, people and
environment are free of defects, errors or flaws.

If the systems, people, and environment in his company are found not
to be free of errors, defects or flaws you will be paid twice your
quoted rate.

Compromise:  Given that flaws already exist within any of the systems
under his control, your software will be tested reasonably and errors
and patches will be tracked appropriatly and made available for
review.  The final software submission will be held against the
Software Development Scope and Proposal and payment will be rendered
based on your meeting the defined objectives provided to you when you
accepted the project.

Probable Result: If this CIO truly believes what he says, he will take
you up on his offer.  If he does not, he is basically confirming that
what he wants from you is clearly impossible, even from himself within
an environment already under his control.  There are simply too many
variables.

My Point: It sounds as if anything you provide in the way of
references may be discounted (based on the attitude of this person). 
Reverse direction and assume what he wants from you is possible. 
Create and envirnment and agreement that demands the same from his
company and find a way to hold him accountable for it.

Truth be told, no developer would ever sign-up for a zero defect
development deal - as previously mentioned in this forum it would be
so time cosuming and expensive that the company using the software
would be broke or no longer need the software by the time it was done.

Just a small comment...

You might be interested in looking into "literate programming", which
was first postulated by Donald Knuth. Literate programming is defined
as "...the combination of documentation and source together in a
fashion suited for reading by human beings," (quote taken from link 2
below).

In other words, it allows non-techie people to get involved in the
programming process because the software is coded like an English
novel! In this way, bugs are easier to catch, and maybe more
importantly, the responsibility of a software project may be shifted
to the client (as long as the client understands English).

Here are a couple links detailing 'literate programming':
1. http://www-cs-faculty.stanford.edu/~knuth/lp.html
2. http://shelob.ce.ttu.edu/daves/lpfaq/faq.html

Good luck.

Hello Seizer,

I would like to provide an answer to your question, but I doubt you
will find one. However there is a lot of material available to improve
the quality of the products you do produce (and perhaps make it
possible to satisfy your customer).

[1] Harlan Mills published an approach generally described as
"Cleanroom" for software development. Used originally by IBM for
several development projects, it was responsible for some extremely
high quality software. Using "Citeseer", you can have references such
as
  http://citeseer.nj.nec.com/context/201910/0
which provide references to several documents on line and in technical
publications (e.g., IEEE Transactions on Software Engineering). For
more references, try searches using phrases such as
  Harlan Mills Cleanroom Software

[2] In addition to his books, Donald Knuth produced a text processing
system called TeX (pronounced "tech") which was well regarded for its
effectiveness and lack of defects. There was even a "bounty" for TeX
defects which was doubled for each defect, the following reference
  http://truetex.com/knuthchk.htm
includes a copy of a check that Knuth sent to a bug finder. As noted
this year, there have been no new bugs found since 1995 and the
current finder's fee is $20.48.

[3] A Discipline for Software Engineering by Watts Humphrey describes
a method by which individual developers can greatly improve the
quality of the software they produce (to a fraction of generally
produced). Here is a reference to it at Amazon.com
http://www.amazon.com/exec/obidos/ASIN/0201546108/ref%3Dnosim/thetechnosphere/102-1928679-8874544
As one of the reviews noted, the material is quite dry - it is really
intended to be a reference for a semester long course. I took the
"train the trainer" course from Watts and will vouch for the
effectiveness of the method.

These are three references to methods (or actual products) where the
defects were greatly reduced (but not eliminated). As others have
noted, getting to defect free software will require a number of
methods, not all of them cost effective.

I hope this helps.
  --Mark

Doug,

I took a good long look for you. Bottom line: Zero-defect software
development is not suggested as a viable possiblility by anyone who is
not selling something.
The Carnegie-Mellon study you mention is not *really* about
zero-defect software.

- Hammer

I second Maniac's recommendation that you look at Cleanroom as a
possibility. I had a wonderful college professor who wrote a text on
it:

Stavely, Allan M. Toward zero-defect programming.  Addison-Wesley,
September 1998, ISBN 0201385953.

Dr. Stavely's book is written in reasonably plain language, and geared
towards actual implementation of cleanroom techniques. I'd say that
this book alone would allow a software team to perform cleanroom
verification of real-world code.

Of course, as a programmer, I'd also say that the process of cleanroom
verification is only slightly less boring than watching grass grow. It
does work, though - our class of about ten people was able to write
useful, real-world code with very, very few errors. It's been a few
years, but if I remember correctly, testing turned up 5 or so very
minor bugs, in a Java program of a few thousand lines... this upon
running it for the very first time, just after we finished verifying
all the components. This was certainly much better than we undergrads
could have done without zero-defect verification!

Here's a review of the book that my boss at the university wrote: 
http://www.nmt.edu/~shipman/reading/stavely.html

Here's a link to the book at Amazon.com:
http://www.amazon.com/exec/obidos/tg/detail/-/0201385953/qid=1036540640/sr=8-1/ref=sr_8_1/103-0058003-8141407?v=glance&n=507846

I eventually concluded that the cleanroom method outlined in the book
was a little more rigorous than I cared to be when coding, but I also
think that some experience with the method helped me quite a bit, and
would be equally helpful to most programmers. At the very least,
cleanroom gets one used to spotting code (written by oneself or
others) that doesn't quite do what was intended. It also gives one
some perspective on common causes of errors, reinforces useful
commenting of code, and provides a framework for collaborative work
amongst a group of programmers.

This last is probably the most impressive aspect of the method, since
in my experience collaborative work amongst programmers is about as
trouble-free as collaborative work amongst cats ^__^

You might try showing this book to your client, as a form of
disagreement couched in an agreeing tone. If I remember correctly, Dr.
Stavely notes early on in the book that it's impossible to prove that
a piece of non-trivial code is bug-free. Maybe the client would react
well to a technique that admits this, but still strives to eliminate
as many bugs as possible.

Good luck, I hope this book will help!

The FastCompany article mentioned above came to mind while reading
your question.  It is, unfortunately, the only article that I recall
on this topic.  Clearly, the article shows that "zero-defect" software
is possible.  However, the same article shows that it is not
practical.  Perhaps the argument is truly over this distinction of
what is possible over what is practical.

You are probably familiar with the "Quality - Time - Money"
development triangle.  Generally, one must balance these three
factors.  Perhaps the next time this conversation comes up with the
CIO, you can refer to "zero-defect" software as cost-prohibitive or
expensive, rather than as impossible.

Unless the CIO is convinced to move to a "closed system" it would be
very difficult to design a cost-effective test plan that would
eliminate all possible issues.

By using the instructions in "A Discipline for Software Engineering" 
(by Watts S Humphrey, author of the CMM)  I have written code that was
defect-free and did not fail while I remained at the company I wrote
it for (which was another 6 months).

It's important to separate the concepts of 'defect-free' and
'failure-immune'.  A defect is a failure of the programmer to meet the
specification, or a fault in the specification itself.  A failure
occurs when the system does not meet the actual requirement of the
user.  Not all failures are caused by defects, and not all defects
cause failures.

IME, following the process in that book was very psychologically
draining - I was unable to keep that level of discipline, as I did not
have the full support of my boss for the annoying and time-consuming
statistical and reporting aspects of it.

The basic method is to write short programs (nothing more than 500
lines or so) and find out how many defects you produce per 100 lines. 
The number is remarkably stable.  You also identify your defect
re-injection rate - the rate at which you put defects in when you're
testing or fixing.  These numbers turned out to be surprisingly
stable.

You can then do testing and have a high level of confidence of how
many defects are going to slip past.  (It's also important to do your
building in very small increments.)

The schedule overhead is smaller than you might think (I'd guess
around 20%), but it is about as enjoyable as forcing your own brain
through a sieve.  Given the cost of maintenance, it's an investment
that pays off pretty quickly, if you can keep the discipline.  (Read
the warnings in the last chapter of the book about organizational
pitfalls trying to implement the PSP, and take them very seriously.)

Alternatively, explain to your customer that you are using SCRUM or
Extreme Programming so that you can respond to their needs faster. 
PSP is more reliable, but it is less enjoyable, and doesn't give a
feeling of agility (although it could be used with XP or SCRUM, just
don't tell them I said that).  Of course, this is much more convincing
if you actually implement XP or SCRUM.

I'm curious what development tools and software you are using. If one
is working in C or C++ for example, tools are very mature and
consistent, and there is a large knowledge base made up of experienced
programmers. On the other hand, using Oracle daily in my job has made
the differences in tools disturbingly clear. Oracle products are
neither consistent nor well documented, frequently buggy, and
information and debuggers of any value must be bought for large sums
of money. The structure of PL/SQL doesn't lend itself to neat coding
practices. A frustrating and overly complicated environment leads to
more bugs, not to mention bugs in the software tools themselves.
Propietary languages and tools tend to have a smaller knowledge base
available for free from which to learn.

Incidentally, I have a friend who works in the aviation industry. 
They take hours of standards training, use strict methods of design,
and must submit all code to a peer-review process. Mission critical
(still not bug free) software is produced, but at a rate of about 2
lines of code per man-hour. Would this be acceptable to your employer?
Probably not.