I plan to run a test with a binomial output: pass or fail. I want to
show that the failure rate is zero, with a 90% confidence level. What
is my required sample size? What is the formula to determine this?

---

Request for Question Clarification by hedgie-ga on 06 Mar 2003 03:15 PST

Hi Leonard

  Would it acceptable to have formula for this,
slightly modified question:

I want to show that failure rate is less then  e  (epsilon),
where e could be a small number... 
.. what sample size I need?

hedgie

---

Clarification of Question by leonardjk-ga on 06 Mar 2003 15:33 PST

Hedgie,

I'm aware of formulas that assume a non-zero failure rate. However, it
seems to me that the confidence level serves the purpose of
uncertainty. Given that we are only 90% certain we are right, why
can't the failure rate be 0? If the confidence level desired was 100%,
then of course the sample size would be infinity. If, through my
ignorance of statistical methodologies, it turns out that what I am
asking is not possible, then I'll be happy to accept your answer. But
I'd like to wait a few days to see what others have to say.

Thanks

---

Request for Question Clarification by hedgie-ga on 07 Mar 2003 06:42 PST

OK Leonard

     I will hold off  until you post another clarification, asking for my
answer. The confidence interval indeed describes the uncertainty,
but the usual application is:  how confident we are that (e.g.)
a mean of a distribution is IN THE INTERVAL  mean +/- std.error.
where mean is observed (estimates) mean=average in this case.

If we are 99% certain, then probability that mean it is really outside that
interval is small  (.01 in this case).  Right?

 So  the confidence level depends on
both, sample size AND size of the interval.

To see why you need interval, rather then a sharp value (a zero) imagine
the failure rate is very very small. Let's say e=E-20 (that 1/10 to power of
20). Clearly, the smaller the e is, the larger N you will need to reach that
same confidence level, that you did not got that string of zeros by chance ....
The answer is available to all.
hedgie

---

Clarification of Question by leonardjk-ga on 27 Mar 2003 13:54 PST

Hedgie,

Sorry for the delay. Based on your input I'd like to revise the
question, and have upped the payment as well for your patience.

Test as follows: A measurement is taken with a pass criteria of 1000,
+/- 5.
Acceptable failure rate is 1 in 1000. How many units do I need to test
to achieve a given confidence level?

And an opinion would be appreciated: in a non-critical application,
what confidence level would you expect to see for someone making this
kind of claim? 90%?

Thanks

No problem Leonard,

 There are many applets on the web which illustrate
 the dependence of confidence level on confidence
 interval size and sample size for a normal distribution.

 Here is one, which seems to me clearer than most:
 (You need to have Java enabled in your browser to see this)

 http://www.rub.rice.ed/~lane/start_SIM/con_interval/

 It shows how confidence level depends on the two
 parameters,  the confidence interval and the sample size.
 
 Confidence Interval is usually stated in the number of sigmas.
 3 sigmas mean the interval is +/- 3 standard deviations above and
below
 the mean.

 This applet , like most, deals with the normal distribution.



 In some cases your problem can be solved aproximately by using such
an approach and the usual tables.  The  reasoning goes like this:

 After N tests, you will get K errors (fails or zeros) and N-K passes
(or ones).
 As you stated the problem, N is large and K is small.  The small
number
 K/N  - the  estimate of the true probability (failure rate) will have
an
approximately normal distribution.
Meaning, if you took your N samples a hundred times you would get
100 different Ks. Let's call them k1, k2,  ... k100 determined by
total N * 100 tests. A histogram of hundred values k1/N, .. K100/N
will approximate the normal distribution centered around the unknown
error rate
you want to estimate. Explanation of 'histogram' is given here:
http://www.statsoftinc.com/textbook/glosh.html

If your N is so large that the sigma gets small when compared with the
error
rate e, so that lower end of the confidence interval is positive, this
approximation would work.


 In general, however, that is not the case:

"When you are calculating confidence intervals for proportions (p =
K/N) and the
proportion is close to 0 or 1, then you should not use the normal
distribution
to calculate confidence intervals (in general, normal distribution
approximation
is okay for .3 < p < .7). You need to calculate tht eexact confidence
intervals
using a different probability model for binomial data. The functions
here
calculate exact confidence limits using a binomial model. For very
large n and
small p, you can use the Poisson model to approximate the binomial
distribution.

To learn more about binomial models used for epidemiologic analysis,
see Selvin
(1996)"  http://www.medepi.org/epitools/rfunctions/cibinom.html

The general formula is here:
http://www.weibull.com/LifeDataWeb/beta_binomial_confidence_bounds.htm

and the  program which solves that equation is described here:
http://www.itl.nist.gov/div898/handbook/prc/section2/prc241.htm



There is additional theory and a 'calculator' on this site

http://members.aol.com/johnp71/javastat.html

under these two links:
 #  Exact C.I.'s for Binomial (observed proportion)
 and Poisson (observed count)

 #  Binomial Test -- whether the number of "successes" differ from
what was
expected based on the number of trials and the probability of success.

The theory looks OK to me and the calculator shows what is possible,
BUT
 I am not sure that it  works correctly  for all values. 
(Note this is not a university site).


Your other question:
 "in a non-critical application,
what confidence level would you expect to see for someone making this
kind of claim? 90%?"
is answered by the Decision Theory
http://pespmc1.vub.ac.be/ASC/DECISI_THEOR.html

 In a nutshell, you quantify that "non-critical" by estimating the
 cost associated with being wrong. The probability of being wrong
 (given by your confidence level) times the cost will give you a
 "mathematical expectation" of the cost. You consider all strategies
 and select the one with the least expected cost.
 This is just a complicated (quantitative) way of saying: The more
 costly the failure, the more confident you need to be it will not
happen.

 If there are real money involved, you may want to post that as a
separate
 question. A Game Theory expert may pick it up (but you may want to
 review the pricing guide first
 http://answers.google.com/answers/faq.html#howmuch)


Search Terms

confidence intervals
confidence level
binominal distribution
decision analysis

    Please, do ask if some links does not work or someting
    is not clear. It is more complex then it seems on the first look.

    hedgie

---

Clarification of Answer by hedgie-ga on 29 Mar 2003 09:57 PST

Leonard,

---

Clarification of Answer by hedgie-ga on 29 Mar 2003 10:04 PST

I was looking around a bit more and found another
calculator, this one from University of Baltimore,
which looks more credible. It does check for unreasonable
inputs (like k>N) and does not require download.
You can find it at
http://www.ubmail.ubalt.edu/~harsham/Business-stat/otherapplets/ConfIntPro.htm?

Parametr names are different (m number of successes is out k)
but otherwise it will give you sample size for given confidence limits.

Here .99 means 99%  etc.

This should allow you to charter the N vs confidence level dependence.