I would like to know how to manually calculate statistical confidence
as shown at www.splittester.com. Would you please walk me through the
calculations as though I have a calculator and a piece of paper? I
believe the test is called a t-test, but I'm not sure. The website
gives an excellent explanation of what I'm trying to accomplish - I
just need to figure out how to make the same calculation by hand.
Please use this example in your answer:
Number of Clicks first ad: 25
Number of Clicks second ad: 45
CTR of first ad: 1.5
CTR of second ad: 2.1
Result should be about 90% confidence

---

Clarification of Question by rgoyette2-ga on 04 Apr 2005 13:38 PDT

Thank you volterwd-ga,

You seem to understand exactly what I need to know. However, I do not
understand your answer because I do not have a statistics background.
Would you mind solving the example I provided as though you were using
a pencil, paper, and a calculator?

Thank you for clarifying - I'm excited to be able to calculate this
example on my own soon!

-Rob

---

Clarification of Question by rgoyette2-ga on 06 Apr 2005 12:45 PDT

Hello,
Thank you for your help so far. I'd like to add $5 to the price for
your trouble. Can you please explain how you got the p-value of
0.1625328 in your example?

I was able to follow everything up to calculating t0. I feel like I
can do that. But how do I get 0.1625328 as a p-value?

Thank you and I'll add $5 to the total because you've been so helpful!

-Rob

This seems like a simple problem.

You have an experiment You have a click through rate and number of ads clicked...

for example click through rate 10% number of clicks 100

so we had 1000 impressions... (100/0.10) = (number of clicks)/(click
through rate[CTR])

assuming that the clicks are all independent trials with the same
probability p then we have that the CTR is approximately normal for
large enough N (where N is the number of impressions)

So basically all your doing is a test of H0: p1 = p2 vs H1: p1 \not = p2 

Im assuming you know how to do this... if not let me know.

So if your wondering what test it is... its a two sample t-test for
sample proportions...

Ok here we go... first we need to calculate N1 and N2 the page impressions

N1 = 25/0.015 = 1666.6666... = 1667 (there is some rounding because
you gave an approximate CTR)
p1hat = 0.015
N2 = 45/0.021 = 2142.857...  = 2143
p2hat = 0.021

Out test is
H0: p1=p2 vs H1: p1 /not = p2

NOTE a two sided test is appropriate not one sided

So under the assumptions i said before (which are just the most simple
assumptions) our test statistic is

t0 = (p1hat - p2hat) / sqrt( p1hat*(1-p1hat)/N1 + p2hat * (1-p2hat)/N2 )

in our case 

t0 = -1.396604

So our p-value is

p-value =  0.1625328

Note for such a large value of N1 and N2 we could just treat t0 as
normal if we wanted to but since the estimated values of p1 and p2 are
so small its not a good idea

So for 10% confidence... we have 0.10<0.1625

Since the p-value is greater than the desired confidence we can not
reject the null hypothesis that they are the same at 10% confidence.

In general even though N1 and N2 are large since p1hat and p2hat are
small you will need more numbers to determine if they are different...

Now dont worry about the assumptions... even though the CTR's may not
be constant for all individuals thats not important... in reality
there is not much you can do about that... and im sure the site you
are refering to does it this way anyways...

so to sum it all up... at 90% confidence you can not say they are different

I would like to point out that if you use his program you will
actually get the wrong result.  That is because he uses a one sided
alternative when in reality you should use a two sided alternative.  I
actually downloaded the excel file he has... everything is the same as
it should be but his p-value is half of the correct one because he is
using a one sided alternative incorrectly.

So he calculates the pvalue of the test as 0.1625328/2 = 0.0812664
which is smaller than 0.10... so at 90% confidence... he says they are
different but he is improperly using that.

So in general i would use your custom program to do this yourself and not use his.

Here is an R function that you can use to do the test.

CTRtest<-function(CTR1,CTR2,clicks1,clicks2){
#CTR    - Click through rate in %
#clicks - Total number of clicks
CTR1<-CTR1/100
CTR2<-CTR2/100
N1<-round(clicks1/CTR1)
N2<-round(clicks2/CTR2)
t0<-(CTR1-CTR2)/sqrt(CTR1*(1-CTR1)/N1+CTR2*(1-CTR2)/N2)
if(N1+N2-2>300){
pvalue<-2*pnorm(abs(t0),lower.tail=F)}
else{
pvalue<-2*pt(abs(t0),df=N1+N2-2,lower.tail=F)}
pvalue
}

Just copy and paste that in and to use it in our previous example just type

CTRtest(1.5, 2.1, 25, 45)

The output should be

[1] 0.1625328

This is the p-value which you compare to the significance... at 90%
confidence... 0.16 > 1-.9 = 0.1.

So the click through rates are 'the same'