I'm curious how to use statistics to distinguish between "luck" and
"influence".

Example A:___________
Lets say I have a website, and there's a link that gets clicked by 5%
of the users on the average.  Then, lets say that I move the link in
hopes to increase that percentage.  How do I determine how long to
leave the link in the new location in order to calculate a trustworthy
percentage for my average?

To illustrate my question further... I could move the link, then
analyze what the next 20 visitors did.  But I wouldn't trust that
number.  Or I could wait and analyze what the next 2 million visitors
did, but I'd be waiting too long.  I could even analyze only 1 million
visitors, because I'm guessing that the average for 1 million would be
pretty much the same as the average for 2 million.  But what math can
I use to determine a good "sample size"?


Example B:___________
What if I somehow rigged a theoretical die so that two numbers never
showed.  How many times would I have to roll that die to ensure that
my die was successfully rigged?  What if I rigged a pair of die?  How
many times would I have to roll the pair to ensure that my cheating
worked, and I wasn't just seeing a "streak"?


Example C:___________
Lets say I determine that customers of my Magic Eel-Face BBQ are
segmented like this: 80% are brown-eyed; 10% are blue-eyed; 10% are
pink-eyed.  Then I run an ad that says "We love pink-eyed people."  As
soon as the ad is printed and released, I start keeping track of my
customers' eye-color... but for how long?  How many customers (after
the ad runs) do I have to analyze in order to be fairly certain that
the ad (assuming no other influences) changed the demographics of my
BBQ?  Would my "minimum reliable sample size" be different if my ad
was "We love brown-eyed people"?


These are just examples of the principles of "certainty" that I want
to learn about.  Feel free to use the examples to teach me, but
remember that I'm want to understand the principles so I can apply
them to other nifty things. I want to know the principles and basic
statistics that can help me distinguish between the natural variance
in life vs. a real change.  When I close this Google Answers question,
I hope to have the knowledge and tools to immediately begin
calculating a) the influence and b) certainty thereof of my actions
(moving a link, rigging the dice, running an ad).

Please define all terms, considering that geometry is as far as I got
in math.  If you can find an Excel spreadsheet that can help me with
this, that'd be excellent.  Free or low-cost software would better. 
Feel free to recommend any software <$150.  I've purchased and perused
"A Cartoon Guide to Statistics".  I already understand these
concepts...

mode
median
mean
variance
standard deviation (conceptual understanding only: ignorant on the
math/numbers)

In order to understand the answer to this question, you would need to
know, in addition to what you already stated, Theory of Probability;
especially, Probability Distributions. If the answer requires a
researcher to explain these topics and then answer your specific
question, I really doubt that someone will answer it for $20.

I strongly suggest that you check the following link. 
http://davidmlane.com/hyperstat/logic_hypothesis.html

What you want to know is well-studied, and is called Hypothesis
Testing. It's not so difficult to grasp; but, as I said, it requires
knowledge of Probability Distributions.


Best regards,

elmarto
Google Answers Researcher

In your first example, if you assume that people fall into two
distinct groups, those that will click the link (5%) and those that
won't (95%), and that they arrive at the site in a random order, and
you count how many of each sequence of 500 visitors click the link,
the mean number of clickers will be 25 (n), and the standard deviation
of the mean number will be 5 (sqrt[n]).  So 'typical' values (within
one standard deviation of the mean) are 20 to 30 clickers per 500;
values outside of the 15-35 range (two standard deviations from the
mean) can be considered unusual.  If the percentage were to jump to
7.2%, the mean value would be 36 clickers out of 500, with a standard
deviation of sqrt(36) or 6.  So again, 'typical': 30-42, 'not
unusual': 24-48.  Since these ranges overlap with those in the 5%
case, you couldn't necessarily determine that the shift was real by
looking at only one group of 500 visitors before, and one after, the
supposed shift.

Now let's say you take groups of 50,000 people.  At 5%, that means the
mean number of clickers is 2500, and the standard deviation about that
mean is sqrt(2500), or 50.  So, 'typical': 2450-2550, 'not unusual':
2400-2600.  Meanwhile, for a rate of 7.2%, the mean is 3600, the s.d.
60, and the ranges 'typical': 3540-3660 and 'not unusual': 3380-3720. 
So by looking at 50,000 visitors, you would be certain (in all
practical terms) that the shift was real.

>>I really doubt that someone will answer it for $20.

Thanks for the feedback.  I really wish such feedback were more
commonplace on Google Answers: the very same ignorance that prompts me
to post a question makes it impossible to set a good price.  :-/

Thank you also for framing the problem and tipping me off to the
appropriate vocabulary.  I'll do more investigation and reprice/repost
the question as needed.  I'll leave the question open for comments,
though.