I'm having a hard time understanding why the taguchi method for
determining the best option for a variable works better than randomly
selecting an option for each variable and testing it.

Here's a primer on the taguchi method:
http://www.offermatica.com/media/articles/landing_page_optimization_using_the_taguchi_method.html

Here a link to a spreadsheet which shows how to make the "recipes":
http://www.kowalick.com/freetaguchi/download.html

After reading those, it seems to me like this method has no advantages
over randomly selecting an option for each variable each time the page
loads. Here's my argument:


Pretend you have 3 variables with 4 options each.
Total number of unique combinations is 4x4x4 = 64.
 
Testing all unique combinations requires 64 tests and gives 16 tests
per any given option.
 
Consider randomness, ie: on each load of the page, there is a random
selection of each of the 4 options for each 3 variables. If we run it
64 times then each variable would have been shown 1/4...0.25 x 64 = 16
times, so there are 16 tests per any given option.
 
Consider taguchi: Let's say it creates 16 recipes on the data.
For a visual, each column is a recipe:
Var A  1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4
Var B  1 1 2 2 3 3 4 4 1 1 2 2 3 3 4 4
Var C  1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
If we run 64 tests, then each recipe is randomly selected probably 4
times. Assuming each option was used 4 times (equal share) then there
were probably 16 tests per any given option.
 
 
So, the results look the same: all of them had a total of 64 tests and
all options have 16 tests each. However, I feel the first 2 are better
than taguchi because every option got a chance with every single other
option. Look at the recipe - the combination (1,4,4) will NEVER be
tested, where with randomness and all, it will be tested.

So, please confirm I am right or explain why I am wrong. Thank you.

Interesting

I skimmed the link you gave on Taguchi and got the impression that it
is for use when you have significantly more options than you have
'lives'.

Obviously exhaustive testing is the optimum method
- and given a large number of 'lives' random selection will tend
towards exhaustive testing

If you don't have many 'lives', then with random selection of tests
one risks just testing a 'related cluster'

I have heard of something similar from neural networks called
'heating', where you radically change all the variables - big time.
And see what happens.

My impression is that the general idea is to 'home in' on results that
are unexpected, and then examine (further test) in more detail.
ie: it is not meant to be used mechanically - rather the results
require intelligent evaluation.

It makes a lot of sense as a means of 'using limited resources'

I'm starting to think that the only situation taguchi would be best is
if it costs resources to make a recipe. For example - if you are
testing out various posters, it costs money to redesign different
ones, print them, etc. However, when the media is a website and it
doesn't cost anything to randomly rotate content, then I don't see the
advantage

Here's a comment from a friend of mine:
 
"The trick is you need to see a number of times for each variable
group to run. So, where on the 4x4x4= 64 that means you have 64
seperate sites to run.
 
Let's say then that you need a group of 100 visitors to give it a
solid test. That is 100 x 64 = 6400 hits needed.
 
Vs Taguchi which gives you 3 recipes x 100 visitors to give solid
test. That is 100 x 3 = 300 hits needed.
 
Each hit costs $0.25
 
That measures the difference between the two in THOUSANDS of dollars."

You don't need 64 separate sites
- just 64 HTML pages
- and even then the pages could be 'assembled on the fly'
- each visitor gets a different page

The 'cost' would be the setup (programming)

The other 'cost' would be lost revenue from putting up unattractive pages.

Personally my opinion is that the 'success' of a web page is largely
down to its content - and that 'over designed' pages simply confuse
people.
But maybe I'm biased - I dislike Flash, animations and Java, although
Style Sheets have their moments.

Good luck with your project.

Here's my response to my friend's argument:

>Let's say then that you need a group of 100 visitors to give it a
solid test. That is 100 x 64 = 6400 hits needed.
You're working it the wrong way - if you get 100 visitors, and each
variable has a 1 in 4 chance of coming up, then you'd get 100/4 = 25
tests for each variable.
 
>Vs Taguchi which gives you 3 recipes x 100 visitors to give solid
test. That is 100 x 3 = 300 hits needed.
First off, you can't have only 3 recipes and be able to get any data
for all variables - 1 would be left out. You need a minimum of 4
recipes. Example given:
Var A  1 2 3 4
Var B  2 3 4 1
Var C  3 4 1 2
So now if we have 100 visitors and each variable has a 1.4 chance of
coming up, there are 100 / 4 = 25 tests for each variable. Where's the
advantage?
 
I still argue that Taguchi only is useful in situations where there is
a cost involved in making the permutations - that's why it's so
popular in manufacturing - but when things can be dynamically rotated
via a webpage at no expense, there is no advantage (in fact there is a
loss of integrity since the combination of variables is limited to
specific variables in their recipe).

<quote>
I still argue that Taguchi only is useful in situations where there is
a cost involved in making the permutations - that's why it's so
popular in manufacturing - but when things can be dynamically rotated
via a webpage at no expense, there is no advantage (in fact there is a
loss of integrity since the combination of variables is limited to
specific variables in their recipe).
</quote>

I agree with you - subject to one small proviso
- eliminating ridiculous combinations

I also suspect that the Taguchi method is what I call a 'benign con'
- a means of preventing research being a slow meticulous dragnet,
rather than testing the 'envelope'
- in other words a justification for trying out 'hunches' and not
allotting blame if they fail.

Thanks for all your input frde, you've been very helpful.