Hi,

I've done a study where I took 17 paired subjects through each of
three conditions. At the end of the session I asked them if they had
to do it again which they would choose, to get an idea which one they
liked best.

So I now have results from all 34 subjects: 6 preferred the first
condition, 11 the second, and 17 the third, and I'd like to know if
these results are significant. A simple Chi Square test is tempting,
were it not for the fact that the 34 subjects were tested in 17 pairs,
so I cannot assume independence for the 34 data points. Nor can I
average each pair's result since this is nominal / categorical data.

The ordering of the conditions was varied across the sessions. The
pairs were not randomly assigned: having recruited one subject we
recruited a friend or colleague as the second subject.

Which test should be using?


			Thanks,


			 Tim.

PS If you are interested, here are the results. Each row is a
different pair of subjects; their preferences are written right
subject’s, comma, and then left subject’s.

3, 3	
2, 1
1, 2
2, 2
1, 1	
3, 2	
2, 3	
3, 3	
3, 2	
1, 3	
2, 2	
2, 3	
3, 3	
2, 3	
3, 3	
3, 3	
1, 3

---

Request for Question Clarification by calebu2-ga on 30 Aug 2002 11:12 PDT

Dumbledad,

I'm not sure if I can sit down and think through the full problem, so
I at least wanted to freely pass on my initial thoughts to whichever
researcher decides to take on the question. (I might come back to it
later, but I normally need a clear mental focus to think through this
stuff :)

I'm also a little bit unsure on a few points, so I figured I'd post
what I had thought up so far and ask some questions to see whether
clarification could prompt further understanding on my part.

Much of what I type comes from knowledge I learned from the following
page of course notes :

Statistics 1B course at Cambridge University (Lecture 9) :
http://www.statslab.cam.ac.uk/~rrw1/stats/S09.pdf

If you are unsure about the independence of the two group responses,
one way to do it would be to arrange the data in the following grid:

              1st respondent
                 1     2     3     TOT
        1       1     1     0       2
2nd  2       1     2     2       5
        3       2     3     5       10
        TOT  4     6     7       17

You then want to test the hypothesis H0: pij = piqj with 0 <= pi, qj
<= 1, sum(pi)=1, sum(qj)=1
vs H1: pij arbitrary s.t. sum(pij) = 1.

The article explains how to do this. The relevant chi squared
statistic is one with 4 d.f.

To answer your main question, I'm stuck on the question of what you
mean by "results are significant"? Do you mean that they are
significantly different from an equal mix ie. pi = p, qj = q?

If so, I believe that what you want to do is test the hypothesis H0:
pij = 1/9 vs H1: pij arbitrary. The d.f. in this case is 8 and the
relevant statistic is the chi-squared distribution.

This approach requires very little assumption about the distribution
of the numbers. If you have some kind of idea of how the two groups
might be related, you are often better off doing a simulation study
and seeing how many times in a group of 10000 trials you get a set of
results that can be categorized by the grid given (or if you decide
what you deem "significant" - a grid of that level of strangeness or
greater).

Let me know whether this is the question you are trying to answer, if
so one of the researchers will give it a closer look (e.g. do the
calculations) - if not, define what you mean by significant and we can
try something else.

Regards

calebu2-ga

---

Clarification of Question by dumbledad-ga on 31 Aug 2002 10:41 PDT

Hi,

Thanks for the comments calebu2-ga. However the Chi Squared test you
describe is not what I'm after. I tried a similar Chi Squared myself
by taking the possible vote patterns (1,1; 1,2; 1,3; 2,1; 2,2; 2,3;
3,1; 3,2; 3,3) and comparing their actual number of occurances (1; 1;
2; 1; 2; 3; 0; 2; 5) with the expected number of occurances (17/9;
17/9; 17/9; 17/9; 17/9; 17/9; 17/9; 17/9; 17/9). This gives a p value
of 0.35, and so shows no sign of preference.

However this isn't testing the significance of 6 people preferring the
first
condition, 11 the second, and 17 the third. Is the third condition
significantly more popular than the first? To answer that question I
think I need a test that will take the dependence between the pairs
into account while still testing individuals preferences accross the
conditions. Does that make sense?

I don't think I can do a simulation study. The three conditions are
three versions of an application I've developped and I really don't
know which one users will prefer. Hence why I bought them into the lab
to try three versions out. The application itself is designed for two
friends to use together - hence the paired subjects.

I did also think of proving that the two subjects within a pair are
unrelated? One plan (suggested by a colleague) was to analyse the data
for correlation between the subjects in a each pair, and hopefully
show that there was none. We could then use that as a justification to
assume independence between the pairs. We used a double entry method (
http://users.rcn.com/dakenny/dyad.htm#Top5 ) and Spearman's ranked
correllation rho to get a correlation of 0.22 with a p-value = 0.21.
That's not good reason to suggest a correlation but neither is it good
reason to reject one.

So I think I'm still looking for the right test. It must be out there
somewhere - there's a statistical test for any eventuality ;-)


         Thanks,


          Tim.