I am a team member in a research in which a group of CS students fill forms
regarding computer companies. the role of the students is as follows:
everyone takes a blank form, which has 45 multiple choice questions,
then the students  enter a web site of a computer company from a given
list, and fills the form by answering the set of 45 questions. most of
the questions are pure facts - what is the name of the company, does
the site secure etc. 5 questions are of different nature - in those
the student writes whether the site is informative enough,
interesting, etc., by choosing a number (1 to 5).
Up to date, half of the students had participated and filled 50 forms
each, and half didn't participate yet.
Our Statistical adviser says that we should fire the half that didn't
help us by now for a statistical purpose as they might damage the
statistical accuracy of the research.

I just don't buy it. I think that it doesn't matter whether the
students that didn't participate yet will join us in the near future.

Please give me a reasoned answer on this matter. I prefer someone that
dealt with statistic research in the past.

Best regards,
R.

---

Clarification of Question by romko-ga on 10 Jul 2006 14:39 PDT

Hi,

We have 18 students.

Up till now only half (9) began working. each of those 9 filled 50 forms.

The other half (9) didn't show up and didn't fill a single form.

The advisor thinks that we better keep 9 students till we finish all the forms,
and fire the 9 that didn't come yet.

I think that we should use them all (the 18 students) and finish the
research faster.

The claim of the advisor is that if the "lazy" 9 start late, the
accuracy of the research will drop.

This smells of homework to me

Essentially it is saying that you have a population of 100 and you
only listen to the views of the first 50 that turn up.

Using Reducto Ad Absurdum, the views of the first to turn up are taken
as the 'sample'.

Nope.

This is the real story from my actual life.

I assure you this is no fiction.

Ro.

Okay, then a couple of questions.

What did the advisor mean by "fire" the non-participants?  Just ignore
them, assume that the survey was complete with 50 participants?

Do you then mean by "join us" that those persons will perhaps soon
submit their questionaires?

I know extremely little about statistics, but I think that
non-response is also a category that will be important for calculating
the range of accuracy of the results you do have.

Thanks for the clarification.  Obviously, I misunderstood the situation.

As I now understand it, the students fill in the questionaires about
different websites, only five of the questions being subjective and
answered on 1 to 5 scale.  Apparently   - my assumption -  each
website is only appraised by one student, so the number of students
participating in the work is not of statistical importance for the
results of the questionaires.  This assumption presumes a measure of
objectivity on the part of the students.

If this is a correct understanding of the situation, I would agree
with the advisor: it is better to let the active 9 students complete
the work, rather than waiting and hoping that the other 9 join in.  If
the latter were offered money to participate, and missed the work
session, they don't deserve a second chance.  The nine more
consciencious students should be allowed to complete the project.

I may have been reading incorrectly between the lines, but if I
haven't, it seems to me that the question is not so much about a
"statistical matter" but rather about a justificaton for the best way
to complete the project.

Dear myoarin,

The question is not about handling our staff on the project.

The real question is of Statistical importance:

Can we use more people, in order to complete the research faster, or not.

R.

Wow - I'm so disappointed.

No answer for this question??

I was sure that this one was easy but I see It is not.

Maybe my expectations were too high...

R.

Romko,
I still have a problem understanding your question.
To most of us it would seem evident that by using more people the
project can be completed faster.   9 students can complete 450 in x
time (call it 1 day);  18 could complete 900 in x time, but 9 didn't
show up.  If they turned up the second day and were as diligent as the
first nine, then 1350 forms could be completed in two days, certainly
better than "firing" them and foregoing their efforts.

BUT, the advisor raise the question of accuracy, a logical but
unfathomable factor from this side.  I expect that this relates to the
students' rating of those five questions and a fear that maybe less
conscience students  - or ones trying to make of for lost time - 
would rate haphazardly, diminishing the quality of the whole project.

But this is just speculation on my part as I try to understand the question.

hello myoarin,

First of all - thanks.

Please note that I'm not trying to find the "common sense" answer,
but trying to get a scientific-statistical one.

the science of statistics is AFAIK a mathematical field with it's own rules
and standards. I would like to get a professional answer on my question.

Thanks,
R.

I suggest your adviser is not so worried about the statistical
"accuracy" of the research; that is really a function of the
experimental design. However, it does seem clear that you have two
pools of observers: one committed and one less so and you do have
subjective questions so the observers influence the observations for
those questions at least. Since the assumption of most statistics is
that all observations are from a common pool, and you know you have
two different pools (based on this criterion) of observers that can
influence the observations, your statistical adviser is advising you
that the basic assumption is incorrect in this case.

Since there are probably many other factors dividing the observers
into different pools (how many males vs. females, different ethnic
groups, etc.) the adviser may be erring on the side of caution.
Reducing the size of a randomly chosen pool of observers post
selection probably also has statistical risks; but that is why you
have a statistical adviser, no?

Thanks dmrmv.

I still think that there is no harm in using the extra force.

R.