swaskexpert1935...
StatPac is a company which creates software for processing surveys.
They have an excellent page on their site which precisely addresses
your question:
http://www.statpac.com/surveys/question-qualities.htm
A summary of their stipulations for a 'good' question is as follows:
1. Evokes the truth. Questions must be non-threatening.
2. Asks for an answer on only one dimension.
(doesn't ask 2 questions in one)
3. Can accommodate all possible answers.
(mutiple choices include all potential responses)
4. Has mutually exclusive options.
(only one appropriate choice is possible)
5. Produces variability of responses.
(not everyone will answer the question the same,
thus producing no useful information)
6. Follows comfortably from the previous question.
(questions don't jump from one topic to another)
7. Does not presuppose a certain state of affairs.
(doesn't assume the content is familiar/known to all)
8. Does not imply a desired answer.
(isn't a 'leading' question)
9. Does not use emotionally loaded or vaguely defined words.
(e.g. quantifying adjectives which mean different things
to different people, e.g. 'several' or 'few')
10. Does not use unfamiliar words or abbreviations.
(takes survey recipients' knowledge into account)
12. Does not ask the respondent to order or rank a series of
more than five items.
(e.g. the percentage of each item - requires too much
thought, threatening the accuracy of obtained info)
There is considerable more discussion available on the page.
-----------------------------------------------------------
Such vagueness in survey questions are often severely
criticized by scholars concerned with the outcomes.
In an article by Andrea Lynn, Humanities Editor of the News
Bureau at the University of Illinois at Urbana-Champaign,
Illinois political scientist Robert Weissberg, argues that:
"'with scant exception' conventional survey questions omit
actual costs of proposed entitlements and 'often avoid
anything to do with money, let alone raising taxes.'
Shunning tangible costs and contexts not only opens the
door to 'immense mischief and misleading information,'
Weissberg argues in the most recent issue of The Public
Interest, but it also renders the data collected from
standard public opinion polling on social welfare
'nearly worthless.'"
Andrea's article goes on to say:
"Another format uses 'smallish, enticing round numbers,'
Weissberg wrote. A 1992 Gallup Poll, for example, asked
respondents if they personally would be willing to spend
$200 yearly to combat air pollution. While this format
appears to be more honest, the price tag seemed to be
'plucked from thin air,' Weissberg noted. 'If the Gallup
organization had done its arithmetic, interviewees would
know that this figure quadrupled the entire EPA budget
while boosting the average tax rate 3.6 percent.'"
http://www.news.uiuc.edu/gentips/02/09surveys.html
-----------------------------------------------------------
A pdf document, on the UK site for the Department of Education
and Skills, reaches this conclusion regarding vague terminology:
"Many surveys are produced within the UK that address, or at
least touch upon, issues to do with skill deficiencies. But
the interpretation of these surveys is bedevilled by
differences in methodology, terminology, and phraseology,
and also by some of the inherent ambiguities in the subject,
and by suspicions over how respondents understand the
terminology of questions. For example, about 60 per cent of
respondents in the CBI manufacturing survey thought that the
question regarding whether or not skilled labour was likely
to limit output over the next four months was concerned with
difficulties in recruiting the right skilled labour, while
45 per cent saw it as reference to problems in relation to
their current workforce."
http://www.skillsbase.dfes.gov.uk/downloads/SKT30.pdf
In summary, it seems that, if questions are not carefully
worded, survey outcomes are not worth the paper they're
printed on, and they risk costing considerably more than
the price of undertaking them, in terms of damage to
public or corporate policies based on erratic data.
Please do not rate this answer until you are satisfied that
the answer cannot be improved upon by means of a dialog with
the researcher through the "Request for Clarification" process.
sublime1-ga
Searches done, via Google:
"survey questions" phraseology
://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=%22survey+questions%22+phraseology |
Clarification of Answer by
sublime1-ga
on
01 Jan 2003 13:55 PST
swaskexpert1935...
You said:
"I was hoping for references to citable academic research
involving experiments with different wordings or presentations."
While this is considerably more specific than your original question,
I have found some examples of this type of research (special thanks
and a tip o' the hat to fellow researcher tehuti-ga).
The following study, published in New Scientist online magazine,
by By Peter Aldhous, Andy Coghlan and Jon Copley, explored
the way in which wording effects the responses of those surveyed
to the very touchy subject of animal testing.
The study is presented here:
http://www.newscientist.com/hottopics/animalexperiments/letthe.jsp
...and graphic results of the study are displayed here:
http://www.newscientist.com/hottopics/animalexperiments/img/costandbenefit.jpg
and here:
http://www.newscientist.com/hottopics/animalexperiments/img/species.jpg
Another expert resource for this area is the author of The
Reseach Methods Knowledge Base. William M.K. Trochim is a
Professor in the Department of Policy Analysis and Management
at Cornell University. The Knowledge Base is substantially
available online, here:
http://trochim.human.cornell.edu/kb/index.htm
Trochim, William M. The Research Methods Knowledge Base, 2nd Edition.
(version current as of August 02, 2000).
Professor Trochim is also the developer of The Concept SystemŽ
and founder of Concept Systems Incorporated:
http://www.conceptsystems.com/
Concept Systems has a contact page here:
http://www.conceptsystems.com/contact.html
...and you may be able to contact Professor Trochim via email at
infodesk@conceptsystems.com
This is an extensive database which includes the construction
and wording of effective surveys and questionnaires.
The Table of Contents is here:
http://trochim.human.cornell.edu/kb/contents.htm
The database is searchable here:
http://trochim.human.cornell.edu/kb/search.htm
A discussion of "multi-option variables" (lists) is here:
http://trochim.omni.cornell.edu/kb/quesresp.htm
He notes the following in making use of a checklist
"Are all of the alternatives covered?
Is the list of reasonable length?
Is the wording impartial?
Is the form of the response easy, uniform?"
The Knowledge Base is available for puchase, in both
online and printed format, here:
http://www.atomicdog.com/BookDetails.asp?BookEditionID=34
You said:
"I am particularly interested in the weakness of the "other"
response as a substitute for an extended list."
While the relative weakness of the use of "other", as opposed
to a comprehensive list, would seem obvious, based on what
we have explored thus far, I can understand your desire to cite
specific academic research which reaches this conclusion.
I did search, extensively, for the existence of such studies,
using search strategies such as:
study phraseology OR terminology "use of other" "extended list"
The 5 results of this search, at the following link, were useless:
://www.google.com/search?num=50&hl=en&lr=&ie=UTF-8&oe=UTF-8&safe=off&q=study+phraseology+OR+terminology+%22use+of+other%22+%22extended+list%22
Unfortunately, databases of such studies are usually available
only by way of paid subscription or membership, to which I do
not have access. Even if I did, I could not direct you to the page
I might quote, as it would ask you for your member ID and password.
One such database is Ingenta <www.ingenta.com>
(special thanks and a tip o' the hat to fellow reseacher
pinkfreud-ga):
"The most comprehensive collection of academic and professional
publications available for online, fax and Ariel delivery."
You may want to consider joining such a database to research
such specific inquiries.
If you feel I have left any digital stone unturned in my efforts
to answer your question, please feel free to request another
clarification prior to rating this answer.
sublime1-ga
|
,
0 Comments
)
