I’m hoping that I have a few helpful academics amongst my readers here.
As I guess most people know, a lot of research in various fields revolves around questionnaires. You get this in politics, in marketing, and in sociology. The matter of questionnaire design is therefore important, because badly designed questions can bias the results, right?
Now suppose you are designing a survey to measure public attitudes towards something, say science fiction. It seems to me self-evident that if your questionnaire is relentlessly negative about the subject then you will a) encourage a negative response and b) leave your respondents with a more negative view of the subject than when they started. To illustrate the point, here are two short sets of questions.
Neutral Questions
On a scale of 1 (low) to 5 (high) please rate the following:
1. How intelligent are science fiction readers compared to readers in general?
2. What is the quality of writing like in science fiction compared to other fiction?
3. How likely are you to want a science fiction reader as a friend?
Less Neutral Questions
One a scale of 1 (strongly disagree) to 5 (strongly agree) please indicate how you feel about these statements:
1. Science fiction readers are stupid people who live in fantasy world.
2. Science fiction books are very poorly written.
3. Science fiction readers are dull, boring people with poor personal hygiene.
See what I mean?
I’m sure that someone, somewhere has done research into the comparative effectiveness of these two strategies. But finding that research isn’t easy, and an academic who has inherited the “less neutral” methodology from previous work in the field isn’t going to be able to challenge those previous methods without proof that they are suspect. Has anyone out there ever done any work on questionnaire design, or read any work on it, that might help?
Of course the survey I’m actually interested in isn’t about science fiction readers, but I think you can probably guess what sort of social minority it is still viewed as acceptable to study in this way.
Coming in with some knowledge of cognitive and situational psychology (one of my big professional interests is human-computer interaction, which means having to know how the human brain works), I’d say the first two questions are still biased. More neutral versions would be:
1. How intelligent are science fiction readers?
2. How good is the writing in science fiction?
(And separate questions about literary readers and prose if the comparison is what you’re after.)
#1 would still probably have a small priming effect, but I’m not sure there’s a way to get around that if that’s really the question you need to ask.
There is a book called “The Power of Survey Design” by Giuseppe Iarossi published by the World Bank which is available free to view on Google Books, and which deals with topics like this. The bibliography can probably lead you in the direction of whatever academic studies of survey design you may need to find, although I’ve been out of academe for too long to be able to advice you more specifically.
As to the issue of comparative effectiveness, I would be surprised if there were any academics who would specifically endorse the idea of using “less neutral” questions in general for determining public opinion. Rather, those who use “less neutral” questions probably either (a) don’t recognize the bias within their questions, (b) are using questions written in the past to measure changes in attitudes over time (see, for example, the General Social Survey at http://www3.norc.org/gss+website/ which has been asking some of the same questions since the 1970s), or (c) are not trying to measure public opinion in an unbiased way.
You are close with b). Basically they’d much rather copy a bad survey done by someone else because a) it means they don’t have to write their own questions and b) all they have to do to justify the questions is point to the previous survey. The fact that they are replicating a useless and actively harmful study doesn’t seem to matter to them.
There’s a lot of academic research on how to construct a good questionnaire/poll/test. They tend to give good examples of “bad” questions – I don’t have the names of these studies to hand, but the husband does and I’ll ask him when he gets home to see if he can put his hands on them by name ;>.
OK – I did a little hunting – here’s some of the better material I found:
Basic primer for writing the Neutral Question Survey: http://www.sciencebuddies.org/science-fair-projects/project_ideas/Soc_survey.shtml
Good examples of Good/Bad Questions: http://www.statpac.com/surveys/question-qualities.htm
More in depth discussion of design for questionnaires: http://www.socsci.uci.edu/ssarc/sshonors/webdocs/designtips.pdf
Academic piece on how to write a good survey/questionnaire (and further reading options): http://www.keene.edu/crc/forms/designingsurveysthatcount.pdf
I hope this is the sort of thing you were looking for – let me know if there’s anything I can clarify.
Thank you!