Is there anyone out there who is an expert in designing surveys? The sort of thing you get on sites like SurveyMonkey. I’ve completed quite a few of these as a subject, and they’ve mostly seemed pretty badly designed. Now that I have to create one myself, I’d like to do better, and that means I need to learn.
5 thoughts on “Survey Design – Advice Needed”
Comments are closed.
Surveys work best with tight objectives, and not to many of them. You need a tree – a sort of timeline, of what you are after. So if you want 40 year old males who buy aftershave, you have to work on price first, and then filter out the ones you don’t want – are you male or female – are in in this range of ages – are you employed as… gets you into the rough area. Only after they’ve answered that, have you found your proposed group. Then you move to subject: how many aftershaves do you own – how often do you buy one…
So you need to know WHO you want, and then WHAT you want them to tell you about. Then you construct a tree that does it logically, each question is a filter on the audience who got through on the last filter. Going down smaller. Then you open up to your specific subject.
Hope that makes sense.
I’m no expert in surveys, but as someone who was in QA and technical support for a while, I did learn what you ask and how you ask it is important. You do want to choose and word your questions carefully. HOW you word your question can influence the outcomes and answers. Your own assumptions can influence the outcomes and answers.
As an example, my dayjob is in the food industry. I was larking about on the web a few weeks ago, looking at articles. One was by the high fructose corn syrup industry, and they were touting that people don’t see HFCS as something they look for in labels, or as something they want to have less of in their diets…except, I guess on this survey they did ask if people wanted to reduce sugar in their diet. Now, I’m guessing the HFCS industry WANTED to tilt the results of their survey, and I’m guessing consumers who took their survey and said they wanted to reduce “sugar” in their food counted HFCS as sugar. But probably a small proportion actually called it HFCS. If that’s the case, it’d be easy for this industry to count up the minuscule number of replies actually spelling out HFCS, proclaim consumers were fine with HFCS, and gloss over the fact that your ordinary consumer might actually consider that to be “sugar”, and in their minds included HFCS as “sugar” when they answered, and thus the survey wasn’t designed properly.
So if there’s a term you use that you’re unsure if YOUR definition of it is the same as the survey-takers, it wouldn’t hurt (in my opinion) to define the term. Even if it’s a term “everyone knows”. Example: “For the purposes of this survey, Anne McCaffrey’s Pern and Marion Zimmer Bradley’s Darkover are considered Sci-fi, while blah and blah and blah are considered Fantasy.” (Some ordinary readers classify Dragonriders of Pern of fantasy, as do some hard-nosed fans, whereas others consider it Sci-Fi–since I’m aware of that, if I was making a survey that would be influenced by the answerer’s perception of genre boundaries, I would offer my definition as a baseline.)
Also, I do agree that you need to target the folks you want to take the survey. But, be careful about making assumptions with your filtering/questions. For example, to use the aftershave thing, it makes sense on the surface that you’d target adult men of a certain age range. But it could be you’re not after USERS but BUYERS. And maybe a woman in the household does all the shopping. In which case, you wouldn’t want to throw out results based on gender. Instead, you might want to put in a question like, “Have you bought aftershave for yourself or for others in your household in the past X months?” Which would be more accurate than a gender question if you were looking for buyers of a product, not users. Once you start throwing people out based on gender and age, you’re bringing preconceptions into the survey…that only people of a certain gender or age (or whatever else) might have valid opinions to bring to the survey. On the other hand, if you’re just gathering that data, instead of using it as a filter, that’s different. Perhaps you want to try to plot how age and income level or whatever influence answers.
So in short…know what you’re looking for in a survey, know what demographics. And be careful of question wording and your own assumptions. Define things for the survey-taker if there’s possibility that their understanding of a term does not match your own. Or, ask them how THEY define a term, if that’s what you’re after.
Some quick clarification: the targets for the survey will be published science fiction writers, so the focus is pretty tight.
As people are self-identifying, needs to be really tight. Technically, I’m a published science fiction writer, due to a story in a gamer’s magazine in 1983!
And published is a bit complicated. So you’ll need to differentiate between novels, short stories, bloggers. Trad published or indie. (Or not, if you don’t care.) But I”d put a timescale in. No idea what your end inquiry is, but I’d say ‘Have you been published within the past five years.
ten years
fifteen years etc.
And also say;
magazines
pro-zines
etc.
Even if you want everyone, it’ll help your results to know which spectra they lie on. This is why so many survey monkeys are terrible – not enough distinct division!
Be aware that if you have to ask for demographic information or other things which tap strongly into self-identification first, that will prime people to conform to expectations or norms associated with whatever group they place themselves in. Although depending on the nature of your survey, there may not be any popular stereotypes you have to worry about invoking.