General Guidelines
1. What is a questionnaire?
• An instrument (form) to
– collect answers to questions
– collect factual data
• Gathers information or measures
• A series of written questions/items in a fixed, rational order
2. Why using a questionnaire?
A well designed questionnaire:
• Gives accurate and relevant information to your research question
• Minimises potential sources of bias
• Will more likely be completed
3. Advantages of questionnaires
• Can reach a large number of people relatively easily and economically
• Provide quantifiable answers
• Relatively easy to analyse
4. Disadvantages of questionnaires
• Provides only limited insight into problem
– Limited response allowed by questions
– Maybe not the right questions are asked
• Varying response
– Misunderstanding/misinterpretation
• Need to get it right first time
– Hard to chase after missing data
5. Types of Questionnaire
· Face to face
· Telephone
- Interviewer administered
· By post
· e-mail/ internet
- Self administered
6. What makes a well designed questionnaire?
• Good appearance → easy on the eye
• Short and simple
• Relevant and logical
Ø High response rate
Ø Easy data summarisation and analysis
7. Basic Rules
• On first page
- Return address
- Study title in bold
• On all pages
- Identifying mark/unique identifier
- Numbered items
- Page numbers
• Directions in bold
• Self-addressed envelope!!
8. Questionnaire Introduction
- Covering letter/ interview introduction
- Who you are/ you work for
- Why you are investigating
- Where you obtained the respondent’s name
- How and where you can be contacted
- Guarantee of confidentiality
- Length of interview (be honest)
⇒ Uusefulness of study should be clear to all respondents
9. Questionnaire Design Steps
- Decide what information is needed
- Search for existing questions
- Focus groups
- Draft new questions/revise existing ones
- Sequence the questions
- Get peer evaluation
- Revise and test on self/co-workers
- Think-aloud interviews
- Revise/eliminate questions
- Prepare interviewer instructions for pilot test
- Pilot test (10-20 cases)
- Revise eliminate questions based on respondent & interviewer comments
- Pilot test again, if necessary
- Prepare final interviewer instructions
- Be prepared to modify questionnaires if interviewer training raises problems
- After interviewing is complete, debrief interviewers for potential problems
- Use experience from one study for future planning
10. How to draft and organize your questionnaire:
Ordering the Questions
• Decide on order of items/questions
– Easy ® difficult
– General ® particular
– Factual ® abstract
• Where to place sensitive questions?
• Group questions by topic/ response options
• Starting questions
– Simple
– With closed format
– Relevant to main subject
– Non-offending
– Neither demographic nor personal questions
• Don’t put most important item last
What Is A Good Question?
§ One that yields a truthful, accurate answer
§ One that asks for one answer on one dimension
§ One that accommodates all possible contingencies of response
§ One that uses specific, simple language
§ One that has mutually exclusive response options
§ One that produces variability in response
§ One that minimizes social desirability
§ One that is pretested
Language of Questions
a) Ask one question at a time
For example:
Bad question:
In the past 6 months, what major appliances has your household purchased new from the store?
Better question:
Now I’m going to read a list of household appliances. As I read each one, please tell me whether or not your household has purchased this type of appliance new from the store during the past 6 months. How about…
a refrigerator?
a kitchen range or oven?
a microwave?
Bad Question:
Compared to one year ago, are you paying more, less, or about the same for your auto and life insurance?
Better Question:
Compared to one year ago, are you now paying more, less, or about the same for …
a. auto insurance?
b. life insurance?
b) Specify
à Specify who, what, when, where and how.
For example, whose income? What’s included? Over what period of time? Example:
“In 2002, what was your total household income, before taxes? Please count income from all members of your household, including wages from employment, disability, social security, and public aid.”
c) Specify through cues
Example
People drink beer in many places – for example, at home, at restaurants, at bars, sporting events, at friends’ homes, etc. During the past 30 days, did you drink any beer?
d) Use words with singular meanings
Ambiguous:
How would you compare how close you are to family in your current neighborhood to how close you were in your old neighborhood? Would you say your family is closer here, further here, or the same?
More Clear:
Compared to your last neighborhood, do you now live closer to your family, are you further from your family, or are you about the same distance?
e) Be appropriate
Are you a drunk?
- Yes
- No
Vs.
How often have you consumed alcoholic beverages during the past 6 months?
- Daily
- 2-6 times/week
- Once a week
- Less than once a week
- Don´t know
f) Be simple
Did you smoke not less than a mean amount
of 7 cigarettes/2 days from 1999 onwards?
· Yes
· No
Vs.
Did you smoke an average of 2 pack of cigarettes/week for the last 5 years?
· Yes
· No
· Don´t know
g) Be accurate
Do you often touch dogs?
· Yes
· No
Vs.
How often did you touch a dog during the
past 3 months?
· Once
· Twice
· Three times or more
· Not at all
· Don’t know
h) Question Placement
· It's a good idea to put difficult, embarrassing or threatening questions towards the end of the interview when the interviewee has gotten more comfortable. This has two benefits. First, it makes them more likely to answer, and, second, if they get mad and leave, at least you've gotten most of your questions asked!
· Put related questions together to avoid giving the impression of lack of meticulousness
i) Filtering "Don't Know"
There are three ways of dealing with "don't know". Standard format. No "don't know" option is presented to the respondent, but is recorded if the respondent volunteers it. Quasi filter. A "don't know" option is included among the possible responses. Full filter. First the respondent is asked if they have an opinion. Then, if yes, they ask the question.
j) Social Desirability
Ø Respondents will try to represent themselves to the interviewer in a way that reflects positively on them
Ø As questions become more threatening, respondents are more likely to overstate or understate behavior, even when the best question wording is used.
Types of Questions
è Two main question formats:
• Closed format ® forced choice
Yes Always
No Sometimes
Don’t know Never
• Open format ® free text
What is your most distressing symptom? Please describe: ____________________________________________________________________________
Closed Questions
1. Straightforward response
What is your age in years? ___ years
How long have you owned a dog? ___ years
What is your sex (gender)?
Male
Female
Did you stay in Hotel X on 25/11/08?
Yes
No
Don’t know
2. Checklist
Which of the following outdoor activities did you do last week?
• Running
• Walking
• Hiking
• Cycling
• Swimming
3. Rating scale
Did you do use sunscreen during the following outdoor activities during the past six months?
Always Sometimes Seldomly Never
Running
Walking
Cycling
4. Rating scale Numerical
How useful would you think that information on the risk of biting from stray dogs would be?
(Please circle)
1 2 3 4 5 6 7
Not at all useful Very useful
5. Scales for measuring attitude (Likert)
Stray dogs carry a higher risk of rabies
No, I strongly disagree
No, I disagree quite a lot
No, I disagree just a little
I’m not sure about this
Yes, I agree just a little
Yes, I agree quite a lot
Yes, I strongly agree
------------------------------------------------------------------------------------------
|
| Advantages | Disadvantages |
| Closed-ended | · Easy and quick to answer · Answers across resps easy to compare · Answers easier to analyze on computer · Response choices make question clearer · Easy to replicate study | · Can put ideas in resp's head · Resps w/ no opinion answer anyway · Resps can feel constrained/frustrated · Many choices can be confusing · Can't tell if resp. misinterpreted the question · Fine distinctions may be lost · Clerical mistakes easy to make · Force respondents into simple responses |
| Open-ended | · Permit unlimited number of answers · Resps can qualify and clarify responses · Can find the unanticipated · Reveal resps thinking processes | · Resps give answers w/ diff. level of detail · Answers can be irrelevant · Inarticulate or forgetful resps are at disadvantage · Coding responses is subjective and tedious · Requires more resp. time and effort · Intimidates respondents · When resp omits a response, can't tell if its because |
Contingency Questions
These are questions of the type:
"If you have done this…. Go to question number such and such…".
In this case you are telling your respondents to skip some questions that do not apply to them and go to another section.
Example:
Was your child breastfed for the first three months after birth?
Yes [ ] No [ ]
If NO please go to questions 25-30 on page 4.
12. Problems and Pitfalls
• Avoid questions that ask two things at once - you won’t know which ‘bit’ people are answering:
Have you ever had stomach ache and diarrhoea?
• Ambiguity.....
Do you go to the woods a lot?
• Avoid jargon/abbreviations/slang
How often do you get up at night to PU? (pass urine)
Should IVDUs be treated in the community?
• Avoid not mutually exclusive options
What age are you?
16-20
20-25
25-30
35-40
• Avoid leading questions
Do you think that the food in the hotel made you sick?
Did the hotel staff seem unhygienic to you?
Do you agree that the hospital staff were close to exhaustion?
• Avoid emotional language, prestige bias and leading questions
o "What should be done about murderous terrorists who threaten the freedom of good citizens and the safety of our children?"
o "Most doctors say that cigarette smoke causes lung disease for those near a smoker. Do you agree"?
o Avoid leading questions like "You don't smoke, do you?" or "I assume you would agree that the teachers do a heroic job for our children".
o Avoid loading questions with extra adjectives and adverbs, like "Should the mayor spend even more tax money trying to keep the streets in top shape?"
• Avoid asking questions beyond a respondent's capabilities
o People have cognitive limitations, especially when it comes to memory of past events. Asking "how did you feel about your brother when you were six years old" is probably useless.
o It is pointless to ask people about things that are not natural ways for them to think. For example, don't bother asking "How many gallons of gasoline did you buy for your car last year?”
• Avoid false premises
Asking "What is the most important thing we should do stop the economy from deteriorating any further?" assumes that the economy is deteriorating, which the respondent may not agree with. This puts the respondent in a tough spot. It would be better to rephrase as "What is the most important thing a government can do to strengthen its economy".
• Avoid asking about future intentions (if you can) Hypothetical questions like "If a new grocery store were to open down the street, would you shop there?" are notoriously unrelated to actual future behavior.
• Avoid negatives and especially double negatives
o Negatives like "Students should not be required to take a comprehensive exam to graduate" are often difficult for many respondents to process, especially if they agree with the predicate, because then they are disagreeing with not doing something, which is confusing!
o Double negatives like "It is not a good idea to not turn in homework on time" yield very unreliable data because people are unsure about whether to put a "yes" or "no" even if it is clear in their minds whether turning homework in on time is a good idea.
• Typographical / spelling errors
Format of Questions
13. Physical Format of the Self-Administered Questionnaire
• Careful formatting is necessary to decrease errors and increase motivation
• Respondent’s needs must always take priority, followed by interviewer and data processors
• The questionnaire should be uncluttered and spread out. Do not skimp on paper. It is better to have a clearly structured questionnaire on several pages than a tightly packed one which is difficult to read.
• Chose a clear format for the respondents to record their answers.
Examples: · Tick boxes - Usually when there is a choice of possible answers you should provide either brackets [ ] or parentheses ( ) for respondents to record their answers. · Circle items - Alternatively you can provide the answers already precoded so that respondents only have to circle the correct answer. Pre-coded answers have the advantage that you can use those codes later in the data-processing stage.
14. Physical Format Checklist
• Number all questions sequentially
• Use large, clear type; don’t crowd
• ‘White space:’ Place more blank space between questions than between subcomponents of questions
• List answer categories vertically instead of horizontally
• Avoid double/triple ‘banking’ of response choices
• Be consistent with direction of response categories
• Be consistent with placement of response categories
• Don’t split questions across pages. If necessary (e.g., question requires 1.5 pages), restate question and response categories on next page
• Put special instructions on questionnaire as needed, next to question
• Distinguish directions from questions
• Precode the questionnaire (vs. check boxes)
