More Related Content
Similar to The Do’s and Don’ts Of Survey Design (20)
The Do’s and Don’ts Of Survey Design
- 1. The Do’s and Don’ts of Survey Design Tips, tricks, and techniques for making your survey research more effective
- 2. What do surveys measure? Surveys measure characteristics of people Behaviors Attitudes Beliefs Opinions Demographics Sociographics
- 3. What do surveys measure? Surveys measure characteristics of people Behaviors Actions that are directly observable by others.
- 4. What do surveys measure? Surveys measure attributes of people: Behaviors Actions that are directly observable by others. Demographics Physical characteristics that are directly observable and directly verifiable.
- 5. What do surveys measure? Surveys measure attributes of people: Behaviors Actions that are directly observable by others. Demographics Physical characteristics that are directly observable and directly verifiable. Sociographics Social characteristics that are directly observable but not directly verifiable (i.e., requires confirmation).
- 6. What do surveys measure? Surveys Attitudes, Opinions, Beliefs Surveys measure preferences of people: Wants and needs Surveys measure mindsets of people: measure constructs of people: Satisfaction, Confidence, Persistence, etc.
- 7. Types of survey questions Open response Partially open response Closed response Semantic differential scales Agreement and rating scales Ranking scales Checklists
- 8. Open response What was your entry status when you started your first semester at UNF? ____________________________
- 9. Partially open response What was your entry status when you started your first semester at UNF? 1. 2. 3. RECENT HIGH SCHOOL GRADUATE TRANSFERRED FROM COMMUNITY COLLEGE OR OTHER INSTITUTION OTHER________________________
- 10. Closed If you could do it all over again, would you choose to come to UNF? 1. 2. 3. YES, I WOULD CHOOSE UNF NO, I WOULD CHOOSE ANOTHER UNIVERSITY NO, I WOULD NOT GO TO COLLEGE
- 11. Semantic differential On a scale of 1 to 5, please indicate the quality level of your instructors by checking one of the boxes below: 11 The Best 11 The Worst 11 11 11
- 12. Agreement scale The quality of my instructors is excellent. 1. 2. 3. 4. 5. Strongly Agree Agree Neutral Disagree Strongly Disagree
- 13. Rating scale Overall, how would you rate the quality of your instructors? 1. 2. 3. 4. 5. Very Good Good Fair Poor Very Poor
- 14. Ranking scale Write the letter of each instructor in the numbered spaces below in order of their relative quality: A. B. C. D. E. Instructor Jones Instructor Smith Instructor Brown Instructor Lane Instructor White 1. __________ 2. __________ 3. __________ 4. __________ 5. __________
- 15. Checklist In which activities did you participate? (Check all that apply): STUDENT GOVERNMENT POLITICAL ACTIVITIES INTRAMURAL SPORTS INTERCOLLEGIATE ATHLETICS COMMUNITY SERVICE
- 16. Why people answer surveys Theory of Social Exchange Perceived importance Costs and rewards Established trust Anonymity Peer pressure
- 17. Common wording problems Bias Objectionable questions Unequal comparisons Bias in more than one direction Unbalanced categories Emotionally charged words Threats to self-esteem Personalization
- 18. Common wording problems Objectionable questions How old are you? ___________ (WRONG) Please indicate your age range: (RIGHT) a. 21 and under b. 22-35 c. 36-49 d. 50 and over
- 19. Common wording problems Unequal comparisons Which of the following poses the greater threat to peace in the Middle East? (WRONG) a. Palestinian attacks against Israeli citizens b. Israeli attacks against Palestinian militants c. Both a and b threaten peace Which of the following must cease for there to be peace in the Middle East? (RIGHT) a. Palestinian attacks against Israeli citizens b. Israeli attacks against Palestinian militants c. Both a and b must cease
- 20. Common wording problems Bias in more than one direction Are you against same sex marriage and in favor of a constitutional amendment to ban it? (WRONG) a. Yes b. No c. Unsure What is your view on same sex marriage? (RIGHT) a. I think marriage is a matter of personal choice b. I’m against it but don’t want a constitutional amendment c. I want a constitutional amendment banning it
- 21. Common wording problems Unbalanced categories Please indicate your age range: (WRONG) a. 18-20 b. 21-30 c. 31-50 d. 51 and over Please indicate your age range: (RIGHT) a. 21 and under b. 22-35 c. 36-49 d. 50 and over
- 22. Common wording problems Emotionally charged words Was the FDC negligent by ignoring the warnings about Vioxx during testing and approving it for sale? (WRONG) a. Yes b. No c. Unsure If the FDC knew that Vioxx caused serious side effects during testing, what should it have done? (RIGHT) a. Ban it from ever being sold. b. Require more testing before approving it c. Unsure
- 23. Common wording problems Threats to self-esteem How often have you driven drunk? (WRONG) a. Never b. Once c. Two or more times Have you ever driven a vehicle after you have had too much to drink? (RIGHT) a. Never b. Yes, on one occasion c. Yes, on more than one occasion
- 24. Common wording problems Personalization Dear Mr. Jones: How would you rate the level of service that you received from Salesman Mr. Smith? (WRONG) a. Excellent b. Good c. Fair d. Poor Dear Valued Customer: How would you rate the level of service that you received from your salesperson? (RIGHT) a. Excellent b. Good c. Fair d. Poor
- 25. The problem with scales of agreement Artificial position statements are made to fit these formats. “Agree/Disagree” statements require respondents to take a position “For” or “Against” an issue. There is no clear-cut way to interpret one’s agreement or disagreement with a statement when the proper response choices are unrelated to agreement. Too many items using agreement scales are built upon “face validity” alone. The underlying assumption is that agreeing with an item indicates a person has a specific opinion about that item. “Agreement-disagreement” is not a semantic differential.
- 26. Agree-Disagree is not a semantic differential scale! Agreeing with a statement worded positively may have a different meaning than disagreeing with the same statement worded negatively. True semantic differential items: Hot Weak Very Good Frequently Very Helpful Cold Strong Very Poor Never Not Helpful at all
- 27. 1. There should be designated parking areas for students. SA A N D SD Strongly agree Agree Neutral Disagree Strongly disagree “I shouldn’t have to waste time looking for parking spaces!” What was the student thinking when he chose “Agree?”
- 28. 1. There should be designated parking areas for students. SA A N D SD Strongly agree Agree Neutral Disagree Strongly disagree “I should have a place to park just like faculty have!” not What was the student thinking when he chose “Disagree?”
- 29. You can improve survey items by using a more direct statement of outcome and a more direct scale of measurement.
- 30. Here is a common example taken from the ISQ survey: Professor clearly explains complex concepts and ideas. Strongly Agree Agree Neutral Disagree Strongly Disagree No response And now….a better way to ask this question: How would you describe your professor’s explanations of complex concepts and ideas? 1. 2. 3. 4. 5. Always clear and concise Mostly clear and concise Sometimes clear and concise & other times obscure and vague Mostly obscurely and vague Always obscure and vague
- 31. The problem with scales of satisfaction: To classify data as "satisfaction information," survey items are often “shoehorned” into a Satisfaction-Dissatisfaction format. These items use the same scale format for expediency sake only, and not because the scale fits the meaning of the statement. Satisfaction surveys are good at identifying general problem areas, but do not lend themselves to specific solutions. All of the statements are typically couched in positive terms thus increasing the likelihood of a positive response bias. In a vague attempt to not be overly positive, many items overuse the words “Adequate” and “Reasonable.”
- 32. Keys to making your surveys better and more useful Use a scale covering the whole range of responses. Don’t be afraid of criticism. Make your items balanced. Use a scale or response list that matches the statement Identify a potential problem first, and then use survey data to define its scope and suggest possible solutions. Use a survey only for its intended purpose. Refine your surveys each time you use them. Don’t forget to include free response input.
- 33. The Do’s and Don’ts of Survey Design The End