Survey Building 101: Six Common Mistakes & How To Avoid Them
The advent of free on-line survey tools has unleashed a powerful tool for many professionals. Once the cloaked domain of the statistically minded, surveys are now being used for an increasing number of different efforts. Unfortunately, many professionals have never designed a survey. The user experience drives little traction, or worse yet, the resulting data is not reliable. Here are some good rules of thumb to keep in mind before publishing your next survey.
Mistake #1 - Expecting Buzz Word Compliance Hopefully the audience you have selected for your survey is relevant to the research you want to conduct. But that does not mean they will interpret industry buzz words the same way you intend them. I once was asked to complete a survey about generational differences in communication styles. As a mother and marketer I was well suited to be a survey participant. Unfortunately, the survey labeled each generation without explanation. Honestly, I can't remember if 18-24 year olds are millennials or generation X and as a result my responses became invalid. This is easy to fix -- if you're not sure if a term is industry specific ask a neighbor to proof read it! Err on the side of over-explaining. The worst that can happen is your participants feel over educated.
Mistake #2 - Duh! I forgot to define my rating scale Just because you ask someone to rank their results from 1 to 5 does not mean they will know you mean 1 is best and 5 is worst! If you are providing a scale be clear about what it means.
Mistake #3 - The Arnold Schwarzenegger Effect: Total Recall This is probably the most common mistake I see in surveys. People offer instructions for answering a question and even post a rating scale, but they do it only once. By the 4th question I forgot what instructions I'm supposed to be following! Don't expect total recall. A little repetition in a survey is smart.
Mistake #4 - You what? Please complete this short survey on governance policies for data retention does not imply a fifty question survey on everything from data retention to fines for non-compliance. And yet, the mismatch is more common than you might imagine. Instead, tell your participants up front what to expect from the survey so they can dedicate the right amount of time and energy to your fact gathering. I always advocated being very specific. Phrases like the below are helpful for setting expectations:
This survey should take no longer than 20 minutes to complete
Please help us by responding to this 10 question survey
We value your time. We will ask you to complete 3 questions before entering the survey to make sure your background is a perfect fit for our research needs. As a thank you for taking the time to match needs with us we want to offer you...
Mistake #5 - All I got was the lousy t-shirt? Have you ever seen those tourist t-shirts: My Grandma toured Italy and all I got was this lousy t-shirt? People's time is incredibly valuable. If you are going to offer a thank you for participation I implore you to match incentives to the level of effort required to complete the survey. A $5 charitable donation, or a copy of the research findings might be a perfect match for a survey that takes 10 minutes to complete. But before offering an incentive consider how much time and energy is required by the participant and if you have earned the right to ask for that time.
Mistake #6 - Who are you again? Sometimes blind surveys are acceptable. However, in most cases the data is most valuable when you know something about the person completing the survey. Make sure to validate relevant characteristics up front.Happy Surveying! With the right planning you can gain incredible insight through this extremely affordable mechanism.