Usually during a final year project or dissertation a university student will need to embark on a new field of study by conducting a literature review and then follow that up with a survey to access new data and insights. Usually the sample is a group that is easily obtained e.g. fellow students, teachers or colleagues from a part-time job (I did this when I worked at Winchester tourist information centre).
However, understandably, experience for students working with surveys is limited and it probably showed in my work back then as it does today on many surveys shared on LinkedIn and Facebook groups. It is a shame especially when so much background research and writing goes into it. Students will either use a survey based platform that offers a free account or one provided by the university. This means routing will be limited (linear) and questions will need extra care and thought to ensure insights can be uncovered. Here are some of the common pitfalls I've seen in some recent student surveys:
Forced answer options - This is the big one! Often student surveys will leave answer options out meaning that the respondent will have to give a false answer to progress or exit the survey entirely. This skews the data and won't be an accurate representation of your sample.
Lack of proof-reading in questions - Often questions are worded incorrectly or are not the most audience or user friendly. They don't make sense, read poorly and the respondent has to guess what the survey is trying to ask of them - all of which is going to limit responses and return poor data.
Wrong question type - Often multi-select questions are used when single-choice question are more appropriate. Sometimes grid questions lend itself to a better experience but are often not selected. Aligning the question text with the question type is so important!
Lack of clarity of what happens to my data - Students need responses quickly and efficiently but one of two things usually happens. Either the respondent is confronted with a long arduous text about the terms and conditions of the research study (which can be very off-putting), that could be put in an alternative link if respondents wish to read it. Or there is no clarity whether the response is confidential, whether the researcher adheres to the MRS code of conduct or what the privacy policy even is. Again confidentiality can encourage respondents to share their views and experiences.
Here are some examples I've found in recent surveys and my thoughts along side them...
To be honest I don't really like at all the wording in the below question, it is a bit wrong! What happens if you have no children; you would have to put yourself down as
a 'Concerned civilian'? What about if you were a parent of a child over the age of 11? In question 2, it asks whether you prefer digital interactive books for your kids? The assumption here is I have children! Also it infers that my children (if I have any) consume 'books' at all. What would I prefer digital interactive books to? Paperback books, Audio books, TV programmes? AI is in mainstream media but what if I didn't know that it stood for Artificial Intelligence. Is it even relevant to the question? Should there be a space for the Maybe answer option. The end result is that the responses given are pretty meaningless and hard to interpret. I understand that in free survey software, routing is limited, so add an instruction here (Much in the same way as if you were doing an offline paper survey) that this question is for parents of children under 12 (or include that in the introduction text).
Here is another example. This survey is about football fan experience and preference of how I watch football matches. The question itself I don't have many issues with; if I was being picky I'd change 'method' to 'way' to make it more audience friendly. The word 'preferred' infers that I should only be able to choose a limited number of options so it could be argued that this should be a single choice question or one where only a few options can
be selected. 'Listening to audio broadcasts' doesn't correspond with 'watching football matches' - again the question wording may need looking at or there may need to be more consideration of answer options on offer. It may be rare but perhaps some fans are unable to watch matches (affordability of tickets, TV subscriptions) so that may need to be taken into consideration too. And also is this ALL football matches or just ones my team is involved with? The background image of the survey is of coral, so I am unsure how that relates to the survey either. PHEW - this was just one question!
Let's continue with the same survey. First of all Q11 below is asking two things in the question text (about your football team and players on social media). What happens if I engage with my team on social media but not any players? A grid type question may be more suitable here so you can answer on both points. Another error is if 'I do not engage with them on social media' is selected, it should exclude other options but it does not - this could be a limitation with the survey software but is unlikely! Again this will be problematic when it comes to survey response cleaning and data analysis.
Let's take a look at a different survey and a common error not just limited to student surveys. Age is an important demographic and can reveal interesting insights when using cross tabs. However, there are a few errors with the question below. First of all we have overlapping categories. If you are 25 (lucky you, if so), are you selecting 18-25 (probably!) or 25-30? This is the issue for individuals aged 30, 35, 40 or 45. The gender question this survey builder used earlier on included a prefer not to say option (good stuff!) but it is needed here too! This can put off respondents from participating! Finally the age bands are okay but are a bit strange. Will there be much of a difference between 40-45 and 45-50? The '35-40' answer option needs some spacing too as does '<18' - consistency is key!
Next example! This survey explores attitudes towards sustainability and luxury in clothing purchasing. Now the instruction leads me to believe I only answer this question if I buy luxury clothes (which I don't as far as I understand the term 'luxury', this could be clarified in the question text). However, when not answering the question and trying to move on the survey alerts me to having to answer the question (make sure this is set to non-mandatory). Unfortunately I don't have a 'I don't buy luxury clothes' option to select either so I either have to give an incorrect answer or drop out the survey entirely. False data or limited data!
Nothing too wrong with this question although there is an assumption on the respondent's part that 0 is not at all important and 5 is very important. It also assumes that I know what sustainability means!
Let's look at another scale question from the same survey. There are a few errors that catch the eye. First of all the incorrect scaling is used in the question text where it should be 0-7 and not 0-5! Also scales should be odd in composition to provide respondents with a middle option; by including 0, this is not achieved! (it has become an 8 point scale question). Again there is an assumption that the respondent understands what 'sustainability' is. There is also an assumption that the respondent knows about these brands! A not applicable option here may be useful. Below I am showing only a few of the options provided, there were about 15 in all. Now with more advanced survey software you could alternate answers shown for respondents (4 per respondent) to ensure a better respondent experience but this may not be applicable with free software. The question text talks about 'level of trust' but then the scale talks about 'level of confidence' - they may be similar but are different things.
Some of these errors aren't limited to student surveys, they happen in bigger organisations too but errors are more likely to be caught if testing processes are in place. Make sure you test your survey with a friend or colleague (see related blogs below). Alternatively I am happy to review and test your survey from as little as £15!
What errors have you seen? Have I missed any?
Comentários