Welcome to the LimeSurvey Community Forum

Ask the community, share ideas, and connect with other LimeSurvey users!

Can I disqualify people when they go too fast with the survey?

  • WinnieWu
  • WinnieWu's Avatar Topic Author
  • Offline
  • New Member
  • New Member
More
4 years 6 months ago #189113 by WinnieWu
I want to get better data qualify of my survey. So I want to disqualify people who are speeding through too fast when doing the survey. Is there any approach to do that in Limesurvey?
The topic has been locked.
  • Joffm
  • Joffm's Avatar
  • Away
  • LimeSurvey Community Team
  • LimeSurvey Community Team
More
4 years 6 months ago #189123 by Joffm
Hi,
Have two hidden questions at the start and near the end where you store the time and calculate the difference.
www.limesurvey.org/forum/can-i-do-this-w...n-is-answered#188876

But: You have the real time used when the respondent submits the survey. So, it's a completed questionnaire.
If you set "save timings" when activating the survey you are able to remove surveys from your analysis which are completed too quickly.

Well, in "nearly each" youtube video about "how to earn money by answering surveys" there are hints how to avoid being screened out. One way is to speed through the survey but at the last question do something else (have a coffee, play a game, feed the baby, ...) and then submit the survey.

Therefore you also have to analyse the timings of each group - which are also stored in the database.

Some more things to consider.
To improve quality (whatever it means) you also should analyse the data (mainly in arrays) for "left justifier", "right justifier", "mid justifier", "happy clicker" and "DKs".

There are ways to do this within LimeSurvey by defining the conditions and calculating a score for each respondent in one of the last questions (ajax call, API).
But in my opinion it's better to do this in your analizing tool. Here you have more choices of statistical tests to decide whether the answer is really suspicious or not.

One more thing: Panel providers don't like it if respondents are screened out at the end of a survey. Some have to pay them as well if they completed more than two thirds.

Joffm

Volunteers are not paid.
Not because they are worthless, but because they are priceless
The topic has been locked.
  • holch
  • holch's Avatar
  • Offline
  • LimeSurvey Community Team
  • LimeSurvey Community Team
More
4 years 6 months ago #189143 by holch
I agree with Joffm. This quality check is usually done with completed responses. I don't think it is viable to do this during the survey itself. This is why you usually get a couple of more responses than you actually need for your sample.

And I also agree on the part that those that do respond with only the "money" in mind will always find a way to trick you. Most of them know how to identify screening questions, they know the dos and don'ts of survey taking, probably a lot better than those that just honestly try to respond, but do not know that certain behaviors could be flagged as problematic.

Even the mentioned "left justifier", "right justifier", "mid justifier", "happy clicker" (not sure what that exactly is) and "DKs" can be totally valid responses. So it is dangerous to just draw conclusions on one of these factors.

E.g. there are certain cultures that give generally more positive answers. Now imagine a survey among Apple fans about the company. On a 1-5 scale, you'll see a lot of 5s (very positive), some might just rate 5 for all. Should we exclude these answers? Probably not.

Same thing: You just got a flight cancelled, got stuck somewhere at the end of the world, the Airline is not giving you solutions, you are frustrated. And there comes the automatic customer satisfaction survey via email. You'll see a lot of 1s coming. If it is at the wrong time you might see all 1s. Should we exclude this response? Probably also not.

Even mid justifiers. I often find myself answering in the middle, because I am quite indifferent about the things ask. I don't see them positive or negative. So depending on the questions, I might answer a lot of things in the middle.

When I check questionnaires before they go online and need to click through the questionnaire a lot of times to find all the errors, run into all filters and screeners, etc. on big array/matrix questions I usually start on the left and for each subquestion go towards the right and once I got there, I go back towards the left step by step. So basically zickzacking my way through the question. This is a lot more difficult to identify automatically than just the left, mid or right clickers.

I think the best way to find out if people are cheating is to include some trick questions. But very expert survey tackers will notice them. Also a good thing is to have a look at the answers of open ended questions.

All in all it is not easy to find those cheaters. The overall time is a good indicator, but as Joffm said, someone who wants to trick you knows how to do it, so you need to be very cautious and simple KPIs mostly don't do the trick anymore. You'll have to look into various factors.

Also: If you use a bad panel and the rate of fraud is high, this will also have an impact on the mean/median time spent, etc.

I answer at the LimeSurvey forum in my spare time, I'm not a LimeSurvey GmbH employee.
No support via private message.

The topic has been locked.

Lime-years ahead

Online-surveys for every purse and purpose