I have a survey that was completed by more than 1000 people.

The layout is done group after group. In fact, some groups have just one question, so the group time here is in fact question time.

Main problem: recorded group times exceed the limits of how long this group is to be displayed.

To put it in the context. I have a group in which a video is displayed. Participants are asked to watch the whole video (231 seconds long) and then proceed to answer 7 questions (each question as a separate group) based on the video.

• video group has a question with a timer set to 250 seconds
• each question group that follows has a timer set to 40 seconds

The participants can skip both the video and the questions, that is why I am checking their times on each group.

1. Video group time should be between 231 (= person watched the full video) and 260 seconds (given slow internet connection, we give a maximum of 10 seconds for the timer object to load, which is already counted as group time for what I understand).
2. For the questions it is the same, each question time must be between 5 (time to read a question) to 45 seconds (we give maximum 5 seconds for the object to load).

The problem I encountered is with times exceeding the said limit. After analyzing the times of each question I realized that times that exceed the limit are somehow multiplications of time taken to answer. For example, the majority of times recorded for questions are truly between 5 and 45 seconds, but then there are some of them concentrated around 85 seconds.

I attach the file with plots of times and outliers (in terms of time limits) - outliers.png and another one but without some huge outliers (more than 3 times the upper time limit).

• FILM is group with video
• TEKST is group with text
• FILM.P1 is group with first question after video and so on

There you also have TEKST.Time (for the group with text to read) and it's questions. Time limit for text was 240 so we set it to 250 in the analysis. You don't need to concentrate on the lower outliers just the top ones. As you can see the problem exists, more or less, for each question. I hoped that this occurs for all times of some individuals, but no, it happens for some people, and if it does happen, it is for some of the times, sometimes for one time, sometimes for 6 or more. I was unable to correlate this behaviour with the device the respondents were using (they specify it in the very beginning of the survey).

You could advise me to drop those people for whom the time exceeds the upper limit for at least one group time. In that case, I would lose around 350 respondents, what's more, this is not just a "badly saved time", the times, when exceeding the upper limit of 45 seconds, are falling in certain boundaries, not some random values.


Do you know of any bugs related to this or how I can deal with this problem?

Thank you in advance!

Mateusz

Matiasko wrote: Do you know of any bugs related to this or how I can deal with this problem?

What version of LimeSurvey was used to conduct the survey? Can you provide a LSS export? Can you try to reproduce the error on your survey? Try your survey with different browsers.

Depending on the used version of LimeSurvey there might be some bugs already reported.

Dear jelo,

Version is 3.13.2+180709.

I attach the lss file. But let me explain how this study works.

1. We randomize respondent to group 1 or 2 (R1 in INFO1)
2. If group 1: information preparing for video is displayed (I1T1 in INFO1)
3. If group 1: respondent watches video (TREAT1a) -> answers 7 questions (T1P1 to T1P8, excluding number 4, all questions have suffix "a") about video (s)he just watched
4. If group 1: information preparing for text is displayed (I2T2 in T2INFO1)
5. If group 1: respondent reads text (TREAT2c) -> answers 7 questions (T1P1 to T1P8, excluding number 4, all questions have suffix "d") about text (s)he just read

If group 2 then it is reversed: information about text -> text -> questions about text (suffx "b") -> information about video -> questions about video (suffix "c").

Rest is the same for all respondents.

I am not able to reproduce this error as I don't know where to start with it. As I said it is a random-like behaviour. In my opinion, if it was browser-based, it would happen for all the times, not only some random number of them. It doesn't happen just for certain questions as well.

Thanks for the infos.

The options "Time limit disable next:" and "Time limit disable prev:" weren't used in the actual survey? Or is that just the case in the export?

Respondents would be able to go back and forth during watching the stimulus.
Might be a issue for correct counting. But that is currently a guess.

Oh, there is such an option. Tbh I feel like there was no option like that when I was setting options of those questions. Nonetheless, I wouldn't use it. First video and text limits were set just so respondents couldn't watch/read it again. If they were restricted to go further after reading the text in 200 seconds, they would probably abandon the survey. Same with the questions about video/text, if someone replies in 7 seconds, there is no need to permit them from going further. Of course, we wanted to analyze the times to correct/wrong answer. In addition we wanted to filter for the movie_time > 231 and text_time > (let's say) 50.

What's important here: they could go further if they want, without reading the whole text, without watching the whole movie, without answering the questions. We assumed it will be easy to filter those respondents based on group times. BUT they couldn't go back (it would be ridiculous) and it was set in the global options.