I am trying to make a survey with the best-worst-scaling method for my thesis where I want people to evaluate 4 different translations.

Not all 4 of the translations are to be evaluated, however, the participants are meant to judge how good/bad the best/worst one is. In other worst, the one they think is the best one is to be rated for how good it is (+1 to +4), while the one the participant thinks is the worse is to be evaluated for how bad it is (-1 to -4).

Conversely, the opposite is equally true. If they rate a translation with, for example, a +3, it is supposed to be the best one and no other of the options should be rated +1 to +4, only negatively. And if they rate one with a negative number, for example, -2, no more negatives are supposed to be chosen. Once 2 out of the 4 translations are rated, no more rating should be possible for that question.

Please explain this to me in as much detail as possible as this is my first time making a survey on any site, and honestly it is a bit overwhelming.

Thanks!



________________+4 +3 +2 +1 0 -1 -2 -3 -4

Translation 1

Translation 2

Translation 3

Translation 4

What if the four translations are all good/bad, just some are better/worse than others.

So I might actually give +4 for the best and +1 for the worst. How do you cover this?

It could be that for the respondent the differences are not that big, but you want to force them to rate negative. Not sure if this is a good idea.

Hi! Thanks for the tip, I will have to discuss this with my mentor, however, I think the terms "good" and "bad" are here to be understood only in relation to the 4 options given. All of the translations could be of very high quality, however, the user has to make a choice of a best and worst one for this type of survey to work.

Apparently, that is how best-worst-scaling is performed in general. It differs from other rating methods exactly because it forces people to choose one option that they think is best and one they think it is worst.

You might think that this type of "forcing" is not really a good idea, however, there are other instances where you limit the choices of the respondents (e. g. making a scale with an even number of options instead of an odd number so that there is no "middle" option).

Basically, good or bad, those are the instructions I have been given:

In this table, the best translation is rated with a number from 0 to +4 and the worst with a number from 0 to -4. Then at the end you will get a meaningful statement / average value for each type of translation. If a translation has already been rated with 0 to +4, it can no longer be selected. If another translation was rated worst, another translation must be rated best. If two are graded, then no further grading is allowed.
 

Apparently, that is how best-worst-scaling is performed in general. It differs from other rating methods exactly because it forces people to choose one option that they think is best and one they think it is worst.

Exactly, but not on a scale. You usually just pick the best and the worst one. You are not qualifying it, how much better, how much worse one is.

Hi,
we always recommend that you send a lss export of this relevant question(s).

So at first you do something like a "MaxDiff" to find the best of the four and the worst of the four
 
This prevents automatically that the same is selected twice (good and bad)

And at the same time you want to rate these translations.

Well, as holch said.
Even the worst could be rated slightly positive.

So you may only rate the four translations with setting "unique" to avoid equal ratings.
This is similar ro a ranking question but with the additional of the differences.

Joffm

So, think it over.
And then send your lss export.

Let's assume you give me 4 translations and in my head I would position them like this:

So they are all fairly good. I would have a hard time to rate any of them with a negative value, because they are all good, but B is worse than C. But still good. It is easy to easy to say which one is the wors and which is the best though.

In your approach, you would force me to give B a negative value. But honestly, really can't. To me, it doesn't make sense to have a Best/Worse approach on a scale like proposed.

But anyway, if this is what you are required to do, it is what you have to do. I just thought I mention this, because I don't feel that this is a good approach.

You might think that this type of "forcing" is not really a good idea, however, there are other instances where you limit the choices of the respondents (e. g. making a scale with an even number of options instead of an odd number so that there is no "middle" option).

Not a fan of forcing people on one side either. When I fill out questionnaires I hate it, when I can't give a more or less "neutral" or "don't know" answer. Yes, you are avoiding those that click everything neutral or don't know. But I rather have those "levelled" out anyway, rather than forcing them to give an answer from which we don't know if it is true or not.

Let's see if someone comes up with an idea on how to technically solve the issue in Limesurvey. I don't really know how to approach this.