Counterbalancing is an experimental procedure to control for the effect of question order. This post shows you how to do it in Qualtrics without having to create duplicate questions.
The procedure is very simple. It involves creating an embedded data variable in the survey flow, and randomly assigning a value. This variable determines which question order your participant will see. Next, place each question in a separate block. Duplicate the blocks in the survey flow, and then specify which order to show using a branch in the survey flow. I’ll give more detail below.
1. Create an embedded data variable
Create an embedded data variable in the survey flow as shown here. Create as many separate embedded data elements as you have orders you would like to counterbalance. In this example, I have two orders I would like to counterbalance. So, I created two embedded data elements. I called both of the elements “counterbalance,” but I assigned each a different value. I used numerical values here, but it could be something else more descriptive such as a word.
Create a randomizer element and drag the embedded data variables under it, as I have done in the image. Set the randomizer to present only one of the elements, evenly. This is true no matter how many orders you have, as participants will be assigned only one of the orders.
2. Create each question in a separate block
Create each question in a separate block. In this example, only the presentation order of the blocks will vary between participants. Each participant will answer each question.
3. In the survey flow, duplicate the blocks
In the survey flow, duplicate the blocks containing the questions. Do this by clicking the “Duplicate” link in the lower right of the block element in the survey flow. Create as many copies as you have orders. This means if you are counterbalancing three orders, you need three copies of the block element for each question.
4. Create branch elements
Create one branch element in the survey flow for each order you wish to present. You will branch on the values of the embedded data variable you created. Below the branch element, drag the blocks in the order you wish to present to that counterbalance condition.
You are now ready to collect data, and analyze. You can use the variable “counterbalance” in your analysis to check for order effects. Additionally, you will not have to duplicate (i.e. cut and paste) any questions before completing your analysis.
Thank you for this post! Very helpful.
Thanks this is helpful, but won’t the number of permutations escalate as the number of questions increases?
So for two questions there will only be 2 potential orders, but for 3 questions there will be 6 potential orderings (e.g. question 1 then question 2 then question 3, or 132, 213, 231, 312, 321) and for 4 questions there will be 24 potential orderings.
This would mean you need to manually create 24 different orderings and drag and drop them for each ordering.
If you wish to cover all permutations, you may be better off randomizing (https://www.qualtrics.com/support/survey-platform/edit-survey/block-options/question-randomization/). Be sure to export the randomization data. You will have to create a variable that contains a value for each order to use it in your analysis.
Ends up looking something like this:
Thanks Kurt! That’s very helpful 🙂
Hi Kurt,
Would Qualtrics not repeat the same order once a participant had a certain order until complete counterbalancing – all permutations? My understanding is that when you randomize, Qualtrics would simply randomize without any consideration of other orders participants might be assigned to. Not sure how the mechanism would work especially if participants would engage in doing the survey at the same time. I hope you could advise me on this because I am doing this as well and I have 24 permutation across 24 participants (limited participants). Would simply randomizing work or do I have to have the 24 orders like what this post mentioned?