Skip to main content
A ranking question asks participants to put a short list of items in order, most preferred first. Optional follow-up probes then ask why they placed one item first and another last. You get each participant’s own order and the reasons for both ends of it from the same interview.

​
Key concepts

  • Item: one of the things being ranked, with a text label.
  • Rank position: where a participant placed an item. Position 1 is the top of the list, the most preferred item.
  • Anchor labels: optional captions shown above and below the list, such as “Matters most” and “Matters least”.
  • Mean rank: an item’s average position across rankings. A lower mean rank means the item was placed closer to the top.

​
When to use ranking or MaxDiff

How many items you have decides which question to use.
  • 2 to 8 items: ranking. Ranking is a short, familiar task, and putting a short list in full order is a real judgement for participants.
  • More than 8 items: MaxDiff. On a longer list, the middle of a full ranking stops being a considered choice and becomes noise. MaxDiff handles 15 to 30 items because it never asks for a full order.
Ranking works best when the ends of the order are what you want to learn about. For example, ranking a handful of shortlisted benefits, or ordering steps of a process by how much friction each causes.

​
Adding a ranking question

  1. Add a Ranking question to the topic guide and write the question. Ranking is listed with a Beta badge in the question type menu.
  2. Under Item library, use Add item to add between 2 and 8 items. You can rename, delete and reorder items. Items are text only; images are not supported.
  3. Optionally fill in the Anchor labels for the top and bottom of the list.
  4. Choose the follow-up setting under Probing:
    • Probe top & bottom (the default): after the ranking, the interviewer asks one follow-up question about the item placed first and one about the item placed last.
    • None: the interview moves on straight after the ranking.
  5. Test the interview on a phone and on a computer.
A ranking question can’t launch with fewer than 2 or more than 8 items with text. Blank items don’t count. Attach any stimulus participants need before they rank. Ranking items people haven’t seen measures how appealing the labels sound, not preference.

​
What participants see

  • The ranking starts empty. All items are listed below it in an order shuffled for each participant.
  • Participants tap an item, or drag it, to add it to their ranking. Tapping adds it to the end.
  • They can drag ranked items to reorder them, or remove an item to send it back to the list.
  • Confirm my ranking stays disabled until every item is ranked. Each item is placed exactly once; ties and partial rankings are not possible, and there is no option to skip the question.
  • After confirming, participants continue to the next step. Changing the order after confirming withdraws the confirmation, so they confirm again.
Every action works by tapping or with a keyboard, not only by dragging.

​
Why item order is shuffled

People tend to rank the first items they see higher. Conveo shuffles the list for each participant and records where each item appeared on screen. The analysis uses this to check whether screen position influenced the results (see Caveats shown with the results).

​
Analyzing ranking results

Open the question in Question Analysis.
  • Distribution chart: one bar per item, split by the share of rankings that placed it at each position. This is the most complete view. A polarizing item that half the sample ranks first and half ranks last looks average on mean rank alone, but stands out here.
  • Table: each item’s Mean rank, its 95% interval, and the share of rankings that put it first (Ranked #1). Items are sorted from the lowest (best) mean rank.
  • Summary: the question summary is based on the calculated figures, such as mean rank and first-choice share.
Each participant’s ordering also appears in their interview transcript.

​
Caveats shown with the results

  • Low sample. When fewer than 30 people completed the ranking, the card warns that the order of the items is not reliable. Read it as a signal to explore, not a result.
  • Presentation bias. If items shown earlier on screen were ranked higher, and the effect is too large to be chance for the number of rankings, the card shows a warning with the correlation. This suggests participants ranked down the screen rather than by preference. Treat the order with caution.
  • Incomplete rankings. Rankings that don’t cover every item are left out, and the card says how many.
Focus on the ends of the order. On a short list, the top and bottom items carry the signal; differences in the middle are rarely a finding. Compare the 95% intervals before calling one item ahead of another. In a diary study, each entry’s ranking is counted. The card shows the number of rankings and the number of people separately, and the low-sample warning is based on people.

​
Talk to Your Data

Talk to Your Data can answer questions about ranking results. It reports mean ranks, the share of rankings that put each item first and last, and how many rankings placed each item at each position, together with the low-sample and presentation-bias caveats. To compare groups, ask about each group separately.

​
Exports

In a custom export, choose Ranking (per item) to add one column per item. Each column holds the position that participant gave the item, where 1 is the top choice. The printable topic guide lists each ranking question’s items under its anchor labels.

​
Limitations

  • 2 to 8 items. Use MaxDiff for longer lists.
  • Full ranking only. No ties, no “rank your top 3”, and no skip.
  • Text items only. Items can’t have images.
  • Not in screeners. Ranking isn’t available as a screener question.
  • Not with randomized stimuli. A ranking question can’t be used on a question, or in a section, that has randomized stimuli attached, because ranking results can’t yet tell the repeated answers apart. Ask the ranking once, after the stimuli.
  • Not a routing condition. You can route to or past a ranking question, but a routing rule can’t be based on its answer. See question routing.