Multiple choice questions are a great way to test a candidate's knowledge and skills for a subject that removes the need for manual or machine learning grading.
To ensure your multiple choice questions are quality and each answer option adds value to the question, review your multiple choice response insights table, under your Assessment Insights tab.
This table displays the number of times response options are selected for single and multiple-choice questions. By visualising the distribution of selected responses, it aims to provide insights into the quality of the content.
For multiple-choice questions (MCQs) to be considered valid and effectively measure a candidate's knowledge, the distribution of responses should reflect a balanced level of difficulty and be informative in terms of test design. Here’s a guideline on what an ideal distribution looks like and how to interpret the results:
Ideal Response Distribution for Multiple Choice Questions
Correct Answer Selection:
- Ideally, about 60-80% of candidates should select the correct answer. This suggests the question is at an appropriate level of difficulty for the group being tested — not too easy and not too hard.
- If 90-100% select the correct answer: The question is likely too easy. It’s testing knowledge that is too basic or well-known to the candidates, making it ineffective for differentiating skill levels.
- If 0-30% select the correct answer: The question may be too difficult or poorly worded, leading to confusion. It could also be testing niche knowledge that only a few candidates are expected to know.
Distractor (Incorrect) Answer Selection:
Distractors (incorrect answers) should ideally be chosen by 10-40% of candidates, with no single distractor being chosen disproportionately.
- If a large proportion (e.g., 40-70%) selects a wrong answer: This could indicate that the distractor is misleading or too similar to the correct answer, which might suggest the question needs to be rephrased to better distinguish between answers.
- If one distractor is selected by a large group (and it’s incorrect): This could indicate that the distractor is overly tempting or poorly worded, creating confusion. Alternatively, it could indicate that the question itself is ambiguous or unclear.
Balanced Distractor Performance:
Ideally, all distractors should have roughly equal likelihood of being selected. A good MCQ has credible distractors that are closely related to the correct answer.
If one distractor is chosen by more than 40% of respondents and it is incorrect, this suggests that either:
- The question is confusing.
- The distractor has elements that make it look like a reasonable answer (e.g., using similar terminology or concepts).
- The question’s difficulty might need to be adjusted.
Interpreting MCQ Performance:
Discriminatory Power:
If most candidates select the correct answer, the question does not discriminate well between candidates who understand the material and those who don’t. This means it might not be an effective question for differentiating skill levels.
Distractor Analysis:
If candidates are selecting a wrong answer frequently, consider if that answer is plausible or if the wording of the question could lead to misinterpretation. If one of the distractors has a significantly high selection rate, it may indicate that the wrong answer is somehow tricking test-takers.
Re-evaluating Questions:
If you notice that most candidates choose the wrong answer, the issue could be:
- The question is ambiguous or unclear.
- The wording of the distractors is too similar to the correct answer.
- The question is too difficult for the group being tested.
In Summary:
- 60-80% correct responses: Ideal distribution for a well-balanced question.
- 10-40% selecting a wrong answer: Indicates the distractors are reasonable but not misleading.
- Equal spread across options: Distractors should be equally plausible and distinct from the correct answer.
Please Note:
If you have a low number of candidates in an assessment, the distribution of scores may not be accurate. For example, if only two candidates complete the assessment and both select the correct answer, the result may show a 100% correct rate. However, this doesn’t necessarily mean the question is too easy — the sample size is just too small to draw conclusions.
A larger group of responses helps ensure a more reliable analysis of how well the question is performing. With more candidates, you can better assess the difficulty of the question and its ability to differentiate between different levels of knowledge.