Introduction

My friend once shared with me a story about his daughter experiencing multiple choice questions for the first time. The story went something like this. Upon encountering multiple choice tests for the first time in first or second grade, she came home bewildered, confused, and a bit incredulous. When greeted with a, “How was school today?” She told her father, “They made us take the stupidest test OF. ALL. TIME. today. All the answers were RIGHT. THERE. We just had to show them which one it was.” He just laughed and tried to explain why multiple choice test items exist - he couldn’t. But upon finishing this chapter, you will be able to succeed where he failed.

Recognized as perhaps the most classic of traditional test items, Multiple Choice test items are used across a wide variety of contexts due to their ability to easily measure various levels of knowledge quickly. Also called “Multiple Guess” questions, these test items are characterized by having two parts: (1) the stem or question, and (2) the choices or responses. The stem consists of the question or incomplete statement at the beginning of the item. This is then followed by possible responses to the question or options to complete the statement. These options consist of one correct option and several distractors. The following sections describe how to make sure you can create quality stems and options for your multiple choice test items.

Affordances and Constraints

Before learning how to best create multiple choice test items, we should discuss when and why we should use them to begin with. Multiple choice test items can be written to assess various levels of learning outcomes, though they are most suited for measuring lower levels of Bloom’s Taxonomy. While multiple choice test items have the ability to measure higher-order thinking skills such as application, analysis, synthesis, and evaluation when used as part of context dependent test items, items which provide some kind of context - chart, diagram, illustration, image, reading, table, etc. - that learners must reference in completing the items, multiple choice test items are primarily used to measure knowledge and understanding. Multiple choice questions cannot be used to measure learning outcomes that rely on creativity, performance, or demonstration of ability. 

Writing a Multiple Choice Test Item

In addition to following the general guidelines from earlier (see Item Creation), writing a good multiple choice test item generally has four steps to follow. These steps should be followed in this order:

1. Identify one idea to be tested by the item.
2. Write the stem or question.
3. Write the correct answer.
4. Write an appropriate number of quality distractors.
5. Review test statsitics and trends.

The first four steps should be completed for each test item that you create. Step 5 should be completed once each test item has been created and after learners have taken your test for the first time. We explore each of these steps in turn.

Step 1: Determine What is Being Tested

The first step is to clearly identify the one idea, concept, or piece of knowledge you want to assess. In general, the stem should not pose more than one problem, although the solution to that problem may require more than one step. Testing over a singular fact helps to ensure clarity and avoid confusion – both of which can invalidate the item’s results. By adhering to this guideline, you provide learners with a clear focus and can gather data that is valid and can reliably be used to guide instruction. Below are both Good and Bad Examples of completing Step 1.

Clear Focus Example

Objective: Identify the verb tense used in a given sentence.

Bad Example: Select the correct verb tense and identify the subject in the following sentence: "She will be studying for her exams all night, while I had already finished mine."

Good Example: Identify the verb tense used in the following sentence: "She will be studying for her exams all night."

In these examples, the problem that needs to be solved is whether learners can identify the verb tense used in a given sentence. The Bad Example creates a stem that has two pieces: (1) selecting the correct verb tense, and (2) identifying the subject. This is bad because it (a) is trying to measure two facets of knowledge and (b) goes beyond the scope of the objective. The Good Example avoids both of these problems, focusing entirely on the objective and giving learners a clear and singular task to demonstrate their ability to meet the objective.

Step 2: Write the Stem

When writing the stem for a multiple choice test item, it is crucial to present a clear and concise question or incomplete statement that provides the necessary context for the options. The stem should be written in a way that avoids nonfunctional words and excessive wordiness. It should also avoid irrelevant material that may make the correct answer too obvious or may make answering the question correctly overly rely on reading comprehension as the determiner of the correct option. Lastly, if the stem is an incomplete statement, it is best for the blank to be at the end of the sentence. Having the blank at the end of the sentence lowers the item’s reliance on reading ability and decreases cognitive load. Below are good and bad examples for both question and incomplete statement stems.

Question Example

Objective: Identify the capital city of a given country.

Bad Example: Which of the following is the capital city of Spain, a city that has a name meaning “water stream”? 

Good Example: What is the capital city of Spain?

Incomplete Statement Example

Objective: Identify the correct usage of a given grammar rule.

Bad Example: It is generally agreed upon by the authors of our textbook, your teacher and William Shakespeare, that you should use the Oxford comma because it _____.

Good Example: The Oxford comma should be used to _____.

The bad example of a question is bad for several reasons. Firstly, it includes irrelevant information about the meaning of the city's name, which is not necessary for the objective of identifying the capital city. Secondly, the stem contains nonfunctional words or phrases, "a city that has a name meaning 'water stream'," that do not contribute to the basis for choosing among the options. Such excessive wordiness can confuse learners and make the stem less clear and focused. To turn the bad example into a good example, we simply remove the extraneous information.

The bad example in the incomplete statement above is also categorized as a bad example due to its wordiness and the inclusion of irrelevant material. The stem starts with a lengthy and unnecessary sentence that is not directly related to the objective of identifying the correct usage of the Oxford comma. It places too much importance on reading comprehension as a determiner of the correct option, which can be a validity concern. The stem should focus on the specific grammar rule being assessed and avoid unnecessary information. Additionally, it attempts to provide a bit of a joke by omitting the Oxford comma in the list of sources who agree the comma is necessary. At best, this joke could provide a hint to the correct answer of the question; and at worst, it could distract learners causing them to lose focus on testing or causing them to rush through later test questions. Either result would invalidate the results from the test item. The good example of an incomplete statement is concise and provides enough context for learners to select the appropriate option to complete the sentence. 

Good stems for multiple choice test items are concise, clear, and focused on the objective without including nonfunctional words, irrelevant material, or excessive wordiness. They provide the necessary context for learners to select the correct option without relying solely on reading comprehension. If a stem is really good, learners should know the answer without having to look at the options. By following these guidelines, stems can effectively assess learners' knowledge and understanding of the targeted concept or objective.

Step 3: Write the Correct Answer

The correct answer should be the option that aligns precisely with the objective being tested. It should be the most accurate and appropriate choice that addresses the specific piece of knowledge being assessed. When writing the correct answer, ensure that it clearly answers the question in the stem or completes the incomplete statement in the stem. If the stem is a question, then each option, including the correct option, should begin with a capital letter. The options should not have ending punctuation unless each option is a complete statement. Such options are generally discouraged, however, as they can increase cognitive load and invalidate the test item data by causing the item to more heavily measure reading ability than content understanding. Conversely, if the stem is an incomplete statement, the options should all be lower case (unless they are proper nouns) and have punctuation after them. If the punctuation is part of the stem, then options do not necessarily need to include punctuation, but if one option includes punctuation then all should. Make sure all of your test items are consistent in their styling. Lastly, correct options should be those that are agreed upon by content-area experts. If there is some level of disagreement concerning “the best answer” then a multiple choice test may not be the best option for measuring understanding of the learning objective.

Science Correct Answer Example

Objective: Identify the correct definition of a given term.

Stem: The process of converting light energy into chemical energy in plants is called _____.

Options:

A) photosynthesis.

B) cellular respiration.

C) water evaporation.

D) soil erosion.

Correct Answer: A) The process of converting light energy into chemical energy in plants is called photosynthesis.

The example above follows best practices for creating stems because it has all of the essential information needed to answer the question, the blank is at the end of the stem, and learners should be able to answer the stem without looking at the options. The correct answer relates to the objective, clearly answers the stem, follows punctuation guidelines, and is agreed upon by content-area experts.

Step 4: Determine the Distractors

Distractors are the incorrect options provided alongside the correct option. They should be plausible and carefully designed to elicit common misconceptions or errors. In order to limit cognitive overload, there should generally be three or four distractors and each distractor should be about the same length, complexity, and grammatical form. Distractors should be appealing to learners who lack a deep understanding of the concept or objective being assessed. Avoid using blatantly incorrect or overly obvious distractors, as they can undermine the validity of the item by lowering the item’s difficulty or causing learners to get distracted and lose their test flow. Using options such as “All of the above” or “None of the above” should also generally be avoided. When these options suddenly appear on a test item, they are more often than not the correct answer. Additionally, they can cause learners to second guess themselves. Remember that a good stem can be answered without looking at the options. If the correct answer is “All of the above” or “None of the above,” it becomes reliant on looking at all options, which unnecessarily increases cognitive load and can therefore invalidate the test item’s data.

English Language Arts Distractor Example

Objective: Identify the correct punctuation mark to end a declarative sentence.

Stem: Which punctuation mark should be used to end a declarative sentence?

Options

A) !

B) ?

C) .

D) ,

This example measures learners’ ability to understand grammatical functions. It relies on learners understanding how different punctuation marks function and what a declarative statement is. The distractors are all the same length  – a single character – and each are punctuation marks that are used to punctuate sentences. It it also important to note that compared to the example in Step 3, the correct answer is in a different location. The correct answer in this example is C.

Step 5: Review Test Statistics and Trends

Once you have created all of your test items, there are two important checks to be made. First, make sure that the levels of Bloom’s taxonomy and that content that you are measuring align with your test plan (see A Guide to Test Planning). Next, make a list of all the correct answers to check whether the location of the correct option is evenly (or nearly evenly) distributed between each location.

After learners have taken a test for the first time, it is important to review the difficulty and discrimination of each item (see Reliability), but you can also do an analysis of response options. An analysis of response options is a comparison of the proportion of learners choosing each response option. You can complete this analysis by dividing the number of learners who selected each option by the total number of learners who answered the test item. This can help you determine if there are some distractors that are not working, or help you to identify whether some distractors are too distracting. In such cases, you should review the learning with learners to understand why the common misconception occurred.

Final Thoughts on Multiple Choice Test Items

Test questions can be likened to lenses that provide a clear view of a learner's knowledge and understanding. Just as a dirty or damaged lens obstructs the view, poorly constructed test questions can hinder our ability to accurately assess what learners have learned. It is crucial to strive for clean lenses in the form of good test questions that effectively measure learners' knowledge and avoid confusion.

A good multiple choice test question is one that allows us to gauge learners' understanding without obscuring their true abilities. If a significant percentage of learners consistently miss a question, or if high-performing learners struggle with it, it indicates a flaw in the question itself. As educators and instructional designers, it is essential to be honest with ourselves about the quality of our test questions and not hesitate to discard those that have caused confusion or misled our learners.

Maintaining credibility with learners lies in our commitment to providing fair and accurate assessments. While discarding the occasional question may be necessary, it is a small price to pay compared to the potential harm caused from standing by misleading or confusing questions. Our primary goal is to support learning, and that includes ensuring that our assessment tools are reliable and reflective of learners’ knowledge and understanding.

By recognizing the importance of clean lenses in the form of well-constructed test questions, we can enhance the validity and reliability of our assessments. Through continuous reflection and improvement, we can create test items that accurately measure what learners have learned, enabling us to provide effective feedback, and to support their educational journey. Let us strive for clarity, fairness, and reliability in our assessment practices, always seeking to refine and enhance the effectiveness of our test questions.