There are many different types of response scales; multiple-choice options are common but Likert scales are used quite often as well. Each method used to obtain a response has limitations and strengths. They also have specific design principles that should be considered.
Multiple Choice response.
In this situation, the item stem is phrased as a question followed by a list of potential answers. Each response option must be mutually exclusive or clearly different from the others. There are a few specific design decisions you need to consider when using the multiple choice response method.
Example of a Multiple Choice responseWhich of the following situations best describes your current relationship status?
- Legal domestic partnership or civil union.
- Single, but cohabitating.
- Single, never married.
- I am unwilling to share this information.
Best answer or Choose all that apply. Normally a respondent is expected to choose the answer that best represents their situation, believe, or condition. However, if the situation warrants, respondents may be asked to select all that apply to them personally. Only use the “all that apply” instruction if you have a good reason for doing so.
In the relationship status example, selecting the best answer is most appropriate. Respondents should currently be in one and only one relationship. The options are mutually exclusive and expected to describe every relevant situation.
Opting Out. Obviously, we want respondents to answer every item on the survey. Unfortunately, ethical considerations and political sensibilities may require you to allow individuals to choose not to answer a specific question. In reality, you can’t force individuals to answer any question whether you provide an opt out option or not. If an individual selects the opt our option or simply chooses not to answer the item, the result is the same. Most research protocols involving the protection of human subjects will require you to allow respondents to opt out at any time. However, failing to answer a key question may equate to refusing to take the survey, even if a respondent answers all the other questions. If the item was to be used as a disaggregation or branching variable, you should require respondent to answer the question. Noting that respondents should still be allowed to opt out if they choose; you just might not be able to use any of their data if they don’t answer key questions.
Consider for example the relationship status item. This item most likely would be included in a survey to provide a basic description of respondent demographics. If this were the case, an individual’s failure to respond may not be a problem. Providing respondents with an opt out option for this item protects the individual’s rights. However, if the results were intended to be disaggregated based on the relationship status, failure to answer the item would make the respondents data unusable. This would be even more problematic if large numbers failed to answer or the response refusal had a systematic pattern. Similarly, if the information was to be used as a branching variable (so you could ask group specific items), failure to capture this information would render this design feature inoperable.
Number of options (option specificity). In most cases, when using a multiple choice response method you want to provide a list of all reasonable responses, but not always. In the relationship status example, the number of possible responses will depend on the various mutually exclusive relationship situations possible. However, the degree to which specific details are required should be determined by the research purposes. You may only need to know if the person is single (never married) or something else. If this were the case, and you had no plans to use the more detailed information, you may only need two response options. In a multiple-response response scale, the researcher attempts to make the list of options exhaustive, otherwise you may need to allow “other” responses. However, If the list gets too long there is the potential for fatigue and option order affect bias.
Allowing OTHER responses. When using the multiple-choice response method, the use of an option allowing respondents to write in another response has its advantages but also has limitations when it comes to, reporting, interpretation, and drawing conclusions. The “other” option is used when the research is unsure whether or not they have an exhaustive list of possible answers. It should be noted that obtaining a comprehensive set of responses should have been obtained in the pilot testing phase of the instrument which would reduce the need for allowing other responses.
To create a comprehensive list of possible answers, you might simple create a list logically. For example, in the relationship status example it is very likely that you could logically determine each possible situation. However, this is not always the case. Consider a situation where you were looking for respondents’ reasons for doing something. In the pilot testing phase you might include an open ended question asking potential respondents to state their reasons. Responses would then be coded and categorized. A comprehensive list of the most common reasons might then be obtained. This would be the basis for the list of answers you use in the final version of the survey. In this situation, you may still wish to allow for other responses. However, you would need to clean the data afterwards by reviewing each response not already listed. This is done by reading each additional response and deciding whether the response is novel (something the research hadn’t thought of), irrelevant/unrelated (the equivalent to a non-answer), or just a different way of saying what was already in the list of options (in which case the response should be recoded).
If the additional response is both new and meaningful, future versions of the survey should be updated to include this answer. However, the interpretation and reporting of the result you obtained with the incomplete set of responses options would be difficult. If you report that one person provided an "other" response (a factual statement) you may be under reporting the importance or significance of the answer based on the frequency or prominence of the response. Had others been presented with the option, they may have also selected it as a response. This is an issue because we just don’t know.
Likert Type response scales.
A Likert response scale is most appropriate when the item stem is a statement. Likert response scales are typically used when designing a scale rather than a questionaire but are also used in questionsaires. The Likert response scale is meant to capture the strength of an individual’s agreement or belief. An adaptation of the Likert scale is often used to group individuals based on some numerical amount (e.g., Income or age grouping). Response options in a Likert scale need to be ordinal in nature and mutually exclusive; Multiple-Choice response options need only be mutually exclusive. There are a few design decisions you need to consider when using this specific type of response scale.
Examples of items using a Likert response scale
Police salaries are quite low.
- Strongly Agree
- Strongly Disagree
- I don’t know or have no opinion
About how much time (in hours) do you typically spent TEXTING each day?
- None (0, I don’t text)
- Less than 1 hour (but not zero)
- 1 to 2 hours
- 3 to 4 hours
- 5 hours or more
Ordinality and Mutual exclusivity. This was mentioned previously but is important enough to explain further. A characteristic of ordinal data is that each point on the continuum represents a measurable increase in the amount or magnitude relative to the previous value on the scale and is some amount less than the next value on the scale. However, unlike interval level data, the difference in the amount between each point on the scale is not neccesarily consistent (i.e., equidistance). For both ordinal and intravel level data, each point must be separate and distinct (mutually exclusive with no overlaps). In the texting example, if you provided the options “1 to 2” and “2 to 3” the options would not meet the mutual exclusive requirement because respondents with exactly 2 hours could select both answers. Likewise, the following set of response options intended to capture the amount of time spent on an activity does not meet the ordinality requirement because “multiple times a week” and “as little as possible” are ambiguous. Respondents may interpret the amounts represented differently than what was intended (e.g, as little as possible may actually be a lot not a little; and multiple times a week might be less than twice a day). Cognitive load is also increased as respondents struggle with trying to determine the amount of time each option represents.
- Multiple times a week
- Twice a day
- Once every day
- Every other day
- A few times a week
- As little as possible
Option Specificity (number of response options). This is an issue for both multiple choice responses and Likert response scales but for different reasons. With Likert response scales, the issue is one of specificity and in some cases ordinality. Typically, a Likert scale will have 4 to 8 options. The specific number should be informed by the respondents’ ability to make clear differentiations between options on the scale. Having too few options (e.g., agree disagree) limits the ability to interpret the data regarding the magnitude of agreement or disagreement. In addition to not being needed, having too many options (e.g, more than 8) makes it difficult to give meaningful labels to each response (i.e., labels that makes them distinct and clearly ordinal in nature). Respondents most likely won't be able to make meaningful differentiations between points on the scale, and less granularity would likely serve the research purposes.
Examples of Option Specificity
Direct Entry Option. One way to alleviate the specificity and ordinality problem, in situation where amounts are used to determine group membership, would be to have participants enter the amount directly or derive the amount base on some other data (e.g. determining one’s age based on their birthday). Group cut points could then be established based on precise amounts rather than broad categories. Category cut points could be established in an a priori manner or based on natural divisions occurring in the data. If this approach is use, you will need to make it clear to respondents what units are being used (e.g., days, hours, or minutes) as well as whether partial amounts can be entered (e.g., decimals or fractions). With online surveys, application software often has features that allow researchers to set restrictions for data entry to ensure amounts are within a valid range and formatted properly.
Odds and Evens. Another often debated topic is the inclusion of a middle point. Having an odd number of options is sometime used which make it possible to have a middle point. Too often however it is used inappropriately. Having a middle point is acceptable when there is a true middle position. For example, when you are attempting to determine the amount of increase respondents believe occurred due to some phenomenon. If there was no increase and there was no decrease, a middle point representing no change is appropriate. Including the “neither agree nor disagree” options is thought by some to be appropriate in certain situations depending on the statement posed in the item stem; others however feel it can lead to lazy responses and central tendency error when respondents don’t consider their choice carefully enough. Removing the middle point by having an even number of options forces respondents to decide whether they agree more than they disagree. It is never appropriate to place an “opt out” option (e.g. “I prefer not to answer”, ”not applicable”, or “I don’t know”) in the middle of the response scale.
Example of Middle Point usage
Item stems as statements. Some practitioners recommend using questions (with multiple choice responses) rather than statements and Likert scales. Both can be used effectively and there are benefits and disadvantages to using either. However, for a statement to work properly with a Likert scale, the statement needs to represent a clear position rather than a middle of the road (ambiguous or noncommittal) declaration. For example, you might ask individuals whether they thought police salaries were OK; however, there is little value in knowing whether respondents strongly agree that salaries were OK because it is difficult to interpret differences between agreeing they were OK and strongly agreeing they were OK. In both cases the respondent feels police salaries are just OK. It like saying, I believe my hair is brown and I strongly believe my hair is brown. The reason for using a statement in the item stem is to determine the strength of an individual’s feeling towards the idea or concept. In addition to making sure the statement presents a clear position, the statement should not be something that everyone might agree with in general (e.g., It would be nice to provide adequate salaries for police).
In addition, information about respondents feeling might require additional information to determine what people thing should be done. You many need two questions because knowing how individual feel about police salaries may not align completely with what they feel should be done about it.
Question vs Statement
What should be done about police salaries? Salaries should be
Police salaries should be increased.
- It is easy to create a flawed items. Carefully considering various design principles will help improve the items you write.
- Two major types of response scales include Mutliple Choice and Likert Type response scales. Ranking and rating, as well as direct imput items require a different method for collecting responses.
- Usually rspondent must pick the single response that best descibe their beliefs. Using a "select all that appl" instruction should be used with caution.
- Allowing participants to opt out may be required but can affect the validity of the survey results.
- Allowing other responses beyond those listed may be necessary but introduce additions problems with interpretation and reporting.
- Option specificity (the number of response options in the scale) is an issue for both mulitple choice and likert type response scales.
- For multiple choice response scales, the options need to be mutually exclusive and distinct.
- For Likert Type response scales, the labels used to identify each options on the scale need to be ordinal in nature and mutually exclusive (clearly different).
- Allowing a middle point should only be used when there is a true middle position participants might take.
- When using a likert type response scale, the statement used in the item stem must be a clear statement that is not ambiguous, ambivolent, or a generally accepted declaration.
- Carefully pilot test all items you create. revise as needed then test them again before administering the survey.
- What ethical issues need to be considered when writing items. How can these affect the quality of the survey.
- In the item stem, when might it be best to use a statement rather than a question.
- What difficulties or problems occur when allowing respondents to select "all that apply" for a specific item?
End-of-Chapter Survey: How would you rate the overall quality of this chapter?
- Very Low Quality
- Low Quality
- Moderate Quality
- High Quality
- Very High Quality