Mistakes Were Made!

The most common mistake many researchers make when conducting survey research is administering the survey before it has been properly vetted. And unfortunately, another common practice that fuels the criticism of survey use is hiding a survey’s flaws—treating the results as adequate when they are not. While it is unlikely that any survey you might design will be perfect, understanding and attempting to mitigate potential problems will increase the usefulness of any survey data you obtain. 

Survey Errors and Response Bias

There are many ways in which a survey research study can go wrong. When a survey goes wrong, we describe the results as having error or bias. Both can invalidate the results of a study. Survey error is caused when researchers make mistakes when creating or administering a survey, as well as when they interpret the results of a survey. Response bias refers to the ways respondents may be unduly influenced when providing answers on a survey. Bias is an issue that affects the accuracy of the survey data obtained and is the result of participants' inability or unwillingness to answer questions precisely or truthfully. There are several specific kinds of error and bias that are important to understand prior to developing your survey.

Common Forms of Survey Error

Survey Scope Error. This error refers to the mistake some make when a survey does not include important items required to fully answer the research questions. Failing to ask important questions will provide an incomplete or inaccurate answer to the research questions posed in the study. This can be difficult, and there will always be some tension between making sure to include all essential survey questions and limiting the length of the survey. Sometimes this error is made because the researcher has not carefully considered what they needed to know. Other times you may only realize a mistake was made once you have the survey results back and find you failed to ask an important question. When a questionnaire or scale does not include items that cover all the important aspects of the topic or construct, survey scope error occurs, reducing the validity of the results.

Purpose Creep Error. At times researchers (or clients) will add items to a survey that don’t directly align with the needs and purposes of the study. This is referred to as purpose creep. Unnecessary survey items, like superfluous demographics or items that “might be nice to know,” may be interesting, but often they are never used. Adding unnecessary or tangentially related items may not affect the validity of the results, but it can still be a problem because it can add to participants’ fatigue and can affect their willingness to thoughtfully answer all the items in a lengthy survey.

Sampling Error. This occurs when mistakes are made selecting a sample. Sampling error typically occurs when a specific subgroup within the population is under- or overrepresented in the sample. When this happens, the results obtained from the survey are not generalizable. This is because the characteristics of respondents in the sample do not match proportionally with the characteristics of those in the population. This can easily happen with small samples, especially when there are a number of small subgroups within the population. As a result, it is generally a good idea to over-sample when possible to maximize the likelihood of obtaining a good representation of the general population and subgroups within that population. However, having a large sample size may not solve the problem if the sample does not proportionally mirror that of the population; in this case, sampling error will still cause the results to be flawed. You will likely never know the degree to which sampling error has affected a study. You should always assume there will be some sampling error and take steps to alleviate the problem as much as possible.

Response Rate Issues (Response Refusal). This is related to sampling error, in that it can affect the generalizability of the results, but it is slightly different. If the group of respondents you invite to take your survey would likely form a representative sample of the population, but several individuals choose not to respond to the survey, the results may not be generalizable. To ensure full ethical transparency, you should always report the sample size and the response rate when writing up results. You should also examine the pattern of respondents who chose not to take the survey to determine if any discernable pattern can be found. If the pattern is random, then there may not be a problem. If the pattern is systematic, meaning one group of potential respondents with similar characteristics is more likely not to complete the survey than another group, then you have a problem.

Item Nonresponse Error. This happens when a participant fails to answer all the questions on a survey. It can happen by accident or intentionally. Nonresponse is unfortunate when you are administering a questionnaire but does not necessarily invalidate the results; the results depend on how many respondents failed to answer a specific question. However, item nonresponse is definitely a problem when administering a scale. This is because the items in the scale work together as a group to provide a measure of a construct. In a scale, each item represents a specific aspect of the construct, and the measure, or score, relies on all the items being answered truthfully and accurately.

Common Forms of Response Bias

Recall Bias. This is common in self-report situations when respondents are asked to provide information retrospectively. Human memory is imperfect. Some information (i.e., perspective and feeling) is more likely to be remembered than others. A person’s ability to recall events and feelings will depend on the metacognitive ability of the individual and the significance of the event to that particular person. Recall often depends on the time interval between the event and when the individual is asked to recall their perceptions. A person may have forgotten the event altogether; they may remember incorrectly, or they might revise their recollection (see prestige bias).

Social Desirability & Conformity Bias. It can be hard for respondents to openly express non-conformity when asked to self-report their behavior, beliefs, and opinions; this is especially true when the respondent believes they may be ridiculed or despised. In such cases, respondents tend to provide a socially acceptable response (sometime subconsciously) over their true feelings. For example, a respondent may tend to agree with a statement more strongly than how they truly feel when the item addresses something that is generally seen in society as desirable or expected.

Prestige Bias. This bias is related to social desirability bias as it is based on an individual’s personal desire to be seen in a positive light. This bias is based on personal feelings, not a general instinct for conformity. For example, respondents may round up their income or report exaggerated amounts of time spent on worthy endeavors (noting the reverse would be true for endeavors the individual feels may diminish how they are perceived). This may not involve outright lying, rather the individual may actually remember the facts inaccurately. Respondents often tend to view or recall their own situation in a more favorable light than is actually the case—subconsciously protecting their self-image or inflating their ego. It is often good practice to assume that, if a question has a potential prestige component, the responses are likely inflated to present the respondents in a more favorable light. Exactly how much they are inflated will depend on the question, context, and respondents.

Acquiescence or Agreement Bias. This bias is like conformity bias. However, unlike conformity bias, in this case the respondent will, in general and inadvertently, agree with statements. With this bias, participants tend to select a positive response option or disproportionately indicate a positive connotation. This bias will skew results towards the positive.

Item and Option Order Effect Bias. Order bias can be the result of both item order and response option order. The order in which survey items are presented can affect a respondent’s answers due to a priming effect. People tend to contextualize their responses. Because of this, survey questions that come just before a particular query may provide information that respondents will use as context in formulating their subsequent answers. If a different primer was presented, the responses may be significantly different.

Two common response biases associated with response option order are primacy and recency bias. Primacy bias is the tendency for respondents to pick one of the first options presented to them. This can happen when a respondent quickly reads through the survey and picks one of the first response options they agree with. Recency bias is the tendency to pick an answer option presented at the end of a list. For example, in a long list of options, the choices respondents read last are more memorable to the respondent as they select an answer.

Mood Bias and Emotional Mind Sets. One’s mood or mindset will affect the way responses are provided. For example, if a participant is exceptionally happy or angry for some reason while taking a survey, their emotional state affects the general pattern of responses provided. Given time, the respondent's current extreme emotions may subside, which will modify the intensity of the responses provided. Emotional responses can be intense in either a positive or negative direction. You will also see this when the survey addresses an emotionally charged topic. Responses may tend to be on the extreme ends of the response scale, possibly because those who choose to complete the survey have strong opinions; however, mood bias becomes a problem when opinions are exaggerated by the respondent's current emotional state.

Central Tendency Bias. This bias refers to the tendency of some individuals to avoid responding in extreme ways. For example, some people may never indicate they strongly agree or are extremely dissatisfied (i.e., nothing is perfect, and nothing is completely without merit). This is the opposite of a mood bias in that responses from those who have this bias will trend closer to the center of the response scale.

Demand Characteristic Bias. A demand characteristic is used to describe specific cues in research that may inadvertently influence a participant’s response. A demand characteristic can manifest in a number of different ways if the researcher is not careful when designing and proceeding with a study. In social science research, demand characteristics can create bias when the subject becomes aware of the purpose of the study. This may potentially bias or invalidate the outcomes. When a respondent becomes aware of the reason or purpose of the study, they may intentionally provide answers they feel would influence the results. For example, if a respondent figured out that the results of a survey will be used to set policy, the individual may attempt to answer in a way that they feel would be beneficial to them.

Random Response Bias. Random response bias can occur when a respondent honestly does not know the answer to the question but answers anyway. This can happen when you ask a respondent to answer a question for which they would not reasonably know the answer. Respondents resort to guessing or speculating rather than reporting factual information. An example of this would be asking someone to indicate the motive of another individual, prompting a random response bias.

Another way this bias can manifest is when an individual has an opinion but hasn’t considered their true feelings carefully. Like a central tendency bias, these individuals also tend to choose options toward the middle of the response scale. At times, people with this bias will choose the exact middle point (on an odd numbered response scale) simply because they don’t want to think about the issue or don’t really care. This bias can also manifest itself maliciously when an individual intentionally responds in a random fashion without actually reading the items. This can happen when there is an incentive involved and people simply want to complete the survey for the promised reward. If you suspect the possibility that a random response bias might occur, trigger items can be added to identify suspect response patterns. For example, adding an item that is the opposite of another item or reverse scoring an item can be used to identify potential problems. Note however, the practice of reverse scoring some items but not all the legitimate items is not advised as this can cause a response bias which can affect the analysis and interpretation of the results (Kulas et al., 2018).

References

Kulas, J. T., Klahr, R., & Knights, L. (2018). Confound it! Social desirability and the “reverse-scoring” method effect. European Journal of Psychological Assessment.