An Overview of Survey Use
Surveys are often used for evaluatons and research. The concept of creating and administering a survey is deceptively simple. You probably have been asked to take several surveys at some point, so you (and many others) most likely feel you understand the idea well enough. Taking a survey means answering questions. Administering a survey involves asking questions. As simple as this process appears though, you probably won’t be surprised to find that there is a bit more to it than that.
Many of the definitions provided for the term survey involve describing the function or purpose of the process. In the social sciences, "to survey" means getting a general picture or description of some identifiable group of individuals. However, a survey also refers to a type of data collection instrument used to gather information from the group of the individuals that you want to better understand. The term survey is used to describe the activity but also the data collection instrument. Survey Research refers to any study or evaluation that uses a survey as its primary data collection method.
Usually when people refer to a survey they mean a sample survey. When using a sample survey, researchers select only a portion of some target population with the intention of generalizing the results to all those in that population. When researchers survey the entire population, they have conducted a census. However, in many cases, surveying the entire population is not feasible; as an alternative, researchers collect information from some individuals to gain insights into the entire group. If done properly, surveying a portion of the population will produce a reasonable description of the entire population.
A survey is a self-report data collection instrument designed to get information from individuals in a specific group.
A sample survey refers to any survey that gathers information from a sample group in order to make generalizations about the group’s population.
A representative sample is one that provides a reasonable representation of the population targeted in the study.
Survey research refers to any study or evaluation that uses a survey as its primary data collection method.
Several aspects of a survey research study determine its quality, but the overall quality of the study will primarily depend on the quality of the survey instrument used to collect information. More specifically, the quality of the items on the survey instruments has a significant impact on the overall quality of the study. It is relatively easy to write survey questions, but it takes effort and skill to craft good ones. It is also important to make sure the survey asks all the right questions, in the right way, and is administered to the right people.
The results from a survey may be used to gain insights about a specific group of individuals, but they also can be used to make generalizations about the population the sample comes from. The degree of validity of the generalizations being made depends on the sampling methods used and the response rate. Basically, the sample must be a representative sample, meaning those reponding to the survey are a reasonable representation of the population targeted in the study.
Survey research is conducted for a variety of purposes. Often the researcher simply wants to better understand the characteristics of individuals in a specific population. Characteristics of interest might include attitudes, opinions, trends, or perceptions. The only way to get these data is to ask individuals directly. This means the respondents must be willing and able to answer the questions truthfully and accurately; the value of any survey research study depends on this. If you can get a more accurate measure of a characteristic in another way, a self-report survey may not be your best choice for data collection.
Understandably, critics of survey research point out these limitations as a reason to question the results of many studies. But with any assessment tool, there will be measurement error. Proponents of survey research point out that having a reasonable estimate, even if not perfect, is often better than having no data at all. However, when conducting survey research you must recognize, attend to, and address threats to validity by designing and creating good surveys.
Types of Surveys
Surveys can take various forms but the most common form is a questionnaire. In this course I will also explain how to create a scale. The basic difference between these two types of surveys is the relationship between the items on the survey instrument.
In a questionnaire, each item stands on its own and is reported separately. Each item provides data that can help answer different parts of the research question. In a scale, the items work together to describe a specific construct or an attribute of an individual participant. Each item in the scale focuses on one aspect of the construct. In a scale, the items responses are combined to create a score. The construct being measured is often affective in nature in that it describes some attribute of the individual respondent. For example, a scale might measure typical feelings, attitudes, or perceptions about something. The resulting score is an indication of the magnitude and direction of the attribute being measured.
With both survey forms, summarizing the results of those individuals participating in the survey can be used to describe those in the population (i.e., the proportion of individuals within a specific category or, in the case of a scale, the average score of individuals within the population). Although, a scale might also be used as a measurement instrument designed to describe some characteristic of an individual but not necessarily a description of the targeted population.
Scales and Questionaires
A questionnaire contains a number of individual items. Each item on a questionnaire is analyzed separately.
A scale is a set of items that measure a specific construct. The items, as a group, provide a measure of the degree to which the construct is present for an individual.
Survey use in Research and Evaluation
Survey research features the use of self-report data collection measures. It is a flexible approach that can be used to study a wide variety of basic and applied research and evaluation questions. Survey use has its roots in applied market research and election polling. More recently surveys are used as a data collection tool in many social science disciplines, including political science, sociology, public health, psychology, and education.
Most survey research is non-experimental (i.e., descriptive research) but surveys are also used in experimental and quasi-experimental research (i.e., comparisons between two randomly assigned or two intact groups). Results obtained through survey research can be used to describe a single variable (e.g., the proportion of individuals who prefer a specific instructional approach or the prevalence of a condition in the defined population). However, surveys are also used to assess relationships between variables (e.g., the potential relationship between socioeconomic status and educational achievement or the factors that likely influence a specific perception). Survey use in educational evaluations often attempts to determine changes in some characteristic that might be attributed to an intervention or program.
Survey research is mostly seen as a quantitative method. Surveys describe things numerically through proportions and means. For example, the results of a survey might report the proportion of individuals who responded a certain way. A survey might also appropriately be used to determine the average amount of time respondents report spending engaged in some activity (although averaging response option is notably a controversial analyis—more on this later). These data do not explain why this phenomenon occurred, just that it occurred. Qualitative methods in contrast describe things holistically using an interpretive lens. Depending on how they are designed, qualitative methods can provide insights and possibly explanations as to why a phenomenon might have occurred.
A survey can be used to capture some qualitative data using open ended item formats or semi-structured interview questions; however, open-ended response items need to be coded and interpreted, thus the utility of such items is limited in that these data are not objective measures. By objective I mean the ability for the results obtained from an item to be analyzed directly without prior interpretation or judgement. Often the utility of a survey is found in the objectiveness of the items in the survey.
With regards to research and evaluation designs, surveys are often used in a mixed methods approach. This approach could be either an explanatory or an exploratory mixed methods design. In an explanatory design, the researcher might use a quantitative survey to determine that a condition exists, and then use qualitative methods in an attempt to understand and explain the phenomenon. In an exploratory design, the research may have used qualitative methods to uncover a phenomenon, and then used a survey or other quantitative methods to verify and determine the degree to which the phenomenon exists in a specific population.
Surveys can be administered in a variety of ways, including paper and pencil forms, personal interviews, and online. Likely due to the increased availability of personal computers, laptops, and smart phones, the most common way to administer a survey is online. With the increased accessibility of technology and the internet, the potential for a survey to capture information in a quick and inexpensive manner leads many researchers to utilize this data collection practice. However, this is not always the case. And, in many ways surveys are overused, leading to additional challenges including response fatigue and lower response rates. Ethical issues and Institutional Review Board approval issues must also be considered.
To some degree there exists skepticism surrounding the ability of any survey (i.e., as a data collection method) to produce information that is not flawed to some degree. This same criticism might be made of most research. It is easy to create flawed data collection instruments which produce results that are inaccurate and cannot be replicated. Poorly constructed surveys often lead to a distorted representation of the phenomena researchers are describing. If you plan to use a survey, it is important to take careful steps to validate the instrument.
With today’s technology it is easy to create and administer surveys. Unfortunately, surveys are not always constructed well. It takes time and resources to create a good survey. It is often difficult to tell the difference between a good survey and a poor one when you look only at how the results are reported. The question of whether survey research is scientific really depends on what you are trying to achieve, the care you take in designing the study, the way you create and administer your survey, as well as the way you analyze and report of the results.
- A survey is a self-report data collection instrument.
- Surveys typically take the form of a questionnaire, but can also be designed as a scale (or a mixture of the two).
- In a questionnaire, each item acts independently of the others and is analyzed and reported separately.
- In a scale, several items work together to provide a score that represents the degree to which an individual possesses some characteristic.
- Surveys are often administered online (for convenience and cost reasons) but they can be administered in other ways, for example, as an interview.
- Survey research includes any study that uses a survey as its main data collection tool.
- Although a survey may not be the ideal method for studying every research question, a well-designed survey can enhance our understanding of just about any issue.
- A survey is often the only means available to get the information needed to answer certain research questions.
- Most survey research is non-experimental (i.e., descriptive research) but surveys are also used in experimental and quasi-experimental research. Survey research is mostly seen as a quantitative research method.
- Surveying the entire population is called a census.
- Sample surveys obtain information from a representative group in the target population then attempt to make generalizable conclusions about those in the population.
- In order to make valid generalizations, the sample used must be a representative sample of the target population.
- Conducting high-quality survey research will depend on asking the right questions, in the right way, to the right people.
- It is easy to create a flawed survey. Without careful attention to overall survey design and the creation of the survey items, the study will likely produce invalid results.
- Under what condition would a survey be a good choice for your research?
- When would a scale be needed rather than a questionnaire?
End-of-Chapter Survey: How would you rate the overall quality of this chapter?
- Very Low Quality
- Low Quality
- Moderate Quality
- High Quality
- Very High Quality