The Design Process
Overview and Importance
The goal for any research is to get accurate and comprehensive information that can be used to answer the research questions. Survey research is no different. However, creating a survey that provides useful results takes time and effort. Unfortunately, many researchers fail to take adequate time to validate their survey instruments. There is not much worse for a researcher than to get survey data back and realize -- I should have done this differently. Without valid data, the study is ruined. And while it is unlikely that any survey you might create will be perfect, following a rigorous survey design process will increase the likelihood that the data you obtain will be useful.
Unfortunately, the survey design process is not understood very well; many do not know how to accomplish the task properly and too often the process is not carried out thoroughly. More importantly, many do not understand that proper survey design involves instrument validation.
The term validity has a slightly different meaning when applied to assessment (i.e., testing) and surveys (i.e., self-report data collection). Validity in assessment generally refers to the degree to which a test measures what it was designed to measure. This definition can apply to surveys as well, however for survey research, validity also refers to the quality, completeness, and usefulness of the data. More specifically, validating a survey instrument is a rigorous process intended to verify that the data obtained from administering a survey will answer the research questions adequately. This does not mean the information obtained from the administration of a validated survey will be perfect. There are many ways in which the information obtained can be biased regardless of the quality of the survey (see Survey Error and Response Bias). Validation simply decreases the potential for response bias and helps us avoid mistakes that can result in survey error. Again, the validation process does not guarantee perfect data; rather, once a thorough validation process is completed, the researcher can be confident that the instrument has the potential to produce useful information. The validation process attempts to make sure the survey can be used to accurately gather all the data it was designed to obtain.
Phases in the Survey Design Process
The survey design (i.e., validation) process involves several steps or phases. They are outlined here. Subsequent chapters then deal with specific aspects of each phase in greater detail.
Conceptualization Phase. This is a theoretical endeavor that affects all the other steps in the process. The aim of this phase is to clearly identify the study’s purpose and research questions. Having a clear understanding of the purpose and research questions helps us figure out what data is needed.
Administration Planning. Although implementation is not technically a part of the design process, this phase involved planning the survey administration to ensure adequate sampling and sufficient response rates. Understanding how the survey will be administer will also affect item creation.
Data Analysis and Report Planning. This is a planning step that will guide how data is collected. Understanding how the data will be analyzed and reported improves the likelihood that you get all the data you need and that the information obtained will be adequate for the statistical analysis needed to answer the research questions.
Item Creation and Response Scale Design. This phase involves development. Both survey items and response scales are created. The goal is to make items that will produce the information required.
Refinement and Pilot Testing. The objective of this phase is to make sure mistakes have not been made before administering the final survey. Based on the results, changes to specific items can be made. Depending on the result, several iterations of the phase may be needed.
- There are no perfect survey instruments but the likelihood that a survey will not be flawed is increased if a rigourous design process is followed.
- A proper survey design process will include instrument validation.
- Survey validation involves several steps or phases intended to verify that the data obtained from administering a survey will answer the research questions adequately.
- The validation process does not guarantee perfect data. However, once a thorough validation process is completed, the researcher can be confident that the instrument has the potential to produce useful information.
- The design phases include conceptulizing the purpose and establishing research questions. It also requires planning the administration, data analsyis and reporting. Instrument creation should also include a refinement and testing cycle.
- Explain how failing to complete each of the steps in the validation process might affect the results of a survey?
- Pick one of the validation steps and explain how skipping or not completing the step may affect the quality of the survey.
End-of-Chapter Survey: How would you rate the overall quality of this chapter?
- Very Low Quality
- Low Quality
- Moderate Quality
- High Quality
- Very High Quality