Survey Administration Planning

Basic Concepts and Principles

After the topic, purpose, and specific research questions have been determined, a few additional aspects of the research need to be planned. The first thing the researcher needs to determine is what data is needed. This is important so you can ask the right questions. The researcher must then decide whether a survey is the best way to collect the required data. You will get better data if you get data in the right way. If a survey is required, a determination needs to be made as to who should be invited to take the survey. Before you start creating survey items, you need to make sure you are asking the right people (i.e., those who can provide the information needed). As a general rule, you need to ask the right questions, to the right people, in the right way. This phase of the planning addresses these aspects of the research. Good planning will attempt to anticipate problems and attempt to prevent them.

What Data Needs to Be Gathered?

Before you can decide whether a survey is a good choice for data collection, you have to determine what information is needed. This decision will also be framed by the purpose of the study. If done carefully, considering the purpose of the study will help you ensure that you get all the data you need to answer the research questions.

During this phase you will want to review existing literature on the topic addressed in the study. By doing this you will get a better idea regarding the data other researchers have gathered and perhaps data that was not but should have been obtained. Not only will this help you gain a better understanding of the topic and issues, but it will also help you understand what instruments others may have already created that can be used during the designing or implementing of your survey. This is especially important when designing a scale. For a scale, the theoretical definition of the construct being measured helps to establish an operational definition which is needed for identifying the component parts or aspects of the construct the survey will attempt to measure.

Considering how the data will be analyzed may also need to be part of the planning. It is one thing to report results descriptively for the entire participant group, but it is another to report differences between subgroups. For example, if you want to disaggregate the data for purposes of comparing responses from different groups, you need to make sure you have sufficient information that will allow you to homogeneously categorize individuals into distinct groups in a mutually exclusive manner. If the purpose of the study is to explore the relationship between two variables, then an accurate measure of the variables in question is needed. This may require a survey, or it may require some other type of measurement instrument.

Data Identification Example

In the counseling services study example from the previous chapter, suppose you wanted to address the issue of need. This issue has various aspects, so you might want to determine the extent of the problem. However, perceptions of need can be personal or general. When a participant is asked about their needs and their perceptions of others' needs, you may get different answers (e.g., "I’m not depressed, but I know a lot of people who are."). A different approach might attempt to better understand whether students feel the available services adequately meet their personal needs and, more generally, others' needs. Information from both these queries (i.e., perceived need and availability) might be required if the study’s purpose was to identify the relationship between personal and general perception (e.g., "Does an individual’s personal situation influence perceived need?" or "Is the relationship different if the need is a personal or a general perception?"). Still another aspect of the issue might explore the perceived need for services that currently are not being provided. In addition, if someone was under considerable stress, doing poorly in school, or in a particularly challenging program, you might hypothesize that their feelings about counseling service adequacy may differ. If this were the case, you will need to obtain specific demographic and situational information. In any case, you need to carefully determine what information you will need to gather.

Is a Survey Needed?

Deciding whether a survey is needed depends on whether self-report is the best way to obtain the information you need. A self-report survey is needed in situations where asking an individual is the only (or best) way to get accurate data. Sometimes the decision to use a survey is clear. Asking individuals about their perspectives, interests, and feelings is not something that can be easily obtained in any other way. This does not mean the data will be accurate (see Response Bias); however, a survey may still be the best way to get that data. In other situations, the need for self-report may be a matter of convenience rather than need. The decision to use a survey may apply to all the information you want or just part of the data you hope to collect. For example, you may be able to get some of the data you need in another way; doing so may provide more accurate results but, for reasons outside of your control, it may be easier to simply ask. For example, the Family Educational Rights and Privacy Act (FERPA) may prohibit you from obtaining information about individuals without specifically getting permission from each individual. Then, even if participants grant you permission, you may find it difficult to obtain the data you need from those who have it for political or practical reasons (e.g., institutions may be unwilling or unable to provide the data). It may be simpler to ask individuals to provide the data even though by doing so you may lose precision or accuracy due to some form of response bias. In either case, the primary issue that will determine the need for a survey is whether self-report data collection is required.

Survey Need Example

Continuing with the example for counseling services, suppose you wanted to determine the relationship between a student’s achievement and their perceptions of counseling services. To do this, you would need to know the current grade point average (GPA) of students, as well as their personal feelings about counseling services. You would definitely need to ask students about their feelings, but it may be best to obtain their GPA in another way. A GPA is often calculated differently depending on the school and purpose for the score; as a result, and for a variety of reasons, when asking about a student’s GPA, the student may be unable to share this with you (e.g., they may not know their current GPA and provide only an estimate, or they may report the calculation based on all subjects instead of a specific set of core academic subjects you are interested in). In either case, getting GPA data from a more accurate source rather than asking students would be preferred. It may be more difficult to accomplish, but it would be more accurate. However, you may need to choose feasibility over accuracy. It may be impossible to get permission from respondents to get grade information, or the institution may not be willing to accommodate your request for the information even if you have student's permission. In such cases, if you are willing to accept the effect additional measurement error will have on the results, then asking students to self-report may be acceptable.

Who Should Be Invited to Take the Survey? Defining the Population

Another important pre-item-development task involves deciding who you feel should be invited to take the survey. Basically, this involves defining the population and understanding basic contextual issues surrounding the population. This step in the process addresses two important factors: accuracy and access. Accuracy involves making sure you ask the right people. Properly defining the population helps with issues of accuracy. Access is a feasibility issue and addresses whether you have access to the right people. Both these factors will influence sample size adjustments and sample selection procedures.

It is essential that you ask the right people. Asking the right people means asking those who have direct knowledge and would best be able to provide accurate information. A common mistake some researchers make when administering a survey is using a convenient indirect sample (i.e., people who do not have a direct knowledge but are associated with those who do). For example, asking parents to speculate what their children might feel or do. Given ethical and feasibility considerations, it might be more convenient to ask parents to complete the survey rather than their children. However, if you ask parents to speculate on things for which they do not have a direct knowledge, their indirect conjecture will likely produce flawed results.

In addition to asking the right people, you need to consider sampling; specifically, how many participants you need to survey. The issue of sampling is addressed in a separate chapter (see Sampling); however, the main concern is getting a representative sample. You need to be confident that the responses you obtain represent the targeted population. This includes any subgroup within that population, should the study’s purpose require disaggregation of data.

Getting access to those you wish to survey is also important. This is a feasibility issue and a question of whether the topic is researchable. If you cannot get reasonable access to those you plan to survey, the topic cannot be studied. This may also determine how the survey might best be administered (e.g., online, by mail, personal interview, or phone). Depending on the situation, you may decide accessing participants might best be accomplished in a specific manner and at a specific time.

Participant Selection Example

Continuing with the example for counseling services, because the target population is first-year university students, it would not be wise to survey instructors who teach undergraduate courses asking what they think first-year undergraduate students think and feel. While this seems pretty obvious, too often similar missteps occur. In addition, you need to make sure you survey a sufficient number of first-year undergraduate students. If you plan to disaggregate results, you also need to make sure your results adequately represent, not just the entire population, but also the subgroups of interest. For example, if a specific demographic is of interest (e.g., gender, age, race, socio-economic status, or program enrollment), you need to make sure that you get a sufficient number of respondents from each subgroup. Gaining access to the participants can also be a challenge. Suppose you plan to administer an online survey; you need to have email addresses for students. Assuming they all have an email address, you also need to make sure you have the email address students typically respond to (e.g., university-issued email or personal email). In addition, you will need to decide when it is best to administer the survey. Sending out surveys during final exams may exaggerate measures of stress. Administering the survey once school has ended for the summer may minimize results and reduce response rates.

Chapter Summary

  • Identifying what data is needed will help determine how to best get that data. 
  • Not all studies should use a survey. 
  • A survey should be used only when self-report is the best way to obtain the information needed. 
  • Data should be obtained from a direct source. 
  • A researcher's ability to gain access to participants will determine whether a topic can be researched properly. 

Discussion Questions

  1. When might a survey not be the best method for collecting data?  
  2. Under what circumstances might a study be untenable?
  3. Under what circumstance would it be appropriate to survey an indirect source for information? Explain your rationale.

Practice Tasks

  1. Based on a study you might complete, identify the population you will need to target. What is the best way to access these individuals?   

This content is provided to you freely by EdTech Books.

Access it online or download it at https://edtechbooks.org/designing_surveys/administration.