Data Analysis Pre-Post Example
A potential reason for conducting survey research is to measure change. Quite often researchers want to be able to determine whether any change can be attributed to a specific intervention but they might also simply be interested in identifying change in general (not attributed to any specific causal relation).
Measuring change usually requires that we have two measures of the variable in question for each of the individuals responding to the survey. Change is calculated by determining the difference between the paired values. This typically means administering two surveys (i..e, pre and post surveys each providing a measure of the variable). It is best practice to use the same individuals for the pre and post surveys; however, in some cases different groups of individual might be surveyed if the samples are randomly selected representative samples of the population. Change in this case can be determined by comparing the average difference obtained from each survey. In some cases, a single survey (posttest only) might attempt to determine change by asking respondents to report the amount of change they experienced, or, by asking respondents to report pre and post levels of the variable in question.
Reporting Amount of Change
When data is interval or ratio level (scale data) the mean differences can be reported. However, in many cases, the data obtain in a survey is best described as ordinal data and should be reported as proportions or frequencies.
Consider the situation where you ask respondents about their level of interest in a topic (pre and post). You might speculate (or hope) that after receiving instruction those involved might become more interested in the topic. In this case the data you obtain is not interval level data because the data points are not equidistance. We can still provide descriptive information about the change that occurred as well as inferential evidence regarding the statistical significance and magnitude of the change.
Reporting average percentages by response categories we see that in general there was a 7% increase in interest post instruction. Using inferential statistics you would be able to determine the statistical significance (probability that this result was due to chance) and the practical significance (whether the magnitude of the change was substantial). A chi-squared goodness of fit test could be use for a pre-post comparison using the post test results as observed values and the pretest results as expected values. Remember that it is best practice to also report the sample size and response rates.
Reporting Breakdown of Change
In this example, reporting overall change in interest can be a bit misleading. You might erroneously conclude that the change in interest was all positive. This is not the case. While the aggregate change has a positive trend, many individuals actually reported a decrease in interest after instruction. This change is not represented in the previous analysis. A more complete analysis would report the breakdown of the change which includes the magnitude and direction.
Based on this analysis, while there was an overall increase in interest, 31% of respondents reported an increase in interest and 20% of the respondents reported a decrease in interest. Non-parametric inferential statistics (e.g., a McNemar-Bowker test of symmetry) could be used to determine the statistical significance of the result.
Post Test Only Example
While this result may best be obtained using pre and posttest surveys, at times this is not possible. You may only be able to ask respondents (in a posttest only survey) to indicate any change in interest they believe occurred after receiving instruction.
- What are the benefits and limitations of measuring change using data from match pairs, representative samples, or posttest only single sample data?
End-of-Chapter Survey: How would you rate the overall quality of this chapter?
- Very Low Quality
- Low Quality
- Moderate Quality
- High Quality
- Very High Quality