7 minutes
CoverEvaluation BasicsEvaluation within a Design ContextEvaluation for Instructional DesignersEvaluation PlanningResearch Principles PrimerEvaluation Approaches for DesignersEvaluation in the Analysis phaseEvaluation in the Design PhaseEvaluation in the Development PhaseEvaluation in the Implementation PhasePersonnel EvaluationData Collection PrimerReporting Evaluation Findings

Evaluation in the Analysis phase

Some typical evaluation approaches and designs used by instructional designers in the analysis phase are summarized here.

Gap Analysis (also called a Needs Analysis)

A search of this topic will likely produce different descriptions of what a needs analysis (or needs assessment) might entail. Some sources refer to a needs assessment as an evaluation whose purpose is to determine the degree to which an objective or goal was met. We refer to these types of evaluations as objectives-oriented or effectiveness evaluations.

What many call a needs analysis might more appropriately be referred to as a gap analysis. The terms need and gap are often used synonymously, but technically, an identified gap necessitates a solution; the gap defines the need. Said another way, we need a solution to fill a gap. These two activities (i.e., the gap and needs analysis) are usually performed together as one process. A gap analysis is used to identify and describe the difference (i.e., gap) between what is and what we want it to be. The needs analysis uses the information from the gap analysis to identify the cause of the problem and evaluate potential solutions. Thus, a gap analysis identifies problems (i.e., gaps), and a needs analysis recommends a potential (needed) solution.

You may detect various kinds of gaps. The gap might be a performance gap, a knowledge gap, a competency or ability gap. Many of these gaps lead to a need for training or practice, and other gaps are reduced by providing an educational product or tool. Sometimes the evaluator recognizes the gap cannot be filled. Before you can confidently recommend a solution, you will need to identify the cause of the gap. You will also need to consider how important it is to fill the gap and the likelihood that any solutions you propose will have the desired effect.

A gap can represent a perceived, present, unrecognized, or potential need. The gap may refer to the difference between intended goals (or objectives) and the actual (or desired) outcome (i.e., perceived need). It may represent the difference between a desired state and the current situation (i.e., a present need). Or, it may refer to something users don't have but could use (need) to accomplish some task (i.e., something a student or provider needs to accomplish a task). In addition, a gap may represent something an individual may need in the future (i.e., a potential need) even though the individual may not appreciate or be aware of the need (i.e., unrecognized need).

Those conducting a needs analysis are often susceptible to an availability heuristic cognitive bias, where decisions are based on the first viable option or solution that comes to mind. For example, it is not uncommon for people to quickly conclude that training must be developed or technology needs to be purchased before identifying the actual cause of the problem and whether that solution would be appropriate.

The basic steps required to accomplish a generic needs analysis are:

Step 1: 

Describe the context and intended users/learners.

Step 2: 

Identify the problem (need or gap).

Step 3: 

Describe the various users experiencing the problem.

Step 4: 

Identify the cause or reason for the problem. 

Step 5: 

Assess how important it is that the problem be solved.

Step 6: 

Recommend solutions.

To a hammer, all problems are a nail. However, in education,

not all learning gaps can be solved by creating more (or better) training.

There are several ways a Needs Analysis might be applied

Performance gap analysis

In their 1997 book, Mager and Pipe present a model for analyzing performance problems. Various depictions of the models exist (flowchart 1, flowchart 2, flowchart 3).

In Mager and Pipe’s model, once the gap (problem) has been identified, the evaluator must identify the cause of the problem, then select an appropriate solution for the performance problem. This is done by asking a series of questions. Each question is intended to provide information so the evaluator can determine the actual cause of the problem and propose a targeted solution.

It is not uncommon for instructional designers to produce training whenever it's asked for, even when training isn't going to solve the performance problem or produce the desired results. Training is only the best intervention when the performance issue is the result of a lack of knowledge and skill. Some issues that would not require training as a solution include:

Being out of practice

Skill is rarely used

Performance is punishing

Its boring, difficult, unpleasant

Performance is unrewarded

Poor compensation, lack of acknowledgment

Nonperformance is rewarding

Conflicting activities are more enticing, lack motivation

Obstacles exist prohibiting the performance

Policies, conditions, facilities, lack of resources, tools

Need to perform well isn’t important

Intended outcomes are trivial, little impact

Training needs analysis

Training is needed when the performance deficit is caused by a lack of knowledge or skill. In these situations, a training needs assessment should be conducted. A training needs analysis identifies the knowledge, skill, and abilities that can be facilitated through instruction. When training is required, gaps may represent knowledge and skills an individual will need in the future or abilities a person needs now to accomplish a particular job or perform a specific task. While a training need may be identified in the analysis phase as part of a gap analysis, some form of task analysis will also be needed, which is often carried out in the design phase.

However, when training is needed, it is not uncommon for instructional designers to spend an exorbitant amount of time and resources developing training for basic knowledge and skills while devoting insufficient time creating instructions for important topics which may be difficult to teach and challenging to develop. A training needs and task analysis will not solve this problem. The designer must use the information provided by a training needs analysis to make design decisions that will facilitate learning in an appropriate, effective, and efficient manner.   

The goal of a training needs assessment is to create learning objectives for the training so assessments, instructional materials, and learning activities can be designed and developed. The basic decision-oriented question that needs to be answered is—what do we need to teach? Some ways we can answer this question include:

Consumer Reviews

A consumer review is often utilized once a gap is revealed and one or more products (i.e., solutions) exist that could potentially fulfill the identified need. In an educational setting, solutions may include a training program, policy, service, or product. The evaluator's role in a consumer review is to provide an expert opinion.

Results from a gap analysis and consumer reviews inform an evaluator's recommendation to create a new product or utilize an existing solution. The purpose of a consumer review is to provide information to the decision-makers about the various solutions so they can decide which, if any, should be employed. If none of the existing solutions are viable, an evaluator might recommend developing a new product or revising an existing product.

The evaluator's judgment of the product might be determined through measurement or personal experience. For example, the value of a product may be measured and compared against specific performance benchmarks. Or, the evaluation may be based on the evaluator's professional opinion, other users' experience and preference, or both. The evaluator must capture and report findings in a way that helps individuals make an informed decision regarding a solution's merit, worth, and value.

Once a significant problem or need has been identified, the basic steps required to accomplish a consumer review are:

Step 1:

Identify existing products (potential solutions).

Step 2:

Establish criteria for judging each solution.

Step 3:

Test each product using the standards (criteria) you selected.

Step 4:

Present results using a comparative analysis of the products. 

Step 5:

Make a recommendation.

Chapter Summary

  • A few evaluation activities that commonly occur in the analysis phase include: gap analysis, needs analysis, and consumer reviews.
  • The goal of gap analysis is to identify problems and deficiencies.
  • A needs analysis is used to identify the cause of a gap and recommend solutions.
  • A consumer review can help evaluate existing educational products that could serve as a solution to an educational problem (a gap). If no viable solutions exist, a new educational product or solution may be needed.
  • Not all educational problems require a training solution—only those involving a lack of knowledge, skill, or ability.
  • A performance analysis is a type of needs analysis that can be used to identify the cause of a performance problem.
  • A training needs analysis is used when the needs analysis suggests training is required. Its purpose is to create learning objectives for the training.

Discussion Questions

  1. Consider an educational product you use or might be interested in using. Identify the purpose of the product and the problem (gap) it solves. Think about how the product works. Who might benefit from this product? Are there alternative (similar) products on the market? Briefly describe how the products differ. Why would someone choose to use one product over another?
  2. Think of a task you or someone else is required to perform, something that people may not do well. Using Mager and Pipe’s model for analyzing performance problems, work through the steps, identify the potential causes of poor performance and suggest specific ways you might solve the problem.

References

Mager, R. F., & Pipe, P. (1997). Analyzing Performance Problems: Or, You Really Oughta Wanna (3 ed.): Center for Effective Performance.