Evaluations are often needed to make informed decisions. The basic process for any evaluation requires that you:
Sounds simple enough. However, the more closely you look at each aspect of the process the more you will appreciate the complexity of planning and carrying out a good evaluation. There are several things to consider.
Evaluator Titles and Jobs
While it is true that everyone conducts informal evaluations all the time, it is also true that those who conduct formal evaluations do not always identify themselves as evaluators.
Most jobs require evaluation skills and abilities; not all job descriptions specifically articulate this as a requirement. People conducting evaluations as a part of their job go by various names and titles. Some of these include – auditor, assessor, analyst, judge, compliance officer, arbitrator, counselor, consultant, director, specialist, manager, supervisor, and advisor. Specific jobs for evaluators in educational settings include – evaluator, teacher, the office of teaching effectiveness and innovation, instructional designer, training and support coordinator, evaluation and research assistant/associate/specialist, director of evaluation and assessment, curriculum specialist/director, program manager/coordinator, assessment/data analyst, and institutional researcher.
Internal and External Evaluators
We should also make a distinction between internal and external evaluators. Internal evaluators, those working within an organization as evaluators, work as full- or part-time employees. When evaluators work as consultants, completing as-needed contract work, they are referred to as external evaluators. There are benefits and disadvantages to both.
When an organization is committed to evaluation and the benefits of continuous improvement, and the organization is large enough to warrant hiring full-time evaluators, it can be cost-effective and efficient to have internal evaluators on staff. Internal evaluators can develop an in-depth understanding of the organization, its purpose, goals, politics, structure, and personnel. They have access to the information and informants they need to do their job and can work on several interrelated evaluation projects (of various sizes and scopes) within the organization. This can be extremely beneficial to an organization. However, internal evaluators can become biased or jaded. They may begin to advocate for specific solutions for political reasons or base findings on unimportant criteria. If internal evaluators do not have some degree of autonomy or do not develop and maintain solid evaluative thinking abilities, professional ethics, appropriate soft skills, and healthy relationships with those they serve, their evaluation efforts can be ineffective and impeded by others in the organization. Above all, evaluators need to be trusted.
External evaluators are sometimes needed for various reasons. At times the evaluation needs of an organization exceed the capacity of internal evaluators. Those in the organization may not have time or expertise to complete a required evaluation. It can be cost-effective and prudent to hire an external evaluator to provide professional services on an as-needed basis. Sometimes, for political reasons, an evaluation needs to be perceived as unbiased and objective. An external evaluator’s reputation as a competent ethical evaluator can provide the organization with results other stakeholders and the general public can trust. There are, however, challenges an external evaluator may face. An external evaluator needs to gain an understanding of the organization’s goals and structure. They (or the evaluation team) need to develop relationships, get access to information and informants, understand how an individual project fits into the overall picture and the reasons the evaluation is being commissioned (i.e., needed).
You will recall that one of the main differences between research and evaluation has to do with the role the principal investigator plays in terms of who is responsible for making final decisions regarding the inquiry’s design (i.e., purpose, questions, and methods). Researchers typically work for themselves (i.e., they are in charge) while evaluators work for clients. There are exceptions, for example when a researcher works on an institutional research team or when an evaluator is conducting evaluation research. Still, evaluators typically work for a client as a service provider, on an evaluation team, or as an independent consultant. We use the term client as a broad description referring to the person commissioning an evaluation project. The client may be a supervisor or manager who assigns the project to an internal evaluator or a team of evaluators. The client may also be an employer hiring an external evaluator to conduct a specific evaluation. In either case, the role of the evaluator can be characterized by the involvement they have in the decision-making process and their responsibilities in implementing the evaluation. The relationship between the evaluator’s involvement in the project and their role is depicted in Figure 1.
Service Providers. When the evaluator is hired as an external consultant or is the director of a department, and they have an extensive say in determining the purpose, goals, and questions for an evaluation, as well as, the methods utilized to complete the evaluation, they might best be described as a service provider. In these cases, the evaluator (or their team) is typically responsible for the implementation of the evaluation. They will then report back to the client once the evaluation is completed.
Evaluation Partner. An evaluator may work as a partner with a client. This is what Patton (2010) calls developmental evaluation. The evaluator may be working as an internal or external evaluator, but when the evaluator’s role is one of partner, the control and the implementation of the evaluation responsibilities are shared between the client and evaluator.
Evaluation Consultants. An evaluator may also be hired solely as a consultant. In these situations, the evaluator lends their expertise to recommend the best way for the client to complete the evaluation. The evaluator may supervise and train individuals as they complete various evaluation activities. The purpose of an evaluator serving in this role is to build capacity and put in place needed evaluation processes so the organization can conduct its own evaluations without the evaluator in the future. Stufflebean (2014) warns however that this type of evaluation may become a pseudo-evaluation if the evaluator is expected to simply sign off on the evaluation as if the evaluation was completed by the evaluator and not the client.
Figure 1: Evaluator Role by Involvement.
Evaluations should not be cast into a single mold [scientific].
For any evaluation, many good designs can be proposed, but no perfect ones.
The main difference between a plan and a proposal is the amount of detail required. Depending on who the evaluator is working for, a proposal need not include a detailed evaluation plan. Suppose you are working as an internal evaluator. In that case, your proposal needs to be clear, concise, and detailed enough that your employer/supervisor (i.e., the client) would sign off on your vision for completing the evaluation. However, if the evaluator is competing for an evaluation through a requestion for proposals (RFP) process, the plan may need to be quite detailed as the proposal will form the basis for an evaluation contract. In cases like this, you must persuade the client that your proposal is the best one submitted, and the prospective client should hire you to complete the evaluation. Once a proposal has been accepted, a more detailed plan may be needed to manage the project properly. Project management plans will include specific details regarding the project tasks, data collection, and personnel assignments.
Before producing a proposal or plan, it is essential to clarify the client's request. It can take some time to fully understand and clarify the context, conditions, and potential value of conducting an evaluation. You will need to consider several things before deciding to proceed with an evaluation. To properly conduct an evaluation, you will need to fully understand the situation. Asking specific questions can facilitate this process.
Often the evaluation context can be complicated. Many educational initiatives and programs are sponsored (funded) by one entity, managed by someone else (often the client), implemented by several individuals (concerned stakeholders), and intended to benefit a diverse group of end-users. Likewise, the reasons for commissioning an evaluation can be varied. Politics is often involved. Sometimes the client has a hidden agenda; they may want the evaluation to justify or support a preferred decision or course of action. Various stakeholders may have different reasons for wanting the evaluation and may intend to use the evaluation results to inform a variety of decisions. To avoid problems, the evaluator must understand the context.
Work with the client to fully understand what is to be evaluated. You might ask:
Evaluators need to gain a thorough understanding of the things they will be evaluating. Not only will they be required to describe the evaluand, but they will not be able to make valid recommendations without understanding the product and how the designer intended it to be used. Some evaluators use a logic model to accomplish this task. When evaluating an educational product, knowing where the product is currently in the design and development process can also inform the kinds of evaluation needed.
It is likely that any evaluation you attempt can be conducted successfully to some extent. We say "likely" and "to some extent" because we do not live in a perfect world; we often lack sufficient resources, and sometimes we lack the desire or ability to complete requisite evaluation activities.
Consider the following common evaluation constraints and issues:
Desire. This can be a constraint when working for a client or with an evaluation research team. You may be willing and interested in completing a rigorous systematic evaluation, but the client (who has commissioned the evaluation), for legitimate reasons, may only be interested in paying for answers to a few specific evaluation questions.
Feasibility. In addition to desire, feasibility can be a constraint. For example, the preferred data collection method may not be possible. When this happens, compromise is required, and, in some cases, the activity may need to be abandoned.
Ability. People often describe evaluators as a generalist. While an evaluator may have many skills and abilities, they may not be an expert when it comes to specific competencies. Skills and abilities can be developed, but if an evaluator cannot competently complete a specific evaluation task, they will need to recruit (hire) someone to help or be satisfied with the possibility of getting a less than perfect outcome.
Time. This is a prominent issue in evaluation practice. Time is usually of the essence; however, a specific window of opportunity often constrains our ability to capture data. For example, you can only get student achievement data once learners have finished taking their course and the final exam. This means an evaluator may have to wait several months to get the required data needed to complete an evaluation. In addition, some data collection practices take time to complete and are more costly to accomplish (e.g., interviews and observations). An impact evaluation can take months, if not years, to track participants longitudinally. If an evaluation needs to be completed within a specific time frame, evaluators may need to prioritize what can and will be done within that time limit.
Money. This is a constraint because clients (employers) have limited funds. Some evaluators are willing to work for free, but others wish to be compensated for their services (soft costs). In addition, there are always some hard costs associated with conducting an evaluation. These include the cost of travel, facilities, equipment, and supplies. These costs are sometimes forgotten when the evaluator is an internal service provider, but they exist nonetheless.
Access. When you plan evaluations, you might proceed under the assumption that you will have unlimited access to the data sources needed to complete the evaluation properly; this would be a mistake. In practice, evaluators must deal with many restrictions associated with data access. Ethical constraints prohibit evaluators from coercing or forcing individuals to provide information. Not everyone is willing to be a study participant, or you may need to get permission to collect data from potential informants (e.g., vulnerable populations). Sometimes the data you wish to use exists, but those controlling access to the data are unwilling to give it to you because of the regulations they must follow or the cost of providing that information.
Additionally, there are times when the data you need does not exist and cannot be easily obtained. You may lack access to key informants, or it is difficult to obtain accurate measures of the data you need because you do not have access to existing instruments or no suitable measurement instruments exist. At other times, those providing the information may not provide accurate information. For example, early elementary-aged children may not have the meta-cognitive capacity to understand and communicate their motives, likes and dislikes, goals and aspirations. Other participants may be unwilling to answer honestly because they are suspicious of an evaluator's motives; they may also provide skewed information for personal reasons. For example, they may (unintentionally or otherwise) downplay flaws in a product they value. Gaining access to and obtaining valid, relevant data can be one of the most significant limitations of any evaluation.
Not all evaluations can or should be conducted
Part of any evaluability assessment involves understanding the evaluand's current stage of development and circumstances surrounding the product's use. The other part involves determining if an evaluation is warranted. Not all evaluations are worth doing. There are several situations where careful consideration should be taken to decide whether conducting a formal evaluation is sensible.
There are certainly many reasons why an evaluation might not be warranted; those presented here are but a few. However, once you determine that an evaluation is viable and needed, you can now start planning the evaluation and developing a proposal.
An evaluation proposal (or plan) often starts with a description of (or introduction to) the evaluand, the client, end-users, and important stakeholders (i.e., those directly associated with the implementation of the product or with a vested interest in the product's use). Information gathered from the pre-planning phase will be indispensable in accomplishing this task.
For evaluation in an educational setting, the evaluator needs to describe the purpose and function of the instructional product, how the designer intended it to be used and who the intended users are. It may include an explanation of the theory and principles supporting the product's design and any contextual aspects relevant to the product's use. If the product has not already been developed and implemented, the introduction section of the proposal should explain the product's current status in the design and development process. Often, evaluators are asked to serve as external evaluators for a funded development project. In these cases, the evaluator may propose various evaluation activities appropriate for each stage of the product's development. For example, it is typical for a developmental evaluation to focus on formative evaluation activities during the project's initial months (or years), then switch to summative evaluation activities once the product's development has been completed and the product has been implemented in its final form. This may include efforts to establish the effectiveness and impact of the product. It may also include a negative-case evaluation to determine which users benefit from the product and which do not.
A purpose statement describes the reason (or need) for the evaluation. This statement will explain the decisions a client needs to make and how evaluation results will be used to inform these decisions.
Sometimes the purpose for an evaluation is provided for you by the client. Other times, all the client knows is that they need or want an evaluation of a specific product. Some clients may be asking for the evaluator's expertise to help them determine a valid and viable purpose for the evaluation. Other times the evaluator needs to suggest utilizing various types of evaluations to answer questions they may not have considered. For example, it is not uncommon for clients or interested stakeholders to ask for a summative evaluation of the effectiveness and impact of an educational product. This may sound reasonable, but if the evaluand has not yet been developed or has yet to be implemented, these types of evaluation may not be possible. Likewise, it is often the case that an evaluator needs to suggest alternative or additional reasons and purposes for evaluating a product. For example, while an objectives-oriented evaluation of the product's effectiveness might be beneficial, an implementation evaluation may be needed as well. An implementation evaluation provides evidence that an educational product can be and is being used as intended or that a program is being implemented properly. Without this, clients, decision-makers, and other interested stakeholders will often misinterpret and possibly misuse the results of an effectiveness evaluation. Misunderstanding results can adversely affect the decisions people make.
When deciding on a purpose for an evaluation, you need to consider all the possibilities then establish an appropriate yet feasible reason for conducting the evaluation. The purpose you eventually decide on will be influenced by context, situation, and existing constraints. However, initially, you should ignore constraints and brainstorm ideas as if anything were possible (divergent thinking). Think about everything you would like to know about the product you plan to evaluate. Consider what you would need to do to answer these questions. After identifying all the possibilities, determine which are essential (need to know objectives) and which are less critical (nice to know objectives). Consider constraints to narrow the evaluation purposes and scope (convergent thinking). The purpose you ultimately decide on will often represent a compromise between what you would like to do and what is possible. The purpose of any evaluation should represent an important reason or need.
An evaluation proposal should list specific evaluation questions needed to inform the decisions proposed in the purpose statement. Your proposal should also include a description of the criteria you will use to judge the merit and worth of the evaluand. Sometimes all the questions and criteria are deemed essential but not always. In these cases, the evaluator should prioritize the questions and criteria by importance. For example, the primary evaluation question might ask how successfully a product facilitates students' achievement of specific expected learning objectives. A secondary question (i.e., an important but not essential question) might ask in what ways the product might be improved to increase its utility. The effectiveness criterion is primary, and the usability criterion is secondary.
You will recall that a standard is a generally accepted set of criteria used to make a judgment. In contrast, a criterion may represent a personal value held by individuals but not generally agreed on as essential or even necessary. Criteria can be classified as rules and requirements, a scoring structure, or specific principles and attributes individuals value. Criteria commonly used to judge educational products include utility, usability, feasibility (viability), availability, cost, effectiveness, efficiency, impact, satisfaction, preference, desirability, relevance, coherence, social acceptability, safety, and sustainability, to name a few. Evaluators may use some or all of these criteria to judge the merit and worth of an educational product or program. However, some of these will be more important than others to decision-makers. For example, decision-makers may value effectiveness and efficiency over user preference and product appeal. Indeed, a product that works is better than one that does not, even if it looks good. Still, individuals may not use a product they find unappealing even if it is more effective than other more attractive products. So while it may be unimportant to decision-makers that users like the look of a product, it still could be an important criterion for evaluating a product – or not.
Once evaluation questions have been established, an evaluator must determine what data is needed to answer the evaluation questions. They must then propose sensible data collection procedures that could be used to capture the required data. Proposed methods will include identifying the sources (e.g., the people, existing data, documents) from which the data will be retrieved. It will also describe the procedures required to obtain the data and how the evaluator will analyze the data. Precise detail need not be included in a proposal. Still, enough specificity should be provided to convince the client that the data collection efforts will likely produce enough relevant information to answer each of the evaluation questions. For example, it would be insufficient for a proposal to only say that interviews and surveys will be used to collect data. The proposed methods should include some details regarding who will be interviewed and surveyed, what instruments will be used, how data collection instruments will be developed, and how the data obtained will be analyzed and used to answer the evaluation questions.
Developing a budget can be tricky, especially when the evaluator is an external evaluator responding to a request for proposals. Budgeting too little will mean you may be doing the evaluation for free, or worse, you may have to pay out of pocket to complete the evaluation you contracted to do. Asking too much may diminish your chances of obtaining work now or in the future. Understanding actual costs and how much you are willing to charge for your services is essential to the budgeting process. However, budgeting issues can be alleviated to some extent by asking clients upfront what they have budgeted for the evaluation. Knowing beforehand any budget restrictions can allow the evaluator to plan activities and limit the scope of the evaluation accordingly. Giving a client the option to include or exclude certain costly activities they are unwilling or unable to pay for can also help.
As mentioned previously, there are soft costs and hard costs that need to be considered. Soft costs represent the evaluator's time, and hard costs represent the cost of travel, facilities, equipment, and supplies. These costs are sometimes forgotten when the evaluator is an internal service provider, but they exist nonetheless. When working as an internal evaluator, budgeting usually entails specifying the personnel required and estimating the time needed to accomplish each task.
Often external evaluators have to submit a proposal and hope for the best. However, the evaluator can sometimes negotiate terms and requirements with the client beforehand or after submitting a draft proposal. Working collaboratively with the client on the proposal's details can be a very productive way to develop an evaluation plan. Once a proposal is approved, a more detailed management plan will likely be needed.
Cronbach, L. J., & Shapiro, K. (1982). Designing evaluations of educational and social programs. Jossey-Bass,.
Patton, M. Q. (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use. Guilford press.
Stufflebeam, D. L., & Coryn, C. L. (2014). Evaluation theory, models, and applications (Vol. 50). John Wiley & Sons.
This content is provided to you freely by EdTech Books.
Access it online or download it at https://edtechbooks.org/eval_and_design/evaluation_planning.