Introduction

The Learning Engineering Evidence and Decision (LEED) tracker is a tool used by members of the Residential Education team at the Massachusetts Institute of Technology (MIT) when designing learning experiences as a means to record, revisit, and iterate upon actionable design decisions in an evidence-grounded way. While the LEED tracker was originally developed and adapted within the MIT context for designing instructor support resources, the general form and function of the tool emphasize writing down and justifying implementable design decisions. Inspired by a learning engineering approach to supporting learners (Goodell & Kolodner, 2022), the goal of maintaining a LEED tracker is to make explicit a team’s thought processes, different channels of influence such as evidence from the learning sciences, instrumented data, and contextual factors in a way that is easy to revisit and update during cycles of design iteration. 

The challenge: Design decisions and managing complexity

Early work exploring design decisions within Instructional Design (ID) focused on understanding how heuristics and other procedural approaches to design work may fail to fully account for design decisions (Kerr, 1983). More recent work has argued that designers employ decision-making in cycles that are contextually situated and often based on constraints (Jonassen, 2008) and that further efforts are needed “to guide how instructional designers engage in decision-making while designing for situated, real-world experiences” (Stefaniak, Tawfik, & Sentz, 2023). Communities outside of ID have noted the importance of design decisions as a way for practitioners to justify and ground their work in research (Kolodner et al., 2003) and from various sources (Hazelrigg, 1998). 

Taken together, it is clear that practitioners who design learning experiences and products are often situated in increasingly complex contexts where defining the learning-related challenge, understanding learners and stakeholders, and being informed by theory and evidence must all be taken into consideration when planning and implementing a solution. Yet the practical work of how such design decisions can be recorded and maintained for research and practical purposes is drastically underreported in the literature. 

The framework: Learning engineering

The Residential Education team at MIT supports instructors and course teams across the institute to improve teaching and learning with digital technologies. Similar to many other industries and organizations, this work is situated in a culturally unique environment where many contextual factors and specific goals for learners and stakeholders shape each learning challenge. Encouraging systematic practices within a contextually conscious frame of designing for learners and learning, the learning engineering process (Kessler et al., 2022) guides the team’s work and facilitates how the team navigates complexity across design iterations. Within a complex system, design decisions are not always made in a linear fashion, nor do they happen completely independently of one another; rather, they occur in nested or concurrent learning engineering processes (see Figure 1). When the complexity of a context grows but keeping learners at the center remains a priority, maintaining a record of design decisions and related justifications becomes necessary for understanding how each piece of the design goes together and why it works.

Figure 1

Nested Learning Engineering Processes Model (Kessler & Totino, 2023)

The solution: LEED tracker

The LEED tracker is one tool that exposes thinking, reasoning, and evidence in a systematic way throughout the design of a learning experience or product. Though adjusted as needed depending on the project, a LEED tracker is a tangible artifact that generally includes labeled columns and rows, such as in Excel or a table in a word processor (see Figure 2). Its form is purposefully low-tech, low-overhead, and shareable to encourage the habit of continually returning to it and consistently using it.

Decisions that get enacted in a design are recorded in the LEED tracker (Figure 2, Column A), as well as the source of influence that supports the decision to include that element or strategy in the learning experience (Figure 2, Column B). Underscoring the importance of evidence and reasoning to inform decisions, a justification is also recorded (Figure 2, Column C), which is often a part of the thought process that loses critical nuance or is difficult to communicate unless written down. When thought processes become explicit, it is easier to know what further evidence needs to be collected from the implementation to understand whether the solution is effective for learning (Figure 2, Column E). The goal is to revisit the tracker on an ongoing basis as teams examine what worked and has not worked in a design (Figure 2, Column D), refer back to why a design element was implemented in the first place (Figure 2, Column C), and iterate while keeping the learners’ needs at the forefront (Figure 2, Column F).

Figure 2

Suggested columns to include in a LEED tracker

LEED tracker key benefits and examples

This section describes three key benefits of the LEED tracker while highlighting specific features of the tracker through examples from the MIT context.

Key benefit 1: Decisions and justifications become concrete

The LEED tracker concretizes decisions grounded in an understanding of pedagogical approaches and context. By giving teams a place to record not just the decisions that get enacted in the design but also the reason behind each decision, the tracker elicits evidence as the source of actions taken. Several sources of evidence can inform the creation of a learning experience or solution, including research, literature, learning design principles, contextual factors, platform data, stakeholder interviews, learner feedback, and one’s own experiences and observations. 

The example in Figure 3 is based on a LEED tracker used for the design of technology training for instructors. Being informed by the learning sciences as well as contextual factors like constraints of the training platform helped generate design ideas and supported decisions that centered on the learners’ (instructors’) needs. Having this information recorded made actions throughout the design process more concrete and justifiable compared to acting on instinct, and additionally gave space to acknowledge any alternatives that were considered. 

Figure 3

Excerpt adapted from a LEED tracker for the design of an instructor training

Some projects, such as the example in Figure 4, involve managing multiple channels of influence on the design of one solution (in this case, a resource for instructors about using a learning management system) that affect decisions of varying “grain size” from overall strategy to specific content choices. In a highly iterative space, such as one in which technology is constantly being updated, LEED tracking helps teams recall the foundations and intended functions of prior decisions before further changes are made to the implementation.

Figure 4

Excerpt adapted from a LEED tracker for the design of an LMS resource site for instructors

Key benefit 2: Decisions become entry points to iteration, supported by new evidence

With these tracked decisions as potential entry points to further improve the learning experience, the LEED tracker facilitates iteration informed by multiple streams of data and feedback collected about the learner experience. The tracker is revisited on an ongoing basis as teams examine what has worked or not worked in a design, refer back to why a design element was implemented in the first place, and iterate while continuing to centralize the learners’ needs. Within the learning engineering framework, instrumenting to collect data from the learning experience is part of the creation process, after which data from the implementation is analyzed to inform improvements to the design. 

Figure 5 shows an example of a LEED tracker from the design of a training for teaching assistants. Recording decisions about the enrollment strategy for the optional training and decisions about the actual training experience allowed the team to easily revisit and rapidly iterate on the enrollment strategy when actionable data related to that particular sub-process became available first. Unlike other approaches, learning engineering opens the door for practitioners to make design decisions on aspects of implementation or the larger systems in which the learning experiences are taking place (Kessler et al., 2022). The example in Figure 5 shows how the LEED tracker can maintain a log of such decisions. 

Figure 5

Excerpt adapted from a LEED tracker for the design of a teaching assistant training

Key benefit 3: Explicit evidence and reasoning guide stakeholder conversations

Because the LEED tracker gives space to track both a decision and its justification, it serves as a communication tool to provide stakeholders from a range of disciplines with a shared understanding of the central challenge for learners. The content of the tracker provides a clear “what” and “why” to shape discourse with subject matter experts, team members, learners, and other stakeholders who have questions, ideas, and feedback about the design and expected outcomes. 

A recent initiative at MIT involved the team working with instructors and student learning technologists to improve the design of courses using digital technologies. Figure 6 shows an example of a LEED tracker from one of these projects that labels decisions to incorporate certain digital tools in the course as being aligned with the instructor’s goals, among other influences such as the learning technologist’s familiarity with the student experience. Since the instructor is ultimately responsible for implementing changes to the course, and since the time between instructor meetings can sometimes be weeks apart, the LEED tracker is helpful in grounding instructor conversations in previously stated goals. 

Figure 6

Excerpt adapted from a LEED tracker for the redesign of a music course at MIT

Conclusion and implications for practitioners

The LEED tracker is a tool that reflects a design process deeply grounded in learning engineering, where decisions, sources of evidence, and justifications are made concretely in order to facilitate iteration and communication in a way that addresses the complex nature of challenges to learners and learning. Through various examples from MIT, practitioners in other industries and organizations are encouraged to think of ways to adapt the LEED tracker to their practice, or consider similar approaches to expose their thinking in a systematic way throughout the design of a learning experience or product for different audiences. While specific elements of the LEED tracker, like column titles, labels, and the content in the tracker, will vary depending on project context, the general form and function of the LEED tracker make it one possible tool for teams to address challenges in an evidence-grounded way.

References

Goodell, J., & Kolodner, J. (Eds.). (2022). Learning Engineering Toolkit: Evidence-Based Practices from the Learning Sciences, Instructional Design, and Beyond (1st ed.). Routledge. 

Hazelrigg, G. A. (1998). A framework for decision-based engineering design. Journal of Mechanical Design, 120(4) (1998), pp. 653-658. 

Jonassen, D. H. (2008). Instructional design as design problem solving: An iterative process. Educational Technology, 48(3), 21-26. 

Kerr, S. T. (1983). Inside the black box: Making design decisions for instruction. British Journal of Educational Technology, 14(1), 45–58.

Kessler, A., Craig, S., Goodell, J., Kurzweil, D., & Greenwald, S. (2022). Learning Engineering is a Process. In J. Goodell & J. Kolodner (Eds.). Learning Engineering Toolkit: Evidence-Based Practices from the Learning Sciences, Instructional Design, and Beyond. New York: Routledge. 

Kessler, A. & Totino, L. (2023, July 24-26). Reality is Much More Complex: Understanding and Exploring Nested Learning Engineering Processes [active learning session]. IEEE ICICLE Learning Engineering Conference 2023, Pittsburgh, PA, United States.

Kolodner, J. L., Camp, P. J., Crismond, D., Fasse, B., Gray, J., Holbrook, J., Puntambekar, S., & Ryan, M. (2003). Problem-Based Learning Meets Case-Based Reasoning in the Middle-School Science Classroom: Putting Learning by DesignTM Into Practice. The Journal of the Learning Sciences, 12(4), 495-547.

Stefaniak, J., Tawfik, A. & Sentz, J. (2023). Supporting Dynamic Instructional Design Decisions Within a Bounded Rationality. TechTrends, 67, 231–244.