3

Theories of Change in Learning Experience (LX) Design

, , &
Learning Experience DesignTheory of ChangeLogic ModelUser Experience DesignDesign ProcessHypothesis Testing
Designers’ theories about how people learn are the keys to impactful educational design. While much effort and attention is appropriately paid to usability in the development of educational tools and materials, the centrality of learning theories is often underappreciated. Learning theories, in combination with considerations of usability, form coherent theories of change. Theories of Change frame how designers expect to shift learners’ knowledge, skills, and abilities. They play out in the features included in digital learning tools and in the activities chosen for learners in technology-enhanced experiences. They are critical to recognizing whether a design is effective. The clearer the theory, the more specific and measurable the indicators; and reliable, focused measures are key to ensuring that a design is working as planned. Additionally, good measures can transform data into launchpads for design iteration. Aligning learning theory, design, and measures, however, is easier said than done. Through illustrative cases of two learning projects, this chapter gives readers useful frameworks and intuitions to approach this process. Designers will be better prepared not only to create effective experiences, but also to communicate their impact to a range of stakeholders including learners, teachers, buyers, and funders.

Author's Note

All three authors contributed equally to this chapter. Names are in alphabetical order.

1. Introduction

Theories about how people learn should drive the process of designing learning experiences. Those theories that designers hold about how learning happens help them to frame their early investigations into a specific learning problem, their successive iterations in learning design, and their repeated testing with targeted learners. Why does this matter? In a world where educational challenges are vast, historically entrenched, and highly complex, such “theories of change” help designers avoid scattershot innovation, serve as powerful anchors for the design process, and help answer the fundamental process questions "Where do I start?" and "How do I know if I’m making progress?"

Working with students in the Learning, Design & Technology (LDT) master's program at Stanford University’s Graduate School of Education (ed.stanford.edu/ldt), we have found that the degree to which students invest in developing and anchoring their design work in well researched and clearly articulated theories of change consistently predicts success. Such theories of change are powerful tools to clarify and guide our work in learning experience (LX) research and design.

2. Theories of Change

The concept of a theory of change is borrowed from the world of program evaluation. It can be used to plan, communicate, manage, and assess interventions. Philanthropists, government agencies, and policy-makers use theories of change as lenses to examine the apparent effectiveness of early-stage interventions for improving lives and livelihoods (e.g., Brest, 2010; California Department of Education, 2019).

Evaluation expert John Mayne notes that theories of change “represent how and why it is expected that an intervention will contribute to an intended result” (2015, p. 127). They can be seen as one kind of logic model. Logic models outline the inputs, activities, outputs, and outcomes that are intended (McLaughlin & Jordan, 2004). Importantly, theory-based logic models (theories of change) emphasize not only "what" will be done but "why" (Mayne, 2015; W. K. Kellogg Foundation, 2004). The approach provides insight into the different assumptions that are embedded in a strategy and should drive user testing. As learning experience designers, the focus on theory prompts us to build on existing research on how people learn. Because new information may challenge assumptions along the way, we encourage designers to revisit their theories of change often and to update them in light of new insights.

In the context of LX design, we like to illustrate the Theory of Change through a simple graphic that shows the learner, the learning outcome, and the approach that informs activities (see Figure 1). Although there are many more details that can be added (Mayne, 2015), this is a useful starting place for designers. Our focus on the learner is key to our learner-centered design philosophy. While keeping the learner at the center, we also consider the contextual factors that make learning possible (Papert, 1987). Because we hope that the outcome is an improvement, we've chosen to illustrate the change with an upward slope. This clarity of purpose helps inform both the design of prototypes and research metrics.

Theory of change diagram
Figure 1

Theory of Change

2.1. Theories Driving LX Design and UX Design

This approach to theories of change helps highlight key parallels and contrasts between user experience design (UX design) and learning experience design (LX design). Both are guided by theories that are well researched, developed, and tested. There is some overlap between the theories that guide both disciplines, but generally they are drawn from different traditions and can be used to formulate different questions.

In the case of UX design, theories from the field of user research, including theories on attention and perception, guide designers in the development of the whole user experience. They apply to educational products and experiences and impact learning, but they also apply more broadly to all products and user experiences, helping to answer core questions such as whether a product is easy to use and will sustain user engagement.

LX design is similarly informed by theories, but, in this case, researchers and designers study and leverage insights from specific theoretical frameworks that focus on how people learn, e.g., behaviorism, cognitivism, and a range of socio-cultural theories, and theories that explain core mechanics of learning at more granular scale, such as worked examples, contrasting cases, and embodied cognition (e.g., Schwartz et al., 2016).

In the Stanford LDT program, we have observed that novice designers tend to favor intuitive outcomes of engagement, likability, and usability to measure progress. All of these measures, while important to user experience (UX) and eventual product viability, should not be the sole focus of early testing. When developing tools intended to foster learning, it is paramount to explicitly define and test the learning theories on which those tools depend through deliberate learning experience (LX) design.

2.2. Key Questions for LX Design

In order to develop and communicate an effective theory of change for learning design, we need to have answers to three key questions:

LX designers seek to help learners by understanding how they learn and help teachers to identify and measure their students’ progress towards specific learning goals. Unfortunately, we have found that when connections between learning theory and design are tenuous, projects sputter and design iterations are haphazard. However, when theory and design are aligned and well-articulated, LX designers have a roadmap to guide the development, analysis, and refinement of learning designs.

In this chapter, we present the stories of two digital learning tools that were informed by very different theories of change. In Section 3, we describe Bounce (a math game tool), and, in Section 4, we describe Prevention Begins With Me (a tool to educate about AIDS prevention). These examples are drawn from projects created by past students in the LDT master’s program at Stanford University. They were selected as cases that illustrate effective testing of both LX and UX assumptions, driving modifications in design that resulted in products enjoying widespread use, documented efficacy, and longevity, which are testament to the design process. While their target learners and contexts are radically different, both projects shared a fundamental commitment to the development of strong, well-informed theories of change to guide deliberate, calculated iteration. This chapter documents the genesis of both projects and the theories of change that drove their user testing, iteration, and evolution into mature products with impact.

3. LX Design in Motion Math: Bounce

Bounce is one of several math games by Motion Math, now part of the i-Ready Learning Games suite (https://edtechbooks.org/-Hfv). Motion Math designs games to engage students in exploring a variety of foundational math concepts in a way that is both engaging and instructive. By mapping abstract concepts onto physical interactions, Motion Math aims to connect abstract ideas with concrete intuitions. Bounce was developed with a mission to help more children understand fractions.

3.1. The Need

In 2010, Motion Math Games Founders Gabriel Adauto and Jacob Klein were looking for an opportunity to set young learners on a successful path in math. Asking teachers to name the most difficult concepts to teach and for children to understand, they learned that fractions are a stumbling block for many elementary students. A review of the literature confirmed that poor or incomplete understanding of fractions is associated with struggles later on. For example, learners who have only a superficial understanding of fractions often find it difficult to wrap their heads around algebra (Booth & Newton, 2012), a subject that has long been a gatekeeper for college readiness (Spielhagen, 2006).

So why are fractions so hard to teach? This problem was not driven by lack of access to curriculum since virtually all comprehensive elementary school math textbooks cover fractions (Alajmi, 2012). However, understanding the magnitude of rational numbers is inherently difficult and their many symbolic representations create confusion (Siegler & Lortie-Forgues, 2017). The Motion Math team wanted to create a tool that would help students develop a strong foundation with fractions and a positive experience in learning mathematics.

3.2. Theory of Change

For Adauto and Klein, translation turned out to be an important concept. By reviewing this body of learning sciences research, they discovered early on that one of the most effective approaches for mathematics learning is to make connections between different representations of a concept. For example, SimCalc software (https://simcalc.sri.com/) shows learners what an equation means in a table of values, a graph, and an animation. As the learner changes a value in one place, the other representations are updated to match. This software, with teacher training, had been shown to substantially increase learners’ understanding of algebraic concepts (Roschelle et al., 2010). Inspired by research like this, the team of LDT students decided to help learners translate the symbols of fractions to concrete representations of physical objects from the real world.

In early grades, translating between concrete to abstract is often done through objects known as manipulatives. Manipulatives are physical objects like blocks or tokens or pie pieces that make the symbolic numbers meaningful. Interacting with objects is one of the ways that learners make sense of math problems (Martin & Schwartz, 2005). This is one example of what has been referred to as “embodied cognition” (Wilson, 2002), the notion that one’s thinking is tied to and enhanced by the body’s physical interactions with its environment.

Unfortunately, the use of blocks can also lead learners to misunderstand the nature of fractions. When the examples are always given as parts of a whole, for example, learners often get stuck when they’re later asked to multiply or divide—it is more helpful if the fractions are seen as parts of a group (Ball, 1992). When devising a learning plan on a topic, it is important that designers choose a range of examples that avoid some of the traps that can box learners into unproductive ways of thinking. And it’s critical to assess whether the students are making the correct connections along the way. Blocks, like all manipulatives, are limited in the feedback they provide.

To sidestep these learning pitfalls, guided practice can offer learners meaningful feedback to develop their expertise (Ericsson et al., 1993). To encourage practice, the challenge should be at the appropriate level. An activity that matches one’s competence to the task’s difficulty level can lead to a pleasant challenge or state of flow, the feeling of losing oneself in an activity (Csikszentmihalyi, 1997). Many games induce this state when they challenge players, while also providing the resources they need to succeed (Gee, 2005). Interactive computer games can provide guided practice with meaningful feedback in a way that matches the learner’s ability and adapts to their growing mastery.

In the late 2000s, a new set of devices transformed the opportunities for digital-physical manipulation. Adauto and Klein realized that sensors in the new Apple iPhone[1] would allow learners to use their bodies to interact with a range of representations of mathematical concepts. In addition, computers can generate tasks based on past performance, to create appropriately challenging prompts. With the knowledge of the device's potential, Adauto and Klein planned to design an engaging smartphone app that would help learners understand what fractions really mean. The challenge was to allow learners to interact with a variety of representations—including fraction symbols, pies, decimals, percents and number lines—that built an accurate foundation to support them in their future math learning. See a simple illustration of this theory of change in Figure 2.

Theory of change in bounce diagram
Figure 2

Theory of Change Applied in the Game Bounce

The team started to prototype a game that would map different representations (pies, percents, decimals) to a number line. Through tilting the phone one way or another, learners would “bounce” a fraction on a number line. The vision was to create a series of fraction-mapping tasks that could be dynamically updated in a game. The game would take learners through levels that explored relationships between different types and representations of fractions and number lines with different ranges. A screenshot of the final game is presented in Figure 3.

Fraction bounce interface
Figure 3

Fractions in the Game Bounce

Note. In this screenshot from the game Bounce, the learner is asked to tilt the smartphone to place the fraction 1/3 on a number line representing 0 to 1 (Motion Math, 2019).

3.3. Testing Hypotheses

There were many ways that this solution could fall short, if not well-designed. A look back at the theory of change shows several testable assumptions. Did the physical interactions work on this new technology? Could the team build an app that was appropriately challenging and, therefore, induce a flow state? Was the feedback appropriate and helpful? And ultimately, were the learners successful in manipulating fractions, developing a positive affect, and persisting in learning math? These questions, derived from their theory of change, led the team to a series of prototype tests.

3.3.1. Did the Physical Interactions Work Using this Technology?

A key assumption in the theory of change for Bounce was that physical movements would aid in understanding math. The first question, then, was whether the movements would work as planned on a digital platform. The team first used a paper cardboard mock-up with a nickel on a string to represent the bouncing ball, which learners could manipulate with ease. However, transitioning to an interactive app presented new challenges. Smartphones were brand-new at the time, and most people had a mental model of clicking on a screen. Early app testing convinced the team that, while tilting a device to interact with a game was novel and unintuitive to some, it was a promising modality that worked well once understood. As a result of this early testing, the designers developed a brief tutorial showing learners how to tilt to aim the ball before they moved on to the first fraction.

3.3.2. Was it Fun?

Another key assumption was that the game would keep learners engaged. If the game induced a flow state—if it was fun—it would propel the development of their mathematical intuitions. The goal would be to find the right level of challenge. To test this, they had kids in 2nd through 6th grade play a simple version of the game. Through these learner tests, they quickly identified that the game was at an appropriate level of challenge for 4th- and 5th-graders. Stakeholder interviews with teachers and curriculum standards helped refine the specific learning goals for the game.

It was important to the team that the mathematics were endogenous to the game, rather than a gateway test that allowed users to advance to the next level (Malone & Lepper, 1987). But would interacting with Bounce actually develop kids’ interest and ability in math? Yes, measurably so. In 2013, a school-based study of Bounce[2] found that learners showed better understanding of fractions, better attitudes, and willingness to complete more math problems after using Bounce than after regular instruction (Riconscente, 2013). But, long before that study's results, the team had evidence that they were on the right track through observations of children playing early versions of the game. Though early prototypes had fewer levels and less polished graphics, the learners smiled and laughed while playing. One girl called it “a kid’s dream of school” (Motion Math Games, 2010). They wanted to play more.

3.3.3. Was the Feedback Actionable?

An important reason for using a digital rather than physical tool was the ability to provide timely and actionable feedback. The team invested a great deal of effort into getting the “hints” right. Powerful scaffolding (Puntambekar & Hubscher, 2005) should include ongoing diagnosis and graduated assistance and should fade out if no longer needed. Through cognitive task analysis interviews with expert educators, the team developed hints that provide progressively more help to guide learners (see Figure 4). User tests were then conducted with learners to identify a small set of useful hints.

Bounce number line
Bounce number line
Bounce number line
Bounce number line
Figure 4

A Number Line in the Game Bounce

Note. These screenshots illustrate how learners are shown progressive hints to guide them toward the correct placement of the fraction 1/3. If not successful, learners will be shown the correct answer before going on to the next problem (Motion Math, 2019).

Through focusing on questions derived from their theory of change, the Motion Math team was able to identify and validate their assumptions early. Their process generated insights to inform their key game mechanics and learning targets, leading to an engaging and effective game.

In the next section we present a tool designed to solve a very different learning challenge. As with Motion Math Bounce, the design and development of TeachAids Prevention Begins With Me illustrates the value of a well-articulated theory of change.

4. LX Design in TeachAids: Prevention Begins With Me

TeachAids (teachaids.org) is a non-profit organization located in Silicon Valley that develops technology-based education products to address persistent global health problems. Prevention Begins With Me[3], the innovative product that launched the organization in 2009, has been used to teach hundreds of millions of people around the world about HIV/AIDS prevention.

4.1. The Need

In 2007, national and international health organizations were struggling to find effective approaches to teach HIV/AIDS prevention in low-income countries. The need for education was overwhelming. In 2007, 33.2 million people around the world carried the HIV virus, and the number was quickly rising (UNAIDS, 2007). Ninety-five percent of those cases came from countries where deep and endemic poverty makes education on any topic difficult (Noble, 2007, as cited in Sorcar, 2009). Misinformation about the disease was rampant, and national and international leaders were eager for solutions.

What makes education on HIV/AIDS especially difficult is the range of cultural taboos on teaching topics related to intravenous drug use and sexual practices, especially homosexuality, pre-marital sex, adultery, and commercial sex work. In India, for example, the central government worked with national and international experts on the disease to develop an official HIV/AIDS curriculum, but many of the country’s states banned it from public schools. To bypass the obstacles in schools, the government also developed mass media campaigns. Unfortunately, these also posed challenges as they lacked depth, could not be targeted to specific learners, and were very expensive (Sorcar, 2009).

4.2. Theory of Change

In the learning sciences, researchers and practitioners study and leverage insights from a range of theoretical frameworks that describe how people learn, e.g., behaviorism, cognitivism, constructionism, and others. In particular, socio-cultural theories of learning build on the recognition that learning is more than a solitary act performed by separate individuals and that learning is a social process inseparable from its cultural context. In 2006, TeachAids founder Piya Sorcar hypothesized that a fundamental barrier to effective education about HIV/AIDS was the range of cultural taboos related to talking about sexual practices, especially in India and other low-income countries (Sorcar, 2009). With socio-cultural learning theories in mind, Sorcar began an extended exploration of the teaching of taboo topics.[4]

Sorcar’s design process aimed at testing a clear theory of change: that to teach HIV/AIDS prevention effectively, especially in culturally conservative countries like India, the program must be designed specifically to circumvent social taboos (Sorcar, 2009). This high-level theory of change is illustrated in Figure 5.

Prevention theory of change

Figure 5

A Simple Theory of Change in Prevention Begins With Me

4.3. Preliminary Hypothesis Testing

The first phase of testing involved an extensive competitive analysis to understand the strengths and weaknesses of existing curricula and further understand what elements contribute to emotional safety. While competitive analysis is best known as a business technique for identifying weaknesses, gaps, and opportunities in markets (Fleisher & Bensoussan, 2015), it can also serve as a powerful early-stage tool for LX designers. In particular, before one has a prototype to test, analyzing existing products and even testing them with users to understand their shortcomings can provide critical insights to inform the theory of change that undergirds one’s own designs. To understand the existing solutions, Sorcar teamed up with a range of collaborators to conduct semi-structured interviews with young adults, schoolteachers and administrators in India, experts in public health and medicine, and others. The interviews were designed to map existing efforts to teach HIV/AIDS prevention, determine what students had gained from those teaching efforts, and identify which topics made them most uncomfortable.

To confirm and build upon insights from these interviews, team members also developed surveys through a process that went through several iterations to make them acceptable to administrators. Surveys were an important method that allowed private and anonymous written responses to culturally sensitive questions. About 200 Indian college students responded, providing crucial insights into their questions and misconceptions. Concurrently, in interviews with medical experts, the team developed its understanding of what young people need to know about HIV/AIDS prevention (Sorcar et al., 2017). This work uncovered several issues.

Testing students showed that misinformation about HIV/AIDS was widespread even after they had examined instructional materials. HIV/AIDS prevention is too often taught as a disconnected set of “DOs” and “DON’Ts”, which puts stress on learners’ cognitive capacity and makes retrieval difficult (Sorcar, 2009). Not only did learners have significant misunderstandings about the nature of the disease and its transmission vectors, but they tended to think of it primarily in moralistic rather than biological terms, associating it with behavioral practices generally condemned in their society. A focus on biology, however, provided an opportunity to address it as a standard topic in Indian schools (Sorcar et al., 2017).

Another challenge was the unwillingness of teachers to discuss the subject with their students. Sorcar found that, due to their discomfort, teachers commonly avoid discussing topics related to sex with their students. In one state in Northern India, teachers burned sex education materials in bonfires. Students also tend to be embarrassed by any discussion of these topics.

Learners’ and administrators’ reactions to various styles of illustration in existing materials showed that many were uncomfortable with more explicit visual depictions. A tension emerged between visual accuracy, which can be obtained through photos and detailed drawings, and the comfort of learners (Sorcar, 2009; Sorcar et al., 2017). Similarly, students were often more comfortable using metaphors and euphemisms than the more accurate but culturally objectionable terms.

4.4. Revised Theory of Change

Sorcar and her team used these insights from early research to develop a more nuanced theory of change (see Figure 6). This deep understanding of the problem inspired the features and frameworks that the team then proceeded to build and test.

Revised prevention theory of change
Figure 6

A Revised Theory of Change in Prevention Begins With Me

The design brief based on this theory of change featured an interactive website rather than live teaching, recall based on biology rather than morality, a visual treatment based on animations rather than live video, and description using comfortable verbal and visual metaphors rather than medically explicit imagery and language. The product also featured cultural, gender, and age affinity between on-screen characters and target learners to increase the sense of emotional comfort while learning about a culturally sensitive topic (see Figure 7).

Game character in prevention begins with me
Figure 7

Characters in Prevention Begins With Me

Note. This screenshot shows characters in Prevention Begins With Me. The design team carefully matched visual and voice characteristics of the animated characters in the program to those of its targeted learners. Designers created separate scenarios based on gender, with a male doctor and a male patient for male learners, and a female doctor and a female patient for female learners (TeachAids, 2019).

The choice of an interactive web site was driven by the desire for learners to study on their own and spare them social discomfort or public embarrassment. This approach also helped ensure uniformity, accuracy, and completeness in the presentation of information to all students. Importantly, the choice of a solution based on networked technology facilitated the design team’s efforts to gather data on various aspects of student use, interaction, and learning, which the team then analyzed for formative evaluation of successive prototypes (Sorcar, 2009).[5]

4.5. Further Hypothesis Testing

As the team began to build out prototypes, they systematically sought feedback on how their designs were received by the target learners. The team began with low resolution paper prototypes, progressively refining them through PowerPoint slide decks and then through higher resolution computer animations. Using about 150 prototypes, team members answered a series of questions related to their theory of change: how comfortable learners were with different visual and verbal metaphors, which ones worked best to communicate accurate information, and which were most effective at building long-term understanding and retention (Sorcar et al., 2017).

4.5.1. Are the Visual Depictions Approachable? Are the Euphemisms and Metaphors Accessible?

Prior to the development of the first version of the product, the team conducted a detailed survey of targeted learners to test their reactions to various styles of visual depiction. The team was able to develop a series of animated drawings to communicate accurately without unsettling learners and school administrators (see Figure 8).

Female doctor
Figure 8

A Female Doctor in Prevention Begins With Me

Note. This screenshot illustrates a female doctor reviewing the 3-point mantra for possible HIV infection (TeachAids, 2019).

The team developed a series of culturally familiar verbal and visual metaphors and euphemisms. In one scene, rather than displaying a couple publicly kissing, the animation shows the couple coming very close to a kiss while the camera gently pans upward to a pair of pecking lovebirds, a Bollywood convention easily recognizable by target learners. Rather than present even a simplified depiction of physical intimacy, illustrators portrayed a woman sitting on a bed, dressed in a gown and surrounded by a traditional floral arrangement indicating that it was her wedding night (see Figure 9).

Pair of doves

Woman sitting on bed

Figure 9

Metaphors in Prevention Begins With Me

Note. These screenshots illustrate how the design team for Prevention Begins With Me leveraged culturally sensitive metaphors and euphemisms taken from Bollywood movies (TeachAids, 2019).

4.5.2. Are the Students Learning From This Approach?

Once the curriculum had been developed to the point where it could be presented on PowerPoint slides, testing included formal learning assessments given before and after the experience to measure knowledge gains and changes in attitudes.

After the first complete version of the project, pre-post experimental studies with 295 students in the 11th and 12th grades showed statistically significant gains in their understanding of HIV/AIDS. In addition, stigma towards the disease was reduced and attitudes toward people with HIV/AIDS became more positive. For example, after the intervention learners were more likely to agree with the statement “It’s okay to be friends with someone who is HIV-positive” and disagree with the statement “People with HIV/AIDS should not be allowed to work/study in public schools.” Follow-up assessments conducted a month after the intervention showed persistence in the learning and attitude gains (Sorcar, 2009).

Thanks in part to this evidence of efficacy, the product has since been deployed in tens of thousands of schools throughout India and beyond (Sorcar et al., 2017). Over time, the theory of change has been further refined and now includes partnering with celebrities to reduce stigma around the topic. The founders of TeachAids have partnered with more than two hundred organizations and adapted Prevention Begins With Me for use in over 80 countries around the world. Recently, the team has adapted the model in the development of a new product, CrashCourse, which provides students, parents, and coaches with the latest medical information about prevention and treatment of concussions.

5. Discussion, Implications, and Conclusion

For those designing educational solutions for learning challenges, this chapter is intended as a guide for answering the essential process questions "Where do I start?" and "How do I know if I’m making progress?" A designer's theory of change leads directly to features and interactions to test out. If the goal is to design a solution with the right activities to yield a particular learning output, early prototypes should be designed to deliberately test the learning theories underlying that solution. Progress should be measured via the prototypes’ failure or success in reaching the desired learning outcomes.

Hypothesis testing is key to designing impactful interventions and can be continued past initial design to product viability. A theory of change is key to developing such hypotheses. In recent years, the value of hypothesis testing has been demonstrated by the startup business community. Perhaps most notably, Eric Ries, in The Lean Startup (2011), argues that hypothesis testing should drive early startup efforts. More recently, a group of Italian researchers performed a randomized controlled trial with 116 startup companies to test the impact of training company founders to use hypothesis-testing to drive their innovation. The trial found hypothesis testing improved business performance, frequency of pivoting, and company longevity (Camuffo et al., 2019). This mindset of using a theory of change to develop hypotheses was key to the long-term success of both Motion Math and TeachAids. In both cases, projects initiated by students in a master’s degree program grew into mature products reaching millions of learners around the world.

A retrospective analytical account, such as this present one, can give the misleading impression that UX and LX design processes are linear and tidy. They generally are not. Rather, design teams often start with theories of change that are full of both productive and misguided assumptions. Through a systematic process of testing each element, teams can ask the right questions, challenge their assumptions, and produce the kinds of insights that propel successful iteration. This highly recursive process requires systematic thinking, humility, and resilience. It also requires avoiding the trap of focusing solely on UX considerations, a mistake all too common among novice designers. In both cases highlighted here, LX considerations were paramount. From project conception to product launch, the designers explicitly defined, tested, and iterated on the learning theories embedded in their theories of change. The interplay of learners, goals, and mechanisms required a long and iterative process of theorizing, prototyping, testing, and refining, as the designers used increasingly specific indicators and measures of both learning and usability to drive their design decisions.

The TeachAids and Motion Math stories shine light on the ways that theories of change and the assumptions behind them can drive the important questions that are asked by LX designers, as well as the iterative designs and learner testing protocols that are developed to answer them. In both cases, it was through continually asking and testing these theory-driven questions that elegant and effective learning tools methodically took form. The designers clearly identified target learners, learning goals, and mechanisms from the learning sciences to develop a theory of change to test and refine over multiple iterations. As different as these tools are, they are similar in their illustration of how a well researched, thoughtfully developed, and clearly articulated theory of change drives productive LX design.

References

Alajmi, A. H. (2012). How do elementary textbooks address fractions? A review of mathematics textbooks in the USA, Japan, and Kuwait. Educational Studies in Mathematics, 79(2), 239–261.

Ball, D. L. (1992). Magical hopes: Manipulatives and the reform of math education. American Educator: The Professional Journal of the American Federation of Teachers, 16(2), 14–18, 46–47.

Booth, J. L., & Newton, K. J. (2012). Fractions: Could they really be the gatekeeper’s doorman?. Contemporary Educational Psychology, 37(4), 247–253.

Brest, P. (2010). The power of theories of change. Stanford Social Innovation Review, 8(2), 47–51.

California Department of Education (2019, July 10). Evidence-based interventions under the ESSA. https://edtechbooks.org/-nibs

Camuffo, A., Cordova, A., Gambardella, A., & Spina, C. (2019). A scientific approach to entrepreneurial decision making: Evidence from a randomized control trial. Management Science, 66(2), 564–586.

Csikszentmihalyi, M. (1997). Finding flow: The psychology of engagement with everyday life. Basic Books.

Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363–406.

Fleisher, C. S., & Bensoussan, B. E. (2015). Business and competitive analysis: effective application of new and classic methods. FT Press.

Gee, J. P. (2005). Learning by design: Good video games as learning machines. E-learning and Digital Media, 2(1), 5–16.

Klein, J. (2010). Motion math: A new genre of learning game [Video]. Vimeo. https://vimeo.com/13612582

Malone, T. W., & Lepper, M. R. (1987). Making learning fun: A taxonomy of intrinsic motivations for learning. Aptitude, Learning, and Instruction, 3, 223‐253.

Martin, T., & Schwartz, D. L. (2005). Physically distributed learning: Adapting and reinterpreting physical environments in the development of fraction concepts. Cognitive Science, 29(4), 587–625.

Mayne, J. (2015). Useful theory of change models. Canadian Journal of Program Evaluation, 30(2). 

McLaughlin, J. A., & Jordan, G. B. (2004). Using logic models. Handbook of practical program evaluation, 2, 7–32. https://edtechbooks.org/-jTLh

Nathan, M. J., & Alibali, M. W. (2010). Learning sciences. Wiley Interdisciplinary Reviews: Cognitive Science, 1(3), 329–345.

Papert, S. (1987). Information technology and education: Computer criticism vs. technocentric thinking. Educational Researcher, 16(1), 22–30. 

Puntambekar, S., & Hubscher, R. (2005). Tools for scaffolding students in a complex learning environment: What have we gained and what have we missed?. Educational Psychologist, 40(1), 1–12.

Riconscente, M. M. (2013). Results from a controlled study of the iPad fractions game Motion Math. Games and Culture, 8(4), 186–214.

Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation to create radically successful businesses. Crown Books.

Roschelle, J., Shechtman, N., Tatar, D., Hegedus, S., Hopkins, B., Empson, S., Knudsen, J., & Gallagher, L. (2010). Integration of technology, curriculum, and professional development for advancing middle school mathematics: Three large-scale studies. American Educational Research Journal, 47(4), 833–878.

Schwartz, D. L., Tsang, J. M., & Blair, K. P. (2016). The ABCs of how we learn: 26 scientifically proven approaches, how they work, and when to use them. WW Norton & Company.

Siegler, R. S., & Lortie-Forgues, H. (2017). Hard lessons: Why rational number arithmetic is so difficult for so many people. Current Directions in Psychological Science, 26, 346–351.

Sorcar, P. (2009). Teaching taboo topics without talking about them: An epistemic study of a new approach to HIV/AIDS prevention education in India [Doctoral dissertation, Stanford University]. Stanford Health Policy. https://edtechbooks.org/-jYa

Sorcar, P., Strauber, B., Loyalka, P., Kumar, N., & Goldman, S. (2017). Sidestepping the elephant in the classroom: Using culturally localized technology to teach around taboos. CHI '17: Proceedings of the 2017 CHI conference on human factors in computing systems (pp. 2792–2804). ACM.

Spielhagen, F. R. (2006). Closing the achievement gap in math: The long-term effects of eighth-grade algebra. Journal of Advanced Academics, 18(1), 34–59.

UNAIDS (2007, July 4). 2.5 million people living with HIV in India. https://edtechbooks.org/-Qht

Wilson, M. (2002). Six views of embodied cognition. Psychonomic Bulletin & Review, 9(4), 625–636.

W. K. Kellogg Foundation (2004). W. K. Kellogg Foundation Logic Model Development Guide. W. K. Kellogg Foundation. https://edtechbooks.org/-Rrox

Acknowledgements

We want to thank the many wonderful students in the Stanford LDT program for inspiring us, and to the LDT cohort of 2020 for providing feedback on early versions of this chapter. Special thanks to Piya Sorcar and Jacob Klein for reviewing the TeachAids and Motion Math cases, and for providing us with images.


[1] The Apple iPhone was first released in June 2007. The first iPad was released in April 2010.

[2] Now known as “Bounce,” the first game was known at the time simply as “Motion Math.”

[3] Now known as “Prevention Begins With Me,” the first app was known at the time simply as “TeachAIDS.” The organization rebranded as “TeachAids” and has since launched an additional product line known as CrashCourse, which was developed through a methodology adapted from the Prevention Begins With Me body of work. It provides students, parents, and coaches with the latest medical information about prevention and treatment of concussions.

[4] Sorcar examined the teaching of taboo topics in other contexts as well for her doctoral dissertation. These included teaching about breast cancer in Native American communities and teaching about maternal health in Mali.

[5] Interestingly, a strong demand arose after the launch of the curriculum for a less interactive, more linear version. As the success of the intervention spread, it was requested in contexts with less access to technology and the internet. In response, the team developed a version that could be played like a movie.

Keith Bowen

Stanford University

Keith Bowen is a researcher and instructor at Stanford University who works at the intersection of technology, Learning Experience Design (LX Design), and education. He has extensive experience developing and studying innovative programs that blend best practices in teaching (e.g., collaborative learning, cross-cultural learning, and problem-based learning) with emerging technologies that are both scalable and sustainable (e.g., online platforms, digital cinema, multiplayer simulation, and virtual reality). He is currently studying the design and impact of educational technology programs that connect students in high-income and low-income countries, with special focus on fragile states and zones of conflict. Working with faculty partners at Stanford, as well as other universities in the US, UK, Australia, Lebanon, and Pakistan, he has designed, developed, and studied a series of Virtual Student Exchange courses in the fields of global health, international education, and international journalism. These programs have resulted in higher student engagement, reduction in stereotypes about the other, improvement in attitudes towards the other, building of relationships across cultures, and development of professional capacity in global environments. To these efforts, Bowen brings over 10 years of experiences training practitioners in Eastern Europe, the Balkans, South Africa, Rwanda, Sudan, Nepal, Iraq, Afghanistan, and Pakistan, as well as teaching students in the U.S. and other Western countries who aspire to work overseas in government, non-profits, education, and the private sector. He has developed educational programs that have been completed by tens of thousands of learners, and he has had several educational documentaries distributed by the Discovery Channel.
Karin S. Forssell

Stanford University

Dr. Karin Forssell is the director of the Learning, Design and Technology (LDT) master's program and lecturer at the Stanford Graduate School of Education. Dr. Forssell also directs the GSE Makery, a maker space where students and faculty can explore how making things helps people learn. In her courses, students learn to use research from the learning sciences and learning-centered design processes to create effective digital tools. Her current interests include teacher technology adoption, maker space education, and parenting in a digital world. Dr. Forssell earned her bachelor's degree in linguistics, master's degree in teacher education, and doctorate in learning sciences and technology design at Stanford University.
Soren Rosier

Stanford University

Soren Rosier is a researcher and instructor at Stanford’s Graduate School of Education. Soren’s research focuses on augmenting classroom learning through building and studying technology that facilitates more meaningful interactions within communities of learners. The specific focus of his research is how children teach each other and how to train effective peer teachers. In 2017, he developed PeerTeach, a web-based application where students practice using evidence-based teaching moves and track their progress as they become increasingly adept at making strategic teaching decisions. PeerTeach is currently being used in classrooms across the Bay Area. After receiving his undergraduate degree from Harvard University, Soren taught middle school in Oakland, California, before joining the Center for Technology in Learning at SRI International. At SRI, his research focused on the development and evaluation of teacher professional development programs and blended learning applications. Through his work as a design based researcher and teacher of graduate students in Stanford’s Learning Design and Technology master’s program, Soren has developed ample insight into the hacks and challenges of developing educational technologies that have real impact on learning.

This content is provided to you freely by EdTech Books.

Access it online or download it at https://edtechbooks.org/ux/lx_theories_of_change.