Conducting a Formative Evaluation on a Course-Level Learning Analytics Implementation Through the Lens of Self-Regulated Learning and Higher-Order Thinking

Self-regulated learning (SRL) and higher-order thinking skills (HOTS) are associated with academic achievement, but fostering these skills is not easy. Scholars have suggested an alternative way to scaffold these important skills through learning analytics (LA). This paper presents a formative evaluation of a course-level LA implementation through the lens of self-regulated learning (SRL) and higher-order thinking skills (HOTS). We explored the changes in students’ SRL, HOTS, and perceptions at the end of the course term. Results indicate an increase in some elements of SRL and HOTS, and positive student perceptions. Discussion on implications and opportunities for informing future teaching strategies and course design reiteration are included.

Fostering students' HOTS and SRL is not simple (Koh et al., 2012;Nouri et al., 2019;Yen & Halili, 2015).Therefore, some scholars adopt learning analytics (LA) to assess to what extent students deploy specific strategies during the learning process (Tabuenca et al., 2015;van Horne et al., 2017;Yamada et al., 2016Yamada et al., , 2017;;You, 2016).Students' digital traces can be analyzed for learning behavior patterns to inform interventions that foster exemplary behaviors (Roll & Winne, 2015).Research has also shown that implementing LA helps foster SRL (Tabuenca et al., 2015;van Horne et al., 2017;Yamada et al., 2016Yamada et al., , 2017;;You, 2016).Despite the benefits of LA, translating data from LA into actionable interventions at the course level is complex and still rare (van Leeuwen, 2019).This paper presents a formative evaluation conducted at the course level.We utilized Learning Management System (LMS) usage data and an LA framework synthesized from existing literature.The LMS data allowed the instructor to decide when to employ interventions that promoted HOTS and SRL.This formative evaluation included an investigation of student SRL and HOTS using pre-and postsurveys, both of which included closed-ended and open-ended items.Essentially, we traced any changes in student SRL and HOTS after the instructor performed data-informed interventions by following a synthesized LA framework based on the works of Ifenthaler and Widanapathirana (2014), Muljana and colleagues (2021), and Muljana and Placencia (2018).Our findings will be incorporated into future instructional strategies and course design reiteration to encourage SRL and HOTS through an LA implementation.

Literature Review
Applications of LA should align with learning contexts; therefore, it is essential to implement LA in conjunction with an existing learning theory or construct (Gašević et al., 2015).In this evaluation, we utilized LA in parallel with promoting students' SRL and HOTS

Description of Learning Analytics
LA refers to "the measurement, collection, analysis, and reporting of data about learners and their contexts, for the purpose of understanding and optimizing learning and the environment in which it occurs" (Siemens & Long, 2011, p.32).This definition yields two key points (Muljana & Luo, 2021;Muljana et al., 2021; Self-Regulated Learning SRL is proactive learning activities or process involving learners' thoughts, behaviors, and affects that systematically and strategically assist them in achieving their goals of improved learning (Zimmerman, 2002, as cited in Dabas et al., 2021).Students who possess self-regulation skills assess the situation, set goals, conduct and monitor their strategies, self-evaluate the outcome, and selfadapt to any improved strategies.In other words, the use of SRL involves students' cognitions, behaviors (Zimmerman, 1989), and affects (e.g., selfsatisfaction) (Zimmerman, 2002), and requires continual iteration.When students self-adapt, they also set new goals for the next learning activity.Among all SRL elements, there is a clear interrelationship between metacognition and regulation (Binbarasan-Tüysüzoğlu & Greene, 2015;Karabenick & Zusho, 2015); students tend to perform better if they continuously regulate their efforts according to their metacognitive awareness about their learning process and progress.In our formative evaluation, we focused on two SRL elements: metacognition and effort regulation.
While an individual's proactive action is essential to SRL, external factors like study environment, available time to study, access to learning resources, instructional guidance, and instructional conditions play a role in SRL development (Gašević et al., 2016;Winne, 2011Winne, , 2017)).Yamada et al. (2017) recommend course elements that intentionally promote students' self-efficacy and cognitive learning strategies because these variables support SRL skills.Broadbent (2017) further recommends scaffolding methods, such as providing learning opportunities and assessments that promote goal setting, planning, and reflection, be integrated into course design to encourage students to adopt SRL strategies.
Several studies connect LA with SRL.For example, student responses to Pintrich et al. 's (1991) Motivated Strategies for Learning Questionnaire (MSLQ) can be analyzed and correlated with the timeliness of assignment submissions (Yamada et al., 2016(Yamada et al., , 2017)).MLSQ items can also be correlated with student SRL and their access frequency to real-time feedback provided by LA dashboards (van Horne et al., 2017).Still, while many LA-related studies focus on measurement purposes, the emphasis on teaching practices using LA to support students deserves further attention (Viberg et al., 2020).

Higher-Order Thinking Skills
We adopted the following overarching description of HOTS: "higher-order thinking occurs when a person takes new information and information stored in memory and interrelates and/or rearranges and extends this information to achieve a purpose or find possible answers in perplexing situations" (Lewis & Smith, 1993, p. 136).Students with HOTS perform beyond the literal interpretation of information to expound and use reason to build representations from it (Newman, 1990;Resnick, 1987).The professional world demands HOTS (Rotherham & Willingham, 2010;Silva, 2009) that instructional strategies and a well-designed learning environment can scaffold (Heong et al., 2011;Yen & Halili, 2015).HOTS allow students to become independent thinkers, problem solvers, and decision makers, and to facilitate the transfer of these abilities into real-life situations in the professional world (Rotherham & Willingham, 2010;Silva, 2009).
Students acquire HOTS during the learning process by identifying tasks and problems, clarifying components required by problems, judging related information, and evaluating information acquired and procedures for problemsolving (Quellmalz, 1985).These activities promote students' self-awareness about their thinking, self-monitoring, and problem-solving strategies (Quellmalz, 1985).HOTS involve the execution of critical, logical, reflective, metacognitive, creative thinking, and self-regulation skills (Mainali, 2012;Resnick, 1987;Zohar & Dori, 2003).Metacognitive thinking, self-regulation, and critical and reflective thinking all overlap with SRL elements.During the learning process, critical thinking helps students select, test, evaluate, adopt, and adapt suitable learning strategies in various learning contexts (Brown et al., 1993;Hadwin & Oshige, 2011).As students evaluate the impact of their learning strategies on learning outcomes, they use reflective thinking to improve their learning process (Isaacson & Fujita, 2006).We focused on investigating critical and reflective thinking during our evaluation.
A small number of recent studies have explored the intersection of HOTS and LA.
For example, visual LA tools have been used to investigate students' activities annotating reading materials and commenting on other annotations, which positively impact critical reading achievement (Koh et al., 2019).Learning assisted by visual LA tools also influences students' higher-order thinking (Zhang & Chan, 2020).However, these studies do not provide practical guidelines to translate LA data into immediate actions.
Using Learning Analytics at the Course Level Ifenthaler and Widanapathirana (2014) developed an LA matrix outlining the benefits of using LA from predictive, formative, and summative perspectives.Predictive LA helps foresee outcomes and determine future strategies when conducted early on.Formative LA uses real-time data to help instructors decide whether to intervene.Summative LA can give insights after learning events.In our two previous works, we built upon Ifenthaler and Widanapathirana's (2014) matrix to develop two similar LA frameworks (Muljana et al., 2021;Muljana & Placencia, 2018) that used three analytic perspectives.For the present evaluation, we synthesized those frameworks and adapted them into three phases: (1) early diagnosis, conducted at the start of the semester; (2) formative diagnosis, conducted throughout the semester; and (3) summative diagnosis, conducted partly during and partly at the end of the semester.Note.The LA approach includes three phases, synthesized from existing frameworks (e.g., (Ifenthaler & Widanapathirana, 2014;Muljana et al., 2021;Muljana & Placencia, 2018).
Instructors can conduct early diagnosis using data from entrance surveys, pre-tests, ice-breaker discussions, and course usage logs.They can review the data to learn students' goals for taking the course and students' prior knowledge and experience.Instructors can then use that information to provide clear expectations and instructions, and offer SRL and HOTS strategy tips through weekly briefings.
Instructors can implement formative diagnosis to detect challenging topic(s), potentially at-risk students, and less-engaged students.They can analyze data like submission timestamps, test difficulty reports, course usage records, and discussion posts.Instructors can then intervene and suggest remedial materials, share SRL and HOTS strategy tips, adjust course design, encourage discussion dialogues, and adjust instruction clarity in subsequent modules.
Instructors can review summative diagnosis such as overall student outcomes to identify students who excel or lag behind so as to observe their engagement level.They can also analyze course usage summaries, participation numbers, and module access to inform decisions for adjusting instructional strategies and course design for future cohorts.

Formative Evaluation Questions
From the lens of SRL and HOTS, we conducted a formative evaluation on an LA implementation performed at a course level.We used an LA framework synthesized from existing literature (Ifenthaler & Widanapathirana, 2014;Muljana et al., 2021;Muljana & Placencia, 2018) to observe students' learning progress and inform the instructor's interventions to adjust teaching strategies and improve student learning (see Table 1).As "the function of formative evaluation is to improve" (Nieveen & Folmer 2013, p. 158), our findings will be used to inform instructional strategies and course redesign for subsequent cohorts.Three questions guided this evaluation: (1) Did the LA implementation increase student SRL by the end of the semester?(2) Did the LA implementation increase student HOTS by the end of the semester?(3) How did the students perceive changes in their SRL and HOTS by the end of the semester?

Evaluation Methods
We adopted a case study approach for this formative evaluation.We use pre-and post-surveys (i.e., closed-and open-ended items) to understand student SRL and HOTS, including their perceived understanding toward their own SRL and HOTS.We selected the case study approach because it allowed us to comprehend a contemporary, complex phenomenon (Yin, 2008).In our context, applying LA at the course level is an emerging practice.The practice of translating data from LA into actionable interventions at the course level is still a rarity (van Leeuwen, 2019), and may require instructors to use complex processes (Molenaar & Knoopvan Campen, 2018;Wise & Jung, 2019).The case study approach guided us to explore how an LA implementation supported student SRL and HOTS, allowing us to highlight the practical significance of the results (Newman & Hitchock, 2011).
In this formative evaluation, we analyzed the data from one instructor who taught two course sections on the same topic: one with an LA implementation, the other without.Analyzing these two cases allowed us to examine each situation (Yin, 2008) by whether the LA implementation contributed to any SRL and HOTS changes or not.Given the small number of participants in this formative evaluation, we only used descriptive statistics to answer the evaluation questions regarding the pre-and post-comparison.These gave insight on how to improve our strategies and course design, as well as to inform readers about a potential LA implementation that can enhance SRL and HOTS.

Participants and Context
After receiving approval from the Institutional Review Board, we recruited students from two identical course sections of an upper-level general education course for an engineering program.This course applied economic theory to solve managerial problems and make decisions related to capital allocation for private, public, and governmental sectors.Twelve students participated in this study: four students from course section 1 and eight from course section 2 (see Table 2).We assured their anonymity, and they signed an informed consent form.

Instrumentation
We used a questionnaire consisting of demographic-related items and selected sub-scales from the MSLQ (Pintrich et al., 1991) to assess students' prior SRL and HOTS, as well as improvements.We specifically chose the following MSLQ subscales: (a) 12 items of Metacognitive Self-Regulation for assessing SRL; (b) four items of Effort Regulation for assessing SRL; and (c) five items of Critical Thinking for assessing HOTS.We also adopted four items from the Reflection sub-scales of the Reflective Thinking Questionnaire (RTQ) by Kember et al. (2000) to assess HOTS.MSLQ is one of the frequently used instruments for assessing SRL strategies (Panadero, 2017;Tong et al., 2020).As cited in Muljana et al. (2021), previous research utilizing MSLQ reported good reliability and validity with Cronbach alpha values between 0.62 to 0.93 (Cho & Shen, 2013;Hederich-Martínez et al., 2016;Kim & Jang, 2015;Li et al., 2020;Stegers-Jager, et al., 2012).RTQ also showed good reliability and validity in several studies, with Cronbach alpha values ranging between 0.62 to 0.91 (Asakereh & Yousofi, 2018;Ghanizadeh, 2017;Ghanizadeh & Jahedizadeh, 2017;Safari et al., 2020;Tsingos-Lucas et al., 2016).In total, we used 25 items from both MSLQ and RTQ in preand post-surveys, but excluded some demographic items from the post-survey and included three open-ended questions.The open-ended questions in the post-survey asked students' whether they perceived any changes in their SRL and HOTS.

Procedures and Data Collection
Formative evaluation took place in two course sections: section 1 (Case 1) and section 2 (Case 2).The same instructor taught both sections on the same topics using the same instructional resources.The instructor conducted the LA phases (as listed in Table 1) in Case 2; but, purposefully, not in Case 1.In week 1 and week 2, the students from both cases completed the pre-survey, and the instructor covered the content of Module 1 and Module 2 that served as the foundation of the more advanced topics in the subsequent modules or weeks.Within Module 3 to Module 12, the instructor included one weekly quiz at the end of each module in Case 1.However, the modules for Case 2 included two weekly quizzes (for midmodule and at the end of a module).The post-survey was made available in week 14, and students had two weeks to complete it.Table 3 lists the overall procedures in both cases.
Table 3 The Formative Evaluation Procedures Note.Data were collected from pre-and post-surveys.

Data Analysis
We exported the pre-and post-survey results into a Microsoft Excel spreadsheet.We, then, analyzed these data using the Statistical Package for the Social Sciences for descriptive statistics.Due to the small sample size, we conducted no inferential statistics.
We analyzed the open-ended responses using the structural coding technique.This coding technique utilizes content-based phrases "representing a topic of inquiry to a segment of data that relates to a specific research question" (MacQueen et al., 2008, p. 124, as cited in Saldaña, 2013, p. 84), and is suitable for analyzing openended survey responses (Saldaña, 2013).Using structural coding, the first author simultaneously coded and categorized students' open-ended responses by identifying segments of responses displaying commonalities (Saldaña, 2013), guided by the third question itself and related topics.As stated by Saldaña (2013), structural coding is "framed and driven by a specific research question and topic" (p.87).
We used three a priori topics related to the third question.For example, students must select and monitor suitable learning strategies when conducting an SRL phase (Zimmerman, 2002).Thus, the first a priori category represented students' perceptions of any changes in their learning strategies.Second, instructors' guidance played an imperative role in SRL development (Gašević et al., 2016;Winne, 2011Winne, , 2017)).So, the second a priori category guided us to analyze students' perceptions about instructional guidance.Third, our literature review indicated students with HOTS could become independent thinkers, problem solvers, decision makers, and facilitate the transfer of these analytical thinking abilities into real-life situations in the professional world (Rotherham & Willingham, 2010;Silva, 2009).Hence, the third a priori category guided us to analyze perceptions regarding transferable, analytical thinking skills that students gained.The first author, next, presented all analyzed categories to the second author for feedback.They discussed the analyzed categories and resolved any disagreements (See Tables 6 and 7 for the highlighted categories resulting from the structural coding techniques).

Changes in SRL
Comparing pre-and post-survey results, Case 2 showed better average score increases for each variable (see Table 4).For example, while the average score of metacognitive self-regulation did not increase in Case 2, it slightly decreased from 3.25 to 3.08 in Case 1. Students' average effort regulation scores also increased from 3.28 to 4.50 in Case 2, more than one point higher than for Case 1.
Table 4 Results of Pre-and Post-Surveys Assessing Self-Regulated Learning  Note.1=Strongly Disagree, 5=Strongly Agree

Changes in HOTS
Students' critical thinking scores decreased slightly in Case 1 (M = 3.75 to M = 3.65), while those of students in Case 2 increased (M = 3.45 to M = 3.78).
Students in both cases self-rated reflection strategy lower by the end of the semester; however, there was a larger decrease among students in Case 1 (M difference = -.62).Table 5 lists pre-and post-survey results for HOTS.Note.1=Strongly Disagree, 5=Strongly Agree The Journal of Applied Instructional Design, 11(1)

Perceived Changes in SRL and HOTS Case 1
To apply SRL, students must select and monitor suitable learning strategies (Zimmerman, 2002).Thus, we asked students whether they changed such strategies.Three out of four students in Case 1 reported they used the same learning strategies since the beginning of the semester.As one student noted, "My strategies are the same as they were.Pay as much attention in class as possible and supplement with textbook or internet knowledge as needed." Students had positive comments about HOTS, despite the absence of the LA implementation.They noted gaining or boosting their skills in the application and analytical-thinking HOTS domains

No change in learning strategy
Students did not change their learning strategy.They still employed the same strategy that they had been using.

3
"My strategies are the same as they were.Pay as much attention in class as possible and supplement with textbook or internet knowledge as needed."

Helpful instructional guidance
Students thought that the instructional guidance provided by the instructor was helpful.Guidance manifested through materials and projects was clear.
3 "The content in this course is fairly intuitive to me, but the instructor offers good explanations for lessintuitive concepts so his guidance is helpful."

Encourage analytical thinking
Students felt they gained analytical and problemsolving skills through the problems posed in the course.There was no particular simple way to solve the problems.
2 "[…,] there's not a preset list of equations to use nor the ability to just plug in values and get an answer.We have to analyze and understand real work applications and that would vary from the formulas."

New applications
One student noted that while they did not feel to learn new information, the course encouraged multiple applications of the topics.
1 "Most of the content in this course is fairly intuitive to me and I was already familiar with a few of the topics, so it doesn't feel like I've learned much new information.However, the course taught me many new applications of these topics that I'm glad I learned."

Case 2
Three out of eight students in Case 2 noticed changes in learning strategies, employing different tactics when approaching a problem (e.g., creating visualizations in Excel to analyze information and solve a problem).One student said, "I find the visual relationship better to understand and have modified that to fit my calculus class as well [another quantitative class]." Six students recognized the instructor selected and applied suitable instructional strategies, displaying awareness about the instructor's teaching and scaffolding strategies.They noted that the instructor promoted student-to-content engagement and provided strategic content sequence.One student expressed "[…,] his [or instructor's] methodology of teaching builds new material on top of the previous material," insight that was not detected in Case 1.
Five students also noticed the transferable skills they gained, displaying a change in their HOTS.These students believed they could transfer what they learned into real-life situations, both personally and professionally.As one student noted, "The class is great for project management positions.[… .]Even just knowing the basics is a great baseline for understanding economics that would come up in future workloads."

Discussion
We have conducted a formative evaluation on an LA implementation performed at a course level from the lens of SRL and HOTS.We used LMS usage data and an LA framework synthesized from existing literature.Results suggest an increase in effort regulation and critical thinking, but not in metacognition nor reflection in Case 2. Although metacognition and reflection did not increase in Case 2, we detected a decrease in these variables in Case 1.In terms of student perception, we positive insights in both cases.Students in Case 2 expressed their insights more analytically (e.g., why the instructor's guidance was helpful).They also noticed instructions were adjusted to match their learning experience (e.g., simpler topics were presented before more complex ones), indicating awareness about their own learning process and progress.We expound these results into several discussion points.

Robust Course Design as a Foundation
Students from Case 1 expressed positive perceptions about changes in their SRL and HOTS at the end of the semester, despite the absence of LA implementation.This may be due to course design and organization.For example, they noted that the course design incorporated instructional guidance.The students also thought learning tasks encouraged analytical thinking beyond simply inputting numbers into formulas.Both cases used similar course structures based on a QM course template developed by the university.Course content was segmented into modules that aligned learning outcomes to individual learning tasks and ordered by complexity.This suggests that well-designed instructional strategies and learning environments can be a foundation for enhancing student SRL and HOTS (Heong et al., 2011;Yen & Halili, 2015).Robust course design can therefore bootstrap effective LA implementation at the course level, allowing instructors to focus more on optimizing learning outcomes (Muljana et al., 2021;Muljana & Placencia, 2018).We intend to continue to practice robust course design in future course cohorts.

Ensuring the Mastery of Prerequisite Topics
Results suggest an increase in HOTS in the critical thinking domain among students in Case 2. This may have been influenced by initial content sequencing and instructional adjustments throughout the semester.Students must apply prior knowledge to adopt critical thinking (Pintrich et al., 1991), meaning instructors may consider ensuring students master prerequisite topics.In this context, the instructor used a mid-module quiz in each module to assess if students understood fundamental concepts.After reviewing quiz results, the instructor analyzed the quiz-item difficulty to determine which topics to review or provide remedial materials.According to existing literature, using LA enables instructors to analyze and diagnose students' current learning progress, struggles, and successes, thereby determining necessary interventions to help them achieve learning outcomes better (Casey & Azcona, 2017;Dietz-Uhler & Hurn, 2013;Macfadyen & Dawson, 2010).We will continue to use mid-module quizzes in the subsequent course cohorts to help instructor ensure mastery of prerequisite topics.

The Role of Early and Formative Diagnoses
Implementing LA may have increased effort regulation in Case 2. From early in the semester, the instructor checked how long students accessed course materials and whether they clicked course pages without reviewing material thoroughly.This data alerted the instructor to students who might have needed learning strategy tips to regulate their efforts in reviewing course materials in the LMS.Furthermore, students who accessed the course less periodically received an email reminding them to regularly access and review materials in the LMS.According to Kim et al. (2016) and You (2016), analyzing LMS usage data early in the semester can help instructors forecast students' course access habits (Kim et al., 2016;You, 2016).
Formative diagnosis in Case 2 may have additionally influenced students' SRL.Formative analysis of LMS data -e.g., student frequency and duration accessing course materials, students' timing (early or late) submitting assignments, and analyzing topic difficulty based on test report data -can support an instructor's decision regarding when to adjust learning conditions based on students' learning progress (Casey & Azcona, 2017;Dietz-Uhler & Hurn, 2013;Macfadyen & Dawson, 2010).In this context, the instructor continuously scaffolded students by adjusting instruction and providing remedial materials and reviews as needed.
Consulting LA data throughout the learning process can help instructors find appropriate strategies to proactively help students perform better (Yen et al., 2015).As expected, students in Case 2 expressed their perception more analytically and in-depth, and were aware when instructors provided deliberate instructional guidance.This resonates with Winne's (2011Winne's ( , 2017) ) and Gašević et al.'s (2016) suggestions to highlight the important role of instructional guidance.
Based on this finding, we will continue to implement analytical diagnoses and provide appropriate instructional guidance and conditions in future course cohorts.

Facilitating Metacognitive and Reflective Learning Activities
Metacognitive self-regulation did not change in Case 2, nor did LMS data appear to capture metacognitive learning activities.In a future course redesign, we plan to ask students to describe their learning habits and strategies during an icebreaker discussion and explain how they overcome challenges while learning.This will provide the instructor an overview of students' awareness, knowledge, and control of their cognition before determining suitable instructional conditions to scaffold metacognition (Gašević et al., 2016;Winne, 2017).
Results suggested the reflective-thinking domain of HOTS did not improve, which may be due to the absence of a reflective assignment.While instructional strategies and a well-designed learning environment can promote HOTS (Heong et al., 2011;Yen & Halili, 2015), students need specific instructions to reflect upon their learning (Kember et al., 2000).We, therefore, plan to add a reflective assignment for the next course design iteration.

Future Work
Our findings highlight how LA may have influenced students' SRL and HOTS in two different course sessions.We recognize that the small number of participants makes it difficult to make a robust comparison or to examine for statistically significant differences on SRL or HOTS due to our LA implementation.Therefore, a future study may include a larger sample size and use an experimental design to investigate such impacts.We would also consider adding alternative data points, such as interviews or focus groups, to enrich our data sources and potentially reveal additional considerations when using LA to support SRL and HOTS.
The findings also give insight on iterative course design.We plan to redesign the course, adjust some assignments, and implement all three phases of LA such as: (1) early diagnosis at the start of the semester; (2) formative diagnosis throughout the semester; and (3) summative diagnosis conducted partly during and at the end of the semester (as shown in Table 1).
Because the practice of translating LA data into actionable interventions at the course level is still emerging (van Leeuwen, 2019), these findings suggest the potential of using LA to foster SRL and HOTS.We, therefore, encourage scholars, instructors, and instructional designers to test the LA framework synthesized from the existing literature for research and teaching purposes to expand the current body of literature at the intersection of LA, SRL, and HOTS.
Pre-SD Post-M Post-SD M difference Pre-M Pre-SD Post-M Post-SD M difference

Table 1 Three Phases of LA Starting point Phase 1: Early diagnosis Phase 2: Formative diagnosis Phase 3: Summative diagnosis Performing sound- pedagogy course design as a foundation Consider assigning: • Entrance survey • Pre-test • Ice-breaker discussion Identify: • Any difficult topics • At-risk students • Less-engaged students
Table 1 details each phase.

Table 2
Demographic and Contextual Information of Evaluation ParticipantsBoth classes met twice a week using a traditional, face-to-face format.The instructor used a Blackboard LMS to host course materials and facilitate both learning tasks and assessments.The courses utilized a Quality Matters (QM) template built by the university to follow quality course design standards.Course content was segmented into 12 modules and sequenced strategically to present the fundamental topics initially before the more complex ones.The LMS includes built-in data-analytics features, such as Course Reports, Performance Dashboard, and Early Warning System, that record overall course usage, students' submission activities, and submission timestamps.The Item Analysis feature within the Grade Center in the LMS allowed the instructor to analyze quiz difficulty and overall students' performance by question.

Table 5
Results of Pre-and Post-Surveys Assessing Higher Order Thinking Skills

Pre-SD Post-M Post-SD M difference Pre-M Pre-SD Post-M Post-SD M difference
. For example, a student noticed merely plugging numbers would not work.Therefore, this student had "[…]to analyze and understand real work applications and that would vary from the formulas."Anotherstudentnoted that the course was already intuitive, but they did not learn new information and claimed, "[…] the course taught me many new applications of these topics that I'm glad I learned."Table6depicts the categories that emerged from open-ended responses from students in Case 1.

Table 6
Categories of Student Insights from Case 1 Table 7 depicts common themes that emerged from open-ended responses by students in Case 2.