Introduction

Learning Management Systems (LMS) have been widely adopted in higher education institutions to support teaching and learning practices (McGill & Klobas, 2009). Previous empirical studies have shown that learners’ LMS usage patterns are associated with, and impact, their learning outcomes (Weaver et al., 2008). Furthermore, researchers have indicated that investigating the association between LMS usage patterns and learning outcomes offers practical implications for future LMS design (Washington, 2019).

Past studies on LMS have delved into the usage patterns among various learner groups across different disciplines (Demmans et al., 2020). These studies have provided valuable insights into refining LMS design to better meet learners’ needs and preferences (Onodipe et al., 2020). However, a literature review conducted by the authors has revealed a lack of studies specifically focusing on pre-service teachers’ LMS patterns and their association with learning outcomes. Given the extensive use of LMS in pre-service education, it is crucial to understand these perspectives to update and enhance course and lesson designs. To bridge this research gap, this study will examine pre-service teachers’ LMS behavioral patterns and their correlation with learning outcomes. The aim is to furnish future instructors or instructional designers with valuable insights for designing and implementing pre-service courses in an LMS.

This study intends to probe into behavioral patterns and associations from both macroscopic and microscopic perspectives. From the macroscopic view, the researchers aim to investigate general usage patterns, such as total page views, on-time assignment percentages, and participation frequency, to discern their correlation with learning outcomes, leading to the following question:

1. How do pre-service teachers’ overall course grades correlate with page views, on-time assignment percentages, and participation frequency on LMS?

From the microscopic lens, the research focuses on how pre-service teachers’ usage patterns differ based on specific content themes and course design features, leading to the following question:

2. What patterns emerge in pre-service teachers’ page views and participation times across various themes and course design elements?

Literature review

Researchers have identified and examined the value of LMS data in understanding students’ learning behaviors and predicting their performance (e.g., Mozahem, 2020). Several studies have investigated students’ behaviors using LMS by analyzing LMS data with various predictive approaches (e.g., Zhang et al., 2020). Haig et al. (2013) detected and visualized the frequency of students’ access to resources and forum discussions in an undergraduate course using data generated from Moodle. Their results indicated students with higher grades accessed resources and forums more frequently. Similarly, Mandalapu et al. (2021) explored patterns in students’ logins and time intervals to determine their influence on learning outcomes. These researchers observed the count and regularity of student logins positively correlated with performance improvement. Chen & Cui (2020) used a deep learning approach to analyze online temporal behaviors, such as daily click frequencies, to predict students’ final grade performance. Unlike earlier research, they presented a trend in online student actions on Moodle for the course and concluded students’ clicks on Moodle could predict their performance, but only during the early weeks of the course.

In conclusion, prior studies have examined the potential of LMS data in predicting students’ performance and achievement. They explored factors like login frequencies and time spent on LMS as impacting course performance. However, more features and predictors within LMS should be evaluated to gain deeper insights into students’ learning behaviors using LMS and their resultant performance.

Methods

Context and data sources

All data were retrieved from an undergraduate course conducted on the Canvas platform during the fall semester of 2021. This course aimed to equip pre-service teachers with the skills to integrate technology into their teaching. It was a blended course, lasting 16 weeks, with no exams scheduled for the final week of the semester.

The primary data sources were reports generated by the Canvas LMS. The access report data consist of 23,156 lines of LMS usage logs, produced by 172 pre-service teachers. Each line indicates the page accessed by a student and the number of times that student viewed and engaged with that page. The course grade report provides insights into students’ overall course grades and their punctuality in submitting assignments. The LMS usage log data were analyzed in conjunction with their performance outcomes.

Analysis

Descriptive analysis displayed the overall learning behavior regarding times of page views, percentage of assignments submitted on time, and times participated on Canvas. Following this, a multiple regression analysis was conducted to explore the relationships between these three predicted variables and overall course grade as a dependent variable. Finally, content themes and course design features were coded and sorted based on the shared characteristics of the page content established for this course. Then, pre-service teachers’ page views and participation times across themes and course design features were visualized to explore their usage pattern differences. 

Results

Table 1 shows the descriptive statistics of behavioral variables available on the Canvas and the students’ course grade. The overall mean course grade was 89.265. Students had chances to earn extra credit if they submitted the bonus assignments on Canvas. The highest final score was 104.5, and the lowest score was 8.8. The range of times the overall pages were viewed is from 176 times to 4206 times, and the average is 1031 times. In addition, the range of assignment on time percent was from 14.81% to 100 %, and the course had an average of 85.27% of on-time submission. As for times of participation on Canvas, the average number of times a student participated on Canvas was 34 times, ranging from 7 times to 57 times.

Table 1

Descriptive Statistics (n=172)

Association among performance and usage patterns

A multiple regression model was conducted with course grade as the dependent variable, and times viewed, percentage of assignments submitted on time, and times participated as predictor variables. As shown in Table 2, times participated and the percentage of assignments submitted on time had statistically significant positive regression weights on course grade. This indicates that students who were more active on Canvas (t = 20.243, p < 0.001) and had a higher on-time submission rate for assignments (t = 2.516, p = 0.013) were expected to have higher final scores, after controlling for other variables in the model. However, page views did not contribute significantly to the multiple regression model, suggesting that the frequency of viewing Canvas pages did not significantly predict students’ course grades.

Table 2

Summary of Regression Analysis for Variables Predicting Overall Course Grade (n=172)

Usage patterns across themes and course design features

To address the second research question, visualizations illustrating pre-service teachers' page views and participation times across various themes and course design features were crafted. Both the content themes and course design features were coded and sorted based on the shared characteristics of the page content established for this course. Figure 1 depicts the proportions of different course design features viewed by students within each theme. Notably, the theme "Virtual Field Experience" displayed a balanced and somewhat anticipated access pattern, with no particular feature standing out. However, the remaining themes revealed a consistent pattern: the features most frequently viewed by students were the Warm-up Quiz and Exit Ticket Quiz. Figure 2 demonstrates the distribution of different themes viewed by students for each feature. A key observation is that the bulk of the Content Materials accessed by students originated from the "Virtual Field Experience" theme. These results suggest that pre-service teachers accessed a greater volume of content materials during the virtual field experience phase.

Regarding participation frequency, Figure 3 showcases the distribution of different course design features in which students participated within each theme. Echoing Figure 1, all themes indicated the Warm-up and Exit Ticket quizzes were the primary features students engaged with. However, within the "Tech-Supported Educational Context" theme, student interaction was notably higher on Announcement pages. This observation is consistent with Figure 4. Concerning the proportion of times students engaged with the Announcement page, the dominant theme was "Tech-Supported Educational Context." Other features exhibited a more balanced and similar participation pattern across themes. These insights reveal that students were more inclined to review announcements when exploring technology-supported educational contexts.

Figure 1

Times Viewed for Theme and Course Design Features

Figure 2

Times Viewed for Course Design Features and Theme

Figure 3

Times Participated for Theme and Course Design Features

Figure 4

Times Participated for Course Design Features and Theme

Discussions

Statistical findings indicate a notable correlation between the number of page views and students' course grades, yet it wasn’t a significant predictor. This contradicts Chen and Cui’s (2020) findings regarding LMS click frequency predicting performance. Students’ submission on time and participation on Canvas was significantly correlated with performance and predicted students’ final course grades. Students’ participation had the highest variable impact on their grades. The findings are consistent with the results in previous studies that late submissions and engagement were revealed to be significant indicators (You, 2016), which suggests the importance of increasing students’ participation and assignment submissions to improve their performance. Therefore, in order to improve students' performance, instructors should monitor student activities, send reminders if necessary, and collaborate more often with them on Canvas. 

Concerning the patterns of pre-service teachers’ page views and participation times across themes and course design features, a salient finding is that pre-service teachers demonstrated completely different LMS behavioral patterns when engaging with the Virtual Field Experience theme. More specifically, pre-service teachers presented a more expected course feature access patterns (Figure 2), and most of the content materials they accessed were related to the Virtual Field Experience theme. These findings indicate that pre-service teachers treated the field experiences differently than the rest of the themes although the whole course is dedicated to preparing their teaching readiness. It is interesting to see pre-service teacher pay more attention to Announcements, especially the content materials, which present more expected learning strategies. However, it also reflects that they might have accessed the content materials in the other themes less than expected since all the content materials occupy a similar proportion in each theme. This result could shed some light on the future design of pre-service courses. Firstly, the learning materials about field experiences can be more integrated into other themes since pre-service teachers seemed interested in it, which might help cultivate a more balanced learning strategy on all the themes. Secondly, when teaching other themes, instructors can assist students in making content more relevant to the field experience since all the knowledge taught in the course will be helpful for their future teaching. 

In all, by examining and analyzing learners’ LMS usage patterns, the outcomes revealed significant yet nuanced learning behaviors which reflect pre-receive teachers’ unique learning preferences and strategies. These findings could be applied to inform future course designs to better support pre-service teachers.

References

Chen, F., & Cui, Y. (2020). Utilizing student time series behaviour in learning management systems for early prediction of course performance. Journal of Learning Analytics, 7(2), 1-17. https://doi.org/10.18608/jla.2020.72.1

Demmans Epp, C., Phirangee, K., Hewitt, J., & Perfetti, C. A. (2020). Learning management system and course influences on student actions and learning experiences. Educational Technology Research & Development, 68(6), 3263–3297. https://doi.org/10.1007/s11423-020-09821-1

Haig, T., Falkner, K., & Falkner, N. (2013, January). Visualisation of learning management system usage for detecting student behaviour patterns. In Proceedings of the Fifteenth Australasian Computing Education Conference,136 (pp. 107-115).

Mandalapu, V., Chen, L. K., Chen, Z., & Gong, J. (2021). Student-centric model of login patterns: A case study with learning management systems. Paper presented at the International Conference on Educational Data Mining (EDM) (14th, Online, Jun 29-Jul 2, 2021). International Data Mining Society.

McGill, T., & Klobas, J. (2009). A task—technology fit view of learning management system impact. Computers & Education, 52(2), 496–508. https://doi.org/10.1016/j.compedu.2008.10.002

Mozahem, N. A. (2020). Using learning management system activity data to predict student performance in face-to-face courses. International Journal of Mobile and Blended Learning (IJMBL), 12(3), 20-31.

Onodipe, G., Keengwe, J., & Cottrell-Yongye, A. (2020). Using learning management system to promote self-regulated learning in a flipped classroom. Journal of Teaching and Learning with Technology, 9(1). https://doi.org/10.14434/jotlt.v9i1.29375

Washington, G. Y. (2019). The learning management system matters in face-to-face higher education courses. Journal of Educational Technology Systems, 48(2), 255-275. http://dx.doi.org/10.1177/0047239519874037

Weaver, D., Spratt, C., & Nair, C. (2008). Academic and student use of a learning management system: Implications for quality. Australasian Journal of Educational Technology, 24(1), 30–41. https://doi.org/10.14742/ajet.1228 

You, J. W. (2016). Identifying significant indicators using LMS data to predict course achievement in online learning. The Internet and Higher Education, 29, 23-30. https://doi.org/10.1016/j.iheduc.2015.11.003

Zhang, Y., Ghandour, A., & Shestak, V. (2020). Using learning analytics to predict students performance in moodle LMS. International Journal of Emerging Technologies in Learning, 15(20), 102-115. http://dx.doi.org/10.3991/ijet.v15i20.15915