Introduction

The COVID-19 pandemic disrupted school student learning and on-site professional development (PD). K-12 educators shifted to online PD for emergency remote teaching (ERT), using synchronous and asynchronous learning (Cavanaugh & DeWeese, 2020; Hartshorne et al., 2020; Sparks, 2020). The EdHub Library, part of the University of Missouri's Network of Educator Effectiveness (NEE) since 2014, has provided 500+ asynchronous activities to around 38,000 educators in Missouri, Kansas, and Nebraska. After the 2019 redesign, EdHub improved PD material searchability and aligned it with Missouri Teacher Standards (Leung, 2021). Initially, subject matter experts identified three ERT resources to support K-12 educators at the pandemic's onset: Best Practices for Remote Instruction, Remote Learning Evaluation, and Google Classroom Help.

Literature review

Although forecasting methods are uncommon in online program management, they are prevalent in higher education and K-12 contexts. In higher education, these models predict student enrollment, demographic shifts, program evaluation, and budget allocation (Langston et al., 2016; Rossi et al., 2018). In K-12 settings, forecasting anticipates school expenditures and student population growth (Hasan et al., 2019; Yang et al., 2020). The COVID-19 pandemic has disrupted these predictions as historical data no longer reliably forecasts future trends due to sudden fluctuations. Uncertainties like economic policies, health mandates, and COVID-19 variants undermine forecasting activities. Scholars suggest new methodologies, employing alternative estimation techniques for outliers and introducing additional variables for stable and accurate forecasts during crises (Ho, 2021; Kumar et al., 2021; Pohlman, 2021).

Problem statement

Outlier and anomaly detection are data mining techniques used to identify unusual patterns in datasets. However, they face limitations due to timestamp considerations when describing resource access probabilities. Resource curation is challenging due to the continuous 24/7 availability of a vast library without defined start and end dates. Although subject matter experts' resource identification played a critical role during the pandemic, a data-driven approach is required by analyzing timestamp attributes for emergency preparedness planning.

Purpose and significance of the study

This study enhances ERT resource planning by forecasting online resource access using timestamps and metrics to ensure K-12 educators’ PD continuity during times of crisis. It employs Facebook Prophet’s changepoint detection to identify pivotal PD resources during the March 2020 COVID-19 crisis, analyzing trends from 2019 to 2021. Findings assist school leaders in evaluating teacher program materials, understanding K-12 educators’ PD needs, and optimizing online PD deployment in the EdHub Library. Notably, there is a gap in the literature on forecasting studies in online teacher PD addressing learner interaction aspects, as initially outlined by Dede et al. (2009). This study explored the following research questions:  RQ1: What are the trend characteristics of Pageviews, New Users, and Returning Users? RQ2: Which online instructional modules are required for the emergency preparedness planning resource?

Methods

Data description

Web analytics data from three academic years (2019, 2020, 2021) was extracted via a Python script in Jupyter Notebooks, connecting to the Google Analytics administrator property. For each academic year, three variables were obtained: New Users (first-time visitors), Returning Users, and Pageviews of resources. Data was reported in Central Standard Time, and the respective metrics were retrieved using the date timestamp. Pageviews represent views of accessed resources in a single session by new or returning users. To safeguard privacy, Google Analytics conceals internet protocol (IP) addresses, preventing individual user identification (UA Dimensions & Metrics Explorer, n.d.).

Facebook Prophet: Open-source time-series tool

Facebook Prophet, an open-source forecasting tool by Meta, was employed for time-series analysis, fitting non-linear trends with yearly, weekly, and daily seasonality and holiday effects (GitHub - Facebook/Prophet, n.d.). This additive regression method decomposes time-series data into trends, seasonalities, cyclic patterns, and random components, predicting future data points. The tool requires two columns: ds for date timestamps in DateTime format and y for the forecasting measurement. Academic years were studied individually using the three Google Analytics metrics to understand user and PD resource access patterns.

Observed trends and changepoints

Facebook Prophet identifies seasonal effects (yearly, weekly, daily) and adeptly handles missing data and outliers (Taylor & Letham 2018). Forecast models, using an 80% uncertainty interval, minimize training error for daily and monthly projections. Predicted values, also known as yhat, fall within a range of lower (yhat_lower) and upper (yhat_upper) bounds. These models reveal dataset trends and abrupt changes at the onset of the pandemic in March 2020. Facebook Prophet calculates daily magnitudes of change, aiding in detecting sudden shifts in Google Analytics variables. This insight informs the curation of online PD modules for emergency preparedness planning.

Findings

RQ1: What are the trend characteristics of Pageviews, New Users, and Returning Users?

After analyzing academic years, the EdHub Library noted a significant decline in Pageviews, New Users, and Returning Users from January to July 2020, attributed to the library’s already substantial user base after the December 2019 interface redesign (Leung, 2021). Yearly, weekly, and daily seasonality analyses revealed an increasing trend at the school year’s start (August-September) and summer (May-June), with peak web traffic on Mondays to Wednesdays and lower on Thursdays and Fridays. Daily user and resource access followed distinct patterns at 7:00 AM, 12:00 PM, and 5:00 PM during the week.

RQ2: Which online instructional modules are required for the emergency preparedness planning resource?

A total of 24 changepoints were identified across the Google Analytics variables amid the COVID-19 pandemic. From 03/02/2020 to 01/08/2021, each variable displayed eight changepoints as school districts resumed library usage at the pandemic’s onset and winter break. Refer to Table 1 for the precise dates of these abrupt upward changes.

Table 1

Changepoints for Pageviews, New Users, and Returning Users

After detecting pandemic-related changepoints in Pageviews, eight dates revealed increased access to educational resources, including Getting Started with EdHub, Teacher Indicator Examples for Kindergarten, Teacher Summative Reports, Scoring Practice videos for English Language and Arts, Dyslexia and Learning, Beginning Teacher Support, and Remote Learning. Notably, K-12 educators engaged with dedicated sitemaps for PD selection, emphasizing Teacher Standards 4 (teaching critical thinking), 5 (creating a positive classroom environment), 6 (using effective communication), and 7 (monitoring the effect of instruction). New Users showed heightened interest in specific content, including Scoring Practice videos for Math and English Language and Arts, Social-Emotional Learning, Teacher Indicator 5.3b video examples (promoting social competence), Instructional Strategies, and Beginning Teacher Support, and performed search queries related to interactive classroom observation scoring training simulations. Returning Users displayed increased access to modules covering Teacher Standard 1 (using content knowledge with appropriate instruction), Beginning Teacher Support, Remote Learning, and various instructional videos. In total, 84 modules in 17 topic categories exhibited upward trends, as shown in Table 2.  Figure 1 depicts the resource interface of all modules with a searchable component and a micro-survey for suggestions.

Table 2

Curated Instructional Modules for the Emergency Preparedness Resource

Figure 1

Emergency Preparedness Resource Interface

Discussion

This study examined user and resource access trends during the COVID-19 pandemic. Despite the downtrend in new users, resource access and web traffic exhibited a weekly uptrend at the start of the week (around 7:00 AM, 12:00 PM, and 5:00 PM), influenced by the start of the school year in August and the start of the summer session.

Implications for program management

These findings have implications for program evaluators, system administrators, and PD material developers. Evaluators should establish a baseline for user behavior, understand school year seasonality, and identify critical PD access times. Administrators should implement retention efforts and early alerts, especially when anticipated seasonality trends differ. PD material developers should strategically release online resources on Thursdays and Fridays. Thus, K-12 educators can access the latest PD releases during breaks on Mondays through Wednesdays of the following week.

Limitations

Although Google Analytics collects several variables, this study only considered Pageviews, New Users, and Returning Users as the most commonly used variables in previous studies (Leung, 2018; Leung, 2019; Leung, 2021). The study identified changepoints in PD resources at the second level of the EdHub Library module, covering all instructional activities. While not all module activities were accessed, the emergency preparedness resource is organized by modules to address K-12 educators’ PD needs.

Conclusion

The study established a baseline for three academic years’ user and resource access patterns. It also helped program managers select 84 online pedagogical resources for emergency preparedness planning to support K-12 educators’ PD continuity efforts.

References

Cavanaugh, C., & DeWeese, A. (2020). Understanding the professional learning and support needs of educators during the initial weeks of pandemic school closures through search terms and content use. Journal of Technology and Teacher Education, 28(2), 233-238.

Dede, C., Jass Ketelhut, D., Whitehouse, P., Breit, L., & McCloskey, E. M. (2009). A research agenda for online teacher professional development. Journal of Teacher Education, 60(1), 8-19. https://doi.org/10.1177/0022487108327554

GitHub - Facebook/Prophet. (n.d.). Facebook Prophet. Retrieved January 22, 2022, from https://github.com/facebook/Prophet

Hartshorne, R., Baumgartner, E., Kaplan-Rakowski, R., Mouza, C., & Ferdig, R. E. (2020). Special issue editorial: Preservice and inservice professional development during the COVID-19 pandemic. Journal of Technology and Teacher Education, 28(2), 137-147.

Hasan, R., Palaniappan, S., Mahmood, S., Shah, B., Abbas, A., & Sarker, K. U. (2019). Enhancing the teaching and learning process using video streaming servers and forecasting techniques. Sustainability, 11(7), 2049. https://doi.org/10.3390/su11072049

Ho, P. (2021). Forecasting in the absence of precedent. Richmond Fed. Retrieved September 13, 2023, from https://www.richmondfed.org/publications/research/working_papers/2021/wp_21-10

Kumar, S., Viral, R., Deep, V., Sharma, P., Kumar, M., Mahmud, M., & Stephan, T. (2021). Forecasting major impacts of COVID-19 pandemic on country-driven sectors: Challenges, lessons, and future roadmap. Personal and Ubiquitous Computing, 1-24.

Langston, R., Wyant, R., & Scheid, J. (2016). Strategic enrollment management for chief enrollment officers: Practical use of statistical and mathematical data in forecasting first-year and transfer college enrollment. Strategic Enrollment Management Quarterly, 4(2), 74-89. https://doi.org/10.1002/sem3.20085

Leung, J. (2018). Discovering utilization patterns in an online K-12 teacher professional development platform: Clustering and data visualization methods. Quarterly Review of Distance Education, 19(3), 17-37.

Leung, J. (2019). The EdHub Library: A microlearning design approach for teacher professional development. C2C Digital Magazine, 1(10), 3. http://scalar.usc.edu/works/c2c-digital-magazine-fall-2018--winter-2019/the-edhub-library-microlearning-design-approach-teacher-prof-dev

Leung, J. (2021). Design features of online teacher professional development: A design case for re-developing the EdHub Library to improve usability and alignment of content with teacher standards. International Journal of Designs for Learning, 12(2), 79-92. https://doi.org/10.14434/ijdl.v12i2.29578

Pohlman, A. (2021, February 1). Why economic forecasting is so difficult in the pandemic. Harvard Business Review. https://hbr.org/2020/05/why-economic-forecasting-is-so-difficult-in-the-pandemic

Rossi, P. H., Lipsey, M. W., & Henry, G. T. (2018). Evaluation: A systematic approach. Sage publications.

Sparks, S. (2020, April 10). Coronavirus and school research: A major disruption and potential opportunity. Education Week. https://www.edweek.org/technology/coronavirus-and-school-research-a-major-disruption-and-potential-opportunity/2020/04

Taylor, S. J., & Letham, B. (2018). Forecasting at scale. The American Statistician, 72(1), 37-45. https://doi.org/10.1080/00031305.2017.1380080

UA Dimensions & Metrics Explorer. (n.d.). Retrieved January 21, 2022, from https://ga-dev-tools.web.app/dimensions-metrics-explorer/

Yang, S., Chen, H. C., Chen, W. C., & Yang, C. H. (2020). Student enrollment and teacher statistics forecasting based on time-series analysis. Computational Intelligence and Neuroscience. https://doi.org/10.1155/2020/1246920