Piloting Programmatic and Instructional Change

{{active_language_id_uc}} {{settings.doi}}

Distance Education as a Pilot Activity

World Education encourages states and programs to view their initial efforts at implementing distance and digital education programs for adult learners as pilot activities. Pilot activities are experimental and give organizations a chance to explore new methods on a small scale. They allow for trial and error and promote innovation. These activities are different from regular, established programs; however, if successful, they may be integrated into an organization’s standard courses. As mentioned earlier in this Handbook, teaching at a distance is significantly different from classroom instruction. Organizations will need to experiment to discover what works best for their staff and students

Here are some recommended piloting principles that teachers and program administrators might follow regardless of whether developing, upscaling, or refining distance and digital education programs:

Consider all aspects of distance or digital education.

Even if you are already offering only a few remote classes or effectively using a proprietary online curriculum, it remains essential to evaluate each key component of the distance and digital education program to identify what, if any, changes are necessary. Resources from the IDEAL Consortium, such as the Distance and Digital Education Site Plan—which is a central activity of this course—and other online materials can assist organizations in considering how to recruit, assess readiness, and orient learners; prepare teachers; deliver instruction; evaluate learners; and measure the program's success.

Expect uncertainty and changes throughout the pilot.

When piloting new programming or a new instructional modality, there is likely to be uncertainty for both teachers and learners. Administrators can help teachers by recognizing the uncertainty that comes with trying something new and understanding if initial efforts don’t meet expectations. Teachers’ worries or hesitation may decrease if they know their administrator doesn’t expect them to become instant experts in the technology, curriculum, and delivery method. In turn, teachers can be open with students about learning something new while piloting distance education, which might encourage students to participate and keep trying.

Determine your measures of success, including student, teacher, and program outcomes.

Before starting a pilot, define what success looks like. Decide if you are making significant changes across all program elements or just trying a new modality with a group of learners. Consider framing success as gaining insights into which approaches work and which are problematic. While positive student outcomes, such as learning gains and goals met as measured by the National Reporting System (NRS), are crucial for adult education, other measures related to teaching and learning can also be considered during a pilot. These might include recruiting new students from different populations, improving students' digital literacy skills, increasing student persistence, boosting teacher confidence in using technology for instruction, developing new instructional materials, and forming new partnerships. Administrators and teachers must agree on which measures beyond NRS outcomes they will use to evaluate success.

Create an environment that fosters experimentation.

Administrators may need to repeatedly remind teachers and staff about the experimental nature of the project. This is a new concept for many pilot participants, and it might take time for them to accept the message. For instance, it took three to four months before Pennsylvania pilot sites were willing to share their problems and the approaches that didn’t work. It also took time for experimenters to fully understand that the focus was on gaining knowledge and that their efforts to try new methods were among the most valued parts of the project.

Determine what works and what does not.

Emphasize that the goal of the project is to gather knowledge about what works and what doesn't. Help participants see that in pilot projects, we learn just as much from apparent failures as from apparent successes. Encourage participants to explore new and creative ideas rather than sticking to strategies they already know.

Be mindful of the approaches you select and the reasoning behind those decisions. Understanding the causes of success is essential for replicating that success (Reeves, 2006, Figure 1.1). As you work to implement your distance education program, ensure you establish ways for staff to reflect on what is working and what isn’t, along with the reasons why. Teacher reflection logs, supervisor check-ins, and professional learning communities are all methods that can help identify what is effective and what needs to be changed or dropped. Some organizations have discovered that regular team meetings focused on successes and challenges were vital for identifying best practices and areas needing more attention or a different strategy. Conducting such meetings enables organizations to adopt a systematic approach to maintaining and expanding their programs more efficiently and effectively.

Pilot activities should assist organizations in determining whether a specific modality of digital education is a viable option for targeted learners and, if so, how they can best support its implementation. Distance and digital education might be more effective for some organizations than for others, just as distance learning is better suited for some learners than for others.  

Setting a Vision and Developing a Plan

Creating a clear vision and a practical plan with timelines for the distance and digital education program can help gain staff support and address concerns that may come up when trying something new. Many administrators find it useful to develop a vision for the use of the modalities they want to offer students and to plan with a team of stakeholders to turn that vision into reality. This approach ensures multiple perspectives are taken into account and encourages program staff involvement in the planning process. Some key questions to consider when setting a vision and developing a plan include:

• What is the purpose of adding a distance and/or digital education program or course offerings? Are you aiming to reach new learners, increase instruction intensity, enhance students’ technology skills, prevent students from dropping out when they can no longer attend in person, improve outcomes, or a combination of these goals? 
• What modalities best support your vision (e.g., pure distance, blended, hybrid, HyFlex, remote live instruction)? 
• Since this is a new initiative, what are the goals for the pilot, And what are the expectations at the end of it? 
• How can the new program enhance your organization’s strengths? How can it support meeting funders’ expectations, such as achieving the Workforce Innovation and Opportunities Act (WIOA) performance standards? 
• How can leadership for the distance and/or digital education program be distributed? 
• What is a realistic timeline for rolling out the new program?

Organizing the Distance Education Program

Each state and/or organization should identify the modalities (as described in Chapter 1, “Setting the Stage”), instructional materials, and technology to deliver the distance and/or education that will best meet the needs of its learners.

Distance and Digital Education Model(s)

The amount of remote or in-person interaction required for distance and digital education programs can vary widely. Some organizations choose to offer a blended education program, where in-person instruction and online learning are seamlessly combined to boost student engagement. Other programs mainly conduct instruction online using asynchronous tools like digital curriculum and activities, or synchronous methods such as webinars or video chats. Some programs feature open enrollment, allowing learners to start on any day, while others find that a managed enrollment, where a group of students begins together, creates efficiencies. HyFlex classes offer learners the flexibility to choose between in-person instruction, joining remotely, or completing online asynchronous work. The structure of your program will depend on your learners, their goals, and the overall vision and objectives of your program. World Education can assist with technical support to help states and organizations determine which distance education model best meets the needs of their programs and students.

Instructional Materials

Another important decision involves choosing instructional materials. Many organizations opt to use a publisher-developed curriculum as their main instructional resource when starting a distance program. Teachers can then find or develop supplemental activities to fill gaps and enhance skills. These often include open education resources available online or in a library maintained by your organization. Your state will provide guidance on available curricular options for teaching at a distance. As mentioned earlier, to count distance learners in the NRS, states must submit a Distance Learning Policy to OCTAE. The curricula approved for distance learning must be specified in the state policy if a program plans to collect proxy contact hours.  

States may allow currently funded Adult Education and Family Literacy Act (AEFLA) grantees to run distance education programs even if the state lacks an established distance learning policy. However, OCTAE encourages states to create or update a policy as soon as possible. Statewide distance learning policies clarify how the state plans to record instructional hours (if it chooses to do so) and outline policies on student assessment. These policies can also provide important information about distance learning curricula that local programs can use to deliver distance education.

Digital Literacy and Technology Access

A final consideration involves exploring how technology can support the expansion of services and which types of technology to use. Technology can be used to reach more learners, motivate them, offer greater instructional flexibility, and increase resources for teaching and learning. Programs should consider what technology students have access to and what technology skills they need for their future employment and postsecondary education goals. For example, smartphone ownership is becoming more common across all demographics, yet adults with less than a high school diploma or earning less than $30,000 are less likely to own a smartphone than the national average (Pew Research Center, 2024b).

Some students may only go online using a smartphone or tablet because they do not own a computer. U.S. adults with less than a high school diploma are the most likely among all educational attainment levels to have a smartphone but lack access to high-speed internet at home (Pew Research Center, 2024a). The COVID-19 pandemic highlighted the need to address the lack of devices and high-speed internet that many learners face at home. Adult education organizations have found creative ways to close this digital divide by partnering with K-12 school districts, libraries, government agencies, and grant organizations. Programs like EveryoneOn also help users find free or low-cost access to computers, high-speed internet, and local digital literacy training for adult learners.

At the state level, Colorado launched their Digital Navigator Program in March 2023 to tackle the digital divide within its communities. The Learning Source, Colorado’s oldest and largest adult education agency, was chosen as one of the grantees and partners with AmeriCorps to serve nine counties across the state. In California, one adult education agency introduced an innovative Community Digital Navigator Career Technical Education pathway, not only creating a training opportunity for new jobs in an emerging field, but also establishing a sustainable student-to-staff pipeline for their in-house Digital Navigator corps (see Yamashita, 2022, p. 92).

One goal of distance and digital education is to make access easier, so the chosen technology shouldn't become a barrier. Some programs also explore non-tech methods to deliver instruction, such as community-based textbook drop-off locations or mailing resources to students. Additionally, there are instructional apps like Cell-Ed that work on any mobile device without needing a smartphone. These low- or no-tech approaches help reach more students with educational services.

Once you select a model and decide on your instructional materials and technology, you will then focus on planning in the five areas discussed throughout this Handbook: (1) recruitment, (2) assessing readiness, (3) orientation for learners, (4) instruction, and (5) assessment of distance and/or digital learners. In the final activity of the course, IDEAL 101: Foundations of Distance and Digital Education, each organization completes a Distance Education and Digital Education Site Plan for its distance or blended learning pilot. We strongly recommend that a staff team, composed of the administrator and two teachers, completes it. Developing this plan as a team offers several advantages:

• All parties involved in the distance education pilot programs have the chance to participate in designing and developing the experimental program. This not only brings a wider range of expertise to the project but also helps all participants feel a sense of ownership of the pilot. 
• Team planning helps administrators better understand what teachers in their organizations will do and the support they will need. 
• Creating the plan as a team fosters a unified, experimental mindset.

Budgeting

Administrators adding distance and digital education programs need to consider costs related to the instructional approach, instructional materials, communication tools, and staffing. Being strategic in selection is important. Administrators should also ensure sufficient staff time for planning, professional development, teaching, and reflecting on the pilot activities. Staff will need time to learn new technology, become familiar with the curriculum, and organize instructional materials for students. Distance teaching time may not be equivalent to traditional classroom teaching. It can require more time for communication, instruction, and progress monitoring, often needing to occur individually with learners or outside of synchronous sessions.

Some questions to consider: What factors influence the format of the delivery service?  How can technology be utilized to enhance organizational efficiencies in communication, instruction, and program management? When is it appropriate to use free open educational resources (OERs), or when is purchasing a product necessary? Are there more efficient ways to replicate and scale your program, such as creating an online course template for all teachers to use so they don’t have to spend time formatting a course in a learning management system?

Identifying and Supporting Teachers

Identifying Teachers

Teaching at a distance or through other technology-rich methods requires different skills from traditional classroom teaching. (See Appendices C and D for resources to assess teacher readiness.) Some classroom teachers adapt well, while others feel less comfortable in this new environment. Successful distance and digital education teachers are innovative, creative, and adaptable. They are open to new experiences, willing to explore multiple approaches to reach their goals, and able to implement new strategies to meet students’ needs. Effective distance teachers need to be technologically adept, knowledgeable about the curriculum, and capable of building rapport with students remotely. It also helps if teachers are enthusiastic about the opportunities for professional growth and the potential of distance learning to benefit their students.  

Therefore, just as distance learning isn't suitable for every student, distance teaching isn't suitable for every teacher. Whenever possible, you should encourage teachers to volunteer for support roles or to try teaching remotely or through modalities like blended or HyFlex; this increases the chances that teachers will possess the qualities described earlier. A teacher with no digital education experience and limited interest in innovative teaching methods is unlikely to succeed. Since distance and in-person teaching are different, teachers need extra training and an open mind toward new educational approaches to succeed.

Some organizations found innovative ways to utilize their staff’s strengths during the pandemic. For instance, teachers who were uncomfortable with online teaching focused on connecting with students via phone or text to keep communication open. Team teaching enabled teachers with less digital skills to learn from more experienced colleagues. Experienced teachers created online lessons that others used in their teaching. These strategies remain relevant today.

Several administrators have also emphasized the importance of considering digital literacy skills when hiring teachers since all teachers may need to deliver some form of online instruction. IDEAL Consortium states are working together to develop frameworks that assess staff’s digital literacy skills, helping to identify opportunities for professional development and growth.

Supporting Teachers

Administrators must understand and be prepared to support the additional responsibilities that teachers will take on, as well as prepare teachers for the new roles they will assume when teaching remotely or through other modalities of digital education modalities. Administrators need to know what to observe when monitoring new instruction. For instance, observing a teacher's work might look different in an online class. A remote instruction observation tool is available for both administrators and teachers. Additionally, the Virginia Adult Learning Resource Center has developed an Adult Education Indicators of Quality Online Courses rubric that can help identify strengths and areas for improvement.

Additionally, teachers in pilot programs take on a dual role: they are both educators and researchers collecting data about the program. Teachers often need to fill out forms, keep records, and gather data to gain insight into how the program is implemented and how effective it is. They should understand why the data is being collected, feel confident using the data collection tools, and recognize the importance of their role as experimenters. This data collection can take a lot of time and should be factored into teachers’ schedules. If both teachers and administrators understand these extra responsibilities, it will help everyone appreciate the time demands the program places on staff.

It is also essential for teachers to be knowledgeable about the technology required for teaching at a distance or in blended learning and HyFlex classes. Since many distance programs include an online or computer-based component, distance teachers need to be technologically proficient. They must not only know how to use their curriculum's delivery mode but also be able to serve as a technology support resource to help students solve their technical issues. Recognizing this need, you might consider surveying teacher technology skills and organizational technology capabilities as part of the selection process for pilot sites. (See Appendix D.)

Professional Development for Teachers

Good teaching is central to effective distance and digital education for adult learners, and teachers need various support mechanisms as they transition from classroom teaching to distance and digital instruction. Providing teachers with professional development, recognition for their efforts, financial compensation, and opportunities to connect with peers are all ways organizations can ease this transition. IDEAL Consortium states recommend that you do the following:

• Offer professional development opportunities for teachers preparing to teach at a distance. This handbook introduces main concerns and serves as a good starting point, especially when combined with IDEAL 101. Some states have created their own training protocols for distance education, and commercial resources are also available. Visit the EdTech Center website for professional development options, such as webinars on blended and mobile learning. Regardless of the training methods and tools used, teachers will need additional training to be equally effective at teaching at a distance as they are in the classroom. 
• Create mentoring groups where experienced teachers support and guide new teachers. This offers a chance for teachers to collaborate on solving challenges and fosters an environment that promotes professional growth. Texas and California have extensive, well-organized mentoring programs for their distance educators. Teachers benefit from their colleagues' experiences and become part of an active community of practice. For example, the Outreach and Technical Assistance Network in California runs a Digital Leadership Academy that brings groups of teachers together and pairs them with a coach. 
• Recognize that switching from classroom teaching to online teaching is a significant change for teachers. Build an institutional environment that supports them during this transition. Offer resources like conference calls, online chats, and websites where teachers can ask questions and work through the many issues they will face. 
• Understand that to teach effectively, teachers must be thoroughly familiar with the instructional resources. Since distance education programs can be customized, students may join the program at various points. Therefore, the teacher cannot simply stay “one day ahead” of the class and meet the students’ needs. Provide curriculum training and planning time for teachers. 
• Provide financial compensation and/or release time from other duties for teachers working with experimental distance education programs. Consider offering flexible working hours for distance teachers and compensation for the nontraditional hours they are likely to work. It is unreasonable to expect teachers to take on such a demanding task during regular work hours or on top of a full workload and still succeed as distance education teacher.

Monitoring Achievement and Evaluating the Pilot Process

In distance and digital education pilot programs, data play a crucial role. While data on enrollment, hours of instruction, and outcomes might not be the main focus of the distance education pilot, they are still important metrics to track. This quantitative data, along with the qualitative reflections from the pilot staff, can be helpful for monitoring progress and assessing what worked and what can be improved.

Accountability

In an ideal situation, states would exempt organizations from their usual accountability requirements during the initial phase of any new pilot program. We believe that distance education for adult learners is so different from traditional classroom programs that it is like "reinventing the school.” It requires organizations to find different students and develop new ways to teach and engage with them. It clearly demands an extended effort and a period of trial and error to identify best practices (Askov et al., 2003, p. 31).

For example, in some states like Pennsylvania, certain pilot sites were exempt from some of their usual accountability requirements to promote experimentation. These sites were required to report the number of students their Workplace Essential Skills distance education programs served, but they did not need to show evidence of educational gains or progress. This was important for several reasons. It reinforced the pilot program’s experimental nature, encouraged sites to try new approaches actively, and gave both the sites and the state more time to address the unique challenges of assessing distance education students.

Other factors may also influence accountability. For example, the U.S. Department of Education Office of Career, Technical, and Adult Education (OCTAE) decided not to make any judgments about performance success or failure based on 2019 program year data due to the widespread impacts of COVID-19 on adult education services (Stump, 2020).

Data Monitoring

No matter how the accountability of distance and digital education pilot programs is assessed, data monitoring remains a crucial part of the pilot. Organizations and states will need to decide what data to collect and how frequently to review it. For example, Arizona Department of Education staff met with pilot programs twice a year to review student and program data and to discuss the successes and challenges of the pilots.

Administrators can collaborate with the pilot team to decide how distance learners will be assessed. They need to ensure that their organization’s assessment plans align with those outlined in the state distance learning policy. Additionally, they should work closely with both state- and organization-level data staff to ensure that relevant information about distance learners is accurately captured in the data systems. Administrators will also need to train teachers on assessment and data reporting policies and requirements. 

The pilot team can also identify if other data could be useful. Some organizations have involved distance students in focus groups or surveys to gather more feedback about the program

A Note about Data Security and Confidentiality

It is crucial to maintain confidentiality and data security practices in distance education programs. Whether staff are working remotely from home or in the office, it is essential for administrations to establish clear expectations and procedures to protect students’ personal information. If staff work from home, it is also important to safeguard their personal data. For example, staff could set up a Google Voice phone number for students to use, so they do not have to share their personal cell phone numbers.

Moving Beyond the Pilot

Pilot programs have a limited lifespan and are eventually likely to be replaced by larger-scale versions. While the growth of distance and digital education programs clearly depends on state policies and support, the changes are typically implemented by local organizations. At the local level, the goal is to offer distance or digital learning as one of the options for adult learners. A good starting point is to raise organization-wide awareness of the program and how it can benefit students. Many organizations find it helpful to combine recruitment, readiness assessment, and orientation for distance students with similar functions for classroom students. This approach not only eases the workload for distance teachers but also helps to legitimize distance learning within the organization. Some examples of how organizations have integrated distance learning into their activities and services include:

• Include distance and/or digital learning as available options on all recruiting materials, such as websites, brochures, and flyers. 
• Training intake staff to recognize students who might benefit from distance learning. 
• Supporting the professional growth of teachers interested in distance education.

Changes in how education is delivered will not be easy or quick. Incorporating distance and digital education into an organization’s range of services should be considered an “organizational change” effort. First and foremost, if you are thinking about adding distance education as a delivery method, you must base your decision on the educational principles and issues that underpin your organization. This way, you'll ensure your decision aligns with the organization’s mission, making its integration smoother and more likely to succeed.

Experience in the IDEAL Consortium indicates that adopting the following approaches may be helpful for organizations transitioning from an experimental to a programmatic implementation of distance and/or digital education:

• Document the lessons learned during the pilot phase and use them as a basis for future planning. Keep the practices that worked well and drop or adjust those that did not. (See Appendix E for a detailed guide on using webinars to reflect on different stages of the pilot.) 
• Write down how the procedures have changed over time and the reasons behind those changes. This helps to standardize the process and ensures that all participants have a shared understanding of the organization’s approach to distance education. 
• Develop an action plan with strategies to assist participants in transitioning from the idea stage to the implementation stage. 
• Write job descriptions for the key players. This may include teachers, organization administrators, technical support staff, recruiters, and others involved in making the organization’s distance learning project a success. Keep in mind that the nature of distance education may require some flexibility in job roles and assignments. 
• Engage with state-level individuals interested in distance education and make policy recommendations based on their experiences.

Connecting Distance and Digital Education with Workforce Innovation and Opportunity Act (WIOA) Outcomes

The Workforce Innovation and Opportunity Act (WIOA) outlines the performance outcomes for adult education organizations that receive funding through this federal legislation. All workforce development and adult education partners funded by WIOA share the same performance outcomes: job placement, job retention, growth in average earnings, attainment of secondary school and postsecondary credentials, measurable skill gains, and effectiveness in serving employers. Here are some ways that distance and digital education can help organizations achieve these performance goals.

• Increasing student persistence and preventing student stop-out

• Increasing skill attainment necessary for employment and postsecondary education
• Modeling and developing digital literacy and fostering independent, lifelong learning skills and mindsets necessary for the workplace and postsecondary education. 
• Integrating academic skills into a training program to create an Integrated Education and Training (IET) model
• Tailoring instruction to include sector-specific activities that prepare students for the workplace.

Support for Distance Education and Digital Education

The IDEAL staff at World Education is available to support you and your program staff as you pilot distance education and work to incorporate it into your program services.

Piloting Programmatic and Instructional Change at Your Site

References

Askov, E., Johnston, J., Petty, L., & Young, S. (2003). Expanding access to adult literacy with online distance education. National Center for the Study of Adult Learning and Literacy. https://www.ncsall.net/fileadmin/resources/research/op_askov.pdf

Bigelow, M., Vanek, J., King, K., & Abdi, N. (2017). Literacy as social (media) practice: Refugee youth and native language literacy at school. International Journal of Intercultural Relations, 60, 183–197. https://doi.org/10.1016/j.ijintrel.2017.04.002

Ghani, S., & Taylor, M. (2021). Blended learning as a vehicle for increasing student engagement. New Directions for Teaching and Learning, 2021(167), 43–51. https://doi.org/10.1002/tl.20458

Hall, G. E., & Hord, S. M. (1987). Change in schools: Facilitating the process. State University of New York Press.

Huang, Q., & Lee, V. W. Y. (2022). Exploring first-year university students’ blended learning experiences during the COVID-19 through the community of inquiry model. The International Journal of Information and Learning Technology, 39(4). https://doi.org/10.1108/ijilt-02-2022-0024

Moore, M. G. (2002). Editorial: Learning the necessary principles. American Journal of Distance Education, 16(3), 129–130. https://doi.org/10.1207/s15389286ajde1603_1

Pew Research Center. (2024a, January 31). Internet, broadband fact sheet. https://www.pewresearch.org/internet/fact-sheet/internet-broadband/

Pew Research Center. (2024b, January 31). Mobile fact sheet. https://www.pewresearch.org/internet/fact-sheet/mobile/

Reeves, D. B. (2006). The learning leader: How to focus school improvement for better results. ASCD.

Shea, P., Hayes, S., Smith, S. U., Vickers, J., Bidjerano, T., Pickett, A., Gozza-Cohen, M., Wilde, J., & Jian, S. (2012). Learning presence: Additional research on a new conceptual element within the community of inquiry (coi) framework. The Internet and Higher Education, 15(2), 89–95. https://doi.org/10.1016/j.iheduc.2011.08.002

Stump, S. (2020). Adult education and family literacy act and COVID-19 – frequently asked questions. U.S. Department of Education, Office of Career, Technical, and Adult Education. https://www2.ed.gov/about/offices/list/ovae/pi/AdultEd/octae-program-memo-20-3.pdf

Vanek, J., King, K., & Bigelow, M. (2018). Social presence and identity: Facebook in an English language classroom. Journal of Language, Identity & Education, 17(4), 236–254. https://doi.org/10.1080/15348458.2018.1442223

Vaughan, N. D. (2010). A blended community of inquiry approach: Linking student engagement and course redesign. The Internet and Higher Education, 13(1-2), 60–65. https://doi.org/10.1016/j.iheduc.2009.10.007

Yamashita, J. H. (2022). Connecting the community: Introducing the digital navigator. COABE Journal: The Resource for Adult Education, 11(2), 87–93. https://www.proquest.com/scholarly-journals/connecting-community-introducing-digital/docview/2892083310/se-2?accountid=9649