Introduction

In 2022, a team of learning experience designers at WGU Labs redefined collaboration as the result of an intricate research project. This project involved working in partnership with both internal and external collaborators to facilitate co-design activities with online learners. This paper will briefly review some background information about the project this research is part of, share key learnings that may serve as a checklist for others who are interested in undertaking similar collaborative projects, provide a narrative of the research activities, and discuss both the research findings and reflections on the processes and practices that were part of this effort.

Previous research at a primarily online university revealed differences in online student graduation rates, especially for those who entered the institution with more transfer credits. Further analysis revealed that students with low incomes were more likely to have fewer credits and therefore less likely to graduate. When evaluating the data based on race and ethnicity, the team found that Black, Indigenous, and students of color (BIPOC) were more likely to fall into this category and were impacted more than white, higher income students. The analysis also identified a key turning point in the student journey. While gaps remained, persistence into the second year led to increased graduation rates across all groups, leading the team to focus on the first year as an opportunity for improvement. 

As a result of these findings, a team of learning experience designers and researchers developed a participatory research study to better understand the experiences of BIPOC and low-income learners during the first year of college in order to identify potential solutions to assist these students into the second year. This paper will discuss the development and implementation of the participatory research design, and highlight opportunities for collaboration across the stakeholder team.

Setting Strong Foundations for Collaboration

Perhaps the most important takeaway from this collaborative effort was the need for dedicated, routine communication early in the project to negotiate, establish, and revisit the motivations for undertaking any project. Krause (2020) offers outlines for these types of discussions along with suggested artifacts to co-create with stakeholders, such as a funding web, which can help visualize the relationships between power, money, and the key decision-makers in any collaborative project. Artifacts like the funding web, and similar variations, can bring visibility to influencing factors that would otherwise go unstated, and help illustrate the connections among those who ultimately have the power to approve or change the course of the project. Power does not always correlate to who has the most money. Krause (2022) emphasizes that often, those with the data should be considered among those with a lot of power in a project, as research project success mostly hinges on access to data. While a funding web might not be the best fit for every project, at the very least, conversations should be had around which individuals or teams hold ultimate responsibility for various stages of a project, and the reasons why should be documented in a way that is clear and easy to revisit as the project unfolds. 

Once the stakeholder team has identified who will be at the proverbial table, and how much power each party will have in the day-to-day tasks of getting the project done, goals and motivations for the project can be discussed and prioritized according to the relationships identified from the funding web discussion. Motivations will shape the goals for the project, and it is important for all relevant stakeholders to have a clear understanding of what the project goals are and are not, and why the team has chosen to include some goals and not others. For example, in addition to providing a rich account of the first-year experience from the perspectives of BIPOC/low-income learners, there were several parallel goals stakeholders had in mind for the research detailed in this paper. First, recognizing no one at the table had the lived experience needed to truly inform a solution for BIPOC and low-income learners, all stakeholders agreed authentic student participation in the identification of possible solutions was a high priority. Second, those who would interact with participants wanted to intentionally commit to mindsets for conducting ethical participatory research with historically marginalized groups in higher education. 

It is also key to understand any constraints at this stage of the work and to think critically about how those constraints may impact different stakeholders and their work towards the project. Use the outcomes from the funding web to develop a process for alerting the rest of the stakeholder team to any unanticipated impacts as a result of constraints, and know which individuals or teams have the ultimate power to approve or deny workarounds that may impact the project plan.

Key Learning: Foundation Conversation Checklist

• Identify all stakeholders.

• Identify relationships among stakeholders with regard to money, power/influence, and other important factors.

• Determine stakeholder motivations.

• Translate motivations into clear project goals.

• Discuss project constraints and how workarounds will be determined.

Regarding the project at the center of this paper, the stakeholder team determined the following during the foundation stage:

• Who is at the project table: The research team (WGU Labs learning experience designers and researchers), partner institution’s Core Team (department leads and relevant staff who will support the research team), and the grant organization (represented by an individual who will receive progress reports at predetermined times throughout the project). 

• Project Motivation: Center the perspectives of first-year BIPOC and low-income students through a participatory research process.

• Project Goal: Develop interventions to support BIPOC and low-income students during the first year at a primarily online institution to increase enrollment into the second year.

• Potential Constraints: Budget, timeline, and access to study participants. 

Theoretical Framework & Rationale

After the foundational conversations have been had, those involved in conducting the actual research can move forward with the research process. At this stage, not all stakeholders will be as involved as they were in the initial conversations, but communicating each phase of the research process is a key activity that should be carefully considered. Ensuring all stakeholders are informed will allow for more efficient collaboration of all groups as needed throughout the project’s duration. Communicating the research questions and theoretical framework clearly, as well as the rationale behind them, is the first step. 

Time invested in the foundation-building conversations allowed for a straightforward translation of project motivations and goals into research questions. As a result, the learning experience designer-led research team generated a theoretical framework for the project which clearly connected to the methodology. When communicating the process of developing the research questions and theoretical framework, stakeholders were most interested in the reasons why the scholarship was appropriate for the goals and motivations of the project, and how that research connected to the methods the team selected for data collection and interpretation of findings. This helped the stakeholders understand what to expect as the project unfolded. 

The following research questions were used to guide the design of methodology and participatory data collection activities.

1. How do students’ experiences during the first year differ based on race, ethnicity, and socioeconomic status? (First-year experiences include: (a) navigating the university; (b) community/belonging; (c) academic experiences; and (d) relationship-building experiences)

2. What experiences do students identify as milestones or pain points?

3. How do students’ perceptions of the first-year experience support or conflict with the goals of proposed pilot initiatives?

After the research questions were developed, the research team identified appropriate scholarship to drive the methodology. Since this research project aimed to understand the specific experiences of students through their intersectional identities in a participatory way, the team explored diversity science literature. Diversity science, as explained by Plaut (2010), considers the subject of differences among people and the ways in which people identify and uphold differences, particularly with regard to race and ethnicity. In addition to studying the boundaries people create through processes and social interactions, diversity science also examines the consequences of these distinctions (Plaut, 2010). “These significant social distinctions are not simply natural, neutral, or abstract,” Plaut (2010, p. 77) argues, and encourages researchers to examine the ways institutions have contributed to inequality through practice and policy decisions. It is this directive in particular that makes diversity science a worthwhile theoretical grounding for the project at hand. Rather than situating the problem of persistence as a deficit of BIPOC and low-income learners, the research team wanted to focus on the barriers created by the system and their effects on BIPOC and low-income learners specifically. 

With this core belief at the forefront of the research process, the team explored ways to weave participatory action research methods and user experience research methods together to create a participatory inquiry process in alignment with the additional goals stated by the stakeholders and research team. The protocol intentionally prioritized students and also recognized the significance of specific staff and faculty groups in any effort to transform the first-year experience. Participatory research is an umbrella term for a variety of methods that share some core values, primarily a commitment to “alternative models for the creation of transformational knowledge” (Bradbury, 2015, p. 6), and is often emergent with a focus on knowledge in action (Bradbury, 2015). These core values drove the protocol design early on and led the research team in choosing appropriate methods from user experience research to make up for the activities participants would partake in during the data collection. At the beginning of the project, the team intended to conduct true participatory action research, which emphasizes collective inquiry with the ultimate goal of cooperative action for change and also prioritizes critical reflection on the ways researchers create knowledge with, rather than about, study participants (Bradbury, 2015). Astute readers may already question the lack of student voice at this stage of the project, and this case study will detail the constraints that led to concessions regarding this goal later in the paper. 

User research, and specifically user experience research, was also selected for this project because of the technology-heavy environment students, staff, and faculty use to interact at a primarily online institution, and because the ultimate project goal was to design solutions that could ultimately be conducted as pilot tests later on. Kuniavsky et al. (2012) describe user research as a “process of figuring out how people interpret and use products and services” (p. 3). Because the research team chose to frame the problem of achievement as an issue with the system and wanted to explore the system’s effects on a specific population of students, who were acting as users, this approach was valuable for providing structure around the activities the team would facilitate with participants. The team selected interviews, surveys, and focus groups as the primary data collection strategies, but emphasized focus groups as an opportunity to apply the participatory research lens. 

Key Learning: Communicating and Establishing Trust in the Research Process

One of the main challenges the team faced from this stage onward was justifying the rigor of the chosen methods for conducting the research and interpreting findings. Rigor in applied research is always challenging to communicate, and the rigor of participatory research is acknowledged as complex (Bradbury, 2015). Stringer (2013) describes rigor in action research as having designated checks for trustworthiness. The outcomes of the research must be questioned and justified to ensure they are not reflections of the biases of the research team (Stringer, 2013). Qualitative research practice offers strategies for ensuring the trustworthiness of findings, including triangulation and member-checking, along with providing thick description (Creswell & Miller, 2000). Thick description can often be in contention with the practical realities of stakeholders’ available time and existing expertise. 

Communicating the rigor or trustworthiness of the research required some time spent educating stakeholders, and a delicate balance of sharing the right amount of information to satisfy the stakeholders without requiring significant and unplanned amounts of their limited time. In hindsight, collaborating with the stakeholder team around the indicators they expected to see signal rigor or trustworthiness in the results would have been a productive effort to include in the foundation conversations described previously in this case study. Consensus on those parameters would have allowed the research team to develop a protocol that addressed concerns ahead of time rather than set up a situation where the stakeholders questioned the results. 

Research Population and Sample

In previous sections of this case study, the research participant population has been broadly identified. Participants were recruited via mass email campaigns to students who met the inclusion criteria outlined below. A recruitment list containing email addresses for students meeting the inclusion criteria was generated by the institution and oversampled to increase the representation of Indigenous students based on findings from the discovery phase of research. This list was then segmented into three smaller lists, and each group received an invitation to participate in either a survey, focus group, or interview. The size of the total recruitment pool was generated based on an estimated participation rate of 15% according to the institution’s internal marketing statistics. In other words, the team expected 15 percent of students who received the email would participate in the study. Using the goal participation numbers below, the institution representative calculated a list size that was most likely to generate the needed student participation numbers. The invitation emails were distributed through the institution’s main email communication channel. The research team utilized Zoom registration to manage sign-ups for focus groups, Calendly to manage sign-ups for interviews, and the survey email containing a direct link to the survey in Qualtrics. The research team provided the text of the email invitations to a representative from the institution who facilitated the distribution process to maintain the security of sensitive student contact information. 

Student Participant Inclusion Criteria

• Enrollment date within the past calendar year.

• Less than 21 transfer credits.

• Self-identified as Black, African American, Indigenous, Hispanic, Latinx, or Hawaiian Pacific Islander in the institution’s student information system. 

• Expected Family Contribution of less than $35,000.

The research team set target participation rates for each data collection strategy based on recommendations for user research methods by Baxter et al. (2005).

• Interviews: 6-10.

• Focus Groups: 5-10.

• Survey: up to 100.

To recruit faculty and staff, a similar approach of inviting participants to take a survey, complete an interview or join a focus group was utilized. The inclusion criteria for staff and faculty are summarized below.

Staff Participant Inclusion Criteria

• Experience advising first-year students.

• Member of Financial Aid Services staff.

• Member of Admissions staff.

• Manager recommendation (to allow work time to be used for participation).

Faculty Participant Inclusion Criteria

• Greater than one year of experience teaching at the institution.

• Instructors in a specific subset of general education courses who are likely to have high first-year student enrollment.

Students and faculty were compensated $50 for their time participating in the research regardless of which activity they completed. Staff were granted release time to participate in the research during work hours. Participation rates are summarized in the table below.

Table 1

Participation rates across data collection activities

Note. Goal participation rates were derived from user research guidance by Baxter, et al. (2005).

Methods

The research team designed participatory activities to be conducted during focus group sessions. McKercher’s (2020) co-design inspired the team to explore activities that would allow for the sharing of power, development of relationships among the participants for the duration of the collaboration, and allow the participants to share insights from their lived experience in spaces designed to maximize the safety of traditionally marginalized participants. Ensuring participation in the research did not subject BIPOC and low-income students to repeated harm was a crucial ethical requirement for the research team, and, as a result, the team decided to host student-only sessions to eliminate the power differential among students and staff/faculty. Staff and faculty participated in comingled focus group sessions. 

Co-design inspired activities were designed to be driven by participant conversations. The team used the initial discovery research detailed in Footnote 1 to develop prompts in the form of “How Might We Statements” (IDEO, n.d.) as conversation starters, but the facilitators were prepared and able to support the students' lead, should sessions shift topics to other sources of frustration in the first year. The same prompts were utilized for the faculty/staff focus group sessions. Miro, a virtual whiteboard tool, housed the protocol. All participants were expected to join sessions virtually, which required an online tool that supported flexible collaboration. A combination of facilitator experience and free guest user access led the research team to select Miro for this purpose.

These co-design focus group sessions began with a practice activity: “What are the different uses for a brick?” This was designed to expose participants to the features of Miro in a low-stakes way and to practice the type of exploratory thinking the facilitators hoped to elicit during the session. After the warm-up, participants were introduced to the first prompt, and the remaining time was divided among identifying pain points, brainstorming solutions, categorizing ideas, and voting on potential solutions. After each session, a follow-up Qualtrics survey was distributed as a way of virtual member-checking. Participants were asked to review the Miro board and reflect on the session. 

A tool that proved highly valuable in thinking through the setup of the collaborative activities as well as the analysis of data was the Creative Matrix (LUMA Institute, 2021). The matrix is primarily used to generate solutions to problems at the intersection of complex topics. The table below is an example of a Creative Matrix. The research team positioned the “How Might We” prompts as column headings, and developed solution categories within each row. One hallmark of the Creative Matrix is the “wildcard” row, which allows for solutions which don’t fit into the pre-designated categories. The Creative Matrix is a way to see relationships and ensure specific intersections of complicated challenges are not unintentionally overlooked. 

Figure 1

Creative Matrix example

Recognizing synchronous group participation would be a barrier for many students who met the inclusion criteria beyond those who required accessibility accommodations, the team realized the need to provide alternative methods to authentically include all interested voices. After the focus group protocol was created, the research team used that protocol to develop an emergent interview guide and survey that covered the same topics. Mimicking the flexible and emergent qualities of the focus group sessions was easier to do for interviews, as interviewers only needed to be skilled in asking appropriate follow-up questions to allow the interviewee the chance to lead the conversation. However, designing a straightforward survey that allowed for similar choice and flexibility proved to be a greater challenge. The Creative Matrix structure inspired a combination of branching logic and open-ended questions to be used to create a survey in Qualtrics to meet the unique needs of the research team. 

Analysis & Findings

After all the sessions were complete and all interviews and surveys had been recorded, the research team divided up the data to conduct a thematic analysis. The goal of this process was to individually identify and interpret any patterns that emerged from the transcripts, memos, and survey responses before coming together as a whole group to interpret findings together. Across a period of three weeks, the team met to derive conclusions from the data collected. Analyzing the data collected across participant groups uncovered some interesting pain points which were described by both students and staff/faculty groups, albeit in different ways. The research team termed these “friction points” and could clearly articulate the impact these friction points were having for both students and staff/faculty, summarized below: 

Friction Point 1: Community and Connection

In this first friction point, students reported a desire for more personalized treatment within the institution. Throughout the co-design focus group sessions and in survey responses, students frequently requested opportunities to connect with faculty and other students to develop relationships, particularly with students and alumni to develop career connections, and with faculty for the purpose of video-based instruction. This friction point was highlighted by staff who pointed out they didn’t know where to refer students when they requested additional academic support when their instructors couldn’t offer real-time connection opportunities and highlighted the lack of social connecting spaces for students to develop their own communities within the institution. 

Friction Point 2: Information Sharing and Transparency

Staff frequently pointed out opportunities to better support students through increased collaboration across departments and a desire for more communication of policies and procedures for referring students to different departments at the institution. In one example, a staff member highlighted the consequences for a student who was stuck in a revolving-door scenario of being referred back and forth between two different departments because neither party understood the procedures, leaving the student frustrated. This was reflected in students’ wishes for more streamlined support and desire for one-stop-shop types of experiences when interacting with multiple departments.

Friction Point 3: Student Expectations vs. Student Reality

The third friction point the data supported was nebulous at first, but as the research team uncovered additional anecdotes, it became clear there was alignment between what one support staff member termed the “tidal wave of reality” and what students described as challenges adapting to the academic demands of school. Many students reported their excitement around school dwindled after their first classes, and this was corroborated by advisors and other support staff who noticed a widespread decline in enthusiasm from students after they got their first final grades back. It was difficult to pinpoint exactly what may cause this misalignment of what students expect their academic experience to be like and what they experience, and further analysis of the data may be needed to uncover additional insights. This is an area the research team plans to prioritize in future participatory research efforts around first-year student retention. 

Pilot Strategies

Recall that the ultimate goal of this phase of the project was to identify possible pilot support strategies to increase the retention of BIPOC and low-income learners into the second year at the partner institution. The study team translated the three friction points into recommendations for intervention to be discussed among the stakeholder team. Some of the recommendations included: 

• Developing intentional social connection opportunities for students to learn from advanced students within their program of study and from recent alumni.

• Exploring the impacts of a community of practice model that could increase transparency and collaboration for staff and potentially lead to reduced administrative barriers for students.

• Designing a risk-free first-term experience to allow prospective students to test the experience of school before committing to enrolling long term.

Discussion of Key Learnings from the Research

The goals of this research were lofty and complex. While some key insights have been discussed in their respective sections, this discussion section will serve as a reflective space to share the lessons learned around carrying out participatory research and the challenges the research team encountered when communicating and coordinating responses to unexpected barriers.

With regard to the research outcomes, it is important to understand the findings were certainly not one size fits all. The facilitators recorded many opposing viewpoints. Some students argued vehemently against the suggestion of synchronous video sessions with faculty, but an equally passionate subset of students and faculty agreed that real-time communication could resolve many of the challenges routinely faced in the learning process. Similarly, many staff viewed their jobs as call center representatives and had little interest in understanding or collaborating beyond their immediate team. These mixed results were expected by the research team, but communicating the nuances of participant voices without watering down the recommended pilot initiatives to external stakeholders proved challenging. 

The design of the co-design focus group sessions was treated with immense care in the form of time spent thinking through different ways to respectfully engage and elicit deep insights from students, staff, and faculty alike, while paying attention to conducting the research in a systematic and rigorous way. At times it seemed the goals of participatory research and traditional research expectations were at odds and navigating those points of conflict is something researchers need to continue wrestling with for future work in this space. The team discussed ad nauseam how much structure was too much structure while trying to avoid the pitfalls of not providing enough instruction or prompting to encourage productive dialogue.

When facilitating the co-design focus group sessions, the research team operated under the assumption that all students would have a laptop to use for the session because laptops or desktop computers were required by the institution to take classes. However, it was quickly discovered that many students who met the inclusion criteria were using mobile phones as their primary means of accessing school-related platforms and services, including the research session. This made the activities difficult for participants to complete and required adept pivoting on the part of the facilitators to ensure sessions were still productive and achieved the research objectives. Luckily, the time spent engaging deeply with the literature of diversity science, Krause’s data equity workshop (2022), and the principles of Design Justice (Costanza-Chock, 2020) provided a strong north star for navigating those moments in real-time.

The process of recruiting the necessary participants for the study was perhaps the most difficult component. As readers may have noticed, the target participation numbers presented in the table above are quite messy. While recruitment was successful in filling the seats for focus groups and interview time slots, when it came to attending the sessions, conversion rates lagged significantly. This had a domino effect on the co-design focus group sessions, which were designed for 10-15 participants. Actual participation rates were around 3-5 students and 5-7 faculty/staff, despite having full registration. This was similar to participation in interviews; it was very common for interviewees to miss the time they signed up for. Due to constraints of timeline and budget, the research team decided to stop rescheduling synchronous engagements and offer the survey to students who were unable to attend focus groups or interviews instead, which led to an unexpected number of survey responses to analyze and increased time allocated to analysis. 

In spite of these challenges, both the research team and stakeholders agreed the sessions were instrumental in shifting longstanding beliefs administrators and other stakeholders held about the desires of students at primarily online institutions.

Conclusion

In summary, despite careful attention to setting appropriate foundations to establish goals and ensure clarity of motivations, this participatory research effort still had its share of significant challenges. The most important finding for the learning designers involved in this study is a greater grasp of what it takes to shift future participatory research projects away from the conventional "transactional" experience and toward the “transformative” experience (McKercher, 2020, p. 17). Students should have had a formal decision-making role on the stakeholder team from the start of this collaborative project, but due to unavoidable yet foreseeable institutional constraints, their input wasn't sought until after the stakeholder team and research team had already made many decisions. The research team suspects this is the fate of many participatory efforts at campuses across the country. While the team acknowledges that some constraints are inevitable, it is important to be clear upfront with stakeholders about what is required to undertake a participatory research project and communicate the tradeoffs when concessions need to be made.

To support a conversation regarding such tradeoffs, Vaughn & Jacquez (2020) provide a model they term “Choice Points” for participatory research which can guide researchers, stakeholders, and participants to areas of opportunity for increased participation of appropriate communities. Their model centers the research process and provides different levels of participation, ranging from “Inform” to “Empower,” along with methods for engaging different groups at each level of participation across all phases of the research process. This is a valuable model that will significantly shape the research team’s approach to future work in this space.

Finally, there are obvious tensions in doing emergent, flexible, student-led work which competes directly with traditional business priorities like deadlines, compensation, return on investment, and other things institutions with financial obligations typically value. It was common throughout this project to encounter questions such as, “Can students really tell us what isn’t working for them?” and “What is the statistical significance of this work?” There was also fear and anxiety about what students may say or request during these engagements when they are empowered to leverage their voices and skills. It is important to name these fears and lean into them. Letting students lead requires intentional disruption of the typical balance of power in traditional higher education. Otherwise, we will continue reproducing our oppressive systems.

References

Bradbury, H. (2015). The SAGE handbook of action research. SAGE Publications Ltd. https://doi.org/10.4135/9781473921290

Baxter, K., Courage, C., & Caine, K. (2015). Understanding your users: A practical guide to user research methods. Elsevier Science & Technology.

Creswell, J. & Miller, D. (2000). Determining validity in qualitative inquiry. Theory into Practice, 39(3), 124-130.

IDEO. (n.d.). How might we. Design Kit. Retrieved from https://www.designkit.org/methods/how-might-we.html

Krause, H. (2020, July 24). Introduction to the funding web. We All Count. Retrieved from https://weallcount.com/2020/07/24/introduction-to-the-funding-web/

Krause, H. (2022, July). We all count data equity bootcamp. [Workshop].

Kuniavsky, M., Goodman, E., & Moed, A. (2012). Observing the user experience: A practitioner's guide to user research (2nd ed.). Morgan Kaufmann.

LUMA Institute. (2021). Creative matrix. LUMA Institute. https://www.luma-institute.com/creative-matrix/

Martin, B. & Hanington, M. (2019). Universal methods of design. Rockport.

McKercher, K. A. (2020). Beyond sticky notes. Doing co-design for Real: Mindsets, Methods, and Movements, 1st Edn. Sydney, NSW: Beyond Sticky Notes.

Plaut, V. C. (2010). Diversity science: why and how difference makes a difference. Psychological Inquiry, 21(2), 77–99. http://www.jstor.org/stable/25704854

Stringer, E. (2013). Action Research (4th ed.). SAGE Publications, Inc.

Vaughn, L. M., & Jacquez, F. (2020). Participatory research methods – Choice points in the research process. Journal of Participatory Research Methods, 1(1). https://doi.org/10.35844/001c.13244

¹ A note about the first-year experiences of interest: the research team derived those four particular areas as a result of the discovery research that took place during the 2021-2022 academic year, prior to the phase of work covered in this paper.