Introduction

Asynchronous Online Course Discussions (AOCDs) continue to be an omnipresent educational activity within online higher education (Fehrman & Watson, 2021). Often compared to in-person class discussions that are spoken and in real-time, AOCDs are typically assigned to span one or two weeks and primarily involve students posting text-based messages to which other students in the course are expected, often required, to read and respond to asynchronously. Originally hypothesized as having potential benefits over in-person discussions as they were “freed of the constraints imposed by time and space” (Henri, 1992, p. 118), AOCDs were later described by critics as “over-cultivated factory farms, in which nothing unexpected or original is permitted to flourish” (Morris & Stommel, 2018). The research conversation about how to best design, implement, assess, and evaluate AOCDs has been taking place at least since Henri’s excitement about the “gold mine of information” they make available (1992, p. 118).

Much AOCD research relies on content analysis. Content analysis assumes that AOCD discussion forum transcripts are an exteriorization of learning processes and that latent variables like critical thinking (Newman et al., 1995), cognitive presence (Garrison et al., 2000), and collaborative knowledge construction (Gunawardena et al., 1997) can be identified and coded in the AOCD message text (de Wever et al., 2006). Research using this approach has measured levels of cognition (Al-Husban, 2020), explored language indicators for students’ feelings of belonging (Zengilowski et al., 2023), compared AOCD content quality between different interfaces (Hou et al., 2015; Z. Sun et al., 2018), and observed the impact of different course design and instructional interventions (Giacumo & Savenye, 2020; Wise et al., 2012). While there are exceptions (Galikyan & Admiraal, 2019), much of the content analysis research has found that when AOCD posts are coded against a taxonomy of low to high quality of thinking, students’ posts tend toward the lower levels (Bai, 2012; Ertmer et al., 2011; Garrison et al., 2000; Gunawardena et al., 1997; Kanuka et al., 2007; Shearer et al., 2015; Wise & Chiu, 2014).

As LMS data has become more accessible (Psaromiligkos et al., 2011), researchers have been able to better consider learners’ behaviors within their AOCDs. Studies conducted within this realm have used social network analysis (SNA) to explore levels of interaction (da Silva et al., 2019) and degrees of closeness as related to knowledge construction (Ye & Pennisi, 2022). LMS trace data analysis has shown that participation requirements significantly shape learners’ AOCD posting behaviors (Lee, 2012) and that habitual learner behaviors informed by the design of the AOCD interface itself can create unanticipated results wherein certain threads persist and others are ignored regardless of discussion quality or learner intent (Hewitt, 2005). Using LMS data to investigate what learners do in their AOCDs has expanded how researchers conceptualize AOCD behavior, moving beyond a singular focus on the text of message posts to also include the construct of “listening,” correlating to the active reading of other messages (Chai et al., 2020; Wise & Hsiao, 2019).

In exploring learners’ perceptions of their AOCDs, researchers have relied largely on quantitative surveys and open-ended text-based questions. The Community of Inquiry (COI) validated quantitative survey (Arbaugh et al., 2008) designed to measure students’ experiences of social presence, cognitive presence, and teaching presence in their AOCDs has been widely deployed (Stenbom, 2018). Such studies often consider the impact of various course design and instructional interventions on measures of one or more of the three COI presences (Sadaf et al., 2021). Others have combined quantitative surveys with open-ended text-based questions to glean more nuanced information on learners’ perspectives regarding their AOCDs (Ebrahimi et al., 2016; Scott & Turrise, 2021; Tibi, 2018).

Despite the proliferation of research into AOCDs, except for the need for structure in online discussions, there are relatively few “settled” areas or agreed upon best practices for their design and implementation (Fehrman & Watson, 2021). Additionally, the bulk of research on AOCDs is conducted without asking learners themselves to describe how they actually experience their course discussions (Rourke & Kanuka, 2007; Sullivan & Freishtat, 2013). Notably, this insufficient attention to learners’ lived experiences has also been observed in online learning overall (Veletsianos, 2013) as well as education more broadly (Seidman, 2013). While useful information about what learners’ do and experience is provided by content analysis, learner analytics data, and surveys, in-depth investigation of learners’ experiences with all the attendant complexity requires qualitative methods like interviewing.

Learner Experience

Considering education through the lens of learner experience goes back at least to Dewey (Dewey, 1938) who articulated a full theory of experience and advocated for its central importance. Learner experiences for Dewey (1938) were always a transaction between the learner and their environment:

An experience is always what it is because of a transaction taking place between an individual and what, at the time, constitutes his environment, whether the latter consists of persons with whom he is talking about some topic or event, the subject talked about being also a part of the situation . . . The environment, in other words, is whatever conditions interact with personal needs, desires, purposes, and capacities to create the experience which is had. (pp. 43-44)

While Dewey wrote well before online education, his experience framework was later built on by instructional design scholars who asked “What would instructional design be like if we gave more attention to how people really encounter and engage in instruction-both learners and facilitators alike?” (Parrish et al., 2011, p. 16). Like Dewey, they emphasized the idea of experience as a transaction between designed elements and individual characteristics.

Most recently within the field of learning, design, and technology conversations are taking place regarding an emerging line of inquiry into learner experience (LX) and learner experience design (LXD) (Schmidt, Tawfik, et al., 2020; Schmidt & Huang, 2022). This developing LXD framework, similar to Dewey (1938) and Parrish et al. (2011), emphasizes experience as a transaction, taking place at the intersection of multiple factors. Schmidt, Tawfik, et al. (2020) state:

LX is not only concerned with the effectiveness of designed learning interventions, but also with the interconnected and interdependent relationship between the learner- (or the teacher-/instructor-) as-user, the designed technology, novel pedagogical techniques or instructional strategies, and the learning context. (n.p.)

There are several diverse areas being considered within an LXD framework including, but not limited to, appropriate research methods (Schmidt, Earnshaw, et al., 2020), socio-technical pedagogical usability considerations (Jahnke et al., 2020), theories of change (Bowen et al., 2020), and the role of needs assessments (Stefaniak & Sentz, 2020). Additionally, the theories in which LXD is grounded emphasize both individual learning experiences (i.e., cognitive load theory) and the collective nature of learning (i.e., activity theory, distributed cognition) (Schmidt, Earnshaw, et al., 2020).

When it comes to considering AOCDs within an LXD framework, there are two takeaways from the broader conversation that should be emphasized. The first is that we need to dedicate more attention, both in instructional design and research, to the user interface and usability of the technology tools with which learners are interacting. This requires adopting and adapting methods from fields like human computer interaction (HCI) and user experience (UX) research, because for too long we have neglected the impact these technologies inevitably have on learning experiences. Secondly, though, an LXD focus cannot stop with UX considerations as our concerns necessarily go beyond the intuitiveness and ease of use of the interface. Consider, for example, a course in which the AOCD interface is pleasing to use, easy to navigate, and facilitates a seamless exchange of messages between its users. Yet, that same course has poorly written discussion prompts, an instructor communication style many students experience as off-putting, and a grading structure that disincentivizes deep thinking. In this example case, the UX might be excellent while the overall LXD is poor. Ultimately, investigating AOCDs through an LXD framework requires simultaneous attention to the variety of elements including, but by no means limited to, the UX of the mediating tool.

Research Focus

For at least the foreseeable future, ACODs will remain a staple of online higher education. And, despite a proliferation of empirical research, there remains much to learn from in-depth qualitative explorations of the learners’ own experiences. LXD offers a useful framework in ensuring that the multiple relevant factors impacting learners’ experiences receive necessary attention. Toward that end, this paper reports on an in-depth qualitative study, made up of over 26 hours of interview data including think-aloud observations, investigating the following:

When considered within an LXD lens, how do online students experience their required course AOCDs and which are the most salient elements mediating those experiences?

This paper then proposes a new conceptual model for future LXD investigations into learners’ experiences of other technology-mediated educational activities.

Study Design

To investigate learners’ experiences with their AOCDs using an LXD lens, this study employed a strategic integration of established interpretative qualitative methods (Creswell & Poth, 2016) and UX evaluation methods (Schmidt, Earnshaw, et al., 2020). The study employed the well-established three-interview sequence designed to facilitate increasing depth and nuance into the participants’ experiences of their AOCDs (Seidman, 2013). In Seidman’s (2013) framework, each of the three interviews is semi-structured, loosely guided with topics but also evolving naturally between the participant and researcher (Brinkmann & Kvale, 2015). In Seidman’s model, the first interview, focused life history, is intended to build rapport between the researcher and participant and establish the relevant context of the participant’s life involving the phenomenon. The second interview, details of the experience, is intended to draw out in as much detail as possible the participant’s experience with the phenomenon. Finally, the third interview, reflections on the meaning, allows for the participant to reflect on their experience of the phenomenon and provide additional insights into it (Seidman, 2013). Because qualitative interviewing alone, however, was not sufficient to access the “design” elements of LXD, the second interview, details of the experience, was extended to also include think aloud observations (TAO) (Gregg et al., 2020; Schmidt, Earnshaw, et al., 2020).

As an interpretative qualitative inquiry, this study did not aim for statistical generalizability but rather transferability wherein readers should be able to transfer relevant findings of the research to their own contexts (Tracy, 2010). The responsibility of phenomenological interviewers is to “come as close as possible to understanding the true ‘is’ of [the] participants’ experience from their subjective point of view (Seidman, 2013, p. 17).” To ensure quality throughout the study, the researcher adhered to established methods within qualitative research for rigor, trustworthiness, and quality. This included extended time with participants and commitment to analytic coherence (Nowell et al., 2017).

Study Setting

This study relied on criterion sampling (Creswell & Poth, 2016), which is recommended for phenomenologically motivated research inquiries (Englander, 2012). The setting was a fully online, asynchronous Master of Professional Studies (MPS) program that was offered by a large state university. This program was selected as the study site because each course included required AOCDs, with participation comprising a non-trivial part of the students’ final grades (typically worth 10% - 20%).

Participant Recruitment

All students who had completed the first course in the recommended program sequence were invited to participate in the study. Participants were offered a $10 gift card for each interview they completed and enrolled into a $50 lottery for completing all three interviews.

Data Collection

Three sequential semi-structured interviews, with the second interview including a TAO, were conducted with eight individual graduate students (see Table 1).

Table 1

Student Participant Cases With Course History

 *Pseudonyms

**All demographic data was self-described

***In-progress refers to courses being taken at time of interviews

In total 24 interviews were conducted and over 26 hours of interview data were captured. All interviews were conducted through web conferencing, fully recorded, 45 to 90 minutes long, and scheduled roughly one to two weeks apart. Examples of topics discussed in the first interview, focused life history, included: “Tell me about yourself.” “What’s your typical week like?” “How did you come to be in this graduate program?” “Please talk about your past education experiences.” “What were your previous experiences with discussions?” “If a friend were to describe the type of student you are, what would s/he say?” “What are the strengths & weaknesses of online learning?”

During the second interview, details of the experience, a TAO was conducted wherein participants shared their computer screen with the researcher and demonstrated their AOCD participation. They were first asked to open a specified lesson in a specified course and to verbalize their internal dialogue while responding to prompts such as: “Walk me through how you approach a lesson in general.” “Let’s look at the discussion for this week – How do you post, read, and respond?” “How did you interact with your instructor?” “How do you feel in the LMS interface?” They were also asked to demonstrate specifically how they would read and write their responses, including opening Word if that is how they initially drafted their AOCD postings.

Examples of the types of questions asked in the third interview, reflections on the meaning, included: “In an ideal situation, what would your course discussions be like?” “How have your experiences in the program compared to this ideal?” “How do you think instructors define discussion quality?” “If you were an instructor in the program, would you require discussions?” “What is the most important thing you want instructors to know about your experiences with online course discussions?”

Student interview data were contextualized with individual interviews with four of the program instructors (see Table 2), two online course design analyses, and participants’ AOCD forum postings. All data were collected for this study prior to the global COVID pandemic.

Table 2

Instructor Interviews

*pseudonyms

Data Analysis

All the interviews, both student and instructor, were fully transcribed and qualitatively coded using holistic and in vivo coding (Saldaña, 2013). The Community of Inquiry framework (Garrison et al., 2010) guided the design analysis of two of the courses. The forum postings provided additional context to the participants’ descriptions of their experiences. Researcher analytic memos were written throughout the process to identify patterns, connections, and points of divergence in the participants’ experiences (Saldaña, 2013). Reflexive thematic analysis (Braun & Clarke, 2006, 2021) was used to analyze the whole data set and the six steps of the data analysis process were: (1) familiarization with the data; (2) generation of initial codes; (3) identification of preliminary themes; (4) review of preliminary themes against the data and each other; (5) refining and naming of themes; (6) production of the final report. Themes that best capture learners’ experiences, both the how and the why, of their AOCDs considered holistically within an LXD framework are presented below.

Findings and Discussion

The MPS program in which the participants were enrolled focused on workplace leadership with an emphasis on practical skills attainment, where what students learned in their coursework could be readily applied to their work environment. It was marketed as relevant to those already working in the field of study as well as those new to it. Robert, the program lead faculty member, who is also a leader in the broader international field, explained that,

there is value in studying and doing at the same time. In other words…we like the idea
that you could take what you learned in class, apply it on the job and then come back
with good questions based on that experience.

While students did not go through the program as part of a strict cohort, there was a recommended sequence of courses and at the time of the interviews, all but one of the participants had already taken, or were taking, the first two suggested courses. The first course, referred to here as Foundations, was an overview of the field, described by Robert, the instructor for that course, as “miles wide and inches deep,” with an emphasis on concepts, skills, and approaches. The second course, referred to here as Consultations, was a deeper exploration of a particular relationship-based methodology for working with clients and described by Bethany, the instructor for that course, as a “next generation” approach in the field. The Foundations and Consultations courses had a consistent visual look and feel and were structured in consistent ways in terms of weekly lessons, inclusion of required AOCDs, similar types of activities and assignments, and the use of video and textual materials.

A key difference between the courses was that in the Foundations course, all the discussions had a predefined instructor prompt to which the students were required to respond. In contrast, the discussion prompts in the majority of AOCDs in the Consultations course were student-generated and these discussions themselves were student-facilitated. The course instructors also had very distinct practices and views related to AOCDs. While they were required in the Foundations course that he taught and throughout the MPS program that he led, Robert did not regard the AOCDs as very useful for interactions. He himself did not read all the posts or actively participate in the discussions, he felt that overall, they were “just not very good,” and believed them to be of low quality: “the students, even when they're given what I consider a pretty tough question, often they give a minimalist answer.” When asked for his perspectives on why these interactions were poor, he discussed concerns about having things in writing and how that potential risk factor might limit people’s willingness to share authentically.

Bethany, in contrast, in the Consultations course she taught described being very diligent about reading the AOCD posts daily and stated that she was “committed to the idea that in an online format, the instructor needs to be present, and if they're not, then the student-- it's a crap shoot whether the student gets the concepts or not.” While she did not participate directly in the student facilitated discussions, she gave individual feedback in her discussion grading comments and broader discussion feedback in her course emails and announcements. For Bethany, the pedagogical purposes of the discussion were the “confirmation of learning, clarification of learning, and three, would be rapport-building.”

Student Participant Cases

All the student participants except for one worked full-time and were somewhat geographically distributed. While one worked full time in Europe, another part time in Africa, and two in different states, the remaining half lived and worked in the same state (see Table 1). All described a primary reason for participating in the program as the potential for career advancement. The following participant case overviews have been intentionally ordered from the most (Carmen) to least (Adele) positive in terms of their AOCD experiences.

Carmen

Carmen was in the program to facilitate a career shift, explaining that she did not want to be a “staff assistant like my mother” for the rest of her life. She saw her professional experiences as very limited compared to that of her classmates and when describing the benefits of the AOCDs said this:

I'm amazed at what we've learned from our classmate in China and how things in China work as compared to our practices here. I think you're really getting-- even across the country, things are different. So, I think the discussion is good, because it's giving everybody's viewpoints, all these different experiences that they have, and it opens our eyes to somebody like me who's barely been out of central and northeastern Pennsylvania to see how the world is really working.

She was by far the most enthusiastic about her AOCDs and appreciated her experiences getting to interact with her classmates through them: “I have fun when I'm online, 2:00 in the morning, sometimes on the discussion board and some of my classmates are up then too.” Notably, when asked explicitly about the quality of dialogue taking place in the AOCDs, Carmen did admit that except for her experiences in the Consultations course, she “wouldn't say there was a tremendous amount of critical thinking. The discussion boards were always sort of going through the motions.” Carmen was the only participant who confidently stated that she read every single AOCD post.

Russ

Russ previously served in the military and described how the program immediately connected to his life in that he applied the skills gained from his courses to the fraternity he advised. Russ, like Carmen, described his AOCD experiences in largely positive terms and regarded himself as being a part of a learning community facilitated by the AOCDs. His biggest critique of the program pertained to instructors he viewed as insufficiently engaged. Russ approached and experienced his discussions through the lens that he described the entirety of his academic career: that of relational connections. More than once he talked about the AOCD experience with his peers, who he also referred to as his “teammates,” by saying “we’re all in this together.” Russ remarked that “I think discussion forums are great. I think it gives us an opportunity to be prompted, to really think, and it gives us a stage to think at a higher level.” In spite of the fact that he spoke positively about his AOCDs, his actual participation was minimal compared to his classmates.

Abby

Abby was doing the program in hopes of career advancement but also felt that even without that possible promotion, “​​it's stuff that I could use to make where I work better.” She was largely positive about the AOCDs and explained that “to put into words, things that you're thinking, re-enforces the concepts that you're learning, and it kind of makes you feel like, ‘I get it,’ that you can have a coherent thought on whatever the topic is.” In spite of the positives she experienced with the AOCDs, she also commented on the fact that “most of them are kind of low quality” and “lots of times people are just pretty much like ‘Oh that’s a great post, that’s a great thought.’”

Susan

Like some of the other participants, Susan appreciated connecting her course learning to her professional world: “I could take what [instructor] gave me and immediately apply it to my job, which was huge.” Susan had a mixed analysis of her AOCD experiences and was the most outspoken in contrasting her AOCDs in the Foundations course with those in the Consultations course. She spoke very positively about the student-facilitated AOCDs in the Consultations course as well as the instructor Bethany. When commenting on critical thinking across her AOCDs she said that while it was low in Foundations, in the Consultations course she “absolutely was using critical thinking skills and what have you to answer the questions and participate in a dialogue.” That said, when ranking her class activities in terms of impact on her learning, she put the AOCDs as the least impactful of all of the activities across the courses she’d taken. Consistent with this, she felt that the discussions were potentially “very valuable, but the current state is not a value add.”

Thomas

Thomas was pursuing this degree as he contemplated “what’s next” in his career. Like Susan, he described his discussion experiences in a mix of positive and negative terms. He explained that the quality of discussions really varied by the course and the types of students. He explained that in some courses they could be great while in others “a waste of time.” He elaborated saying that “When you get a whole bunch of people that don't want to be involved with the discussion, the discussion board basically goes flat. It's almost lifeless, you know what I mean?” The consistent issue Thomas raised related to instructors who he experienced as taking a hands-off approach in both the course overall and the AOCDs specifically. As he saw it, “You can't just put it in cruise control…you got to steer them [the discussions].”

Mary

Mary enrolled in the program out of a desire to take her career “to the next level.” Overall, she did not evaluate her online discussion experiences very positively in terms of her learning, though she did see discussions in face-to-face classes as a definite “value add.” She described the requirement to participate as a form of “busy work” that would not be a factor if she were in a “bricks and mortar program.” Regarding what she hoped her instructors would understand about the AOCDs, she explained that “From my perspective, I understand what they're trying to affect with the discussions, but I don't think it's having the effect that they want it to have.”

John

John’s motivation for the program was in part to gain more practical knowledge for his job and also because the country in which he worked part-time had a high regard for higher education degrees. John was quite critical not only of his AOCD experiences but ultimately of the overall expectations for dialogue and critical thinking throughout the program. John also recognized, as did others, that the AOCDs could be mechanistic in that “you can copy and paste from the outline, the summary, and more or less copy and paste that in a Word document, and reformat a little bit and nobody's going to mark you off for not submitting that discussion.” Despite his critiques, John also suggested that “the discussion forum has the potential of being the most important part of the course and I would just say that it's not reaching its full potential.” John believed the discussions were so common in the courses because the instructors felt compelled by protocol to include them.

Adele

Adele explained that part of her motivation for the program was the monetary increase she would get on completing the degree but that she was also inherently interested in the curriculum as she was “curious about how things work and how they don't work and why they don't work.” Adele was the only participant who had nothing positive to say about her AOCD experiences and remarked that she “was and continue[s] to be underwhelmed by this method of interaction.” Notably, at the time of her interviews, she was enrolled in an elective course from a different program that did not require discussions and remarked in regard to the absence of discussions that “I have to say, Andrea, I do not miss it. Oh my God. I so don’t miss it.” Importantly Adele also craved meaningful connections and dialogue with her classmates that arguably, in large part, should have been taking place in the AOCDs.

Experiential Themes

While there was some range in valence in the participant cases presented above in terms of their AOCD experiences, overall, they were generally weighted more towards the negative. What follows here, then, are salient themes that showed up across participants and provide more depth as to the how and why of these AOCD experiences. The first theme addresses the key ways in which participants felt that their required AOCD activities were not actually authentic engagements with their peers.

Not "Real" Discussions

Rather than being experienced as an interaction taking place with others in their course, participants often positioned their AOCDs as individual assignments to complete and move on from. Several participants noted that they did not approach the discussions as being “real.” For example, Susan stated, “the way that I typically approach the discussion is I'm checking off the box to make sure that I've satisfied the criteria for the week.” Adele agreed with this, stating, “this feels like ticking the box to me.” Similarly, Abby stated that “I think a lot of times the discussion experiences are just going through the motions,” or as Mary stated, it “seems to me a little bit like busywork.” Even Carmen, the most positive among the participants, explained that except for the Consultations course, her other AOCDs were “sort of going through the motions.” There was also explicit discussion about how their AOCDs were not real discussions. Adele noted that “a lot of times it's not a dialogue.” Susan concurred, “it's really not a discussion quite honestly.” Mary also stated that “because it’s not like you’re chatting with each other… there's not really much of a synergy there in my experience.”

Abby offered more detail as to how students were generally writing their replies to each other.

I think people have a standard response that they write to postings and the words are different but the general sentiment of the posting is the same, like, “Oh, great post. I've had this same experience in my blah blah blah.” And, “Oh, I do that all the time. Thanks for sharing.” It's just like this standard: change a few words in a post. It does signify, “Yes, I read your post and yes, I can relate that to an experience that I've had,” but there's nothing meaningful.

This tendency was recognized by at least one instructor. Jessie stated that “probably 20% [of students] see it as busy work and kind of checking the box, and it's just something that I need to do, so I do it.” Based on the experiences of the participants in this study, Jessie’s 20% is arguably a low estimation.

Adele explains her sense of isolation in the AOCDs in terms of not “knowing” her classmates:

[I]t's so bad that I don't even know who they are, but I know that I'm dismissing it. That's pathetic to me. That just doesn't feel very human to me...half the time I don't remember the names of these people that are in my class. I don't know what their names are. I have no idea what their life looks like. Honestly, I don't really care. And if I think about that, it makes me a little bit sad, because that's not really who I am. I'd rather have a real relationship with someone.

Thomas, on the other hand, positioned himself as the unknown in the AOCDs, in contrast to his Facebook experiences.

They [audience on Facebook] are very familiar with me. In a college setting, that's not necessarily the case. If I have any discussion with you on the discussion board, you don't know me from Adam. Maybe you've seen my name on a couple posts. You don't know what I'm like. You don't know what I think is funny. You don't know what I think. You don't know when I'm serious.

Both Thomas and Adele speak to a lack of social presence, where social presence is in part defined as students’ ability to experience themselves, and by extension their classmates, as “real people” in the virtual environment (Garrison et al., 2000).

Deterministic Participation Requirements

In their positioning of their discussions as a deliverable, it is clear participants’ experiences were shaped by the posting and response requirements wherein they typically had to post at least once and reply at least twice. As observed in the TAOs, all participants had an intentional way of authoring their initial post, with essentially two distinct approaches among the participants. Each approach suggested some of the learners’ implicit views about the discussion’s purpose.

The majority approach was to draft the post without looking at any others, which suggested that they saw part of the purpose as the articulation of independent thoughts. The second approach, described by three of the participants, was to rely heavily on the posts of others to determine how to craft their own. While on the surface the second approach might seem a more passive approach, it aligns with the collaborative knowledge construction framework (Gunawardena et al., 1997) and is more akin to the experience of a face-to-face discussion, wherein before speaking a participant hears what others say and then ideally builds on it.

An important element of the participation requirement was that their minimum two replies had to be made to students who had not already received at least two replies. For some this completely determined how they read, and more importantly did not read, the discussion posts of others. Mary and Russ both described this common practice (see Figure 1 for the practices they describe below). Mary stated,

I know that I also have a responsibility to respond to two of my fellow teammates. What I do is I come over here to the replies, and would typically look to see who had not been replied to. If this person had four replies, I'd skip right over that and I'd probably go the ones that have one and/or zero, and I then I would try to reply to that.

Russ stated,

In this case, you look to see how many posts there are. In this case, [student name] had two people respond to her already. It's clear that [instructor name] says if there were more than one-- if two posts had been made, you should try to find somebody who hasn't had any post or are less than two. So in this case, [student name] doesn't have two responses so I'd open up hers, with the thought is, what her thoughts are, click on reply.

Figure 1

Intersection of Assignment Requirements and AOCD UX

John recognized that this habit ultimately stifled any authentic exchange. He stated,

If for example, somebody satisfies their criteria in this course that means they've made their post and they've commented on two other colleagues, you won't expect to hear from them because they've moved on and if you comment and ask them questions or whatever, you can't expect them to actually come back because they don't have to.

Notably, not all learners participated this way. For instance, Carmen and Abby described in detail checking the posts every day and often having the LMS open to see if anyone had responded to them after initially posting. At the same time, though, just like with the other participants, Abby and Carmen did mention multiple times the required nature of the participation and how it shaped their practices. Abby stated, “It's like when you're forced to have conversation and you don't really have input into it...it's like it's so forced.”

There is not yet a settled answer to how AOCDs should be structured in terms of requirements for posting and replying. Many who have taught and/or researched this topic recognize that there are challenges to either not requiring participation—in that it can lead to AOCDs with minimal contribution—or overly prescribing participation—which can lead to AOCDs with learners participating in rote, predictable ways as the above demonstrates.

Instructor Presence (Mostly) Lacking

The area in which participants were most outspoken and consistent was the importance of the instructor’s active participation with their AOCDs. Both Susan and Thomas contacted me through email over a month after the interviews were over to share additional feedback about how important the instructor’s role was to their experience. Thomas also spoke at length on this during the interviews. His sentiments encapsulated how most participants discussed it.

The instructor involvement is huge. That is by far, in my opinion, the number one determining factor of how well the discussion boards go, the instructor involvement. . . . If you have an instructor that's involved and they're telling you how much they appreciate your discussion and they talk a little bit about it, more people are going to be involved. When you have an instructor that doesn't pay any attention to the discussion board, simply gives you your ten points or whatever it is for discussing that week, well, then that tells you right from jump street he could care less whether I post five paragraphs or two sentences. That's number one in my opinion for a successful discussion board.

The model participants consistently offered as the ideal for both AOCD structure and instructor participation was the Consultations course. As already described, the discussion prompts for and facilitation of most of the AOCDs in the Consultations course, unlike the others, were done by the students. In describing why the Consultations course was typically a better AOCD experience, a repeated reason was the role of Bethany, the instructor. Carmen stated, “I would say we have the most in-depth dialogue [in the Consultations course]. We have the best discussion boards. We get a long running conversation going about topics. We have engaged faculty, so that helps us.” Susan goes further in making explicit how her level of effort aligned with her perception of Bethany’s care. She stated,

I think I put more effort into it when the instructor was more visible. Because as long as I satisfied it in [course instructor’s for Foundations course] world, I don't think it would have made a difference. So I think it was just the mere fact that I answered the question. If [the Consultations course instructor] was more visible then I should really be thoughtful how I'm managing this. . . .Because she cared. If she was taking the time to read it, she really cared.

It is important to note here that instructor facilitation does not have to take place in the discussion forum itself to be impactful. In general, Bethany did not post anything in the AOCDs as she wanted to preserve that space for student dialogue. Instead, she gave feedback to students on their AOCD contributions in her individual grading and commented more broadly on the discussion in course announcements and emails to the entire class. The “where” of the instructor involvement was ultimately not what mattered to students. As an example, Susan had misremembered that Bethany was highly active in the AOCDs themselves. But when she went to show examples of Bethany's participation during the TAO interview, she realized that she was mistaken and that in actuality Bethany had not participated at all in the AOCDs but instead in her grading. This is good news for faculty committed to preserving the AOCD space for student dialogue but also demonstrates that instructor involvement very much impacts not only how students experience their AOCDs but also how much effort they put into them.

Notably, the Consultations course seemed to be the exception that demonstrated the larger rule of a lack of AOCD instructor involvement. “I feel like I'm writing for my classmates more than the instructor. I honestly, I don't even know if some of the instructors read these” (Abby), “I had one instructor that was absolutely absentee . . . But in this forum, you really need more” (Russ), and “I guess my take on it is if they don't respond to the post, then the post-- I mean, there are certainly learning opportunities, but they are just shy of busy work” (Mary). Many of the participants wanted more engagement with their instructors, and when asked what the one thing they would want instructors to know about their course discussion experiences, the responses again confirmed the need for more instructor involvement: “I think that the most important thing should be that during discussion has to in some capacity include the professor” (John) and “Involvement; they have to be involved in this process” (Thomas).

This lack of instructor involvement in AOCDs was confirmed by some of the faculty. Consider Barry’s description: “I do try to read some of the posts, but it's a little overwhelming, because with 20 students in the class and everyone required to post, and not all the posts are that interesting” or Robert’s explanation that he found “the discussions and blogs in the course to be mind-numbingly boring and unhelpful. I am not sure what to do to fix them.”

While the importance of instructor involvement was one of the strongest points made by the participants, it did not fully explain their experiences of “checking the box” and “going through the motions” in their AOCDs. For instance, even those who acknowledged that Bethany’s involvement in the Consultations course was far superior to that of their other instructors still ranked their AOCDs as having the lowest impact on their learning when compared with other course elements.

AOCD UX Limitations

As part of the second interview, participants were asked to open an AOCD from a course and describe their thoughts and feelings while looking at it. Russ described its lack of aesthetic appeal quite elaborately in comparison to the “most generic post-it note yellow, ugly one that you can find.” He went on to add,

The discussion forum is black and white to-- it's a communication apparatus. It's very non-fuzzy. There's nothing warm about it. You press this button you go to here, you type in whatever, press this button, you're done. There is no room for your picture, there is no room for colors to make it warm and fuzzy and appealing. It's a wall that you're sticking something to for everybody to see, and then people all go to that wall and see it, and just stick other things on top of it until you got-- it's almost like post-it notes. The most generic post-it note yellow, ugly one you can find, and then you got a string of them that are all stuck together. That's what it's sort of like. I'm a visual person.

Others described its frustrating interface in terms of seamless navigation. Abby stated,

On a discussion board, when people post on a discussion board in your class, you have to click through like a million times to get to that discussion board, and then you have to click through on that class, then you have to click on what's new, then you have to wait for something else to load, then you have to click a little plus sign and drop it down, then you have to click the specific forum. It's just a lot of clicks. Whereas on a Facebook news-feed it just comes up, or basically just comes up in your news-feed and it's maybe one click to see what people have said about something.

Adele also noted,

So the one thing you have to be aware of is to find your way back; find your way back to the original question and make sure you're commenting on what you want to comment on rather than--things get to be red herrings. Not red herrings exactly, but you may end up commenting on something and it'll get lost. It'll just be mired.

This sentiment was echoed by the instructor Robert as well.

Well, I have to go and click in [the LMS], click down through the threaded discussions, do all that kind of stuff. It's very tedious and it's a pain. That is a pain compared to in a classroom where they have the conversations.

The UX will naturally impact how one experiences not just the tool itself, but also the learning activity taking place in the tool. While one might rightly inquire as to whether these experiences were unique to the specific LMS used in this program, Susan, Mary, and Carmen all had previous AOCD experiences at other universities that took place within different LMSs. When speaking about their experiences, they tended to group them together, regardless of LMS. This parallels prior research considering students’ experiences writing in open spaces where it was purported that the experience of writing online in an asynchronous threaded environment transcends the specifics of the LMS (Adams & van Manen, 2017) Similarly, the discussion forum has been described as “a ubiquitous component of every learning management system and online learning platform from Blackboard to Moodle to Coursera” and marked by “one relatively standardized interface” (Morris & Stommel, 2018). One might surmise that some of these experiences are shaped by the UX of the AOCDs housed within industry standard LMSs. As with the previous themes related to the impact of instructor participation and assignment specifications, the UX alone is not sufficient to explain the complexity of learners’ experiences with their AOCD.

Yet, AOCDs Still Important

The final theme to come out of this analysis is also the most surprising. Despite their strong feelings on the challenges and limitations with AOCDs, when asked directly as to whether they should be kept in the program, all but one of the participants felt they should not be eliminated. In terms of reasons given for why they shouldn’t be eliminated. John described that

I think that discussion is the only opportunity you'll ever get to actually say something… It's a vehicle that creates an opportunity for that to take place, and if I took the time to present my weekly discussion comment, I learn from that.

For others like Thomas, “it is the only activity really that you share anything with the other classmates” or like Susan, “So at least, the discussions give you a glimpse of maybe how a person thinks. But if you took that away, you're missing the diversity piece of the whole class learning from one another.” Carmen’s explanation was more nuanced and addressed the need for them to be “done correctly.”

Because I feel, if they're done correctly and maintained by the faculty, interjecting when they see it stagnating. If it's done correctly, I think they can be a big learning tool, because we're learning so much about, not just the topics, but about each other, about how the different businesses work, how these ideas translate to other people's thinking, other cultures. So, I definitely think they're of value if they're done the right way.

When asked to further reflect on how they should be structured, John and Mary felt they should be re-designed so that the emphasis was not on interactions with one’s classmates but instead with the course instructor. Many of the participants, however, would not change much about the format of the discussions, including their value in the final grade, their typical format with an instructor-provided prompt, or the requirement that each student respond to two comments that did not already have two responses. Whether this was simply because they couldn’t envision a better way to structure the AOCDs to be more effective or because participants believed the discussions were already ideally designed at present is difficult to say. This was an especially interesting finding given the sense of drudgery the participants often expressed regarding their AOCD experiences.

Implications

The findings of this study considered in the context of both the AOCD and the LXD literature to date suggest a potential new model for how we conceptualize and analyze AOCDs within an LXD framework. There are five key factors that intersect to ultimately shape learners’ experiences of their AOCDs: academic culture, course design, AOCD UX, student traits, and time-bound dynamics (see Figure 2). With slight modification, this model can also be applied to the exploration of other technology mediated educational experiences.

Figure 2

LXD Model for Design and Research

LXD Model: Key Elements

In the following section, I will briefly describe the key elements of the model while also discussing it in the context of the specific findings from this study.

Academic Culture

Academic culture refers to elements that transcend but necessarily impact individual course design. For example, discussion norms will generally vary based on things like discipline (e.g., philosophy, mechanical engineering), academic level (e.g., first year, PhD level), and even type of school (e.g., small liberal arts emphasis, large state school). Considering academic culture in the context of analyzing the LXD of AOCDs also means accounting for things like similarities and differences across courses in the same program. For example, if discussions are not commonplace in a particular major, a course that includes them will be experienced differently than a course with AOCDs that is part of a major where AOCDs are ever-present. In the context of this study, the program itself was an MPS with a pragmatic emphasis that valued experience-sharing in the AOCDs. Additionally, the students were working adults, most of them in the same or a related field.

Course Design

Course design refers to the ways in which the course itself situates AOCDs. This includes the presence or lack of an explanation of the pedagogical purpose of the AOCDs. It also considers how the AOCDs are evaluated, how much they are worth, and other elements like discussion prompts, participation requirements, and role assignments. In the context of this study, the course design was largely consistent across the courses, using the same LMS, lesson structure, overall look-and-feel, language for different lesson pages (e.g., roadmap), and the inclusion of required AOCDs worth 10-20% of the final grade. An important difference was between the Foundations course and the Consultations course in the structure of the AOCDs. The prompts in the Foundations course were provided and students were to respond directly to them whereas in the Consultations course, the students rotated a leadership role in creating the prompt and facilitating the AOCD for that lesson.

Student Traits

Student traits are all the unique elements that individuals bring with them to their courses. These characteristics necessarily impact their experiences of AOCDs and include, among other things, areas like motivation, technology self-efficacy, socialization needs, writing abilities and style, beliefs about the purpose of course discussions, and cultural background regarding communication norms. In the case of this study, there were some key student traits participants had in common as they were all working adults interested in career advancement. There were also important differences pertaining to things like how they understood the purpose of the discussions, professional backgrounds, and approaches to their course requirements. As an example, John viewed the AOCDs as a site for productive disagreement where in contrast, due to her limited professional experiences, Carmen appreciated learning from the experience-sharing of others. Some of the participants diligently read everything that was assigned (Adele) while others only read if they found they couldn’t complete the activities without reading (Susan).

AOCD UX

AOCD UX refers to all the elements that mediate the experience of participating in the discussion and requires accounting for the affordances and constraints of the mediating educational technology. Research has suggested that assigning discussions in spaces like Facebook (Hou et al., 2015) or social annotation tools (Y. Sun & Gao, 2017) lead to different outcomes than the standard LMS AOCD. Examples of features and functionality of the AOCD interface that can ultimately influence learners’ AOCD experiences include aesthetics/look-and-feel, how messages are threaded, how messages are displayed, the use of images, and how the interface displays elements like read/unread and number of responses. In the case of this study, the way that the AOCD provided information about how many replies a message had impacted student reading behavior (see Figure 1). Participants' experiences of unnecessary clicking contributed to a sense that the discussion wasn’t seamless.

Time-Bound Dynamics

Lastly, time-bound dynamics recognizes that there are elements of the course that will be unique each time the class runs. For instance, there will always be a new combination of students, unique instructor styles, and external events that can influence the context in which students participate in their AOCDs. As anyone who has ever taught the same course more than once can attest, no matter how much they hold constant in terms of their teaching style and course design elements, there will be always be some unique elements—whether a group of students that seems more engaged/disengaged, global external events, or more isolated internal events like a software upgrade—making the course different than the last time it was taught. The communication styles and patterns of the individual students and the instructor for that class will impact the discussion. In the case of this study, the different styles of Bethany and Robert as instructors in terms of how they facilitated, or not, the AOCDs clearly impacted students’ experiences. If the students had taken the courses at a different point in time and they were being taught by different instructors, that too would have an impact.

Intersecting Elements

While the above five factors have been discussed distinctly, they necessarily come together to shape the learners AOCD experiences. For instance, the course design and the AOCD UX worked in combination to ultimately determine many of the participants’ viewing, reading, and replying patterns (see Figure 1). Student traits interacted with academic culture to influence their overall attitude toward the discussions: John was frustrated while Carmen appreciated what she was learning. Also, students were required to post once, reply twice and because the overall emphasis was on experience-sharing rather than critical dialogue, students were not encouraged to go beyond the minimum expectations. Academic culture and course design interacted in that while the discussions were an important element of students’ overall performance and were integrated throughout the online courses in the program, in a culture that respects faculty independence, individual faculty determined their own level of participation and in all but the Consultations course, it was experienced by the students as low to “nonexistent.”

Using the LXD Model to Improve AOCDs

In using this LXD model to improve AOCDs, it is necessary to think through each of the model’s elements while recognizing those which can be realistically influenced and those which instead need to be designed for. Whether or not one can influence a particular element of the model will, like many things, be largely dependent on one’s position within the college or university. As an example, when it comes to the AOCD UX, some instructors will be required to use a particular interface, such as the one in the LMS, where others will have more flexibility in terms of which discussion platform they choose to implement. An educational technology division of a college or university might have a great impact on the AOCD UX if they determine which platforms the school adopts but have no impact on course design. This is why it is so important that technologists get input from faculty, learning designers, and students as to the ways in which the UX of different AOCD interfaces affords and constrains natural discussion flow.

As another example, the program lead of an online academic program would typically have more ability to impact academic culture as it pertains to discussion norms across courses than would an individual instructor hired to teach a single course section. Given the importance of instructor AOCD involvement, program expectations for teaching a course with discussions should emphasize that instructors actively facilitate course discussions. For instructors concerned that their involvement might overly influence the direction of the discussion, recall that Bethany in the Consultations course was regarded as being quite involved without posting in the discussion space itself. Instead, she provided individualized feedback to students demonstrating her participation while helping guide the discussions productively.

Student traits are often mentioned as an area instructors have little to no influence over; yet, while one may not be able to influence student traits, one can often design for them. When it comes to discussions, this study echoes the findings of Rourke and Kanuka (2007) in that students often view the purpose of the discussions quite differently. This can lead to a situation where one student regards himself as “provocative,” while others view him as problematically “negative” or “difficult” if they, in contrast, think of the discussion as a social space. That students will view the purpose of discussions differently is something to anticipate and the course design should be clear as to expectations for how students should be communicating in the AOCD space. Is the purpose of the AOCD to challenge each other’s ideas using an argumentative model? Is the purpose to share experiences, reading about and learning from each other? Is the purpose something else entirely? Making these communication norms explicit, with examples, is important as is instructor feedback regarding discussion communication expectations.

As a final example, when considering academic culture and course design as it relates to discussions, it is important to reflect on the pedagogical purpose of the discussions. It is likely AOCDs are at times implemented when there is a more appropriate tool. The nature of the AOCD UX combined with course design elements, such as the requirement that one replies only to a post that doesn’t already have two replies (see Figure 1), can put pressure on students to arbitrarily focus on particular posts over others. If the goal is for students to have a place to share their experiences with others in the class, AOCDs without required replies, blog posts, or even individual assignments that allow group viewing would all achieve that outcome without unnaturally constraining reading and posting behaviors. Similarly, if the primary goal is individual idea articulation, an individual reflection assignment would be more appropriate than an AOCD.

Other Applications for this Model

While the model discussed here has been presented in the context of AOCDs, with a few modifications it can be applied to considering learners’ experiences with other technology mediated educational activities. Of the five key factors of academic culture, course design, AOCD UX, student traits, and time-bound dynamics, the only one that would need to be changed would be the AOCD UX. Instead, that component would be the UX of whatever was the mediating technology.

Conclusion

This study provides valuable insights into students' experiences with AOCDs in an online graduate program. The findings highlight key challenges and limitations of AOCDs, including the prescriptive nature of the requirements, limited instructor involvement, and problematic UX design. At the same time, it demonstrates that both students and faculty continue to value online course discussions for idea articulation and peer learning. The study was conducted in the context of AOCDs as a persistent activity in online higher education and a gap in the extensive AOCD literature base regarding in-depth qualitative studies. The research inquiry was an investigation into students’ AOCD experiences from their own perspectives considered within a broader LXD lens. The study site was a fully online asynchronous graduate program and data included 26+ hours of interviews, including TAOs, with eight student participants, four instructor interviews, two course design reviews, and a review of participants’ AOCD posts. All the data were analyzed through a reflexive thematic approach to identify salient themes. This study found that learners tended to experience their discussions as a lower quality deliverable where participation was required. There were individual variations among participants, however, with some clearly more positive than others. Their experiences were shaped by the specific course requirements and limitations of the AOCD interface. Learners were also highly aware of the participation of their instructors and felt that more often than not, their instructors were absent from the discussion, which also impacted the quality of the AOCDs. All that said, the majority of participants would not suggest removing the AOCD as an activity and described valuing the opportunities they provided for social connections, idea articulation, and collaborative group processes.

This paper also proposed a new model for considering learners’ experiences with AOCDs through an LXD lens based on a synthesis of the findings and both the AOCD and LXD literature. The model is made up of the five components of academic culture, course design, AOCD UX, student traits, and time-bound dynamics. With slight modification, this proposed new model can be extended to apply to student experiences with other technology mediated educational activities. While distinct, the components of academic culture, course design, AOCD UX, student traits, and time-bound dynamics also inevitably intersect in interesting ways to shape students’ experiences. Future research should implement and evaluate this new model in the context of both AOCDs and other educational technology mediated environments.

References

Adams, C., & van Manen, M. A. (2017). Teaching phenomenological research and writing. Qualitative Health Research, 27(6), 780–791. https://doi.org/10.1177/1049732317698960

Al-Husban, N. A. (2020). Critical thinking skills in asynchronous discussion forums: A case study. International Journal of Technology in Education, 3(2), 82. https://doi.org/10.46328/ijte.v3i2.22

Arbaugh, J. B., Cleveland-Innes, M., Diaz, S. R., Garrison, D. R., Ice, P., Richardson, J. C., & Swan, K. P. (2008). Developing a community of inquiry instrument: Testing a measure of the Community of Inquiry framework using a multi-institutional sample. The Internet and Higher Education, 11(3-4), 133-136.
https://doi.org/10.1016/j.iheduc.2008.06.003

Bai, H. (2012). Students’ use of self-regulatory tool and critical inquiry in online discussions. Journal of Interactive Learning Research, 23(3), 209–225. https://www.learntechlib.org/primary/p/37588/

Bowen, K., Forssell, K. S., & Rosier, S. (2020). Theories of change in learning experience (LX) design. In M. Schmidt, A. A. Tawfik, I. Jahnke, & Y. Earnshaw (Eds.), Learner and user experience research. An introduction for the field of learning design & technology. EdTech Books. https://edtechbooks.org/ux/lx_theories_of_change

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77–101. https://doi.org/10.1191/1478088706qp063oa

Braun, V., & Clarke, V. (2021). One size fits all? What counts as quality practice in (reflexive) thematic analysis?Qualitative Research in Psychology, 18(3), 328–352. https://doi.org/10.1080/14780887.2020.176923

Brinkmann, S., & Kvale, S. (2015). InterViews: Learning the craft of qualitative research interviewing (3rd ed.). SAGE Publications Inc.

Chai, H., Liu, Z., Hu, T., & Li, Q. (2020). A new conceptual framework for measuring online listening in asynchronous discussion forums. Artificial Intelligence Supported Educational Technologies, 59–73.
https://doi.org/10.1007/978-3-030-41099-5_4

Creswell, J. W., & Poth, C. N. (2016). Qualitative inquiry and research design: Choosing among five approaches (4th ed.). SAGE.

da Silva, L. F. C., Barbosa, M. W., & Gomes, R. R. (2019). Measuring participation in distance education online discussion forums using social network analysis. Journal of the Association for Information Science and Technology, 70(2), 140–150. https://doi.org/10.1002/asi.24080

de Wever, B., Schellens, T., Valcke, M., & van Keer, H. (2006). Content analysis schemes to analyze transcripts of online asynchronous discussion groups: A review. Computers and Education, 46(1), 6–28.
https://doi.org/10.1016/j.compedu.2005.04.005

Dewey, J. (1938). Experience and education. Macmillan.

Ebrahimi, A., Faghih, E., & Marandi, S. S. (2016). Factors affecting pre-service teachers’ participation in asynchronous discussion: The case of Iran. Australasian Journal of Educational Technology, 32(2), 115–129. https://doi.org/10.14742/ajet.2712

Englander, M. (2012). The interview: Data collection in descriptive phenomenological human scientific research. Journal of Phenomenological Psychology, 43(1), 13–35. https://doi.org/10.1163/156916212X632943

Ertmer, P. A., Sadaf, A., & Ertmer, D. J. (2011). Student-content interactions in online courses: The role of question
prompts in facilitating higher-level engagement with course content. Journal of Computing in Higher Education, 23(2–3), 157–186. https://doi.org/10.1007/s12528-011-9047-6

Fehrman, S., & Watson, S. L. (2021). A systematic review of asynchronous online discussions in online higher
education. American Journal of Distance Education, 35(3), 200–213.
https://doi.org/10.1080/08923647.2020.1858705

Galikyan, I., & Admiraal, W. (2019). Students’ engagement in asynchronous online discussion: The relationship
between cognitive presence, learner prominence, and academic performance. The Internet and Higher
Education, 43, 100692. https://doi.org/10.1016/j.iheduc.2019.100692

Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical thinking in a text-based environment: Computer
conferencing in higher education. Internet and Higher Education,2(2–3), 87–105. https://doi.org/10.1016/S1096-7516(00)00016-6

Garrison, D. R., Anderson, T., & Archer, W. (2010). The first decade of the community of inquiry framework: A retrospective. Internet and Higher Education, 13(1–2), 5–9. https://doi.org/10.1016/j.iheduc.2009.10.003

Giacumo, L. A., & Savenye, W. (2020). Asynchronous discussion forum design to support cognition. Effects of
rubrics and instructor prompts on learner’s critical thinking, achievement, and satisfaction. Educational Technology Research and Development, 68(1), 37–66. https://doi.org/10.1007/s11423-019-09664-5

Gregg, A., Reid, R., Aldemir, T., Gray, J., Frederick, M., & Garbrick, A. (2020). Think-aloud observations to
improve online course design: A case example and “how-to” guide. In M. Schmidt, A. Tawfik, I. Jahnke,
& Y. Earnshaw (Eds.), Learner and user experience research: An introduction for the field of learning design & technology. EdTech Books. https://edtechbooks.org/ux/15_think_aloud_obser

Gunawardena, C. N., Lowe, C. A., & Anderson, T. (1997). Analysis of a global online debate and the development
of an interaction analysis model for examining social construction of knowledge in computer conferencing. Journal of Educational Computing Research, 17(4), 397–431.
https://doi.org/10.2190/7MQV-X9UJ-C7Q3-NRAG

Henri, F. (1992). Computer conferencing and content analysis. In A. Kaye (Ed.), Collaborative learning through
computer conferencing: The Najaden papers (pp. 117–136). Springer-Verlag.
https://doi.org/10.1007/978-3-642-77684-7_8

Hewitt, J. (2005). Toward an understanding of how threads die in asynchronous computer conferences. Journal of the Learning Sciences, 14(4), 567–589. https://doi.org/10.1207/s15327809jls1404

Hou, H.-T., Wang, S.-M., Lin, P.-C., & Chang, K.-E. (2015). Exploring the learner’s knowledge construction and
cognitive patterns of different asynchronous platforms: comparison of an online discussion form and Facebook. Innovations in Education and Teaching International, 52(6), 610–620. https://doi.org/10.1080/14703297.2013.847381

Jahnke, I., Schmidt, M., Pham, M., & Singh, K. (2020). Sociotechnical-pedagogical usability for designing and
evaluating learner experience in technology-enhanced environments. In M. Schmidt, A. A. Tawfik, I. Jahnke, & Y. Earnshaw (Eds.), Learner and user experience research: An Introduction for the field of learning design & technology. EdTech Books. https://edtechbooks.org/ux/sociotechnical_pedagogical_usability

Kanuka, H., Rourke, L., & Laflamme, E. (2007). The influence of instructional methods on the quality of online
discussion. British Journal of Educational Technology, 38(2), 260–271.
https://doi.org/10.1111/j.1467-8535.2006.00620.x

Lee, J. (2012). Patterns of interaction and participation in a large online course: Strategies for fostering sustainable discussion. Educational Technology and Society, 15(1), 260–272. https://www.jstor.org/stable/jeductechsoci.15.1.260

Morris, S. M., & Stommel, J. (2018). The discussion forum is dead; long live the discussion forum. In S. M Morris &
J. Stommel (Eds.), An urgency of teachers: The work of critical pedagogy. Hybrid Pedagogy Inc. The book is licensed under the CC BY-NC-ND 4.0 license (https://creativecommons.org/licenses/by-nc-nd/4.0/).

Newman, D. R., Webb, B., & Cochrane, C. (1995). A content analysis method to measure critical thinking in face-to-face and computer supported group learning. Interpersonal Computing and Technology, 3(2), 56–77. https://www.learntechlib.org/p/80700/

Nowell, L. S., Norris, J. M., White, D. E., & Moules, N. J. (2017). Thematic analysis: Striving to meet the
trustworthiness criteria. International Journal of Qualitative Methods, 16, 1–13.
https://doi.org/10.1177/1609406917733847

Parrish, P. E., Wilson, B. G., & Dunlap, J. C. (2011). Learning experience as transaction: A framework for instructional design. Educational Technology, 51(2), 15–22. https://doi.org/10.1080/10643389.2012.728825

Psaromiligkos, Y., Orfanidou, M., Kytagias, C., & Zafiri, E. (2011). Mining log data for the analysis of learners'
behaviour in web-based learning management systems. Operational Research, 11(2), 187–200. https://doi.org/10.1007/s12351-008-0032-4

Rourke, L., & Kanuka, H. (2007). Barriers to online critical discourse. International Journal of Computer-Supported
Collaborative Learning, 2, 105–126. https://doi.org/10.1007/s11412-007-9007-3

Sadaf, A., Kim, S. Y., & Wang, Y. (2021). A comparison of cognitive presence, learning, satisfaction, and academic performance in case-based and non-case-based online discussions. American Journal of Distance Education, 35(3), 214–227. https://doi.org/10.1080/08923647.2021.1888667

Saldaña, J. (2013). The coding manual for qualitative researchers. Teachers College Press.

Schmidt, M., Earnshaw, Y., Tawfik, A. A., & Jahnke, I. (2020). Methods of user centered design and evaluation for
learning designers. In M. Schmidt, A. A. Tawfik, I. Jahnke, & Y. Earnshaw (Eds.), Learner and user
experience research: An introduction for the field of learning design & technology. EdTech Books.
https://edtechbooks.org/ux/ucd_methods_for_lx

Schmidt, M., & Huang, R. (2022). Defining learning experience design: Voices from the field of learning design &
technology. TechTrends, 66(2), 141–158. https://doi.org/10.1007/s11528-021-00656-y

Schmidt, M., Tawfik, A. A., Jahnke, I., Earnshaw, Y., & Huang, R. T. (2020). Introduction to the edited volume. In
M. Schmidt, A. A. Tawfik, I. Jahnke, & Y. Earnshaw (Eds.), Learner and user experience research: An
introduction for the field of learning design & technology. EdTech Books. https://edtechbooks.org/ux/introduction_to_ux_lx_in_lidt

Scott, M., & Turrise, S. L. (2021). Student perspectives: Discussion boards as learning strategies in online
accelerated nursing courses. Journal of Nursing Education, 60(7), 419–421.
https://doi.org/10.3928/01484834-20210616-12

Seidman, I. (2013). Interviewing as qualitative research: A guide for researchers in education and the social sciences (3rd ed.). Teachers College Press.

Shearer, R. L., Gregg, A., & Joo, K. P. (2015). Deep learning in distance education: Are we achieving the goal?
American Journal of Distance Education, 29(2), 126–134. https://doi.org/10.1080/08923647.2015.1023637

Stefaniak, J. E., & Sentz, J. (2020). The role of needs assessment to validate contextual factors related to user experience design practices. In M. Schmidt, A. A. Tawfik, I. Jahnke, & Y. Earnshaw (Eds.), Learner and user experience research: An introduction for the field of learning design & technology. EdTech Books.
https://edtechbooks.org/ux/role_of_needs_assessment

Stenbom, S. (2018). A systematic review of the community of inquiry survey. The Internet and Higher Education
39, 22–32. https://doi.org/10.1016/j.iheduc.2018.06.001

Sullivan, T. M., & Freishtat, R. (2013). Extending learning beyond the classroom: Graduate student experiences of
online discussions in a hybrid course. Journal of Continuing Higher Education, 61(1), 12–22. https://doi.org/10.1080/07377363.2013.758555

Sun, Y., & Gao, F. (2017). Comparing the use of a social annotation tool and a threaded discussion forum to support
online discussions. The Internet and Higher Education, 32, 72–79. https://doi.org/10.1016/j.iheduc.2016.10.001

Sun, Z., Lin, C. H., Wu, M., Zhou, J., & Luo, L. (2018). A tale of two communication tools: Discussion-forum and mobile instant-messaging apps in collaborative learning. British Journal of Educational Technology, 49(2). https://doi.org/10.1111/bjet.12571

Tibi, M. H. (2018). Computer science students’ attitudes towards the use of structured and unstructurered discussion forums in fully online courses. Online Learning, 22(1), 93–106.
https://doi.org/10.24059/olj.v22i1.995

Tracy, S. J. (2010). Qualitative quality: Eight “big-tent” criteria for excellent qualitative research. Qualitative Inquiry, 16(10), 837–851. https://doi.org/10.1177/1077800410383121

Veletsianos, G. (2013). Learner experiences with MOOCs and open online learning. Hybrid Pedagogy.
http://learnerexperiences.hybridpedagogy.com

Wise, A. F., & Chiu, M. M. (2014). The impact of rotating summarizing roles in online discussions: Effects on
learners’ listening behaviors during and subsequent to role assignment. Computers in Human Behavior, 38, 261–271. https://doi.org/10.1016/j.chb.2014.05.033

Wise, A. F., & Hsiao, Y.-T. (2019). Self-regulation in online discussions: Aligning data streams to investigate
relationships between speaking, listening, and task conditions. Computers in Human Behavior, 96, 273–284. https://doi.org/10.1016/j.chb.2018.01.034

Wise, A. F., Saghafian, M., & Padmanabhan, P. (2012). Towards more precise design guidance: Specifying and
testing the functions of assigned student roles in online discussions. Educational Technology Research and Development, 60(1), 55–82. https://doi.org/10.1007/s11423-011-9212-7

Ye, D., & Pennisi, S. (2022). Analysing interactions in online discussions through social network analysis. Journal of
Computer Assisted Learning, 38(3), 784–796. https://doi.org/10.1111/jcal.12648

Zengilowski, A., Lee, J., Gaines, R. E., Park, H., Choi, E., & Schallert, D. L. (2023). The collective classroom "we":
The role of students’ sense of belonging on their affective, cognitive, and discourse experiences of online and face-to-face discussions. Linguistics and Education, 73, 101142. https://doi.org/10.1016/j.linged.2022.101142