Introduction

Automated writing evaluation (AWE) systems are predominantly used for providing feedback for writing. Inadequate attention has been devoted to investigating the potential of such systems for enhancing speaking (Shadiev & Feng, 2023). Studies (e.g., Ranalli, 2021; Thi et al., 2022) exploring learner engagement with automated feedback (AF) have mainly considered engagement with in-class instructional activities. Out-of-class engagement has gone unaccounted for. Considering the prevalent application of AWE systems for language learning and the scant attention given to the innovative use of AWE systems for improving speaking, as well as the scarcity of investigations studying transformative experiences in language learning and teaching contexts, we aimed to explore the effect of the innovative use of Grammarly feedback on improving the speaking skills of English as a Foreign Language (EFL) learners. EFL learners are individuals who learn English in countries where English is not the primary language. Moreover, this study attempted to uncover learners’ perceptions of the efficacy of Grammarly feedback for enhancing speaking. Additionally, we sought to examine learners’ transformative engagement experiences as a result of Grammarly feedback provision. Conducting this study is important as it not only fills the lacuna in the extant empirical literature but also expands the line of research on engagement from in-class contexts to out-of-class settings. Moreover, the results can provide insights into the innovative use of AWE systems for developing speaking skills.  According to these objectives, the following research questions guided the study:

• RQ1: Does the use of Grammarly feedback on EFL learners’ transcribed speech significantly enhance their speaking performance?
• RQ2: What are EFL learners’ perceptions towards the use of Grammarly feedback on their transcribed speech for improving their speaking performance?
• RQ3: Do learners have any transformative engagement experiences as a result of receiving feedback from Grammarly?

Relevant Literature

The advent of advanced technological tools has given an ever-growing rise to the implementation of AWE systems, such as Grammarly, in language learning contexts (Koltovskaia, 2020). A strand of previous investigations on Grammarly (e.g., Fahmi & Cahyono, 2021; Koltovskaia, 2020; ONeill & Russell, 2019) indicates this AWE system has been mainly utilized for feedback concerning writing. Scholars have predominantly explored teachers' and learners’ perceptions of the efficacy of Grammarly in this regard. As Shadiev and Feng (2023) maintain, the use of automated corrective feedback should not be focused solely on writing and can be extended to other language skills, such as speaking. Another strand of studies (e.g., Ranalli, 2021; Thi et al., 2022; Tian & Zhou, 2020; Zhang & Hyland, 2018) focused on learner engagement with automated feedback. These studies, however, have dealt with engagement from an instructional and educational lens and failed to consider engagement beyond the classroom from a transformative engagement experience (TEE) perspective.

TEE refers to the involvement with learning experiences outside the classroom based on the ideas learned in class to experience everyday life in the world differently (Pugh et al., 2017). As Pugh et al. (2017) maintain, transformative experience encompasses three dimensions: motivated use, expansion of perception, and experiential value. Motivated use refers to applying the in-class learned ideas in free-choice settings; expansion of perception constitutes viewing the issues and events via the perspective of in-class learned content; and experiential value pertains to valuing the content for enriching everyday life experiences. A review of investigations on TEE shows this construct has been explored mainly in science education (e.g., Girod et al., 2010), and higher education (Clifford & Montgomery, 2015). As for language teaching and learning contexts, inadequate attention has been paid to transformative experiences (Abdollahzadeh et al., 2022).

Methods

This study adopted a mixed-methods design. The population consisted of 456 intermediate-level learners studying at three branches of Safir Language Academy in Tehran, Iran. Out of the 456 learners, 107 expressed their willingness to take part in this study. Thus, a convenience sampling technique was used. The target participants were comprised of 63 Iranian EFL learners who were selected out of the initial pool of 107 students based on their performance on a Preliminary English Test (PET). To this aim, only those learners whose PET scores fell within the range of +/-one standard deviation from the mean were selected. The 63 selected learners were within the age range of 18 to 41 (M=27.30, SD=4.23), and Persian was their mother tongue. Table 1 displays the participants’ demographic information.

Table 1

Demographic Information of the Participants

The participants were divided randomly into two groups: experimental (N=33) and control (N=30). The speaking test scores obtained on PET, administered for homogeneity purposes, were used as pretest scores (see Table 2). To make sure that the two groups were homogenized before the treatment, an independent samples t-test was run. With t (61) =1.02, p=0.31>.05, it was concluded that the mean difference between the speaking pretest scores for the experimental and control groups was not statistically significant. After making sure that the two groups were homogenized regarding their speaking pretest, the treatment was administered.

Table 2

Descriptive Statistics for Pretest Scores

In the experimental group, the participants were given a speaking topic and asked to deliver a 3–5-minute speech each session. Their speech presentations were audio-recorded, and learners were required to transcribe the speech content and submit the text to Grammarly to receive feedback. Learners were asked to give a second revised presentation on the same topic, drawing on Grammarly feedback. The learners in the control group were also asked to give a 3–5-minute speech presentation each session. They received feedback from the teacher and were required to deliver a second presentation based on the teacher’s feedback. However, they were not asked to transcribe their speech and/or submit it to Grammarly. The treatment lasted for 15 sessions over two months. Upon finishing the treatment, the learners in both groups were given the speaking test of PET from another version as a posttest. Fifteen participants from the experimental group, who expressed their consent to attend interviews, took part in semi-structured interviews (see Table 3). The same 15 learners were also interviewed to uncover how EFL learners relate Grammarly feedback to their everyday life experiences from a transformative engagement perspective.

Table 3

The Demographic Information for the 15 Interviewees

To develop the semi-structured interview questions, initially, the literature on transformative engagement (e.g., Abdollahzadeh et al., 2022; Pugh et al., 2017) and learners’ perceptions towards Grammarly (e.g., Fahmi & Cahyono, 2021; Koltovskaia, 2020; Oneill & Russell, 2019) was reviewed. Then, an initial list of interview questions was developed and reviewed by an expert panel consisting of three Ph.D. holders in the field of Teaching English as a Foreign Language (TEFL). TEFL experts teach English to non-native speakers in environments where English is not the primary language. Afterward, the questions were piloted on a sample of five participants to enhance clarity.

Results and Discussion

To investigate any significant difference between the speaking posttest scores for the experimental and control groups, an independent samples t-test was run (see Table 4). With t (61) =7.15, p =0.00<.001, it was concluded the mean difference between the speaking posttest scores for the experimental and control groups was statistically significant. Moreover, the posttest score mean for the experimental group was higher than that of the control group (11.54>9.20). Thus, it was concluded that the use of Grammarly feedback significantly enhanced EFL learners’ speaking performance.

Table 4

Descriptive Statistics for Speaking Posttest Scores

To analyze the interview contents, steps provided by Auerbach and Silverstein (2003) were followed. The results suggested that learners held positive views towards Grammarly feedback as it provided them with motivation, engagement, satisfaction, and convenience in the feedback process (see Table 5).

Table 5

Content Analysis for Participants’ Perceptions Towards Grammarly Feedback

Concerning motivation, for example, one of the interviewees stated, “Grammarly feedback was good because it made me want to speak more since the feedback was full of points.” With respect to engagement, one of the interviewees pointed out that “The feedback from Grammarly made me pay more attention to my speaking because I had everything recorded, and I really wanted to have fewer mistakes.”

The content analysis results also revealed traces of motivated use, expansion of perception, and experiential value in relation to the application of in-class ideas and contents related to Grammarly feedback in learners’ everyday lives (see Table 6). As for motivated use, as an example, one of the interviewees wanted to receive similar feedback for a presentation in his biology course at university, even though his instructor emphasized the importance of content over grammar. He decided to transcribe his speech and use another similar application to check his speaking and receive feedback. He stated, “When I learned that Grammarly could give me such great feedback, I wanted to receive this feedback for my presentation. I wanted to check my speaking myself.”

Table 6

Content Analysis for Participants’ Transformative Engagement Experiences

Overall, the results of the current study corroborate the findings of previous investigations (e.g., Fahmi & Cahyono, 2021; ONeill & Russell, 2019; Thi et al., 2022) concerning learners’ positive views toward AWE systems and the effectiveness of Grammarly for fostering language performance. Furthermore, the findings shed light on how AWE feedback can be related to EFL learners’ everyday life experiences. Based on the findings, EFL teachers are encouraged to employ Grammarly feedback to improve EFL learners’ speaking. Future research concerning the integration of Grammarly feedback with teachers’ feedback on transcribed speech texts for improving speaking fluency, accuracy, and complexity is recommended.

References

Abdollahzadeh, E., Farsani, M. A., & Zandi, M. (2022). The relationship between L2 motivation and transformative engagement in academic reading among EAP learners: Implications for reading self-regulation. Frontiers in Psychology, 13. https://doi.org/10.3389/fpsyg.2022.944650 

Auerbach, C. E., & Silverstein, L. B. (2003). Qualitative data: An introduction to coding and analysis. New York: NYU Press.

Clifford, V., & Montgomery, C. (2015). Transformative learning through internationalization of the curriculum in higher education. Journal of Transformative Education, 13(1), 46-64. https://doi.org/10.1177/1541344614560909 

Fahmi, M. A., & Cahyono, B. Y. (2021). EFL students’ perception on the use of Grammarly and teacher feedback. JEES (Journal of English Educators Society), 6(1), 18-25. https://doi.org/10.21070/jees.v6i1.792

Girod, M., Twyman, T., & Wojcikiewicz, S. (2010). Teaching and learning science for transformative, aesthetic experience. Journal of Science Teacher Education, 21, 801-824. http://dx.doi.org/10.1007/s10972-009-9175-2

Koltovskaia, S. (2020). Student engagement with automated written corrective feedback (AWCF) provided by Grammarly: A multiple case study. Assessing Writing, 44. https://doi.org/10.1016/j.asw.2020.100450 

ONeill, R., & Russell, A. (2019). Stop! Grammar time: University students’ perceptions of the automated feedback program Grammarly. Australasian Journal of Educational Technology, 35(1). https://ajet.org.au/index.php/AJET/article/download/3795/1514/ 

Pugh, K. J., Bergstrom, C. M., & Spencer, B. (2017). Profiles of transformative engagement: Identification, description,  and  relation  to  learning  and  instruction.  Science  Education,  101(3),  369-398. https://doi.org/10.1002/sce.21270 

Ranalli, J. (2021). L2 student engagement with automated feedback on writing: Potential for learning and issues of trust. Journal of Second Language Writing, 52. https://doi.org/10.1016/j.jslw.2021.100816 

Shadiev, R., & Feng, Y. (2023). Using automated corrective feedback tools in language learning: a review study. Interactive Learning Environments, 1-29. https://doi.org/10.1080/10494820.2022.2153145 

Thi, N. K., Nikolov, M., & Simon, K. (2022). Higher-proficiency students’ engagement with and uptake of teacher and Grammarly feedback in an EFL writing course. Innovation in Language Learning and Teaching, 1-16. https://doi.org/10.1080/17501229.2022.2122476 

Tian, L., & Zhou, Y. (2020). Learner engagement with automated feedback, peer feedback and teacher feedback in an online EFL writing context. System, 91. https://doi.org/10.1016/j.system.2020.102247 

Zhang, Z. V., & Hyland, K. (2018). Student engagement with teacher and automated feedback on L2 writing. Assessing Writing, 36, 90-102. https://doi.org/10.1016/j.asw.2018.02.004