Department of English Language and Literature, Faculty of Foreign Language, Yazd University, Yazd, Iran.
Abstract: (17 Views)
obective: This study evaluated AI detection tools’ efficacy in assessing EFL students’ writing, focusing on distinguishing AI-generated from human text without misclassifying L2 work.
Methods: A two-phase quantitative design was used. Phase 1 involved 30 BA students from Yazd University producing one non-AI and one ChatGPT-assisted essay on identical topics. Phase 2 compared two MA thesis corpora: 30 from the pre-AI era (2010–2015) and 30 from the post-AI era (2020–2025). All texts were analyzed using Turnitin and Grammarly Premium.
Results: Results, analyzed via Wilcoxon Signed-Rank Tests and one-sample t-tests, revealed significant performance divergence. Turnitin demonstrated near-perfect accuracy in identifying AI-assisted essays, whereas Grammarly showed a statistically significant false negative rate, failing to detect some AI content. Both tools accurately analyzed short, human-written essays. However, the thesis analysis revealed increased AI-detection scores in the post-AI corpus, contrasting with near-zero scores in the pre-AI corpus.
Conclusion: This finding suggests either actual, undisclosed AI use or algorithmic bias misclassifying Iranian EFL linguistic patterns as AI-generated. The study concludes that while Turnitin is more robust than Grammarly, uncritical tool application risks systematic misjudgment against L2 writers. False positives and negatives threaten academic equity. Thus, AI-detection outputs should trigger pedagogical dialogue and human-led investigation rather than serving as sole misconduct evidence.
Methods: A two-phase quantitative design was used. Phase 1 involved 30 BA students from Yazd University producing one non-AI and one ChatGPT-assisted essay on identical topics. Phase 2 compared two MA thesis corpora: 30 from the pre-AI era (2010–2015) and 30 from the post-AI era (2020–2025). All texts were analyzed using Turnitin and Grammarly Premium.
Results: Results, analyzed via Wilcoxon Signed-Rank Tests and one-sample t-tests, revealed significant performance divergence. Turnitin demonstrated near-perfect accuracy in identifying AI-assisted essays, whereas Grammarly showed a statistically significant false negative rate, failing to detect some AI content. Both tools accurately analyzed short, human-written essays. However, the thesis analysis revealed increased AI-detection scores in the post-AI corpus, contrasting with near-zero scores in the pre-AI corpus.
Conclusion: This finding suggests either actual, undisclosed AI use or algorithmic bias misclassifying Iranian EFL linguistic patterns as AI-generated. The study concludes that while Turnitin is more robust than Grammarly, uncritical tool application risks systematic misjudgment against L2 writers. False positives and negatives threaten academic equity. Thus, AI-detection outputs should trigger pedagogical dialogue and human-led investigation rather than serving as sole misconduct evidence.
Send email to the article author
| Rights and permissions | |
 |
This work is licensed under a Creative Commons — Attribution-NonCommercial 4.0 International (CC BY-NC 4.0). |
This work is licensed under a Creative Creative Commons — Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)