← 研踪 PaperTrailarXiv:2606.12801 · v2
Evaluating open-weight LLMs for scoring L2 essays: Accuracy, prompt transfer, and deployment of AiAWE
首次提交 2026-06-11 09:55
版本更新 2026-10-28 08:23
雷达观测 2026-10-30 10:23
摘要
This study presents AiAWE, an open-source automated writing evaluation system that scores argumentative essays with a LoRA-adapted large language model deployed on local hardware. Using 480 TOEFL Independent Writing essays from a restricted-access Educational Testing Service (ETS) dataset, we fine-tuned Gemma-3-27B and LLaMA-3.3-70B under identical LoRA configurations on 120 essays from one prompt, with all hyperparameters fixed in advance, and evaluated them on the remaining 360 essays. Gemma achieved a root mean square error (RMSE) of 0.474 and a quadratic weighted kappa (QWK) of 0.828 on the half-point grid against the ETS scores. In paired comparisons, it significantly outperformed a fine-tuned GPT-3.5 baseline from prior work on every metric tested, showing that a locally deployable open-weight model can achieve competitive scoring performance in this setting. It also had significantly lower error than the larger LLaMA configuration, although, because the two models differ in family as well as size, this comparison does not isolate the effect of scale. Neither open-weight model showed a statistically detectable increase in error on the 240 essays written to a second, unseen prompt from the same task family. On a single consumer-grade GPU, the scoring model returned a complete response in a median of 14.2 s, demonstrating technical feasibility under the tested conditions. Only the scoring component was evaluated, not the generated feedback. The system is publicly accessible at https://app.awade.gec.waseda.ac.jp, and its code, LoRA adapters, and training configurations are openly released.
作者备注：27 pages with 14 tables and 1 figure
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-10-28 08:23 · 最新观测版本
Evaluating open-weight LLMs for scoring L2 essays: Accuracy, prompt transfer, and deployment of AiAWE
This study presents AiAWE, an open-source automated writing evaluation system that scores argumentative essays with a LoRA-adapted large language model deployed on local hardware. Using 480 TOEFL Independent Writing essays from a restricted-access Educational Testing Service (ETS) dataset, we fine-tuned Gemma-3-27B and LLaMA-3.3-70B under identical LoRA configurations on 120 essays from one prompt, with all hyperparameters fixed in advance, and evaluated them on the remaining 360 essays. Gemma achieved a root mean square error (RMSE) of 0.474 and a quadratic weighted kappa (QWK) of 0.828 on the half-point grid against the ETS scores. In paired comparisons, it significantly outperformed a fine-tuned GPT-3.5 baseline from prior work on every metric tested, showing that a locally deployable open-weight model can achieve competitive scoring performance in this setting. It also had significantly lower error than the larger LLaMA configuration, although, because the two models differ in family as well as size, this comparison does not isolate the effect of scale. Neither open-weight model showed a statistically detectable increase in error on the 240 essays written to a second, unseen prompt from the same task family. On a single consumer-grade GPU, the scoring model returned a complete response in a median of 14.2 s, demonstrating technical feasibility under the tested conditions. Only the scoring component was evaluated, not the generated feedback. The system is publicly accessible at https://app.awade.gec.waseda.ac.jp, and its code, LoRA adapters, and training configurations are openly released.
查看此版本解读 →查看此版本原文 ↗