Evaluating open-weight LLMs for scoring L2 essays: Accuracy, prompt transfer, and deployment of AiAWE
John Maurice Gayed
This study presents AiAWE, an open-source automated writing evaluation system that scores argumentative essays with a LoRA-adapted large language model deployed on local hardware. Using 480 TOEFL Independent Writing essays from a restricted-access Educational Testing Service (ETS) dataset, we fine-tuned Gemma-3-27B and LLaMA-3.3-70B under identical LoRA configurations on 120 essays from one prompt, with all hyperparameters fixed in advance, and evaluated them on the remaining 360 essays. Gemma achieved a root mean square error (RMSE) of 0.474 and a quadratic weighted kappa (QWK) of 0.828 on the half-point grid against the ETS scores. In paired comparisons, it significantly outperformed a fine-tuned GPT-3.5 baseline from prior work on every metric tested, showing that a locally deployable open-weight model can achieve competitive scoring performance in this setting. It also had significantly lower error than the larger LLaMA configuration, although, because the two models differ in family as well as size, this comparison does not isolate the effect of scale. Neither open-weight model showed a statistically detectable increase in error on the 240 essays written to a second, unseen prompt from the same task family. On a single consumer-grade GPU, the scoring model returned a complete response in a median of 14.2 s, demonstrating technical feasibility under the tested conditions. Only the scoring component was evaluated, not the generated feedback. The system is publicly accessible at https://app.awade.gec.waseda.ac.jp, and its code, LoRA adapters, and training configurations are openly released.
尚无 AI 解读，展示原文摘要