跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2610.17895 · v4

Interaction-as-a-Judge

首次提交 2026-10-13 15:49
版本更新 2026-11-18 03:18
雷达观测 2026-11-19 19:08
arXiv ↗PDF ↗

摘要

Coding agents are tasked with producing sophisticated interactive applications such as websites, desktop applications, and games. While humans assess these applications interactively, current automated LLM evaluations primarily focus on code analysis and unit tests. This mismatch can cause LLM judges to miss failures that are immediately apparent to human users. We study Interaction-as-a-Judge, an evaluation paradigm where computer-use agents (CUAs) directly interact with an application. We demonstrate that Interaction-as-a-Judge can improve both evaluation and generation of interactive applications. To this end, we collect and release two complementary datasets: 6,8586,858 pairs of applications with human preferences and corresponding agent traces, and 222222 applications with human-verified requirement labels. Relative to standard non-interactive approaches, we find that Interaction-as-a-Judge improves evaluation in terms of predicting human preferences (65.6%→69.1%65.6\% \to 69.1\%) and requirement verification (77.0%→81.6%77.0\% \to 81.6\%). We also observe scaling benefits: across both preference alignment and requirement verification, higher-quality interactions improve performance. Finally, through both inference- and training-time experiments, we demonstrate that Interaction-as-a-Judge can also improve the quality of generated artifacts. At inference time, Interaction-as-a-Judge repairs 75.5%75.5\% of broken requirements within five attempts, compared to 55.4%55.4\% for code-only self-feedback. At training time, using Interaction-as-a-Judge as a reward signal improves win-rate over the base model by 6.76.7pp while reducing both generated code length by 35.7%35.7\% and build failure rates by 7.27.2pp compared to a code-only reward signal.

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v4,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v4 · 2026-11-18 03:18 · 最新观测版本

Interaction-as-a-Judge

Coding agents are tasked with producing sophisticated interactive applications such as websites, desktop applications, and games. While humans assess these applications interactively, current automated LLM evaluations primarily focus on code analysis and unit tests. This mismatch can cause LLM judges to miss failures that are immediately apparent to human users. We study Interaction-as-a-Judge, an evaluation paradigm where computer-use agents (CUAs) directly interact with an application. We demonstrate that Interaction-as-a-Judge can improve both evaluation and generation of interactive applications. To this end, we collect and release two complementary datasets: 6,8586,858 pairs of applications with human preferences and corresponding agent traces, and 222222 applications with human-verified requirement labels. Relative to standard non-interactive approaches, we find that Interaction-as-a-Judge improves evaluation in terms of predicting human preferences (65.6%→69.1%65.6\% \to 69.1\%) and requirement verification (77.0%→81.6%77.0\% \to 81.6\%). We also observe scaling benefits: across both preference alignment and requirement verification, higher-quality interactions improve performance. Finally, through both inference- and training-time experiments, we demonstrate that Interaction-as-a-Judge can also improve the quality of generated artifacts. At inference time, Interaction-as-a-Judge repairs 75.5%75.5\% of broken requirements within five attempts, compared to 55.4%55.4\% for code-only self-feedback. At training time, using Interaction-as-a-Judge as a reward signal improves win-rate over the base model by 6.76.7pp while reducing both generated code length by 35.7%35.7\% and build failure rates by 7.27.2pp compared to a code-only reward signal.

查看此版本解读 →查看此版本原文 ↗