← 研踪 PaperTrailarXiv:2604.26020 · v2
Training Computer Use Agents to Assess the Usability of Graphical User Interfaces
首次提交 2026-04-29 02:04
版本更新 2026-11-11 04:28
雷达观测 2026-11-13 11:12
摘要
Recent HCI research leverages AI agents to assess graphical user interfaces (GUIs) usability, but current agents still struggle. Computer use agents (CUAs) that are used in these assessments are predominantly developed using objective task trajectories with verifiable outcomes and lack an operationalized definition of usability and necessary training data. This work presents a novel training objective that operationalizes usability to fine-tune CUAs to assess GUI usability through relative comparisons between interface variants. We apply our method to CUA models using a large-scale dataset of fully interactive UIs paired with usability labels mixed with human preferences.Two empirical evaluations across real and synthetic GUIs show that our method i) improved relative usability assessment accuracy across three off-the-shelf CUA models, ii) enabled our best model to outperform proprietary baselines, and iii) produced grounded usability assessments. Our work lays a data-driven foundation for agent-based usability assessment, enabling exploration of new forms of scalable usability evaluation and interface design.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-11 04:28 · 最新观测版本
Training Computer Use Agents to Assess the Usability of Graphical User Interfaces
Recent HCI research leverages AI agents to assess graphical user interfaces (GUIs) usability, but current agents still struggle. Computer use agents (CUAs) that are used in these assessments are predominantly developed using objective task trajectories with verifiable outcomes and lack an operationalized definition of usability and necessary training data. This work presents a novel training objective that operationalizes usability to fine-tune CUAs to assess GUI usability through relative comparisons between interface variants. We apply our method to CUA models using a large-scale dataset of fully interactive UIs paired with usability labels mixed with human preferences.Two empirical evaluations across real and synthetic GUIs show that our method i) improved relative usability assessment accuracy across three off-the-shelf CUA models, ii) enabled our best model to outperform proprietary baselines, and iii) produced grounded usability assessments. Our work lays a data-driven foundation for agent-based usability assessment, enabling exploration of new forms of scalable usability evaluation and interface design.
查看此版本解读 →查看此版本原文 ↗