← 研踪 PaperTrailarXiv:2611.06922 · v2
Towards Comprehensive Mobile Application Testing for User-centric Feature Coverage via Simulating Real-World Users with Personas
首次提交 2026-07-12 09:25
版本更新 2026-11-06 12:01
雷达观测 2026-11-09 10:16
摘要
LLM agents are increasingly used for black-box mobile application testing, where the ultimate goal is the comprehensive coverage of the features users rely on. However, most existing studies focus on Activity Coverage, which measures the proportion of distinct application screens visited during a test session but does not directly measure whether agents actually test the features users care about. To address this, we focus on (1) how to automatically measure whether a testing framework covers the features users care about, and (2) how to design a framework that achieves more comprehensive feature coverage. To answer (1), we introduce the User-Centric Feature Coverage benchmark, which consists of two components: a manually crafted dataset of 3,012 user-centric target features based on real reviews and app descriptions, and an automated evaluation pipeline that translates agent-generated test cases and target features into a shared semantic space, then uses an LLM-as-a-judge to compute coverage. To answer (2), we introduce ECS (Exploration, Critical User Journeys, and Synthetic Reviews). ECS is a framework that explores the application, then uses an LLM to convert exploration traces into Critical User Journeys (CUJs) to bring the semantic meaning of the trace closer to a distinct feature. Each CUJ consists of an LLM-inferred user intent paired with reproducible test steps. ECS then aggregates multiple CUJs using persona-augmented synthetic reviews to capture more complex features, simulating real users who build knowledge of an application across multiple screens before writing a review. ECS achieves 64.1% feature coverage, a 23.8% gain over the state-of-the-art baseline, and covers 72.5% of features in app descriptions, capturing the most prominent features. Our failure analysis shows statelessness and exploration discoverability as main reasons of failures, providing future directions.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-06 12:01 · 最新观测版本
Towards Comprehensive Mobile Application Testing for User-centric Feature Coverage via Simulating Real-World Users with Personas
LLM agents are increasingly used for black-box mobile application testing, where the ultimate goal is the comprehensive coverage of the features users rely on. However, most existing studies focus on Activity Coverage, which measures the proportion of distinct application screens visited during a test session but does not directly measure whether agents actually test the features users care about. To address this, we focus on (1) how to automatically measure whether a testing framework covers the features users care about, and (2) how to design a framework that achieves more comprehensive feature coverage. To answer (1), we introduce the User-Centric Feature Coverage benchmark, which consists of two components: a manually crafted dataset of 3,012 user-centric target features based on real reviews and app descriptions, and an automated evaluation pipeline that translates agent-generated test cases and target features into a shared semantic space, then uses an LLM-as-a-judge to compute coverage. To answer (2), we introduce ECS (Exploration, Critical User Journeys, and Synthetic Reviews). ECS is a framework that explores the application, then uses an LLM to convert exploration traces into Critical User Journeys (CUJs) to bring the semantic meaning of the trace closer to a distinct feature. Each CUJ consists of an LLM-inferred user intent paired with reproducible test steps. ECS then aggregates multiple CUJs using persona-augmented synthetic reviews to capture more complex features, simulating real users who build knowledge of an application across multiple screens before writing a review. ECS achieves 64.1% feature coverage, a 23.8% gain over the state-of-the-art baseline, and covers 72.5% of features in app descriptions, capturing the most prominent features. Our failure analysis shows statelessness and exploration discoverability as main reasons of failures, providing future directions.
查看此版本解读 →查看此版本原文 ↗