← 研踪 PaperTrailarXiv:2608.30685 · v2
ATLAS: Dual-Horizon Diagnostic Evaluation for Tool-Use Agents in Meituan Xiaotua
首次提交 2026-08-31 20:25
版本更新 2026-10-29 16:57
雷达观测 2026-10-30 13:28
摘要
LLM-based assistants on web-scale service platforms increasingly address local-service needs through multi-step planning and tool use. Under dynamic business conditions and evolving user needs, outcome-level evaluation offers limited evidence of where execution fails or how service drifts across continued interaction. We present ATLAS, a dual-horizon diagnostic evaluation framework for industrial tool-use agents. Within-turn evaluation uses trajectory-wise signals across thinking and reflection, tool and skill execution, and response generation to identify execution failures and unmet capability requirements. Across-turn evaluation uses user-wise signals spanning multiple lifecycles to diagnose failures in maintaining or updating user intent, constraints, and corrections. Fine-grained signals use rule-based checks and LLM judgments with explicit evidence scopes and decision boundaries. To support reliable diagnosis at production scale, LLM judges are calibrated against dynamically maintained references from business logs, and selected judges are distilled into compact diagnostic models. The resulting diagnostic profiles help developers attribute failures, prioritize targeted interventions, and reassess improvements. Applied in Meituan Xiaotuan, a million-scale DAU local-services assistant, ATLAS achieves high diagnostic fidelity across calibrated judges and distilled models. Blinded human auditing and a 14-day online A/B test with nearly two million users per arm confirm that the ATLAS-guided agent improves recommendation relevance, factual reliability, user engagement (+2.84% answer read-through rate), and order volume (+6.13% paid orders), leading to full production deployment.
作者备注：19 pages; The project's homepage is https://atlas-eval.github.io/
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-10-29 16:57 · 最新观测版本
ATLAS: Dual-Horizon Diagnostic Evaluation for Tool-Use Agents in Meituan Xiaotua
LLM-based assistants on web-scale service platforms increasingly address local-service needs through multi-step planning and tool use. Under dynamic business conditions and evolving user needs, outcome-level evaluation offers limited evidence of where execution fails or how service drifts across continued interaction. We present ATLAS, a dual-horizon diagnostic evaluation framework for industrial tool-use agents. Within-turn evaluation uses trajectory-wise signals across thinking and reflection, tool and skill execution, and response generation to identify execution failures and unmet capability requirements. Across-turn evaluation uses user-wise signals spanning multiple lifecycles to diagnose failures in maintaining or updating user intent, constraints, and corrections. Fine-grained signals use rule-based checks and LLM judgments with explicit evidence scopes and decision boundaries. To support reliable diagnosis at production scale, LLM judges are calibrated against dynamically maintained references from business logs, and selected judges are distilled into compact diagnostic models. The resulting diagnostic profiles help developers attribute failures, prioritize targeted interventions, and reassess improvements. Applied in Meituan Xiaotuan, a million-scale DAU local-services assistant, ATLAS achieves high diagnostic fidelity across calibrated judges and distilled models. Blinded human auditing and a 14-day online A/B test with nearly two million users per arm confirm that the ATLAS-guided agent improves recommendation relevance, factual reliability, user engagement (+2.84% answer read-through rate), and order volume (+6.13% paid orders), leading to full production deployment.
查看此版本解读 →查看此版本原文 ↗