← 研踪 PaperTrailarXiv:2607.06873 · v2
Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
首次提交 2026-07-08 08:11
版本更新 2026-11-16 23:21
雷达观测 2026-11-17 16:10
摘要
Failures of conversational LLM agents can have real-world consequences, for example when an agent completes a transaction without the user's confirmation. An important class of such failures occurs at workflow boundaries, such as identity checks and confirmation steps, which a test can reach only after several turns of conversation. We present AgentEval, a black-box testing framework that discovers and tests these boundaries. AgentEval interacts with an agent to mine a conversational workflow graph, a model of its behavior whose nodes are the agent's activities and whose edges are the user actions observed between them. It generates tests at each node of this graph, replaying the conversational path to that node before applying a perturbation. AgentEval then runs each test, and a separate LLM judge decides whether it passes or fails using only the conversation turns. We benchmark AgentEval against a privileged, white-box auditor with access to the agent's underlying source code, which AgentEval never sees. On four -bench agents, the boundary tests of AgentEval cover 38.8-46.0 distinct boundaries per agent on average over five repetitions. In ablation studies, generating tests without targeting the graph's nodes lowers the distinct-boundary count by 2.0-7.0 in every agent and raises the duplicate rate in most agents.
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-16 23:21 · 最新观测版本
Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
Failures of conversational LLM agents can have real-world consequences, for example when an agent completes a transaction without the user's confirmation. An important class of such failures occurs at workflow boundaries, such as identity checks and confirmation steps, which a test can reach only after several turns of conversation. We present AgentEval, a black-box testing framework that discovers and tests these boundaries. AgentEval interacts with an agent to mine a conversational workflow graph, a model of its behavior whose nodes are the agent's activities and whose edges are the user actions observed between them. It generates tests at each node of this graph, replaying the conversational path to that node before applying a perturbation. AgentEval then runs each test, and a separate LLM judge decides whether it passes or fails using only the conversation turns. We benchmark AgentEval against a privileged, white-box auditor with access to the agent's underlying source code, which AgentEval never sees. On four -bench agents, the boundary tests of AgentEval cover 38.8-46.0 distinct boundaries per agent on average over five repetitions. In ablation studies, generating tests without targeting the graph's nodes lowers the distinct-boundary count by 2.0-7.0 in every agent and raises the duplicate rate in most agents.
查看此版本解读 →查看此版本原文 ↗