arXiv:2605.29253 · v2
OpenClawBench: Benchmarking Process-side Anomalies in Real-world Agent Execution Trajectories
摘要
Task success can hide process anomalies in real-world agent executions. An agent may pass the final task oracle while still accumulating unresolved ambiguity, unsafe external writes, ignored errors, weakly grounded commitments, or capability-boundary overcommitment. We study this mismatch as the Outcome-Process Gap and introduce OpenClawBench, a large-scale dataset for measuring and supervising process-side anomalies in real agent execution processes. OpenClawBench is built from BFCL-driven OpenClaw sessions produced by 6 source models and contains 31,264 annotated trajectories. FullTax converts the aligned trajectories into structured anomaly supervision: binary labels, supporting evidence, onset/span localization, severity, recoverability, and a 5-class anomaly taxonomy. Using OpenClawBench, we make the Outcome-Process Gap measurable. Among 31,135 oracle-passing executions, 2,904 (9.33%) are still labeled process-anomalous under FullTax. Within oracle-passing executions that contain high-risk process evidence, 1,765 of 1,904 (92.70%) are labeled anomalous. On Gold-300, a test subset labeled independently by two human annotators, silver labels match the final human binary label on 93.3% of trajectories. A LoRA-fine-tuned Gemma 3 12B detector trained on the high-confidence FullTax supervised pool reaches binary F1=0.729 on the cleaner-labels silver test split (n=2,646). It outperforms the GPT-5.4 frontier reference by +0.302 absolute, the no-fine-tuning base by +0.357, and wins on all six source-model agent slices. The gain comes from calibration rather than higher recall: zero-shot detectors over-flag at 42-50% against a 14.8% label rate, while the fine-tuned detector predicts anomalies 18.2% of the time. Together, OpenClawBench turns real agent execution logs into auditable and reusable supervision for studying, diagnosing, and operationally monitoring runtime agent reliability.
作者备注：Accepted to the NeurIPS 2026 Evaluations & Datasets Track. 42 pages, 1 figure. Dataset: https://huggingface.co/datasets/YangzeLiu/OpenClawBench Code: https://github.com/YangzeLiu/OpenClawBench