← 研踪 PaperTrailarXiv:2603.00468 · v3
Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems
首次提交 2026-02-28 13:04
版本更新 2026-11-06 10:02
雷达观测 2026-11-09 10:16
摘要
LLM agents are explored for automating root cause analysis (RCA) in cloud-native systems, creating a need to evaluate both diagnostic correctness and the quality of the supporting investigation. Existing static benchmarks offer repeatable inputs but limited system-facing interaction, while live testbeds expose tools but hinder controlled comparison because incident evidence varies across runs; both paradigms focus primarily on final answers. To address these limitations, we present Cloud-OpsBench, an evaluation infrastructure for interactive and evidence-grounded cloud RCA. It comprises 754 runtime-verified cases across 57 fault types on two microservice workloads spanning application services and Kubernetes platform layers. Each fault is captured as a state snapshot and replayed through standard diagnostic interfaces, with outcome labels and diagnostic evidence graphs that enable matched comparisons and process-level analysis. Evaluating eleven LLMs on Cloud-OpsBench reveals that strong outcome accuracy often masks incomplete investigations: while Joint RCA Accuracy (JRA) reaches up to 0.78 on OnlineBoutique and 0.68 on TrainTicket, models identify the correct root cause without achieving evidence closure (e.g., in 33.1% and 29.4% of evaluated cases for GPT-5). These results demonstrate that final-answer accuracy alone cannot establish whether an agent has traced the failure and collected the evidence needed to confirm its root cause.
作者备注：28 pages, 6 figures
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-06 10:02 · 最新观测版本
Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems
LLM agents are explored for automating root cause analysis (RCA) in cloud-native systems, creating a need to evaluate both diagnostic correctness and the quality of the supporting investigation. Existing static benchmarks offer repeatable inputs but limited system-facing interaction, while live testbeds expose tools but hinder controlled comparison because incident evidence varies across runs; both paradigms focus primarily on final answers. To address these limitations, we present Cloud-OpsBench, an evaluation infrastructure for interactive and evidence-grounded cloud RCA. It comprises 754 runtime-verified cases across 57 fault types on two microservice workloads spanning application services and Kubernetes platform layers. Each fault is captured as a state snapshot and replayed through standard diagnostic interfaces, with outcome labels and diagnostic evidence graphs that enable matched comparisons and process-level analysis. Evaluating eleven LLMs on Cloud-OpsBench reveals that strong outcome accuracy often masks incomplete investigations: while Joint RCA Accuracy (JRA) reaches up to 0.78 on OnlineBoutique and 0.68 on TrainTicket, models identify the correct root cause without achieving evidence closure (e.g., in 33.1% and 29.4% of evaluated cases for GPT-5). These results demonstrate that final-answer accuracy alone cannot establish whether an agent has traced the failure and collected the evidence needed to confirm its root cause.
查看此版本解读 →查看此版本原文 ↗