← 研踪 PaperTrailarXiv:2610.14381 · v2
ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence
首次提交 2026-10-10 07:01
版本更新 2026-11-08 08:32
雷达观测 2026-11-10 12:16
摘要
Large language models are increasingly used as scientific assistants, yet we still lack reliable ways to evaluate whether they can investigate a system rather than merely answer a prompt. A useful evaluation must let agents choose experiments, interpret finite evidence, and make causal decisions, while preserving objective ground truth. We introduce ACED (Active Causal Evidence and Decision Benchmark), a neuro-symbolic benchmark that separates the readable scientific surface from the formal causal world: natural-language scenarios describe the system, while a hidden causal model defines the data, the effect of every intervention, and the oracle answers. In ACED-Repair, agents diagnose a failing production system: within 15 experiments they must decide which controls to test, read confounded and filtered data, and recommend a fix, which we re-simulate on a hidden true objective. In ACED-Query, agents answer causal questions and make constrained causal decisions in worlds backed by categorical probabilistic graphical models. Acceptance on ACED-Repair ranges from 43% to 0% across 12 LLMs, while strong agents approach ceiling on ACED-Query's qualification questions. Tracing every experiment shows where investigations stop: most open-weight models test a true lever on its own in fewer than half of the worlds, the strongest models test the right levers but often miss the working dose or combination, and agents that do run a working fix almost always recommend it. No model reliably repairs systems with two independent causes or systems that need a subgroup-specific policy. These results suggest that reliable AI co-scientists need explicit support for choosing experiments, keeping track of evidence, and composing interventions. Code and data are available at https://github.com/Rose-STL-Lab/ACED-Bench.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-08 08:32 · 最新观测版本
ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence
Large language models are increasingly used as scientific assistants, yet we still lack reliable ways to evaluate whether they can investigate a system rather than merely answer a prompt. A useful evaluation must let agents choose experiments, interpret finite evidence, and make causal decisions, while preserving objective ground truth. We introduce ACED (Active Causal Evidence and Decision Benchmark), a neuro-symbolic benchmark that separates the readable scientific surface from the formal causal world: natural-language scenarios describe the system, while a hidden causal model defines the data, the effect of every intervention, and the oracle answers. In ACED-Repair, agents diagnose a failing production system: within 15 experiments they must decide which controls to test, read confounded and filtered data, and recommend a fix, which we re-simulate on a hidden true objective. In ACED-Query, agents answer causal questions and make constrained causal decisions in worlds backed by categorical probabilistic graphical models. Acceptance on ACED-Repair ranges from 43% to 0% across 12 LLMs, while strong agents approach ceiling on ACED-Query's qualification questions. Tracing every experiment shows where investigations stop: most open-weight models test a true lever on its own in fewer than half of the worlds, the strongest models test the right levers but often miss the working dose or combination, and agents that do run a working fix almost always recommend it. No model reliably repairs systems with two independent causes or systems that need a subgroup-specific policy. These results suggest that reliable AI co-scientists need explicit support for choosing experiments, keeping track of evidence, and composing interventions. Code and data are available at https://github.com/Rose-STL-Lab/ACED-Bench.
查看此版本解读 →查看此版本原文 ↗