跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2606.16613 · v2

CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies

首次提交 2026-06-15 20:04
版本更新 2026-11-13 21:22
雷达观测 2026-11-16 11:41
arXiv ↗PDF ↗

摘要

LLM agents are increasingly capable of handling complex, long-horizon tasks and may eventually take on business management, a role traditionally performed by humans. Deploying them safely requires monitoring their capabilities and behavior in environments that reflect real economies. We introduce CoffeeBench, a benchmark built on a compact simulation of a coffee supply chain with realistic business mechanics. Six autonomous LLM coding agents run its farmers, roasters, and retailers, and negotiate and trade with one another to maximize their own net income over 90 simulated days. We evaluate eight recent proprietary and open-weight models by letting each run one roaster while a fixed background model runs the other five firms, and scoring that roaster's net income. Claude Sonnet 5.5 earns the highest net income, closely followed by GPT-6 Astra. The leaders sell at higher prices, and the best model also buys its inputs far more cheaply, whereas weaker models trade at thin margins. Since performance targets are a common cause of accounting fraud in human firms, we also test whether agents under KPI pressure inflate their numbers. Under a revenue target, circular trades occur for some pairs of models: one agent proposes that the two rival firms sell goods back and forth at inflated prices so that both book revenue, and the rival accepts. We hope CoffeeBench serves as an early step toward a future in which LLM agents manage firms without misconduct. We release our code and agent trajectories.

作者备注:Accepted to NeurIPS 2026 (Evaluations and Datasets Track)

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-13 21:22 · 最新观测版本

CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies

LLM agents are increasingly capable of handling complex, long-horizon tasks and may eventually take on business management, a role traditionally performed by humans. Deploying them safely requires monitoring their capabilities and behavior in environments that reflect real economies. We introduce CoffeeBench, a benchmark built on a compact simulation of a coffee supply chain with realistic business mechanics. Six autonomous LLM coding agents run its farmers, roasters, and retailers, and negotiate and trade with one another to maximize their own net income over 90 simulated days. We evaluate eight recent proprietary and open-weight models by letting each run one roaster while a fixed background model runs the other five firms, and scoring that roaster's net income. Claude Sonnet 5.5 earns the highest net income, closely followed by GPT-6 Astra. The leaders sell at higher prices, and the best model also buys its inputs far more cheaply, whereas weaker models trade at thin margins. Since performance targets are a common cause of accounting fraud in human firms, we also test whether agents under KPI pressure inflate their numbers. Under a revenue target, circular trades occur for some pairs of models: one agent proposes that the two rival firms sell goods back and forth at inflated prices so that both book revenue, and the rival accepts. We hope CoffeeBench serves as an early step toward a future in which LLM agents manage firms without misconduct. We release our code and agent trajectories.

查看此版本解读 →查看此版本原文 ↗