← 研踪 PaperTrailarXiv:2606.03918 · v3
Hedge-Bench: Can Agents Reason Their Way to Better Predictions?
首次提交 2026-06-03 01:11
版本更新 2026-10-28 12:15
雷达观测 2026-10-30 09:53
摘要
Modern AI agents are increasingly capable of routine financial-analysis tasks such as retrieving documents and updating spreadsheets. A harder challenge is forecasting an unknown future from incomplete information. Public markets provide a natural testbed: analysts routinely predict discrete events with information from that point in time. We present Hedge-Bench, a benchmark of 92 real-world environments derived from hedge fund analysts' reasoning traces paired with subsequently resolved prediction questions. Agents reason from frozen information snapshots, and their forecasts are evaluated against real-world outcomes. Across 11 frontier models and three trials per environment, joint success rates, which require both a correct forecast and over 60% research coverage, range from 2.54 to 31.52%. Using GRPO with turn-level credit assignment, we train Muse-Glimmer on 165 environments, increasing coverage on 22 held-out environments from 17.17% to 33.36% at low reasoning effort and from 30.19% to 47.02% at extra-high effort. The trained model at low effort also exceeds the base model's extra-high-effort coverage while using 47.9% fewer tokens. We also observe evidence of generalization: held-out forecasting accuracy at high effort improves by 10.06 percentage points, while out-of-distribution evaluation scores improve by 3.94 points on GDPval, 2.29 on GDP_pdf, 7.07 on Harvey LAB, and 4.96 on Frontier Finance under their respective metrics. Code: https://github.com/Trata-Inc/trata-hedge-bench-verified.
作者备注：Dataset and evaluation harness available at https://github.com/Trata-Inc/trata-hedge-bench-verified
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v3 · 2026-10-28 12:15 · 最新观测版本
Hedge-Bench: Can Agents Reason Their Way to Better Predictions?
Modern AI agents are increasingly capable of routine financial-analysis tasks such as retrieving documents and updating spreadsheets. A harder challenge is forecasting an unknown future from incomplete information. Public markets provide a natural testbed: analysts routinely predict discrete events with information from that point in time. We present Hedge-Bench, a benchmark of 92 real-world environments derived from hedge fund analysts' reasoning traces paired with subsequently resolved prediction questions. Agents reason from frozen information snapshots, and their forecasts are evaluated against real-world outcomes. Across 11 frontier models and three trials per environment, joint success rates, which require both a correct forecast and over 60% research coverage, range from 2.54 to 31.52%. Using GRPO with turn-level credit assignment, we train Muse-Glimmer on 165 environments, increasing coverage on 22 held-out environments from 17.17% to 33.36% at low reasoning effort and from 30.19% to 47.02% at extra-high effort. The trained model at low effort also exceeds the base model's extra-high-effort coverage while using 47.9% fewer tokens. We also observe evidence of generalization: held-out forecasting accuracy at high effort improves by 10.06 percentage points, while out-of-distribution evaluation scores improve by 3.94 points on GDPval, 2.29 on GDP_pdf, 7.07 on Harvey LAB, and 4.96 on Frontier Finance under their respective metrics. Code: https://github.com/Trata-Inc/trata-hedge-bench-verified.
查看此版本解读 →查看此版本原文 ↗v2 · 2026-10-27 11:27
Hedge-Bench: Benchmarking Agents on Hard, Realistic Tasks Pertaining to Financial Reasoning
Modern AI agents are increasingly capable of routine financial-analysis tasks such as retrieving documents and updating spreadsheets. A harder challenge is forecasting an unknown future from incomplete information. Public markets provide a natural testbed: analysts routinely predict discrete events with information from that point in time. We present Hedge-Bench, a benchmark of 92 real-world environments derived from hedge fund analysts' reasoning traces paired with subsequently resolved prediction questions. Agents reason from frozen information snapshots, and their forecasts are evaluated against real-world outcomes. Across 11 frontier models and three trials per environment, joint success rates, which require both a correct forecast and over 60% research coverage, range from 2.54 to 31.52%. Using GRPO with turn-level credit assignment, we train Muse-Glimmer on 165 environments, increasing coverage on 22 held-out environments from 17.17% to 33.36% at low reasoning effort and from 30.19% to 47.02% at extra-high effort. The trained model at low effort also exceeds the base model's extra-high-effort coverage while using 47.9% fewer tokens. We also observe evidence of generalization: held-out forecasting accuracy at high effort improves by 10.06 percentage points, while out-of-distribution evaluation scores improve by 3.94 points on GDPval, 2.29 on GDP.pdf, 7.07 on Harvey LAB, and 4.96 on Frontier Finance under their respective metrics. Code: https://github.com/Trata-Inc/trata-hedge-bench-verified.
查看此版本解读 →查看此版本原文 ↗