← 研踪 PaperTrailarXiv:2608.16951 · v2
The Little Scientist: What Does the Scientific Method Buy an LLM Agent?
首次提交 2026-08-16 09:32
版本更新 2026-11-04 04:23
雷达观测 2026-11-05 11:11
摘要
Scientific discovery emerges from cycles of hypothesis, implementation, empirical testing, and feedback. Can this process be automated? We approach automated algorithm design through the lens of the scientific method, where an LLM-based agent goes through each step of the process in an iterative fashion. Results: We present The Little Scientist, a framework that maps the scientific method onto an LLM agent in a loop: a Scientist agent iterates hypothesis, prediction, code, test, and reconciliation inside an evaluation environment that returns structured diagnostics, and a Kuhn agent injects paradigm-shifting reframings at plateaus. On DNA motif discovery, the process wrote an algorithm from scratch (DALE) that outperforms STREME across 132 ENCODE transcription factors on the shuffled-negative benchmark (mean AUROC 0.842 vs. 0.803) while running 11x faster. We then asked why this method worked, and that question became the paper's scientific core: a preregistered, controlled teardown of the framework's own scaffolding consisting of ten ablation arms against one-shot bandit nulls. The answer overturned the premise. Seven of ten loop arms fail to beat a bandit drawing samples from the same model and prompt, the full scaffold performs below blind sampling at 7-8x the token cost, and a floor-prompt arm with no framework at all wins on both score and value per token. On a held-out genomic benchmark, the best loop artifact and the best bandit draw agree to within 0.002, and the DALE algorithm from the initial experiment scored similarly. The convergent picture is that LLM agents do not reason their way past what training placed in the weights. Autoresearch, on this evidence, is sampling, selection, and remixing, which is the bet the Evolve-family architectures made when they chose populations and external selection over scratchpad reasoning, and why those choices were right.
作者备注：Code: https://github.com/travis42/little-scientist-dale and https://github.com/travis42/little-scientist-delta-v (Apache 2.0)
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-04 04:23 · 最新观测版本
The Little Scientist: What Does the Scientific Method Buy an LLM Agent?
Scientific discovery emerges from cycles of hypothesis, implementation, empirical testing, and feedback. Can this process be automated? We approach automated algorithm design through the lens of the scientific method, where an LLM-based agent goes through each step of the process in an iterative fashion. Results: We present The Little Scientist, a framework that maps the scientific method onto an LLM agent in a loop: a Scientist agent iterates hypothesis, prediction, code, test, and reconciliation inside an evaluation environment that returns structured diagnostics, and a Kuhn agent injects paradigm-shifting reframings at plateaus. On DNA motif discovery, the process wrote an algorithm from scratch (DALE) that outperforms STREME across 132 ENCODE transcription factors on the shuffled-negative benchmark (mean AUROC 0.842 vs. 0.803) while running 11x faster. We then asked why this method worked, and that question became the paper's scientific core: a preregistered, controlled teardown of the framework's own scaffolding consisting of ten ablation arms against one-shot bandit nulls. The answer overturned the premise. Seven of ten loop arms fail to beat a bandit drawing samples from the same model and prompt, the full scaffold performs below blind sampling at 7-8x the token cost, and a floor-prompt arm with no framework at all wins on both score and value per token. On a held-out genomic benchmark, the best loop artifact and the best bandit draw agree to within 0.002, and the DALE algorithm from the initial experiment scored similarly. The convergent picture is that LLM agents do not reason their way past what training placed in the weights. Autoresearch, on this evidence, is sampling, selection, and remixing, which is the bet the Evolve-family architectures made when they chose populations and external selection over scratchpad reasoning, and why those choices were right.
查看此版本解读 →查看此版本原文 ↗