跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2512.18436 · v3

VeruSAGE-Bench: A Benchmark Suite for Rust System Verification

首次提交 2025-12-21 01:22
版本更新 2026-11-15 06:57
雷达观测 2026-11-17 12:06
arXiv ↗PDF ↗

摘要

Large language models (LLMs) have shown impressive capability to understand and develop code. However, their capability to rigorously reason about and formally prove the correctness of software systems remains in question. Current benchmarks for formal proof generation focus on isolated tasks, typically algorithmic-style problems utilizing basic data structures. We curate a new benchmark suite for system-level proof generation, VeruSAGE-Bench, which consists of 849 proof tasks extracted from eight open-source Verus-verified Rust systems, covering operating systems, storage systems, memory allocators, and more. These tasks are much more complicated than those in previous benchmarks, containing on average more than 20x the formal specification (in Lines of Code) and requiring a much broader array of verification techniques. We evaluate six LLMs on VeruSAGE-Bench (o4-mini, GPT-5, Sonnet 4, Sonnet 4.5, Opus 4.5, Qwen3.5), and demonstrate that different models and agentic setups lead to vastly different success rates and costs. We also show that the best models possess impressive capability in writing correctness proofs and hence should inspire new ways of developing trustworthy software. Meanwhile, some difficult tasks in VeruSAGE-Bench stress even the best model, forcing long reasoning time (over an hour on the hardest tasks, 15.3 min per task on average), high token cost (USD 13.65 per task on average), and proof failures, which we hope will guide future improvements in models and agents. This is particularly true for a harder no-lemma version of VeruSAGE-Bench, where the helper lemmas written by human experts are removed: there, the best model proves only 41% of the tasks from the hardest project, at USD 96 per task.

作者备注:NeurIPS 2026 Evaluations & Datasets Track

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v3,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v3 · 2026-11-15 06:57 · 最新观测版本

VeruSAGE-Bench: A Benchmark Suite for Rust System Verification

Large language models (LLMs) have shown impressive capability to understand and develop code. However, their capability to rigorously reason about and formally prove the correctness of software systems remains in question. Current benchmarks for formal proof generation focus on isolated tasks, typically algorithmic-style problems utilizing basic data structures. We curate a new benchmark suite for system-level proof generation, VeruSAGE-Bench, which consists of 849 proof tasks extracted from eight open-source Verus-verified Rust systems, covering operating systems, storage systems, memory allocators, and more. These tasks are much more complicated than those in previous benchmarks, containing on average more than 20x the formal specification (in Lines of Code) and requiring a much broader array of verification techniques. We evaluate six LLMs on VeruSAGE-Bench (o4-mini, GPT-5, Sonnet 4, Sonnet 4.5, Opus 4.5, Qwen3.5), and demonstrate that different models and agentic setups lead to vastly different success rates and costs. We also show that the best models possess impressive capability in writing correctness proofs and hence should inspire new ways of developing trustworthy software. Meanwhile, some difficult tasks in VeruSAGE-Bench stress even the best model, forcing long reasoning time (over an hour on the hardest tasks, 15.3 min per task on average), high token cost (USD 13.65 per task on average), and proof failures, which we hope will guide future improvements in models and agents. This is particularly true for a harder no-lemma version of VeruSAGE-Bench, where the helper lemmas written by human experts are removed: there, the best model proves only 41% of the tasks from the hardest project, at USD 96 per task.

查看此版本解读 →查看此版本原文 ↗