跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2608.05180 · v3

The Nuclear Decision-Making Benchmark: Evaluating Frontier LLM Tendencies in Escalation, Proliferation, Non-Proliferation, and Arms Control Scenarios

首次提交 2026-06-29 09:45
版本更新 2026-11-19 01:05
雷达观测 2026-11-19 15:04
arXiv ↗PDF ↗

摘要

Large language models (LLMs) are entering defense and national-security workflows. Seven of the nine nuclear-armed states have committed to keeping humans in control of decisions to use nuclear weapons, but LLMs may still shape the assessments and options that reach decision-makers. Scrutinizing that influence calls for evidence about how these models behave on nuclear-policy questions. Such evidence should cover more than crisis escalation and should separate a model's tendencies from artifacts of a single phrasing, ordering, or run; evaluations to date have not focused on these questions. We introduce the Nuclear Decision-Making Benchmark (NDM Bench), a dataset and reproducible framework for testing how the choice of model and the design of a question shape LLMs' answers on nuclear policy. Its 151 scenarios, written by international relations experts, span four domains: nuclear escalation (76), arms control (25), non-proliferation (25), and proliferation (25). Each scenario can be posed with different states in the decision-maker's role and with added framing sentences. Eight frontier LLMs answered every prompt five times, about 380,000 responses in all. Results highlight that models' choices differ by model and domain, and that in escalation, how often they choose a nuclear threat or nuclear use depends on the state being advised and, for some models, the framing. However, agreement across identical runs is often low, with all five runs agreeing on only 43% of escalation prompts. We release the scenarios, the prompts with their answer orders, and the parsed responses, along with documentation of how they were built and recommended practices for evaluations where there is no correct answer to grade against.

作者备注:46 pages, 7 figures

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

仅对比实际观测的元数据,不代表 PDF 正文或实验变化。版本可能不连续;带删除线为移除,下划线为新增。

摘要变化

Large language models (LLMs) are entering defense and national-security workflows. Seven of the nine nuclear-armed states have committed to keeping humans in control of decisions to use nuclear weapons, but LLMs may still shape the assessments and options that reach decision-makers. Scrutinizing that influence calls for evidence about how these models behave on nuclear-policy questions. Such evidence should cover more than crisis escalation and should separate a model's tendencies from artifacts of a single phrasing, ordering, or run; evaluations to date have not focused on these questions. We introduce the Nuclear Decision-Making Benchmark (NDM Bench), a dataset and reproducible framework for testing how the choice of model and the design of a question shape LLMs' answers on nuclear policy. Its 151 scenarios, written by international relations experts, span four domains: nuclear escalation (76), arms control (25), non-proliferation (25), and proliferation (25). Each scenario can be posed with different states in the decision-maker's role and with added framing sentences. Eight frontier LLMs answered every prompt five times, about 380{,}000 responses in all. Results highlight that models' choices differ by model and domain, and that in escalation, how often they choose a nuclear threat or nuclear use depends on the state being advised and, for some models, the framing. However, agreement across identical runs is often low, with all five runs agreeing on only 43\% of escalation prompts. We release the scenarios, the prompts with their answer orders, and the parsed responses, along with documentation of how they were built and recommended practices for evaluations where there is no correct answer to grade against.

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v3 · 2026-11-19 01:05 · 最新观测版本

The Nuclear Decision-Making Benchmark: Evaluating Frontier LLM Tendencies in Escalation, Proliferation, Non-Proliferation, and Arms Control Scenarios

Large language models (LLMs) are entering defense and national-security workflows. Seven of the nine nuclear-armed states have committed to keeping humans in control of decisions to use nuclear weapons, but LLMs may still shape the assessments and options that reach decision-makers. Scrutinizing that influence calls for evidence about how these models behave on nuclear-policy questions. Such evidence should cover more than crisis escalation and should separate a model's tendencies from artifacts of a single phrasing, ordering, or run; evaluations to date have not focused on these questions. We introduce the Nuclear Decision-Making Benchmark (NDM Bench), a dataset and reproducible framework for testing how the choice of model and the design of a question shape LLMs' answers on nuclear policy. Its 151 scenarios, written by international relations experts, span four domains: nuclear escalation (76), arms control (25), non-proliferation (25), and proliferation (25). Each scenario can be posed with different states in the decision-maker's role and with added framing sentences. Eight frontier LLMs answered every prompt five times, about 380,000 responses in all. Results highlight that models' choices differ by model and domain, and that in escalation, how often they choose a nuclear threat or nuclear use depends on the state being advised and, for some models, the framing. However, agreement across identical runs is often low, with all five runs agreeing on only 43% of escalation prompts. We release the scenarios, the prompts with their answer orders, and the parsed responses, along with documentation of how they were built and recommended practices for evaluations where there is no correct answer to grade against.

查看此版本解读 →查看此版本原文 ↗
v2 · 2026-11-17 06:15

The Nuclear Decision-Making Benchmark: Evaluating Frontier LLM Tendencies in Escalation, Proliferation, Non-Proliferation, and Arms Control Scenarios

Large language models (LLMs) are entering defense and national-security workflows. Seven of the nine nuclear-armed states have committed to keeping humans in control of decisions to use nuclear weapons, but LLMs may still shape the assessments and options that reach decision-makers. Scrutinizing that influence calls for evidence about how these models behave on nuclear-policy questions. Such evidence should cover more than crisis escalation and should separate a model's tendencies from artifacts of a single phrasing, ordering, or run; evaluations to date have not focused on these questions. We introduce the Nuclear Decision-Making Benchmark (NDM Bench), a dataset and reproducible framework for testing how the choice of model and the design of a question shape LLMs' answers on nuclear policy. Its 151 scenarios, written by international relations experts, span four domains: nuclear escalation (76), arms control (25), non-proliferation (25), and proliferation (25). Each scenario can be posed with different states in the decision-maker's role and with added framing sentences. Eight frontier LLMs answered every prompt five times, about 380{,}000 responses in all. Results highlight that models' choices differ by model and domain, and that in escalation, how often they choose a nuclear threat or nuclear use depends on the state being advised and, for some models, the framing. However, agreement across identical runs is often low, with all five runs agreeing on only 43\% of escalation prompts. We release the scenarios, the prompts with their answer orders, and the parsed responses, along with documentation of how they were built and recommended practices for evaluations where there is no correct answer to grade against.

查看此版本解读 →查看此版本原文 ↗