跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.20447 · v1

The Logic Chain Breaking Point: A Controlled-Variable Benchmark for Reasoning Degradation in Open-Weight Language Models

首次提交 2026-11-16 11:16
版本更新 2026-11-16 11:16
雷达观测 2026-11-17 13:07
arXiv ↗PDF ↗

摘要

We introduce The Logic Chain Breaking Point, a controlled-variable benchmark for deductive reasoning in open-weight language models. It spans three phases: (i) pure modus-ponens chain scaling, (ii) distractor resilience with a fixed core chain, and (iii) contradiction detection under primacy versus recency placement. Independent variables follow a powers-of-two sequence v∈{1,2,4,8,16,32,64,128}v \in \{1,2,4,8,16,32,64,128\}, with ground truth known by construction. We evaluate sixteen open-weight models across instruction-tuned and reasoning-tuned panels, including a small-model suite and a larger local-model extension. All models are evaluated with greedy decoding under model-appropriate context and output budgets, while context- and length-limited trials are quarantined from the valid accuracy denominator. Our results reveal a sharply non-monotonic difficulty surface: pure deductive chains up to L=128L=128 remain near saturation for most models, and distractor noise causes only model-specific degradation at high DD, while the inconsistent premise condition has the lowest accuracy. Instruction-tuned models, including larger 24B--27B baselines, score essentially zero on inconsistent premise sets, while reasoning-tuned models recover only partially and remain below reliable accuracy. Within this benchmark, inconsistent premise sets are associated with the strongest degradation under the benchmark's abstention-based scoring rule.

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-16 11:16 · 最新观测版本

The Logic Chain Breaking Point: A Controlled-Variable Benchmark for Reasoning Degradation in Open-Weight Language Models

We introduce The Logic Chain Breaking Point, a controlled-variable benchmark for deductive reasoning in open-weight language models. It spans three phases: (i) pure modus-ponens chain scaling, (ii) distractor resilience with a fixed core chain, and (iii) contradiction detection under primacy versus recency placement. Independent variables follow a powers-of-two sequence v∈{1,2,4,8,16,32,64,128}v \in \{1,2,4,8,16,32,64,128\}, with ground truth known by construction. We evaluate sixteen open-weight models across instruction-tuned and reasoning-tuned panels, including a small-model suite and a larger local-model extension. All models are evaluated with greedy decoding under model-appropriate context and output budgets, while context- and length-limited trials are quarantined from the valid accuracy denominator. Our results reveal a sharply non-monotonic difficulty surface: pure deductive chains up to L=128L=128 remain near saturation for most models, and distractor noise causes only model-specific degradation at high DD, while the inconsistent premise condition has the lowest accuracy. Instruction-tuned models, including larger 24B--27B baselines, score essentially zero on inconsistent premise sets, while reasoning-tuned models recover only partially and remain below reliable accuracy. Within this benchmark, inconsistent premise sets are associated with the strongest degradation under the benchmark's abstention-based scoring rule.

查看此版本解读 →查看此版本原文 ↗