跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2609.22226 · v2

Swiss-Knife: A Framework for Reconfigurable Externalised Multi-Objective Alignment at Decode Time

首次提交 2026-09-03 09:31
版本更新 2026-11-18 05:43
雷达观测 2026-11-19 19:08
arXiv ↗PDF ↗

摘要

Decode-time alignment methods steer a frozen language model by scoring candidate continuations with an external reward and selecting the maximiser. We argue that this shared design is a single degenerate point in a much larger space. We introduce Swiss-Knife, a framework for externalised multi-objective alignment in which the alignment specification is a first-class runtime object: hot-swappable scoring blades, a batch normaliser, a pairwise aggregation operator, and a selection rule. Six axioms characterise the admissible aggregation operators, and we prove a representation theorem: every operator satisfying them has the form Ri=∑j≠ig((μi−μj)/s(σi,σj))R_i = \sum_{j \neq i} g((μ_i - μ_j)/s(σ_i,σ_j)), a two-parameter family containing probit and logistic comparison rules and pointwise argmax as named coordinates. Within it, pairwise aggregation is Lipschitz-stable under adversarial reward contamination while argmax is not, and Candidate-Batch Normalization (CBN) makes the weight simplex invariant to the rescalings under which reward models are only ever identified. Our reference instantiation pairs DPO-LoRA blades with an uncertainty-aware pairwise tournament. Sweeping the helpfulness/honesty/harmlessness simplex, it attains the best balanced frontier of six decode-time methods (harmonic F1F_1 0.797 vs. 0.750 for the strongest baseline, p<10−14p < 10^{-14}) with the lowest refusal rate and highest helpfulness of any arm, and reconfigures its objectives in 0.05 ms with no gradient computation. Ablating CBN costs 0.217 F1F_1, and the metadata confirms the predicted mechanism: the lowest-variance blade retains 9% of its nominal 33% influence, and the collapse follows the predicted ordering.

作者备注:19 pages, 3 figures, includes technical appendix and proofs

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-18 05:43 · 最新观测版本

Swiss-Knife: A Framework for Reconfigurable Externalised Multi-Objective Alignment at Decode Time

Decode-time alignment methods steer a frozen language model by scoring candidate continuations with an external reward and selecting the maximiser. We argue that this shared design is a single degenerate point in a much larger space. We introduce Swiss-Knife, a framework for externalised multi-objective alignment in which the alignment specification is a first-class runtime object: hot-swappable scoring blades, a batch normaliser, a pairwise aggregation operator, and a selection rule. Six axioms characterise the admissible aggregation operators, and we prove a representation theorem: every operator satisfying them has the form Ri=∑j≠ig((μi−μj)/s(σi,σj))R_i = \sum_{j \neq i} g((μ_i - μ_j)/s(σ_i,σ_j)), a two-parameter family containing probit and logistic comparison rules and pointwise argmax as named coordinates. Within it, pairwise aggregation is Lipschitz-stable under adversarial reward contamination while argmax is not, and Candidate-Batch Normalization (CBN) makes the weight simplex invariant to the rescalings under which reward models are only ever identified. Our reference instantiation pairs DPO-LoRA blades with an uncertainty-aware pairwise tournament. Sweeping the helpfulness/honesty/harmlessness simplex, it attains the best balanced frontier of six decode-time methods (harmonic F1F_1 0.797 vs. 0.750 for the strongest baseline, p<10−14p < 10^{-14}) with the lowest refusal rate and highest helpfulness of any arm, and reconfigures its objectives in 0.05 ms with no gradient computation. Ablating CBN costs 0.217 F1F_1, and the metadata confirms the predicted mechanism: the lowest-variance blade retains 9% of its nominal 33% influence, and the collapse follows the predicted ordering.

查看此版本解读 →查看此版本原文 ↗