超越模态偏好：在多模态大模型模态冲突下控制模态仲裁
Beyond Modality Preference: Controlling Multimodal Arbitration under Modality Conflict in MLLMs
Zhaoying Pan, Shenyu Lu, Jun Hui Wong, Junyi Chai, Xiaoqian Wang
AI 速读 基于摘要
作者识别出少数对视觉与文本证据取舍有因果影响的“仲裁注意力头”，并提出一种免训练的推理时干预方法，用于按任务指定的可信模态偏置模型预测。
cs.CV
查看原文摘要
Multimodal large language models (MLLMs) integrate information across modalities, but become unreliable when those modalities disagree. Under conflicting inputs, MLLMs may favor modalities that are not the most reliable for the task, thus motivating inference-time control of modality arbitration. We identify a small set of attention heads, termed arbitration heads, that causally influence whether predictions follow visual or textual evidence. Building on this, we introduce a lightweight, training-free intervention that biases these heads toward a designated trusted modality at inference time. The identified heads are stable across calibration set sizes and generalize across settings, and steering them enables effective bidirectional control of modality reliance across diverse benchmarks. Our findings identify an effective control mechanism to align MLLM predictions with task-specified modality priorities. The code is available at https://github.com/zhaoyingpan/modality-arbitration.