同样的词，不同的判断：偏好如何随模态变化
Same Words, Different Judgments: How Preferences Vary Across Modalities
Aaron Broukhim, Nadir Weibel, Eshin Jolly
AI 速览：作者报告了一项基于ICC的受控跨模态研究，比较同一语义内容在文本与音频两种模态下的人类及合成偏好标注，发现两种模态的偏好判断存在显著差异。
Preference-based reinforcement learning (PbRL) is the dominant framework for aligning AI systems to human preferences. However, evaluation protocols for such data were designed for text and have not been validated for speech. We present the first intraclass correlation coefficient (ICC)-based, controlled cross-modal study of human and synthetic preference annotations, comparing text and audio evaluations of identical semantic content across 100 prompts. We show that achieving \textit{good} (audio) to \textit{near good} (text) agreement within either modality (ICC(2,$k$) $\approx$ .81 audio, .74 text) requires $\sim$9 raters and modalities show marked differences in how people report preferences: audio raters exhibit narrower decision thresholds, reduced length bias, and more user-oriented evaluation criteria, with near-chance cross-modality agreement. We demonstrate that synthetic ratings can be used to effectively predict inter-rater agreement, thus serving as an early signal for stimulus selection and proxy for human annotations. Together, these findings argue that evaluation protocols for audio preference data require modality-specific design rather than direct adaptation from text. Code, data, and the audio corpus are available at: https://huggingface.co/datasets/sciminds/same-words-different-judgements