← 研踪 PaperTrailarXiv:2611.21372 · v1
Learning What Pairs Miss for Higher-Order Multimodal Alignment
首次提交 2026-11-17 00:36
版本更新 2026-11-17 00:36
雷达观测 2026-11-17 16:10
摘要
Contrastive learning beyond two modalities now scores the whole tuple with multilinear products, volumes, and fusion heads, critics that can represent the joint distribution. In practice these critics still behave like bags of pairs. Their negatives break at least two of the three pairs, a single pairwise check rejects them, and when most samples can be solved by pairs the higher-order term receives almost no gradient. To solve this problem, we propose CHORD (Contrastive Higher-Order Residuals with Decoys), which trains a residual against decoys drawn from a calibrated pairwise model, negatives that the pairwise model finds plausible, so that rejecting them often needs all three modalities. Drawing decoys costs one inner-product query over a memory bank, and the optimal residual is the log-ratio between the data and the pairwise model, so the decoy loss also bounds what that model misses. On M3-Mix, where one in ten training samples needs all three modalities, our CHORD improves accuracy on those samples by 21.2 percentage points over the strongest of eight baselines. CHORD also gives the best two-to-one results on every real dataset, and its residual improves other retrievers without retraining.
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-17 00:36 · 最新观测版本
Learning What Pairs Miss for Higher-Order Multimodal Alignment
Contrastive learning beyond two modalities now scores the whole tuple with multilinear products, volumes, and fusion heads, critics that can represent the joint distribution. In practice these critics still behave like bags of pairs. Their negatives break at least two of the three pairs, a single pairwise check rejects them, and when most samples can be solved by pairs the higher-order term receives almost no gradient. To solve this problem, we propose CHORD (Contrastive Higher-Order Residuals with Decoys), which trains a residual against decoys drawn from a calibrated pairwise model, negatives that the pairwise model finds plausible, so that rejecting them often needs all three modalities. Drawing decoys costs one inner-product query over a memory bank, and the optimal residual is the log-ratio between the data and the pairwise model, so the decoy loss also bounds what that model misses. On M3-Mix, where one in ten training samples needs all three modalities, our CHORD improves accuracy on those samples by 21.2 percentage points over the strongest of eight baselines. CHORD also gives the best two-to-one results on every real dataset, and its residual improves other retrievers without retraining.
查看此版本解读 →查看此版本原文 ↗