← 研踪 PaperTrailarXiv:2605.16834 · v2
ReLATE: Learning Relative Representations for Post-Hoc Multimodal Alignment
首次提交 2026-05-16 14:33
版本更新 2026-11-01 20:45
雷达观测 2026-11-04 23:43
摘要
Post-hoc multimodal alignment connects independently pretrained vision and language encoders while keeping them frozen, offering an alternative to joint pretraining when paired data are scarce. A central design challenge is to construct comparable representations across heterogeneous feature spaces while retaining local information relevant to distinct semantic concepts within each input. We propose Relative Alignment with Learnable Anchors and Token Evidence (ReLATE), which learns modality-specific anchors that jointly define relative representations and determine token contributions to each relative coordinate. Image patches and text tokens are represented by their similarities to anchors within their respective frozen feature spaces. Anchor-wise Similarity Pooling (ASP) then weights and aggregates these similarities separately for each anchor to construct the corresponding relative coordinate. A sample-level symmetric contrastive objective aligns the resulting image and text representations by optimizing only the anchors, without updating the encoders. Experiments on 13 general-domain benchmarks across four vision-language tasks demonstrate improvements over the compared post-hoc alignment methods under matched paired-data budgets, with further evaluations across data scales and encoder combinations. Beyond the general domain, we evaluate ReLATE in pathology as a specialized-domain application. By aligning frozen pathology image and biomedical language encoders, ReLATE matches or exceeds the evaluated pathology-specific vision-language models in balanced accuracy on three zero-shot classification benchmarks.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-01 20:45 · 最新观测版本
ReLATE: Learning Relative Representations for Post-Hoc Multimodal Alignment
Post-hoc multimodal alignment connects independently pretrained vision and language encoders while keeping them frozen, offering an alternative to joint pretraining when paired data are scarce. A central design challenge is to construct comparable representations across heterogeneous feature spaces while retaining local information relevant to distinct semantic concepts within each input. We propose Relative Alignment with Learnable Anchors and Token Evidence (ReLATE), which learns modality-specific anchors that jointly define relative representations and determine token contributions to each relative coordinate. Image patches and text tokens are represented by their similarities to anchors within their respective frozen feature spaces. Anchor-wise Similarity Pooling (ASP) then weights and aggregates these similarities separately for each anchor to construct the corresponding relative coordinate. A sample-level symmetric contrastive objective aligns the resulting image and text representations by optimizing only the anchors, without updating the encoders. Experiments on 13 general-domain benchmarks across four vision-language tasks demonstrate improvements over the compared post-hoc alignment methods under matched paired-data budgets, with further evaluations across data scales and encoder combinations. Beyond the general domain, we evaluate ReLATE in pathology as a specialized-domain application. By aligning frozen pathology image and biomedical language encoders, ReLATE matches or exceeds the evaluated pathology-specific vision-language models in balanced accuracy on three zero-shot classification benchmarks.
查看此版本解读 →查看此版本原文 ↗