UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval
Libiao Chen, Xiyang Liu, Yanheng Wei, Zhenyu Tang
原文摘要
Modern Web information access increasingly involves heterogeneous multimodal content, including images, videos and visually rich documents. Users may seek not only globally similar content but also candidates that satisfy fine-grained attributes, relations or answer evidence under complex instructions. Supporting such access requires both efficient corpus-scale retrieval and fine-grained relevance reasoning. Multimodal large language model (MLLM)-based embedding methods derive representations from MLLM hidden states, and recent work further uses Chain-of-Thought (CoT) reasoning to improve embedding capability by injecting intermediate semantic evidence into the representation. However, existing CoT methods typically use item-wise reasoning over queries and candidates in isolation, providing no explicit evidence to distinguish a positive from a semantically confusable hard negative. Moreover, contrastive embeddings capture global similarity but struggle with meta-tasks requiring answer verification, category judgment or fine-grained reasoning. In this paper, we propose UMER, a Unified Multimodal Embedding and Ranking framework for universal multimodal retrieval. UMER replaces item-wise reflection with PairAware Discriminative Reasoning, which compares query-candidate pairs to identify instruction-relevant matching and discrepancy evidence. UMER jointly learns contrastive embeddings for efficient global matching and discriminative ranking for explicit pairwise relevance judgment within a single MLLM. A complementary mutual distillation strategy further transfers reliable pairwise preferences between the embedding and ranking functions. On MMEB-V2, UMER-R scores 65.5/70.6 with Qwen2-VL-2B/7B, outperforming the strongest included matched-backbone baselines by 1.4/2.0 points. UMER also supports budget-adjustable inference from efficient embedding retrieval to optional pair-aware reranking.
cs.AI