← 研踪 PaperTrailarXiv:2603.17450 · v3
VLM2Rec: Resolving Modality Collapse in Vision-Language Models for Sequential Recommendation
首次提交 2026-03-18 15:46
版本更新 2026-11-02 14:54
雷达观测 2026-11-04 23:42
摘要
Sequential Recommendation (SR) learns sequence-level Collaborative Filtering (CF) from interactions, but ID-based models struggle with sparse data and cold-start items. In contrast, item titles and images can reveal preference continuity, such as a recurring color or style. Yet existing approaches miss visual continuity: small encoders take a single item as input, and Large Language Models (LLMs) read only titles. We therefore shift our focus to Large Vision-Language Models (VLMs), which can read a user's entire image sequence, not just its titles. However, VLMs inherently suffer from modality collapse: they over-rely on text and, as a result, overlook visual continuity for recommendation. We find that standard contrastive Supervised Fine-Tuning (SFT), essential for adapting VLMs to SR, further amplifies this collapse: updates follow the text gradient, visual discrimination weakens, and adding image embeddings to the text embeddings consistently lowers accuracy. We propose VLM2Rec with two objectives: Weak-modality Penalized Contrastive Learning strengthens each user's weaker modality, and Cross-modal Relational Topology Regularization aligns each user's relative similarity structure over the same items across the two modality spaces. Its variant, $\text{VLM2Rec}_{\text{plug}}$, reuses the learned item embeddings in SR backbones and thus serves at the latency of conventional SR models. Experiments show that VLM2Rec mitigates the collapse and consistently improves accuracy and robustness.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-02 14:54 · 最新观测版本
VLM2Rec: Resolving Modality Collapse in Vision-Language Models for Sequential Recommendation
Sequential Recommendation (SR) learns sequence-level Collaborative Filtering (CF) from interactions, but ID-based models struggle with sparse data and cold-start items. In contrast, item titles and images can reveal preference continuity, such as a recurring color or style. Yet existing approaches miss visual continuity: small encoders take a single item as input, and Large Language Models (LLMs) read only titles. We therefore shift our focus to Large Vision-Language Models (VLMs), which can read a user's entire image sequence, not just its titles. However, VLMs inherently suffer from modality collapse: they over-rely on text and, as a result, overlook visual continuity for recommendation. We find that standard contrastive Supervised Fine-Tuning (SFT), essential for adapting VLMs to SR, further amplifies this collapse: updates follow the text gradient, visual discrimination weakens, and adding image embeddings to the text embeddings consistently lowers accuracy. We propose VLM2Rec with two objectives: Weak-modality Penalized Contrastive Learning strengthens each user's weaker modality, and Cross-modal Relational Topology Regularization aligns each user's relative similarity structure over the same items across the two modality spaces. Its variant, $\text{VLM2Rec}_{\text{plug}}$, reuses the learned item embeddings in SR backbones and thus serves at the latency of conventional SR models. Experiments show that VLM2Rec mitigates the collapse and consistently improves accuracy and robustness.
查看此版本解读 →查看此版本原文 ↗