MMCLIP: Cross-Modal Attention Masked Modelling for Medical Language-Image Pre-Training
Biao Wu, Yutong Xie Mohamed bin Zayed University of Artificial Intelligence, Zeyu Zhang Australian National University, Vu Minh Hieu Phan, Qi Chen Adelaide University, Ling Chen University of Technology Sydney, Qi Wu The University of Adelaide
Vision-and-language pretraining (VLP) in medicine leverages contrastive learning on image–text pairs, often enhanced with masked modeling. However, existing methods face two challenges: difficulty reconstructing key pathological features due to limited data, and reliance on either paired or image-only datasets without combining both. To address this, we propose MMCLIP (Masked Medical Contrastive Language–Image Pre-training), which introduces two modules: AttMIM: Masks image features highly correlated with text to improve reconstruction of fine medical details. EntMLM: Masks key medical entities in text and reconstructs them using visual cues. Furthermore, MMCLIP incorporates unpaired data through disease-kind prompts, achieving state-of-the-art performance in zero-shot and fine-tuning across five benchmarks.
尚无 AI 解读，展示原文摘要
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2026 · 正式发表