← 研踪 PaperTrailarXiv:2606.19882 · v2
Multimodal Concept Bottleneck Models
首次提交 2026-06-18 15:41
版本更新 2026-11-09 08:29
雷达观测 2026-11-10 13:18
摘要
Concept Bottleneck Models (CBMs) enhance the interpretability of deep neural networks by aligning the features extracted from images with human-interpretable concepts. However, conventional CBMs typically use a linear classifier that maps concept activations to a fixed set of predefined classes, limiting their flexibility in open-vocabulary settings. In this paper, we propose Multimodal Concept Bottleneck Model (MM-CBM) to address these issues and extend CBMs into CLIP structure. MM-CBM utilizes dual Concept Bottleneck Layers (CBLs) to align both the image and text embeddings into interpretable features. This allows us to perform new vision tasks like zero-shot classification or image retrieval in an interpretable way. Compared to existing methods, MM-CBM achieves 23.28% accuracy improvement on average across four standard benchmarks. Our method maintains competitive performance compared with black-box CLIP model while offering interpretability. Under classification tasks, MM-CBM's accuracy is on average 3.94% lower than CLIP in the target-dataset adaptation setting and 9.84% in the zero-shot transfer setting. Under image-text retrieval tasks, it achieves an average gap of 2.48% for image retrieval and 9.22% for text retrieval.
作者备注：Accepted to TMLR 2026
期刊信息：Transactions on Machine Learning Research, 10/2026
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-09 08:29 · 最新观测版本
Multimodal Concept Bottleneck Models
Concept Bottleneck Models (CBMs) enhance the interpretability of deep neural networks by aligning the features extracted from images with human-interpretable concepts. However, conventional CBMs typically use a linear classifier that maps concept activations to a fixed set of predefined classes, limiting their flexibility in open-vocabulary settings. In this paper, we propose Multimodal Concept Bottleneck Model (MM-CBM) to address these issues and extend CBMs into CLIP structure. MM-CBM utilizes dual Concept Bottleneck Layers (CBLs) to align both the image and text embeddings into interpretable features. This allows us to perform new vision tasks like zero-shot classification or image retrieval in an interpretable way. Compared to existing methods, MM-CBM achieves 23.28% accuracy improvement on average across four standard benchmarks. Our method maintains competitive performance compared with black-box CLIP model while offering interpretability. Under classification tasks, MM-CBM's accuracy is on average 3.94% lower than CLIP in the target-dataset adaptation setting and 9.84% in the zero-shot transfer setting. Under image-text retrieval tasks, it achieves an average gap of 2.48% for image retrieval and 9.22% for text retrieval.
查看此版本解读 →查看此版本原文 ↗