arXiv:2603.02865 · v2
Signals Are Present, Predictions Fail: Probing Diagram Representations in Large Vision-Language Models
摘要
Large vision-language models (LVLMs) demonstrate strong performance on diagram understanding benchmarks, yet they still struggle to understand relationships between elements, particularly those represented by directed edges between nodes. To investigate the causes of this limitation, we probe the internal representations of LVLMs using a synthetic diagram dataset based on directed graphs. Our linear probing experiments reveal that signals about nodes and edges are linearly decodable from the hidden states of LVLMs, even for aspects that LVLMs struggle with in VQA, such as edge direction and connectivity between nodes. These results reveal a gap between internal signals and model outputs. We further perform causal interventions on the model's hidden states, showing that, for some aspects, the signals identified through probing causally contribute to the model's final predictions. These findings also hold on flowcharts, not only on synthetic diagrams.
作者备注：Accepted to EMNLP 2026 Main; 38 pages, 58 figures, 18 tables
期刊信息：Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (2026), pp. 61160-61197