Mixture of Cognitive Experts in Large Vision-Language Models
Robert Wijaya, Ngai-Man Cheung
原文摘要
Large Vision Language Models (LVLMs) require strong reasoning over both visual and textual inputs. Recent work suggests that cognitive elements, especially diverse representations and metacognition, correlate with better performance. Many required perceptual functions are already provided by specialized domain-specific computer vision models, which can serve as the perceptual subsystem for detecting objects, localizing them, inferring states, recovering spatial layout, and reading text. The key challenge is integrating these multi-encoder experts into a trustworthy, interpretable, and coherent representation that improves verifiability and reduces hallucinations. This is difficult because vision-language questions span different cognitive levels, yet most LVLM pipelines apply the same perception-reasoning routing regardless of the query's demands. We propose an evidence-driven multimodal reasoning framework that uses a Bloom-inspired taxonomy as a hierarchical reasoning protocol. Our two-stage cognitive verbalization first produces a Literal Evidence Summary by decomposing expert outputs into short, atomic evidence statements, then performs Bloom Verbalization to convert these items into a staged reasoning trace. A lightweight Reasoning Trace Module quantitatively analyzes the trace, making evidence usage and reasoning progression explicit. Experiments show improvements in both perception and reasoning and diagnostics indicate that different queries induce different cognitive entry levels and evidence-use trajectories.
cs.CV