Zhi Li, Hau Phan, Matthew Emigh, Austin J. Brockmeier
Pretrained vision models encode semantic information in latent embedding spaces that are useful for downstream tasks. We hypothesize that this information can be disentangled to separate the content of complex scenes by decomposing each embedding into multiple concept-specific components. We propose Sparse Linear Concept Cones (SLiCC), a supervised dictionary learning approach that estimates a linear synthesis model consisting of sparse, non-negative combinations of groups of dictionary atoms, whose group-wise activity is guided by multi-label information. Each concept-specific component is a non-negative combination of atoms associated with a label and lies in a corresponding concept cone. We introduce an alternating optimization algorithm and an autoencoder formulation of SLiCC. We apply SLiCC to three types of CLIP and DINOv2 embeddings across five datasets. For CLIP, we additionally exploit image-text alignment to enable a self-supervised SLiCC. We also introduce a fully unsupervised autoencoder that enforces group sparsity based on the norm of neuron activations. Quantitative results show that supervised SLiCC improves concept-filtered image retrieval over unfiltered embeddings, including retrieval of finer-grained information not provided during training. Beyond retrieval of individual concepts, the disentangled components enable concept deletion and recombination across images. We further explore conditional image generation using Image-to-Prompt, with qualitative results illustrating retrieval and generation from manipulated concept representations.
Vision-language models are shifting from offline passive understanding toward online proactive interaction. However, prevailing methods largely rely on explicit requests while overlooking implicit user intents (e.g., "where should I click next?" when a user is navigating a website) and fail to continuously update their understanding as user intents evolve. These limitations make them less suitable for intent-aware interaction in real-world scenarios. To address these challenges, we introduce StreamVibe-Bench, a new benchmark that serves as a testbed for user intent understanding and interaction, comprising 4.1K evaluation samples and 85K training samples spanning explicit, implicit, and dynamic intents. Building on this benchmark, we develop StreamVibe-VL, a streaming proactive model that infers implicit user intents from continuous observations and dynamically adapts its interactions as these intents evolve. We further introduce StreamVibe-Harness, which maintains user intents as persistent states with flexible lifecycles through hierarchical memory, enabling it to track the latest user intents and environmental states. Our experiments reveal three key findings: (i) StreamVibe-VL preserves strong general video understanding capabilities while improving response timing and answer quality by 10.7% over existing baselines; (ii) existing proactive models remain limited in understanding user intents and determining when to respond, especially under implicit and dynamic intents; and (iii) StreamVibe-Harness further improves response consistency and adaptation during sustained interactions, resulting in a 12.3% gain in proactive interaction performance.
Qian Chen Alibaba Group, Xianyin Zhang Fudan University, Yanzhi Liu Alibaba Group, Lifan Guo, Feng Chen Alibaba Group, Chi Zhang Alibaba Group
The emergence of Large Vision-Language Models (LVLMs) has substantially expanded model capabilities beyond text-only understanding, enabling unified inference across both visual and textual modalities and supporting a broader range of real-world applications. To comprehensively evaluate the perception, understanding, reasoning, and cognition capabilities of LVLMs throughout the entire financial business workflow in Chinese contexts, we introduce CFMME, a novel Chinese financial multimodal evaluation benchmark. CFMME comprises 6,052 instances spanning from fundamental academic knowledge to complex real-world applications, covering eight primary financial image modalities and four core multimodal tasks. On CFMME, we conduct a thorough evaluation of representative LVLMs. The results show that the state-of-the-art model attains an overall accuracy of 66.11% on the question answering task and an average score of 77.18 on the detection, recognition, and information extraction tasks, indicating substantial room for improvement in current LVLMs. In addition, we conduct detailed analyses of error causes, cross-modal capabilities, and multi-orientation settings, yielding valuable insights for future research. We hope that CFMME will spur further progress in LVLMs, especially by improving their performance on multiple multimodal tasks in the financial domain.
尚无 AI 解读，展示原文摘要
cs.CL · cs.CV
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2026 · 正式发表