StreamVibe: Proactive Video Agents for Evolving User Intent

摘要变化

Vision-language models are shifting from offline passive understanding toward online proactive interaction. However, prevailing methods largely rely on explicit requests while miss theoverlooking implicit user intentintents (e.g., "where should I click next?" whilewhen a user navigateis navigating a website), meanwhileand fail to updatecontinuously theupdate statetheir simultaneouslyunderstanding as user requests might updateintents continuouslyevolve. These limitations make them less suitedsuitable tofor intent-aware interaction in practicalreal-world scenarios. To address these challenges, we introduce StreamVibe-Bench, a new benchmark servingthat serves as a testbed for user intent understanding and interaction, withcomprising 4.1K evaluation samples and 85K training samples spanning explicit, implicit, and dynamic intents. Building on this benchmark, we develop StreamVibe-VL, a streaming proactive model that infers implicit user intents from continuous observations and dynamically tailorsadapts its interactions as these intents evolve. We further introduce StreamVibe-Harness, which maintains user intents as persistent states with a fleixbleflexible lifecycles via athrough hierarchical memory, enabling it to track the latest user intentintents and environmental statestates. Our experiments reveal three key findings: (i) StreamVibe-VL preserves strong general video understanding capabilities while improving response timing and answer quality by 10.7% over otherexisting baselines; (ii) existing proactive models remain limited in understanding user intentintents and decidingdetermining when to respond, especially under implicit and dynamic intents; and (iii) StreamVibe-Harness further improves response consistency and adaptation overduring sustained interactioninteractions, resulting in a 12.3% gain in proactive interaction performance.