Temporal Grounding of Video Large Language Model via Holistic Event Perception and Matching
Jiahao Nie, Wenbin An, Gongjie Zhang, Yicheng Xu, Yap-Peng Tan, Alex C. Kot, Shijian Lu
Despite recent advances in Video Large Language Models (Vid-LLMs), Temporal Video Grounding (TVG) targeting precise localization of time segments of query events remains a grand challenge. Recent studies attempt to address this challenge by matching two separate tokens to all frame features and identifying the start and end frames of events based on the highest matching similarities. However, this approach relies heavily on the exact start and end timestamps, which are often ambiguous without capturing the semantic continuity and integrity of the event. To this end, we design E.M.Ground, a temporal grounding Vid-LLM that tackles the TVG challenge from a holistic and coherent event perception perspective. E.M.Ground introduces three novel designs: (i) It introduces a special <evt> token that aggregates information from all frames of query events with varying lengths, thereby preserving semantic continuity and integrity for accurate event matching. (ii) It introduces Savitzky-Golay smoothing to suppress the noise and abrupt changes in token-to-frame similarities across adjacent timestamps, thereby improving prediction accuracy. (iii) It aggregates multi-grained frame features to enhance the reliability of token-to-frame matching and improve temporal understanding, thereby compensating for the compression-induced information loss. Extensive experiments on widely adopted TVG benchmarks, along with in-depth analyses, demonstrate that E.M.Ground achieves strong overall performance, with the most pronounced improvements on single-event grounding and boundary-integrity tasks.
尚无 AI 解读，展示原文摘要