Déjà Cue: Localizing States in Object Histories via Vocabulary-Relative Coordinates
Haofan Cao, Zhichao You, Yunkai Yang, Liang Guo, Jie Wang, Chongshou Li
原文摘要
Object tracking preserves identity through visual changes, but cannot tell when an object is whole, cut, or peeled. We study state-moment retrieval: locating when a tracked object exhibits each state in a supplied set of descriptions. However, absolute image--text similarity can favor shared object semantics, obscuring the distinctions needed to localize individual states. Our insight is that the alternatives themselves provide a reference for measuring state evidence. We introduce Déjà Cue, a training-free framework that centers description embeddings on a state-balanced vocabulary mean, producing query directions that compare each state against its alternatives. Using frozen vision--language encoders, Déjà Cue calibrates frame scores and retrieves intervals at multiple durations within contiguous visible runs. We construct MS-VOST by annotating 620 object histories with 2,465 states. With the encoder and temporal search fixed, Déjà Cue improves all three localization metrics over absolute scoring across eight encoder configurations, raising Top-1 temporal IoU by 0.42--1.14 percentage points. These gains persist across four text constructions, and candidate-ranking analysis shows that the vocabulary reference brings relevant intervals forward within the same search space. These results establish the supplied state vocabulary as an object-specific reference for localizing semantic states from frozen visual representations.
cs.CV · cs.LG · cs.MM