AdaLoop:以视觉编码器进行思考
AdaLoop: Thinking with the Vision Encoder
Xinran Nicole Han, Yiyang Lu, Katelyn Gan, Todd Zickler
AI 速览:AdaLoop 提出一种按问题条件进行视觉循环计算的框架,让冻结的预训练视觉编码器在多次迭代中复用其变换,以适配不同问题所需的视觉处理深度。
Most vision-language models (VLMs) treat the vision encoder as a static feature extractor: it encodes an image once, independently of the question, and leaves subsequent reasoning to the language model. Yet different questions may require different visual operations and processing depths. We present AdaLoop, a framework for question-conditioned visual recurrence in VLMs. AdaLoop adapts visual computation to the question and reuses the encoder's transformations across multiple iterations, with the pretrained vision weights frozen. We also introduce VisRO, a procedural dataset and generation pipeline for training and evaluating vision-centric reasoning. On six VisRO tasks, AdaLoop outperforms standard fine-tuning and question-conditioned single-pass baselines with matched training data and optimizer updates. These gains hold across three VLM families on both in-distribution and out-of-distribution examples. Our findings suggest that pretrained vision encoders can contribute to reasoning through task-directed, iterative visual computation.