← 研踪 PaperTrailarXiv:2605.29894 · v2
Train the Agent, Not the Expert: Learning to Harness Heterogeneous Experts for Multi-Turn Visual Reasoning
首次提交 2026-05-28 21:18
版本更新 2026-10-29 12:52
雷达观测 2026-10-30 11:25
摘要
Recent advances in computer vision have produced powerful specialized models for detection, segmentation, counting, and other visual tasks. However, these models are typically optimized for isolated tasks, limiting their ability to support general-purpose visual intelligence, particularly when complex language understanding and dense small-object perception are required. We propose VisHarness, a trainable visual agent that decouples high-level perception, reasoning, and decision-making from low-level task execution. Rather than training a task-specific model, we train VisHarness to harness a set of carefully designed heterogeneous visual experts. This paradigm preserves the agent's general intelligence while leveraging the precision of specialized visual experts for specific tasks. With only lightweight training, VisHarness learns a generalizable policy for harnessing visual experts, enabling it to solve fundamental vision tasks in diverse complex scenarios through multi-turn interactions with these experts. To enable efficient on-policy reinforcement learning in a live environment, we introduce dynamic visual memory archiving to reduce the visual-token overhead that rapidly accumulates during multi-turn interactions with visual experts. Experiments on four representative benchmarks spanning reasoning segmentation, generalized referring expression segmentation, dense small-object detection, and referring expression counting show that VisHarness substantially outperforms existing general-purpose models and is competitive with or superior to task-specific models. Our code, model weights, and training data are publicly available at https://github.com/fyw1999/VisHarness.
作者备注：This work has been accepted to NeurIPS 2026
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-10-29 12:52 · 最新观测版本
Train the Agent, Not the Expert: Learning to Harness Heterogeneous Experts for Multi-Turn Visual Reasoning
Recent advances in computer vision have produced powerful specialized models for detection, segmentation, counting, and other visual tasks. However, these models are typically optimized for isolated tasks, limiting their ability to support general-purpose visual intelligence, particularly when complex language understanding and dense small-object perception are required. We propose VisHarness, a trainable visual agent that decouples high-level perception, reasoning, and decision-making from low-level task execution. Rather than training a task-specific model, we train VisHarness to harness a set of carefully designed heterogeneous visual experts. This paradigm preserves the agent's general intelligence while leveraging the precision of specialized visual experts for specific tasks. With only lightweight training, VisHarness learns a generalizable policy for harnessing visual experts, enabling it to solve fundamental vision tasks in diverse complex scenarios through multi-turn interactions with these experts. To enable efficient on-policy reinforcement learning in a live environment, we introduce dynamic visual memory archiving to reduce the visual-token overhead that rapidly accumulates during multi-turn interactions with visual experts. Experiments on four representative benchmarks spanning reasoning segmentation, generalized referring expression segmentation, dense small-object detection, and referring expression counting show that VisHarness substantially outperforms existing general-purpose models and is competitive with or superior to task-specific models. Our code, model weights, and training data are publicly available at https://github.com/fyw1999/VisHarness.
查看此版本解读 →查看此版本原文 ↗