TEMPO:面向视觉-语言-动作模型的语义-动作解耦强化学习后训练
TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models
Ziheng Liu, Quantao Yang
AI 速读 基于摘要
TEMPO 提出一种语义与动作解耦的双时间尺度强化学习后训练框架,冻结预训练视觉-语言主干,分别以不同频率更新语义投影层和动作专家,以缓解潜在动作漂移并提升策略性能。
cs.RO · cs.CV
查看原文摘要
Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning(RL) post-training. SFT is prone to distribution mismatch, and existing RL approaches typically optimize all trainable components within a single RL loop at a uniform update rate, although these components play distinct functional roles. We propose TEMPO, a semantic-action decoupled, two-timescale RL post-training framework for VLA models. TEMPO freezes the pretrained vision-language backbone and adapts two components with dedicated RL loops: the semantic projection layer, which produces the latent action conditioning the action expert, and the low-level action expert itself. The semantic projection layer is updated infrequently and the action expert frequently, which aims to limit latent-action drift while allowing the action expert to learn from online feedback. Our analyses show that, compared with equal update frequencies, less frequent semantic updates reduce latent-action drift and improve policy performance. Experiments on the CALVIN benchmark and real-world manipulation tasks demonstrate that TEMPO outperforms both pretrained state-of-the-art VLA models and the RL post-training baseline, while reaching and maintaining higher evaluation rewards on three real-world tasks.