跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2606.23641 · v2

Flatness Preserves Instruction Following in Vision-Language-Action Models

首次提交 2026-06-23 01:30
版本更新 2026-11-19 03:09
雷达观测 2026-11-21 12:18
arXiv ↗PDF ↗

摘要

Vision-language-action (VLA) models have the potential for open-world generalization by leveraging pretrained vision-language representations, yet downstream finetuning on limited robot data often degrades these representations. This leads to brittle action policies that ignore language instructions in favor of visual shortcuts, despite the base VLM having this knowledge, which is a failure mode we term instruction blindness. We hypothesize that standard finetuning with limited data applies gradients to a sparse set of points, which manifests as a sharp loss landscape with high-curvature minima. We analyze this effect, then show that regularization through explicit flatness-preserving optimization is a simple yet effective technique while finetuning on the exact same data. Specifically, we demonstrate that applying sharpness-aware minimization (SAM) during VLA finetuning significantly improves instruction following by over 55% across multiple simulation and real-world benchmarks without additional data, architectural modification, or pretraining. Explicitly preserving flatness better preserves pretrained knowledge compared to other regularization techniques that simply anchor model weights or constrain learning. We further analyze the effect of selective sharpness, quantify its effects, and show that such approaches are complementary to existing guidance techniques.

作者备注:This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-19 03:09 · 最新观测版本

Flatness Preserves Instruction Following in Vision-Language-Action Models

Vision-language-action (VLA) models have the potential for open-world generalization by leveraging pretrained vision-language representations, yet downstream finetuning on limited robot data often degrades these representations. This leads to brittle action policies that ignore language instructions in favor of visual shortcuts, despite the base VLM having this knowledge, which is a failure mode we term instruction blindness. We hypothesize that standard finetuning with limited data applies gradients to a sparse set of points, which manifests as a sharp loss landscape with high-curvature minima. We analyze this effect, then show that regularization through explicit flatness-preserving optimization is a simple yet effective technique while finetuning on the exact same data. Specifically, we demonstrate that applying sharpness-aware minimization (SAM) during VLA finetuning significantly improves instruction following by over 55% across multiple simulation and real-world benchmarks without additional data, architectural modification, or pretraining. Explicitly preserving flatness better preserves pretrained knowledge compared to other regularization techniques that simply anchor model weights or constrain learning. We further analyze the effect of selective sharpness, quantify its effects, and show that such approaches are complementary to existing guidance techniques.

查看此版本解读 →查看此版本原文 ↗