← 研踪 PaperTrailarXiv:2605.17204 · v2
Event-Grounded Sparse Autoencoders for Vision-Language-Action Policies
首次提交 2026-05-17 08:20
版本更新 2026-11-05 12:44
雷达观测 2026-11-06 11:06
摘要
Vision-Language-Action (VLA) policies translate language and visual inputs into robot actions, where their hidden representations directly shape closed-loop behavior. However, mechanistic interpretability tools from language and vision-language models do not transfer cleanly to VLAs: outputs are robot actions rather than human-readable tokens, and validating internal changes requires expensive closed-loop rollouts. We propose an event-grounded interpretability pipeline that anchors SAE feature analysis to behavioral events rather than text contexts. We extract keyframes from end-effector trajectories and cluster them within each task using visual, robot-state, and temporal cues. The resulting events guide SAE feature ranking, with optional VLM annotations providing descriptions of the events. To our knowledge, our pipeline is among the first to ground SAE-based VLA analysis in closed-loop behavioral events. Across two simulation architectures and a real-robot study, event-grounded ranking yields the strongest causal effects on OpenVLA and transfers to the continuous action chunks of $π_{0.5}$. SAE is a sparse but imperfect intervention basis: usability varies with architecture and intervention site, and aggressive intervention reveals safety and interpretability limits. Overall, event-grounded SAE analysis emerges as a practical starting point for behavior-anchored VLA interpretability, motivating future work on SAE analysis beyond end-effector motion, finer-grained closed-loop evaluation, and safe interventions for high-stakes VLA deployments.
作者备注：NeurIPS 2026 camera-ready version
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-05 12:44 · 最新观测版本
Event-Grounded Sparse Autoencoders for Vision-Language-Action Policies
Vision-Language-Action (VLA) policies translate language and visual inputs into robot actions, where their hidden representations directly shape closed-loop behavior. However, mechanistic interpretability tools from language and vision-language models do not transfer cleanly to VLAs: outputs are robot actions rather than human-readable tokens, and validating internal changes requires expensive closed-loop rollouts. We propose an event-grounded interpretability pipeline that anchors SAE feature analysis to behavioral events rather than text contexts. We extract keyframes from end-effector trajectories and cluster them within each task using visual, robot-state, and temporal cues. The resulting events guide SAE feature ranking, with optional VLM annotations providing descriptions of the events. To our knowledge, our pipeline is among the first to ground SAE-based VLA analysis in closed-loop behavioral events. Across two simulation architectures and a real-robot study, event-grounded ranking yields the strongest causal effects on OpenVLA and transfers to the continuous action chunks of $π_{0.5}$. SAE is a sparse but imperfect intervention basis: usability varies with architecture and intervention site, and aggressive intervention reveals safety and interpretability limits. Overall, event-grounded SAE analysis emerges as a practical starting point for behavior-anchored VLA interpretability, motivating future work on SAE analysis beyond end-effector motion, finer-grained closed-loop evaluation, and safe interventions for high-stakes VLA deployments.
查看此版本解读 →查看此版本原文 ↗