SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation
Zhou Liu, Chenxu Zhang, Ligang Huang, Zeli Su, Zewei Pan, Changhang Tian, Zhaoyang Han, Xing Chen 等 10 位作者
Graphical user interface (GUI) agents can reuse past interaction experience. We study how illustrated procedures support action prediction and students deployed without retrieval. We introduce SkillLens, which represents procedures, applicability cues, and interface views in a shared Visual Skill Card (VSC) schema. Retrieved cards provide a frozen executor with reference evidence alongside the current screen. CardDistill uses this schema to construct privileged teacher context from labeled training steps for on-policy distillation; the student learns from native inputs and requires no retrieval at deployment. With matched training data and update counts, students trained with full-card teachers improve step success rate by an average of 5.04 percentage points over those trained with compact-text teachers across three training seeds on 119 tasks from ten websites absent from training. An independent runtime control uses training-source memory and pre-action retrieval queries: full cards achieve 12/64 correct actions (18.75%) versus 4/64 (6.25%) for text-only cards. These results establish the value of the complete VSC teacher-context package for retrieval-free students and provide evidence for pre-action retrieval of illustrated procedures.
尚无 AI 解读，展示原文摘要