Learning to Learn from Multimodal Experience
Xingyu Sui, Weixiang Zhao, Yongxin Tang, Yanyan Zhao, Yang Wu, Dandan Tu, Bing Qin
原文摘要
Multimodal agents increasingly interact with real-world-like environments and generate rich trajectories spanning perception, reasoning, and action. A key research focus is how to transform these raw multimodal trajectories into reusable experience that can improve future task performance. Existing experience-driven learning methods are mostly developed in textual settings and typically rely on manually designed memory schemas, making them less effective for multimodal experience that requires composing heterogeneous signals across modalities, temporal scales, and abstraction levels. We propose AutoMMemo, a general framework that enables agents to automatically design memory mechanisms for learning from multimodal experience. AutoMMemo represents each memory mechanism as an executable memo program, which defines how past episodes are stored, organized, and retrieved. It discovers effective memo programs through an iterative process of update-then-retrieve evaluation, reflection-guided mutation, and budget-aware tree search. Experiments across GUI/Web navigation and multimodal visual reasoning benchmarks show that AutoMMemo consistently improves agent performance over no-memory agents and fixed memory baselines, while maintaining reasonable token and interaction costs. Transfer and search-process analyses further demonstrate that the learned memory designs capture reusable experience structures and progressively refine retrieval relevance. These results highlight adaptive memory design as a key optimization target for multimodal experience-driven learning. Code is available at https://github.com/XingYuSSS/AutoMMemo.
cs.AI