Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations
Runyi Yu, Xiaoyi Lin, Hok Wai Tsui, Yinhuai Wang, Zhijie Peng, Hui Zhang, Qihan Zhao, Ke Fan 等 13 位作者
原文摘要
We present a system for learning generalizable hand-object tracking controllers purely from synthetic data, without requiring any human demonstrations. Our approach makes two key contributions: (1) HOG, a highly efficient Hand-Object data generator that rapidly synthesizes diverse, large-scale, and uniformly distributed trajectories; and (2) HOT, a Hand-Object Tracker that bridges synthetic-to-physical transfer through reinforcement learning and interaction imitation learning. Our method extends to diverse object shapes and hand morphologies. HOT also enables cross-simulator transfer. Through extensive evaluations, we show that our approach enables dexterous hands to track challenging, long-horizon sequences including object re-arrangement and agile in-hand reorientation. These results represent a significant step toward scalable foundation controllers for manipulation that can learn entirely from simple, scalable synthetic data, breaking the data bottleneck that has long constrained progress in dexterous manipulation.
cs.RO · cs.GR