← 研踪 PaperTrailarXiv:2510.15786 · v3
DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation
首次提交 2025-10-18 00:08
版本更新 2026-11-03 13:24
雷达观测 2026-11-04 23:40
摘要
A dexterous hand succeeds or fails through contact, yet every scalable record of human manipulation (video, vision-based hand tracking, motion capture) observes how the hand moves and misses how it pushes. We present DexCanvas, a hybrid real-synthetic dataset of 7,000 hours of dexterous hand-object interaction with per-frame contact points and contact forces, grown from 70 hours of optical motion capture that spans 21 manipulation types, from power and precision grasps to in-hand manipulation. The forces come from a real-to-sim pipeline: one reinforcement learning (RL) policy, conditioned on object geometry and manipulation type, drives an actuated MANO hand in physics simulation to reproduce each captured object trajectory, and the simulator measures the contacts that produce the reproduced motion. The policy reproduces 87.5% of the demonstrations and still succeeds on 80.9% of rollouts from perturbed initial object poses; these perturbed rollouts expand the 70 captured hours a hundredfold. On six YCB objects absent from the capture set it succeeds 72.5% of the time. The contact channels make better policies: a Diffusion Policy that observes DexCanvas contact features succeeds 82.2% of the time, against 67.5% without them, over 18,522 trials. Contact-driven retargeting carries the same demonstrations to two dexterous robot hands, where the RL stage again produces stable rollouts. A first release of 111,857 force-annotated trajectories is publicly available on Hugging Face.
作者备注：22 pages, 9 figures. DexGEM Lab technical report. Dataset: https://huggingface.co/datasets/Dex-GEM/DexCanvas
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-03 13:24 · 最新观测版本
DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation
A dexterous hand succeeds or fails through contact, yet every scalable record of human manipulation (video, vision-based hand tracking, motion capture) observes how the hand moves and misses how it pushes. We present DexCanvas, a hybrid real-synthetic dataset of 7,000 hours of dexterous hand-object interaction with per-frame contact points and contact forces, grown from 70 hours of optical motion capture that spans 21 manipulation types, from power and precision grasps to in-hand manipulation. The forces come from a real-to-sim pipeline: one reinforcement learning (RL) policy, conditioned on object geometry and manipulation type, drives an actuated MANO hand in physics simulation to reproduce each captured object trajectory, and the simulator measures the contacts that produce the reproduced motion. The policy reproduces 87.5% of the demonstrations and still succeeds on 80.9% of rollouts from perturbed initial object poses; these perturbed rollouts expand the 70 captured hours a hundredfold. On six YCB objects absent from the capture set it succeeds 72.5% of the time. The contact channels make better policies: a Diffusion Policy that observes DexCanvas contact features succeeds 82.2% of the time, against 67.5% without them, over 18,522 trials. Contact-driven retargeting carries the same demonstrations to two dexterous robot hands, where the RL stage again produces stable rollouts. A first release of 111,857 force-annotated trajectories is publicly available on Hugging Face.
查看此版本解读 →查看此版本原文 ↗