EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation
Ziheng Xi, Jiayi Yu, Yanbo Duan, Yitao Wang, Jinyang Yu, Zhanwei Xu, Jianjiang Feng, Jie Zhou
原文摘要
Hand pose estimation from egocentric video is challenging under adverse visual conditions, such as occlusion. Wrist-worn surface electromyography (sEMG) measures muscle activity without requiring visual access to the hand, offering a wearable complement to egocentric cameras. Evaluating their combined use requires synchronized physiological and visual recordings with continuous pose annotations, which existing EMG-based hand pose datasets do not jointly provide. We introduce EgoEMG, a multimodal dataset pairing bilateral wrist sEMG with egocentric video and motion-capture-derived hand poses, supplemented by external multi-view RGB-D recordings. EgoEMG contains 25.9 hours of recordings from 68 participants, covering 60 bare-hand gesture classes and interactions with 10 everyday objects. We establish EMG-only, vision-only, and fusion benchmarks with shared 22-dimensional pose targets and evaluations on unseen users, gestures, and objects. For EMG-only estimation, EMGFormer achieves the lowest overall MAE among the compared methods on EgoEMG and competitive results on EMG2Pose. Our residual fusion model combines ViT and EMGFormer features. Adding EMG reduces error for both single-frame and temporal vision models, with MAE decreasing from $7.39^\circ$ to $7.06^\circ$ for ViT-L. The largest absolute reduction occurs when evaluating interactions with an unseen object. Generalization to unseen users remains challenging. Project page: https://zhenqis123.github.io/egoemg-site/.
cs.CV