EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras
Luming Wang, Hao Shi, Jiajun Zhai, Kailun Yang, Kaiwei Wang
原文摘要
Egocentric 3D hand pose estimation and gesture recognition are fundamental to understanding hand interactions from first-person observations. Robust perception remains challenging: frame-based cameras suffer from motion blur and limited dynamic range, while event-based approaches face ego-motion interference, monocular depth ambiguity, and limited real-world datasets. We propose EgoEV-HandPose, an end-to-end framework for joint 3D bimanual hand pose estimation and gesture recognition from stereo event streams. Its core module, KeypointBEV, combines sparse 3D joint hypotheses with geometry-aware stereo fusion and iterative reprojection-guided refinement to progressively resolve depth uncertainty and improve pose accuracy. We also introduce EgoEVHands, the first large-scale real-world stereo event dataset for egocentric 3D bimanual hand pose estimation and gesture recognition. Built upon EgoEvGesture recordings, it adds dense 2D/3D hand annotations and comprises 2,498 annotated stereo sequence pairs across 38 gesture classes. Extensive experiments demonstrate that EgoEV-HandPose achieves state-of-the-art performance on EgoEVHands, obtaining a mean per-joint position error (MPJPE) of 30.54mm and a gesture recognition accuracy of 86.87%, significantly outperforming RGB-based stereo and previous event-based methods. The framework exhibits exceptional robustness under low illumination and severe bimanual occlusion. Experiments on DHP19 further demonstrate the cross-dataset and cross-architecture generalizability of the KeypointBEV refinement design. Dataset and code will be released at https://github.com/ZJUWang01/EgoEV-HandPose.
cs.CV · cs.RO · eess.IV