Gen Li, Kaifeng Zhao, Siwei Zhang, Xiaozhong Lyu, Mihai Dusmanu, Yan Zhang, Marc Pollefeys, Siyu Tang
Understanding the world in first-person view is fundamental in Augmented Reality (AR). This immersive perspective brings dramatic visual changes and unique challenges compared to third-person views. Synthetic data has empowered third-person-view vision models, but its application to embodied egocentric perception tasks remains largely unexplored. A critical challenge lies in simulating natural human movements and behaviors that effectively steer the embodied cameras to capture a faithful egocentric representation of the 3D world. To address this challenge, we introduce EgoGen, a new synthetic data generator that can produce accurate and rich ground-truth training data for egocentric perception tasks. At the heart of EgoGen is a novel human motion synthesis model that directly leverages egocentric visual inputs of a virtual human to sense the 3D environment. Combined with collision-avoiding motion primitives and a two-stage reinforcement learning approach, our motion synthesis model offers a closed-loop solution where the embodied perception and movement of the virtual human are seamlessly coupled. Compared to previous works, our model eliminates the need for a pre-defined global path, and is directly applicable to dynamic environments. Combined with our easy-to-use and scalable data generation pipeline, we demonstrate EgoGen's efficacy in three tasks: mapping and localization for head-mounted cameras, egocentric camera tracking, and human mesh recovery from egocentric views. EgoGen will be fully open-sourced, offering a practical solution for creating realistic egocentric training data and aiming to serve as a useful tool for egocentric computer vision research. Refer to our project page: https://ego-gen.github.io/.
Gen Li, Yutong Chen, Yiqian Wu, Kaifeng Zhao, Marc Pollefeys, Siyu Tang
Understanding multimodal signals in egocentric vision, such as RGB video, depth, camera poses, and gaze, is essential for applications in augmented reality, robotics, and human-computer interaction, enabling systems to better interpret the camera wearer's actions, intentions, and surrounding environment. However, building large-scale egocentric multimodal and multitask models presents unique challenges. Egocentric data are inherently heterogeneous, with large variations in modality coverage across devices and settings. Generating pseudo-labels for missing modalities, such as gaze or head-mounted camera trajectories, is often infeasible, making standard supervised learning approaches difficult to scale. Furthermore, dynamic camera motion and the complex temporal and spatial structure of first-person video pose additional challenges for the direct application of existing multimodal foundation models. To address these challenges, we introduce a set of efficient temporal tokenizers and propose EgoM2P, a masked modeling framework that learns from temporally-aware multimodal tokens to train a large, general-purpose model for egocentric 4D understanding. This unified design supports multitasking across diverse egocentric perception and synthesis tasks, including gaze prediction, egocentric camera tracking, and monocular depth estimation from egocentric video, and also serves as a generative model for conditional egocentric video synthesis. Across these tasks, EgoM2P matches or outperforms specialist models while being an order of magnitude faster. We will fully open-source EgoM2P to support the community and advance egocentric vision research. Project page: https://egom2p.github.io/.
Time-series forecasting is foundational to a wide spectrum of web-scale services, spanning personalized recommendations, energy scheduling, and infrastructure capacity planning. Accurate long-term time series forecasting (LTSF) requires models to capture long-range temporal dependencies while adapting to periodic patterns that vary across variables. Although frequency-aware methods offer a global view of temporal dynamics, they typically rely on inflexible heuristic designs, such as predefined period lengths, manually selected or randomly sampled top-k frequency components, and separate spectral transformations. Consequently, they fail to account for time-frequency heterogeneity, where variables that appear synchronized in the time domain can differ substantially in the frequency domain. Meanwhile, Mamba-based forecasters, despite their efficiency in long-range modeling, introduce frequency information only as auxiliary signals, leaving spectral evidence disconnected from state transitions. In this work, we propose AdaMamba, a novel approach that explicitly internalizes input-dependent frequency information into the modeling of temporal dependencies. AdaMamba comprises an interaction learning module and an adaptive frequency-gated state space module: the former captures cross-variable dependencies and multi-scale temporal context, while the latter generates context-dependent frequency bases and unifies temporal forgetting into a single time-frequency gate that directly modulates state evolution. Extensive experiments on seven public LTSF benchmarks and four domain-specific datasets demonstrate that AdaMamba consistently outperforms state-of-the-art baselines while maintaining competitive computational efficiency.