EgoAERO: Learning Dexterous Manipulation from a Single Egocentric Video without Object Assets

A new framework called EgoAERO can teach robots dexterous manipulation skills from a single egocentric RGB-D video of a human performing a task, without requiring pre-scanned 3D models of the objects involved, according to a paper submitted in 2026 [1]. The system reconstructs contact-consistent hand-object trajectories using asset-free object tracking and reconstruction, ego motion compensation, and adaptive contact optimization [1][3]. It then converts these trajectories into executable robot policies through a two-stage residual learning process: a hand-tracking policy learns to follow the reconstructed human hand motion, and a residual policy uses object pose and contact feedback to produce dexterous manipulation [3]. Simulation and real-world experiments demonstrated that EgoAERO enables single-demonstration dexterous manipulation and achieves downstream performance close to methods that rely on CAD-based reconstructions when evaluated on the HOI4D benchmark [1][3]. The researchers also introduced EgoDex-R, a large-scale egocentric dataset containing 4.3 million RGB-D frames for dexterous policy learning, and an online quality assessment mechanism [1][2]. This work addresses a persistent bottleneck in robot learning: the scarcity of manipulation data. Unlike natural language or 2D computer vision, there is no internet-scale corpus for dexterous manipulation [4]. Egocentric human video offers a passively scalable alternative, but existing large-scale datasets such as Ego4D lack native hand pose annotations and do not focus on object manipulation [4]. A separate effort, the EgoDex dataset, collected 829 hours of egocentric video with paired 3D hand and finger tracking data across 194 tabletop tasks using Apple Vision Pro, providing 90 million frames and 338,000 task demonstrations [4]. EgoAERO's asset-free approach distinguishes it from prior work that required pre-scanned object meshes or CAD models [2][3]. The framework combines lightweight MLLM semantic initialization with asset-free object tracking and reconstruction under hand occlusion [3]. Other recent research has explored complementary directions for learning from egocentric video. EgoAVFlow, for instance, addresses the problem of active viewpoint control by learning manipulation and camera trajectories from egocentric videos through a shared 3D flow representation, using reward-maximizing denoising to maintain task-critical visibility during robot execution [5]. EgoHumanoid extends the paradigm to humanoid loco-manipulation by co-training vision-language-action policies on abundant egocentric human demonstrations and limited robot data, incorporating view alignment and action alignment to bridge the embodiment gap [6].