MagicSim: A Unified Infrastructure for Executable Embodied Interaction

A new infrastructure called MagicSim aims to unify the fragmented simulation pipelines used in robot learning by providing a single, deterministic runtime for control, skills, and planning, according to a paper submitted to arXiv on 16 Jun 2026 [1][2]. The system, detailed in a preprint on the open-access repository arXiv, is built around one deterministic batched runtime and a shared Markov decision process (MDP) [1][2]. The authors argue that existing simulation pipelines split control, skills, and planning across disconnected environments or rely on non-reproducible “magic” actions, making it difficult to evaluate and annotate the same episode consistently [1][2]. arXiv, which was founded in 1991 and now hosts over two million e-prints, serves as the primary distribution channel for such pre-peer-review manuscripts in fields like computer science and robotics [6]. MagicSim constructs executable worlds from YAML-first specifications that decouple contents, placement, behavior, and agent exposure [1][2]. These worlds span task families, interaction regimes, physics, layouts, sensors, avatars, and robot embodiments, all operating within a single reset-and-step loop [1][2]. A common execution interface grounds high-level commands through controllers, atomic skills, planner primitives, and asynchronous planning, translating them into robot actions rather than simulator-side state edits [1][2]. One task definition supports three capabilities: benchmark and reinforcement learning evaluation, an autocollect interface that automatically turns commands into grounded trajectories, and agent or vision-language-model (VLM) interaction [1][2]. For automatic execution, commands flow through a Command-to-Skill-to-Planner-to-Robot-to-Record pipeline, while per-environment states for command, skill, planning, retry, annotation, and episode progress advance independently above the shared physics tick [1][2]. Successful rollouts are saved as structured multimodal trajectories that align language supervision, action representations, visual and geometric representations, and task-level status with the executed episode [1][2]. The paper’s abstract page on arXiv also features the arXivLabs framework, a set of experimental community-built tools that appear as tabs below the abstract [3][4]. arXivLabs, formalized in 2020, allows third-party collaborators to develop features such as the Bibliographic Explorer and CORE Recommender, provided they adhere to arXiv’s values of openness, community, excellence, and user data privacy [4][5]. The framework is currently on a temporary hiatus for new proposals while the arXiv development team focuses on migrating systems to the cloud [3].