Learning a Particle Dynamics Model with Real-world Videos

A new machine-learning framework can train neural object dynamics models directly from unlabeled real-world videos, bypassing the need for simulated data and labeled particle states, according to research published on arXiv [1]. The approach, detailed in a paper submitted on 22 May 2026, introduces a particle-based dynamics model that operates within a Gaussian splatting framework [1]. The model works on dense particles derived from Gaussians — particles that carry scale and rotation information — and predicts how their positions and rotations change over time [1]. Training is driven by rendering supervision, which means the system learns by comparing rendered outputs to the original video frames rather than requiring ground-truth particle-level labels [1]. This design eliminates the need for heuristic subsampling of anchor points, a step that earlier methods often relied upon [1]. Data-driven physics simulators, sometimes called world models, have shown strong results in predicting the motion of rigid and non-rigid objects in multi-body scenes [2]. However, these models have historically been confined to simulated environments because obtaining perfect state information — such as complete scene point clouds and temporal point correspondences — is difficult in real-world settings [2]. The resulting sim-to-real gap can limit the usefulness of models when they are deployed outside the synthetic domains they were trained on [2]. The new framework directly addresses that gap by learning from real-world video alone [1]. To support the work, the researchers also assembled a real-world dataset of approximately 500 videos capturing diverse object interactions [1]. The dataset provides a benchmark for training and evaluating dynamics models without synthetic data [1]. While this work focuses on object dynamics, the broader field of learned simulators draws on concepts from multiple domains. Molecular dynamics, for instance, has long used numerical solutions to Newton’s equations of motion to simulate the physical movements of atoms and molecules, a process that is mathematically ill-conditioned and accumulates integration errors over long timescales [4]. Diffusion models, another class of generative models widely used in computer vision as of 2024, learn to reverse a noising process and have been applied to tasks including image generation and video generation [3]. The new particle-dynamics framework does not use a diffusion formulation but shares the goal of learning complex physical processes from data [1][3].