Segment to Focus: Guiding Latent Action Models in the Presence of Distractors

A new method called MaskLAM improves how AI agents learn from video by ignoring visual distractions, according to research posted on arXiv. The technique forces latent action models to focus on the agent itself, substantially reducing errors when the agent is later trained to perform tasks. Latent action models (LAMs) are designed to pre-train embodied agents using video that lacks explicit action labels, inferring the actions that cause changes between frames [1]. These inferred actions can then be mapped to real control commands with only a small set of labeled examples [2]. The approach has shown promise, but recent work identified a critical weakness: LAMs fail when videos contain action-correlated visual distractors such as dynamic backgrounds or camera movement [1]. In these cases, the model's standard reconstruction objective mistakenly encodes the motion of distractors rather than the agent's own dynamics, leading to poor performance when the agent is fine-tuned for a task [2]. The researchers behind MaskLAM observed that the agent and the distracting elements are usually separated in space within the video frame [1]. Control-relevant changes are concentrated on the agent's pixels, while exogenous motion occurs elsewhere [2]. MaskLAM exploits this spatial separation by restricting the model's reconstruction objective to only the pixels belonging to the agent, forcing the inferred latent actions to explain agent-controlled dynamics [1]. The method obtains the required agent mask without any additional training by using off-the-shelf segmentation foundation models, such as the Segment Anything Model (SAM) [2]. Crucially, MaskLAM requires no architectural changes to the underlying model, no auxiliary loss functions, and no action labels during the pre-training phase [1]. The technique was evaluated on two continuous-control benchmarks: the Distracting Control Suite and Distracting Meta-World [2]. On these tests, MaskLAM reduced the normalized linear-probe mean squared error by up to 3.51 times compared to the existing LAPO model [1]. It also improved the normalized return, a measure of task performance, by up to 4.97 times over LAPO [2]. The results significantly narrowed the performance gap to a model called LAOM-Labels, which relies on ground-truth action supervision during training [1]. The paper, titled "Segment to Focus: Guiding Latent Action Models in the Presence of Distractors," was submitted by Marcus Fechner on February 2, 2026, and a revised version was posted on May 27, 2026 [1].