ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model

A new framework called ThinkJEPA pairs dense-frame dynamics modeling with semantic guidance from a vision-language model to improve latent world models, according to a paper posted on arXiv [1]. The dual-pathway design targets a known weakness: predictors that rely on short observation windows often miss long-horizon context [1]. Latent world models such as V-JEPA2 have demonstrated an ability to forecast future world states from video, but their dense prediction over brief windows can bias them toward local, low-level extrapolation [1]. Vision-language models offer broader semantic grounding by reasoning over uniformly sampled frames, yet they are not well-suited as standalone dense predictors because of compute-driven sparse sampling, a language-output bottleneck, and a data-regime mismatch on small action-conditioned datasets [1]. The ThinkJEPA framework, authored by Haichao Zhang and colleagues, addresses this gap with a dual-temporal pathway: a dense JEPA branch captures fine-grained motion and interaction cues, while a uniformly sampled VLM “thinker” branch operates at a larger temporal stride to supply knowledge-rich guidance [1][2]. To transfer the VLM’s reasoning signals, the team built a hierarchical pyramid representation extraction module that pools visual tokens from the VLM’s visual encoder and intermediate hidden states from selected language-model layers [3][5]. These multi-depth features are projected into the predictor space, preserving both low-level visual cues and high-level semantic traces [5]. The paper reports that ThinkJEPA outperforms a strong VLM-only baseline and a JEPA-predictor baseline on hand-manipulation trajectory prediction and yields more robust long-horizon rollout behavior [1][4]. Beyond the primary hand-manipulation experiments, the HTML version of the paper states that ThinkJEPA was evaluated across EgoDex, EgoExo4D, BAIR Robot Pushing, and Physion, where it surpassed diverse latent world model and trajectory-prediction baselines on egocentric trajectory prediction, long-horizon rollout, robotic latent prediction, and physical-scene forecasting [6]. The work builds on the V-JEPA lineage: V-JEPA2 was pre-trained on over 1 million hours of internet video and later adapted into an action-conditioned world model, V-JEPA2-AC, which used less than 62 hours of unlabeled robot video to enable zero-shot picking and placing on Franka arms [7][8]. ThinkJEPA extends that direction by injecting semantic reasoning directly into the latent forecasting loop rather than treating language as a post-hoc interface. The first version of the ThinkJEPA manuscript was submitted on 23 March 2026 at a size of 162 KB; a revised version followed on 16 June 2026 at 144 KB [1].