LaLa-World: natural-language actions let one latent world model span robots, sims, and video — 6× cheaper than an RAEv2 baseline
NAVER AI Lab's language-conditioned JEPA predictor trains jointly on control and vision data and plans 13.9× faster than a matched diffusion baseline — but no code or weights are out yet.
LaLa-World, a new preprint from NAVER AI Lab with Georgia Tech, Brown, and Mila, is a JEPA-style latent world model that writes actions as natural language: a single predictor — about 0.8B parameters in the matched comparison — trains jointly on robot and simulation trajectories plus plain image and video data (arXiv abstract, alphaXiv summary). The authors report 6× less training cost and 13.9× faster MPC planning than a matched RAEv2-based generative baseline, and a fine-tuned policy competitive with an adapted Cosmos Policy at up to 5.5× faster inference (arXiv abstract).
What shipped
Latent world models (LWMs) like V-JEPA 2-AC and DINO-WM predict future visual representations rather than reconstructing pixels, which makes them cheap to roll out for planning (paper HTML). Their action-conditioned predictors are typically trained per domain on numerical action vectors whose dimensionality and physical meaning differ everywhere — [2, 4] may encode a displacement in one environment and [4] force in another — and image/video data, a rich source of world observations, usually lacks action annotations. That is the scaling wall LaLa-World attacks.
Two design choices do the work (paper HTML):
- Language is the shared interface for actions and visual change. Fixed templates convert each domain's numerical action vector into text that preserves direction and magnitude —
move 2 left and 4 up,push with force 2 downward. Image and video transitions are described the same way, via captions or editing instructions (alphaXiv summary). Everything becomes one text channel. - The predictor is itself a language model. A frozen self-supervised vision encoder maps frames to representations; a predictor built on Qwen3.5 reads the current latent states alongside the language condition and outputs the next representation, trained to match the target (alphaXiv summary).
Because all actions share one interface, a single predictor plans across domains whose action spaces have nothing in common, and it accepts qualitative actions like move slightly left with no numerical value at all (paper HTML).
Planning results. The headline evaluation covers five simulation tasks — Wall, PointMaze, Push-T, MetaWorld-Reach, MetaWorld-Reach-Wall — plus DROID robot data. Averaged over the five sim tasks, one jointly trained LaLa-World predictor reaches 72.1% success versus 64.6% for JEPA-WM baselines trained separately per domain — a 7.5-point gap in this comparison (alphaXiv summary).
The sharper test is a matched comparison against an RAEv2-based generative world model, whose predictor is a diffusion Transformer (paper HTML, RAEv2 project page). Both models use the same language conditions, training data, and vision encoder, with predictors of roughly 0.8B parameters. LaLa-World scores 59.4 in average planning performance against 48.0 for five-step RAEv2, and it matches RAEv2's 12-hour performance after 2 hours of training on eight H100s — the "6× less training cost" (alphaXiv summary).
On planning speed, LaLa-World runs 2.9× faster per trajectory than one-step RAEv2 and 13.9× faster than five-step RAEv2, timed on a single H100 (alphaXiv summary).
Adding image and video transitions to the training mix helps: the planning average rises from 59.3 to 61.6, and paraphrase robustness improves sharply. When the action templates are rewritten at evaluation time (words changed, values and directions kept), the control-only model drops 6.8 points, while the all-data model drops only 0.6 (alphaXiv summary).
LaLa-Policy. The pretrained world model also initializes a policy. LaLa-Policy is fine-tuned to jointly predict an action chunk and a future latent representation, with a separate action generator producing the chunk (alphaXiv summary). On RoboCasa with 50 demonstrations per task, it reaches 57.3% success versus 56.0% for an adapted Cosmos Policy initialized from Cosmos-Predict2, cutting average inference time per trajectory from 11.71 seconds to 2.12 seconds — the ~5.5× figure (alphaXiv summary, NVIDIA Cosmos Policy).
Why it matters
The design move worth stealing is the shared text interface for heterogeneous dynamics data. Every LWM builder hits the same wall: multi-environment datasets have incompatible action spaces, and unlabeled video is unusable for action-conditioned prediction. LaLa-World shows a single jointly trained predictor absorbs all of it, and the results are consistent with the extra image/video data buying robustness to how actions are worded.
The efficiency argument for latent over generative world models is now measured in a matched setup rather than asserted. Planning means repeatedly evaluating candidate action sequences, and skipping pixel reconstruction is what makes each evaluation cheap. Against a same-data, same-encoder, similar-size diffusion predictor, the latent predictor planned competitively at a fraction of the wall-clock training time — though the comparison is narrower than the abstract's "6×" wording suggests (see below).
The policy result matters for a different reason: it shows the LWM weights carry useful priors for action prediction, in the same way Cosmos Policy adapts a video-generation model. The speed gap (2.12s vs 11.71s per trajectory) is the consequence of doing that joint prediction in latent space instead of in pixels.
Caveats
- Nothing to reproduce yet. This is an unreviewed preprint (arXiv v1, submitted Oct 20, 2026). The linked project page (
lala-world.github.io) returns 404, the paper's code links point at anchors on that page, and we found no published code or weights. The headline numbers are entirely author-reported. - The efficiency figures are wall-clock time, and the axes matter. The "6×" is 2h vs 12h of elapsed training on the same 8×H100 setup (so the same ratio holds in GPU-hours), not FLOPs or measured compute efficiency; a latent architecture doing the same wall-clock work could still use the hardware differently. The "13.9×" is against five-step RAEv2 — against one-step RAEv2 it is 2.9×.
- The Cosmos Policy comparison is against an adapted baseline. To make it fair, the authors adapted Cosmos Policy to the same joint action-plus-future prediction setup; it is not NVIDIA's shipped model, which reports a higher RoboCasa average (67.1%) in its own documentation. LaLa-Policy beating the adapted baseline 57.3% to 56.0% is not the same as beating the released Cosmos Policy.
- Detail numbers need a grain of salt. Figures like 72.1/64.6, 59.4/48.0, and 2.12s/11.71s come from the paper's figures and tables as rendered by alphaXiv's AI-generated summary; the arXiv HTML truncates before the experiments section in our reader, so these are paper-derived but not independently verified by us. The matched RAEv2 setup also controls encoder, conditions, data, and approximate predictor size, but does not isolate latent prediction as the only cause of the gain.
- Scope is deliberately small. The predictor is trained for next-state prediction only (no multistep supervision), the MPC horizons tested are at most six steps, and the policy results are simulation-only; the authors name real-robot evaluation as next work.