Counterfactual world models: a single subtraction step recovers "what if" reasoning with no counterfactual data — R² from 38–46% to 86–96%
Balestriero and LeCun prove factual world models can't answer individual counterfactuals — then recover them with residual reuse: two forward passes and a subtraction, no retraining.

A new preprint from Randall Balestriero and Yann LeCun proves that world models trained only on factual transitions cannot answer individual counterfactual questions — then shows that a single subtraction step on top of an ordinary factual predictor recovers them, with no counterfactual data, extra parameters, or retraining. On four rendered simulators the authors report counterfactual R² rising from 38–46% to 86–96% using the same world models.
What shipped
The paper (arXiv:2611.24778, by Balestriero, LeCun, and Kerda Varaku) targets a specific problem. A world model operating under a hidden disturbance — a current pushing a boat, a gust of wind — observes one outcome, and you want to know what would have happened under a different action, subject to the same disturbance.
The first result is negative. A world model trained only on factual transitions cannot answer individual counterfactuals on its own, "whatever their data or capacity": distinct counterfactual worlds can induce exactly the same factual distribution, so the counterfactual is not identifiable from factual data alone. The authors back this with a minimax impossibility result and an explicit gauge ambiguity — the latent noise can be relabeled per action without changing any factual prediction.
The constructive result rests on a physical assumption the authors call natura non facit saltus — nature makes no leaps. When actions and disturbances act as separate forces, their interaction is second-order in time. To first order, the counterfactual outcome is the observed outcome plus the difference between the world model's predictions under the alternative and factual actions:
ẑ_{t+1}(a′) = z_{t+1} + μ(z_t, a′) − μ(z_t, a_t)
where z_t = f(o_t) is the encoder representation of the observation and μ(z_t, a) is a factual world model predicting the next representation under action a. The observed outcome already records the realized disturbance's effect; the world model supplies the effect of changing the action. This "residual reuse" needs only two evaluations of an ordinary factual predictor plus one subtraction — no paired counterfactual data, no extra parameters, no explicit disturbance inference, no retraining. The authors prove an O(h²) bound on the counterfactual error relative to disturbance variance for transitions of duration h, and a matching minimax lower bound with a sharp leading constant for estimators using unpaired factual transitions and one observed outcome.

Why it matters
The validation is pixel-based, using JEPA (Joint-Embedding Predictive Architecture) world models that predict in the feature space of self-supervised encoders rather than reconstructing pixels.
On four rendered simulators, the authors report counterfactual R² improving from 38–46% to 86–96% with the same world models, with the correction applied after a readout fitted on factual frames. Applied directly to image features (no state labels), they report 73–94%. On a fifth simulator — a BlueROV2 underwater vehicle — they report 90–94% with sixteen times more factual data. Four further frozen encoders (DINOv3, SigLIP 2, CLIP, EVA-02) give 81–95% on the first four environments.
The strongest result is in hindsight planning: the counterfactual model finds the action that would have reached the goal in 35–69% of events, against 14–23% for an oracle planner that knows the simulator but not the observed outcome. It even outperforms an oracle predictor that knows the true simulator and full state but not the disturbance (which predicts the average outcome).
For engineers: counterfactual reasoning about hidden disturbances bolts onto an existing JEPA stack with two extra forward passes and no retraining. The authors ship a fully reproducible pipeline (MIT-licensed repo, ~150 GPU-hours on H100/H200, with third-party files under their own licenses) across seven rendered simulators — HandReach, Ball-in-Cup, FetchReach, liquid transport (PassWater), Racetrack, Push-T, and BlueROV2 — each with a bounded, physically meaningful disturbance (command bias, force, position-goal offset, or thrust).
Caveats
Every number above is the authors' own claim from a v1 preprint that has not been peer-reviewed. All results come from simulation, and the broad success ranges indicate the method helps far more on some tasks than others.
The method's boundary is explicit. The authors characterize failures at discontinuities (contacts), coordinate folds (folded representations), and disturbances that scale or multiply the command — for example a worn actuator whose effect grows with the command, which is smooth yet interacts with the action at first order. "Separate channels" is an assumption about how the disturbance enters the dynamics, not a consequence of smoothness. The paper says it recovers these predicted failure modes but does not quantify the error in those cases.
One note: some secondary coverage adds a "51–97% with multiple attempts" figure that does not appear in the paper's abstract. I have not relied on that figure here.