GUI-JEPA picks mobile actions with no token decoding — 2.8–4.0× faster than a 7B VLM at 6% of the params
A decoder-free JEPA policy scores accessibility-tree candidates in latent space — 2.8–4.0× faster than a fine-tuned 7B VLM on AndroidControl, at 86–92% of its accuracy.
A new preprint, GUI-JEPA, makes action selection in mobile GUI agents cheaper: it scores accessibility-tree candidates in the latent space of a frozen JEPA encoder and — to the authors' knowledge the first decoder-free GUI agent policy — drops token decoding out of the loop entirely. On AndroidControl, the authors report the decoder-free policy is 2.8–4.0× faster per step than a Qwen2.5-VL-7B agent fine-tuned on the same inputs, reaching 86–92% of its in-distribution accuracy with 6.0% of its parameters.
What shipped
GUI-JEPA (Wang et al., arXiv, 27 Oct 2026) is, in the authors' words, "to our knowledge the first decoder-free GUI agent policy built on a Joint-Embedding Predictive Architecture (JEPA)". The dominant mobile GUI agent is a 7B-scale VLM that emits each action token by token from a screenshot; even candidate-scoring approaches keep a generative LLM or VLM in the loop. GUI-JEPA removes it:
At every step it enumerates the executable actions from the a11y tree, embeds the screenshot once with a frozen JEPA encoder, transforms that screen latent once per candidate with a residual predictor, scores each candidate-conditioned latent and the candidate's text against the text condition, and executes the highest-scoring candidate. Since no token, pixel or line of code is generated, a decision is one batched forward pass whose cost grows with the number of candidates.
The pieces, per the paper:
- Screen encoding. A frozen "JEPA ViT-L pretrained on GUI screenshots" embeds the screen once per step; the latent is 196 patch tokens of 1024 dims.
- Candidate-conditioned prediction. A 39.9M-parameter residual predictor transforms that screen latent once per candidate (each an a11y element or global action), conditioned on the candidate's action and region of interest (ROI).
- Scoring heads. A cosine structural head scores the candidate-conditioned latents; a 33M-parameter text matcher adds a score for how well the candidate's text matches the instruction, and the agent executes the argmax. The two text rankers together add 3.8 out-of-distribution AC-High points (Table 2).
Training runs in three stages (Figure 2): (1) JEPA pretraining on unlabelled GUI screenshots, after which the encoder is frozen; (2) policy training — a step-level offline policy on AndroidControl that predicts the recorded human action from one step's inputs and the ground-truth history; (3) training the text rankers on the fixed policy. No RL stage is described in the sections we could read; this is supervised imitation plus the encoder's self-supervised pretraining.
The headline numbers, all author-reported on AndroidControl, measured on one NVIDIA H20 at batch size 1 (median over 100 real AC-High steps):
| Metric | Value |
|---|---|
| Latency vs Qwen2.5-VL-7B baseline (same inputs) | 2.8–4.0× faster |
| In-distribution accuracy | 92.4% (Low) / 86.1% (High) of the 7B agent's |
| Parameters | 6.0% of the 7B agent's |
| GPU memory under HF serving | 4.6–6.4× less |
| Baseline per-step cost | 207 ms (vLLM) / 507 ms (HF transformers) to emit a ~12-token JSON action |
For scale, the paper cites an external benchmark where 8B-class agents average 2.6–10.6 s per step on live devices.
Why it matters
The obvious takeaway is cost: for a GUI agent, each decision currently means decoding a short JSON action with a multi-billion-parameter model. GUI-JEPA turns the decision into one batched forward pass over however many candidates the accessibility tree exposes, with the generative model gone. That is the difference between live on-device interaction and something that only works on a cluster — at the cost, per the authors, of roughly 8–14% of the 7B agent's in-distribution accuracy.
The more interesting finding is an internal tension the paper states plainly. JEPA's point is predictive world modeling: I-JEPA and V-JEPA 2 learn representations by predicting, in latent space, parts of an input they have not seen. Yet the ablation that matters cuts the dynamics pretraining out of the policy: "a randomly initialised predictor does just as well as one pretrained on screen dynamics," which the authors summarize as "a JEPA policy for GUIs needs no dynamics pretraining."
The predictor is trained for the decision itself, not for dynamics. Read against the abstract's note that "the scorer relies on this latent pathway," the architecture's value sits in the frozen visual encoder plus the candidate-conditioned latent pathway the scorer reads; the predictor's world-model pretraining is surplus to requirements. (The paper does not ablate the encoder itself, so whether the JEPA objective is what makes it work as a backbone is untested.)
That decoupling is a design lesson for anyone building an agent over enumerated options (UI trees, DOM nodes, tool lists, menu items). You do not need a generative policy, and you may not need a learned dynamics model: a small latent scorer on a frozen encoder inherits the encoder's perception for a fraction of the cost, and pretraining compute spent on prediction the decision never uses is wasted. The failure mode to plan around is candidate quality — the agent inherits the accessibility tree's coverage and completeness, since it can only ever pick between enumerated candidates. GUI-JEPA is also a rebuttal to the "candidate-scoring always means a big model in the loop" pattern: prior scorers keep a generative model around — V-Droid verifies a11y candidates with a prefill-only LLM, LiMAC calls Florence-2 for text arguments — while GUI-JEPA's loop is entirely decoder-free.
Caveats
- Every number here is author-reported from a single, unreplicated v1 preprint. Nothing has been independently reproduced, and we found no independent press coverage of the paper yet.
- AndroidControl (Li et al., NeurIPS 2024 D&B — 15,283 demonstrations, 14,548 tasks over 833 apps) is the only benchmark named in the portions of the paper we could retrieve. The headline accuracy ratio is in-distribution only; out-of-distribution AC-High needs the two extra text rankers, which add 3.8 points.
- The code repo is a placeholder — a 42-byte README, no code, weights, or licence. You cannot run this yet, despite the abstract linking to it.
- The paper's experimental tables were not fully recoverable from the arXiv HTML (it truncates mid-§3), so absolute success/match rates for either agent, the ablation tables, and GUI-JEPA's own absolute ms-per-step figures are not public in a form we could verify. The "6.0% of parameters" counting basis is likewise not stated in the retrievable portion.
- Speed comes with a scaling caveat: per-step cost grows with the number of candidates, and everything is gated on accessibility-tree extraction quality.