VISTA: MIT's visual harness keeps every frame in memory — Claude Opus 5.0 hits perfect 100 RHAE on ARC-AGI-3 without program synthesis

MIT researchers released VISTA, a "visual harness" that stores every environment frame losslessly in external memory and lets a vision-language agent retrieve, zoom into, and read pixels from any past frame. On ARC-AGI-3 it lifts Claude Opus 5.0's Relative Human Action Efficiency (RHAE) from 40.68 to a perfect 100.00 — the authors report it is the first vision-based system to reach perfect or near-perfect scores on the benchmark without program synthesis.

What shipped

VISTA (paper: arXiv:2610.02200, code: github.com/joshhhhhan/VISTA) is a harness, not a model. It wraps a general-purpose multimodal model — Anthropic Claude Opus 5.0 or OpenAI GPT-5.6 Sol — and changes only the interface between the model and the environment. The model is never retrained.

The design has three components, per the paper:

• Visual observations. The agent receives a 512×512 PNG — a nearest-neighbor 8× upscale of the official 64×64 frame with one-pixel grid lines. It is never told the world is a 64×64 grid.
• Lossless visual memory. Every returned frame, including intermediate animation frames, is stored in its original form, indexed by turn and frame number. Nothing is embedded and discarded.
• Model-directed visual inspection. The model decides when to revisit old frames, zoom into a region, or read exact pixel values.

The harness exposes four tools: play (execute an environment action), inspect (revisit frames or regions, with zoom), read_pixels (read exact pixel values), and history (revisit prior actions and their results). Two markdown files serve as memory. GUIDE.md holds durable cross-level knowledge (e.g. "the difference between a brown block and a black background"); WORKING.md holds per-level state (e.g. "the character currently has a power-up").

The per-turn loop: given the latest frame and the list of available keyboard actions, the model chooses to (i) read or update notes, (ii) retrieve previous frames, (iii) retrieve previous actions and results, or (iv) execute an action. It repeats until it acts. After the environment updates, only the latest frame and the new action list are added to the input.

Why it matters

The core claim is that for multimodal agents, memory should be storage, not compression. Typical agentic systems embed each frame and keep the embedding until context fills, then delete it. Embeddings are lossy, and once deleted the information is gone. VISTA instead keeps the original evidence and lets the model decide what to re-examine — storage kept separate from attention.

On ARC-AGI-3's 25 public games, VISTA with Claude Opus 5.0 at xhigh reasoning effort completes all 25 with an RHAE of 100.00, using 57.4% fewer actions than first-time human players. VISTA with GPT-5.6 Sol at max effort scores 99.00. The official non-program baselines are 40.68 (Opus 5.0 high) and 13.33 (GPT-5.6 Sol max). Earlier systems that hit perfect RHAE, such as Tycho, did so by generating programs that simulated the environments; the authors report VISTA is the first to do it by actually playing.

The gains extend beyond ARC-AGI-3. On GameWorld (170 tasks in 34 browser games), VISTA with GPT-5.6 Sol completed 63.3% of tasks versus 55.3% for first-time humans and 40% for the native harness. On AI GameStore it scored about 40% better than first-time players. On BabyVision it went from 41.0% to 63.2%.

The ablation shows the frames are doing the work: denied the ability to store frames, VISTA with GPT-5.6 Sol drops to 70 RHAE.

For engineers, the practical takeaway is that a large chunk of this gain comes from the interface, not the model. If you are building a vision-language agent that interacts with a screen or game, storing raw frames and giving the model retrieval and zoom tools is a cheap, adoptable change — no fine-tuning, no new weights.

Caveats

• All benchmark numbers are author-reported in a non-peer-reviewed tech report. There is no independent replication, and the ARC-AGI-3 runs are expensive (the README notes roughly 1.5–2 hours per game) and need API keys.
• The efficiency metric flatters the harness. Under ARC-AGI-3's RHAE protocol, only environment actions count toward the score; internal reasoning and read-only inspection are free. So the retrieval, zoom, and pixel reads that make VISTA work do not count against its efficiency number.
• A small discrepancy in the efficiency figure: the arXiv paper and The Batch say 57.4% fewer actions, while the project page says 56.0%. The difference is immaterial to the result but worth noting.
• The 70 RHAE no-frame-storage ablation is reported by The Batch; I could not confirm that exact figure in the paper text.
• The GameWorld human baseline of 55.3% is a success-rate figure and comes from The Batch; other summaries report a different fresh-human success figure, so treat it as one source's number rather than a settled baseline.

Sources

• VISTA: A Visual Harness for Reasoning in an Interactive World (arXiv abstract): https://arxiv.org/abs/2610.02200
• VISTA paper, full HTML: https://arxiv.org/html/2610.02200v1
• VISTA code repository (MIT license): https://github.com/joshhhhhan/VISTA
• VISTA project page: https://vista-research.github.io
• The Batch: Context Engineering for Images: https://deeplearning.ai/the-batch/context-engineering-for-images