Vision encoders: the pretraining is optional, the dedicated capacity is not — and it can live inside the backbone
A controlled ablation from NYU and Meta FAIR isolates what a vision encoder actually contributes — and finds it can be trained jointly and placed inside the backbone.

A controlled study from NYU and Meta FAIR argues that a vision encoder's value in a multimodal LLM comes from its dedicated parameters, not its pretraining — and that those parameters don't need to sit in a separate encoder before the backbone. Training every model from scratch on common data at matched lifetime compute, the authors report that an encoder trained jointly with the language model matches a pretrained one. An encoder-free model trails even when handed the encoder's parameter count.
The study
The paper, Unbundling Multimodal Vision Encoding (arXiv:2611.17334v1, submitted 12 Nov 2026), is a preprint from NYU and Meta FAIR. It starts from the observation that a pretrained vision encoder bundles three design choices:
- Visual prior — knowledge from the encoder's upstream pretraining (image recognition or image–text alignment).
- Dedicated vision capacity — parameters used only by vision tokens.
- Pre-fusion placement — running that capacity in a separate encoder before vision and text tokens interact in the backbone.
Published models differ in data, architecture, and compute at once, so comparing them can't isolate any one choice. The authors instead train every stage from random initialization, holding everything fixed except the vision-encoding recipe, and compare at matched "lifetime compute" — encoder pretraining, text-only training, and multimodal training counted together, across budgets from 6.5×10²⁰ to 2.9×10²² FLOPs.
The backbone is modeled after Qwen3 (dense, decoder-only, pre-norm, RoPE, SwiGLU, GQA), compared at 1.7B and 4B with vision capacity fixed at 0.4B parameters. The scratch encoder is a randomly initialized 27-layer, 0.4B-parameter Vision Transformer modeled after MoonViT-V2. Evaluation uses Avg. VQA (four category means over 16 Cambrian benchmarks) and Avg. Text (12 language benchmarks).
The three headline results:
The visual prior is optional. At 1.7B and 4B, an encoder pretrained with contrastive MetaCLIP trails the from-scratch encoder at every budget — by 1.8 Avg. VQA points at the largest 4B budget. More encoder pretraining raises ImageNet accuracy but not VQA, even when treated as free.
Dedicated capacity is what matters. Encoder-free models (image patches fed straight into the backbone via a linear embedding) trail the encoder by 4.1–6.4 Avg. VQA points at 1.7B and 2.8–6.8 at 4B, and across the 1.7B–6.4B size ladder need roughly 3× the compute to match it. The gap isn't just missing parameters: giving the encoder-free backbone the encoder's 0.4B parameters as extra width, depth, or both moves Avg. VQA by at most 0.4 points. A richer two-layer patch embedding helps (40.5 vs 39.0 Avg. VQA) but still sits below the encoder's 44.4.
That capacity can move in-fusion. Instead of a pre-fusion encoder, the authors place the 0.4B-parameter budget as vision-only modules inside the shared backbone. A vision-only FFN branch in the first eight layers matches the scratch encoder (45.0 vs 44.4 Avg. VQA); other placements do no better than encoder-free. Early placement helps most on OCR and chart questions (36.0 vs 25.2 for encoder-free). In-fusion stays within one Avg. Text point of the encoder.
The results hold when adding vision to a pretrained 1.7B language model (midtraining): in-fusion, early-layer mixture-of-transformers, and the scratch encoder perform comparably, all ahead of encoder-free.
Why it matters
The takeaway inverts a widely held assumption: a team building a multimodal model may not need a separate, pretrained vision encoder at all. The encoder's contribution is dedicated vision capacity trained jointly with the backbone — capacity that can live early inside the backbone rather than in a serial pre-fusion stage.
That placement has a concrete serving benefit. A separate encoder must finish each image before the backbone can start, adding a serial step to inference. On a 1.7B backbone on one H200 serving 16 concurrent requests (one 448-px image plus 256 text tokens each), the authors report that in-fusion serves images faster than the encoder at every vision-only budget, with the advantage growing as the budget scales. The encoder-free model is fastest but, at 1.7B, trails the encoder by 5.4 Avg. VQA points — the speed/comprehension trade-off made explicit.
For engineers this is a design-space result, not a drop-in recipe: train dedicated vision capacity jointly with the backbone and place it early, and question the default of bolting on a pretrained encoder.
Caveats
This is a preprint with author-reported numbers and no independent replication. Treat every benchmark figure as the authors' claim, not established fact.
The controlled protocol is also its own limitation. Everything is trained from scratch on the authors' data with relatively small backbones (0.6B–6.4B), and no external pretrained models are used. Whether the conclusions transfer to large production setups — real Kimi K3, SigLIP-based systems, or encoders trained on corpora like SigLIP 2's 40B image–text samples — is untested. The "pretraining treated as free" robustness check still relies on the authors' own pretraining data, not the much larger corpora real encoders use. The paper is future-dated, and as of the time of writing we found no third-party commentary on it to weigh.