Richland Technology
Richland Technology
← Back to Blog
researchdiffusioninterpretabilityrlactivation-steering

RL post-training of diffusion transformers is mostly a constant shift: 0.29M steering vectors recover 90–112% of the reward gain of RL-tuned LoRAs

A frozen base model plus 188 fixed vectors (0.29M params, no reward in the fitting objective) recovers 90–96% of the gain across six image models and 89.8%/111.9% on two video models — and RL-tuned from scratch, the vectors beat weight-based RL on OCR and PickScore.

Richland AI Desk·21 November 2026·4 min read
RL post-training of diffusion transformers is mostly a constant shift: 0.29M steering vectors recover 90–112% of the reward gain of RL-tuned LoRAs

A new preprint argues that most of what reinforcement learning post-training does to a diffusion transformer is a single, prompt-independent nudge: a fixed set of steering vectors added to a frozen model's block outputs recovers 90–112% of the reward gain of RL-tuned image and video generators, at 65× fewer parameters than a rank-32 LoRA on SD3.5-M. The paper (arXiv:2611.26627) reframes RL post-training as "activation steering" rather than new input-dependent skills — and a much cheaper way to reproduce the gains.

What shipped

The authors (Virginia Tech, fal, Georgia Tech, IBM Research) fit a small set of steering vectors to each RL-tuned checkpoint — 188 vectors on SD3.5-M. Added to the block outputs of the frozen base model, the vectors are identical for every prompt, token, and seed; they vary only by block, guidance pass, step bin, and stream. On SD3.5-M that's 0.29M parameters against 18.8M for the rank-32 RL-tuned LoRA (paper, Fig. 2).

The vectors are fit to match the velocity predictions of each public RL-tuned checkpoint, with no reward in the objective — a regression, not RL. Matching velocity matters: simply adding the average activation change RL induces recovers far less of the gain than matching the target model's velocity (intro).

With that single constant shift, the frozen base model recovers:

  • 90.0–96.3% of the RL reward gain across six image models spanning SD3.5-M and FLUX.1-dev, multiple RL algorithms, LoRA and full fine-tuning, and rewards including PickScore, HPSv2.1, OCR, and GenEval (abstract).
  • 89.8% and 111.9% of the gain on two RL-tuned video generators (abstract).

The steering vectors add the same vector to every token of every prompt; in the RL-tuned model, the weight update gives each input its own shift.

Then the authors test the stronger version: can RL learn the vectors from scratch? With the generator frozen and only the steering vectors trainable, initialized at zero, Flow-GRPO reaches 104.6% of the OCR-tuned SD3.5-M gain and 103.6% of the PickScore-tuned gain — exceeding weight-based RL without updating a single pretrained weight (intro).

An ablation localizes the gain: vectors fit to the text stream, the conditional guidance pass, and the early sampling steps alone — 531× fewer parameters than the rank-32 LoRA — still recover 89.5% of the OCR gain and 73.7% of the PickScore gain (intro).

Why it matters

The headline claim is counterintuitive. Capabilities like rendering requested text or composing multiple objects are inherently prompt-dependent, so the natural assumption is that improving them requires complex, input-dependent transformations. A constant vector carries no prompt information at all — so how does it produce prompt-specific improvements?

The frozen generator does the prompt-specific work. Attention, normalization, and MLP layers mix the constant vector with prompt-dependent representations, so its effect on the output depends strongly on the prompt (intro).

The authors trace the mechanism on SD3.5-M: the shift reaches the image through what image tokens read from text tokens in joint attention. When image tokens read the text keys/values of the unsteered base model instead — everything else unchanged — most of the OCR and PickScore gains disappear. The same replacement removes most of the gains of the actual RL-tuned models, showing that the steering vectors and RL reach the image by the same route (intro).

For engineers this suggests a cheap, interpretable alternative to RL post-training: instead of running Flow-GRPO over LoRA or full weights — expensive and finicky — fit or train a few hundred thousand constant parameters on a frozen base. The paper also fits a growing line of LLM results: RL keeps a model close to its base, raises the probability of answers the base already samples, and can be matched by RL-trained steering vectors (related work). This extends that story to diffusion transformers, where the reward gain is a prompt-specific change of a whole image or video.

It also reframes what RL post-training does: it steers the pretrained generator toward what it can already compute, rather than teaching new input-dependent transformations.

Caveats

  • This is a preprint, not yet peer-reviewed. Every headline number (0.29M vs 18.8M, 90.0–96.3%, 104.6%, 103.6%, 89.5%) is the authors' own, not independently replicated.
  • Fitting the vectors is not free: it requires the RL-tuned checkpoint you want to match and a fitting pass over its velocity predictions. The method is parameter-efficient at inference; the paper does not report the fitting cost.
  • The two video models, Wan2.1 and Wan2.2-5B, are identified only in the supplementary material. The promised project page is a dead placeholder in the arXiv comments rather than a working link, and we could not locate a code repository.
  • The claim is about policy-gradient RL post-training. The authors' own contrast case is supervised fine-tuning such as DreamBooth subject-fidelity, where a same-size LoRA does better than the vectors — the constant-shift story is not expected to hold there.
  • The online-RL result (104.6%/103.6%) is on SD3.5-M only, for OCR and PickScore rewards.

Sources

Start a Conversation

Ready to build something like this?

Tell us about your project and we'll respond with a clear plan within one business day.

Tell Us About Your Project

Takes 3 minutes. We read every submission personally.