Richland Technology
Richland Technology
← Back to Blog
open-sourceresearchembeddingsaudio

EmbeddingGemma-2 hits #1 on MTEB audio with a 2.37 MB frozen PCA head — no training, just the null space of the text projection

Injecting acoustic features from early Conformer layers into the 256 dimensions text vectors never touch lifts a 570M model past 8–9B omni models on audio-to-audio tasks — author-reported, pending leaderboard merge.

Richland AI Desk·29 October 2026·5 min read

A frozen 2.37 MB PCA head — no training, no gradient updates — took Google's 577M-parameter EmbeddingGemma-2 from rank #26 to #1 on MTEB's MAEB audio leaderboard, outscoring 7–9B multimodal models on audio-to-audio tasks, by the author's own measurements. The trick is to inject acoustic features from early Conformer layers into the 256 dimensions of the 768-d embedding space that text vectors never touch.

What shipped

Bruno Hays released BrunoHays/embeddinggemma-2-nullspace-audio on Hugging Face: google/embeddinggemma-2 with fully frozen weights, plus a 256-dimensional "null-space head" — a single 2.37 MB safetensors file with no trainable parameters. Text, image, and video inputs take the same code path and produce identical outputs to the base model; only audio embeddings shift (model card).

Author-reported MAEB results, measured with the mteb library:

  • Audio-only (19 tasks): mean score 47.79 → 54.82, rank #26 → #1 across 70 audio models.
  • Full MAEB (30 tasks): 49.31 → 54.40 (raw average 49.95 → 55.09), rank #16 → #1.
  • The models it displaces are 8–9B: Qwen2-Audio-7B (8.40B, #2 at 50.99 on audio-only), e5-omni-7B and LCO-Embedding-Omni-7B (8.93B each, #2/#3 on full MAEB).

The gains concentrate exactly where text-aligned audio models are known to be weak. Per-task, versus the measured base model: VoxPopuliGenderClustering 0.05 → 63.09, JamAltArtistA2ARetrieval 55.20 → 93.82, CREMA_D 19.70 → 40.16, IEMOCAPGender 66.38 → 85.77, GTZANGenre 57.70 → 71.50 (per-task table).

Why it matters

The result rests on two facts most embedding pipelines leave on the table.

Early layers still know acoustics. Layer-wise analyses of speech encoders (Pasad et al., 2021; Bouziane, Mdhaffar & Estève, 2026) show speaker, pitch, and emotion traits peak in early and middle layers, then get filtered out as the model converges on what was said rather than how it sounded. In EmbeddingGemma-2's 12-layer Conformer audio tower, layer 2 still carries gender, timbre, and pitch; layer 6 still carries vocal emotion (CREMA_D nearly doubles from 19.7% at the final output to 39.5% at layer 6).

The null space is free real estate. EmbeddingGemma-2's text backbone is 512-d, projected to 768-d by a terminal 768×512 matrix W. Text embeddings therefore live entirely in the column space of W; the orthogonal complement — 256 perpendicular directions — is never touched by any text vector. The head fills exactly those directions: it pools layer 2 (mean + std, 2048-d) and layer 6 (mean, 1024-d), soft-whitened-PCA-reduces each block to 128-d, concatenates to a 256-d vector z, and computes

v_final = normalize(v_base + γ · (z @ U_⊥ᵀ))

with γ = 1.0 and U_⊥ a 768×256 matrix spanning the null space. Because text vectors are orthogonal to U_⊥ by construction, the injection cannot move text-to-text similarities — the author verifies this on MSEB/SVQ, where cross-modal retrieval moves by at most ~0.3 points (architecture section).

The PCA states were fitted only on 350 clips from each of four external corpora (expresso, AudioSet, music_genres, esc50) — no MAEB or MSEB audio — which the author calls "zero-leakage Protocol A."

For engineers building audio search or embedding infrastructure, three things follow:

  • Edge audio search just got cheaper. A 570M-parameter model plus a 2.37 MB add-on outranks 8–9B omni models on audio-to-audio tasks, and Google designed the base model to run on consumer hardware such as laptops. If you only need audio-to-audio similarity (artist/song search, voice clustering, emotion), the serving footprint is dramatically smaller.
  • It is a template, not a one-off. Any embedding model whose final layer projects down to a lower dimension has an unused orthogonal complement — free capacity for auxiliary signals (acoustic, visual, or otherwise) without touching the primary embedding space.
  • It quantitatively confirms the layer-wise story. The "final layers keep words, early layers keep sound" intuition from the speech-representation literature is now a working recipe with leaderboard numbers attached.

Caveats

  • Rank #1 is author-reported, not yet on the live leaderboard. The MTEB submission PRs (mteb#5775, results#810) were still open and unmerged as of October 28, and the benchmark itself is labeled "MAEB(beta)" pending peer review. The per-task numbers in the model card match the submitted result files, but as of this writing no independent replication has surfaced.
  • "Zero training" is narrower than it sounds. No gradients were used, but the author discloses that the PCA mode and γ were selected by looking at test scores across a small grid, and that γ = 1.0 was the largest value tested — the optimum may lie beyond it. He says to treat the MAEB numbers as optimistic.
  • Gains are concentrated, and some tasks regress hard. Several spoken-language tasks lose accuracy: NMSQAPairClassification (question answering) −19.5 pp, SIBFLEURS (language ID) −35.9 pp, MInDS14 (intent) −9.3 pp (97.63 → 88.34), VoxCelebSA (sentiment) −9.8 pp. The headline "beats 9B models" really means "beats them on audio-to-audio and acoustic-attribute tasks."
  • Full 768-d required. Because there is no retraining, audio embeddings need all 768 dimensions — Matryoshka truncation to 512/256/128 is not supported.
  • Small discrepancies to note. The author's 577M parameter figure differs from Google's 570M text+audio figure on the base model card, and the model card's citation block points at the v1 EmbeddingGemma paper rather than the EmbeddingGemma 2 technical report. None of this changes the technique, but it is a reminder that this is a single-run, self-measured result awaiting review.

Sources

Start a Conversation

Ready to build something like this?

Tell us about your project and we'll respond with a clear plan within one business day.

Tell Us About Your Project

Takes 3 minutes. We read every submission personally.