A 2.4 MB PCA head puts EmbeddingGemma 2 at the top of MAEB

On 28 October 2026 developer Bruno Hays published an add-on for EmbeddingGemma 2 on Hugging Face that lifts the model from 26th to 1st place in the audio part of the MAEB benchmark without a single epoch of training. The whole add-on is one 2.37 MB file, and the model carrying it outranks systems more than fifteen times its size.

Key takeaways

• Add-on size: 2.37 MB, with zero parameters trained at inference time
• MAEB (v1, audio-only, 19 tasks): a jump from 26th to 1st among 70 audio models, mean score 47.79 → 54.82
• MAEB (v1, full 30 tasks): a jump from 16th to 1st, mean score 49.31 → 54.40
• Models left behind: Qwen2-Audio-7B (8.40B parameters) and LCO-Embedding-Omni-7B (8.93B)
• The submission to the MTEB repository (PR #5775) remains open, so the results have not yet been verified by the benchmark team

Let us picture the situation

A team is building a search tool over an archive of call-centre recordings and needs one vector of numbers per recording, able to surface conversations with a similar emotional tone. EmbeddingGemma 2 looks ideal, because a single model handles text and audio in the same space, so a text query can land on a recording.

The problem only shows up in the numbers. On CREMA_D, a task measuring emotion recognition in speech, the model is correct in 19.7 percent of cases.

But a question arises: is the emotional information genuinely absent from the model, or does it simply never reach the output?

The answer is that it exists and gets filtered out along the way. That observation is the whole basis of the construction described below — with it attached, the same frozen model reaches 40.16 percent on CREMA_D.

What exactly does this head do?

The add-on is called embeddinggemma-2-nullspace-audio and it changes not a single weight of the base model. It acts only on audio input, and for text, image and video the computation path stays exactly as it was in the original.

The mechanism has three stages. First, two intermediate layers are pulled from the audio tower — a 12-layer Conformer encoder — instead of the final output: layer 2 (mean and standard deviation over time, 2048 dimensions) and layer 6 (mean only, 1024 dimensions). Each block is then projected through a frozen PCA matrix down to 128 dimensions, and the two are joined into a single 256-dimensional unit vector^?unit vector: a vector of length 1, carrying direction only and no information about magnitude. That vector is added to the base 768-dimensional embedding.

The choice of layers 2 and 6 is not arbitrary. According to the layer-wise analyses of speech encoders cited in the model card, different audio attributes peak at different depths. Early layers carry speaker gender, timbre and pitch, while middle layers carry emotion. The final layers, tuned for text alignment, filter that out, because matching audio to a description does not require it.

Why does this not break text embeddings?

Here lies the most interesting part of the idea. The model ends with a projection from 512 to 768 dimensions, so every text embedding sits inside a 512-dimensional subspace of those 768. The remaining 256 directions are never used by text.

The audio information is injected into exactly those 256 unused directions. Because they are orthogonal to the text subspace, adding anything along them changes no text embedding and cannot alter text-to-text similarities. That is a mathematical property, not an empirical result.

Cross-modal retrieval^?Cross-modal retrieval: retrieval in which the query and the result take different forms, for example text finding a recording confirms it. On the Google SVQ set, averaged over 26 locales, scores shift by at most roughly 0.3 points for audio-to-text retrieval and about 0.05 points in the other direction.

What do these names mean?

• Embedding — a list of numbers describing the content of a recording or a sentence. The equivalent of a fingerprint that makes it possible to search for similar things.
• Null space — the set of directions a given operation never uses. It can be compared to unused columns in a table, where data may be written without disturbing the rest.
• PCA — a method for finding the few directions along which data varies the most. It allows 2048 numbers to be squeezed into 128 without losing what matters.
• MAEB — the Massive Audio Embedding Benchmark, a set of 30 audio tasks hosted on Hugging Face that measures the quality of audio embeddings.
• Conformer — a type of audio encoder combining convolutions with an attention mechanism, used here as a 12-layer audio tower.

These pieces add up to one trick: PCA extracts from the Conformer layers what MAEB measures, and the null space provides somewhere to store it without harming the rest of the model.

Why not simply fine-tune the model?

The obvious approach is to fine-tune the audio tower on the tasks that underperform. That costs GPU time, requires a training set, and every change of weights means re-checking whether the model has damaged its own text-to-audio alignment.

There is also a second, harder problem. A model like this usually feeds a vector database^?vector database: a database that stores embeddings and searches by vector similarity instead of exact matching already holding millions of computed embeddings. Changing the weights invalidates all of them, so the entire archive has to be recomputed from scratch.

A null-space head^?head: a lightweight module attached to a model's output, leaving its weights unchanged sidesteps both problems. Text embeddings stay identical bit for bit, so the text side of the database needs no recomputation, and the add-on weighs about as much as a medium-sized photograph. One restriction does appear: audio vectors require the full 768 dimensions and cannot be shortened by Matryoshka truncation to 512, 256 or 128.

Is it working yet?

Yes as a downloadable model, no as a confirmed result on the official leaderboard. The files and a run script have been publicly available on Hugging Face since 28 October 2026, and the model can be launched with a single command. The submission adding it to the MTEB repository, PR #5775 opened on 24 October 2026, was still open and unmerged at the time of checking.

The model card itself flags two caveats. The method variant and the injection strength were selected after looking at test scores, so the numbers should be treated as optimistic. Several tasks clearly regress — SIBFLEURS drops by 35.9 percentage points and NMSQAPairClassification by 19.5.

Why does it matter?

The result undercuts the assumption that embedding benchmarks are won by scale. The gap between 26th and 1st place turned out to be a question of access to information the model already held, rather than its absence. For teams running on their own hardware, it suggests that sub-billion-parameter models are sometimes underrated, and that examining what their final layers discard can be cheaper than swapping in a larger model.

Table: size versus position in the audio part of the MAEB benchmark.

What next?

• Merging PR #5775 into the MTEB repository will decide whether the ranking reaches the official leaderboard and whether an independent recomputation confirms the numbers
• The model card notes that an injection strength of 1.0 was the highest value tested, so the optimum may lie higher
• The method relies only on frozen intermediate layers, so it can carry over to other multimodal embedding models with a dimension-narrowing projection

Sources

• Hugging Face — BrunoHays/embeddinggemma-2-nullspace-audio
• Hugging Face — google/embeddinggemma-2
• GitHub — model: add BrunoHays/embeddinggemma-2-nullspace-audio (PR #5775)
• arXiv — Layer-wise Analysis of a Self-supervised Speech Representation Model
• arXiv — EmbeddingGemma: Powerful and Lightweight Text Representations