An independent project has turned EmbeddingGemma 2 — Google DeepMind's multimodal embedding model — into a speech recognition system. An hour of audio is transcribed in 12.6 seconds on a base MacBook Air, on device, even though the model was never trained for transcription.
Key takeaways
- One hour of audio in 12.6 s on a base M3 MacBook Air — 288× faster than real time
- Only a small "reader" is trained: under two hours on eight H100s
- The transcript is built by picking the nearest of roughly 229,000 English words
- Word error rate around 5% against 2–3% for leading speech models
- EmbeddingGemma 2 has 740M parameters, 300M of them in the audio encoder, under Apache 2.0
Reading words out of a vector space
EmbeddingGemma 2 from Google DeepMind maps text, images, video and audio into a single shared 768-dimensional space — a spoken word lands close to the same word written down.
The project took that literally: the audio encoder is untouched, and a small reader?reader: A thin layer translating the vector of an audio slice into a vector in text space. The only component this project trained at all. moves a short slice of the recording into the model's text space. The nearest of roughly 229,000 English words wins, and the slices add up to a transcript.
The diagram shows the order of steps described by the authors. It does not depict the audio encoder's architecture or how the reader is trained.
Training cost: two hours on eight cards
Training touches only the reader — the audio encoder stays frozen. It took under two hours on eight H100 cards and used nothing but openly licensed English speech. That is the entire budget for giving an embedding model a new output modality.
Speed against accuracy
On the same MacBook Air the project reaches 288× real time against roughly 150× for IBM Granite Speech 5.0 TurboCTC, which runs on the GPU via MLX. Everything runs on the Neural Engine, with no audio leaving the machine.
| Solution | Hardware | Multiple | One hour of audio |
|---|---|---|---|
| Embedding Gemma Transcribe | Neural Engine, M3 MacBook Air 8 GB | 288× | 12.6 s |
| IBM Granite Speech 5.0 TurboCTC | GPU (MLX), same Mac | ~150× | 25.7 s |
| Parakeet TDT 0.6B v3 (FluidAudio) | Neural Engine, M4 Pro | 156× | — |
| Parakeet TDT 0.6B v3 (FluidAudio) | Neural Engine, M5 Pro | 335× | — |
| Phonon-2 (FluidAudio) | M5 Pro | 480× | — |
The authors are explicit about it: Parakeet TDT 0.6B v3 from NVIDIA and its fine-tune Phonon-2 were measured on different hardware. The price of the speed does show up in quality — around 5% word error rate against 2–3% for leading speech models.
Why it matters
A new modality in AI usually means a new model, a new dataset and a new training budget. Here the cost came down to two hours on eight cards, because the hard part — understanding audio — already sat inside the embeddings. Five percent error is not production-ready, but the route to the result is repeatable for other pairs of modalities, not just speech and text.
What's next
- The authors have announced another iteration with a better-trained reader to close the 5% gap
- The app ships as version 1.6 for macOS 14+ on Apple Silicon, with 715 MB of models
- The project supports English only and is not affiliated with or endorsed by Google
Sources
- Embedding Gemma Transcribe — We turned EmbeddingGemma 2 into a speech-to-text model
- Hugging Face — google/embeddinggemma-2 model card
