Qwen has released Qwen3.8-Flash-Next — a model with 125B parameters of which only 6B are active per request. The company describes it plainly as an early preview of the architecture heading into Qwen4. It takes text, images and video.
Key takeaways
- 125B total parameters, 6B active per request
- Of those, 51B are n-gram embedding parameters and 4B go to multi-token prediction
- 262,144-token native context, extensible to one million
- Hybrid attention: Gated DeltaNet combined with Qwen Sparse Attention
- qwen-community-1.0 licence, thinking mode on by default
Attention over micro-blocks, not tokens
Qwen Sparse Attention does not pick individual tokens — it works at the micro-block level. That cuts long-context latency, because the model does not walk the whole sequence token by token.
The second pillar is n-gram embedding. It scales the parameter count without the overhead a classic mixture of experts carries, which is where 51B parameters in embeddings alone come from. Training ran two optimisers side by side, Muon and AdamW, selected by weight category.
Of 125B parameters, only a fraction works on any one request:
Results: agents, code and the phone screen
Agentic tasks are the model’s strongest suit. It scores 84.5 driving an Android interface and 88.5 on real-world image understanding. Coding results are solid rather than record-breaking — 58.7 on DeepSWE 1.1 and 62.5 on SWE-bench Pro.
The contrast with its sibling is sharp. Qwen 3.8 27B, which we covered last week, is a dense model. Here, with four times the parameters, fewer than 5% of them do the work.
Simon Willison ran quantised builds on a DGX Spark. He settled on the 78.9 GB variant as the best compromise, which means the model fits on a single developer machine.
Why it matters
Previewing an architecture before the next generation ships is rare and genuinely informative. If Qwen4 really rests on micro-block sparse attention and n-gram embeddings rather than pure MoE, that is a turn away from the direction most labs took. Teams planning their own infrastructure get a year of lead time on hardware decisions.
What’s next?
- Qwen presents this architecture as the basis for Qwen4, with no launch date given
- The model runs on vLLM, SGLang and Transformers, so deployment needs no custom runtime
- The million-token context remains a claim — no public tests at that length yet
Sources
- Simon Willison — Qwen3.8-Flash-Next
- Hugging Face — Qwen/Qwen3.8-Flash-Next
