The idea that the future of agentic AI belongs to small language models (SLMs) is no longer just an academic provocation. NVIDIA Research laid it out in a June 2025 paper, revised in September, and 2026 is bringing real-world confirmation — from Hugging Face's open SmolLM3 to Gemma 3, which Google now runs locally on a palm-sized board.
Key takeaways
- NVIDIA Research (arXiv:2506.02153, June 2025, revised September 2025) argues SLMs are "sufficiently powerful, inherently more suitable, and necessarily more economical" for most calls in agentic systems
- Hugging Face SmolLM3 is a fully open 3B-parameter model with a 128k-token context, support for six languages, a reasoning mode and tool calling
- Google Gemma 3 270M (270 million parameters) runs entirely on the Coral Board — a Coral NPU on RISC-V architecture, 2 GB RAM, 1 TOPS, no cloud
- NVIDIA proposes an LLM-to-SLM agent conversion algorithm and heterogeneous architectures: SLMs for routine work, LLMs for the hardest reasoning
From "bigger is better" to "good enough"
For several years, language-model progress ran on a single rule: more parameters, more data, more compute. NVIDIA Research challenges its universality. In "Small Language Models are the Future of Agentic AI," the authors argue that in agentic systems — where a model performs a narrow, repetitive set of tasks with little variation — a large general-purpose model is usually overkill. SLMs are, as they write, "sufficiently powerful, inherently more suitable, and necessarily more economical" for most calls.
The economic argument is central. An agent does not hold a free-flowing conversation — it parses data, invokes tools, formats responses. That does not require a model that has read all of world literature. NVIDIA proposes a concrete conversion algorithm: moving an agent's routine calls from a large model to a specialized small one, alongside heterogeneous architectures in which an SLM handles the everyday work and an LLM is called only for the hardest reasoning.
SmolLM3: what an open 3B model can do
The clearest illustration of how far the frontier has moved is SmolLM3 — released by Hugging Face in July 2025. It has 3 billion parameters and was trained on roughly 11 trillion tokens, yet handles a context of up to 128,000 tokens, six European languages, and two modes: fast answers and explicit step-by-step reasoning. It also supports tool calling, essential for agentic work.
The numbers show where the balance between cost and capability lies. Turning on the reasoning mode lifts the score on the hard AIME 2025 math test from 9.3 to 36.7 percent — nearly fourfold. On general benchmarks SmolLM3 outperforms same-class models such as Llama-3.2-3B and Qwen2.5-3B, and closes in on larger 4-billion-parameter models. The entire training recipe is public, and the model was built on 384 H100 GPUs in 24 days — a fraction of the cost of training a frontier model.
The real edge: a model that fits inside the device
The second driving force is locality. In late May 2026 Google showed the Coral Board — a palm-sized single-board computer on which Gemma 3 270M (270 million parameters) runs entirely on-device, with no cloud connection. At its heart is the open Coral NPU built on RISC-V architecture, with 2 GB of memory and 1 TOPS of compute. Demos at Google I/O included real-time translation and voice control. It targets devices where the cloud is a non-starter: headphones, AR glasses, smartwatches. The Gemma 3 family was designed so that — as Google advertises — it can run "on a single GPU or TPU."
What a small model won't do
Enthusiasm does not suspend physics, though. NVIDIA itself concedes that where free-flowing, general conversation matters, the advantage still lies with large models — which is why the researchers recommend heterogeneous systems rather than replacing LLMs outright. The SmolLM3 data show it too: the 4-billion-parameter Qwen3-4B generally scores higher. A small model has to be tuned to a specific task — its strength is specialization, not versatility. The trade-off is clear: lower cost, lower latency and privacy in exchange for a narrower scope and the need for adaptation.
Why it matters
The shift toward small models changes the economics of the whole industry. If a large share of calls in agentic systems can be handled by a model that fits on a single GPU — or even inside the user's device — the cost of running those systems drops by orders of magnitude, and with it the dependence on massive data centers and cloud providers. For companies, that means building agents without paying for every frontier-model API call. For users, it means real privacy, because data never leaves the device, plus operation that works offline and without the latency of a server round trip. The openness of models like SmolLM3 adds a third dimension: full control and the ability to audit. This is not a forecast of the end of large models — those stay essential where general intelligence counts. It is a market maturing, one that matches model size to the task instead of reaching for the biggest option by default.
What's next
- The Coral Board with Gemma 3 270M is expected to ship in summer 2026, though Google has not yet announced a price
- NVIDIA invited critique of its position and committed to publishing incoming correspondence, so the debate over the limits of SLMs is only just unfolding
- Hugging Face released SmolLM3's full training recipe and checkpoints, lowering the barrier for further open 3-billion-parameter models
Sources
- NVIDIA Research — Small Language Models are the Future of Agentic AI
- arXiv — Small Language Models are the Future of Agentic AI
- Hugging Face — SmolLM3: smol, multilingual, long-context reasoner
- The Decoder — Google launches a tiny board that runs Gemma 3 locally
