Qwen3-4B

Qwen3-4B is an open (Apache 2.0) large language model (LLM) from the Qwen3 series developed by the Qwen team at Alibaba Cloud, released on 29 April 2025. It is a dense model with 4.0B parameters (3.6B non-embedding), built on a Transformer architecture with Grouped Query Attention (32 query heads and 8 key-value heads), SwiGLU, RoPE and RMSNorm with pre-normalization, QK-Norm stabilization and 36 layers.

A defining feature of the Qwen3 series is its hybrid operation: the model switches between a "thinking" mode for complex reasoning, math and coding, and a non-thinking mode for fast, general-purpose dialogue. The native context window is 32,768 tokens and can be extended to 131,072 tokens using the YaRN technique. The model supports over 100 languages and dialects and tool use in both modes.

Qwen3 was trained on roughly 36 trillion tokens across 119 languages, and post-training used a four-stage pipeline: long chain-of-thought cold start, reasoning-based reinforcement learning, thinking-mode fusion, and general reinforcement learning. According to the developer, even the small Qwen3-4B can rival the results of Qwen2.5-72B-Instruct. Weights are available under the Apache 2.0 license, enabling local deployment, fine-tuning and commercial use.