Hunyuan-Large

Hunyuan-Large is a large mixture-of-experts (MoE) language model developed by Tencent, released on 5 November 2024. It is one of the largest open-source Transformer-based MoE models: it has 389 billion total parameters, of which 52 billion are activated per token.

The model supports very long context — the pre-trained version up to 256K tokens and the instruct version up to 128K tokens. It is a text model that excels at language understanding and generation, logical reasoning, mathematics, coding and multilingual tasks (with an emphasis on English and Chinese).

Its key technical innovations include: large-scale synthetic-data training, a mixed expert-routing strategy, key-value (KV) cache compression, and expert-specific learning-rate scaling. The authors also studied MoE scaling laws and learning-rate schedules.

According to its technical report, Hunyuan-Large outperforms LLaMA 3.1-70B and achieves performance comparable to the much larger LLaMA 3.1-405B — despite activating only 52B parameters. The code and weights are released on GitHub and Hugging Face under the Tencent Hunyuan Community License Agreement; the model is also available via Tencent Cloud.