Evolutionary Model Merge

Evolutionary Model Merge is a method introduced by Sakana AI on March 21, 2024 that automates foundation model development by combining existing open-source models using evolutionary algorithms (CMA-ES). The method operates in two complementary spaces: Parameter Space (PS) — evolutionary mixing of weights from multiple models — and Data Flow Space (DFS) — evolutionary selection and ordering of layers from different source models. Both approaches can be combined to discover non-trivial, counter-intuitive combinations that human experts would unlikely find manually.

The first release produced three Japanese foundation models: EvoLLM-JP (a Japanese math LLM in 7B and 10B variants, merged from Shisa Gamma 7B v1, WizardMath-7B-V1.1 and Abel-7B-002, plus an EvoLLM-JP-A variant using Arithmo2-Mistral-7B), EvoVLM-JP (a 7B Japanese vision-language model merged from LLaVA-1.6-Mistral-7B and Shisa Gamma 7B v1), and EvoSDXL-JP (a Japanese-aware 4-step SDXL diffusion model).

EvoLLM-JP scored 52.0% (7B) and 55.6% (10B) on MGSM-JA, surpassing all Japanese LLMs below 70B parameters and previous 70B SOTA models. EvoVLM-JP achieved 19.70 ROUGE-L on JA-VG-VQA-500 and 51.25 on JA-VLM-Bench-In-the-Wild, beating the base LLaVA-1.6-Mistral-7B and prior Japanese VLMs. The work was published in Nature Machine Intelligence on January 27, 2025.