llama-modchip

llama-modchip is an open-source set of ggml/llama.cpp kernel patches, quantization transcoders, and runtime tools that run sparse Mixture-of-Experts (MoE) checkpoints entirely in system RAM on x86_64 (AVX-512 / AVX2) and ARM64 (NEON / SVE / Apple Metal) processors, with no GPU.

It fits the 284B DeepSeek-V4-Flash model (13.17B active parameters) into 64 GB of RAM via 1.71 bits-per-weight quantization (56.56 GiB on disk), and runs 30B models (e.g. Nemotron-3.5-30B-A3B, Qwen3-30B-A3B) on 16–32 GB laptops. Reported throughput is 44.4 tok/s for 284B prefill and 25–111 tok/s for 30B decode, depending on the CPU.

Key techniques include multi-level register/cache tiling for SIMD kernels, register-stationary FlashAttention, Fractal-64 quantization (64-weight block), and self-speculative decoding with a lightweight Stockfish-PVS drafter plus near-instant agent context restore from /dev/shm (~13 ms). Access is via CLI (./modchip chat), a local web UI, and Claude Code integration. License: Apache-2.0.