Robots Atlas>ROBOTS ATLAS
Artificial Intelligence

284B parameters on 64 GB of RAM, no GPU — just cache tiling

Sir Robot19 November 2026 · 7 min read
284B parameters on 64 GB of RAM, no GPU — just cache tiling

The open-source llama-modchip project runs the 284.33B-parameter DeepSeek-V4-Flash model on an ordinary desktop with 64 GB of RAM and not a single graphics card. Version 2.4.0 of the repository, last updated on 8 November 2026, reports 44.36 tokens per second for prompt processing and 9.06–14.7 tokens per second for generation. Models at this scale have until now been treated as the preserve of accelerator-equipped server rooms.

Key takeaways

  • DeepSeek-V4-Flash: 284.33B total parameters, 13.17B active, compressed from 76.87 GiB down to 56.56 GiB
  • Memory footprint: 57.61–58.9 GiB VmRSS including a cache for 256K tokens of context, with no disk access
  • Prefill sped up by 77.1 percent (25.0 → 44.36 tok/s) on a 16-core AMD Ryzen 9 9955HX, zero GPUs
  • The default mode is bit-exact: KL divergence of 0.0000 and 100 percent Top-1 agreement against the unmodified model
  • Apache-2.0 licence, 620 stars and 33 forks on GitHub (figures as of 9 November 2026)

Picture the following situation

A developer has a desktop with 64 GB of RAM and wants to run a model comparable to commercial services locally, without sending code outside. They download the official DeepSeek-V4-Flash file in UD-IQ1_S quantization. The file weighs 76.87 GiB, while Linux on a 64 GB machine makes roughly 62.5 GiB usable. The model does not fit.

The system starts reading missing fragments from disk on every token, or the process is killed by the OOM handler. Formally the model “runs”, but in practice it cannot be used.

But a problem appears: the boundary is not gradual — either everything fits in RAM, or performance drops by an order of magnitude. There is no soft landing here.

And it is exactly that threshold llama-modchip aims to move.

How does the mechanism work?

llama-modchip is a set of patches for the ggml and llama.cpp kernels, quantization transcoders and runtime tools. The repository splits the work into three independent layers.

The first layer is quantization. Rather than compressing the whole model uniformly, the project varies precision per tensor type: embeddings, layer zero and 43 shared experts stay in the GGUF file at Q4_K/Q5_K/F32 precision, while part of the attention projections and 256 routed experts move to its own Fractal-64 format. The result is 56.56 GiB at an average 1.71 bits per weight.

That last figure is not a claim but a plain quotient — file size in bits divided by parameter count:

…

The second layer is the kernels. Instead of processing one matrix column at a time, the patches unpack a weight superblock once into AVX-512 registers and reuse it across up to eight token columns at once. The tiling schedule spans four memory levels — ZMM registers, 32 KB L1D, 1 MB L2 and 64 MB L3 — and streaming expert weights with the _MM_HINT_NTA hint is meant to keep the actively used KV cache from being evicted out of L3.

The third layer is speculative decoding and state snapshotting. A small 1.64 MB n-gram drafter (named Stockfish-PVS v2) proposes upcoming tokens, while KV cache snapshots held in /dev/shm allow context to be restored in 13.1–32.6 ms.

Input
Official GGUF file
llama-modchip
Transcode selected tensors
Fits in 62.5 GiB of usable RAM?
YES
56.56 GiB on diskAllow
NO
76.87 GiB on diskDeny
llama-modchip
Cache-tiled kernelsAllow

This is a simplified flow based on figures from the project's documentation, not a full description of the implementation.

What do these names mean?

The project's documentation leans on abbreviations that mean little outside inference engineering.

  • bpw (bits per weight) — the average number of bits per model parameter after compression. The equivalent of bitrate in an audio file: fewer bits mean a smaller file and a greater risk of quality loss.
  • Mixture of Experts — an architecture where the model holds many specialised sub-networks but uses only a few per token. That is why DeepSeek-V4-Flash has 284B parameters in total yet computes on 13.17B.
  • prefill and decode — the two phases of a model's work. Prefill is the one-off read of the prompt, decode is generating the answer token by token. They have different bottlenecks, hence different tok/s figures.
  • KV cache — stored attention states for already-processed tokens, so they need not be recomputed. It grows with context length and occupies RAM alongside the model itself.
  • VmRSS — the amount of physical memory a process actually holds, read from the operating system. That is the number deciding whether a model fits, not the file size on disk.

These terms form one equation: the post-quantization size plus the KV cache must fit within available VmRSS, while throughput in both phases depends on how well the kernels exploit the processor's cache hierarchy.

What does it cost in quality?

That is the standard objection to aggressive quantization, and the project supplies concrete measurements against the unmodified model.

The default mode, labelled “Lossless”, evaluates all six routed experts per layer. Reported logit cosine similarity is exactly 1.00000, KL divergence: a measure of the difference between two probability distributions — zero means the model returns exactly the same probabilities as the original is 0.0000, and Top-1 and Top-3 agreement are both 100 percent. In other words: compression changes the size, not the outputs.

Only the optional modes trade quality for speed.

ModeDecode (tok/s)Logit similarityTop-1 agreement
Unpatched UD-IQ1_S (76.87 GiB)5.911.00000 (reference)100%
Lossless — Top-6, default9.06–9.951.00000100%
Top-5 with L2 quadrature9.68–9.870.9971495.8%
Top-5 + Gate-First SwiGLU10.150.99385100%
Turbo — Top-4 + SwiGLU10.12–10.740.99126100%

Prefill stays identical across modes, because expert pruning affects only the generation phase. Perplexity in Top-5 mode rises by 0.42 percent.

For the smaller NVIDIA-Nemotron-3.5-30B-A3B model, the project reports a 100/100 score on its own quality gate for both compression variants — 4.25 and 3.75 bits per weight — while the memory footprint drops from 21.41 to 15.40 GiB.

Why not simply buy a GPU?

The question is fair: accelerators exist to compute models, and system RAM exists for everything else.

The problem is capacity, not speed. Fitting 56 GiB of weights into card memory requires hardware far beyond the consumer shelf — typical gaming graphics cards carry 8–24 GB of VRAM. Adding 64 GB of DDR5 to an existing desktop costs orders of magnitude less.

Second: the MoE architecture suits CPUs precisely because it uses a small fraction of the weights per token. The project's documentation also describes a hybrid variant — a hot attention trunk on the GPU, cold experts in system RAM — which reaches 80.8 tok/s for a 30B model on a 16 GB card.

The third problem is one a GPU does not address at all: time to return to context. With coding agents that push dozens of tool definitions on every call, the cost is not generation itself but recomputing the same prompt. The project measures restoring such a state for a 40-tool Claude Code prompt at 13.1–32.6 ms.

Is it working already?

Yes — as publicly available open-source software, but with benchmarks supplied by the project itself. The Apache-2.0 repository is available on GitHub, version 2.4.0 was released in November 2026, and the latest code update is dated 8 November 2026. The project has gathered 620 stars and 33 forks.

Benchmarking a machine comes down to a single command, and the project asks for submissions:

Bash
./modchip bench

The caveat concerns verification. All quoted results come from the repository's own documentation and were measured on one specific configuration — a 16-core AMD Ryzen 9 9955HX. There are no published independent replications yet. The project itself solicits submissions through a reporting template and maintains a community leaderboard, which is an admission that figures on other hardware remain an open question.

Why does this matter?

The line between “a data-centre model” and “a desktop model” is shifting not through newly sold hardware but through how well code exploits the cache hierarchy of an existing processor. If the reported figures hold up on other machines, the assumption that running frontier models requires accelerators stops applying. That changes the calculation for anyone who cannot send data to the cloud.

44.36 tok/sprefill of a 284-billion-parameter model on a 16-core processor, with no graphics card at allllama-modchip v2.4.0

What next?

  • The project runs an open leaderboard and accepts benchmark submissions through a GitHub issue template — only that data will show whether the figures repeat beyond the Ryzen 9 9955HX
  • The documentation claims ARM64 support (NEON, SVE, Apple Metal) alongside x86-64, but the reported benchmarks cover AMD processors only — no published measurements exist for the other platforms
  • Identified risk: the expert-pruning modes lower Top-1 agreement to 95.8 percent, so the trade-off between speed and model fidelity sits with the user and is not presented as the default

Sources

Share this article