RouteCrack
How it works
Stage 1 — draft: the RouteCrack predictor cascade (rc_predictor.h) proposes next tokens without a draft model, using two-zone in-context n-gram matching (16K hot zone + 24K prefix), an 8-level FNV-64 hash cascade (k=10..1), PPM, skipgrams, a macro codebook (macro_codebook.bin, 11,382 chains), and an online session trie. Stage 2 — verify/prune: Stockfish-PVS v2 (rc_stockfish.h) scores drafts online, hashing context with 64-bit Zobrist keys and maintaining multi-horizon transposition tables (tt5/tt4/tt3). Every accepted token increments "wins" and boosts history (history gravity); every rejected token is recorded as a killer/refutation move and the entry is penalised. The horizon M is held conservatively at 1-5 tokens and extended to M=7 (batch of 8 tokens) only on a "singular extension" (true_consensus_hits >= 2 or a very long exact context match, confirmed in the transposition table and unrefuted). Profitability is governed by WallGain = A − 0.30·D and MambaGain = A − 0.50·rollbacks (A = accepted tokens, D = total drafts). The whole drafter working set (~192 KB) fits in L2 cache, giving <3 µs/step with zero L3 eviction.
Problem solved
Classic speculative decoding assumes a GPU where verifying a batch of D draft tokens costs nearly the same as generating one token, and it additionally requires a separate memory-hungry draft model. On CPU with sparse MoE and hybrid Mamba-2 this assumption breaks: each draft token has a real marginal cost (MoE expert routing ~+0.30·T1) and rejecting a token forces an expensive Mamba-2 recurrent-state rollback (restoring ~18.4 MB of state). RouteCrack addresses accelerating decoding on CPU without a separate draft model and without wasting compute on rejected drafts.
Components
Draftless speculative-token generator combining several statistical engines: in-context n-gram matching, an 8-level FNV-64 hash cascade, PPM, skipgrams, a macro codebook, and an online session trie.
Chess-engine-style verification and pruning layer: 64-bit multi-horizon Zobrist transposition tables (tt5/tt4/tt3), history gravity, killer/refutation moves, countermove table, 2-ply continuation history, and principal-variation-search late move reduction (PVS LMR). Its ~192 KB working set fits in L2.
In-context match scanner that splits the window into a hot zone (16K) and a prefix zone (24K); it computes multi-occurrence consensus (true_consensus_hits) that gates horizon extension.
Official
A set of 11,382 multi-token chains extracted offline; "latched" across subsequent tokens while the transposition table reports no refutation, enabling multi-token drafts.
Official
Implementation
Extending every match to M=7 wastes MoE expert-routing time (~+0.30·T1 per rejected token) and forces Mamba-2 rollbacks. The horizon should be extended only on a certified singular extension.
Rejecting a draft token forces restoring ~18.4 MB of recurrent state from the spec checkpoint and replaying it — unlike the zero-copy pointer decrement of a Transformer KV cache.
Hyperparameters (configurable axes)
Maximum draft length. Conservative base of 1-5 tokens; extended to M=7 (batch of 8 tokens) only on a singular extension.
Certainty threshold a draft must pass to be emitted.
Minimum number of distinct in-context occurrences agreeing on the continuation, required to extend the horizon.
Computational complexity
Time complexity: O(1) na krok draftu (amortyzowane). Space complexity: ~192 KB zbiór roboczy draftera (L2).
Execution paradigm
Draft length and acceptance depend on the match of the current context (input-dependent certification).
Draft candidates from the cascade are gated by a certainty threshold and by confirmation/refutation in the Stockfish multi-horizon transposition tables; horizon extension requires a "singular" certification.
Parallelism
The predictor runs sequentially per step, while draft verification is batched (across tokens). On CPU the batch cost grows linearly with the number of drafts (MoE routing), hence horizon pruning.
Hardware requirements
Designed CPU-first; the drafter working set fits in L2/L3, it uses AVX-512/AVX2, and it eliminates a separate draft-model forward pass.
The key assumption (the WallGain equation) concerns CPU, where every rejected draft token costs time. On GPU batch verification is nearly free, so the horizon-pruning advantage diminishes.