1. The source model processes the input context and produces a KV cache (key-value states for every layer and attention head). 2. Offline, a closed-form linear mapping (ridge regression) is calibrated per head to transform the source model's KV states into the target model's KV space. 3. Source layers with the highest predictive power for the target layers are selected. 4. Positional encodings are removed so the mapped states are position-independent and reusable. 5. At inference time, instead of running a full prefill, the target model receives the mapped KV states and proceeds directly to decoding. For challenging model pairs, nonlinear variants can improve accuracy.
In production deployments that dynamically switch between differently sized models within one family (e.g. to balance cost and quality), each switch requires the new model to reprocess the entire context (prefill) from scratch. This increases latency (time-to-first-token) and GPU usage. Cross-model KV cache transfer removes this redundancy by porting the already-computed context.
A closed-form linear mapping (ridge regression) learned separately for each attention head, transforming the source model's KV states into the target model's KV space.
A mechanism for selecting the source-model layers with the highest predictive power for the target-model layers.
Removing positional encoding from the KV states so the mapped states are position-independent and reusable.
A small dataset used offline to fit the coefficients of the linear mapping.
The transfer works for models sharing architecture and tokenizer within a family; it does not port freely between unrelated models.
For some model pairs accuracy drops below the full 73–98% range; the linear mapping may be insufficient.
To make mapped states reusable, positional encoding must be removed, which requires compatibility with the model's positional scheme.
First LLM serving engine supporting cross-model prefix KV cache reuse between a base model and its adapters.
Discovery of linear structure in cross-model KV pairs and introduction of a closed-form per-head mapping that replaces full prefill.
The linear mapping is applied independently per attention head and layer, so the transfer is fully parallelizable — unlike sequential prefill.
The technique targets GPU-based LLM serving; the mapping itself is a lightweight linear operation (matrix multiply) well suited to tensor cores.
