NCP-ArchPreview Hits OLMo-3's Loss on Half the Training Tokens

Shanghai AI Lab and the LUMIA Lab at Shanghai Jiao Tong University have released NCP-ArchPreview — an 8.94B-parameter language model that predicts the next "concept" in a learned latent space alongside the next token. It reaches OLMo-3-7B’s final pretraining loss while consuming 51.3% of that model’s token budget. The weights ship under Apache-2.0.

Key takeaways

8.94B parameters, trained on 5.73T tokens from the Dolma-3 corpus
OLMo-3-7B’s loss reached on 51.3% of the token budget — 1.95x convergence
Benchmark macro-average: 49.04 against 46.59 for OLMo-3-7B
+5.99 percentage points on GSM8K, +4.28 on HumanEval
Both stages on Hugging Face under Apache-2.0

51.3%of OLMo-3-7B’s token budget is enough for NCP-ArchPreview to reach its final pretraining lossNCP-ArchPreview model card (Stage 1)

Three modules instead of one stack

The architecture splits into three parts. Mean-pooling four consecutive encoder states gives one continuous concept, and a separate module predicts the next concept before the decoder returns to the token level.

Token path

Token encoder

Mean-pool four consecutive states

Concept path

Concept Module

Token path

Token decoder

Next-token distributionAllow

\dots ▶ Symbol meaning \dots length of the concept sequence \dots length of the token sequence \dots token states mean-pooled into one concept

What that number does not mean

The model card raises the caveat itself: the convergence comparison measures tokens needed to reach a given loss. It does not measure training wall-clock time or generation throughput. At 85% of the standard compute budget the model approaches the loss of a strictly parameter-aligned 8.9B baseline — a narrower and more honest version of the same claim.

Seventeen million parameters as an interface

The learned latent space stays useful after pretraining. Updating the VQ module alone — 17M parameters, with the token backbone frozen — gives a lightweight interface for domain adaptation. Injecting concept representations into the DFlash2 drafterDrafter: A small, fast model that proposes several upcoming tokens at once, which the main model only accepts or rejects. raised the mean accepted length by 4.17% with negligible overhead.

Why it matters

The idea of letting a model plan above the token level keeps returning and usually stalls at small scale. Here the scale is 8.9B parameters and 5.73T tokens, with open weights and open evaluation code. If the result holds under independent checking, some of the gain the field currently buys with a bigger data budget can be taken from architecture instead.