NCP-ArchPreview

NCP-ArchPreview is an 8.94B-parameter language model developed by Shanghai AI Lab and LUMIA Lab (Shanghai Jiao Tong University). It goes beyond traditional next-token prediction by introducing Next Concept Prediction (NCP): alongside the next token, the model also predicts the next 'concept' — a learned latent representation spanning roughly four tokens.

The architecture has three modules: a Token Encoder (16 layers) that processes input text, a Concept Module (8 layers) that predicts next concepts using product quantization with 32 codebooks, and a Token Decoder (16 layers) that generates token distributions informed by concept predictions. Cross-module residual connections let information flow while preserving causality; generation keeps the standard next-token interface.

The Stage 1 release was pretrained on the Dolma 3 Mix dataset (5.73 trillion tokens) with three joint objectives: next-token prediction, next-concept prediction, and vector-quantization fitting. It achieves an Overall AVG of 49.04 (versus 46.59 for OLMo-3-7B), with notable gains on math and code tasks. Weights are Apache 2.0 licensed.