GPT-BERT

GPT-BERT is a hybrid language model developed by the Language Technology Group at the University of Oslo. It merges two learning paradigms within a single transformer stack: causal language modeling (as in GPT) and masked language modeling (as in BERT). As a result, the same model can operate both as an autoregressive (generative) model and as a masked (understanding and classification) model.

The model was introduced in the paper "GPT or BERT: why not both?" (Charpentier and Samuel, 2024). In the 2024 BabyLM Challenge — out of 31 submissions — the hybrid causal-masked architecture outperformed masked-only and causal-only approaches. The submission was trained on two tracks: Strict (100M words, a ~119M-parameter variant) and Strict-Small (10M words, a ~30M-parameter variant).

Both variants are 12-layer transformers (hidden size 768 for the base variant, 384 for the small one). The models were trained on a 1:1:1 mixture of corpora (BabyLM, FineWeb-Edu and Cosmopedia) with a BPE tokenizer. The code is released under the MIT license in the ltgoslo/gpt-bert repository, and the weights are available on Hugging Face.