A GGUF file consists of a header with a magic number and format version, a section of key-value pairs holding metadata (architecture, hyperparameters, tokenizer configuration), a tensor-info table (name, shape, data/quantization type, offset) and the actual tensor data stored as quantized blocks or in full precision. An inference engine (e.g. llama.cpp) memory-maps the file, reads the metadata, reconstructs the model graph and computes directly on the quantized blocks, dequantizing values on the fly on CPU or GPU.
Earlier formats (GGML/GGMF/GGJT) were unstable and required external files and manual specification of hyperparameters, which made distribution and backward compatibility difficult. GGUF consolidates weights, metadata and tokenizer into a single self-describing file, providing stability, extensibility and simple, portable deployment of quantized models.
Start of the file with the 'GGUF' magic number and format version, identifying the file and its layout.
Standardized set of key-value pairs describing the model architecture, hyperparameters and tokenizer configuration.
Table describing each tensor: name, shape, data/quantization type and offset within the file.
The actual model weights stored as quantized blocks or in full precision, aligned for memory mapping.
GGUF replaced the GGML/GGMF/GGJT formats in llama.cpp, introducing a self-describing, extensible binary file.
Weight quantization level and scheme determining the trade-off between file size, memory usage and model quality.
GGUF and llama.cpp were designed for efficient CPU inference using vector instructions and memory mapping.
GGML-based engines support GPU acceleration (layer offloading), speeding up inference of GGUF models.
