This course will guide learners through a practical Transformer implementation in PyTorch: from token representations and attention to a full trainable model. Chapters and lessons will be added in later stages.
You will learn why the Transformer architecture was created, the core concepts of sequences and tokens, and the differences between encoders, decoders and decoder-only models.
You will learn the practical PyTorch foundations needed to implement a Transformer: tensor shapes, broadcasting, axis transforms, modules, masks, padding and GPU execution.
You will build intuition for self-attention, learn the roles of Query, Key and Value, derive scaled dot-product attention, and prepare to implement a single attention head in PyTorch.
You will learn why the Transformer uses multiple attention heads, how Q, K and V projections work across heads, how to merge head outputs, and how to build a MultiHeadAttention module in PyTorch.
You will assemble a complete Transformer block from residual connections, LayerNorm, the feed-forward network and attention into a stable implementation pattern.
You will learn how token IDs become vectors, how position information is added, and which masks are needed for padding and autoregressive decoding.
You will assemble a mini-GPT from embeddings, a decoder-only block stack, a language modeling head, and a full forward pass returning logits and loss.
You will move from preparing training sequences through the loss function and PyTorch training loop to validation, checkpoints, and core language-model metrics.
This chapter shows how to run a language model in generation mode: from the autoregressive loop through sampling, KV cache, and quality debugging.
This chapter covers practical extensions of the classic Transformer: RoPE, FlashAttention, MQA/GQA, and techniques for fine-tuning and scaling modern models.
