Transformers: Research-Level Mastery

Module 1: Attention as Memory

• Attention as Soft Lookup in a Key-Value Store
• Inductive Biases: What Transformers Assume About the World
• Universal Approximation and Expressivity of Transformers

Module 1: Quadratic Complexity and Flash Attention

• The O(n²) Problem: Why Long Sequences Break Standard Attention
• Flash Attention: IO-Aware Exact Attention via Tiling
• Sparse, Linear, and Efficient Attention Variants

Module 1: BERT GPT T5 and Beyond

• BERT: Bidirectional Encoders and Masked Language Modeling
• GPT: Autoregressive Decoders and In-Context Learning
• T5 and the Text-to-Text Unification Framework

Module 1: Building Transformers from Scratch

• Implementing Scaled Dot-Product Attention in PyTorch
• Building the Full Encoder Stack End-to-End
• Debugging Transformers: Gradient Flow, NaN, and Attention Collapse

Module 1: The Research Frontier

• Vision Transformers: Patches, Position, and Cross-Modal Attention
• State Space Models: Mamba and the Challenge to Transformers
• Mechanistic Interpretability: What Attention Heads Are Actually Doing