Introduction to frame-autoregressive video models

A reading list and collection of resources for building up to frame-autoregressive video generation and world models. The progression goes: autoregressive transformers, then flow matching for images, then flow matching for video, and finally combining both with KV caching for efficient frame-autoregressive generation.

read essential reading   skim worth skimming   bonus for deeper exploration   code implementation

1. Decoder-only transformer + KV caching

Build a decoder-only transformer from scratch and understand KV caching for efficient autoregressive generation.

• tutorial ARENA Chapter 1.1: Transformer from Scratch -- step-by-step implementation of a transformer
• code minGPT / nanoGPT -- Karpathy's clean, minimal GPT implementations
• bonus Coding the KV Cache in LLMs from Scratch -- Sebastian Raschka's hands-on walkthrough of KV caching

2. MNIST & class-conditional MNIST using flow matching

Learn diffusion and flow matching by generating images on MNIST. The references below trace the evolution from the original diffusion formulation to modern rectified flow matching.

• bonus Denoising Diffusion Probabilistic Models -- the original DDPM paper
• bonus Denoising Diffusion Implicit Models (DDIM) -- deterministic sampling; for a while everyone used this
• bonus Latent Diffusion Models -- computational trick: run diffusion in a learned latent space to make everything much faster
• read Scalable Diffusion Models with Transformers (mmDiT) -- the transformer architecture and conditioning mechanism that everyone is using now
• read Scaling Rectified Flow Transformers for High-Resolution Image Synthesis -- nowadays everyone is doing rectified flow matching
• code Rectified flow matching on MNIST -- beautiful minimal implementation on MNIST
• code minRF: Rectified flow matching on CIFAR -- minimal implementation on CIFAR

3. Pong using flow matching

Move from images to video: generate pong games using flow-matching-based video models.

• skim WAN: Scalable Bidirectional Text-to-Video Generation -- bidirectional text-to-video generation
• skim Diffusion Forcing -- bridges autoregressive and diffusion models for sequence generation
• bonus CausVid -- causal video generation (advanced)
• bonus Self-Forcing -- training autoregressive video models without teacher forcing (advanced)
• code Minimal diffusion forcing implementation -- a toy implementation to get started

4. KV caching for frame-autoregressive transformers

Combine KV caching (from step 1) with flow-matching video generation (from step 3) for efficient frame-autoregressive world models.