Explainers
One Token’s Journey Through the Stack
Vaswani et al. (arXiv 1706.03762) followed end to end — one token from embedding through positional encoding, eight heads, the residual stream and cross-attention, with a live attention playground.
Run it → jsxQuery, Key & Value
An interactive walk through the three self-attention projections, from intuition to the full scale-and-softmax pipeline.
Run it → jsxAttention Sinks Induce Gradient Sinks
Stepped paper explainer (Chen & Yao, 2026): attention mass piling on the BOS token creates a matching sink in the backward pass.
Run it →Unread — 5
MiniMax Sparse Attention
Production blockwise sparse attention from a frontier lab, with kernel-level design and ultra-long-context speedups.
arXiv 2606.13392 paper · Apr 2026Attention Sink in Transformers: A Survey
Synthesises 200+ studies on why sink tokens emerge — the direct sequel to your attention-gradient-sinks explainer.
arXiv 2604.10098 paper · Feb 2024A phase transition between positional and semantic learning in dot-product attention
A solvable statistical-physics model shows a single attention layer undergoes a sharp transition between positional and content-based learning — a rare exact result.
arXiv 2402.03902 paper · Jun 2017Attention Is All You Need
The paper that introduced the Transformer, on which essentially all modern LLMs are built — Shazeer is a co-author.
arXiv 1706.03762 paper · Feb 2020GLU Variants Improve Transformer
A short, sharp paper introducing the gated-linear-unit variants (SwiGLU) now used in the feed-forward layers of most state-of-the-art LLMs.
arXiv 2002.05202