Attention Variants and Memory Efficiency

Overview This post is divided into eight parts; they are: • The Checkpoint Fixes the Attention Architecture • A Closer Look at Attention Memory • Multi-Head Attention • Multi-Query and Grouped-Query Attention • Sliding-Window and Mixed Attention • Multi-Head Latent Attention • What the Inference Runtime Can Change • From Cache Size to Serving Capacity The Checkpoint Fixes the Attention Architecture The checkpoint fixes the model configuration; the inference engine does not change it.