Attention was the bottleneck; a clever memory strategy broke it. A 2022 algorithm reordered the computation to avoid writing huge intermediate matrices to slow memory. It made long-context models dramatically faster without changing their output.
Memory, not flops
The insight is about bandwidth. Attention was limited by memory traffic, not arithmetic, so the fix minimizes reads and writes. It fuses operations to stay on fast chip memory.
Exact, not approximate
Nothing is sacrificed. The method computes the same attention as before, only faster and with less memory. Accuracy is untouched.
Longer contexts
Windows expanded. Cheaper attention let models handle far longer inputs, from books to codebases. Long-context applications flourished.
Hardware awareness
Design met silicon. The algorithm is tuned to the memory hierarchy of modern accelerators. Co-design of software and hardware paid off.
Rapid adoption
Uptake was swift. Major training frameworks integrated it within months. It became a default component.
Successive versions
Improvement continued. Later versions squeezed more speed from newer chips. The line of work keeps advancing.
The bottom line
FlashAttention removed the memory bottleneck in Transformers without approximating. It enabled far longer contexts and faster training. The technique is now standard in large-model infrastructure.
