A new architecture questions whether attention is essential. State-space models process sequences with a recurrence that scales linearly rather than quadratically. In 2023 one such model matched Transformers on language while handling very long sequences efficiently.
Linear scaling
The appeal is cost. Compute grows in proportion to sequence length, not its square. Very long inputs become tractable.
Selective memory
The model chooses what to keep. A gating mechanism decides which information to propagate through the state. It focuses capacity where it matters.
Long-range strength
Distant dependencies are captured. The recurrence carries signal across thousands of steps. Tasks with long context benefit.
Competitive quality
Performance holds up. On several benchmarks the model rivals attention-based peers. The gap has narrowed sharply.
Hardware efficiency
Throughput is high. A hardware-aware scan makes training and inference fast. It runs well on modern accelerators.
An open question
The field is watching. Whether such models fully displace attention remains unsettled. Hybrids may capture the best of both.
The bottom line
State-space models offer linear-time sequence modeling with selective memory. They match Transformers on language while scaling to long contexts. They are a serious architectural alternative.
