A clever trick makes big language models generate text faster without changing output. A small draft model proposes several tokens that the large model verifies in parallel. It cut latency substantially.
Draft and verify
Two models cooperate. A small model guesses ahead; the large one checks. Correct guesses are kept.
Parallel gains
Speed rises. Multiple tokens are confirmed at once. Latency drops.
Same output
Nothing changes. The final text matches the large model alone. Quality is preserved.
Cheap drafter
Overhead is small. The draft model is lightweight. Net cost falls.
Wide adoption
Uptake spread. Serving systems integrated the method. It became common.
Active variants
Refinement continues. Many speedup schemes followed. The area is busy.
The bottom line
Speculative decoding uses a small draft model to propose tokens that a large model verifies in parallel, cutting latency without changing output. It was widely adopted. Refinements keep coming.