The paper says the speedup comes from running more computation concurrently, so total arithmetic operations can rise, and that the method is not helpful for configurations where additional computation resources are not available.

The exact words it rests on

latency is improved through increased concurrency at the cost of an increased number of arithmetic operations. Thus, our method is not helpful for configurations where additional computation resources are not available.

What the source said when we opened it, on 30 Oct 2026.

The source

Fast Inference from Transformers via Speculative Decoding
arXiv (Leviathan, Kalman, Matias) · 2022-11-30

Open the sourcearchived copy

Checked

Checked by the notis newsroom on 30 Oct 2026, against the source above.

In the story

How speculative decoding lets a small model draft and a large one check 30 Oct 2026