Speculative Decoding Explained: How AI Models Generate Text 2 to 3 Times Faster

Every extra second an AI model takes to answer is a second a user spends staring at a loading indicator. Standard LLM inference generates one token at a time, and each token requires a full forward pass through every layer of the model. For a large model, that adds up fast. Speculative decoding is the technique that changed this, and it did it without changing a single output the model produces, cutting response time by 2 to 3 times according to the original research [1].

This post walks through exactly how speculative decoding works, why it provably produces identical output to normal decoding, and how production systems like EAGLE and Medusa push the idea further.

If you would rather show this mechanism than describe it, here is a short video generation prompt built around the same approve-or-reject idea, written for a Wan-style video model:

bash
A small pastel-toned mechanical figure quickly stamps a row of paper slips on a conveyor belt, most slips glowing soft green as they pass a second, larger checkpoint, one slip glowing red and falling away. The belt visibly speeds up each time more slips glow green in a row. Clean scientific motion-graphics style, precise geometric shapes and soft pastel lighting, no readable text, no logos, no people, smooth steady camera pan following the belt.

What Speculative Decoding Actually Does

The core idea comes from two papers published within months of each other in late 2022 and early 2023. Google Research's "Fast Inference from Transformers via Speculative Decoding" demonstrated a 2 to 3 times speedup on a T5-XXL model with output identical to standard decoding [1]. DeepMind's "Accelerating Large Language Model Decoding with Speculative Sampling" showed a 2 to 2.5 times speedup on Chinchilla, a 70 billion parameter model, in a distributed serving setup [2].

Both papers describe the same core pattern, now generally called draft-then-verify. A small, fast draft model proposes several tokens ahead. The large target model then checks all of those proposed tokens in a single parallel forward pass instead of generating them one at a time.

Step 1: The Draft Model Proposes Tokens

The draft model is a much smaller, much faster version of the same general capability as the target model. It generates a short sequence of candidate tokens autoregressively, the normal slow way, but because it is small this is cheap. Typically 3 to 8 tokens are proposed per round.

Step 2: The Target Model Verifies in Parallel

Here is the part that makes the technique lossless rather than just an approximation. The large target model does not generate tokens one at a time to check the draft. It runs a single forward pass over the entire proposed sequence at once, since transformers can score multiple positions in parallel just as easily as one. That single pass produces the target model's true probability for every proposed token simultaneously.

Each draft token is then accepted or rejected using a rejection sampling rule: a token is accepted with probability equal to the ratio of the target model's probability for that token over the draft model's probability for it, capped at 1. If a token is rejected, everything after it is discarded, since it would have been conditioned on a token the target model would not have generated, and a corrected token is resampled directly from an adjusted version of the target distribution at that position.

This is the mathematical guarantee both original papers prove: the resulting output distribution is exactly identical to what you would get sampling from the target model alone, token by token, the slow way. Speculative decoding is not an approximation or a quality tradeoff. It is a way of computing the exact same result with fewer expensive forward passes through the large model.

Here is a simplified version of the draft-then-verify logic:

python
/code
import random

def draft_tokens(draft_model, prefix, k=4):
    """Small, fast model proposes k candidate tokens autoregressively."""
    tokens = []
    context = prefix
    for _ in range(k):
        token = draft_model.sample_next_token(context)
        tokens.append(token)
        context = context + [token]
    return tokens

def verify_tokens(target_model, prefix, draft_tokens):
    """Large target model scores all draft tokens in a single parallel forward
    pass, then accepts or rejects each one using the same rejection-sampling
    rule the original speculative decoding papers describe, so the final
    output distribution exactly matches sampling from the target model alone."""
    accepted = []
    context = prefix
    for token in draft_tokens:
        p_target = target_model.prob(token, context)
        p_draft = draft_model.prob(token, context)
        accept_prob = min(1.0, p_target / max(p_draft, 1e-9))
        if random.random() < accept_prob:
            accepted.append(token)
            context = context + [token]
        else:
            # Reject and resample this position directly from an adjusted
            # target distribution, then stop, since everything after a
            # rejection would have been conditioned on a wrong token anyway.
            corrected_token = target_model.sample_adjusted(context, rejected=token)
            accepted.append(corrected_token)
            break
    return accepted

The actual speedup you get depends heavily on the acceptance rate, meaning how often the draft model's guesses match what the target model would have picked anyway. A rough way to reason about expected speedup looks like this:

python
/code
def expected_speedup(acceptance_rate: float, draft_tokens_per_step: int, cost_ratio: float) -> float:
    """Rough expected speedup from speculative decoding, given the fraction of
    draft tokens typically accepted, how many tokens the draft model proposes
    per step, and the draft model's cost relative to the target model.
    Matches the intuition behind the 2-3x figures reported in the original
    papers: higher acceptance rate and more draft tokens per step both help,
    but a slow or inaccurate draft model erodes the gain."""
    expected_tokens_per_step = sum(
        acceptance_rate ** i for i in range(draft_tokens_per_step + 1)
    )
    draft_cost = draft_tokens_per_step * cost_ratio
    target_cost = 1.0
    return expected_tokens_per_step / (draft_cost + target_cost)

A higher acceptance rate and more draft tokens proposed per step both help, but a draft model that is too slow, or too inaccurate, relative to the target model erodes the benefit quickly. Picking or training a good draft model is most of the engineering work in a real speculative decoding system.

Case Study: Medusa and EAGLE Take Different Approaches

Two widely deployed production techniques solve the draft model problem differently, and the contrast is useful for understanding the design space.

Medusa, from "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads," skips a separate draft model entirely. It adds several extra prediction heads directly onto the target model, each one trained to predict a token further ahead than the last, then uses a tree-based attention structure to verify multiple candidate continuations from those heads in a single pass. The paper reports Medusa-1 achieving over 2.2 times speedup with no generation quality loss, and Medusa-2 reaching 2.3 to 3.6 times [3].

EAGLE, from "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty," takes a different angle. Instead of drafting at the token level, it drafts autoregressively at the feature level, one layer below the final output, which reduces the uncertainty a normal token-level draft model struggles with. The EAGLE paper reports being roughly 3 times faster than vanilla decoding and about 1.6 times faster than Medusa on MT-Bench [4].

Case Study: What This Looks Like in Production

These techniques are not just research artifacts. vLLM, one of the most widely used open source inference engines, ships built-in support for both EAGLE-style and Medusa-style speculative decoding, documented in its speculative decoding guide [5]. On Llama-3.3-70B chat, EAGLE-3 based drafting has been reported to produce a 3.0 to 3.4 times decode speedup over a non-speculative baseline at batch size 1 [6].

That single-user, batch-size-1 detail matters. Speculative decoding delivers its biggest wins specifically in low-batch, latency-sensitive serving, which happens to be exactly the situation a single person waiting on a chat response or a generation job is in. At very high batch sizes, where a GPU is already fully utilized processing many requests at once, the relative benefit shrinks, since there is less idle compute for the parallel verification pass to take advantage of.

If you want a walkthrough of the mechanism in video form, this explainer on speculative decoding covers the draft-and-verify loop visually, which helps if the rejection sampling math above needs a second pass to click.

Why Inference Speed Is Not Just a Backend Detail

It is easy to treat inference speed as a pure infrastructure concern, but it shapes what a product can actually offer. A tool that generates a script, a set of visuals, and a voiceover in one pipeline, the way Text2Shorts in Miraflow AI turns a topic into a finished short, only feels fast if every model in that pipeline responds quickly. The same applies to iterating on an idea in the AI image generator in Miraflow AI or generating variations in the cinematic AI video generator: the difference between a tool that feels instant and one that feels sluggish often comes down to inference-level engineering like this, not just model quality. The same is true of the AI Music Generator, where a track needs to render in under a minute to stay usable in a real editing session, and of AI Clipping, where scanning a long video for viral moments only feels worth doing if the turnaround is fast. We ran into a similar tradeoff comparing video models directly in MiniMax H3 vs Veo 3.1, where generation speed mattered as much as output quality, and again when comparing music models in our look at Lyria 3.5. You can browse more breakdowns like this on the Miraflow AI blog, and every tool named above lives at miraflow.ai.

Common Mistakes and Misunderstandings

A few misunderstandings show up often when people first encounter speculative decoding.

• Assuming it changes model outputs. The entire point of the rejection sampling step is that it does not, when implemented correctly.
• Assuming a bigger draft model is always better. A draft model that is too close in size to the target model erases the speed advantage, since drafting becomes nearly as expensive as just running the target model directly.
• Assuming the speedup is constant across workloads. Acceptance rate varies by task, and batch size changes how much benefit is available, as the production numbers above show.
• Confusing speculative decoding with model distillation. Distillation trains a smaller model to replace a larger one and can change output quality. Speculative decoding uses a smaller model only as a proposal mechanism and mathematically preserves the target model's exact output distribution.
• Assuming EAGLE and Medusa are interchangeable. They solve the same problem with meaningfully different architectures, feature-level extrapolation versus multiple prediction heads, and perform differently depending on the model and workload.

A Practical Checklist Before Adopting Speculative Decoding

• Measure acceptance rate on your actual workload before committing to a draft model choice, since published numbers from a different task or dataset may not transfer directly.
• Check whether your serving engine already supports it, since vLLM and similar engines ship EAGLE and Medusa support rather than requiring a custom implementation.
• Prioritize it for latency-sensitive, low-batch scenarios first, where the benefit is largest.
• Re-benchmark after any target model upgrade, since a draft model tuned for one target model version will not necessarily transfer its acceptance rate to a new one.

Frequently Asked Questions

Does speculative decoding reduce output quality? No. When implemented with correct rejection sampling, the output distribution is mathematically identical to standard decoding from the target model alone.

Do I need to train a custom draft model? Not necessarily. Methods like Medusa add extra heads to an existing model rather than requiring a separate draft model, and EAGLE-style drafting is available out of the box in engines like vLLM.

Why does the speedup vary so much between reports? It depends on acceptance rate, how many tokens are drafted per step, batch size, and the relative cost of the draft versus target model. The 2 to 3.4 times range cited throughout this post reflects results across different models and setups, not a single fixed number.

Is speculative decoding only useful for chat applications? No. Any autoregressive generation task benefits, including code generation, structured data generation, and the script and prompt generation steps behind AI video and image tools.

Is this the same as parallel decoding or non-autoregressive generation? No. Non-autoregressive methods generate multiple tokens without an autoregressive draft step and typically do change output distribution. Speculative decoding is specifically designed to be output-preserving.

Where can I see this kind of speed difference for myself? Compare a fast, well-optimized generation tool against a slower one directly. The AI Image Generator in Miraflow AI and the YouTube Thumbnail Maker are both good places to notice how much inference speed shapes whether a tool feels usable for fast iteration.

Conclusion

Speculative decoding is one of the rare optimizations that gives you speed without asking you to trade away anything. A small model proposes, a large model verifies in parallel, and rejection sampling guarantees the result is exactly what the large model would have produced alone. What started as a 2 to 3 times speedup in a 2022 paper has grown into production systems like EAGLE and Medusa delivering 3 to 3.6 times gains, and it is now a standard feature in serving engines rather than a research curiosity. Understanding the draft-then-verify mechanism is worth the time whether you are optimizing your own inference stack or just trying to understand why some AI tools respond so much faster than others. If you are curious about the training side of the same models this speeds up, our breakdown of how to build the best dataset for code generation AI covers the other half of the pipeline.

References and Sources

[1] Leviathan, Kalman, Matias. "Fast Inference from Transformers via Speculative Decoding."

[2] Chen, Borgeaud, Irving, Lespiau, Sifre, Jumper. "Accelerating Large Language Model Decoding with Speculative Sampling."

[3] Cai et al. "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads."

[4] Li et al. "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty."

[5] vLLM Documentation, Speculative Decoding.

[6] EAGLE-3 decode speedup benchmark on Llama-3.3-70B chat, reported via vLLM's EAGLE draft models documentation.