Spend cheaply on hypotheses, expensively on verification
Speculative decoding, RAG with reranking, multi-sample voting, and chain-of-thought are all the same pattern: fast hypothesis generation followed by expensive verification. It is the fundamental structure of scalable LLM systems.
Speculative decoding, RAG with reranking, and multi-sample voting look like three unrelated tricks from three different corners of the stack. They're the same trick, run at three different layers: generate a pile of cheap candidates, then spend the expensive model on checking them instead of producing them.
The pattern: spend cheaply on generating hypotheses, spend expensively on verifying them.
Four systems, one pattern
Speculative decoding uses a small, fast draft model to generate candidate tokens. A large, expensive model then verifies those candidates in parallel, accepting the ones that match what it would have generated itself. The small model is wrong sometimes. The large model corrects it. Net result: 2-3x faster generation with identical output quality, because verification is cheaper than generation when you're checking rather than creating.
The draft model proposes 4 tokens per round. The target model verifies all 4 in a single forward pass and keeps the longest correct prefix — the rest of that batch is thrown away.
Retrieval with reranking uses fast vector search (cheap, approximate) to generate a set of candidate documents. A cross-encoder then verifies relevance by reading each candidate carefully. The vector search is wrong sometimes. The reranker corrects it. Net result: the quality of a cross-encoder with the cost of a vector search, because you only pay the cross-encoder on a small candidate set.
Chain-of-thought with sampling generates multiple short candidate solutions at low temperature. A judge then verifies which one is correct. The individual samples are wrong sometimes. The judge corrects it. Net result: more reliable than a single long chain-of-thought at lower total cost, because sampling in parallel is cheaper than generating one very long sequence.
RAG itself is this pattern. Fast embedding retrieval generates candidate context. The full language model then verifies relevance by reading and generating from that context. The retrieval is imprecise sometimes. The model corrects it in generation. This is why RAG works at all: the generator recovers from retrieval noise more often than it fails because of it.
Why this pattern works
Generation is exploration. Verification is exploitation. Exploration needs speed and breadth: you want to cover the hypothesis space quickly, even if you make some mistakes. Exploitation needs precision: you want to pick the best hypothesis carefully, even if it takes longer.
When you try to do both at once (generating carefully, one shot, with a slow expensive model) you are neither fast nor maximally precise. You are paying the cost of exploitation for a step that has the economics of exploration.
The pattern separates these jobs cleanly. The fast model, the fast retriever, the low-temperature sampler does not need to be right. It needs to be right enough that the answer is in the candidate set. Then the expensive verifier does the precise work on a small, filtered problem.
This is also why throwing a bigger model at a problem often underperforms the generate-then-verify architecture at the same cost. A single call from a model twice as expensive is not the same as cheap generation plus targeted verification.
| Use case | Cheap generation step | Expensive verification step |
|---|---|---|
| Token generation speed | Small draft model (7B) | Large target model verifies in parallel (70B) |
| Document retrieval | Dense vector search (ANN) | Cross-encoder reranker reads full query-doc pair |
| Reasoning quality | 3–5 low-temperature completions | LLM judge selects or aggregates best answer |
| Code correctness | Fast completion at τ=0.3 | Unit tests as verifier — deterministic |
| Factual accuracy | RAG candidate retrieval | Generator constrained to cite; verifier checks citation exists |
| Agent tool selection | Lightweight planner LLM | Executor validates plan against available tools before running |
The design question
Whenever you are deciding how to improve a system, the first question is: where is the boundary between exploration and exploitation in my pipeline? Everything before that boundary should be optimized for speed and coverage. Everything after it should be optimized for precision.
Most teams draw this boundary in the wrong place, or don't draw it at all. They use one model for both jobs, tune it for precision, and wonder why it's slow and expensive. The insight is not to find a better model. It is to separate the jobs.
Speculative decoding is this exact pattern shipping in production:
We just released LM Studio 0.3.10 with Speculative Decoding support! It's a technique that pairs together a large model, and a more efficient (smaller) "draft" model that can generate tokens fast (draft generation). The larger model then verifies/rejects the draft tokens. If Show more
LM Studio 0.3.10 is here with 🔮 Speculative Decoding! This provides inferencing speedups, in some cases 2x or more, with no degradation in quality. - Works for both GGUF/llama.cpp and MLX models! - Easily experiment with different draft models - Visualize accepted draft token
References: Speculative Decoding (Leviathan et al., 2022), Cross-encoders for Re-ranking (Nogueira et al., 2020), Self-Consistency (Wang et al., 2022)