Speculative Multi-Agent Reasoning
Speculative decoding uses a small draft model to propose tokens that a large model verifies in parallel. This project lifts that idea to the reasoning level: a lightweight draft agent proposes the next N reasoning steps, and a larger verifier agent accepts or rejects each in a single forward pass. The target is a wall-clock speedup on long multi-step traces with no accuracy loss, the same way token-level speculative decoding works; that target is not yet backed by a public benchmark.
Chain-of-thought makes models more accurate by letting them "think out loud", but every token of that thinking is generated serially, one forward pass at a time, by the same expensive model that's also producing the final answer. On a long reasoning trace, most of the latency is spent on steps that turn out to be straightforward.
The problem
Speculative decoding solved an analogous problem at the token level: a small, fast draft model proposes the next several tokens, and the large model verifies them in a single forward pass, accepting the ones it agrees with. Because verification is parallelizable across the proposed tokens, this gives a wall-clock speedup with no change to the large model's output distribution.
Reasoning traces have the same shape: a sequence of steps, most of which are "obvious" continuations that a smaller model could draft correctly. But nothing applies the draft-and-verify pattern at the step level instead of the token level.
How it works
The diagram above shows the loop:
- A draft agent (a smaller, faster model) proposes the next N reasoning steps in one go, given the problem and the steps accepted so far.
- A verifier agent (the larger model) evaluates all N proposed steps in a single forward pass, not by re-deriving each step from scratch, but by checking whether each step follows validly from the previous state.
- Accepted steps are appended to the shared scratchpad. The first rejected step (if any) is discarded along with everything after it, and the verifier generates the correct replacement for that one step itself, exactly as in token-level speculative decoding, where the large model "corrects" at the first point of disagreement and the draft resumes from there.
Status check before the numbers below: these are targets derived from the token-level speculative decoding literature, not a published benchmark of this system. The orchestration loop (draft → verify → accept/reject → resume) is implemented; the systematic benchmark that would turn "target" into "observed" is the open work item.
Based on the analogous speedup reported for token-level speculative decoding (Leviathan et al., 2022), not measured for this step-level variant.
Key design decisions
- Step-level, not token-level, granularity. Verifying individual tokens of a reasoning trace doesn't capture what actually matters: whether a step (a claim, a calculation, a sub-conclusion) is valid. A token can be locally "correct" while contributing to a step that's logically wrong. Operating at the step level lets the verifier do something closer to checking a proof than checking a sentence.
- The verifier never trusts a step it didn't check. This is the property that preserves accuracy: the draft agent can propose anything, but nothing enters the scratchpad without the verifier's sign-off in the same forward pass. The system can only get faster, not less careful: speed comes from parallelizing verification, not from skipping it.
- Reject-and-resume instead of reject-and-restart. When the verifier rejects a step, only that step and what follows it are discarded. Everything accepted before the rejection point stays, so a single bad draft step costs roughly one step's worth of wasted work, not the whole trace.
| Task shape | Expected effect |
|---|---|
| Long arithmetic / algorithmic traces with repetitive structure | High speedup — draft agent predicts the pattern reliably |
| Short reasoning chains (1-3 steps) | Little benefit — verification overhead dominates |
| Highly novel, non-formulaic reasoning | Lower acceptance rate — falls back toward verifier-only speed |
Status
Prototype: the orchestration loop, batched draft generation, and reject-and-resume logic are implemented and run end-to-end on multi-step reasoning tasks. What's explicitly not done yet: a systematic benchmark with a published dataset, task list, and acceptance-rate breakdown by task type. No code is public yet either. Until both of those exist, the speedup and accuracy numbers above are targets carried over from token-level speculative decoding, not measured properties of this system.
Limitations
- No public benchmark yet. The 2-3× / ~0 figures are hypotheses based on the analogous token-level technique, not a result from this implementation. Treat them as a target to validate, not a claim to cite.
- Speedup is acceptance-rate dependent. If the draft agent's proposals are rejected often, the system pays the cost of drafting and the cost of full verification; there's a crossover point below which this is slower than verifier-only. Characterizing that crossover by task type is the open work.
- Two models in the loop is two things to maintain. Any change to the verifier's standards (a prompt change, a model upgrade) can shift the draft agent's acceptance rate and needs to be re-checked, not assumed stable.
For the underlying idea that chain-of-thought is "renting" extra computation, see Chain-of-thought rents computation with tokens.