Draft, Then Verify
From speculative decoding to DSpark
Separate drafting, target verification, and confidence budgets to advance several tokens at once.
A draft’s value depends on accepted tokens and verification cost, not draft length alone.
01Let a cheaper draft move ahead
Calling a large model for every token incurs fixed overhead. Speculative decoding proposes several candidates with a draft mechanism, then verifies them with the target model, hoping one expensive pass accepts multiple tokens. Greedy decoding can compare candidates with target argmax choices; stochastic sampling needs proper acceptance and correction rules. Simple matching is not a general algorithm, and rejected drafts must still lead to a target-correct choice.
# Framework pseudocode for draft-verification planning.
query_budget = confidence_policy.choose_budget(request_states)
plan = scheduler.plan_verification(request_states, query_budget)
main_output = model_runner.execute_main_model(plan)
sampler_output = model_runner.execute_sample(main_output, plan)
step_state = StepState(plan, sampler_output)
# Accepted lengths control committed token positions.
# Rejected positions must not remain visible as accepted state.
ready_outputs = post_process_when_ready(step_state)
publish(ready_outputs)
# Variable draft lengths can share one scheduling round.
# Inspect mtp_varlen.hpp for upstream allocation rules.02After rejection, restore the right cache position
Verification processes candidates before knowing which will remain. Rejected positions must not become visible context, and draft state must also be reconciled. EOS, length limits, and accepted lengths determine committed results. The framework tracks planned queries, accepted counts, and state rollback separately rather than advancing positions by draft length.
03DeepInfer also allocates verification by confidence
Upstream combines DSpark with confidence-scheduled verification. mtp_varlen.hpp chooses a total query budget using candidate-acceptance estimates and execution-cost models, then distributes it across requests rather than fixing one draft length for everyone. This pseudocode marks budgeting, verification, sampling, and commit boundaries; inspect upstream for the concrete policy.
04Confidence schedules verification; verification decides acceptance
Official production design: DSpark in the official deployment article uses candidate confidence for variable-length verification: requests can receive different numbers of verification positions within one round’s budget. Confidence estimates acceptance. It can prioritize candidates for verification, but cannot turn an unverified draft into committed output. Target-model verification and sampling rules still determine what is accepted. Public-code check: MTPVarlen.schedule maintains separate acceptance-rate queues and historical snapshots for total-budget selection and per-request lengths. It explicitly requires later candidates’ confidence not to determine whether an earlier candidate is verified. Once the budget is selected, remaining verification positions are allocated by acceptance-rate order. Keep confidence estimates, query lengths, and accepted lengths separate when tracing verification and rollback.
Make it yours
Give each of two requests three candidate confidences with budget for three draft positions. Compare equal allocation with expected-acceptance allocation and identify state corrections for different outcomes.