S06 · Schedule Every Step — Let the batch change at every step

Scheduling one step at a time lets requests finish independently and new work enter available slots.

01Stop waiting for the slowest batch member

In a static batch, short requests leave unused slots when they finish, but new requests cannot immediately enter. Continuous batching selects work again each round: finished requests leave and waiting requests enter when resources permit. Requests retain their own KV and progress; what changes is the current execution set. A request’s lifetime is therefore separate from membership in one forward-pass batch.

s06_continuous_batch.py

# Framework pseudocode; phase selection is simplified.
for round in engine_rounds:
    reclaim_finished_requests()
    budget = max_query_tokens
    selected = []
    for request in eligible_requests(waiting, running):
        cost = planned_query_tokens(request)
        if cost <= budget and cache_can_fit(request):
            reserve_planned_state(request)
            selected.append(request)
            budget -= cost
    ready_outputs = execute_scheduled_round(selected)
    apply_ready_updates(ready_outputs)
    # Some sampling/post-processing may still be in flight.
# Inspect upstream NORMAL-mode prefill/decode coordination.

02A budget must count more than requests

A long prompt prefill and a one-token decode have different compute and memory needs. A scheduler limits both sequence count and total query tokens for the step, while checking cache availability. Chunked prefill spreads a long input over several steps instead of consuming the entire budget at once. Fairness, first-response latency, and throughput need explicit policies rather than accidental queue ordering.

03Read DeepInfer’s step-type constraints carefully

In NORMAL mode, the upstream C++ Scheduler decides whether a step is prefill or decode and uses voting to keep the DP group on the same step type. This is more specific than a generic loop that freely mixes both kinds of tokens. ALL and ANY attempt points balance idle participation against waiting bounds. Start by tracing waiting, running, cache allocation, and completion cleanup before following those branches.

TRY IT YOURSELF

Make it yours

Combine a long prompt, a short prompt, and an already-decoding request. Plan five rounds under a fixed token budget and state exactly why an excluded request cannot enter each round.