S01 · Follow a Request — Follow a request through the engine

An engine round is a coordination boundary, not exactly one token for one request.

01Entry and execution have different responsibilities

LLM.generate receives inputs, adds requests to the engine, and delegates progress to LLM._run_engine. The entry layer creates requests and collects results; LLMEngine turns current request state into the next round of work. Separating them prevents confusing one user-facing generate call with one model forward.

s01_request_flow.py

# Framework pseudocode, not an executable implementation.
for request in inputs:
    llm_engine.add_request(request)

while llm_engine.has_unfinished_requests():
    ready_outputs = llm_engine.step()
    collect(ready_outputs)

collect(llm_engine.flush_inflight_reqs())

# Inside a scheduled round:
# Scheduler -> Worker -> ModelRunner -> sampling
# -> ready state updates and request outputs
# A round may cover several requests and token positions.

02Which boundaries one step crosses

LLMEngine.step asks the C++ Scheduler to select sequences and a token budget, then delegates execution to Worker and ModelRunner. Sampling chooses or accepts candidates; post-processing updates request state. A step can involve multiple requests, prefill chunks, or several draft positions, so it is not a fixed one-token loop.

03Output readiness can cross round boundaries

With ZOS, current forward, previous-step post-processing, and current sampling follow a defined order, and some updates are deferred to the next round. _run_engine must also drain in-flight outputs and flush at the end. This pseudocode marks responsibilities; inspect pinned upstream source for concrete arguments, branches, and lifetimes.

TRY IT YOURSELF

Make it yours

Map entry, scheduling, execution, sampling, and state updates to their modules. Explain why outputs returned by a step need not all come from its current sampling.