Learn DeepInfer.
Understand DeepInfer. Follow the request.
Start at request entry and trace each scheduling and execution round. Explore how DeepInfer connects caching, sampling, state updates, and experts.
One request. A connected framework.
LLMEngine.step coordinates scheduling, execution, sampling, and state updates. Follow inputs, outputs, and shared state to understand each module boundary.
# Framework pseudocode for reading responsibilities.
for request in inputs:
llm_engine.add_request(request)
while llm_engine.has_unfinished_requests():
ready_outputs = llm_engine.step()
collect(ready_outputs)
collect(llm_engine.flush_inflight_reqs())
# step: schedule -> execute -> sample -> state updates
# ZOS may commit a prior round and defer current state.
# One round may involve multiple requests and token positions.Framework pseudocode · a round may cover several requests and token positions
One request, step by step
engine.step()Prepare: preset token IDs are in request state, waiting for an execution plan.
KV cache 0 positionsOne mechanism at a time.
12 progressive lessons that follow requests through DeepInfer’s architecture and mechanisms.
Follow a Request
Start at LLM.generate and trace scheduling, execution, sampling, and state updates.
Inside an Execution Step
Understand how Worker and ModelRunner pass input metadata, cache references, and sampling results.
Remember with KV
Follow CacheEngine and request lifetimes to see how historical state is retained, located, and released.
Choose the Next Token
Trace model output through sampling into request state that can be committed.
Work in Batches
Use static batching to understand request identity, input layouts, and independent stopping conditions.
Schedule Every Step
Understand continuous batching through waiting and running queues plus token budgets.
Page the KV Cache
Use a block table to separate logical token positions from physical cache blocks.
Reuse a Prefix
Use request forks to understand prefix reuse, reference counts, and copy-on-write.
Draft, Then Verify
Separate drafting, target verification, and confidence budgets to advance several tokens at once.
Overlap the Pipeline
Read ZOS forward, postprocess, and sampling order, including one-step-late state correction.
Route to Experts
Follow a MoE pass through gating, dispatch, expert computation, and combine.
Balance the Hot Experts
Understand EPLB through historical token counts, then evaluate placement on the next window.
Every lesson is open. Reading progress stays in your browser.
Put each mechanism in context.
Understand DeepInfer through four layers of responsibilities and state.
Inference core
Connect request entry, sampling, and state updates through the framework.
State & memory
Understand cache ownership, block mappings, and prefix reuse.
Batching & scheduling
Plan each round under request, token-budget, and execution dependencies.
Distribution & experts
Trace expert routes, physical replicas, and distributed coordination.
Keep reading: official system design and source map
Understand it through the source.
Chapter 1: map entry, scheduling, execution, and state updates.