One mechanism at a time.
12 progressive lessons that follow requests through DeepInfer’s architecture and mechanisms.
Follow a Request
Start at LLM.generate and trace scheduling, execution, sampling, and state updates.
Inside an Execution Step
Understand how Worker and ModelRunner pass input metadata, cache references, and sampling results.
Remember with KV
Follow CacheEngine and request lifetimes to see how historical state is retained, located, and released.
Choose the Next Token
Trace model output through sampling into request state that can be committed.
Work in Batches
Use static batching to understand request identity, input layouts, and independent stopping conditions.
Schedule Every Step
Understand continuous batching through waiting and running queues plus token budgets.
Page the KV Cache
Use a block table to separate logical token positions from physical cache blocks.
Reuse a Prefix
Use request forks to understand prefix reuse, reference counts, and copy-on-write.
Draft, Then Verify
Separate drafting, target verification, and confidence budgets to advance several tokens at once.
Overlap the Pipeline
Read ZOS forward, postprocess, and sampling order, including one-step-late state correction.
Route to Experts
Follow a MoE pass through gating, dispatch, expert computation, and combine.
Balance the Hot Experts
Understand EPLB through historical token counts, then evaluate placement on the next window.