Put each mechanism in context.
Understand DeepInfer through four layers of responsibilities and state.
Inference core
Connect request entry, sampling, and state updates through the framework.
State & memory
Understand cache ownership, block mappings, and prefix reuse.
Batching & scheduling
Plan each round under request, token-budget, and execution dependencies.
Distribution & experts
Trace expert routes, physical replicas, and distributed coordination.
Follow a request through DeepInfer.
LLM.generateLLMEngine.stepSchedulerWorkerModelRunnerSamplingState updatesFollow entry, scheduling, execution, sampling, and state updates. One step may involve several requests and tokens; ZOS can defer post-processing to the next round. Source links use a pinned revision.
deepinfer/entrypoints/llm.pydeepinfer/engine/llm_engine.pycsrc/engine/scheduler.hppdeepinfer/worker/worker.pydeepinfer/worker/model_runner.pydeepinfer/model_executor/layers/sampler.pyRead the design. Understand the tradeoffs.
DeepSeek’s official article “DeepSeek V4.1 Flash Large-Scale Inference Deployment” explains the system design behind DeepInfer. Connect that design to the course through five questions.
Why separate prefill and decode?
Prefill processes the input context; decode generates subsequent tokens. PD disaggregation gives the stages separate resources and scheduling boundaries, connected by a state handoff. In the source, identify a request’s current stage and the state required to enter the next one.
Stages can be scheduled independently; the state handoff remains a dependency.
request → prefill → state_handoff → decode → outputREADING SCOPEProduction design; cache services and load-balancing infrastructure are not fully open-sourced. Source entry points follow the course’s pinned revision.8b16149