Route to Experts
Follow tokens through expert routing
Follow a MoE pass through gating, dispatch, expert computation, and combine.
EP distributes expert parameters; tokens must travel to the devices holding them.
01Sparse activation does not remove large parameter sets
MoE selects a small number of routed experts per token, activating only part of the parameters for that computation while the full expert set still needs storage. Expert parallelism spreads expert parameters across devices, and the gate supplies logical expert IDs and weights. Unlike data parallelism, where replicas handle independent requests, one token’s MoE computation may cross several devices.
# Framework pseudocode for EP routing and return correspondence.
expert_ids, weights = gate.topk(hidden, experts_per_token)
physical_ids = resolve_replicas(expert_ids, replica_map)
routes = pack_routes(hidden, physical_ids, weights)
received, handle = dispatch_to_owners(routes)
local_outputs = empty_like(received.hidden)
for expert in local_experts:
rows = received.rows_for(expert.id)
local_outputs[rows] = expert(received.hidden[rows])
returned = combine_to_origins(local_outputs, handle)
output = zeros_like(hidden)
for route in returned:
output[route.token] += route.weight * route.hidden
if shared_expert is not None:
output += shared_expert(hidden)02Communication must preserve enough information to return
Dispatch sends tokens and routing metadata to expert-owning devices; combine returns expert results to their original positions and merges weighted contributions. A token may be copied to multiple experts, so ordering, origin indices, and weights must survive. Real systems also handle uneven per-device token counts, communication buffers, and compute overlap. Communication can offset part of the benefit from sparse computation.
03Logical and physical experts upstream
DeepInfer’s DeepseekMoE connects dispatch/combine paths. The model selects logical experts, then the framework resolves physical experts, including redundant replicas; their count must satisfy EP allocation constraints. Attention DP/CP and MoE EP serve different roles. Trace route origins and return order alongside process-group responsibilities.
Make it yours
Place four experts on two ranks and choose two experts for each of three tokens. Draw dispatch messages, expert outputs, and weighted combine while retaining each message’s origin token.