S12 · Balance the Hot Experts — Place and replicate hot experts

The busiest device limits group progress; a good average load is not enough.

01Equal expert counts do not imply equal work

Gate decisions depend on inputs, so experts can receive very different token counts. Even with equal expert counts per device, hot experts may cluster on one device while others wait. EPLB uses historical routing counts to add physical replicas of hot logical experts and rearrange their placement. Additional parameter replicas also consume additional storage.

s12_expert_balance.py

# Framework pseudocode for expert placement policy.
history, future = split_windows(expert_token_counts)
assert physical_experts % ep_size == 0
local_capacity = physical_experts // ep_size

placement, replicas, counts = rebalance_experts_global(
    history,
    num_physical_experts=physical_experts,
    num_local_physical_experts=local_capacity,
)

for name, layout in candidate_layouts(placement):
    # Equal sharing between replicas is an explicit proxy assumption.
    load = estimate_rank_tokens(future, layout)
    bottleneck = load.max(axis=-1).sum()
    imbalance = load.max(axis=-1) / load.mean(axis=-1)
    report(name, bottleneck, imbalance)

export_case(history, future, placement, policy_revision)

02Window-level complementarity is richer than totals

The upstream policy first assigns redundant replicas, then greedily places experts using their load vectors across historical steps. Its objective involves the sum of each step’s busiest-rank load. Experts with equal totals may peak together or alternate, making their co-location behave differently. Grouping by totals alone discards that timing structure, while stable sorting and tie-breaking matter for reproducible experiments.

03How policy output meets runtime responsibilities

policy.py proposes replica counts and placement from historical counts; runtime responsibilities include counting, communication, parameter movement, and logical-to-physical mapping. Hotspots can change between windows. Read policy input, layout output, and later routing separately. Maximum token load is a proxy for placement decisions, not the behavior of the whole framework.

04Request balancing and EPLB act on different objects

Official production design: Request balancing in the official deployment article chooses which serving-instance group receives a request and distributes work among prefill and decode service replicas. EPLB acts inside expert execution, choosing physical replicas of logical experts and their placement. After a request reaches a service replica, its tokens can still undergo expert routing. These decisions require separate state and statistics. Public-code check: rebalance_experts_global consumes historical expert token counts and returns replica counts and logical/physical maps, establishing the EPLB policy. The README explicitly lists balancing across data-parallel prefill and decode replicas as missing service infrastructure. The public EPLB policy and a single engine’s Scheduler therefore do not establish a complete production request load balancer.

TRY IT YOURSELF

Make it yours

Create two traffic windows whose hotspot moves from expert 0 to expert 3. Compare static, random, and history-optimized layouts on the second window, explaining why the best history layout can lose its advantage.