Work in Batches
Put multiple requests into one computation
Use static batching to understand request identity, input layouts, and independent stopping conditions.
A batch shares computation, not each request’s progress or stopping condition.
01What a one-request-at-a-time loop leaves unused
Model weights are large, while one decode token often provides too little work to use a device well. Combining requests in one forward pass can share weight reads and matrix-operation overhead. A static batch fixes its members at the start and accepts the next group only after all members finish. This is simple, but the longest request determines when the batch can end.
# Framework pseudocode for fixed batch membership.
members = choose_initial_requests(waiting)
while not all_finished(members):
metadata = prepare_batch_metadata(members)
outputs = execute_scheduled_round(metadata)
apply_ready_updates(outputs)
for request in members:
if request.finished:
preserve_finished_status(request)
# Do not admit waiting requests into this fixed group.
release_group_references(members)
# Static batching is a comparison model for the scheduler.
# Upstream scheduling has additional phase and budget rules.02Aligned shapes still need independent semantics
Different input lengths require padding or variable-length layouts, with positions and visibility carried by metadata. Every request retains its own cache length, output limit, and stop state. Finished requests stop accumulating output without ending their peers; fixed membership and independent lifetimes are distinct.
03Upstream automatic batching goes beyond this loop
DeepInfer’s LLM.generate adds inputs individually; Scheduler forms work under token, sequence-count, and cache constraints rather than stacking arbitrary input lengths. Static batching is a conceptual comparison. Focus on metadata retained by requests versus execution sets formed by scheduling.
Make it yours
Draw requests with output lengths of 2, 5, and 8. Mark active rows on each static-batch round and how many rounds a fourth request waits. Compare rounds rather than inventing timings.