Reuse a Prefix
Compute a shared prefix once
Use request forks to understand prefix reuse, reference counts, and copy-on-write.
Reuse requires identical computation state, not merely similar-looking text.
01Shared beginnings cause repeated prefill
Requests may share a system message, long document, or few-shot examples. If tokens, positions, and model computation conditions match, the prefix’s KV can be shared. Each request continues from its own suffix instead of recomputing the beginning. Similar-looking or identically rendered text is not sufficient: chat formatting and special tokens can change the actual prefix.
# Framework pseudocode for explicit prefix-state sharing.
# The prefix state is already available; no automatic lookup is assumed.
child = create_child_request(suffix_request)
child.block_table = prefix_state.block_table.copy()
for block in child.block_table:
pool.retain_reference(block)
shared_tail = child.block_table[-1]
if append_would_modify_shared_state(shared_tail):
private_tail = pool.copy_for_write(shared_tail)
child.block_table[-1] = private_tail
pool.release_reference(shared_tail)
# Inspect scheduler.fork_seq usage in examples/benchmark.py.
# Sharing also requires matching computation conditions.02Shared state needs ownership rules
Sharing read-only blocks increments references; finishing a request decrements them, and reclamation waits for the final reference. If a fork shares a partially filled final block, later writes must not overwrite another request’s state, requiring copying or equivalent isolation. Lookup keys should also distinguish model revision, positions, and cache format. Omitting those conditions can turn a high hit rate into incorrect computation.
03What the upstream example actually demonstrates
The upstream prefix-reuse example computes a common cached prefix and uses scheduler.fork_seq to create later requests, demonstrating explicit forks and state reuse. This alone does not establish a complete KV service with automatic lookup for arbitrary requests. Separate the fork mechanism from prefix indexing, cross-machine transfer, and service-level eviction. This lesson follows sharing and lifetime management only.
04A prefix hit still needs ready SWA state
Official production design: The official deployment article distinguishes a prefix KV-block hit from ready sliding-window attention (SWA) state. When some SWA state was not retained with the prefix, a bounded suffix can be replayed to approximately recover the window state needed to continue. Track the hit position, valid state range, and replay boundary together before starting decode. Public-code check: StateBlock separates stored and dropped state and states that consumer SWA recovery depends on prefill recomputation. The truncation metadata entry requires the retained suffix to cover the SWA window plus backtracking allowance. These recovery mechanisms are public, while the README still lists a complete KV cache service as missing infrastructure. They do not establish that the full production prefix-lookup and recovery service is released.
Make it yours
Two requests share one full block and one half-full block, then append different tokens. Draw reference counts and copy-on-write, marking when the old block can safely be released.