S03 · Remember with KV — Connect execution state with the cache

Request state, processed positions, and cache ownership must advance together.

01Caching makes past computation request state

Later positions read previously processed context. The framework retains model-specific historical state and tracks its owning request and covered positions. CacheEngine handles storage, while sequence and block management track addressing and lifetimes. Returned text length alone does not determine cached length.

s03_kv_cache.py

# Framework pseudocode for request-owned cache state.
state = sequence_table[request_id]
blocks = block_manager.resolve(state.block_table)
metadata = build_input_metadata(
    query_tokens=planned_tokens,
    cached_positions=state.computed_positions,
    cache_blocks=blocks,
)
worker.execute_model_step(metadata, cache_engine.cache)

# Execution consumes input positions and updates cache state.
# A selected token becomes input only if the request continues.
# Reclaim blocks only when their references are no longer needed.

02Account for input positions and selected outputs separately

A forward processes current inputs; sampling results follow. A newly selected token contributes its cache state only when consumed by a subsequent forward. Draft verification adds accepted lengths and rollback positions. Track planned positions, computed positions, accepted outputs, and published results separately.

03Model and runtime define the cache layout

Upstream CacheEngine includes model-specific layouts and state management, while block management handles allocation, sharing, and reclamation. Plain K/V-array shapes do not describe every upstream path. This pseudocode locates framework responsibilities; inspect cache_engine.py and its callers for formats, references, and valid positions.

TRY IT YOURSELF

Make it yours

A request has processed its input and selected an output before the next round begins. Mark computed positions, selected outputs, and cache coverage, then add a draft rejection.