acceptodds
Desk rejected at ICLR 2027

PGCR: Directional Continuation Substitution for Recurrent LLM Serving

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Reusing eight transformer layers across different requests requires a valid autoregressive handoff, not merely a nearby hidden state. The handoff must jointly replace the post-block residual and every key/value write produced inside the bypassed block, and its validity depends on the ordered source–target pair. Provenance-Gated Continuation Routing (PGCR) learns this directional relation: it searches only completed requests at the same observable workflow position, retrieves with a contrastive subroutine key, reconstructs a full residual/KV continuation from a 512-dimensional prototype, and authorizes substitution with a target-conditioned utility governor. On CodeContests-Edit, SWE-bench-Lite, and ToolBench-Recurring with Llama-3.1-8B-Instruct, PGCR delivers a median end-to-end speedup while keeping signed quality changes within points. Against a common-interface LLMCache-style similarity route, it lowers the accepted-request 1-to-0 regression incidence from 18.7% to 10.7% on CodeContests-Edit while increasing speed from to ; the all-request regression incidence falls from 6.4% to 2.8%. The lower accepted-request incidence appears in every template- and family-separated comparison, including 11.8% versus 19.6% on unseen Code templates. ToolBench-Recurring reaches versus for exact speculative decoding, and its high-recurrence stratum reaches 1.31. Replication on Qwen2-7B and H100 preserves the method ordering, establishing complete continuation substitution as a practical compute-sharing primitive for provenance-rich recurrent services.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
foundational causal formal and efficient methods, PGCR

Related papers

Discussion (0)

Sign in to comment.