acceptodds
Under review as a conference paper at ICLR 2027

StateBridge: State-Transition Shortcuts for Context Memory Reuse in SSM Serving

Abstract

State-space sequence models (SSMs) have emerged as a promising alternative to Transformers for efficient long-context language modeling, compressing context into a fixed-size recurrent state. However, unlike Transformers, which have well-established solutions for reusing precomputed KV caches of repeated text blocks across requests, SSMs lack any analogous mechanism. The reason is structural: Transformers handle context in an append-based manner that naturally supports modular reuse, whereas SSMs operate in an in-place manner where each block is fused into the recurrent state inseparably. This suggests a natural question: can we learn to approximate the effect of a fixed block on a different recurrent state, without actually prefilling it? In this paper, StateBridge answers this affirmatively, learning a compact input matrix for each reusable context block that approximates its in-place state update for different preceding contexts. At serving time, SSM scans this learned input matrix from the live recurrent state, applying a shared transition shortcut in place of the raw block to approximate the original behavior. Across four benchmarks, StateBridge retains %–% of performance and achieves up to speedup in time to first token (TTFT).

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
State-space models, Context reuse, LLM Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qzzpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies non-prefix context reuse in frozen state-space language models, where the same document follows different request-specific prefixes. StateBridge learns a short matrix of continuous input vectors for each reusable document and scans this matrix from the live recurrent state instead of processing the full document. Training combines teacher-answer distillation with document-end convolutional and SSM state matching, using augmented prefixes. Across four QA benchmarks, the paper reports F1 retention of 88.69%–101.07% on the main evaluation sets and whole-prompt TTFT speedups of 1.62×–2.71× under the tested configurations. Additional studies examine different Mamba backbones, state supervision, document length, and transfer to unseen question types.

Critical strengths

1.The learned surrogate provides a compact, executable mechanism for updating prefix-dependent recurrent states through a frozen SSM.

2.Evaluations across four QA benchmarks demonstrate useful quality–latency trade-offs, with whole-prompt TTFT speedups reaching 2.71× in the tested configurations.

3.The state-anchor ablation raises held-out HOW-question F1 from 7.14 to 37.50, providing direct evidence for the benefit of state supervision.

Critical weaknesses

1.Missing comparisons with relevant state-composition and compression methods, particularly order-preserving CASO, leave the method’s additional contribution unclear.

2.The substantial decline on unseen question types and lack of controlled prefix-transfer tests leave generalization beyond the construction distribution insufficiently established.

Questions for authors

1.How does StateBridge compare with order-preserving CASO and a document-compression baseline under the same frozen backbone, considering quality, preprocessing cost, storage, and online latency?

2.Can the authors separately evaluate prefix-distribution shift and question-type shift, with document-level confidence intervals, to clarify the observed OOD degradation?

Reviewer_H2D4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses non-prefix context reuse in state-space models, where a repeated text block must update different incoming recurrent states. StateBridge learns a compact sequence of continuous input vectors for each reusable block, using two-stage optimization with behavior and state supervision to approximate the block’s effect on a frozen model. At serving time, the model scans this shorter sequence from the current recurrent state instead of processing the original block. Across four document-grounded QA benchmarks with Mamba backbones, StateBridge retains 88.69%–101.07% of raw-document F1 and achieves up to 2.71× speedup in time to first token.

Critical strengths
  • State reuse is an important problem, and this paper proposes a clear and effective solution by representing a reusable text block’s state transition with a compact sequence of learned input vectors.
Critical weaknesses
  • The evaluation focuses only on document-grounded question answering. Its applicability to agent workloads, where contexts and interactions can change dynamically, remains unclear and may not fit the paper’s assumptions about reusable blocks. In addition, the substantial performance degradation on unseen query types limits the method’s practicality.
  • The writing could be substantially improved. Some symbols are not clearly defined when first introduced, and terminology is not always used precisely or consistently, making the formulation and method harder to follow.
Questions for authors
  1. Why do the experiments primarily use a fixed number of learned input vectors mm, rather than a fixed compression ratio? Reporting document lengths and the corresponding compression ratios ∣x∣/m|x|/m alongside Table 2 would make the trade-off clearer across tasks.
  2. Section 5.1 specifies the hardware and basic timing definitions, but could the authors clarify the exact prefill kernel and execution path used? How are warm-up, GPU synchronization, and repeated measurements handled when measuring TTFT and block-prefill latency?
  3. How well does the method perform on Gated DeltaNet (GDN), and how could it be applied to hybrid models that combine recurrent and attention layers? This would help assess its applicability beyond Mamba backbones.
  4. Could the authors ablate the two-stage optimization procedure to establish why Stage A is necessary? Table 5 examines the contribution of state supervision, but does not directly isolate the benefit of Stage A compared with optimizing the Stage B objective from initialization under a matched optimization budget.
  5. It would be helpful to report cosine similarity alongside the squared-error-based StateMatch score in Table 1.
Reviewer_V3GHOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

Transformers handle context in an append-based manner that naturally supports modular reuse, where SSMs operate in an in-place manner where each block is fused into the recurrent state separately. The authors ask: can we learn to approximate the effect of a fixed block on a recurrent state without actually prefilling it? As a response, they propose StateBridge which learns a compact input matrix for each reusable context block.

Critical strengths
  • The speed ups attained by the method are quite impressive, with an average of 2×2\times speedup for TTFT while almost always preserving >90%>90\% of the retention.

  • I found the empirical evaluation of the paper to be quite thorough. More specifically, the authors always made sure to establish the correct baseline to be compared against, and extensively ablated their method. For instance, in Table 2, they compare retrieval results compared to the standard approach, as well as when there is no document in the context, or there is a mismatched document. They also carried out ablations about the different parts of their proposed approach, different Mamba backbones, different document lengths, and different query types.

Critical weaknesses
  • I found the problem statement itself a bit hard to follow. This is not really helped by figure 1 (which only uses abstract symbols and a single block for the SSM in-place state update problem, upper right), or by observation 1 on page 3 where I found the notation slightly awkward and hard to follow. My understanding of the problem is as follows, please correct my if i'm wrong: the problem is that if I have two prefixes u0 and u1, and then I receive a prompt x then I can't reuse any part of H(u0,x)H(u0, x) to process H(u1,x)H(u1,x)?

  • I find the need to learn a matrix for every reusable block under multiple prefixes to be a very big limitation of the approach. As it stands, I cannot really think of a scenario where the cost of such training would be offset by the saving during inference time.

  • I found the different parts of the approach in section 4.3 not very well motivated and therefore quite disjointed/hard to follow. The authors seem to combine several losses computed at different parts of the pipeline, which seems to work. But I have no intuition as to why these different losses were combined to begin with, or what the overarching takeaway is (hence my score of 1 for informativeness)

Questions for authors
  • how does the method compare to PICASO, for instance, which to my understanding sets out with a similar goal as the authors? Furthermore, part of the appeal of SSMs is that the per-layer state-space transition has a tractable structure. Does that not buy us anything in terms of being able to reuse the state for xx in any meaningful way? I asked because it could be the case that we do have to process reprocess O(n) tokens, but that could be okay if that can be carried out in parallel, which traditional complexity does not account for.

  • I'm not sure what the authors mean by "the rows sis_i are [...] scanned by the SSM, and thus induce their own cache updaate" on lines 160-161.

  • Small nitpick: I find the use of the notation uu and u0u_0 to be confusing. If i'm correct in my understanding that they are alternatives, I would recommend renaming uu to u1u_1.

Please also respond to my concerns in the weaknesses section

Comments on the paper

3 comments

Dear Reviewer,

I hope this message finds you well. I am writing to kindly remind you that the authors’ rebuttal has been posted and is now available for your review in the system.

We would greatly appreciate it if you could review the rebuttal and provide your assessment at your earliest convenience, as your input is important for us to move forward with a timely decision.

Thank you very much for your time and continued support.

ACs

As only 24 hours remain in the author rebuttal period, I sincerely invite the reviewers to review the author's response and promptly raise any remaining questions or indicate whether you believe the author has addressed the issues you have raised.

AuthorsOpenReview ↗

Summary of the Rebuttal and Additional Results

Dear Area Chair and Reviewers,

Below we summarize the main results and clarifications from our responses to qzzp, H2D4, and V3GH.

  1. Comparison with state reuse and direct token compression (qzzp, V3GH). On DuoRC with the same frozen Mamba-2 2.7B backbone, StateBridge at m=64m=64 retains 96.93% of full-document F1, compared with 83.64% for CASO, an order-preserving state reuse baseline. StateBridge uses about 1/264 of CASO's per-document storage and reduces time to first token by 7.4–7.8%. At matched input budgets of 32–128 tokens or learned vectors, StateBridge improves F1 by 12.94–16.16 points over LLMLingua-2, a direct token compression baseline that selects tokens from the original input.

  2. Generalization across prefixes and question types (qzzp, H2D4). We evaluated prefix shift and question-type shift separately and reported document-level bootstrap confidence intervals. StateBridge retains 93.81% of full-document F1 under prefix shift. With document-grounded QA augmentation, it retains 89% of full-document F1 on unseen question types while preserving F1 on the original questions.

  3. Applicability to Gated DeltaNet and the role of Stage A (H2D4, V3GH). On Gated DeltaNet, StateBridge retains 90.39–98.56% of full-document F1 across DuoRC and CoQA. Under approximately matched optimization budgets, removing Stage A reduces F1 by 20.82–54.75% relative to the two-stage procedure. We also report state cosine similarity and compression ratios, and clarify the supervision used in each stage and the latency measurement protocol.

  4. Practical workloads and preprocessing cost (qzzp, H2D4, V3GH). StateBridge addresses a recurring pattern in production retrieval-augmented generation (RAG), where requests retrieve the same documents or chunks after different prefixes. In production workloads, RAGCache (ACM TOCS) found that 2% of documents accounted for 60% of requests, and Cache-Craft (SIGMOD 2025) reported more than 12 billion tokens of repeated processing in one month. Prior work has studied context reuse across different prefixes in Transformer serving (CacheBlend, EuroSys 2025 Best Paper; EPIC, ICML 2025; FusionRAG, SIGMOD 2026). A related opportunity arises in multi-step agentic search, where successive queries and reasoning steps access a shared corpus (Q-RAG, ICLR 2026 Oral; MC-Search, ICLR 2026 Oral; DCI, NeurIPS 2026 Spotlight). Our evaluation focuses on document-grounded QA. For Mamba-2 2.7B at m=64m=64, stage-wise H800 measurements yield an estimated one-time optimization time of 3.2 seconds per document under ideal eight-GPU parallelism.

We thank the Area Chair and all reviewers for their time and constructive feedback on our paper. Following the reviewers' follow-up comments, we ran additional experiments and shared the results in the discussion threads. We believe these results and our clarifications address the questions raised during the discussion. We hope the reviewers will consider our follow-up responses and additional results when revisiting their scores.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.