acceptodds
Under review as a conference paper at ICLR 2027

Weighted State Souping for Efficient Long-Context Retrieval

Abstract

State space models (SSMs) combine fast inference and fixed state size with competitive language-modeling performance. Recently, *state souping* has been proposed as a procedure whereby an SSM independently encodes chunks of tokens like documents, tool definitions, or memory records. The resulting state encodings may be cached and recombined to cheaply construct a context for downstream queries. However, both state souping and vanilla SSMs deteriorate on long, crowded contexts, limiting their utility for long-horizon domains. To address this issue, we propose *weighted state souping*, a simple method that learns query-dependent weights for composing cached states. The resulting models improve long-context performance substantially across both single- and multi-hop document question answering, needle-in-a-haystack retrieval, variable tracking, and tool use. Our method can be further adapted into an efficient sparse variant, yielding near-constant encoding cost with respect to context length while preserving (and sometimes *improving*) performance. We also present simple theoretical arguments that illustrate why souping works and when it may fail. Weighted state souping thus combines reusable context representations with selective retrieval, offering an efficient approach to language modeling over large collections of documents, tools, and other forms of structured context that increasingly dominate modern agentic workloads.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
state souping, state space models, long-context retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_h6wMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes weight state souping, where an SSM encodes separate documents into different states, and later linearly combines those states using the learned linear combination. Experiments are conducted on several long-context benchmarks.

Critical strengths

Improving the memory capacity of SSMs is an important topic.

Critical weaknesses

The method uses more state memory than the vanilla SSM baseline. For instance, they mention that SSM's state size is 33.6 MB. With 128 chunk states, their method requires 33.6 MB * 128 ~= 4.3GB. Although this does not imply 128 times slower decoding, because they first linearly combine those states before use, those states should still live on HBM.

Relevant prior works [1,2] are not cited and compared against. These works also have multiple chunk states, and for each token, a router network selects states and read/write to those selected states only. The routing decision can be better than the proposed method, where which state a token is written in is pre-defined and not learned.

Almost all practical LLM agent use cases do not have clear document boundaries; an LLM input is often a single long stream of tokens. This issue is explicitly acknowledged by the authors: "On the other hand, independently encoded chunks cannot incorporate information from other chunks during encoding." On simple NIHA-style tasks such as RULER, this may look fine, but the effectiveness may diminish on a more realistic task with no clear document boundaries, especially compared to [1,2].

[1] Mom: Linear sequence modeling with mixture-of-memories

[2] Scaling Linear Attention with Sparse State Expansion

Questions for authors

See weaknesses.

1 comment

Thanks for the helpful feedback! Responding to your specific comments below:

The method uses more state memory than the vanilla SSM baseline...

Thanks for raising this point. Weighted state souping is a novel research architecture, so we prioritized establishing its strong benefits across diverse benchmarks rather than optimizing every aspect of its implementation. The state size may be substantially reduced through additional quantization (going down to 4-bit quantization leaves us ~8.4 MB) and moreover, we conjecture that significant portions of the state may likely be discarded or compressed, leading to significant savings still. Having established the practical value of this method, we look forward to future work exploring useful optimizations like reducing the state size.

Separately, one of the strengths of our approach is that the states do not have to live in HBM, and can live in CPU memory or even on disk. Figure 3 shows the continued efficiency gains of weighted state souping even when states live on disk, when encoding costs for processing a fresh sequential input exceed the retrieval latency for states living on disk. Since CPU memory and especially disk are significantly larger than HBM, states can be efficiently cached off HBM.

Finally, it’s worth noting that our unoptimized state is nonetheless in the same ballpark as an equivalently-sized Transformer’s KV cache. To encode 128 chunks with 512 tokens per chunk (the setting of Figure 3a), our 1.5B Transformer model in the paper requires roughly 10.7 GB of KV cache, over double the 4.3 GB you note for our naive state. Of course, the Transformer’s memory requirement also grows linearly with the length of each chunk (while weighted state souping remains constant), so increasing to 1024 token chunks now pushes the Transformer to over 20 GB of KV cache, underscoring another dimension along which an unoptimized state souping approach is more efficient. We will expand our discussion of computational cost to include these considerations.

Relevant prior works [1,2] are not cited and compared against...

Thanks for pointing us to these additional references, we’ll expand our related work to cover this domain.

These works explore different settings than ours, and their solutions are motivated by different considerations. Both learn token-level routing into multiple recurrent memories or state partitions, addressing interference through selective writes and reads. Our focus instead is on independently encoding documents or tools into reusable cached states, then learning query-dependent weights to combine those states once before generation. This permits cached chunks to be reused across different queries and contexts without re-encoding their tokens, leading to both efficiency gains and strong long-context performance across diverse benchmarks. Our design and considerations are therefore distinct from token routing, but there may be ways to productively combine these token routing mechanisms with a state caching, souping architecture, taking the best of both approaches. We look forward to exploring these schemes further in future work.

Almost all practical LLM agent use cases do not have clear document boundaries; an LLM input is often a single long stream of tokens... On simple NIHA-style tasks such as RULER, this may look fine, but the effectiveness may diminish on a more realistic task with no clear document boundaries, especially compared to [1,2].

Thanks for raising this point. Our setting focuses on reusable, chunked context, which occurs naturally in many practical applications, particularly in agentic settings. When you prompt Claude Code or Codex, under the hood your query is prepended with a long list of tool definitions, memory artifacts, documents retrieved through retrieval-augmented generation (RAG), and other discrete chunks with clear boundaries, used when orchestrating the agent’s response.

While true that the user query itself may not be organized into chunks with clear boundaries, in modern agentic systems, the user query typically makes up a very small fraction of the total input context (see for instance this blog post). Tool definitions, memory artifacts, RAG documents, and other discrete artifacts dominate the input context. Processing them as cached chunks is precisely our target, and our results suggest substantial efficiency and performance gains are possible through weighted state souping. The user query itself is not chunked, and is processed from an initialization constructed through our souping procedure. We will ensure our framing and background is clear in the manuscript.

Thanks again for the valuable feedback. Please let us know if you have any additional questions or concerns.

Reviewer_QckkOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Mamba-2 encodes each passage once. To answer a question, it starts from a weighted sum of the cached states. The authors fine-tune a router together with Mamba-2 to assign these weights based on the question. A sparse variant fetches only the highest-ranked states. The authors evaluate both versions on retrieval, document question answering, variable tracking and tool use.

Critical strengths
  1. The router scores the much smaller passage keys first, so only selected states need to be fetched in the sparse model. On document question answering, the router learns from the answers without labels for the supporting passages.
  2. The key–value example pinpoints what can be lost at a chunk boundary. If every key and value occupies its own chunk and positions are discarded, swapping two values changes the answer but leaves the model with exactly the same collection of chunks.
Critical weaknesses
  1. The MuSiQue comparison changes both the passage selection and the reader. Sparse Mamba-2 reads four selected states. Sequential Mamba-2 processes every passage, including the distractors that the router discarded. Uniform averaging is a useful control for the weights. What is missing is a sequential reader given just the four retrieved passages.
  2. The counting argument does not establish the suggested factor-nn storage separation for the endpoint-only task. Proposition 6 lower-bounds the memory needed to recover all CnCn edge associations, whereas the actual queries require only the CC start–endpoint associations. The argument therefore does not establish that independently encoded edges must preserve all intermediate associations. A formal comparison would need an endpoint-only lower bound and a clearly specified memory measure, distinguishing the merged state from the total cached states and routing keys.
Questions for authors
  1. Which reader makes better use of the same four passages on MuSiQue? Keep the router’s choices fixed and compare learned weights, equal weights and the fine-tuned sequential Mamba-2 reading those passages in their original order. Please report exact match overall and separately on questions for which the four passages contain all the annotated evidence.
  2. For the two-hop endpoint-only task, can the authors establish a storage lower bound for independently encoded edges under a clearly specified memory model? In particular, does the claimed separation concern the merged recurrent state or the total cached representations, including routing keys? If no such bound is available, should the factor-nn comparison be interpreted as a conjecture rather than an established separation?
1 comment

Thanks for the helpful feedback! Responding to your specific comments:

The MuSiQue comparison changes both the passage selection and the reader... What is missing is a sequential reader given just the four retrieved passages.

Thanks for suggesting this baseline. Our comparison evaluates the complete weighted state souping architecture, including its learned selection mechanism. All methods receive the same input context. Sparse souping selects relevant states internally, without access to gold evidence labels, and its ability to suppress distractors is a primary benefit of the method.

Regarding your question on which reader makes better use of the same four passages, we've run your suggested baseline and found that they perform about the same. This is true both when the sequential model is provided the same four passages selected by the sparse souping weights, and when both models are provided the set of four passages containing all the annotated evidence (the “gold” passages, represented by the K=1 point in Figure 2c, also printed below). The specific numbers are:

MuSiQue exact match (%, 500 examples). Within each row, all readers receive the same router-selected passages. k=1 corresponds to the gold-only input control.

Pool sizeSparse soup, learned weightsSparse soup, equal weightsSequential on the same passages
k=1 (gold only)39.228.639.6
k=1237.020.635.8
k=6431.217.828.8
k=12828.615.827.6
Native hard distractors21.615.820.4

Interestingly, souping with equal weights weakens performance, indicating utility to the specific learned weights, highlighting the strength of our end-to-end approach that learns the weights in conjunction with retrieval. Note, souping is not intended to be a superior state mixer than sequential input; rather, it’s intended to preserve performance while being more efficient, and the comparable performance with sequential input in this setting is evidence that souping achieves this goal. Altogether, these results indicate that a primary benefit of weighted souping is in its more flexible capacity for attention-like selection, rather than because the model reads passages differently. We will add this baseline and discussion to our additional results in Appendix A.

The counting argument does not establish the suggested factor- storage separation for the endpoint-only task. Proposition 6 lower-bounds the memory needed to recover all edge associations, whereas the actual queries require only the start–endpoint associations...

Thanks for raising this point. You’re correct that Proposition 6 lower-bounds the memory needed to recover all associations, and does not establish that an independently encoded representation must retain all intermediate associations for the endpoint-only task. We will ensure that our language is clear on how the proposed factor-nn advantage is an intuitive conjecture, and not intended to be a full theoretical discussion on a capacity separation. We also support our intuition with an additional experiment, below.

The underlying observation is that sequential encoding permits cross-chunk preprocessing. In the favorably ordered setting, our construction forms start-endpoint associations during encoding, allowing the query to retrieve its answer directly. Independently encoded single-edge chunks cannot perform this same preprocessing. The same mechanism motivating the proposed storage advantage also yields a depth advantage in our constructions. This motivates testing whether trained models exhibit a corresponding difference in depth efficiency.

We therefore train sequential and mean-souping models on a four-hop task with three disjoint chains. Each query names a chain’s start and asks for its endpoint. As in the paper, ordered inputs present each chain consecutively in forward order, while random inputs shuffle all edges. We record test accuracy on 8192 held-out examples below:

ModelLayersOrderedRandom
Sequential299.95%43.38%
Sequential5100.00%99.95%
Mean souping233.39%33.75%
Mean souping599.49%99.96%

These results correspond precisely with our theoretical intuitions. The shallow sequential model solves the ordered task, while shallow souping remains at chance. Random ordering substantially reduces shallow sequential accuracy, and both deep models solve both orderings. These results demonstrate a substantial empirical depth-efficiency advantage for sequential encoding on ordered inputs under matched training conditions, consistent with the benefit of cross-chunk preprocessing. We will add these empirics to our manuscript to support our theoretical intuitions about cross-chunk preprocessing and efficiency advantages with sequential models.

Thanks again for the valuable feedback. Please let us know if you have any additional questions or concerns.

Comments on the paper

3 comments
Area_Chair_xDuEOpenReview ↗

Third Review from Area Chair

This paper proposes query-dependent state souping for Mamba. Each chunk is encoded into a state offline. At inference time, a router weights and combines these states to answer a query.

Reasons to accept

  1. The idea is simple and well motivated. Query-dependent weighting clearly improves over mean state souping across several tasks.
  2. The paper includes useful analyses of sparse state selection and the cost of loading states from different memory tiers.

Reasons to reject

  1. Novelty. The broader idea of precomputing document representations and loading them at inference time is not new. Parametric RAG and block-attention methods explore related ideas. The paper should better explain what is fundamentally new, rather than simply applying the aforementioned mechanisms to Mamba states.
  2. Weak long-context baselines. If the paper claims a competitive long-context architecture, it should compare against stronger linear attention models such as Gated DeltaNet and KDA, instead of the vanilla Transformer baseline .
  3. Unclear benefit of state mixing. The experiments do not compare against a model that reads the same top-N chunks selected by the router. Without this baseline, it is hard to tell how much comes from mixing states and how much simply comes from filtering irrelevant chunks.
  4. Efficiency is not yet established end to end. The microbenchmarks assume precomputed states. Cache construction, total storage, generation latency and comparable retrieval-based systems are not fully accounted for.

Questions

  1. How does the method compare with Gated DeltaNet or KDA under similar training conditions?
  2. What happens if a sequential model reads the same top-N chunks selected by the router?
  3. Could the authors compare against a Transformer with a proper length extrapolation method, and clarify its pretraining context length?
  4. How does the method differ from Parametric RAG and related block attention in novelty, storage and inference cost?

[1] W Su, Y Tang, Q Ai, J Yan, C Wang, H Wang, Z Ye, Y Zhou, Y Liu. Parametric retrieval augmented generation. [2] Dongyang Ma, Yan Wang, Lan Tian. Block-Attention for Efficient Prefilling.

Rating: 2: Weak rejection Confidence: 4

AuthorsOpenReview ↗

Response (1/2)

Thanks for the helpful feedback, and furnishing a review on short notice! Responding to your specific comments below:

Novelty. The broader idea of precomputing document representations and loading them at inference time is not new...

How does the method differ from Parametric RAG and related block attention in novelty, storage and inference cost?

Thanks for raising this point, and directing us to Parametric RAG and block attention. Document caching is certainly a broad topic. Our specific focus is on the unique opportunities enabled by state space models in providing substantial efficiency gains while preserving (and in some cases even improving) performance. These techniques are sparsely explored in SSMs, and in particular our weighted state souping procedure is a novel method with a fundamentally different design from Parametric RAG and block attention.

For example, Parametric RAG relies on an expensive encoding procedure that requires augmenting documents with question-answer pairs and training a unique LORA adapter for every individual document. Such an approach is difficult to adapt for agentic applications, as it would require individualized data augmentation and re-training for every single new tool definition and memory artifact. In contrast, our state embeddings are encoded through a single inference-time forward pass, making our method substantially cheaper and realistic to deploy for agentic settings.

Block attention approaches, in which we store and retrieve the KV cache for different documents, is similar in spirit to our approach but weakened by the computational inefficiencies of Transformers. To produce and save the KV cache for a long document, Transformers require quadratic compute and linear memory. In contrast, an SSM requires just linear compute and constant memory to save a state embedding, leading to substantial savings particularly as the document size increases.

Finally, while our approach is similar to RAG, the motivation is much broader. By learning an end-to-end system that distributes attention-like weights to different input documents, measurable performance gains are possible over a simple RAG-like approach. For instance, in our results below, we retrained and evaluated on MuSiQue comparing three different settings: 1) learned weights on sparse souping selected documents (an end-to-end system), 2) fixed equal weights on sparse souping selected documents (only the retrieval stage is learned), and 3) learned weights on BM25 retrieved documents, a common baseline for RAG approaches (only the weighting stage is learned). The end-to-end system handily outperforms both ablations, illustrating the strength of an end-to-end weighted souping approach.

MuSiQue exact match (%, 500 examples). Each method combines up to four selected passage states.

Passage poolRouter selection, learned weightsRouter selection, equal weightsBM25 selection, learned weights
k=1237.020.633.2
k=6431.217.825.4
k=12828.615.822.0
Native hard distractors21.615.88.2

Weak long-context baselines. If the paper claims a competitive long-context architecture, it should compare against stronger linear attention models such as Gated DeltaNet and KDA, instead of the vanilla Transformer baseline .

How does the method compare with Gated DeltaNet or KDA under similar training conditions?

Could the authors compare against a Transformer with a proper length extrapolation method, and clarify its pretraining context length?

We’ve retrained a set of models on our RULER NIAH task, adding baselines for Gated Delta Net and a Transformer with YaRN position extrapolation. The results are:

Single-needle NIAH exact match (%, mean of two seeds). All models are trained from scratch on up to 2K-token contexts. Evaluation uses 100 examples per length

Model2K8K32K64K128K
Transformer + plain RoPE (48.4M)10010000
Transformer + YaRN (48.4M)10098565221
Sequential Mamba-2 (46.6M)100100965427
Gated DeltaNet (53.3M)10099875214
Mamba-2 + weighted soup (full / top-4 / top-20)10010099100100

All models are trained up to 2K context length. Both GDN and a Transformer with length extrapolation deteriorate with longer contexts. Only weighted state souping maintains strong performance at all context lengths. We’re unable to provide exhaustive remeasurements across all tasks within the limited time span of the rebuttal, but will add these baselines to all our experiments in the revision.

Note, while our method exhibits strong length generalization, it is not the sole purpose of our architecture. Rather, our focus is on delivering an efficient and performant caching approach for SSMs, and long-context benefits are a serendipitous additional benefit.

AuthorsOpenReview ↗

Response (2/2)

Unclear benefit of state mixing. The experiments do not compare against a model that reads the same top-N chunks selected by the router. Without this baseline, it is hard to tell how much comes from mixing states and how much simply comes from filtering irrelevant chunks.

What happens if a sequential model reads the same top-N chunks selected by the router?

Thanks for raising this point. The souping procedure itself is not intended to be a stronger mixing procedure than sequential input. Rather, it aims to be more efficient than sequential input while at least preserving performance. Indeed, the stronger long-context performance of weighted souping is primarily in its benefit to filter out irrelevant chunks. If a sequential model is provided with the same set of passages selected by the sparse souping model, the two methods perform about the same.

Below, we compare sparse souping and sequential on MuSiQue, where both models are provided the set of passages selected by the sparse souping model:

MuSiQue exact match (%, 500 examples). Both readers receive the same router-selected passages, up to four per example.

Passage poolSparse soup, learned weightsSequential on the same passages
k=1237.035.8
k=6431.228.8
k=12828.627.6
Native hard distractors21.620.4

Performance is about the same, and a positive signal that souping is preserving the performance of standard sequential input. We will add this baseline and discussion to our manuscript.

Efficiency is not yet established end to end. The microbenchmarks assume precomputed states. Cache construction, total storage, generation latency and comparable retrieval-based systems are not fully accounted for.

Thanks for raising this point. The timings plotted in Figure 3 are indeed the end-to-end timings required to process the entire query, generate weights for souping, and construct the state initialization. Generation in a souping model is precisely the same as a sequential model, since souping influences only the state initialization. As a result, souping inherits all the computational benefits of SSMs for generation (linear in compute, constant in memory).

In our setting, the document encoding itself is constructed once offline, and the cost of this initial encoding is amortized across indefinitely many future uses. Documents are not re-encoded for every input. Hence, the latency in weighted state souping is determined by the document retrieval time rather than encoding. We discuss the comparison to Parametric RAG and block attention above, in our response to your first point. We will clarify our setting and discussion on these topics in Section 4.2.

Thanks again for the valuable feedback. Please let us know if you have any additional questions or concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.