- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes weight state souping, where an SSM encodes separate documents into different states, and later linearly combines those states using the learned linear combination. Experiments are conducted on several long-context benchmarks.
Critical strengths
Improving the memory capacity of SSMs is an important topic.
Critical weaknesses
The method uses more state memory than the vanilla SSM baseline. For instance, they mention that SSM's state size is 33.6 MB. With 128 chunk states, their method requires 33.6 MB * 128 ~= 4.3GB. Although this does not imply 128 times slower decoding, because they first linearly combine those states before use, those states should still live on HBM.
Relevant prior works [1,2] are not cited and compared against. These works also have multiple chunk states, and for each token, a router network selects states and read/write to those selected states only. The routing decision can be better than the proposed method, where which state a token is written in is pre-defined and not learned.
Almost all practical LLM agent use cases do not have clear document boundaries; an LLM input is often a single long stream of tokens. This issue is explicitly acknowledged by the authors: "On the other hand, independently encoded chunks cannot incorporate information from other chunks during encoding." On simple NIHA-style tasks such as RULER, this may look fine, but the effectiveness may diminish on a more realistic task with no clear document boundaries, especially compared to [1,2].
[1] Mom: Linear sequence modeling with mixture-of-memories
[2] Scaling Linear Attention with Sparse State Expansion
Questions for authors
See weaknesses.
Thanks for the helpful feedback! Responding to your specific comments below:
Thanks for raising this point. Weighted state souping is a novel research architecture, so we prioritized establishing its strong benefits across diverse benchmarks rather than optimizing every aspect of its implementation. The state size may be substantially reduced through additional quantization (going down to 4-bit quantization leaves us ~8.4 MB) and moreover, we conjecture that significant portions of the state may likely be discarded or compressed, leading to significant savings still. Having established the practical value of this method, we look forward to future work exploring useful optimizations like reducing the state size.
Separately, one of the strengths of our approach is that the states do not have to live in HBM, and can live in CPU memory or even on disk. Figure 3 shows the continued efficiency gains of weighted state souping even when states live on disk, when encoding costs for processing a fresh sequential input exceed the retrieval latency for states living on disk. Since CPU memory and especially disk are significantly larger than HBM, states can be efficiently cached off HBM.
Finally, it’s worth noting that our unoptimized state is nonetheless in the same ballpark as an equivalently-sized Transformer’s KV cache. To encode 128 chunks with 512 tokens per chunk (the setting of Figure 3a), our 1.5B Transformer model in the paper requires roughly 10.7 GB of KV cache, over double the 4.3 GB you note for our naive state. Of course, the Transformer’s memory requirement also grows linearly with the length of each chunk (while weighted state souping remains constant), so increasing to 1024 token chunks now pushes the Transformer to over 20 GB of KV cache, underscoring another dimension along which an unoptimized state souping approach is more efficient. We will expand our discussion of computational cost to include these considerations.
Thanks for pointing us to these additional references, we’ll expand our related work to cover this domain.
These works explore different settings than ours, and their solutions are motivated by different considerations. Both learn token-level routing into multiple recurrent memories or state partitions, addressing interference through selective writes and reads. Our focus instead is on independently encoding documents or tools into reusable cached states, then learning query-dependent weights to combine those states once before generation. This permits cached chunks to be reused across different queries and contexts without re-encoding their tokens, leading to both efficiency gains and strong long-context performance across diverse benchmarks. Our design and considerations are therefore distinct from token routing, but there may be ways to productively combine these token routing mechanisms with a state caching, souping architecture, taking the best of both approaches. We look forward to exploring these schemes further in future work.
Thanks for raising this point. Our setting focuses on reusable, chunked context, which occurs naturally in many practical applications, particularly in agentic settings. When you prompt Claude Code or Codex, under the hood your query is prepended with a long list of tool definitions, memory artifacts, documents retrieved through retrieval-augmented generation (RAG), and other discrete chunks with clear boundaries, used when orchestrating the agent’s response.
While true that the user query itself may not be organized into chunks with clear boundaries, in modern agentic systems, the user query typically makes up a very small fraction of the total input context (see for instance this blog post). Tool definitions, memory artifacts, RAG documents, and other discrete artifacts dominate the input context. Processing them as cached chunks is precisely our target, and our results suggest substantial efficiency and performance gains are possible through weighted state souping. The user query itself is not chunked, and is processed from an initialization constructed through our souping procedure. We will ensure our framing and background is clear in the manuscript.
Thanks again for the valuable feedback. Please let us know if you have any additional questions or concerns.