acceptodds
Under review as a conference paper at ICLR 2027

States as Messages: Latent Communication for State Space Models

Abstract

State space models (SSMs) are increasingly used in language modeling, and when model instances hold different contexts, one may need information that another has already processed. Transferring this information through natural-language summaries requires autoregressive generation at the source and re-encoding at the receiver. Latent communication can reduce this overhead by directly exchanging internal representations such as the fixed-size recurrent states of SSMs. Using these states for communication requires the receiver to combine the source's state with its own while retaining information from both contexts. We introduce a dynamics-grounded fuser for pretrained multilayer SSMs. The fuser learns a lightweight per-channel correction to a single-layer state decomposition while keeping the base model weights frozen. Across five scales each of Mamba and Mamba-2, our method closes up to 75% and 84% of the receiver-only to full-context perplexity gap on WikiText-103, respectively. The method generalizes across sequence lengths, partition ratios, and domains without retraining. On separate-document QA with HotpotQA and MuSiQue, task-specific state communication achieves exact-match scores comparable to natural-language summaries and closes up to 74% of the output-distribution gap to the full-context teacher. On HotpotQA, state communication runs approximately faster end to end than the summary baselines on a single GPU, with no generated message tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, state space models, latent communication, multi-agent communication

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yRNsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes communication between two models sharing a frozen state-space language model by transferring and merging their internal recurrent states. Its main contribution is a learned elementwise correction to an existing decay-based state-composition rule, trained through distillation from full-context predictions without updating the backbone. Evaluation spans ten Mamba/Mamba-2 checkpoints, language modeling, commonsense reasoning, and multi-hop question answering. The results generally improve upon receiver-only context, while question-answering performance is comparable to text-summary communication with lower measured communication-stage computation and latency. Full-context processing remains stronger, and the efficiency measurements exclude transmission of the relatively large state payload. Ablations examine alternative fusion rules, context lengths, and document partitions.

Critical strengths

1.Simple and decently motivated method: The single-layer decomposition is correct under its stated assumptions, and the distinction between this identity and approximate multilayer fusion is clearly explained. Learning an elementwise correction while freezing the base checkpoint is an economical adaptation strategy. Appendix H.1 also clearly specifies that the QA answer-format adapter is shared across conditions and frozen during fuser training.

  1. Broad and consistent language-modeling evidence: Perplexity improves over receiver-only processing at all ten checkpoints on both WikiText-103 and PG-19. The learned correction improves on the analytic prior in the reported comparisons, including all five Mamba-2 scales (Table 8). ARC-Easy and HellaSwag point estimates also improve across both model families, although the magnitude varies by task and scale.

  2. Design ablations: Table 4 compares the method with the analytic prior, a parameter-matched linear fuser without decay, and a parameter-matched MLP. The improvement over the no-decay linear baseline, together with Figure 3b's transfer to shorter sequences, supports the value of retaining accumulated decay as an input-dependent factor. The MLP results also expose an informative difference between perplexity and downstream accuracy.

  3. Disclosure of limitations: The paper reports negative results at 8192 tokens, degradation with additional independently processed source segments, mixed PIQA results, and the omission of convolution-state fusion. It also explicitly limits the latency measurements to a single GPU and excludes inter-device transmission. These disclosures help delimit the contribution.

Critical weaknesses
  1. The practical efficiency advantage is not established against raw-text transfer:

Table 3 compares state communication with pipelines that generate summaries autoregressively. However, the introduction also motivates avoiding the receiver's reprocessing of raw source text, and this alternative is not timed. Also, The full-context condition already achieves higher HotpotQA EM than state communication, 0.541 versus 0.507, but its latency and computational cost are absent. This is my big concern; I don't feel motivated enough to use this method anywhere unless the latency is significantly lower, 2x is still not worth it, given that we loose interpretability. The comparison should use the same assumptions about which source and receiver computations have already occurred. This would establish whether state fusion offers a useful quality–latency tradeoff over simply transferring the paragraph and processing the combined prompt.

The reported 180 ms versus 278/292 ms measurements support the stated comparison with summary pipelines. Their broader practical significance remains uncertain without the raw-text baseline and a stage-level breakdown. The 20 MB FP32 payload also makes deployment conditions relevant. For illustration, transmitting 20 MB at ideal link throughput takes roughly 16 ms at 10 Gbps or 160 ms at 1 Gbps, before other overheads or overlap. The former would reduce the reported advantage; the latter could eliminate it. A bandwidth sensitivity analysis would clarify where the method is useful.

  1. The incremental contribution of the learned correction needs stronger task-level evidence:

The decay-based composition rule and its approximate application to stacked Mamba layers already appear in State Soup, including experiments on independently processed sequence chunks. Pióro et al., 2024 The main methodological addition here is therefore the learned correction, alongside the broader evaluation and communication setting. Its perplexity benefit is demonstrated, but the analytic prior with Λ = 1 is missing from the QA comparison in Table 3. This leaves its incremental value unclear on the task most directly supporting the communication claim.

The design comparison also requires uncertainty estimates. In Table 4, MLP-CW achieves better WikiText/PG-19 perplexity than the proposed fuser, 20.33/18.75 versus 20.44/18.78, while the proposed fuser has higher downstream accuracy point estimates. Its accuracy improvements over the analytic prior are 0.30, 0.21, and 0.15 percentage points on ARC-Easy, PIQA, and HellaSwag, respectively. These differences may be meaningful, but their statistical reliability cannot be determined from aggregate accuracies alone. Paired confidence intervals and the number of training seeds are needed to substantiate the claim of a better downstream inductive bias. The existing confidence intervals in Appendix H.4 provide useful evidence of improvement over receiver-only on a separate HotpotQA sample, but do not resolve these design comparisons.

Sum and Average are reasonable ablations under the frozen-checkpoint constraint. Their poor performance in this setting should not be interpreted as a comparison with the complete training procedures of prior state-aggregation methods.

  1. The QA comparison does not isolate the benefit of the communication channel from task-specific adaptation:

the QA fuser receives 80,000 sampled training examples per benchmark and supervision from the full-context teacher, whereas the summary mechanism uses fixed prompts without comparable task-specific summarization training. The source lacks the question in both conditions, and the answer-format adapter is shared, so those aspects are controlled. Nevertheless, matching these prompted summaries does not establish that state communication would remain competitive with an adapted summarizer. A stronger summary baseline would make the comparison more informative.

The distinction between distribution recovery and answer quality also matters. Held-out KL is a valid metric for the stated distillation objective, but it does not by itself establish equivalent task-level recovery. From Table 3, EM gap closure is approximately 35% on HotpotQA and 44% on MuSiQue, compared with KL closure of 74% and 68%. The paper acknowledges this distinction in 4.3; the abstract would be more informative if it also conveyed the extent of answer-quality recovery.

  1. Evaluation ambiguities and an unexplained table discrepancy limit the reproducibility:

In appendix H.1 describes both QA benchmarks as having two supporting paragraphs per question and reports evaluation on all 2,417 MuSiQue development examples. However, the original Musique development set contains 1,252 two-hop, 760 three-hop, and 405 four-hop questions. The paper needs to explain how supporting paragraphs for three- and four-hop questions are mapped into its two-document setting. The original training set contains 19,938 questions, so the stated 80,000 sampled training examples also require an explanation of replacement sampling or other example construction. These choices could be valid, but they materially affect interpretation of the experiment. Trivedi et al., 2022, Table 2

Table 14 reports receiver/fused WikiText perplexities of 20.861/19.987 for Mamba-130M at split 0.5 and length 1024, whereas Table 4 reports 21.42/20.44 under the same stated configuration. If the layer analysis uses a different subset or protocol, this should be specified; otherwise the discrepancy needs correction.

The zero-shot evaluation description also lacks the precise prompt partitioning and scoring procedure, including treatment of answer choices. For QA, the token positions and aggregation used to compute KL closure should be stated explicitly. These details are necessary to reproduce the results and understand what information each model receives.

  1. The scope of demonstrated generalization is narrower than the abstract suggests

The successful length-transfer experiments cover 64–512 tokens, all shorter than the 1024-token training sequences. At 8192 tokens, fusion performs worse than receiver-only at every evaluated Mamba scale. Full-context processing also degrades, so this experiment does not isolate fusion error from the base models' length extrapolation limits. Evaluation at an intermediate length such as 2048 would be more informative about extension beyond the fuser's training length.

Tables 12–13 contain lower ARC-Easy point estimates after fusion when the source holds little context. The paper reports these limitations, but its broad generalization statements should distinguish the demonstrated perplexity transfer from less consistent task-level behavior.

Questions for authors

What are the latency and computational cost of raw-text transfer/full-context processing and receiver-only under the Table 3 protocol? Please report the stages included, precision, batch size, actual summary lengths, and timing variation. How is decay accumulation accounted for in the 0.02 GFLOPs figure, including any quantities reused from the receiver's forward pass?

Can you add the Λ = 1 analytic prior to Table 3 and report its EM and KL closure under the identical QA setup? This would directly establish the learned correction's contribution?

Can you provide paired confidence intervals for the main QA and downstream comparisons, particularly the proposed fuser versus the analytic prior and MLP-CW, and report the number of training seeds?

How does state communication compare with a summary mechanism given comparable task-specific adaptation? If such a comparison is unavailable, please delimit the conclusions to the prompted summary baselines evaluated here?

How are MuSiQue's three- and four-hop examples represented by two documents, and how are 80,000 training examples constructed? Please also explain the Table 14/Table 4 discrepancy and specify the zero-shot partitioning/scoring procedure and QA KL evaluation positions.

3 comments
AuthorsOpenReview ↗

Author's Response to Reviewer yRNs (1/2)

Thank you for suggesting the direct comparison with raw text.

W1 / Q1. Raw-text latency. Reading both documents and the question in one pass was fastest for the short paragraphs on both datasets. On HotpotQA, Table 3 also gives higher EM for full context than for fusion (54.1% versus 50.7%). Fusion reached the first answer token sooner on longer sources when both agents had already read their documents.

Both clocks stop at the last answer token. Total includes any initial document reads the method uses, with independent reads running in parallel. Warm starts after each agent has read its own document. All timings exclude network transfer. One-pass raw text reads both documents and the question together. The re-read baseline reads both documents together, then processes the question in a separate call. Both start from raw text, so their total and warm times are identical. The append baseline reuses the receiver’s state, reads the source text, then processes the question.

We timed 1,000 questions per dataset on one A100 with FP16, batch size one, and greedy decoding. The table gives mean latency in milliseconds, with standard deviations in parentheses. With the paper’s prompts, short and medium summaries averaged 5.2 and 5.6 tokens on HotpotQA and 12.0 and 18.1 on MuSiQue.

MethodTotal, HotpotQATotal, MuSiQueWarm, HotpotQAWarm, MuSiQue
Receiver only236 (85)250 (80)186 (84)200 (80)
Raw text, one pass178 (81)196 (84)178 (81)196 (84)
Raw text, re-read then question237 (81)253 (79)237 (81)253 (79)
Raw text, append then question294 (81)309 (79)245 (81)260 (79)
State fusion239 (90)253 (84)189 (90)204 (84)

Receiver-only processes the question in a separate call from the saved state, which adds overhead at these short lengths. When raw text also processed the question separately, its total latency was close to fusion on both datasets. Fusion added 3 to 4 ms over receiver-only. In a separate timing run with the same harness, HotpotQA fusion took 234 ms. About 49 ms went to parallel document reads, 2.7 ms to packing, copying, and fusing the states, 58 ms to processing the question, and 123.5 ms to decoding the answer.

Source-length sweep. To simulate sending a longer document, we padded the sharer’s paragraph with unrelated text and evaluated 200 HotpotQA questions per length. Both agents had already read their documents. Raw text requires the receiver to process the sharer’s tokens again, while fusion reuses the sharer’s state. We timed the first answer token because output lengths varied. At long source lengths, raw text sometimes generated long, incorrect answers, which would add decoding time to the comparison.

Source tokensRaw text, one pass (ms)State fusion (ms)
1305362
5718062
1,07712962
2,07222962
4,06941862
8,07173661

With both states available, raw text was faster at 130 tokens and fusion was faster at every tested length from 571 onward. This applies, for example, when the sharer reuses a document state across questions. Including the initial reads made raw text faster at shorter lengths and brought first-token times within about 3% at 4,069 and 8,071 tokens. Appending to the receiver’s saved state was slower than one-pass raw text at all six lengths.

Transfer time. At ideal link speeds of 1 and 10 Gb/s, sending the 10 MiB FP16 state takes an estimated 84 and 8.4 ms. INT4 reduces this to 26 and 2.6 ms. At 1 Gb/s, the 26 ms transfer exceeds fusion’s 18 ms local saving at 571 tokens. At 2,072 tokens, it brings fusion to about 88 ms versus 229 ms for one-pass raw text, a projected saving of 141 ms.

FP16 matched FP32 in aggregate EM on both datasets. For INT4 relative to FP16, the paired 95% confidence intervals for the change in EM were within ±0.2 percentage points on both datasets. We will add the raw-text baselines to Table 3 and specify when the speed comparison assumes existing document states.

Compute accounting. The 0.02 GFLOPs in Table 3 counts final fusion on the receiver. Two multiplications and one addition per state entry across 64 × 5,120 × 16 entries give 0.016 GFLOPs. This step took 2.30 to 2.37 ms on an A100 (Table 15). Eq. 8 reuses the receiver’s step sizes. For a 134-token receiver paragraph, forming the accumulated decay takes 43.6 million additions, 5.24 million multiplications, and 5.24 million exponentials. Counting each exponential once gives 0.054 billion scalar operations for decay and 0.070 billion including fusion. We will correct Table 3, Section 4.4, and Appendices F.1 and H.3. For raw text and receiver-only, we measured latency only.

AuthorsOpenReview ↗

Author's Response to Reviewer yRNs (2/2)

W2 / Q2–Q3. Learned correction, paired intervals, and training seeds. Learning Λ\Lambda improves EM over the fixed decay rule by 1.7 points on HotpotQA (paired 95% CI 1.1 to 2.2) and 4.0 points on MuSiQue (2.9 to 5.0). These gains use paired question-level differences. With Λ\Lambda = 1, HotpotQA reaches 49.0% EM and 64.0% KL closure, and MuSiQue reaches 18.0% EM and 56.9% KL closure. We will add the Λ\Lambda = 1 row to Table 3.

In Table 4, our fuser scores 39.27% on ARC-Easy versus 37.79% for the parameter-matched MLP, a gain of 1.48 percentage points. Below are the paired 95% intervals from per-question Mamba-130M predictions, in percentage points. We trained each fuser once, so the intervals capture variation across evaluation questions.

Learned Λ\Lambda compared withARC-EasyPIQAHellaSwag
Λ\Lambda = 1[−0.42, +1.01][−0.44, +0.82][−0.09, +0.51]
Parameter-matched MLP[+0.34, +2.53][−0.27, +1.41][+0.11, +0.99]

The gains over the MLP on ARC-Easy and HellaSwag have intervals above zero. Against Λ\Lambda = 1, all three downstream intervals include zero. We will distinguish these smaller downstream differences from the QA gains and describe Sum and Average as ablations on the frozen checkpoint.

W3 / Q4. Supervision for summaries. All methods share the answer-format adapter, but only the fuser receives communication-specific training from the full-context teacher. We will limit the comparison to the prompted summaries in Table 3. Training a summarizer with the same teacher and examples remains future work. In the abstract, we will also report how much of the EM gap fusion recovers on HotpotQA and MuSiQue (about 35% and 44%) alongside KL closure (74% and 68%).

W4 / Q5. Reproducibility.

  • MuSiQue. We use the first two supporting paragraphs in dataset order, giving the first to the sharer and the second to the receiver. For the 1,165 three- and four-hop questions, this leaves out supporting evidence in every condition, including full context. For the 1,252 two-hop questions, the two paragraphs contain all supporting evidence. EM on this subset is 0.224 for receiver-only, 0.305 for fusion, and 0.403 for full context.
  • Training examples. The 80,000 comes from 20,000 training steps with four questions per step. A question counts each time it is used. MuSiQue has 19,938 distinct training questions, so this amounts to about four passes through the dataset, with reshuffling between passes. We will replace the one-epoch description in Appendix H.1 with the step count.
  • Tables 14 and 4. The layer analysis in Table 14 used a 200-sequence subset of the WikiText test set. Table 4 used 279 sequences. We will specify the subset in the Table 14 caption.
  • Scoring. We split the tokenized prompt in half, keeping answer choices out of the prompt, and score each choice as a continuation. We normalize by choice length on ARC-Easy and HellaSwag and use summed log probability on PIQA. For QA, we teacher-force each method on the gold answer and end-of-turn token. We average token KL within each question first, then across questions.

W5. Intermediate context length. Fusion improves PG-19 perplexity over receiver-only at 2,048 tokens across all five Mamba scales. This is twice the fuser’s training length and matches the pretraining length of the checkpoints. The table gives the percentage of the gap from receiver-only to full context that fusion closes.

Total sequence length130M370M790M1.4B2.8B
1,024 tokens64%41%51%33%35%
2,048 tokens51%28%44%19%18%

Both rows use the Appendix D.4 long-document protocol. At each length, we evaluated 200 sequences and split the first 75% of each sequence equally between the agents.

At 8,192 tokens, fusion and full context both give worse perplexity than receiver-only at every scale. We will distinguish these perplexity results from the mixed ARC-Easy results when the sharer receives only a small part of the prompt.

Dear Reviewer yRNs, as the reviewer/author discussion period is coming to an end, we wanted to check whether we have addressed all your questions and concerns. We are happy to clarify anything further.

Reviewer_AFCEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is well motivated and addresses an important challenge in multi-agent language model systems: transferring information between models that have independently processed different contexts.

Conventional communication through natural language requires the source model to generate a summary and the receiver to re-encode it, introducing computational overhead and potentially losing information. Existing state-fusion techniques, such as addition and averaging, do not adequately account for the underlying state-space dynamics.

The authors propose a dynamics-grounded fusion mechanism that combines independently computed SSM states using accumulated receiver-side decay and a lightweight learned correction term. The correction is trained to minimize the divergence between the fused receiver's output distribution and that of a full-context model.

The approach is evaluated across multiple configurations of Mamba and Mamba-2, with experiments covering language modeling, zero-shot benchmarks, and multi-document question answering.

Critical strengths
  1. Computational efficiency. The approach introduces fewer than 1% additional trainable parameters while keeping the pretrained model frozen. Fusion requires inexpensive elementwise operations, avoiding autoregressive message generation and receiver-side re-encoding.

  2. Competitive performance against natural-language communication. On HotpotQA, the method achieves 50.7% exact match compared with 50.1% for the strongest natural-language summary baseline, while reducing latency from 278–292 ms to 180 ms on a single GPU. On MuSiQue, it matches the strongest summary baseline.

  3. Principled fusion mechanism. Unlike naive addition or averaging, the method incorporates the accumulated decay of the receiver's SSM dynamics. This provides a theoretically grounded inductive bias that consistently improves downstream performance relative to simpler fusion mechanisms.

  4. Evaluation across architectures and model scales. The authors demonstrate improvements across five scales each of Mamba and Mamba-2. Their ablations help establish the respective contributions of the analytic prior and learned correction.

Critical weaknesses
  1. Limited improvement in actual task performance.

Despite closing 74% of the output-distribution gap on HotpotQA, the method improves exact-match accuracy by only 1.8 percentage points (48.9% → 50.7%). On MuSiQue, it only matches the strongest natural-language baseline. This raises the question of whether the transferred information meaningfully improves reasoning or simply makes the model's predictions more similar to the full-context model.

  1. Poor generalization to longer contexts and multiple sources.

At 8,192 tokens, state fusion performs worse than using the receiver alone across all tested Mamba scales. Performance also deteriorates when information is split across multiple independent sources. Although the base model's long-context limitations may partly explain the results, these failures raise concerns about the robustness of the fusion mechanism in more realistic multi-agent settings.

  1. Restrictive assumptions about the models and evaluation setting.

The method requires identical model checkpoints, with matching layers and state dimensions. It therefore cannot directly support heterogeneous agents. Moreover, the QA evaluation uses only two gold supporting paragraphs, and separate fusion parameters are trained for each benchmark. It remains unclear whether the approach generalizes to multi-agent systems processing larger, noisier, and more diverse documents.

  1. Limited ability to recover information lost during independent processing.

The source and receiver process their contexts separately, and their states are combined only afterward. A simple elementwise correction cannot reconstruct all the interactions that would have occurred if both contexts had been processed together. The method also ignores Mamba's convolution states. This may impose a fundamental ceiling on how closely fusion can approximate full-context processing.

  1. Efficiency gains may not hold in distributed settings.

The reported 1.6× speedup is measured on a single GPU and excludes communication between devices. For Mamba-2.8B, the method transfers approximately 20 MB of hidden states, substantially more data than a short text summary. Without accounting for network transmission, it is unclear whether the method is faster in realistic distributed deployments.

  1. Limited evidence of superiority over stronger alternatives.

The natural-language baselines are restricted to short, question-agnostic summaries. Furthermore, the paper's MLP-based fusion baselines achieve better perplexity, although they perform worse on downstream tasks. Stronger communication baselines and comparisons under matched computational budgets would help establish when the proposed fusion mechanism is actually preferable.

  1. Low novelty

The main idea builds directly on existing state-composition methods, particularly State Soup, which already provides the mathematical decomposition used to combine SSM states.

The primary new component is a learned elementwise correction that rescales the source state. However, the paper's own results suggest that much of the improvement comes from the existing analytic rule:

  • The analytic fusion rule without learned parameters already closes 46% of the WikiText perplexity gap on Mamba-130M, compared with 69% for the proposed method.
  • 85.5% of the learned correction parameters remain within 5% of their initial value of 1.

These results suggest that the learned correction provides an incremental improvement rather than introducing a fundamentally new fusion mechanism.

The paper's main contribution is therefore the application and empirical validation of existing state-composition principles for communication between pretrained SSMs, rather than substantial methodological novelty.

Questions for authors
  1. Long-context performance: Why does state fusion perform worse than receiver-only at 8,192 tokens? Is this caused by the base model or the fusion mechanism?

  2. Heterogeneous models: Can the approach work between models with different architectures, layer counts, or state dimensions?

  3. Limited accuracy gains: Why does closing 74% of the output-distribution gap on HotpotQA improve actual answer accuracy by only 1.8 percentage points? Is the receiver effectively using the transferred information?

  4. Stronger baselines: How does the method compare against more advanced learned-fusion techniques and question-aware natural-language summaries under similar computational budgets?

  5. Real-world efficiency: Does the reported 1.6× speedup hold in distributed settings when accounting for the 20 MB state transfer and network latency?

2 comments
AuthorsOpenReview ↗

Author's Response to Reviewer AFCE

Thank you for raising the question of what the receiver gains from the source. We looked at where the gold answer appears in 1,000 HotpotQA questions.

W1 / Q3. Does the receiver use the source information? Fusion improves EM over receiver-only by 20 percentage points on the 229 questions where the gold answer appears only in the source paragraph (paired 95% CI 15 to 26). Full context gains about 36 points on those same questions, so fusion recovers 56% of that gain. On the 523 questions where the answer appears only in the receiver paragraph, fusion lowers EM by 8 points and full context lowers it by 9. Answer location helps explain which questions benefit from sharing the source, although it cannot identify the reasoning steps the reader used.

Across the full HotpotQA evaluation, fusion raises EM from 48.9% to 50.7%, recovering about 35% of the EM gap to full context. The Λ\Lambda = 1 ablation illustrates why this fraction differs from KL closure. The decay rule alone closes 64% of the KL gap while changing EM by only 0.1 points. We will report EM recovery alongside KL closure in the abstract.

W2 / Q1. Long contexts and multiple sources. At 8,192 tokens, Mamba-2.8B struggles even when it reads the full sequence directly. PG-19 perplexity is 118.44 with full context, 28.72 with fusion, and 16.25 with receiver-only. The checkpoint’s long-context limitation is visible without fusion, though fusion also hurts relative to receiver-only. At 2,048 tokens, the Mamba pretraining length and twice the fuser’s training length, fusion closes 18% to 51% of the perplexity gap across all five scales.

Splitting the source into more segments also lowers performance. Each segment starts from an empty state, and the fuser saw only single sources during training. The current experiment cannot separate those effects.

W3 / Q2 and W4. Model compatibility and independent processing. Our method assumes a shared checkpoint. Eq. 9 combines corresponding state entries at each layer, so models with different layer counts or state dimensions would need a mapping between those entries. We tested transfer across QA benchmarks with gold supporting paragraphs. A fuser trained on the other benchmark scored 0.7 EM points above Λ\Lambda = 1 on HotpotQA and 1.9 points above it on MuSiQue.

By the time the source state arrives, the receiver has already processed its paragraph. Fusing the final states cannot reproduce all the interactions that would have occurred during a joint read. Our fuser also omits the convolution states.

W5 / Q5. Distributed efficiency. Table 3 measures speed on one GPU. At 1 Gb/s, sending the state would add an estimated 84 ms in FP16 or 26 ms in INT4. FP16 matched FP32 in aggregate EM on both datasets. The table gives the change from FP16 on a 0–1 EM scale, with paired 95% intervals over questions.

FormatSizeHotpotQA ΔEM [paired 95% CI]MuSiQue ΔEM [paired 95% CI]
INT8 (per-channel scale)5.6 MiB+0.000 [−0.000, +0.001]+0.000 [+0.000, +0.000]
INT4 (per-channel scale)3.1 MiB+0.000 [−0.001, +0.002]+0.000 [−0.002, +0.002]

INT4 reduces the payload from 10 MiB to 3.1 MiB and preserves 99.3% of FP16 answers on HotpotQA and 98.0% on MuSiQue. At 10 and 100 Gb/s, ideal transfer times are 8.4 and 0.84 ms for FP16, or 2.6 and 0.26 ms for INT4. These estimates exclude network startup and quantization.

At natural HotpotQA paragraph lengths, reading both documents and the question in one pass took 178 ms per complete answer on an A100, compared with 239 ms for fusion. Reading both documents first and processing the question in a separate call took 237 ms. Table 3 reports 54.1% EM for full context and 50.7% for fusion. With both document states already available and a 2,072-token source, fusion took 62 ms to the first answer token versus 229 ms for full-prompt replay. Adding the estimated 26 ms INT4 transfer at 1 Gb/s leaves about 141 ms of that saving. At 571 tokens, the same transfer uses up the 18 ms local saving. We will add the raw-text comparison to Table 3 and qualify the speed claim in the abstract.

W6 / Q4. Stronger baselines. Table 3 compares fusion with prompted summaries generated before the sharer sees the question. A matched-budget comparison with question-aware senders remains open. Among the learned fusers, the parameter-matched MLP gives lower perplexity. On Mamba-130M, our fuser gives higher accuracy on ARC-Easy and HellaSwag, with paired 95% intervals of 0.34 to 2.53 and 0.11 to 0.99 percentage points for the respective gains. The PIQA interval includes zero.

W7. Contribution of the learned correction. We learn a per-layer correction to the existing decay rule while keeping both model instances frozen. With the QA reader fixed, learning Λ\Lambda improves EM over Λ\Lambda = 1 by 1.7 points on HotpotQA (paired 95% CI 1.1 to 2.2) and 4.0 points on MuSiQue (2.9 to 5.0).

Dear Reviewer AFCE, as the reviewer/author discussion period is coming to an end, we wanted to check whether we have addressed all your questions and concerns. We are happy to clarify anything further.

Reviewer_ZKHoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies efficient context transfer between two instances of the same pretrained State Space Model (SSM) that have processed different inputs. Building on a single-layer SSM state decomposition, the authors propose a dynamics-based fusion method in which the receiver applies its accumulated decay to the source state and then uses a lightweight learned per-channel correction Λ\Lambda, while updating only the correction parameters during fuser training. The method is evaluated on five sizes each of Mamba and Mamba-2, covering language modeling, zero-shot downstream tasks, and separate-document question answering on HotpotQA and MuSiQue. At the standard 1024-token setting, the method improves perplexity on WikiText-103 and PG-19 and accuracy on ARC-Easy and HellaSwag across all evaluated model sizes, while the PIQA results are mixed. In the QA experiments, state communication achieves EM scores comparable to natural-language summaries while avoiding generated message tokens and using much less communication-related computation. The main contributions are the directed source-to-receiver state communication setup, the frozen-model learned correction to the analytic fusion rule, and the broad evaluation across model scales and tasks.

Critical strengths
  1. The method is clear and well motivated. The receiver first adjusts the source state using its accumulated decay, then applies a small learned correction Λ\Lambda. Only the correction parameters are updated during fuser training. For the evaluated Mamba checkpoints, the fuser adds less than 1% of the base model parameter count; the paper does not report the corresponding parameter ratio for Mamba-2. The ablations show that accumulated decay is useful, and the proposed method gives better downstream accuracy than the tested MLP fusers, even though the MLPs sometimes achieve lower perplexity.
  2. The experiments are broad and mostly consistent. The paper evaluates five sizes each of Mamba and Mamba-2, two language-modeling datasets, several zero-shot tasks, and two multi-document QA benchmarks. Improvements appear across all model sizes on WikiText-103, PG-19, ARC-Easy, and HellaSwag. The Mamba-2 results also show that learning Λ\Lambda consistently improves over the parameter-free rule with Λ=1\Lambda=1.
  3. The efficiency direction is useful, and the paper reports its limitations honestly. State communication avoids generating message tokens and re-encoding a summary. On one GPU, it is faster than the tested summary pipelines and has much lower communication-related arithmetic. The paper also clearly reports the 20 MB state payload, the failure at 8192 tokens, and the loss of performance when one source is split into several segments.
Critical weaknesses
  1. The new technical contribution is narrower than the overall idea of state communication. The decay-based rule in Eq. (6), including its exactness for a single SSM layer and its approximation in a multilayer model, is closely related to A-decay mixing in State Soup and to the two-context case of CASO in PICASO. The paper cites this earlier idea, so this is not a missing-citation problem. The main new contribution is the learned correction Λ\Lambda, the directed source-to-receiver setting, and the larger evaluation. The language-modeling experiments compare learned Λ\Lambda with Λ=1\Lambda=1, but the HotpotQA and MuSiQue experiments do not provide this comparison.
  2. The practical speed comparison is incomplete. The reported 1.6x speedup is measured on a single H100 and excludes transferring the 20 MB state between devices. The paper also does not report the latency of directly sending the original source paragraph and processing it as full context. Therefore, the experiment shows a local compute advantage, but it does not yet establish an advantage in a general distributed setting.
  3. The QA comparison uses different amounts of communication-specific supervision. After a rank-16 answer-formatting LoRA adapter is trained on 5,000 examples and shared across all evaluation conditions, the state fuser receives an additional 80,000 benchmark-specific training examples for each QA benchmark using a full-context teacher. The natural-language baselines use fixed summarization prompts and receive no comparable communication-specific training. The comparison is still useful, but the similar EM scores may partly result from the additional training received by the state fuser.
  4. The demonstrated generalization has clear limits. At 8192 tokens, state communication is worse than receiver-only at every tested Mamba scale. Performance also decreases when one source is divided into multiple independently processed segments. In addition, all transfers use the same model checkpoint, and most controlled experiments split one continuous sequence artificially. The claims about long-context and multi-source generalization should therefore remain limited.
Questions for authors
  1. Can the authors report the parameter-free setting Λ=1\Lambda=1 on HotpotQA and MuSiQue? This would show how much of the QA improvement comes from the new learned correction rather than the existing decay-based composition rule.
  2. What is the end-to-end latency of directly sending the source paragraph and processing the full context? How would the comparison change after including state-transfer time at representative network speeds? Results with FP16, FP8, or INT8 state transmission would also help evaluate the payload-accuracy trade-off.
  3. Does a fuser trained on general text, or on one QA benchmark, transfer to the other QA benchmark? Alternatively, can the authors compare against a natural-language communication baseline with similar task-specific supervision?
  4. For the multi-source experiment, can training with multiple source segments separate the effect of training mismatch from the effect of additional segment boundaries? Since all source states are combined in one expression, repeated numerical fusion does not seem to explain the degradation.
  5. In the probing experiment, is the best layer selected using data separate from the test split used to compute the confidence interval? Selecting the best layer and computing the interval on the same test data may make the result look stronger than it is. (This is a minor issue and does not affect the main experimental conclusions.)
2 comments
AuthorsOpenReview ↗

Author's Response to Reviewer ZKHo

Thank you for suggesting the Λ\Lambda = 1 ablation and the transfer experiment. We ran both on HotpotQA and MuSiQue.

W1 / Q1. What does learning Λ\Lambda add? With the QA reader fixed, learning Λ\Lambda improves exact match over the decay rule alone by 1.7 percentage points on HotpotQA (paired bootstrap 95% CI 1.1 to 2.2) and 4.0 points on MuSiQue (2.9 to 5.0).

MethodHotpotQA EMHotpotQA KL closureMuSiQue EMMuSiQue KL closure
Receiver only0.4890%0.1630%
Λ\Lambda = 10.49064%0.18056.9%
Learned Λ\Lambda0.50774%0.22168%

On HotpotQA, Λ\Lambda = 1 already closes 64% of the KL gap to full context, yet changes EM by only 0.1 points over receiver-only (paired 95% CI −0.7 to 0.8). We will add this row to Table 3.

W2 / Q2. Raw text and state transmission. One-pass raw text was faster for the short paragraphs in Table 3. On one A100, reading both documents and the question together took 178 ms per complete answer on HotpotQA and 196 ms on MuSiQue. Fusion took 239 and 253 ms, respectively. When the receiver read both documents first and processed the question in a separate call, raw text took 237 and 253 ms. At these lengths, rereading the sharer’s paragraph is inexpensive.

These timings exclude network transfer. FP16 matched FP32 in aggregate EM on both datasets. The table gives the change in EM from FP16 on a 0–1 scale, with paired 95% intervals over questions.

FormatSizeHotpotQA ΔEM [paired 95% CI]MuSiQue ΔEM [paired 95% CI]
INT8 (per-channel scale)5.6 MiB+0.000 [−0.000, +0.001]+0.000 [+0.000, +0.000]
INT4 (per-channel scale)3.1 MiB+0.000 [−0.001, +0.002]+0.000 [−0.002, +0.002]

INT4 preserved 99.3% of FP16 answers on HotpotQA and 98.0% on MuSiQue. At ideal link speeds of 1, 10, and 100 Gb/s, sending the 10 MiB FP16 state takes an estimated 84, 8.4, and 0.84 ms. INT4 reduces those times to 26, 2.6, and 0.26 ms.

When both agents had already read their documents, fusion reached the first answer token 18 ms sooner than full-prompt replay at 571 source tokens and 167 ms sooner at 2,072 tokens. Adding 26 ms for INT4 transfer at 1 Gb/s uses up the saving at 571 tokens and leaves about 141 ms at 2,072. These estimates exclude network startup and quantization. We will add the raw-text baselines to Table 3 and report estimated transfer time separately from measured latency.

W3 / Q3. Cross-benchmark transfer and supervision. The MuSiQue-trained fuser scored 0.7 EM points above Λ\Lambda = 1 on HotpotQA. In the other direction, the HotpotQA-trained fuser scored 1.9 points above Λ\Lambda = 1 on MuSiQue. We kept the target reader fixed and transferred only the fuser. Each QA fuser was trained with its benchmark’s answer-format adapter. The WikiText fuser was trained without that adapter, so it does not apply directly to the adapted QA reader (Appendix H.1).

Target datasetΛ\Lambda = 1 EM in transfer experimentTransferred fuser EMIn-domain learned fuser EM (Table 3)
HotpotQA0.4900.4970.507
MuSiQue0.1800.1990.221

We agree that the supervision differs. Only the fuser receives communication-specific training from the full-context teacher. We will limit the comparison in Table 3 to the prompted summaries we evaluated.

W4. Length generalization. At 2,048 tokens, fusion improves PG-19 perplexity over receiver-only at all five Mamba scales, closing 18% to 51% of the gap to full context. This matches Mamba’s pretraining length and is twice the fuser’s training length. At 8,192 tokens, Mamba-2.8B perplexity reaches 118.44 when the model reads the full sequence directly. Fusion gives 28.72, still worse than the 16.25 from receiver-only. The checkpoint itself struggles at this length, and using the source state also hurts.

Q4. Multiple source segments. Our experiment cannot separate the effect of additional segment boundaries from the effect of training on a single source. As you point out, all source states enter one fusion expression. Each segment still starts from an empty state. Training a fuser on multiple segments and evaluating it on the same partitions as the single-source fuser would let us measure the effect of the training mismatch.

Q5. Probe layer selection. We now select the layer on one half of the test set and estimate the accuracy gain on the other half. With this split, fusion improves mean probe accuracy by 5.1 points on Mamba-130M (95% CI 4.2 to 6.0) and 6.2 points on Mamba-370M (5.2 to 7.2). We will include these results in Appendix G.

Dear Reviewer ZKHo, as the reviewer/author discussion period is coming to an end, we wanted to check whether we have addressed all your questions and concerns. We are happy to clarify anything further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

To the Area Chair and Reviewers

Dear Reviewers and Area Chair,

Thank you for the detailed reviews. We responded to the weaknesses and questions raised in each review and ran the experiments the reviewers requested. This comment summarizes the main points. The tables are in the individual threads.

Raw text and state transfer (ZKHo W2, AFCE W5, yRNs W1). We timed raw text on 1,000 questions per dataset on one A100. One-pass raw text is the full-context reference in Table 3, where one model reads both documents and the question together. It took 178 ms per complete answer on HotpotQA versus 239 ms for fusion. We treat it as a best-case reference, since it assumes the receiver has not yet read its own document. In our setting each agent reads its document before the question arrives. When raw text also processes the question in a separate call, it took 237 ms, close to fusion. With both document states already available, fusion reached the first answer token sooner from 571 source tokens onward, at 62 ms versus 229 ms for a 2,072-token source. INT4 reduces the state from 10 MiB in FP16 to 3.1 MiB and adds an estimated 26 ms at 1 Gb/s. The paired 95% intervals for the change in EM stay within ±0.2 percentage points. We will add the raw-text baselines to Table 3 and qualify the speed claim in the abstract.

What learning Λ\Lambda adds (ZKHo W1, AFCE W7, yRNs W2). We evaluated Λ\Lambda = 1 in the Table 3 QA setup. With the QA reader fixed, learning Λ\Lambda improves EM over the decay rule alone by 1.7 points on HotpotQA (paired 95% CI 1.1 to 2.2) and 4.0 points on MuSiQue (2.9 to 5.0). On HotpotQA, Λ\Lambda = 1 closes 64% of the KL gap but changes EM by only 0.1 points.

KL closure and answer quality (AFCE W1, yRNs W3). On the 229 HotpotQA questions where the gold answer appears only in the source paragraph, fusion improves EM over receiver-only by 20 points (paired 95% CI 15 to 26), which is 56% of the full-context gain. We will report EM recovery (about 35% and 44%) alongside KL closure in the abstract.

Supervision and transfer (ZKHo W3, yRNs W3). Only the fuser receives communication-specific training from the full-context teacher, so we will limit the comparison to the prompted summaries in Table 3. A fuser trained on the other benchmark scored 0.7 EM points above Λ\Lambda = 1 on HotpotQA and 1.9 points above it on MuSiQue.

Length generalization (ZKHo W4, AFCE W2, yRNs W5). At 2,048 tokens, the Mamba pretraining length and twice the fuser’s training length, fusion closes 18% to 51% of the perplexity gap at all five Mamba scales. At 8,192 tokens, fusion and full context both give worse perplexity than receiver-only at every scale.

Paired intervals and reproducibility (yRNs W2 and W4, ZKHo Q5). We added paired 95% intervals for the Table 4 comparisons. The gains over the parameter-matched MLP on ARC-Easy and HellaSwag have intervals above zero, and all three intervals against Λ\Lambda = 1 include zero. We also specified the MuSiQue paragraph assignment, the training-example count, the Table 14 subset, and the scoring procedure, corrected the compute accounting for Table 3, and repeated the probe analysis with layer selection on a held-out half.

We are happy to clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.