- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes communication between two models sharing a frozen state-space language model by transferring and merging their internal recurrent states. Its main contribution is a learned elementwise correction to an existing decay-based state-composition rule, trained through distillation from full-context predictions without updating the backbone. Evaluation spans ten Mamba/Mamba-2 checkpoints, language modeling, commonsense reasoning, and multi-hop question answering. The results generally improve upon receiver-only context, while question-answering performance is comparable to text-summary communication with lower measured communication-stage computation and latency. Full-context processing remains stronger, and the efficiency measurements exclude transmission of the relatively large state payload. Ablations examine alternative fusion rules, context lengths, and document partitions.
Critical strengths
1.Simple and decently motivated method: The single-layer decomposition is correct under its stated assumptions, and the distinction between this identity and approximate multilayer fusion is clearly explained. Learning an elementwise correction while freezing the base checkpoint is an economical adaptation strategy. Appendix H.1 also clearly specifies that the QA answer-format adapter is shared across conditions and frozen during fuser training.
-
Broad and consistent language-modeling evidence: Perplexity improves over receiver-only processing at all ten checkpoints on both WikiText-103 and PG-19. The learned correction improves on the analytic prior in the reported comparisons, including all five Mamba-2 scales (Table 8). ARC-Easy and HellaSwag point estimates also improve across both model families, although the magnitude varies by task and scale.
-
Design ablations: Table 4 compares the method with the analytic prior, a parameter-matched linear fuser without decay, and a parameter-matched MLP. The improvement over the no-decay linear baseline, together with Figure 3b's transfer to shorter sequences, supports the value of retaining accumulated decay as an input-dependent factor. The MLP results also expose an informative difference between perplexity and downstream accuracy.
-
Disclosure of limitations: The paper reports negative results at 8192 tokens, degradation with additional independently processed source segments, mixed PIQA results, and the omission of convolution-state fusion. It also explicitly limits the latency measurements to a single GPU and excludes inter-device transmission. These disclosures help delimit the contribution.
Critical weaknesses
- The practical efficiency advantage is not established against raw-text transfer:
Table 3 compares state communication with pipelines that generate summaries autoregressively. However, the introduction also motivates avoiding the receiver's reprocessing of raw source text, and this alternative is not timed. Also, The full-context condition already achieves higher HotpotQA EM than state communication, 0.541 versus 0.507, but its latency and computational cost are absent. This is my big concern; I don't feel motivated enough to use this method anywhere unless the latency is significantly lower, 2x is still not worth it, given that we loose interpretability. The comparison should use the same assumptions about which source and receiver computations have already occurred. This would establish whether state fusion offers a useful quality–latency tradeoff over simply transferring the paragraph and processing the combined prompt.
The reported 180 ms versus 278/292 ms measurements support the stated comparison with summary pipelines. Their broader practical significance remains uncertain without the raw-text baseline and a stage-level breakdown. The 20 MB FP32 payload also makes deployment conditions relevant. For illustration, transmitting 20 MB at ideal link throughput takes roughly 16 ms at 10 Gbps or 160 ms at 1 Gbps, before other overheads or overlap. The former would reduce the reported advantage; the latter could eliminate it. A bandwidth sensitivity analysis would clarify where the method is useful.
- The incremental contribution of the learned correction needs stronger task-level evidence:
The decay-based composition rule and its approximate application to stacked Mamba layers already appear in State Soup, including experiments on independently processed sequence chunks. Pióro et al., 2024 The main methodological addition here is therefore the learned correction, alongside the broader evaluation and communication setting. Its perplexity benefit is demonstrated, but the analytic prior with Λ = 1 is missing from the QA comparison in Table 3. This leaves its incremental value unclear on the task most directly supporting the communication claim.
The design comparison also requires uncertainty estimates. In Table 4, MLP-CW achieves better WikiText/PG-19 perplexity than the proposed fuser, 20.33/18.75 versus 20.44/18.78, while the proposed fuser has higher downstream accuracy point estimates. Its accuracy improvements over the analytic prior are 0.30, 0.21, and 0.15 percentage points on ARC-Easy, PIQA, and HellaSwag, respectively. These differences may be meaningful, but their statistical reliability cannot be determined from aggregate accuracies alone. Paired confidence intervals and the number of training seeds are needed to substantiate the claim of a better downstream inductive bias. The existing confidence intervals in Appendix H.4 provide useful evidence of improvement over receiver-only on a separate HotpotQA sample, but do not resolve these design comparisons.
Sum and Average are reasonable ablations under the frozen-checkpoint constraint. Their poor performance in this setting should not be interpreted as a comparison with the complete training procedures of prior state-aggregation methods.
- The QA comparison does not isolate the benefit of the communication channel from task-specific adaptation:
the QA fuser receives 80,000 sampled training examples per benchmark and supervision from the full-context teacher, whereas the summary mechanism uses fixed prompts without comparable task-specific summarization training. The source lacks the question in both conditions, and the answer-format adapter is shared, so those aspects are controlled. Nevertheless, matching these prompted summaries does not establish that state communication would remain competitive with an adapted summarizer. A stronger summary baseline would make the comparison more informative.
The distinction between distribution recovery and answer quality also matters. Held-out KL is a valid metric for the stated distillation objective, but it does not by itself establish equivalent task-level recovery. From Table 3, EM gap closure is approximately 35% on HotpotQA and 44% on MuSiQue, compared with KL closure of 74% and 68%. The paper acknowledges this distinction in 4.3; the abstract would be more informative if it also conveyed the extent of answer-quality recovery.
- Evaluation ambiguities and an unexplained table discrepancy limit the reproducibility:
In appendix H.1 describes both QA benchmarks as having two supporting paragraphs per question and reports evaluation on all 2,417 MuSiQue development examples. However, the original Musique development set contains 1,252 two-hop, 760 three-hop, and 405 four-hop questions. The paper needs to explain how supporting paragraphs for three- and four-hop questions are mapped into its two-document setting. The original training set contains 19,938 questions, so the stated 80,000 sampled training examples also require an explanation of replacement sampling or other example construction. These choices could be valid, but they materially affect interpretation of the experiment. Trivedi et al., 2022, Table 2
Table 14 reports receiver/fused WikiText perplexities of 20.861/19.987 for Mamba-130M at split 0.5 and length 1024, whereas Table 4 reports 21.42/20.44 under the same stated configuration. If the layer analysis uses a different subset or protocol, this should be specified; otherwise the discrepancy needs correction.
The zero-shot evaluation description also lacks the precise prompt partitioning and scoring procedure, including treatment of answer choices. For QA, the token positions and aggregation used to compute KL closure should be stated explicitly. These details are necessary to reproduce the results and understand what information each model receives.
- The scope of demonstrated generalization is narrower than the abstract suggests
The successful length-transfer experiments cover 64–512 tokens, all shorter than the 1024-token training sequences. At 8192 tokens, fusion performs worse than receiver-only at every evaluated Mamba scale. Full-context processing also degrades, so this experiment does not isolate fusion error from the base models' length extrapolation limits. Evaluation at an intermediate length such as 2048 would be more informative about extension beyond the fuser's training length.
Tables 12–13 contain lower ARC-Easy point estimates after fusion when the source holds little context. The paper reports these limitations, but its broad generalization statements should distinguish the demonstrated perplexity transfer from less consistent task-level behavior.
Questions for authors
What are the latency and computational cost of raw-text transfer/full-context processing and receiver-only under the Table 3 protocol? Please report the stages included, precision, batch size, actual summary lengths, and timing variation. How is decay accumulation accounted for in the 0.02 GFLOPs figure, including any quantities reused from the receiver's forward pass?
Can you add the Λ = 1 analytic prior to Table 3 and report its EM and KL closure under the identical QA setup? This would directly establish the learned correction's contribution?
Can you provide paired confidence intervals for the main QA and downstream comparisons, particularly the proposed fuser versus the analytic prior and MLP-CW, and report the number of training seeds?
How does state communication compare with a summary mechanism given comparable task-specific adaptation? If such a comparison is unavailable, please delimit the conclusions to the prompted summary baselines evaluated here?
How are MuSiQue's three- and four-hop examples represented by two documents, and how are 80,000 training examples constructed? Please also explain the Table 14/Table 4 discrepancy and specify the zero-shot partitioning/scoring procedure and QA KL evaluation positions.
3 comments
Author's Response to Reviewer yRNs (2/2)
W2 / Q2–Q3. Learned correction, paired intervals, and training seeds. Learning improves EM over the fixed decay rule by 1.7 points on HotpotQA (paired 95% CI 1.1 to 2.2) and 4.0 points on MuSiQue (2.9 to 5.0). These gains use paired question-level differences. With = 1, HotpotQA reaches 49.0% EM and 64.0% KL closure, and MuSiQue reaches 18.0% EM and 56.9% KL closure. We will add the = 1 row to Table 3.
In Table 4, our fuser scores 39.27% on ARC-Easy versus 37.79% for the parameter-matched MLP, a gain of 1.48 percentage points. Below are the paired 95% intervals from per-question Mamba-130M predictions, in percentage points. We trained each fuser once, so the intervals capture variation across evaluation questions.
| Learned compared with | ARC-Easy | PIQA | HellaSwag |
|---|---|---|---|
| = 1 | [−0.42, +1.01] | [−0.44, +0.82] | [−0.09, +0.51] |
| Parameter-matched MLP | [+0.34, +2.53] | [−0.27, +1.41] | [+0.11, +0.99] |
The gains over the MLP on ARC-Easy and HellaSwag have intervals above zero. Against = 1, all three downstream intervals include zero. We will distinguish these smaller downstream differences from the QA gains and describe Sum and Average as ablations on the frozen checkpoint.
W3 / Q4. Supervision for summaries. All methods share the answer-format adapter, but only the fuser receives communication-specific training from the full-context teacher. We will limit the comparison to the prompted summaries in Table 3. Training a summarizer with the same teacher and examples remains future work. In the abstract, we will also report how much of the EM gap fusion recovers on HotpotQA and MuSiQue (about 35% and 44%) alongside KL closure (74% and 68%).
W4 / Q5. Reproducibility.
- MuSiQue. We use the first two supporting paragraphs in dataset order, giving the first to the sharer and the second to the receiver. For the 1,165 three- and four-hop questions, this leaves out supporting evidence in every condition, including full context. For the 1,252 two-hop questions, the two paragraphs contain all supporting evidence. EM on this subset is 0.224 for receiver-only, 0.305 for fusion, and 0.403 for full context.
- Training examples. The 80,000 comes from 20,000 training steps with four questions per step. A question counts each time it is used. MuSiQue has 19,938 distinct training questions, so this amounts to about four passes through the dataset, with reshuffling between passes. We will replace the one-epoch description in Appendix H.1 with the step count.
- Tables 14 and 4. The layer analysis in Table 14 used a 200-sequence subset of the WikiText test set. Table 4 used 279 sequences. We will specify the subset in the Table 14 caption.
- Scoring. We split the tokenized prompt in half, keeping answer choices out of the prompt, and score each choice as a continuation. We normalize by choice length on ARC-Easy and HellaSwag and use summed log probability on PIQA. For QA, we teacher-force each method on the gold answer and end-of-turn token. We average token KL within each question first, then across questions.
W5. Intermediate context length. Fusion improves PG-19 perplexity over receiver-only at 2,048 tokens across all five Mamba scales. This is twice the fuser’s training length and matches the pretraining length of the checkpoints. The table gives the percentage of the gap from receiver-only to full context that fusion closes.
| Total sequence length | 130M | 370M | 790M | 1.4B | 2.8B |
|---|---|---|---|---|---|
| 1,024 tokens | 64% | 41% | 51% | 33% | 35% |
| 2,048 tokens | 51% | 28% | 44% | 19% | 18% |
Both rows use the Appendix D.4 long-document protocol. At each length, we evaluated 200 sequences and split the first 75% of each sequence equally between the agents.
At 8,192 tokens, fusion and full context both give worse perplexity than receiver-only at every scale. We will distinguish these perplexity results from the mixed ARC-Easy results when the sharer receives only a small part of the prompt.
Dear Reviewer yRNs, as the reviewer/author discussion period is coming to an end, we wanted to check whether we have addressed all your questions and concerns. We are happy to clarify anything further.
Author's Response to Reviewer yRNs (1/2)
Thank you for suggesting the direct comparison with raw text.
W1 / Q1. Raw-text latency. Reading both documents and the question in one pass was fastest for the short paragraphs on both datasets. On HotpotQA, Table 3 also gives higher EM for full context than for fusion (54.1% versus 50.7%). Fusion reached the first answer token sooner on longer sources when both agents had already read their documents.
Both clocks stop at the last answer token. Total includes any initial document reads the method uses, with independent reads running in parallel. Warm starts after each agent has read its own document. All timings exclude network transfer. One-pass raw text reads both documents and the question together. The re-read baseline reads both documents together, then processes the question in a separate call. Both start from raw text, so their total and warm times are identical. The append baseline reuses the receiver’s state, reads the source text, then processes the question.
We timed 1,000 questions per dataset on one A100 with FP16, batch size one, and greedy decoding. The table gives mean latency in milliseconds, with standard deviations in parentheses. With the paper’s prompts, short and medium summaries averaged 5.2 and 5.6 tokens on HotpotQA and 12.0 and 18.1 on MuSiQue.
Receiver-only processes the question in a separate call from the saved state, which adds overhead at these short lengths. When raw text also processed the question separately, its total latency was close to fusion on both datasets. Fusion added 3 to 4 ms over receiver-only. In a separate timing run with the same harness, HotpotQA fusion took 234 ms. About 49 ms went to parallel document reads, 2.7 ms to packing, copying, and fusing the states, 58 ms to processing the question, and 123.5 ms to decoding the answer.
Source-length sweep. To simulate sending a longer document, we padded the sharer’s paragraph with unrelated text and evaluated 200 HotpotQA questions per length. Both agents had already read their documents. Raw text requires the receiver to process the sharer’s tokens again, while fusion reuses the sharer’s state. We timed the first answer token because output lengths varied. At long source lengths, raw text sometimes generated long, incorrect answers, which would add decoding time to the comparison.
With both states available, raw text was faster at 130 tokens and fusion was faster at every tested length from 571 onward. This applies, for example, when the sharer reuses a document state across questions. Including the initial reads made raw text faster at shorter lengths and brought first-token times within about 3% at 4,069 and 8,071 tokens. Appending to the receiver’s saved state was slower than one-pass raw text at all six lengths.
Transfer time. At ideal link speeds of 1 and 10 Gb/s, sending the 10 MiB FP16 state takes an estimated 84 and 8.4 ms. INT4 reduces this to 26 and 2.6 ms. At 1 Gb/s, the 26 ms transfer exceeds fusion’s 18 ms local saving at 571 tokens. At 2,072 tokens, it brings fusion to about 88 ms versus 229 ms for one-pass raw text, a projected saving of 141 ms.
FP16 matched FP32 in aggregate EM on both datasets. For INT4 relative to FP16, the paired 95% confidence intervals for the change in EM were within ±0.2 percentage points on both datasets. We will add the raw-text baselines to Table 3 and specify when the speed comparison assumes existing document states.
Compute accounting. The 0.02 GFLOPs in Table 3 counts final fusion on the receiver. Two multiplications and one addition per state entry across 64 × 5,120 × 16 entries give 0.016 GFLOPs. This step took 2.30 to 2.37 ms on an A100 (Table 15). Eq. 8 reuses the receiver’s step sizes. For a 134-token receiver paragraph, forming the accumulated decay takes 43.6 million additions, 5.24 million multiplications, and 5.24 million exponentials. Counting each exponential once gives 0.054 billion scalar operations for decay and 0.070 billion including fusion. We will correct Table 3, Section 4.4, and Appendices F.1 and H.3. For raw text and receiver-only, we measured latency only.