- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
MARCH stores checkpoints of the recurrent state at chunk boundaries. This leverages an attention-like structure to select which past state should be used (or no past states at all via null option) to augment the current query-to-output map. This work demonstrates the effectiveness of MARCH across commonsense reasoning tasks and (especially) long-context modeling.
Critical strengths
-
This method is well-targeted at solving problems with forgetting in modern SSMs / Linear Attention. Since the linear-state is often constraining, this method provides a parallelizable and nonlinear way to recall past information without significant overhead.
-
Overall, the method clearly improves over the baselines, especially in long-context scenarios in Table 3.
Critical weaknesses
-
Table 3, in some other form, should be combined with the main results of the paper. The first comparison of GDN, Transformers, and Log-Linear Attention does not provide convincing enough evidence on its own that this method is useful. This is especially relevant since Log-Linear Attention often underperforms GDN in some settings.
-
There should be more ablations on the central innovation of recalling previous state iterations. While the router ablations are nice, these are less critical to supporting the method. For example, it is unclear if storing the exact state is needed, or if the hybrid attention structure is performing any heavy lifting. A simple ablation could be replacing with a data-dependent “value”, rather than the output of the state.
-
Table 4 should be moved to the appendix. This subsection of positional embeddings does not seem very relevant to the main story of your method. If anything, Table 9 in the appendix is much more relevant.
-
It is unclear if the downstream task performance gains outweigh the costs of slowing down training throughput.
Questions for authors
-
In response to Appendix G, Q1: "Is it actually preferable to have overlapping content between checkpoints vs disjoint information? Are we giving up memory expressivity this way?"
-
My central question for W2’s ablation comment: “Why exactly do we need to store the past state and not some other form of data-dependent / state-agnostic information?”
7 comments
Thank you for your response. I have a few additional questions before I can draw an informed conclusion from the rebuttal.
W2) I am finding it difficult to understand the exact computation used for each row in the table. Additionally, could you explain why we would want to be averaging over h?
W4) Could you also report language modeling & real-world recall results (SWDE, FDA, SQuAD) under these time-matched training budgets? This should be straight-forward as these models are already trained. I find it difficult to believe that this improvement of MARCH holds outside of synthetic recall and worry these results are cherry-picked.
Additionally, I am also very concerned in the large difference in time-to-first-token and throughput compared to GDN. The performance gains over GDN are not significant; however, the increase in compute time is.
Thank you for the follow-up questions. We clarify the ablation setups and report additional results on downstream performance and inference cost.
W2. Ablation details
Let denote the recurrent checkpoint for segment and head . The default historical readout is . Routing weights are obtained by applying softmax to the routing logits over visible checkpoints and the null option.
To construct the suggested data-dependent value, we use . The average runs over text token positions in segment and preserves the hidden dimension. Mean pooling lets every token contribute directly to a single segment summary. A learned projection then produces one fixed value vector per checkpoint and head. This ablation tests whether a simple whole-segment summary can serve as the historical payload under the same routing rule and checkpoint schedule.
| Variant and construction | Mean NIAH6 (4K–32K) |
|---|---|
| Anchor-derived keys. , where is the normalized anchor representation informed by aligned checkpoint reads in earlier layers. | 48.67 |
| Pooled-segment keys. . | 39.67 |
| State-derived keys. , where is a learned probe for head . | 44.79 |
| Pooled-vector values. Replace with the head- slice of and retain anchor-derived routing. | 33.7 |
| Uniform history. Set for each of the visible checkpoints, preserving the null probability. | 33.93 |
W4. Downstream performance and inference cost
We evaluated the same checkpoints as in our NIAH comparison. Based on measured training throughput, GDN at ~34B tokens and MARCH at ~17B tokens have comparable estimated training times.
| Model | Commonsense Avg. ↑ | SQuAD ↑ | SWDE ↑ | FDA ↑ |
|---|---|---|---|---|
| GDN (~34B tokens) | 40.09 | 29.59 | 24.57 | 14.07 |
| MARCH (~17B tokens) | 40.13 | 30.97 | 43.11 | 29.95 |
The gains extend beyond synthetic recall: MARCH improves SWDE by 18.54 points, FDA by 15.88 points, and SQuAD by 1.38 points, with similar commonsense performance.
In a separate inference benchmark, we used one GPU, BF16, batch size 4, and 1,024 generated tokens. Each context-length cell reports time to first token (ms) / aggregate decode throughput (tokens/s) / analytical cache size at the end of prefill (GiB).
| Configuration | Mean NIAH6 (4K–32K, %) ↑ | 4K | 16K | 32K | 64K |
|---|---|---|---|---|---|
| GDN | 34.6 | 56.1 / 2,168 / 0.072 | 213.2 / 2,168 / 0.072 | 419.9 / 2,168 / 0.072 | 847.7 / 2,168 / 0.072 |
| Dense MARCH, chunk 512 | 48.7 | 197.2 / 1,548 / 0.600 | 727.3 / 1,251 / 2.182 | 1,660.2 / 989 / 4.291 | 4,288.6 / 720 / 8.511 |
| Top-4 MARCH, chunk 512 | 41.3 | 195.1 / 1,127 / 0.600 | 654.6 / 1,121 / 2.182 | 1,280.5 / 1,112 / 4.291 | 2,563.8 / 1,101 / 8.511 |
| Transformer (RoPE) | 52.6 | 87.0 / 914 / 2.215 | 548.7 / 606 / 8.859 | 1,700.6 / 423 / 17.719 | 6,035.2 / 260 / 35.438 |
| Hybrid MARCH + full attention | 77.8 | 159.0 / 1,618 / 0.920 | 631.3 / 1,216 / 3.516 | 1,506.2 / 911 / 6.977 | 4,110.7 / 605 / 13.899 |
Dense MARCH improves mean NIAH6 from 34.6% to 48.7% over GDN, at the cost of higher first-token latency, lower decode throughput, and a larger cache. Compared with the Transformer, dense MARCH scores 48.7% vs. 52.6%, while delivering 2.77× the decode throughput and using 76.0% less estimated cache at 64K.
Hybrid MARCH achieves the highest mean NIAH6 accuracy (77.8% vs. 52.6% for the Transformer). At 64K, it also delivers 2.33× the Transformer’s decode throughput, with 31.9% lower first-token latency and 60.8% less estimated cache.
Overall, dense MARCH offers a middle ground between GDN’s inference efficiency and the Transformer’s retrieval accuracy. Combining MARCH with full attention improves retrieval accuracy over the Transformer while retaining lower latency, higher decode throughput, and a smaller estimated cache at 64K.
W1/W3. Presentation of the main evidence
We agree that the paired comparisons across GDN, KDA, and GDN2 in Table 3 deserve greater prominence. These comparisons show average improvements in long-context retrieval across all three backbones under the shared training protocol.
We will integrate Table 3 into the main results section and move the detailed positional-encoding study in Table 4 to the appendix. We will also bring Table 9 into the main text, as checkpoint intervals and state-bank organization directly characterize the method’s memory–retrieval trade-off.
W2/Q2. Why retain historical recurrent states?
Each checkpoint preserves an associative memory that subsequent queries can read differently through Amqt. Both state-based and vector-based memories support query-dependent routing. Retaining a matrix-valued state additionally allows the current query to determine which associations to extract within each checkpoint.
To evaluate this design, we trained a variant that replaces historical state readouts with learned projections of segment-mean hidden representations,
vm=WvMeani∈Bm(hi).
The variant retains the online GDN, anchor-based routing, null route, and additive fusion, using the same backbone and nominal 50B-token training budget.
Averaged over six NIAH tasks at 4K, 8K, 16K, and 32K, MARCH achieves 48.67, compared with 33.70 for the pooled-value variant, a 14.97% gain. This comparison supports the benefit of the historical-state representation and readout design over the pooled-vector alternative. Detailed task- and length-wise results are provided in the appendix.
We also evaluated how the historical states are indexed. The two routing-key comparisons use the same checkpoint bank and reader.
Q1. Overlapping versus disjoint checkpoints
Overlapping checkpoints involve a trade-off between distinct local coverage and contextual continuity. We preserve full-sequence recurrence so that checkpoints capture successive stages of the evolving memory, including associations that later decay and delta updates may weaken or overwrite.
Under matched cache capacity, cumulative and disjoint checkpointing achieve 46.4 and 45.1, respectively. The 1.3-point improvement supports cumulative checkpointing under the evaluated budget.
Separating memory evolution from checkpoint placement also provides a basis for future adaptive caching based on memory changes or retrieval utility.
W4. Performance relative to training cost
We agree that retrieval gains should be assessed alongside training cost. Using measured throughput to estimate training time, we compare MARCH trained on approximately 17B tokens with GDN trained on approximately 34B tokens.
Under these comparable estimated training-time budgets, MARCH improves average NIAH accuracy across 4K, 8K, and 16K from 31.1 to 36.8, a 5.7% gain, with improvements at all three lengths. These results support the value of the additional computation for long-context retrieval in this setting.
From a practical perspective, hybrid models such as Qwen3.5 combine Gated DeltaNet with full-attention layers, providing an example of complementing efficient recurrence with richer context access. MARCH explores a related trade-off through retrieval from historical recurrent states.