acceptodds
Under review as a conference paper at ICLR 2027

MARCH: Scaling Recurrent Memory with Content-Routed State Anchors

Abstract

Transformers owe much of their strong long-context retrieval capability to a token-level memory that grows with context length. This flexibility, however, incurs quadratic computational complexity during training and a key-value cache that grows linearly during autoregressive inference. Recurrent alternatives offer efficient decoding by compressing the entire history into a fixed-size state, but often underperform on recall-intensive tasks because earlier associations are usually overwritten by subsequent updates, leaving only the most recent contextual information available. In this paper, we introduce Memory-Anchor Routing across Context History (MARCH), a network architecture that effectively scales state-space models beyond a fixed-size dimension while maintaining computational efficiency over long sequences. MARCH periodically caches cumulative recurrent-state checkpoints as state anchors and associates each anchor with a compact, content-conditioned anchor key. This allows MARCH to maintain a memory bank that grows with context length, providing a controllable trade-off between historical resolution and memory cost. At each token, MARCH produces an anchor query to attend to all causally available state anchors and computes the output by aggregating the historical-anchor readouts together with the current-state readout in an attention-style manner. We show that, after standard pretraining, MARCH consistently outperforms multiple linear-attention variants across commonsense reasoning, LongBench, and in-context retrieval. These results demonstrate that content-routed state caching substantially strengthens recurrent long-range memory while preserving its native computational path.

Primary area:
foundation or frontier models, including LLMs
Keywords:
linear attention, recurrent memory, long-context modeling, state expansion

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_KVYdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

MARCH stores checkpoints of the recurrent state at chunk boundaries. This leverages an attention-like structure to select which past state should be used (or no past states at all via null option) to augment the current query-to-output map. This work demonstrates the effectiveness of MARCH across commonsense reasoning tasks and (especially) long-context modeling.

Critical strengths
  1. This method is well-targeted at solving problems with forgetting in modern SSMs / Linear Attention. Since the linear-state is often constraining, this method provides a parallelizable and nonlinear way to recall past information without significant overhead.

  2. Overall, the method clearly improves over the baselines, especially in long-context scenarios in Table 3.

Critical weaknesses
  1. Table 3, in some other form, should be combined with the main results of the paper. The first comparison of GDN, Transformers, and Log-Linear Attention does not provide convincing enough evidence on its own that this method is useful. This is especially relevant since Log-Linear Attention often underperforms GDN in some settings.

  2. There should be more ablations on the central innovation of recalling previous state iterations. While the router ablations are nice, these are less critical to supporting the method. For example, it is unclear if storing the exact state is needed, or if the hybrid attention structure is performing any heavy lifting. A simple ablation could be replacing om(l)o_m^{(l)} with a data-dependent “value”, rather than the output of the state.

  3. Table 4 should be moved to the appendix. This subsection of positional embeddings does not seem very relevant to the main story of your method. If anything, Table 9 in the appendix is much more relevant.

  4. It is unclear if the downstream task performance gains outweigh the costs of slowing down training throughput.

Questions for authors
  1. In response to Appendix G, Q1: "Is it actually preferable to have overlapping content between checkpoints vs disjoint information? Are we giving up memory expressivity this way?"

  2. My central question for W2’s ablation comment: “Why exactly do we need to store the past state and not some other form of data-dependent / state-agnostic information?”

7 comments

W1/W3. Presentation of the main evidence

We agree that the paired comparisons across GDN, KDA, and GDN2 in Table 3 deserve greater prominence. These comparisons show average improvements in long-context retrieval across all three backbones under the shared training protocol.

We will integrate Table 3 into the main results section and move the detailed positional-encoding study in Table 4 to the appendix. We will also bring Table 9 into the main text, as checkpoint intervals and state-bank organization directly characterize the method’s memory–retrieval trade-off.

W2/Q2. Why retain historical recurrent states?

Each checkpoint preserves an associative memory that subsequent queries can read differently through AmqtA_mq_t. Both state-based and vector-based memories support query-dependent routing. Retaining a matrix-valued state additionally allows the current query to determine which associations to extract within each checkpoint.

To evaluate this design, we trained a variant that replaces historical state readouts with learned projections of segment-mean hidden representations,

vm=WvMean⁡i∈Bm(hi).v_m = W_v\operatorname{Mean}_{i\in B_m}(h_i).

The variant retains the online GDN, anchor-based routing, null route, and additive fusion, using the same backbone and nominal 50B-token training budget.

Averaged over six NIAH tasks at 4K, 8K, 16K, and 32K, MARCH achieves 48.67, compared with 33.70 for the pooled-value variant, a 14.97% gain. This comparison supports the benefit of the historical-state representation and readout design over the pooled-vector alternative. Detailed task- and length-wise results are provided in the appendix.

We also evaluated how the historical states are indexed. The two routing-key comparisons use the same checkpoint bank and reader.

MethodNIAH Avg. ↑ (4K–32K)
MARCH — anchor-derived keys48.7
MARCH — pooled-segment keys39.7
MARCH — state-derived keys44.8
MARCH — uniform history33.9

Q1. Overlapping versus disjoint checkpoints

Overlapping checkpoints involve a trade-off between distinct local coverage and contextual continuity. We preserve full-sequence recurrence so that checkpoints capture successive stages of the evolving memory, including associations that later decay and delta updates may weaken or overwrite.

Under matched cache capacity, cumulative and disjoint checkpointing achieve 46.4 and 45.1, respectively. The 1.3-point improvement supports cumulative checkpointing under the evaluated budget.

Separating memory evolution from checkpoint placement also provides a basis for future adaptive caching based on memory changes or retrieval utility.

W4. Performance relative to training cost

We agree that retrieval gains should be assessed alongside training cost. Using measured throughput to estimate training time, we compare MARCH trained on approximately 17B tokens with GDN trained on approximately 34B tokens.

Under these comparable estimated training-time budgets, MARCH improves average NIAH accuracy across 4K, 8K, and 16K from 31.1 to 36.8, a 5.7% gain, with improvements at all three lengths. These results support the value of the additional computation for long-context retrieval in this setting.

From a practical perspective, hybrid models such as Qwen3.5 combine Gated DeltaNet with full-attention layers, providing an example of complementing efficient recurrence with richer context access. MARCH explores a related trade-off through retrieval from historical recurrent states.

Thank you for your response. I have a few additional questions before I can draw an informed conclusion from the rebuttal.

W2) I am finding it difficult to understand the exact computation used for each row in the table. Additionally, could you explain why we would want to be averaging over h?

W4) Could you also report language modeling & real-world recall results (SWDE, FDA, SQuAD) under these time-matched training budgets? This should be straight-forward as these models are already trained. I find it difficult to believe that this improvement of MARCH holds outside of synthetic recall and worry these results are cherry-picked.

Additionally, I am also very concerned in the large difference in time-to-first-token and throughput compared to GDN. The performance gains over GDN are not significant; however, the increase in compute time is.

Thank you for the follow-up questions. We clarify the ablation setups and report additional results on downstream performance and inference cost.

W2. Ablation details

Let Am,jA_{m,j} denote the recurrent checkpoint for segment mm and head jj. The default historical readout is rt,m,j=Am,jqt,jhistr_{t,m,j}=A_{m,j}q^{\mathrm{hist}}_{t,j}. Routing weights πt,j,m\pi_{t,j,m} are obtained by applying softmax to the routing logits over visible checkpoints and the null option.

To construct the suggested data-dependent value, we use hˉm=1∣Bm∣∑i∈Bmhi\bar h_m=\frac{1}{\lvert B_m\rvert}\sum_{i\in B_m}h_i. The average runs over text token positions in segment BmB_m and preserves the hidden dimension. Mean pooling lets every token contribute directly to a single segment summary. A learned projection then produces one fixed value vector per checkpoint and head. This ablation tests whether a simple whole-segment summary can serve as the historical payload under the same routing rule and checkpoint schedule.

Variant and constructionMean NIAH6 (4K–32K)
Anchor-derived keys. κm=Wκum\kappa_m=W_\kappa u_m, where umu_m is the normalized anchor representation informed by aligned checkpoint reads in earlier layers.48.67
Pooled-segment keys. κm=Wκhˉm\kappa_m=W_\kappa\bar h_m.39.67
State-derived keys. κm=WsRMSNorm⁡(Concat⁡j(Am,j⊤pj))\kappa_m=W_s\operatorname{RMSNorm}\bigl(\operatorname{Concat}_j(A_{m,j}^{\top}p_j)\bigr), where pjp_j is a learned probe for head jj.44.79
Pooled-vector values. Replace rt,m,jr_{t,m,j} with the head-jj slice of WvhˉmW_v\bar h_m and retain anchor-derived routing.33.7
Uniform history. Set π~t,j,m=(1−πt,j,∅)/Mt\tilde{\pi}_{t,j,m}=(1-\pi_{t,j,\varnothing})/M_t for each of the MtM_t visible checkpoints, preserving the null probability.33.93

W4. Downstream performance and inference cost

We evaluated the same checkpoints as in our NIAH comparison. Based on measured training throughput, GDN at ~34B tokens and MARCH at ~17B tokens have comparable estimated training times.

ModelCommonsense Avg. ↑SQuAD ↑SWDE ↑FDA ↑
GDN (~34B tokens)40.0929.5924.5714.07
MARCH (~17B tokens)40.1330.9743.1129.95

The gains extend beyond synthetic recall: MARCH improves SWDE by 18.54 points, FDA by 15.88 points, and SQuAD by 1.38 points, with similar commonsense performance.

In a separate inference benchmark, we used one GPU, BF16, batch size 4, and 1,024 generated tokens. Each context-length cell reports time to first token (ms) / aggregate decode throughput (tokens/s) / analytical cache size at the end of prefill (GiB).

ConfigurationMean NIAH6 (4K–32K, %) ↑4K16K32K64K
GDN34.656.1 / 2,168 / 0.072213.2 / 2,168 / 0.072419.9 / 2,168 / 0.072847.7 / 2,168 / 0.072
Dense MARCH, chunk 51248.7197.2 / 1,548 / 0.600727.3 / 1,251 / 2.1821,660.2 / 989 / 4.2914,288.6 / 720 / 8.511
Top-4 MARCH, chunk 51241.3195.1 / 1,127 / 0.600654.6 / 1,121 / 2.1821,280.5 / 1,112 / 4.2912,563.8 / 1,101 / 8.511
Transformer (RoPE)52.687.0 / 914 / 2.215548.7 / 606 / 8.8591,700.6 / 423 / 17.7196,035.2 / 260 / 35.438
Hybrid MARCH + full attention77.8159.0 / 1,618 / 0.920631.3 / 1,216 / 3.5161,506.2 / 911 / 6.9774,110.7 / 605 / 13.899

Dense MARCH improves mean NIAH6 from 34.6% to 48.7% over GDN, at the cost of higher first-token latency, lower decode throughput, and a larger cache. Compared with the Transformer, dense MARCH scores 48.7% vs. 52.6%, while delivering 2.77× the decode throughput and using 76.0% less estimated cache at 64K.

Hybrid MARCH achieves the highest mean NIAH6 accuracy (77.8% vs. 52.6% for the Transformer). At 64K, it also delivers 2.33× the Transformer’s decode throughput, with 31.9% lower first-token latency and 60.8% less estimated cache.

Overall, dense MARCH offers a middle ground between GDN’s inference efficiency and the Transformer’s retrieval accuracy. Combining MARCH with full attention improves retrieval accuracy over the Transformer while retaining lower latency, higher decode throughput, and a smaller estimated cache at 64K.

Reviewer_KAywOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper augments a recurrent model with cached state checkpoints and content-based historical reads. The aim is to recover associations that later recurrent updates would otherwise overwrite.
Critical strengths
  • I like the separation between the unchanged recurrent update and the historical read branch. The improvements extend beyond synthetic recall to LongBench and in-context retrieval, and the tests on multiple recurrent backbones are useful.
Critical weaknesses
  • The experiments do not isolate the proposed anchor-key construction from the benefit of storing more states. The uniform baseline in Table 8 removes both learned weighting and the null route. A comparison using the same checkpoints and null route, but simpler routing keys, would make the architectural contribution clearer.

  • The quality comparisons do not match cache capacity or compute. Also, dense quality and Top-4 training speed are separate operating points, as Appendix G acknowledges. I would like a quality-versus-memory/latency comparison, including autoregressive decoding.

Questions for authors
  • How much of the gain remains with pooled-segment routing keys under the same memory budget? This also seems important for establishing the difference from the closely related Memory Caching approach.
1 comment

Response to W1 — Does learned historical weighting help when the null route is retained?

We agree that the original uniform baseline couples the effects of historical weighting and the null route. We address this through two comparisons. First, on a fixed trained model, we preserve each query/head’s null probability and uniformly redistribute the remaining probability mass across visible historical checkpoints.

Inference-time aggregationMean NIAH accuracy (%) ↑, 4K–16K
Learned routing + null55.7
Uniform history + preserved null probability30.5

The 25.2% reduction shows the trained model’s sensitivity to removing learned preferences among checkpoints while preserving the null probability and total historical probability mass.

Second, to complement this inference-only intervention, we trained a uniform-history variant from scratch with the null route retained. It achieves 33.9% versus 48.7% for MARCH over 4K–32K. This comparison supports learned historical weighting when each variant is trained with its own aggregation rule.

Response to Q1 and W1 — Routing alternatives at matched historical-state capacity

We compare anchor-based routing with position-based routing, pooled-segment keys, and state-derived keys. All variants share the backbone, checkpoint construction, historical-state capacity, read layers, null route, fusion mechanism, and training settings.

Position-based routing directly projects each token representation to a coefficient for each visible prefix-state checkpoint. Pooled-segment keys use mean-pooled token representations; state-derived keys use learned probes of cached state matrices.

MethodMean NIAH accuracy (%) ↑
GDN34.6
MARCH — anchor-derived keys48.7
MARCH — pooled-segment keys39.7
MARCH — state-derived keys44.8
MARCH — uniform history + null33.9
MC-GRM36.7
MARCH — position-based routing43.6

Scores average six NIAH tasks at 4K, 8K, 16K, and 32K. Differences are calculated before rounding. GDN and MC-GRM provide reference baselines.

Pooled-segment keys retain a 5.1-point gain over GDN. Anchor-derived keys add a further 9.0 points at the same historical-state capacity, reaching 48.7%. Position-based routing achieves 43.6%, compared with 48.7% for anchor-derived keys. Anchor-derived keys also outperform state-derived keys by 3.9 points.

The key-construction ablations support the benefit of anchor-derived keys at matched historical-state capacity. MC-GRM scores 36.7%, compared with 48.7% for MARCH.

Response to W2 — Quality–cost comparison and autoregressive decoding

We report accuracy and inference costs together for GDN, dense MARCH, Top-4 MARCH, Transformer (RoPE), and hybrid MARCH with full-attention layers.

For the MARCH components, we retain the original chunk length C=512C=512, state dimensions, checkpoint schedule, and learned null route. Benchmarks use a single GPU, batch size 4, BF16 operators, and 1,024 generated tokens. Timing includes archive construction and updates, routing-key computation, index scanning, and historical reads.

Each context-length cell reports first-token latency (ms) / aggregate decode throughput (tokens/s across the batch) / analytical cache (GiB). Mean NIAH uses the six-task, 4K–32K protocol above.

ConfigurationMean NIAH (%) ↑4K16K32K64K
GDN34.656.1 / 2168 / 0.072213.2 / 2168 / 0.072419.9 / 2168 / 0.072847.7 / 2168 / 0.072
Dense MARCH, chunk 51248.7197.2 / 1548 / 0.600727.3 / 1251 / 2.1821660.2 / 989 / 4.2914288.6 / 720 / 8.511
Top-4 MARCH, chunk 51241.3195.1 / 1127 / 0.600654.6 / 1121 / 2.1821280.5 / 1112 / 4.2912563.8 / 1101 / 8.511
Transformer (RoPE)52.687.0 / 914 / 2.215548.7 / 606 / 8.8591700.6 / 423 / 17.7196035.2 / 260 / 35.438
Hybrid MARCH + full attention77.8159.0 / 1618 / 0.920631.3 / 1216 / 3.5161506.2 / 911 / 6.9774110.7 / 605 / 13.899

Cache values are analytical BF16 storage estimates at the end of prefill. They include online recurrent states, archived states, routing keys, and full-attention KV caches as applicable.

Dense MARCH improves mean NIAH accuracy from 34.6% to 48.7% over GDN, with additional storage and execution costs. Relative to Transformer, it achieves 2.77× decode throughput with 76.0% less analytical cache at 64K. Mean NIAH accuracy over 4K–32K is 48.7% versus 52.6%.

Top-4 trades mean accuracy (41.3% versus 48.7%) for 40.2% lower first-token latency and 1.53× decode throughput at 64K, while retaining the same historical archive. Its decode throughput exceeds dense MARCH at 32K and 64K and is lower at 4K and 16K.

Hybrid MARCH achieves 77.8% mean NIAH accuracy, requiring 13.899 GiB of analytical cache at 64K, compared with 8.511 GiB for dense MARCH and 35.438 GiB for Transformer.

Reviewer_7H5fOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MARCH (Memory-Anchor Routing across Context History), a recurrent architecture that improves long-range retrieval by caching and retrieving historical recurrent states. MARCH saves cumulative recurrent states as state anchors and associates each anchor with a content-conditioned routing key derived from an anchor token. At each token, the model scores all causally available anchors, computes a weighted combination of their historical state readouts, and adds it to the current-state output. A learned null route allows the model to skip historical retrieval when the current state is sufficient. The model is trained end to end with the standard language-modeling objective. The authors report that MARCH outperforms Gated DeltaNet and related recurrent variants on commonsense reasoning, LongBench, in-context retrieval, and NIAH benchmarks, and that it extrapolates beyond the training context length.

Critical strengths
  1. This paper introduces a simple mechanism to address an important bottleneck of recurrent long-context models.
    Linear attention and modern recurrent sequence models compress the full history into a fixed-size recurrent state, enabling constant-memory decoding but limiting long-range memory. MARCH addresses this limitation by periodically saving cumulative recurrent states and letting later tokens retrieve from them through content-based routing. Since historical retrieval is introduced as a residual branch without modifying the underlying recurrence, the method is easy to integrate with different recurrent backbones. The improvements reported on GDN, KDA, and GDN2 further suggest that the approach is not specific to a single architecture.

  2. The gains are strongest on the tasks where recurrent models are expected to struggle most.
    The improvements are particularly substantial on long-context retrieval, which directly targets the limitation motivating the paper. On GDN, the LongBench average improves from 11.9 to 14.9. I found the 32K extrapolation results especially interesting: At 32K, both Transformer baselines and Log-Linear GDN drop to near zero on most NIAH tasks, while MARCH keeps non-zero accuracy on all six.

Critical weaknesses
  1. My main concern is that the paper does not clearly isolate the value of its most distinctive component. Keeping and revisiting historical recurrent states has already been explored in related and concurrent work, including Memory Caching. What seems more specific to MARCH is the use of anchor tokens to produce content-conditioned routing keys. However, the paper does not show whether this choice actually works better than simpler ways of indexing the same cached states. This makes it hard to tell how much of the improvement comes from the anchor-key design itself, as opposed to simply having access to multiple historical states.
  2. Some of the reported improvements are quite small, with several differences across the evaluated baselines being only a few tenths of a point. The paper also does not report variability across runs or confidence intervals. Without such information, it is hard to know whether these small differences are meaningful or just reflect normal evaluation variation.
Questions for authors
  1. Could you add an ablation that isolates the anchor-token keys, e.g., replacing them with keys that are not conditioned on the checkpoint, position-based keys, or keys pooled directly from the cached states, while keeping the same checkpoint bank? If MARCH’s anchor keys clearly outperform these alternatives, it would address my main concern.
  2. How many runs were performed for the main results, and were they averaged over multiple seeds? If possible, could you report variance for the key comparisons, especially those with small differences (e.g., commonsense reasoning in Table 1)?
3 comments

Response to W1/Q1 — Contribution of anchor-derived keys

We agree that the anchor-key design should be evaluated separately from the benefit of retaining historical states. We have added two key-construction ablations and a position-based routing baseline, preserving the checkpoint-bank design and capacity, readout mechanism, and training settings.

Position-based routing directly projects each token representation to a coefficient for each visible prefix-state checkpoint, with output coordinates indexed by checkpoint position. It remains token-dependent but does not construct checkpoint-specific content keys. Segment pooling constructs keys from mean-pooled token representations within each segment. State projection constructs keys directly from cached state matrices using learned probes. We also include uniform aggregation, which averages historical state readouts without learned selection.

VariantMean NIAH accuracy ↑
MARCH: anchor-derived keys48.7
Position-based routing43.6
Uniform aggregation33.9
Segment pooling39.7
State projection44.8

The average covers six NIAH tasks at 4K, 8K, 16K, and 32K contexts. Uniform aggregation provides a simple reference for using historical states without learned selection. Anchor-derived keys achieve 48.7, compared with 43.6 for position-based routing, and outperform segment pooling and state projection by 9.0% and 3.9%, respectively.

Checkpoint-aligned reads refine anchor representations across layers, allowing retrieval descriptors to incorporate information retained in recurrent memory. These results support an additional contribution from anchor-based indexing beyond retaining the same number and size of historical states. Detailed results, variant definitions, and training controls are provided in the revised manuscript.

Response to W2/Q2 — Run counts and variability

We agree that across-seed statistics would help assess the small differences in Table 1. Our main results are based on one full pretraining run per configuration. This follows the reporting practice in Gated DeltaNet and Log-Linear Attention, whose main pretrained-model comparisons also report point estimates without across-seed statistics. We will state the run count explicitly.

We agree that some differences in Table 1 are small and may vary across training seeds. Still, the results show that MARCH remains competitive on short-context benchmarks. The longest evaluation sequence contains 269 tokens, below the 512-token checkpoint interval, so historical retrieval is inactive throughout this evaluation. We therefore interpret Table 1 as evidence that MARCH preserves short-context capability. The larger gains are observed on long-context and recall-intensive tasks, where historical retrieval is active.

As complementary evidence, we conducted 1,000-step pretraining runs from scratch for MARCH and GDN across three seeds (42, 43, and 44), using matched training settings with a 16K context length and the original learning-rate schedule. The training cross-entropy loss, averaged over the final 10 optimization steps, was 2.3944 ± 0.0032 for MARCH and 2.3971 ± 0.0044 for GDN (mean ± sample standard deviation across seeds, in nats/token). Both methods exhibited similar early-stage convergence with small across-seed loss variation.

Thanks to the authors' detailed replies which address some of my concerns. I will maintain my original score.

Thank you for considering our responses and additional experiments. We appreciate your feedback. If you have any further questions or concerns, we would be happy to address them.

Reviewer_Moh6OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper is presenting MARCH and targets the limitations caused by fixed-capacity bottleneck of linear recurrent models. The main idea is to periodically checkpointing cumulative recurrent states at fixed intervals (C=512C=512) and treat them as state anchors. For each boundary, it considers a dedicated shared anchor token and inserts it into the sequence without modifying the state of the model. In layer ℓ\ell, it reads the aligned recurrent checkpoint A(m,ℓ)\mathbf{A}^{(m,\ell)}, and then passes the readout through the residual stream. The authors evaluate a roughly 790M-parameter models trained on 50B tokens across commonsense reasoning, LongBench, RULER NIAH, and in-context retrieval tasks.

Critical strengths
  1. The paper proposes a hardware-aware producer–reader implementation that that reuses the efficient kernels of existing architectures. This can be important for scaling the models.
  2. The paper is easy to follow overall and understandable.
  3. The experiments show partial improvement over the selected baselines.
Critical weaknesses
  1. Novelty: I have not quite understand what is the main novelty of MARCH. The paper is well-written in general, but when it comes to novelty of contributions, the text has left unclear. What is the main difference of memory caching with MARCH? Even, based on the authors wording in both main text and appendix, these two works are quite similar in terms of how they expand the state over time, and the differences seems to be based on using anchor tokens, which can be an arbitrary choice without ablation studies. The authors refer to memory caching as "concurrent" but that paper seems to be on OpenReview from last year ICLR 2026, and also published 7 months before the submission deadline. It would be good if the authors could clarify how they have considered this work as concurrent?
    To my understanding, the concept of memory caching is to cache the states and then define read process to use the cached states. How this is different from MARCH? Is MARCH an instance of memory caching? If not, it would be great if the authors could clarify why MARCH cannot be written as the above? If it is, then it would be great to clarify the main text and also discuss how it is different from its three initial instances (i.e., the use of anchor tokens), and then clarify how they contribute to the improvement by providing both the intuition and ablation studies.
  2. Mixed Results: The results are mixed and MARCH does not consistently outperform the selected baselines. Can the authors comment on this? In particular, it seems that when context is large, the throughput of MARCH is worse than the base model. How the authors justify this trade-off?
    Also, again there is no comparison with similar studies of RAT, and memory caching. Does the use of anchor tokens improve the results? How much of the contribution comes from caching, and how much comes from the novel contributions?
  3. Selected and Missing Baselines: The baseline of the papers are very limited. It would be great if the authors could clarify why they have chosen these baselines, given the fact that there are architectures in either group of (1) more powerful models, (2) simpler models, (3) newer models, and/or even (4) faster models. Which aspects convinced the authors that these are the proper baselines? In addition to the base architecture, there are missing competitors. How MARCH is compared to RAT/RAT+/memory caching/MoM with the same state-size, ... The current experiments are more seem like a preliminary results for the start, but are not convincing that there is a need for MARCH and it can improve the existing methods. In particular, at least there is a need for comparison with RAT and memory caching variants.
Questions for authors

Please see weaknesses

1 comment

Response to Point 1 — Novelty and relationship to Memory Caching

We agree that MARCH’s relationship to Memory Caching (MC) and its specific contribution need clearer explanation. MARCH can be viewed as an instantiation of the general cache-and-read framework. Our contribution concerns how historical checkpoints are indexed for retrieval.

The contextual weighting or selection mechanisms described for GRM, Memory Soup, and SSC use pooled segment descriptors. MARCH constructs routing keys from learned anchor representations that receive checkpoint-aligned state readouts. These reads refine the anchor representations across layers, allowing routing keys in subsequent layers to incorporate information retained in the recurrent checkpoints.

Our oracle analysis in Appendix E further motivates this design. Using the known needle location to select a historical snapshot and combining its readout with the current-state output improves long-range recall. This motivated our focus on identifying relevant snapshots and fusing their readouts with the current-state output. Point 2 tests anchor-derived keys.

By “concurrent,” we meant overlapping research development: we had been exploring this design before MC became publicly available, although our manuscript was submitted later. We acknowledge MC’s earlier public availability and will clarify the wording and the relationship between the two methods in the revised manuscript.

Response to Point 2 — Mixed results, additional baselines, and ablations

As discussed in the paper, MARCH improves long-context performance over the evaluated recurrent baselines, while a gap remains to the Transformer baseline.

(1) Additional baseline comparisons. We have added comparisons with RAT, MoM-GDN, and MC-GRM. RAT constructs compressed key/value vectors through within-chunk recurrence and applies softmax attention across chunks. MoM-GDN routes tokens to multiple independent matrix-valued recurrent memories for selective updates. MARCH maintains a continuous matrix-valued recurrent process and caches its historical states, allowing the current query to read different information from each checkpoint.

The table reports mean NIAH accuracy across six tasks at 4K, 8K, 16K, and 32K contexts. Full results and configurations will also be included in the revised manuscript.

MethodMean NIAH accuracy ↑
GDN34.6
MC-GRM36.6
MARCH with uniform aggregation33.9
MARCH with pooled-segment keys39.7
MARCH with state-derived keys44.8
MARCH with anchor-derived keys48.7
MoM-GDN (Fixed Multi-State)45.3
RAT (chunk-based attention)47.9
Transformer (full attention)52.6
MARCH (hybrid with full attention)77.8

(2) Contribution of anchor-derived keys. To isolate key construction, we keep the checkpoint-bank capacity, archival policy, null route, readout mechanism, and training settings fixed across the key variants. Pooled-segment keys achieve 39.7, compared with 34.6 for GDN. Replacing pooled-segment keys with anchor-derived keys further improves accuracy to 48.7, a gain of 9.0%. Anchor-derived keys also outperform state-derived keys by 3.9%, with improvements at all four evaluated lengths.

Uniform aggregation achieves 33.9, further showing the importance of how historical readouts are weighted. MARCH also outperforms the evaluated MC-GRM configuration by 12.0%. Differences are calculated before rounding.

Training throughput. MARCH adds routing and readout costs to access historical checkpoints, increasing overhead at longer contexts. Practical architectures such as Qwen3.5 also combine GDN with full attention, illustrating the trade-off between recurrent efficiency and richer context access.

We therefore assess retrieval quality under comparable estimated training-time budgets, using measured throughput. MARCH trained on approximately 17B tokens achieves 36.8 average NIAH accuracy over 4K, 8K, and 16K, compared with 31.1 for GDN trained on approximately 34B tokens. This 5.7% gain supports the additional computation for long-context retrieval, even when GDN receives twice as many training tokens.

Response to Point 3 — Baseline selection and missing competitors

Our original baselines were selected to compare different ways of organizing and accessing memory: a single evolving recurrent state in GDN, a temporal hierarchy of states in Log-Linear GDN, and content-addressable historical checkpoints in MARCH. Using the same GDN backbone across these models helps control differences in the underlying architecture. The Transformer provides a strong full-attention reference with direct access to past tokens.

We agree that comparisons with closely related methods are also necessary. We have therefore added MC-GRM, MoM-GDN, and RAT, as reported in Point 2. We will include detailed configurations, parameter counts, training budgets, and cache usage at the evaluated context lengths in the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response from the Authors

General Response from the Authors

We thank the reviewers for their constructive feedback. In response, we clarified MARCH’s relationship to prior work, added controls for its key design choices, and expanded the evaluation of retrieval quality and computational cost.

  • Relationship to prior work and additional baselines (Moh6 W1–W3). We acknowledge the earlier public release of Memory Caching and clarify what MARCH adds to the shared idea of caching recurrent states: checkpoint-conditioned anchor keys and a historical readout with a learned null option. We added comparisons with MC-GRM, MoM-GDN, and RAT. MARCH achieves higher mean NIAH6 than MC-GRM and MoM-GDN over 4K–32K.
  • Anchor keys and learned routing (Moh6 W1/W2; 7H5f W1/Q1; KAyw W1/Q1). At the same historical-state capacity, anchor-derived keys achieve 48.7% mean NIAH6, compared with 39.7% for pooled-segment keys, 44.8% for state-derived keys, and 43.6% for position-based routing. A separately trained uniform-history variant with the null route retained scores 33.9%, showing the benefit of learned weighting among checkpoints in this setting.
  • Historical memory representation (KVYd W2/Q1/Q2). Replacing the query-dependent matrix readout with a pooled vector per checkpoint and head reduces mean NIAH6 from 48.67% to 33.70%. In a separate comparison at matched cache capacity, cumulative checkpoints score 46.43% versus 45.09% for chunk-independent checkpoints, with the largest gain at 32K.
  • Training and inference costs (Moh6 W2; KAyw W2; KVYd W4). At comparable estimated training times, MARCH improves mean NIAH6 over GDN from 31.1% to 36.8% across 4K–16K and improves SWDE and FDA by 18.54 and 15.88 points, respectively, with similar commonsense performance. We also report first-token latency, decode throughput, and analytical BF16 cache size across 4K–64K. Hybrid MARCH achieves 77.8% mean NIAH6 versus 52.6% for the Transformer and delivers 2.33× its decode throughput with 60.8% less estimated cache at 64K.
  • Run counts and presentation (7H5f W2/Q2; KVYd W1/W3). The main results use one full pretraining run per configuration. Three additional 1,000-step runs per model show similar early training loss for MARCH and GDN. We will state the run counts explicitly, bring the cross-backbone results and checkpoint-interval analysis into the main evaluation, and move the detailed positional-encoding ablation to the appendix.

We look forward to further discussion with the reviewers and would be happy to address any additional questions or comments to further improve the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.