acceptodds
Under review as a conference paper at ICLR 2027

CHARTA: A Recurrent Language Model with Routed Writes and Addressable Memory

Abstract

Fixed-state recurrent models provide constant-memory processing, but maintaining structured, persistent task-relevant state under memory load is not generally an explicit objective. We introduce CHARTA, an attention-free language model that structures delta-rule recurrent memory through routed and addressable writes. CHARTA retains the delta update while adding a block-level organiser that routes write strength across blocks, an event router that conditions write content and strength, and contextual and token-stable addressing paths. It can be trained with auxiliary supervision of task-relevant state while its recurrent matrices, which we call boards, remain accessible through fitted or learned readouts. We evaluate CHARTA along three complementary axes: persistence under load, state accessibility and causal relevance, and data-efficient language modelling. After training with at most four bindings, CHARTA reaches 96.7% accuracy with sixteen, versus 82.3% for DeltaNet, and retains once-stated facts across unseen window boundaries. A linear decoder recovers held-out item locations from boards at 49.0%, versus 26.1% for DeltaNet; board transplantation causally changes predictions. On BabyLM strict-small, CHARTA achieves a higher aggregate score than the top entry (55.9 vs. 53.6). Together, our results show that fixed-size recurrent state can be structured to retain, expose, and reuse task-relevant information without attention over the preceding context.

Primary area:
foundation or frontier models, including LLMs
Keywords:
recurrent language models, linear attention, delta rule, fixed-size memory, memory addressing, interpretability, linear probing, activation transplantation, data-efficient language modelling, entity tracking, BabyLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_6Y2uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CHARTA is an attention-free language model whose recurrent state is a set of per-head delta-rule matrices ("boards") that persist across windows. Three additions sit on top of the delta update: a block-level organiser that softmax-routes each token's write strength across the six blocks from the raw embedding; an event router with eight learned channels that conditions write content and strength; and two address paths, a token-stable identity path and a contextual path. Training adds an auxiliary linear grader on the final hidden vector that supervises all currently correct story facts. Evaluation proceeds along three axes: persistence under concurrent load (person-code binding trained on at most four pairs and tested to sixteen, plus long-gap recall to 768 tokens), state accessibility (a frozen linear decoder recovers held-out item locations; donor transplantation), and data-efficient language modelling (BabyLM strict-small, MAD). Headline results are 96.7% versus 82.3% for DeltaNet at sixteen pairs, 49.0% versus 26.1% held-out-item decoding, and a BabyLM aggregate of 55.85 versus 53.59 for the top archived entry.

Critical strengths

S1. Internal state is tested at three levels, not just at the output. A frozen linear decoder recovers the location of item words never seen in decoder training at 49.04%, against 26.08% for DeltaNet, 21.83% for KDA and 20.56% for Gated DeltaNet (chance 20%), while transplanting all boards flips the answer to the donor's item in 99.8% of 1,000 stories, with the contextual heads carrying the effect (99.9%) and the object-file heads not (0.4%). The controls are the part that matters: a donor holding a deliberately wrong item moves the answer (99.8%), an unchanged box keeps it (99.7%), and a different-box donor leaves the target unchanged (0.0%). This is what makes the memory claim testable rather than illustrative, and it is consequential because the paper's contribution is about the state itself rather than perplexity.

S2. The high-load gain survives a write-matched control, and the paper says where it does not. With two writes per token and near-identical parameters (2.41M vs 2.39M), CHARTA reaches 96.7/94.2% at sixteen pairs in-view/carried against 90.0/85.8% for DeltaProduct; with a single write it is 90.7% against 82.3% for DeltaNet in-view. This pre-empts the most obvious rejection ground (that the result is only "DeltaNet with a second write"), and the paper discloses that single-write carried recall does not improve (79.0 vs 82.0). Consequential because the remaining disagreement is then about the size and interpretation of an increment that demonstrably exists.

Critical weaknesses

W1. The default configuration cannot do the thing the paper is about, and the failure is attributed to a mechanism the experiments cannot identify. Selecting a moved fact succeeds on 0.5% of relocation stories for the full model, against 81% once the object-file heads are removed and 42% for DeltaNet. In-place replacement at 99.7% is a different operation (overwriting an existing address) and does not speak to revision. The paper reads the gap as "a read-arbitration problem rather than a failed write" because a linear decoder still recovers the updated fact, but a decoder fitted on model-native q/k/v features can read information that the native read path does not address, redistribute it across blocks, or exploit residual correlations; recovering a value therefore does not localise the fault to arbitration, and the long list of attempted repairs in Appendix O that produced no improvement is consistent with either story. Without a delay/repetition curve, "slower relocation" is also not distinguishable from a configuration that never relocates. This is the decisive weakness: persistence and accessibility are the paper's selling points, and the default model trades the first for the second while the abstract presents both.

W2. Several claims that carry the "new memory architecture" framing have no matched evidence behind them. (a) "Attention decodes faster only up to 4k tokens, and CHARTA is faster at 16k" is stated in Section 6 but there is no timing table, hardware, batch size, sequence length, kernel, or throughput number anywhere in the paper; structural complexity is not a measurement. (b) The bounded-content argument ("later writes cannot dominate earlier ones through scale alone") holds only for the tanh base term, since the event correction is added afterwards and its norm is learned. (c) There is no equal-total-state-budget comparison against routed-bank or slot-style memory (MoM, Recurrent Entity Networks), which is the nearest alternative way to get addressable persistent state, and the auxiliary objective is not factorised from the architecture: a matched DeltaNet trained on the same mixture is a good start but does not isolate grader-on/off at fixed tokens. (d) The component ranking and the scarce-data advantage rest on single seeds in places, and the BabyLM headline changes data mixture, objective, and architecture simultaneously, so it supports "this entry scores higher" rather than "this architecture is more data-efficient".

Questions for authors
  1. On the same relocation stories that fail, what does the frozen decoder return for the moved box, and can you restore selection by changing only the read path (e.g. re-fitting or ablating the object-file read, or a targeted intervention on the contextual heads)? Do you have a repetition/elapsed-token curve showing that 0.5% recovers with more mentions? If neither is available, will you narrow the claim from "read-arbitration failure" to "recoverable by a separate reader but not selected by the native path"?
  2. Is there a single configuration that both binds a once-stated fact and revises it, for example a learned gate that disables the token-stable address after a move is detected? At what revision frequency does the default configuration stop being the better choice, quantitatively?
  3. Can you run one equal-budget control against a routed-bank or slot-memory baseline (same total state size, parameters, and write compute) and report load, relocation, and language-model quality together, plus grader on/off at fixed token budget?
  4. Please either provide the 16k decoding protocol and measurements or remove the claim; reconcile Appendix O ("stack twice") with Table 10 ("three times"), both 100%->32%; and add a table mapping each reported result to its configuration ID, seed, and evaluation set.
Reviewer_MnrbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CHARTA, an attention-free recurrent language model that maintains a fixed-size delta-rule memory. It proposes structuring memory writes through a block-level organiser, an event router, and object-file heads (stable addressing). The model is evaluated on synthetic state-tracking tasks (persistence, deduction, binding) and BabyLM. The authors claim that CHARTA improves memory load capacity compared to DeltaNet, and that its memory matrices ("boards") are highly accessible (via linear decoding) and causally relevant (via board transplantation).

Critical strengths
  1. Novel Block-Level Routing Mechanism: The block-level organiser that distributes write strength across different memory blocks is an interesting architectural innovation. It provides a new perspective on how to handle memory load conflicts in fixed-state models.
  2. Causal Interpretability: The board transplantation experiment is a compelling way to demonstrate that the recurrent state causally controls the model's predictions, going beyond mere correlation or simple readout accuracy.
Critical weaknesses
  1. Potential Unfair Comparisons and Missing Ablations: I noticed in Section 5.1 that CHARTA applies two writes per token, whereas DeltaNet uses only one. According to Appendix A, single-write CHARTA achieves 79.0% in the carried state scenario, which is slightly lower than DeltaNet's 82.0%. This makes it difficult to ascertain if the reported superiority stems from the architecture itself or the additional compute. Furthermore, the lack of an ablation on the "Grader" auxiliary loss leaves it unclear whether the strong performance (e.g., 98.2% position accuracy in Table 1) is primarily driven by the architectural design or simply the auxiliary teaching signal.
  2. Limited Baseline Comparisons in Main Text: The evaluation heavily relies on synthetic tasks that are closely tailored to the architecture. In the BabyLM evaluation (Table 3), the results for matched baselines (such as DeltaNet and Transformer) are missing, which makes the overall comparison incomplete. Additionally, the specific model corresponding to the "Archived entry" is not clearly identified.
  3. Clarity, Presentation, and Formulation Issues: I found the presentation to require substantial effort. Specifically, Figure 1 has overlapping text, arrows, and graphics that hinder readability. Key experimental data (such as the component ablations in Section 6) are scattered throughout the text and would be much clearer if consolidated into a table. Finally, the auxiliary loss (Grader) lacks a formal mathematical definition, making it difficult to understand how it is computed and integrated.
Questions for authors
  1. Could the authors clarify the performance of CHARTA if the number of writes is strictly matched (e.g., single-write vs. single-write, or dual-write vs. DeltaProduct)? Including these compute-matched results in the main text would significantly strengthen the paper's claims.

  2. Since the Grader explicitly supervises the full task state, could the authors provide an ablation of CHARTA without the Grader loss? This would help isolate the pure architectural contribution.

  3. How does the Grader supervision extend to regular, ambiguous natural language corpora? Furthermore, as prior context compression literature suggests (e.g., Liu et al., 2026, "Autoencoding-free context compression for LLMs via contextual semantic anchors"), highly compressed or densely supervised representations might sometimes be harder to utilize in downstream tasks. I would be interested in the authors' thoughts on this.

  4. Is there a specific reason for choosing 8 event channels? A sensitivity analysis would be helpful. Also, the paper mentions CHARTA is faster at 16k tokens; could the authors provide more insights on its performance and latency scaling at 128k tokens, and whether it can be combined with Full Attention?

  5. Could the authors clarify what the "Archived entry" in Table 3 refers to, and provide the missing baseline results in the table?

  6. It would be beneficial to discuss how CHARTA relates to recent attention-based fixed-state or memory-augmented models, such as Recurrent Memory Transformer (Bulatov et al., 2022), HMT (He et al., NAACL 2025), and CoMeT (Zhao et al., ACL 2026).

1 comment

Thank you for the additional experiments and revisions. My specific technical questions in the initial review—regarding matched writes, the Grader ablation, BabyLM baselines, presentation of figures and equations, event channel and latency analyses, and related work—have been substantively addressed.

My main remaining concern lies in the scope of the paper's contribution. Based on the current evidence, CHARTA’s advantages are primarily concentrated on small-scale, controlled synthetic memory tasks such as person–code binding. On BabyLM, matched DeltaNet performs essentially on par with CHARTA (53.91 vs. 53.95); the strongest teaching results rely on symbolic Grader supervision that is difficult to obtain from natural text; and under unseen wording, relocation performance remains noticeably lower than DeltaNet (24.2 vs. 53.2). Therefore, even if the contribution is framed strictly as a diagnostic study of fixed-state memory mechanisms, I remain concerned that the insights and broader applicability of such mechanistic diagnostics to general-purpose language model architectures are relatively limited.

I encourage the authors to further narrow the claims in the Abstract and Introduction and to clearly report the trade-offs noted above. While the technical issues raised in my initial review are largely resolved under a strictly mechanistic-diagnostic framing, this limited scope itself remains the primary reason for maintaining my rating. Accordingly, I maintain my score of 2 (if intermediate scores were available, my assessment would be around 2.5).

Reviewer_VcK4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper tries to improve DeltaNet's associative memory system.

Critical strengths

S1- The proposed method appears to be new.

S2- The experiments also reveal some interesting observations

Critical weaknesses

W1- The novelty is somewhat limited. Learned gating, memory routing, and entity-based addressing have each been explored in prior work. The paper integrates these ideas within DeltaNet. It is unclear what fundamentally new memory-learning principle the proposed architecture introduces. The paper should distinguish an effective engineering combination from a genuinely novel memory mechanism.

W2- Your wrote "The full model selects the moved item on 0.5% of stories; removing the object-file heads raises this to 81%, while DeltaNet reaches 42%." Does this imply that the proposed architecture performs poorly when facts change. This is a major concern.

W3- It is unclear whether the main improvement is primarily attributable to additional memory writes. The paper's headline comparison reports 96.7% for CHARTA versus 82.3% for DeltaNet on sixteen-pair binding. However, Appendix A shows that CHARTA performs two delta writes per token, whereas DeltaNet performs one.

W4- The strongest results appear to come from relatively simple synthetic tasks involving a small number of entities and associations. The BabyLM evaluation is useful, but its training mixture contains synthetic entity-tracking examples. The authors acknowledge that DeltaNet trained with abundant entity data achieves similar entity-tracking performance. The paper also reports that a matched Transformer achieves lower perplexity within the context window. It seems that the proposed method is mainly effective when the entity training data is scarce, which is also acknowledged in the paper. The experiments therefore do not yet convincingly demonstrate that CHARTA is a better general-purpose language model.

Questions for authors

See above.

Reviewer_7YsaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present CHARTA, which is an attention-free recurrent language model built on the standard DeltaNet recurrence and is augmented with how writes are formed, addressed, and scaled across layers. In particular, CHARTA: (1) to scale write strength βt(ℓ)\beta_t^{(\ell)} across blocks, adds a block-level "organiser" that computes a softmax over raw token embeddings; (2) to assign static token-identity addresses, it adds an event router that mixes 8 learned bias vectors into vt(ℓ)v_t^{(\ell)} and βt(ℓ)\beta_t^{(\ell)}; (3) to bypass layer context and depthwise convolution, it adds "object-file heads" that compute kt,qtk_t, q_t directly from raw token embeddings and (4) a training-only linear "state grader" head that supervises the final hidden state with ground-truth entity states. The model is also evaluated on very small scales of <14M parameters on synthetic box/person-code stories, MAD, and BabyLM strict-small.

Critical strengths
  • Moving beyond output-only accuracy to test whether frozen recurrent matrices linearly encode unseen item locations and whether transplanting donor boards into recipient runs causally flips model predictions is an interesting mechanistic evaluation setup.
  • Demonstrating that CHARTA generalizes from 4 training bindings to 16 evaluation bindings and improves MAD fuzzy recall shows that bounding content with tanh⁡\tanh and coordinating write gates across layers reduces associative interference at tiny scales.
Critical weaknesses

In the paper there are severe experimental flaws and unfair comparisons:

  • In the Abstract and Section 5.4, the authors claim CHARTA beats the top BabyLM strict-small entry (55.9 vs. 53.6 average) primarily by scoring 60.76 vs. 27.95 on Entity Tracking. However, CHARTA was trained on a custom corpus mixture where 10% of the data (1.0M words, sampled at a 0.3 fraction) consists of synthetic entity-tracking stories generated by the authors' own scripts! In Section 5.4 and Appendix L, the authors bury the admission that when matched DeltaNet is trained on the exact same synthetic mixture, it achieves 60.21 on Entity Tracking (statistically indistinguishable from CHARTA's 59.97±0.8459.97 \pm 0.84). Furthermore, in Appendix B, a pure-text CHARTA control without synthetic examples scores chance (3–7%) on the recall tasks. Accordingly, the claim for BabyLM improvement is an artifact of training on synthetic test-distribution templates, not the CHARTA architecture.
  • In Table 4, CHARTA claims a MAD average of 73.2 vs. 71.8 for DeltaNet, which is driven entirely by Fuzzy Recall (43.7 vs. 35.7). Yet Appendix Q reveals that CHARTA used expanded 128-dimensional keys (dk=128d_k=128) instead of the standard dk=64d_k=64. When CHARTA is run with the fair setup defaults (dk=32d_k=32, RMSNorm, no key expansion), its Fuzzy Recall drops to 23.9 and its MAD average drops to 69.8 (below DeltaNet's 71.8).
  • All experiments are run at tiny scales (2M parameters on TinyStories, about 14M on BabyLM, 256-token windows). Moreover, the "state grader" requires ground-truth symbolic state annotations at the exact end of every state-changing sentence, which is impossible to obtain in real-world unstructured pretraining corpora.
Questions for authors

Please see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.