- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CHARTA is an attention-free language model whose recurrent state is a set of per-head delta-rule matrices ("boards") that persist across windows. Three additions sit on top of the delta update: a block-level organiser that softmax-routes each token's write strength across the six blocks from the raw embedding; an event router with eight learned channels that conditions write content and strength; and two address paths, a token-stable identity path and a contextual path. Training adds an auxiliary linear grader on the final hidden vector that supervises all currently correct story facts. Evaluation proceeds along three axes: persistence under concurrent load (person-code binding trained on at most four pairs and tested to sixteen, plus long-gap recall to 768 tokens), state accessibility (a frozen linear decoder recovers held-out item locations; donor transplantation), and data-efficient language modelling (BabyLM strict-small, MAD). Headline results are 96.7% versus 82.3% for DeltaNet at sixteen pairs, 49.0% versus 26.1% held-out-item decoding, and a BabyLM aggregate of 55.85 versus 53.59 for the top archived entry.
Critical strengths
S1. Internal state is tested at three levels, not just at the output. A frozen linear decoder recovers the location of item words never seen in decoder training at 49.04%, against 26.08% for DeltaNet, 21.83% for KDA and 20.56% for Gated DeltaNet (chance 20%), while transplanting all boards flips the answer to the donor's item in 99.8% of 1,000 stories, with the contextual heads carrying the effect (99.9%) and the object-file heads not (0.4%). The controls are the part that matters: a donor holding a deliberately wrong item moves the answer (99.8%), an unchanged box keeps it (99.7%), and a different-box donor leaves the target unchanged (0.0%). This is what makes the memory claim testable rather than illustrative, and it is consequential because the paper's contribution is about the state itself rather than perplexity.
S2. The high-load gain survives a write-matched control, and the paper says where it does not. With two writes per token and near-identical parameters (2.41M vs 2.39M), CHARTA reaches 96.7/94.2% at sixteen pairs in-view/carried against 90.0/85.8% for DeltaProduct; with a single write it is 90.7% against 82.3% for DeltaNet in-view. This pre-empts the most obvious rejection ground (that the result is only "DeltaNet with a second write"), and the paper discloses that single-write carried recall does not improve (79.0 vs 82.0). Consequential because the remaining disagreement is then about the size and interpretation of an increment that demonstrably exists.
Critical weaknesses
W1. The default configuration cannot do the thing the paper is about, and the failure is attributed to a mechanism the experiments cannot identify. Selecting a moved fact succeeds on 0.5% of relocation stories for the full model, against 81% once the object-file heads are removed and 42% for DeltaNet. In-place replacement at 99.7% is a different operation (overwriting an existing address) and does not speak to revision. The paper reads the gap as "a read-arbitration problem rather than a failed write" because a linear decoder still recovers the updated fact, but a decoder fitted on model-native q/k/v features can read information that the native read path does not address, redistribute it across blocks, or exploit residual correlations; recovering a value therefore does not localise the fault to arbitration, and the long list of attempted repairs in Appendix O that produced no improvement is consistent with either story. Without a delay/repetition curve, "slower relocation" is also not distinguishable from a configuration that never relocates. This is the decisive weakness: persistence and accessibility are the paper's selling points, and the default model trades the first for the second while the abstract presents both.
W2. Several claims that carry the "new memory architecture" framing have no matched evidence behind them. (a) "Attention decodes faster only up to 4k tokens, and CHARTA is faster at 16k" is stated in Section 6 but there is no timing table, hardware, batch size, sequence length, kernel, or throughput number anywhere in the paper; structural complexity is not a measurement. (b) The bounded-content argument ("later writes cannot dominate earlier ones through scale alone") holds only for the tanh base term, since the event correction is added afterwards and its norm is learned. (c) There is no equal-total-state-budget comparison against routed-bank or slot-style memory (MoM, Recurrent Entity Networks), which is the nearest alternative way to get addressable persistent state, and the auxiliary objective is not factorised from the architecture: a matched DeltaNet trained on the same mixture is a good start but does not isolate grader-on/off at fixed tokens. (d) The component ranking and the scarce-data advantage rest on single seeds in places, and the BabyLM headline changes data mixture, objective, and architecture simultaneously, so it supports "this entry scores higher" rather than "this architecture is more data-efficient".
Questions for authors
- On the same relocation stories that fail, what does the frozen decoder return for the moved box, and can you restore selection by changing only the read path (e.g. re-fitting or ablating the object-file read, or a targeted intervention on the contextual heads)? Do you have a repetition/elapsed-token curve showing that 0.5% recovers with more mentions? If neither is available, will you narrow the claim from "read-arbitration failure" to "recoverable by a separate reader but not selected by the native path"?
- Is there a single configuration that both binds a once-stated fact and revises it, for example a learned gate that disables the token-stable address after a move is detected? At what revision frequency does the default configuration stop being the better choice, quantitatively?
- Can you run one equal-budget control against a routed-bank or slot-memory baseline (same total state size, parameters, and write compute) and report load, relocation, and language-model quality together, plus grader on/off at fixed token budget?
- Please either provide the 16k decoding protocol and measurements or remove the claim; reconcile Appendix O ("stack twice") with Table 10 ("three times"), both 100%->32%; and add a table mapping each reported result to its configuration ID, seed, and evaluation set.
Thank you for the additional experiments and revisions. My specific technical questions in the initial review—regarding matched writes, the Grader ablation, BabyLM baselines, presentation of figures and equations, event channel and latency analyses, and related work—have been substantively addressed.
My main remaining concern lies in the scope of the paper's contribution. Based on the current evidence, CHARTA’s advantages are primarily concentrated on small-scale, controlled synthetic memory tasks such as person–code binding. On BabyLM, matched DeltaNet performs essentially on par with CHARTA (53.91 vs. 53.95); the strongest teaching results rely on symbolic Grader supervision that is difficult to obtain from natural text; and under unseen wording, relocation performance remains noticeably lower than DeltaNet (24.2 vs. 53.2). Therefore, even if the contribution is framed strictly as a diagnostic study of fixed-state memory mechanisms, I remain concerned that the insights and broader applicability of such mechanistic diagnostics to general-purpose language model architectures are relatively limited.
I encourage the authors to further narrow the claims in the Abstract and Introduction and to clearly report the trade-offs noted above. While the technical issues raised in my initial review are largely resolved under a strictly mechanistic-diagnostic framing, this limited scope itself remains the primary reason for maintaining my rating. Accordingly, I maintain my score of 2 (if intermediate scores were available, my assessment would be around 2.5).