- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether compressed memory tokens carry document information that a language model actually uses, or mainly absorb attention as sinks. It compares dropping a memory slot with swapping its keys and values for those computed on another document. Combined with value norms, this audit assigns operational roles to slots. The paper then proposes Assigned-Role Memory (ARM): a zero-valued sink slot with a token-like key, plus a fixed set of memory slots rewritten after each chunk. Controlled training experiments, audits of released checkpoints, PG19 language modeling, synthetic associative recall, and efficiency measurements support several practical advantages of this design.
Critical strengths
-
The distinction between presence and content is useful. A slot can be important to numerical behavior without carrying document-specific information needed for prediction. The contrast between drop and swap in Figure 2 and Table 13 makes this concrete. The most-attended slot has a substantial removal cost but almost no measured swap effect. This is more informative than using attention weights or removal alone.
-
The mechanism is examined through substantial controls. The paper compares retained BOS tokens, learned sink tokens, scalar logits, KV biases, zero-valued variants, and projected keys (Table 2). Additional seeds, grouped-query attention, training trajectories, key geometry, and released AutoCompressor/RMT checkpoints extend the evidence beyond one small model (Appendices C.1–C.14). These controls make the central observation credible, even though the interpretation of every slot verdict needs qualification.
-
The design has a favorable measured quality–memory tradeoff. At 124M parameters, ARM with 16 memory slots reports long-range perplexity 528 versus 529 for accumulating ICM with 128 slots per chunk, while the extra positions read at 4k are 17 versus 384 (Table 3). The reported synthetic recall improvement, 78.6% versus 53.5%, is also substantial. The equal-held-cache comparison against a 1024-token sliding window is particularly helpful (Table 22); the paper does not rely solely on comparing extra memory positions.
-
The appendix addresses many likely evaluation questions. It includes multiple training seeds, paired sequence bootstrap intervals for language-model comparisons, an explicit rare-token metric, longer training contexts, and a separate last-layer-feedback variant for longer recall (Appendices F.1–F.9). These should remain visible in the main presentation.
Critical weaknesses
-
The audit's operational null is interpreted too strongly. Section 3 defines operational slot roles, but Appendix C.2 interprets a small average next-token loss change as evidence of an input-independent slot. That implication does not follow in general: effects can cancel across targets, redundant paths can compensate, and rare uses can be diluted by averaging. Swapping a key can also change softmax weights, so the statement that swap preserves the “mass structure” needs a precise qualification. The paper itself supplies a useful boundary case: slot 2 of the 124M accumulating model is classified as housekeeping with swap +0.0035 and value ratio 0.33, yet its final-layer chunk readout is 60% with cross-sequence cosine 0.43 (Tables 13–14, p. 25). Table 14 probes final-layer hidden states, whereas later attention reads per-layer cached K/V. The readout shows that an operationally housekeeping slot can have an input-dependent final hidden state; it does not by itself establish informative or used cached K/V. The near-constant, most-attended slot is a much cleaner case. Section 3 already defines the sink tax through the evaluated loss, thresholds, and housekeeping verdict. Please retain this operational definition while limiting its interpretation to negligible measured predictive use on the audited population; the input-independence conclusion requires additional assumptions. The existing threshold sensitivity and geometry controls are valuable; they do not establish the stronger logical implication.
-
The memory horizon of the one-pass design needs to be stated. Section 5.2 and Appendix D.1 use fresh learned slot embeddings in each chunk, carry KV states at each layer, and implement all chunks with one forward-pass mask. Under standard pre-attention KV projection and this mask, forwarding a document signal through another chunk consumes another layer. For fixed supplied input tokens and no last-layer feedback, an L-layer output therefore has a depth-bounded cross-chunk dependency, rather than an unrestricted recurrent state. This is a derivation from the described computation graph, not an observed implementation failure. Stable perplexity at 128k establishes stable processing, but does not establish retention of a fact from arbitrarily early in that context. The authors already distinguish ordinary ARM's four-update recall from the additional feedback variant's 1–15-chunk result (Table 32 and p. 40). Please state the corresponding receptive-field bound and explain which variant supports each long-memory claim. A gain in the 9–15-chunk rare-token bucket (Table 25) alone does not isolate causal use of the specific earlier occurrence.
-
Recall uncertainty does not reflect all sampling levels. Table 30 pools queries across sequences and training seeds into Wilson intervals. Four queried values share a sequence and memory state, and seed pooling does not estimate training-run uncertainty. The pooled-query intervals do not account for possible within-sequence dependence or training-run variation; their actual effect on coverage has not been measured here. Appendix F.5 already reports ARM G=4 recall for three seeds (51.6%, 51.6%, 51.2%). Please extend per-seed reporting to the principal comparisons, particularly G=16, and provide paired sequence-level intervals using existing outputs. This issue is unlikely by itself to erase the large 78.6%-versus-53.5% gap. If the interval overlap relative to a common sliding-window baseline is intended to substantiate that ARM matches accumulating ICM, it is not a formal equivalence test between those two models (p. 38). The paper reports overlap rather than a formal equivalence test. Descriptive similarity is supported; a stronger equivalence claim would require a direct comparison and a stated tolerance.
Questions for authors
- In the saved audit results, how often do per-sequence or target-level swap effects cancel? Can the interpretation be limited to negligible average predictive use, while explicitly distinguishing the final hidden states probed in Table 14 from the per-layer cached K/V swapped in Table 13?
- What exact KV/hidden-state recurrence is used in the default ARM implementation? Does it contain any path from the final layer of one chunk to the input of the next? If not, what is its depth-dependent content horizon, and which existing results use the feedback variant?
- Which operations are included in the 39× timing? Do the recorded timings include all model blocks, output projection, and register rewriting during generation, or only the attention computation with the corresponding layer shapes?
- Beyond the G=4 per-seed values already in Appendix F.5, can the principal recall comparisons be summarized by paired sequence resampling and per-seed values? For the long-range perplexity comparison described as matching, what is the direct paired difference and its interval?
4 comments
Response to Reviewer rokC (Part 2/4). We thank the reviewer for W2 and Q2. We state the L-1 chunk horizon and the recurrence of the default ARM, measure the use of a chunk by distance, and name the fed-back variant at every result that uses it.
W2/Q2. The memory horizon of the one-pass design, the exact recurrence of the default ARM, the depth-dependent content horizon, and which results use the fed-back variant.
We state the bound in Section 5.2 and Appendix D in the revised PDF (page 29): the default ARM carries the register set only as per-layer key-value states without final-layer feedback, as Section 5 of the submission states, and a query of an L-layer model therefore depends on raw tokens at most L-1 chunks back, 11 chunks at 124M and 23 at 340M. In the default ARM, a query in chunk c reads the registers of chunk c-1 at the same layer, and the final layer of one chunk does not feed the input of the next. We chose this recurrence because one attention mask then implements the whole sequence in a single forward pass, as in Gisting (Mu et al., 2023), whereas a path from the final layer to the next input makes the chunks sequential during training. Information propagates one chunk per layer in this cache, which gives the bound of L-1 chunks, 11,264 tokens at 124M.
A swap of the register set by chunk distance shows that every design uses the content of a chunk mainly in the next two chunks. Within these two chunks, the default ARM carries three times the content of Accumulating ICM one chunk later, 0.638 compared to 0.215 on the long-range targets, and 1.5 times two chunks later, 0.102 compared to 0.067. The fed-back variant carries the most content of every design at distances 1 to 4 and stays above the noise level up to 8 chunks with the same 5 cache positions. For the swap by distance the reviewer asks for, we replace the register set written after chunk j by the set of another sequence, in one forward pass at 16 chunks on 32 PG19 test sequences (Table 33, page 39). The swap effect of the default ARM vanishes after three chunks, and Accumulating ICM, which keeps the set of every chunk, retains an effect of +0.006 at 9 to 15 chunks with a cache that grows with the number of chunks, whereas the rewritten set keeps 5 positions:
Table R10. Loss change after swapping the set written after chunk j, by distance.
| design | tokens | chunks after the swapped set: 1 | 2 | 3 | 4 | 5 to 8 | 9 to 15 |
|---|---|---|---|---|---|---|---|
| Accumulating ICM (R=4) | all tokens | +0.017 | +0.005 | +0.001 | 0.000 | -0.001 | -0.001 |
| Accumulating ICM (R=4) | long-range targets | +0.215 | +0.067 | +0.039 | +0.024 | +0.010 | +0.006 |
| ARM (G=4) | all tokens | +0.064 | +0.006 | 0.000 | 0.000 | 0.000 | 0.000 |
| ARM (G=4) | long-range targets | +0.638 | +0.102 | +0.010 | 0.000 | 0.000 | 0.000 |
| ARM (G=4), fed-back variant | all tokens | +0.179 | +0.031 | +0.010 | +0.003 | 0.000 | 0.000 |
| ARM (G=4), fed-back variant | long-range targets | +0.938 | +0.202 | +0.085 | +0.030 | +0.011 | -0.001 |
We agree that each result should name the variant behind it, and we have clarified this in the revision. All results of the main text are obtained with the default ARM, and the fed-back variant appears only in the recall beyond four updates, in Table 32 of the submission and in Table R11. To clarify, Table 32 of the submission labels the variant as the last-layer output fed into the next update. The revision defines the fed-back variant in Section 5.2 beside the default recurrence, labels its rows in Table 42 (page 44) and Table 33 (page 39), and states that every result of the main text uses the default recurrence. The fed-back variant adds the final-layer state of the memory slots to the input of the next set and keeps the same 5 cache positions. Continued on 64 chunks of 256 tokens and evaluated on the same 256 sequences, the default ARM recalls 47.5% of the facts with one to four updates between the fact and the query and at most 1.2% of the facts beyond four updates. The fed-back variant recalls 22.6% of the facts with 32 to 63 updates between the fact and the query, with the same 5 cache positions, as Table R11 shows.
Table R11. Recall at 64 chunks of 256 tokens, by update distance.
| design | recall (%) by the number of updates between the fact and the query: 1 to 4 | 5 to 7 | 8 to 15 | 16 to 31 | 32 to 63 |
|---|---|---|---|---|---|
| Accumulating ICM R=4, all register sets | 0.0 | 0.9 | 0.4 | 0.0 | 1.2 |
| ARM (G=4) | 47.5 | 0.9 | 0.4 | 0.4 | 1.2 |
| ARM (G=4), fed-back variant | 49.0 | 50.9 | 50.4 | 42.6 | 22.6 |
Table 42 (page 44) reports Table R11. The 128k perplexity of the abstract is obtained with the default ARM as well.
Response to Reviewer rokC (Part 3/4). We thank the reviewer for Q3. We list what the decoding step behind the 39x throughput includes and report the time of the per-chunk update next to the step time.
Q3. Which operations the 39x timing includes.
The 39x is the decoding throughput of ARM (G=4) divided by the decoding throughput of Accumulating ICM (R=128), both at 128k tokens under a 40 GB cache budget on one A6000, and the timed decoding step includes, in every layer, the QKV projection of the new token, the attention of that token over the cached keys and values of its request, the output projection and the MLP. Under a fixed cache budget, the cache of one request sets how many requests decode concurrently and consequently the throughput (Kwon et al., 2023), and the constant cache of ARM is the strength that the 39x measures. For both designs at 128k tokens, in bf16 with the layer shapes of the 124M model, we try batches of concurrent requests up to the largest batch whose caches fit in the budget. Every request holds its own cache of the positions that the design retains, 1,029 for ARM and 17,280 for Accumulating ICM (R=128). We report the batch with the highest throughput. The throughput of ARM saturates at 512 requests and stays within 1% up to the largest batch that fits, 1,054 requests, whereas Accumulating ICM reaches the budget at 62 requests. We measure the median step time over 20 steps after 5 warm-up steps, and Table 3 reports the median of three measurements, 12.6 ms per step at 512 requests for ARM and 59.7 ms at 62 requests for Accumulating ICM, 40.8 and 1.04 thousand tokens per second. The token embedding, the LM head, sampling and the CPU-side loop are identical for both designs and are excluded from the timing.
The timed step excludes the update of the rewritten set, which runs once per chunk. At 512 requests the update takes 162 ms, and averaged over the 1,024 tokens decoded in a chunk it adds 1.3% to the step time of ARM, which lowers the 39x to 38.8x. ARM updates the rewritten set once per chunk as one forward pass of the G+1 register tokens over the raw tokens of the chunk and the slots of the previous set, under the same four operations and the same kernel. We time the update and the step in the same run and report both in Appendix E.3 in the revised PDF (page 32):
Table R12. Time of the per-chunk update of ARM (G=4), timed with the same four operations and kernel as the step. Share of step is the update time averaged over the 1,024 tokens of a chunk, divided by the step time.
| model size | requests | step time (ms) | update time (ms) | share of step (%) |
|---|---|---|---|---|
| 124M | 1 | 1.7 | 3.2 | 0.2 |
| 124M | 512 | 12.4 | 162 | 1.3 |
| 340M | 1 | 4.3 | 8.1 | 0.2 |
| 340M | 395 | 27.9 | 293 | 1.0 |
In the revised PDF, Appendix E.3 (page 32) lists the four timed operations and reports the time of the update next to the step time.
Response to Reviewer rokC (Part 4/4). We thank the reviewer for W3 and Q4. We report every training run of every design, replace the two-run recall mean by the mean over all runs, and compute paired sequence-level bootstrap intervals for the principal comparisons.
W3/Q4. Recall uncertainty does not reflect all sampling levels, the principal comparisons need per-seed values and paired sequence-level intervals, and the match of long-range perplexity is an overlap rather than an equivalence test.
The principal comparison of Table 3, ARM (G=4) against Accumulating ICM (R=4), holds in every training run: both recall 51 to 53% of the placed facts in each of their three runs. At 16 slots the recall varies across training runs for both families, and the revision therefore replaces the two-run mean of ARM (G=16), 78.6%, by the mean over all 12 training runs, 50.7%, and reports every run in Table 37 in the revised PDF (page 41) and Figure 16 (page 41). Following the suggestion, the revision reports every training run of every design in a per-seed table with the mean over the runs, and Table 3 of the revision carries the mean over every run (Table 4, page 8). Over the runs, ARM (G=16) recalls between 0.0 and 94.4% of the facts, Accumulating ICM (R=16) between 52.0 and 67.7% and Accumulating ICM (R=128) between 53.5 and 99.7%, and Figure 16 (page 41) plots every run against the number of memory slots. Table R13 summarises the runs of each design.
Table R13. Recall of the placed facts over the training runs of each design: 3 runs each at 4 slots, 12 runs of ARM (G=16) and 3 each of Accumulating ICM (R=16) and (R=128). Cache positions count the positions read beyond the window at 4 chunks.
| design | cache positions | recall of the placed facts (%): lowest run | highest run | mean |
|---|---|---|---|---|
| ARM (G=4) | 5 | 51.2 | 53.0 | 51.9 |
| Accumulating ICM (R=4) | 12 | 51.3 | 52.9 | 52.4 |
| ARM (G=16) | 17 | 0.0 | 94.4 | 50.7 |
| Accumulating ICM (R=16) | 48 | 52.0 | 67.7 | 57.6 |
| Accumulating ICM (R=128) | 384 | 53.5 | 99.7 | 83.8 |
We compute paired sequence-level bootstrap intervals over the 192 evaluation sequences, and ARM (G=16) matches Accumulating ICM (R=128) within 0.04 in long-range loss at 4k tokens. For the long-range comparisons, we compute the paired difference of the loss between two designs on the same 192 evaluation sequences, with a 95% interval from 2,000 bootstrap resamples over sequences, the paired-sequence design the reviewer suggests. The paired bootstrap over evaluation units is the standard test for comparing two systems (Koehn, 2004). An interval that excludes zero means that the sign of the paired difference holds in at least 97.5% of the resamples over sequences (Table 39, page 42):
Table R14. Paired differences of the long-range loss, ARM minus the compared design, on the same 192 evaluation sequences, with 95% intervals from 2,000 bootstrap resamples over sequences.
| ARM | compared design | tokens | long-range loss: ARM | compared | difference | 95% interval |
|---|---|---|---|---|---|---|
| ARM (G=16) | Accumulating ICM (R=128) | 4k | 6.269 | 6.264 | +0.005 | [-0.026, +0.036] |
| ARM (G=4) | Accumulating ICM (R=4) | 4k | 6.341 | 6.416 | -0.075 | [-0.108, -0.042] |
| ARM (G=4) | sliding window | 4k | 6.341 | 6.601 | -0.259 | [-0.309, -0.215] |
| ARM (G=16) | Accumulating ICM (R=128) | 16k | 6.391 | 6.761 | -0.370 | [-0.394, -0.346] |
The interval for ARM (G=16) and Accumulating ICM (R=128) at 4k tokens contains zero and lies within 0.04 in loss, which states the match of Table 3 as a paired margin instead of an overlap. The other three intervals exclude zero in favor of ARM: ARM (G=4) lowers the long-range loss by 0.075 compared to Accumulating ICM (R=4) and by 0.259 compared to the sliding window at 4k tokens, and ARM (G=16) lowers it by 0.370 compared to Accumulating ICM (R=128) at 16k tokens, differences of 2 to 15 times the half-width of their intervals. We have added these paired intervals to the revision.
In the revised PDF, Appendix F.7 (page 42) and Table 39 (page 42) replace the overlap statement by the paired margin of 0.04 in loss, about 4% in perplexity, and Table 37 (page 41) and Table 4 (page 8) report every run and the mean over every run.
Response to Reviewer rokC (Part 1/4). We thank the reviewer for W1 and Q1. We limit the reading of the swap test to negligible measured predictive use, separate it from input independence in Appendix C.2, and report the swap effects at the sequence level and at the token level.
We thank Reviewer rokC for the review and for noting that the contrast between drop and swap is more informative than attention weights or removal alone. Below we respond to each point in four parts, beginning with W1 and Q1.
We agree with the reviewer and limit the interpretation of the swap test to negligible measured predictive use on the audited sequences. Section 3.1 of the revision defines a content-free position as one whose content has a negligible effect on the loss of the audited sequences (page 3), Appendix C.2 reads a content effect near zero as content that the model does not use on the evaluated sequences, and the abstract, Section 1 and Section 7 state that, in the accumulating designs, the memory slot that receives the most attention stores no content that the model uses.
We agree that a mean over sequences alone cannot rule out cancellation, and we have therefore computed the swap effects at the sequence level and at the token level, as Q1 asks. Neither level shows cancellation. The content effect of the paper averages over 64 audited sequences. We have computed the swap effect separately for each sequence, the change of its mean next-token loss when the cached key-value states of one slot are replaced at every layer by those computed on another document (Table 8 in the revised PDF, page 18). When we swap the memory slot that receives the most attention, which the audit marks as annexed, this change lies between -0.006 and +0.006 on every one of the 64 sequences, within the 0.005 that a repeated audit with other swap documents produces (Table R9). We have also kept the change of every raw token of the 64 sequences, 196,608 tokens per model (Table R9 and Table 9 (page 18)). When we swap the annexed slot of Accumulating ICM or the sink slot of ARM, 99% of the tokens change by less than 0.1 in loss, and the tokens that change by more than 0.1 split evenly between the two signs, 0.14 to 0.27% up and 0.13 to 0.24% down. When we swap a content slot of the same models, 9.5 to 17.4% of the tokens go up by more than 0.1 and 7.0 to 14.5% go down.
Table R9. Swap effects of one slot per sequence and per token on the 64 audited sequences, on PG19 and on FineWeb under chunk-local masking, each design the mean over its training runs (3, 3, 3 and 2). The largest change is the maximum over the runs, and up and down count the tokens whose loss changes by more than 0.1 in that direction.
We thank the reviewer for separating the two readings and agree that the submission blurred them: Appendix C.2 read a small swap effect as input independence, which only the probe of Table 14 of the submission measures. We have revised Appendix C.2 to keep the two apart. Table 13 of the submission (Table 16 of the revised PDF) swaps the cached key-value states of one slot at every layer, the states that the queries read, and its swap effect is the change in next-token loss on the audited sequences when those states are replaced, which is the measured predictive use of the slot. Table 14 probes the final-layer hidden state of the slot without any query and asks two questions, whether that state identifies its chunk (top-1 accuracy) and whether it varies across sequences (self-cosine). For the annexed slot of Accumulating ICM (R=4) they agree: its swap effect is -0.0000, and its self-cosine of 1.00 shows a final state that is the same for every document. For the first memory slot of ARM (G=4) in the same audit they disagree: its swap effect stays at or below 0.005, yet its self-cosine of 0.60 and top-1 accuracy of 57% show a final state that varies with the input and identifies its chunk. A small swap effect therefore does not imply an input-independent slot, and the revised Appendix C.2 reads the swap effect as predictive use and takes input independence from Table 14 alone (page 25).