acceptodds
Under review as a conference paper at ICLR 2027

The Sink Tax: Auditing Memory Tokens and Assigning Their Roles in Long-Context Language Models

Abstract

In-context memory (ICM) systems reduce the key-value cache of long contexts by splitting the context into chunks, compressing each chunk into a few memory tokens, and evicting the raw tokens. However, whether the model uses the information that each memory token stores has not been examined. In this paper, we introduce a slot audit that tests whether a memory token stores the information of its chunk by replacing its key-value states with those of another document. We apply the audit to designs that evict the raw tokens, from models trained under the mask of gist and beacon tokens to the official checkpoints of AutoCompressor and RMT. We find that the attention sink shifts to the memory tokens, and across these existing designs the memory token that receives the most attention consistently stores no information of its chunk that the model uses. Yet, removing this token still raises the loss under the mask that hides each chunk from later queries. Therefore, the removal test, which existing work uses to validate memory tokens, counts the sink as memory. We term the fraction of memory tokens that the sink takes the sink tax. Based on these findings, we introduce Assigned-Role Memory (ARM), which assigns one role to every cache position, whereas in prior designs the roles emerge during training. ARM assigns the sink of the memory slots to one sink slot with a value fixed at zero and a token-like key. This sink slot removes the sink from every memory slot at 124M, 340M, and 0.8B parameters. Once the roles are fixed, ARM keeps one set of memory slots for the entire context and updates the memory slots after every chunk, whereas prior designs accumulate one set per chunk. The cache size therefore remains constant as the context length grows. In long-context language modeling on PG19, ARM matches the long-range perplexity of per-chunk accumulation with 17 cache positions instead of 384 and recalls 79% of the facts placed in earlier chunks compared to 54%. With 17 cache positions, ARM lowers the long-range perplexity of a sliding window by 29%. ARM maintains its perplexity at 128k tokens, 32× the training length, while per-chunk accumulation degrades beyond the training length. At 128k tokens, ARM runs at the decoding throughput of a sliding window with a lower perplexity and at 39× that of per-chunk accumulation at the ratio of prior work.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long-context language models, KV cache compression, in-context memory, attention sink, interpretability, efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rokCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether compressed memory tokens carry document information that a language model actually uses, or mainly absorb attention as sinks. It compares dropping a memory slot with swapping its keys and values for those computed on another document. Combined with value norms, this audit assigns operational roles to slots. The paper then proposes Assigned-Role Memory (ARM): a zero-valued sink slot with a token-like key, plus a fixed set of memory slots rewritten after each chunk. Controlled training experiments, audits of released checkpoints, PG19 language modeling, synthetic associative recall, and efficiency measurements support several practical advantages of this design.

Critical strengths
  1. The distinction between presence and content is useful. A slot can be important to numerical behavior without carrying document-specific information needed for prediction. The contrast between drop and swap in Figure 2 and Table 13 makes this concrete. The most-attended slot has a substantial removal cost but almost no measured swap effect. This is more informative than using attention weights or removal alone.

  2. The mechanism is examined through substantial controls. The paper compares retained BOS tokens, learned sink tokens, scalar logits, KV biases, zero-valued variants, and projected keys (Table 2). Additional seeds, grouped-query attention, training trajectories, key geometry, and released AutoCompressor/RMT checkpoints extend the evidence beyond one small model (Appendices C.1–C.14). These controls make the central observation credible, even though the interpretation of every slot verdict needs qualification.

  3. The design has a favorable measured quality–memory tradeoff. At 124M parameters, ARM with 16 memory slots reports long-range perplexity 528 versus 529 for accumulating ICM with 128 slots per chunk, while the extra positions read at 4k are 17 versus 384 (Table 3). The reported synthetic recall improvement, 78.6% versus 53.5%, is also substantial. The equal-held-cache comparison against a 1024-token sliding window is particularly helpful (Table 22); the paper does not rely solely on comparing extra memory positions.

  4. The appendix addresses many likely evaluation questions. It includes multiple training seeds, paired sequence bootstrap intervals for language-model comparisons, an explicit rare-token metric, longer training contexts, and a separate last-layer-feedback variant for longer recall (Appendices F.1–F.9). These should remain visible in the main presentation.

Critical weaknesses
  1. The audit's operational null is interpreted too strongly. Section 3 defines operational slot roles, but Appendix C.2 interprets a small average next-token loss change as evidence of an input-independent slot. That implication does not follow in general: effects can cancel across targets, redundant paths can compensate, and rare uses can be diluted by averaging. Swapping a key can also change softmax weights, so the statement that swap preserves the “mass structure” needs a precise qualification. The paper itself supplies a useful boundary case: slot 2 of the 124M accumulating model is classified as housekeeping with swap +0.0035 and value ratio 0.33, yet its final-layer chunk readout is 60% with cross-sequence cosine 0.43 (Tables 13–14, p. 25). Table 14 probes final-layer hidden states, whereas later attention reads per-layer cached K/V. The readout shows that an operationally housekeeping slot can have an input-dependent final hidden state; it does not by itself establish informative or used cached K/V. The near-constant, most-attended slot is a much cleaner case. Section 3 already defines the sink tax through the evaluated loss, thresholds, and housekeeping verdict. Please retain this operational definition while limiting its interpretation to negligible measured predictive use on the audited population; the input-independence conclusion requires additional assumptions. The existing threshold sensitivity and geometry controls are valuable; they do not establish the stronger logical implication.

  2. The memory horizon of the one-pass design needs to be stated. Section 5.2 and Appendix D.1 use fresh learned slot embeddings in each chunk, carry KV states at each layer, and implement all chunks with one forward-pass mask. Under standard pre-attention KV projection and this mask, forwarding a document signal through another chunk consumes another layer. For fixed supplied input tokens and no last-layer feedback, an L-layer output therefore has a depth-bounded cross-chunk dependency, rather than an unrestricted recurrent state. This is a derivation from the described computation graph, not an observed implementation failure. Stable perplexity at 128k establishes stable processing, but does not establish retention of a fact from arbitrarily early in that context. The authors already distinguish ordinary ARM's four-update recall from the additional feedback variant's 1–15-chunk result (Table 32 and p. 40). Please state the corresponding receptive-field bound and explain which variant supports each long-memory claim. A gain in the 9–15-chunk rare-token bucket (Table 25) alone does not isolate causal use of the specific earlier occurrence.

  3. Recall uncertainty does not reflect all sampling levels. Table 30 pools queries across sequences and training seeds into Wilson intervals. Four queried values share a sequence and memory state, and seed pooling does not estimate training-run uncertainty. The pooled-query intervals do not account for possible within-sequence dependence or training-run variation; their actual effect on coverage has not been measured here. Appendix F.5 already reports ARM G=4 recall for three seeds (51.6%, 51.6%, 51.2%). Please extend per-seed reporting to the principal comparisons, particularly G=16, and provide paired sequence-level intervals using existing outputs. This issue is unlikely by itself to erase the large 78.6%-versus-53.5% gap. If the interval overlap relative to a common sliding-window baseline is intended to substantiate that ARM matches accumulating ICM, it is not a formal equivalence test between those two models (p. 38). The paper reports overlap rather than a formal equivalence test. Descriptive similarity is supported; a stronger equivalence claim would require a direct comparison and a stated tolerance.

Questions for authors
  1. In the saved audit results, how often do per-sequence or target-level swap effects cancel? Can the interpretation be limited to negligible average predictive use, while explicitly distinguishing the final hidden states probed in Table 14 from the per-layer cached K/V swapped in Table 13?
  2. What exact KV/hidden-state recurrence is used in the default ARM implementation? Does it contain any path from the final layer of one chunk to the input of the next? If not, what is its depth-dependent content horizon, and which existing results use the feedback variant?
  3. Which operations are included in the 39× timing? Do the recorded timings include all model blocks, output projection, and register rewriting during generation, or only the attention computation with the corresponding layer shapes?
  4. Beyond the G=4 per-seed values already in Appendix F.5, can the principal recall comparisons be summarized by paired sequence resampling and per-seed values? For the long-range perplexity comparison described as matching, what is the direct paired difference and its interval?
4 comments
AuthorsOpenReview ↗

Response to Reviewer rokC (Part 1/4). We thank the reviewer for W1 and Q1. We limit the reading of the swap test to negligible measured predictive use, separate it from input independence in Appendix C.2, and report the swap effects at the sequence level and at the token level.

We thank Reviewer rokC for the review and for noting that the contrast between drop and swap is more informative than attention weights or removal alone. Below we respond to each point in four parts, beginning with W1 and Q1.

W1/Q1. A small average swap effect does not imply an input-independent slot, and the interpretation should be limited to negligible measured predictive use on the audited population.

We agree with the reviewer and limit the interpretation of the swap test to negligible measured predictive use on the audited sequences. Section 3.1 of the revision defines a content-free position as one whose content has a negligible effect on the loss of the audited sequences (page 3), Appendix C.2 reads a content effect near zero as content that the model does not use on the evaluated sequences, and the abstract, Section 1 and Section 7 state that, in the accumulating designs, the memory slot that receives the most attention stores no content that the model uses.

We agree that a mean over sequences alone cannot rule out cancellation, and we have therefore computed the swap effects at the sequence level and at the token level, as Q1 asks. Neither level shows cancellation. The content effect of the paper averages over 64 audited sequences. We have computed the swap effect separately for each sequence, the change of its mean next-token loss when the cached key-value states of one slot are replaced at every layer by those computed on another document (Table 8 in the revised PDF, page 18). When we swap the memory slot that receives the most attention, which the audit marks as annexed, this change lies between -0.006 and +0.006 on every one of the 64 sequences, within the 0.005 that a repeated audit with other swap documents produces (Table R9). We have also kept the change of every raw token of the 64 sequences, 196,608 tokens per model (Table R9 and Table 9 (page 18)). When we swap the annexed slot of Accumulating ICM or the sink slot of ARM, 99% of the tokens change by less than 0.1 in loss, and the tokens that change by more than 0.1 split evenly between the two signs, 0.14 to 0.27% up and 0.13 to 0.24% down. When we swap a content slot of the same models, 9.5 to 17.4% of the tokens go up by more than 0.1 and 7.0 to 14.5% go down.

Table R9. Swap effects of one slot per sequence and per token on the 64 audited sequences, on PG19 and on FineWeb under chunk-local masking, each design the mean over its training runs (3, 3, 3 and 2). The largest change is the maximum over the runs, and up and down count the tokens whose loss changes by more than 0.1 in that direction.

modelswapped slotattention (%)largest per-sequence changesequences positive (%)tokens up (%)tokens down (%)
Accumulating ICM (R=4), PG19annexed3.40.0058530.270.24
Accumulating ICM (R=4), PG19content1.70.06409917.3714.45
ARM (G=4), PG19sink slot25.20.0036540.220.20
ARM (G=4), PG19content1.40.05259812.4510.39
Accumulating ICM (R=4), FineWebannexed9.70.0011460.160.14
Accumulating ICM (R=4), FineWebcontent1.20.083010012.618.94
ARM (G=4), FineWebsink slot41.40.0014550.140.13
ARM (G=4), FineWebcontent1.00.05541009.467.00

We thank the reviewer for separating the two readings and agree that the submission blurred them: Appendix C.2 read a small swap effect as input independence, which only the probe of Table 14 of the submission measures. We have revised Appendix C.2 to keep the two apart. Table 13 of the submission (Table 16 of the revised PDF) swaps the cached key-value states of one slot at every layer, the states that the queries read, and its swap effect is the change in next-token loss on the audited sequences when those states are replaced, which is the measured predictive use of the slot. Table 14 probes the final-layer hidden state of the slot without any query and asks two questions, whether that state identifies its chunk (top-1 accuracy) and whether it varies across sequences (self-cosine). For the annexed slot of Accumulating ICM (R=4) they agree: its swap effect is -0.0000, and its self-cosine of 1.00 shows a final state that is the same for every document. For the first memory slot of ARM (G=4) in the same audit they disagree: its swap effect stays at or below 0.005, yet its self-cosine of 0.60 and top-1 accuracy of 57% show a final state that varies with the input and identifies its chunk. A small swap effect therefore does not imply an input-independent slot, and the revised Appendix C.2 reads the swap effect as predictive use and takes input independence from Table 14 alone (page 25).

AuthorsOpenReview ↗

Response to Reviewer rokC (Part 2/4). We thank the reviewer for W2 and Q2. We state the L-1 chunk horizon and the recurrence of the default ARM, measure the use of a chunk by distance, and name the fed-back variant at every result that uses it.

W2/Q2. The memory horizon of the one-pass design, the exact recurrence of the default ARM, the depth-dependent content horizon, and which results use the fed-back variant.

We state the bound in Section 5.2 and Appendix D in the revised PDF (page 29): the default ARM carries the register set only as per-layer key-value states without final-layer feedback, as Section 5 of the submission states, and a query of an L-layer model therefore depends on raw tokens at most L-1 chunks back, 11 chunks at 124M and 23 at 340M. In the default ARM, a query in chunk c reads the registers of chunk c-1 at the same layer, and the final layer of one chunk does not feed the input of the next. We chose this recurrence because one attention mask then implements the whole sequence in a single forward pass, as in Gisting (Mu et al., 2023), whereas a path from the final layer to the next input makes the chunks sequential during training. Information propagates one chunk per layer in this cache, which gives the bound of L-1 chunks, 11,264 tokens at 124M.

A swap of the register set by chunk distance shows that every design uses the content of a chunk mainly in the next two chunks. Within these two chunks, the default ARM carries three times the content of Accumulating ICM one chunk later, 0.638 compared to 0.215 on the long-range targets, and 1.5 times two chunks later, 0.102 compared to 0.067. The fed-back variant carries the most content of every design at distances 1 to 4 and stays above the noise level up to 8 chunks with the same 5 cache positions. For the swap by distance the reviewer asks for, we replace the register set written after chunk j by the set of another sequence, in one forward pass at 16 chunks on 32 PG19 test sequences (Table 33, page 39). The swap effect of the default ARM vanishes after three chunks, and Accumulating ICM, which keeps the set of every chunk, retains an effect of +0.006 at 9 to 15 chunks with a cache that grows with the number of chunks, whereas the rewritten set keeps 5 positions:

Table R10. Loss change after swapping the set written after chunk j, by distance.

designtokenschunks after the swapped set: 12345 to 89 to 15
Accumulating ICM (R=4)all tokens+0.017+0.005+0.0010.000-0.001-0.001
Accumulating ICM (R=4)long-range targets+0.215+0.067+0.039+0.024+0.010+0.006
ARM (G=4)all tokens+0.064+0.0060.0000.0000.0000.000
ARM (G=4)long-range targets+0.638+0.102+0.0100.0000.0000.000
ARM (G=4), fed-back variantall tokens+0.179+0.031+0.010+0.0030.0000.000
ARM (G=4), fed-back variantlong-range targets+0.938+0.202+0.085+0.030+0.011-0.001

We agree that each result should name the variant behind it, and we have clarified this in the revision. All results of the main text are obtained with the default ARM, and the fed-back variant appears only in the recall beyond four updates, in Table 32 of the submission and in Table R11. To clarify, Table 32 of the submission labels the variant as the last-layer output fed into the next update. The revision defines the fed-back variant in Section 5.2 beside the default recurrence, labels its rows in Table 42 (page 44) and Table 33 (page 39), and states that every result of the main text uses the default recurrence. The fed-back variant adds the final-layer state of the memory slots to the input of the next set and keeps the same 5 cache positions. Continued on 64 chunks of 256 tokens and evaluated on the same 256 sequences, the default ARM recalls 47.5% of the facts with one to four updates between the fact and the query and at most 1.2% of the facts beyond four updates. The fed-back variant recalls 22.6% of the facts with 32 to 63 updates between the fact and the query, with the same 5 cache positions, as Table R11 shows.

Table R11. Recall at 64 chunks of 256 tokens, by update distance.

designrecall (%) by the number of updates between the fact and the query: 1 to 45 to 78 to 1516 to 3132 to 63
Accumulating ICM R=4, all register sets0.00.90.40.01.2
ARM (G=4)47.50.90.40.41.2
ARM (G=4), fed-back variant49.050.950.442.622.6

Table 42 (page 44) reports Table R11. The 128k perplexity of the abstract is obtained with the default ARM as well.

AuthorsOpenReview ↗

Response to Reviewer rokC (Part 3/4). We thank the reviewer for Q3. We list what the decoding step behind the 39x throughput includes and report the time of the per-chunk update next to the step time.

Q3. Which operations the 39x timing includes.

The 39x is the decoding throughput of ARM (G=4) divided by the decoding throughput of Accumulating ICM (R=128), both at 128k tokens under a 40 GB cache budget on one A6000, and the timed decoding step includes, in every layer, the QKV projection of the new token, the attention of that token over the cached keys and values of its request, the output projection and the MLP. Under a fixed cache budget, the cache of one request sets how many requests decode concurrently and consequently the throughput (Kwon et al., 2023), and the constant cache of ARM is the strength that the 39x measures. For both designs at 128k tokens, in bf16 with the layer shapes of the 124M model, we try batches of concurrent requests up to the largest batch whose caches fit in the budget. Every request holds its own cache of the positions that the design retains, 1,029 for ARM and 17,280 for Accumulating ICM (R=128). We report the batch with the highest throughput. The throughput of ARM saturates at 512 requests and stays within 1% up to the largest batch that fits, 1,054 requests, whereas Accumulating ICM reaches the budget at 62 requests. We measure the median step time over 20 steps after 5 warm-up steps, and Table 3 reports the median of three measurements, 12.6 ms per step at 512 requests for ARM and 59.7 ms at 62 requests for Accumulating ICM, 40.8 and 1.04 thousand tokens per second. The token embedding, the LM head, sampling and the CPU-side loop are identical for both designs and are excluded from the timing.

The timed step excludes the update of the rewritten set, which runs once per chunk. At 512 requests the update takes 162 ms, and averaged over the 1,024 tokens decoded in a chunk it adds 1.3% to the step time of ARM, which lowers the 39x to 38.8x. ARM updates the rewritten set once per chunk as one forward pass of the G+1 register tokens over the raw tokens of the chunk and the slots of the previous set, under the same four operations and the same kernel. We time the update and the step in the same run and report both in Appendix E.3 in the revised PDF (page 32):

Table R12. Time of the per-chunk update of ARM (G=4), timed with the same four operations and kernel as the step. Share of step is the update time averaged over the 1,024 tokens of a chunk, divided by the step time.

model sizerequestsstep time (ms)update time (ms)share of step (%)
124M11.73.20.2
124M51212.41621.3
340M14.38.10.2
340M39527.92931.0

In the revised PDF, Appendix E.3 (page 32) lists the four timed operations and reports the time of the update next to the step time.

AuthorsOpenReview ↗

Response to Reviewer rokC (Part 4/4). We thank the reviewer for W3 and Q4. We report every training run of every design, replace the two-run recall mean by the mean over all runs, and compute paired sequence-level bootstrap intervals for the principal comparisons.

W3/Q4. Recall uncertainty does not reflect all sampling levels, the principal comparisons need per-seed values and paired sequence-level intervals, and the match of long-range perplexity is an overlap rather than an equivalence test.

The principal comparison of Table 3, ARM (G=4) against Accumulating ICM (R=4), holds in every training run: both recall 51 to 53% of the placed facts in each of their three runs. At 16 slots the recall varies across training runs for both families, and the revision therefore replaces the two-run mean of ARM (G=16), 78.6%, by the mean over all 12 training runs, 50.7%, and reports every run in Table 37 in the revised PDF (page 41) and Figure 16 (page 41). Following the suggestion, the revision reports every training run of every design in a per-seed table with the mean over the runs, and Table 3 of the revision carries the mean over every run (Table 4, page 8). Over the runs, ARM (G=16) recalls between 0.0 and 94.4% of the facts, Accumulating ICM (R=16) between 52.0 and 67.7% and Accumulating ICM (R=128) between 53.5 and 99.7%, and Figure 16 (page 41) plots every run against the number of memory slots. Table R13 summarises the runs of each design.

Table R13. Recall of the placed facts over the training runs of each design: 3 runs each at 4 slots, 12 runs of ARM (G=16) and 3 each of Accumulating ICM (R=16) and (R=128). Cache positions count the positions read beyond the window at 4 chunks.

designcache positionsrecall of the placed facts (%): lowest runhighest runmean
ARM (G=4)551.253.051.9
Accumulating ICM (R=4)1251.352.952.4
ARM (G=16)170.094.450.7
Accumulating ICM (R=16)4852.067.757.6
Accumulating ICM (R=128)38453.599.783.8

We compute paired sequence-level bootstrap intervals over the 192 evaluation sequences, and ARM (G=16) matches Accumulating ICM (R=128) within 0.04 in long-range loss at 4k tokens. For the long-range comparisons, we compute the paired difference of the loss between two designs on the same 192 evaluation sequences, with a 95% interval from 2,000 bootstrap resamples over sequences, the paired-sequence design the reviewer suggests. The paired bootstrap over evaluation units is the standard test for comparing two systems (Koehn, 2004). An interval that excludes zero means that the sign of the paired difference holds in at least 97.5% of the resamples over sequences (Table 39, page 42):

Table R14. Paired differences of the long-range loss, ARM minus the compared design, on the same 192 evaluation sequences, with 95% intervals from 2,000 bootstrap resamples over sequences.

ARMcompared designtokenslong-range loss: ARMcompareddifference95% interval
ARM (G=16)Accumulating ICM (R=128)4k6.2696.264+0.005[-0.026, +0.036]
ARM (G=4)Accumulating ICM (R=4)4k6.3416.416-0.075[-0.108, -0.042]
ARM (G=4)sliding window4k6.3416.601-0.259[-0.309, -0.215]
ARM (G=16)Accumulating ICM (R=128)16k6.3916.761-0.370[-0.394, -0.346]

The interval for ARM (G=16) and Accumulating ICM (R=128) at 4k tokens contains zero and lies within 0.04 in loss, which states the match of Table 3 as a paired margin instead of an overlap. The other three intervals exclude zero in favor of ARM: ARM (G=4) lowers the long-range loss by 0.075 compared to Accumulating ICM (R=4) and by 0.259 compared to the sliding window at 4k tokens, and ARM (G=16) lowers it by 0.370 compared to Accumulating ICM (R=128) at 16k tokens, differences of 2 to 15 times the half-width of their intervals. We have added these paired intervals to the revision.

In the revised PDF, Appendix F.7 (page 42) and Table 39 (page 42) replace the overlap statement by the paired margin of 0.04 in loss, about 4% in perplexity, and Table 37 (page 41) and Table 4 (page 8) report every run and the mean over every run.

Reviewer_V6VBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the attention sink in in-context memory (ICM), where each chunk is compressed into a few memory tokens and the raw tokens are evicted. A slot-level audit, which swaps a memory slot's KV states with those computed on another document, shows that the sink shifts onto the memory tokens: the most-attended slot stores no usable content, yet the standard removal test counts it as memory. The authors then propose Assigned-Role Memory (ARM), which combines a zero-valued sink slot with a single set of memory slots rewritten after every chunk, so the KV cache stays constant. ARM is evaluated on 124M–0.8B models pre-trained from scratch on PG19.

Critical strengths
  • The distinction between token presence and actual information storage is interesting, and the swap-based audit is a cleaner test than the removal test.
  • The method is simple, and the gains in cache efficiency and long-range perplexity are promising (17 cache positions match per-chunk accumulation with 384).
Critical weaknesses
  1. Limited novelty and motivation. Attention sinks are already well studied, and recent architectural designs such as gated attention (Qiu et al., 2025) can substantially mitigate this phenomenon. In fact, the paper's own Appendix E.4 shows that the output gate alone also leaves no memory slot annexed in Accumulating ICM, i.e., an existing remedy already removes the sink tax. The argument that the gate cannot bound the cache is about the memory layout (the rewritten set), not about the sink slot, and the decisive control, the rewritten set with the gate but without the sink slot, is not reported. The paper also sets aside gate/sigmoid remedies because they require a model trained with them, but ARM likewise requires training from scratch. The paper therefore does not convincingly establish why a dedicated sink slot remains necessary.

  2. Insufficient evaluation. ARM is evaluated only on small models (124M–0.8B) pre-trained from scratch on PG19. The audit covers 7B checkpoints, but ARM itself is never applied to a modern pretrained LLM, and there is no evaluation on realistic long-context benchmarks (e.g., RULER, LongBench). The downstream evidence is limited to PG19 perplexity and a synthetic key-value recall task whose pairs are also injected into the training text. This significantly limits the practical relevance of the results.

  3. Questionable memory design motivation. The paper does not adequately justify why compressing historical context into a small fixed set of latent slots is preferable to external memory with selective retrieval, particularly for tasks requiring precise information recall. Retrieval-based memory is neither discussed nor compared. Even the best configuration, ARM (G = 16), recalls 78.6% of the facts versus 100% for full attention, and 75.4% for facts written two updates back.

  4. Unclear attribution of improvements. ARM combines sink handling with recurrent memory updates, an existing architectural idea; the paper itself notes that the rewritten set without the sink slot follows the recurrence of RMT (Appendix F.7). The cumulative ablation in Table 3 does not isolate the contribution of the sink slot to the headline results. At G = 4, adding the rewritten set on top of the sink slot slightly worsens long-range perplexity (568 → 582), and its recall (51.5%) is close to Accumulating ICM with R = 4 (52.9%). The headline numbers (long-range perplexity 528, recall 78.6%) come from G = 16. The rewritten set without the sink slot is reported only in the slot audit and validation loss, never in long-range perplexity, recall, or the 128k evaluation, and has no G = 16 counterpart. It therefore remains unclear how much of the improvement comes from addressing the sink rather than from recurrent memory compression with more slots.

Minor: formatting and writing. The body text is set in Computer Modern rather than the Times font of the official ICLR template, which suggests that the template was modified. The writing is also hard to follow: many sentences are terse, repetitive, and mechanically phrased, and the paper does not appear to have been carefully proofread.

The architectural contribution is relatively incremental, and the evaluation is insufficient to establish its relevance to modern long-context LLMs. I do not find the empirical evidence convincing enough to support the paper's broader claims.

Questions for authors

Clarifications / missing controls (could change my assessment):

  1. What are the long-range perplexity, recall, and 128k perplexity of the rewritten set without the sink slot, at G = 4 and G = 16, under the PG19 setup of Table 3? If these are close to ARM, the sink slot's contribution would be mainly to the audit rather than to downstream performance.
  2. Table 23 reports ARM (G = 4) + output gate, which keeps the sink slot. What does the rewritten set with the output gate but without the sink slot achieve? This is the direct test of whether the sink slot is needed once a gate is used.

Suggestions (would substantially strengthen the paper):

  1. Can ARM be applied to an existing pretrained LLM (e.g., via continued training of a 7B/8B model) and evaluated on realistic long-context benchmarks such as RULER or LongBench?
  2. How does ARM compare with external memory with selective retrieval over the evicted context at a matched memory budget, especially on precise recall?
5 comments
AuthorsOpenReview ↗

Response to Reviewer V6VB (Part 1/5). We thank the reviewer for W1, the output gate as the alternative to the sink slot. We train the gate as a control: it lowers the loss, and its memory slots return none of the placed facts, whereas the sink slot keeps the recall at a constant cache.

We thank Reviewer V6VB for the review and for noting that our swap-based audit provides a cleaner test than the removal test. Below we respond to each point in five parts, beginning with W1.

W1. Gated attention already removes the sink and ARM also needs training from scratch, which leaves the necessity of a sink slot unestablished.

The output gate removes the sink tax, as the reviewer states, and the ablation the reviewer asks for, the rewritten set with the output gate and without the sink slot, is reported in Part 3. We need the sink slot because in-context memory exists to carry the content of the evicted chunks to later queries, which the memory slots of the gate variant do not: ARM (G=4) recalls the placed facts in every training run, whereas the gate variant recalls none. In the recall task of Table 3, a fact is one key followed by one value, placed in an earlier chunk and queried after the model evicts its raw tokens.

The lower perplexity of the gate variant comes from the attention over the raw tokens and not from its memory slots. The output gate reaches a lower perplexity than the sink slot, by 2.7 to 2.9 points at 4k tokens at every G (Part 3). The gate acts on every attention block, and its gain can come from the raw tokens or from the memory slots. To separate the two, we add the same gate to a sliding window without memory slots. The gate lowers the loss of the window by 0.023 and that of the rewritten set (G=4) by 0.027 (Table 26 in the revised PDF, page 36). The memory slots therefore account for 0.004 of the gain, close to the 0.003 that the sink slot gives on the same layout.

ARM (G=4) returns the placed facts on the recall task, on the first query of each sequence and on RULER-format needles in every training run, whereas the gate variant returns none on the recall task in any run. A far query is a query whose key-value pair stands in an earlier, evicted chunk, which only the memory slots can answer. In the recall task of Table 3, every training run of ARM (G=4) recalls 51 to 53% of the placed facts, whereas the gate variant recalls 0.0 to 0.2% in its three runs (Part 3). The four far queries of a sequence are written consecutively at the end of a later chunk, as in multi-query associative recall (Arora et al., 2024). From the second query on, the model also sees the answers to the earlier queries. The first far query is therefore the only one that must take its value from the memory slots alone, like the single query of a needle test, and we report it separately. On it, ARM answers 22 to 25% correctly, whereas the gate variant and the window answer 0.0 to 0.6%, the accuracy of a random choice among the 256 values. On RULER-format needles sampled at evaluation time on held-out sequences, ARM (G=4) retrieves 22.8 to 30.3% in its three runs, and the gate variant 0.0% in one run and 25.6% in the other (Part 4).

Table R1. Retrieval of the placed facts at 124M, G=4, means over the training runs. Rows marked + add the component to the first row.

design, PG19 124M, W=256cache sizerecall of the placed facts (%)first far query (%)RULER needles exact match (%)
Sliding window00.20.30.0
Rewritten set (G=4)452.227.528.5
+ output gate40.10.112.8
+ sink slot (ARM)551.924.326.3
+ sink slot + output gate50.10.27.6

The output gate empties the memory slots of the placed facts both alone and together with the sink slot. When we combine the sink slot and the output gate, the gate takes the sink duty, and the sink slot receives only 2.2% of the attention at recall time instead of 63%. The combined model recalls 0.1 to 0.2%, the level of the gate variant alone. The memory slots of the gate variant reach a swap effect of 0.057, close to the 0.059 of ARM. They carry context that lowers the loss on the text but not the key-value pairs.

Neither the sink slot nor the gate requires training from scratch, but only the sink slot leaves the attention computation of a pretrained model unchanged. Both are added to a pretrained model by continued training, and the sink slot adds one cache position whereas the gate adds a module to every attention block (Part 4). In sum, the output gate removes the sink by modifying the attention block and empties the memory slots of facts, whereas the sink slot removes the sink with one cache position and preserves fact recall.

In the revised PDF, we state the two component roles in the abstract and Section 5 and report in Section 6.3 (page 9) that the memory slots of the gate variant recall none of the placed facts. We also add the rows of the rewritten set with the output gate and without the sink slot at G=4 and G=16 to Table 27 (page 36).

AuthorsOpenReview ↗

Response to Reviewer V6VB (Part 2/5). W1 continued: where and why the gate variant loses the facts, with the gate values and the attention of the far queries measured in every layout (Table R2).

A memory unit performs two operations, a write that updates the memory from the chunk and a read that retrieves it at the query (Behrouz et al., 2025). We measure the attention of both operations per layer and head on the same held-out recall sequences and trained models as in Part 1.

The read of the memory slots is present at the far queries in every design: one head places an attention mass close to 1 on the memory slots, and in the gate variant the gating score of that head is 0.97 at G=4 and 0.94 at G=16. The swap effect of these slots is 0.057 (Part 1), and a gating score set to 1 at every position at evaluation time leaves the recall at 0.0% at both sizes.

The write fails: at the memory tokens of the chunk that holds the facts, the gate closes the heads that attend to the fact tokens. As the reference, we take the attention mass that the 17 fact tokens receive from a memory token that spreads its attention evenly over the 1,024 raw tokens of the chunk, 0.015 at G=4. Without the gate, the memory tokens of the rewritten set (G=4) place 0.056 on the fact tokens, 3.7 times the reference, and the largest head places 0.25. With the gate, the memory tokens place 0.017, the reference level, the mean gating score at the memory tokens is 0.33, and the largest attention mass multiplied by the gating score falls from 0.25 to 0.12 at G=4 and from 0.26 to 0.04 at G=16. After gate in the table denotes the largest attention mass multiplied by the gating score.

Table R2. Attention at the write and the read, 124M. Rows marked + add the component to the first row of their block.

designmemory tokens of the fact chunk (write): massreferencemax headgateafter gatefar queries (read): max headgate
Rewritten set (G=4)0.0560.0150.2510.250.991
+ sink slot (ARM)0.0490.0130.2510.250.991
+ output gate0.0170.0160.240.330.121.000.97
+ output gate initialised at 0.980.0420.0150.430.490.181.001.00
Rewritten set (G=16)0.0440.0150.2610.261.001
+ sink slot (ARM)0.0260.0140.1510.150.991
+ output gate0.0150.0160.220.280.040.970.94
+ output gate initialised at 0.980.0290.0150.240.440.141.000.90

The gate variant never writes the facts during training: its accuracy on the far queries stays at the accuracy of a random choice in every evaluation of all six training runs, whereas every design without the gate ends at 52.5 to 57.1%. We evaluate 256 far queries of 64 validation sequences every 131M training tokens. Across the three runs at G=4 and the three at G=16, the gate variant answers 0.0 to 1.6% of the far queries at every evaluation, compared to 0.4% for a random choice.

With the sink slot, the far-query accuracy rises later and ends at the level of the rewritten set without it, and the sink slot takes the sink early in training. The audit of the four ARM (G=4) recall runs at five saved states (Table 29, page 37) shows the sink slot taking the sink by 0.62B tokens, no memory slot annexed at any state, and the content effect of the memory slots above that of the rewritten set without the sink slot from 0.62B tokens on.

The gate closes the write heads because their output does not lower the loss on the text tokens, whereas the sink slot leaves every head output in place. The gating score is a learned function of the input, trained by the same loss as the rest of the model, and Qiu et al. (2025) show that the trained scores concentrate near zero: the gate passes the output of a head only where that output lowers the loss. At the raw tokens, this filtering removes the sink and lowers perplexity. At the memory tokens, the heads that attend to the fact tokens copy random key-value pairs, which lower no loss on the text. The only loss that rewards the copy is the recall loss at 8 of the 4,096 positions, and it rewards the copy only once the far queries read the slots. The gate therefore closes these heads. Even a gate initialized at 0.98 closes to 0.49 at the memory tokens by the end of training, its largest attention mass multiplied by the gating score falls to 0.18, and its recall reaches 17.3%, between the 0.2% of the default initialization and the 52% without the gate. The sink slot removes the sink without touching any head output, and ARM stores both the key-value pairs (51.9%) and the RULER-format needles (26.3%) at G=4.

In the revised PDF, we add the attention measurements in Table 28 (page 37), the training-time audit in Table 29 (page 37), the accuracy trajectories in Table 30 (page 37) and the analysis of the write heads in Appendix E.4 (page 35).

AuthorsOpenReview ↗

Response to Reviewer V6VB (Part 3/5). We thank the reviewer for Q1, Q2 and W4. We report the rewritten set without the sink slot and with the gate at G=4 and G=16, isolate the contribution of the sink slot in every layout pair, and state the cache trade-off of the rewritten set.

Q1/Q2. The long-range perplexity, recall and 128k perplexity of the rewritten set without the sink slot, and of the rewritten set with the output gate, at G=4 and G=16.

The rewritten set alone already carries the memory at a constant cache size, and the sink slot frees every memory slot of the sink in every run and lowers the long-range perplexity at G=16. Without the sink slot, the rewritten set reaches a long-range perplexity of 590, a recall of 52.2% and a perplexity of 78.2 at 128k tokens at G=4, and 570, 53.1% and 78.1 at G=16. With the sink slot, which is ARM, the three values are 582, 51.9% and 78.0 at G=4 and 540, 50.7% and 77.1 at G=16, means over the training runs of Table 3 (Table 27 in the revised PDF, page 36). Table 23 of the submission labelled its last row "ARM (G=4) + output gate" although that row lacks the sink slot, which the revision corrects.

The rewritten set with the output gate and without the sink slot reaches the lowest perplexity of every design and recalls 0.1% of the placed facts at both G. The gate added to ARM recalls 0.1 to 0.2% as well (Table R3).

Table R3. Rewritten set with and without the sink slot and the output gate, 124M. Rows marked + add the component to the first row of their block.

designcache sizerecall (%) : all far queriesannexed slotsjoint content effectRULER needles exact match (%)perplexity : 4k128klong- range
Sliding window00.200.0000.083.080.0736
+ output gate00.200.0000.080.177.1694
Rewritten set (G=4)452.210.05128.581.878.2590
+ sink slot (ARM)551.900.05926.381.578.0582
+ output gate40.100.05712.878.675.2550
+ sink slot + output gate50.100.0557.679.075.6565
Rewritten set (G=16)1653.100.06734.981.778.1570
+ sink slot (ARM)1750.700.07255.980.977.1540
+ output gate160.100.07072.078.274.6514
+ sink slot + output gate170.200.09254.478.975.3497

W4. The ablation of Table 3 does not isolate the sink slot, and the rewritten set does not add long-range perplexity or recall over per-chunk accumulation.

The rewritten set trades 14 points of long-range perplexity at 4k tokens, 568 to 582, for a constant cache of 5 positions instead of one that grows to 635, which lowers the perplexity at 128k tokens from 123.2 to 78.0.

The sink slot raises the content that the memory slots store and lowers the long-range perplexity in every layout pair and fixes the position of the sink. Without the sink slot, the sink annexes one of the four memory slots of the rewritten set at G=4 and two of the four per chunk of Accumulating ICM (Table 2), and an annexed slot has a negligible effect on the loss (Table 35, page 40). Because the annexed slot differs across training runs, a cache policy that protects the sink, as StreamingLLM protects the initial tokens, has no fixed position to protect. Table R4 isolates the contribution of the sink slot: in every pair the joint content effect of the memory slots rises and the long-range perplexity falls, from 590 to 582 at G=4 and from 570 to 540 at G=16, where the retrieval of RULER-format needles rises from 34.9 to 55.9%. At G=16 the gate variant returns 72.0% of the needles, whose format its continuation trains on, and 0.1% of the facts of the recall task in every run.

Table R4. Each layout with and without the sink slot, 124M.

designjoint content effect : withoutwith sink slotlong-range perplexity : withoutwith sink slot
Accumulating ICM (R=4)0.0510.063621568
Rewritten set (G=4)0.0510.059590582
Rewritten set (G=16)0.0670.072570540

Without the output gate, Accumulating ICM, the rewritten set and ARM with four memory slots recall 51 to 53% of the facts, the accuracy of storing the values without their keys, and the submission's 51.5 and 52.9% lie there. Each answer is visible to the later queries of the sequence. A model that stores the four values without their keys and selects at random among the unused values is correct with probability 1/4 on the first query and 1 on the fourth, 0.52 on average, the bound under complete feedback (Diaconis and Graham, 1981). A recall above 52% requires the memory slots to store which key a value belongs to (Appendix F.6, page 40).

In the revised PDF, Table 27 (page 36) reports these rows relabelled, Section 6.3 (page 9) states the cache trade-off, Appendix F.6 (page 40) derives the 52% level, and Figure 17 (page 43) plots every training run.

Reviewer_fvp3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper introduces a systematic study of In-context-memory (ICM) methods and leverages insights from the study to propose a novel method: assigned-role-memory (ARM) that outperforms existing ICM methods on a cache-quality Pareto frontier. The paper uncovers that slot-based register methods, where chunks of the input sequence are assigned memory "slots", have the following interesting properties: (1) certain slots ocupy a large portion of the attention weights, however, their content may not be relevant, (2) dropping such slots hurts performance, but retaining them and swapping them out for something else surprsingly has little impact to quality, (3) these slots occupy a "sink tax" where such slots do not store important information but nevertheless occupy a large portion of weights in the attention logits space. The authors leverage these insights to propose their ARM method.

Critical strengths

The paper takes a systematic approach to first understand the problem space: proposing motivating experiments (Section 3 & 4) to support each hypothesis then exploits these observations to propose their technique. In this light the paper is informative.

Critical weaknesses

Several key weaknesses that I see with this paper. (1) The paper is incredibly dense and hard to read. Lacking figures explaining their technique. The paper would do well to aid figures in Section 3, explaining each of the concepts as well as Section 5, when introducing ARM. (2) The evaluation is quite thin. I appreciate the use of PG19, which is quite long-context. However, the classical long-context benchmarks are RULER and LongBench. Can the authors please run these as well to demonstrate good quality on these tasks. Since these tasks are diverse (encompassing NIAH, and other such complex retrieval tasks) this will be more convincing. (3) Although I appreciate Section 3 and 4, the motivating example is restricted to a 124M parameter model, which is frankly too small to generalize these insights. However, the authors claimed to have run larger audits (on up to 0.8B) and placed the results in the Appendix. This seems inverted to me: the paper would become stronger if larger models demonstrate the same issues as presented in Section 4.

Questions for authors

(1) Can the authors run LongBench and RULER to show the quality on these benchmarks. (2) Can the authors demonstrate Section 4's motivating observations using larger model sizes, preferably 1B?

1 comment
AuthorsOpenReview ↗

Response to Reviewer fvp3. We thank the reviewer for W1, W3 with Q2, and W2 with Q1. We add schematics of the audit and of ARM, extend the audit of Section 4 to 1.1B and to continued training at 1.5B and 7B, and evaluate RULER-format needles and LongBench on those models.

We thank Reviewer fvp3 for the review and for the reading that the paper takes a systematic approach to first understand the problem space. Below we respond to each point.

W1. The paper is dense and hard to read, and lacks figures explaining the audit of Section 3 and ARM in Section 5.

We agree and add a schematic figure to each section. Figure 2 in the revised PDF (page 3) illustrates the drop and swap interventions and the verdict rule of Section 3, and Figure 5 (page 7) illustrates the zero-valued sink slot inside the softmax and the update of the rewritten set across chunks in Section 5.

W3/Q2. The motivating audit is restricted to a 124M model, the larger audits sit in the appendix, and the observations of Section 4 should be shown up to 1B parameters.

We agree that the placement was inverted, and the revision moves the audits at the further scales into Section 4 (Table 1, page 5) and extends them to 1.1B parameters. Across the scales, the sink takes two of the four memory slots of Accumulating ICM at 124M, 340M and 0.8B, and three of the four at 1.1B.

Table R8. Slot audit of Accumulating ICM (R=4) under chunk-local masking from 124M to 1.1B.

scaletraining tokensAccumulating ICM R=4, chunk-local masking: annexed slots of 4content effect
124M2.5B2+0.075
340M2.5B2+0.079
0.8B5B2+0.100
1.1B2.5B3+0.087

W2/Q1. Can the authors run LongBench and RULER to show the quality on these benchmarks?

We add RULER-format needles and the single-document QA group of LongBench to the revision. We evaluate LongBench on Qwen2.5-1.5B-Instruct, because LongBench requires instruction following (Bai et al., 2024), with the four designs trained for 250M tokens under the protocol of Activation Beacon (Appendix F.13, page 45). The single-document QA group has three tasks, NarrativeQA, Qasper and MultiFieldQA-en, with 150 examples each. The mean F1 over the three tasks is 15.3 for Accumulating ICM (R=4), 15.5 for the rewritten set (G=4), 16.2 with the output gate and 15.0 for ARM (G=4) (Table 46, page 46). ARM reaches the F1 of Accumulating ICM on every task, with paired differences between -1.1 and +0.2 F1 whose 95% intervals over the examples include zero, while it keeps 5 positions beyond the window instead of 16 at the 8k prompt. The output gate leads ARM by 1.6 F1 on MultiFieldQA-en, the one difference whose interval excludes zero. The audit explains the small differences: after 250M tokens, the memory slots of every design pass the removal test and fail the swap test, and the memory slots therefore do not separate the designs on this benchmark.

On held-out RULER-format needles, ARM (G=4) retrieves 22.8 to 30.3% in each of its three runs with 5 cache positions, the sliding window retrieves none, and the gate variant retrieves 0.0% in one run and 25.6% in the other. We port the multi-key needle task of RULER (Hsieh et al., 2024) to our 124M models, because the RULER suite needs a pretrained instruction-following model and the 1.5B designs retrieve no needle after 250M tokens (Table 45, page 46). The needles are placed in PG19 text beyond the window of raw tokens, the recall runs continue for 250M tokens with needles in the RULER sentence format, and we score the exact match on 256 held-out sequences (Table 43, page 44). ARM matches the exact match of Accumulating ICM (R=4) with 5 cache positions instead of 12, as Table R15 shows, and returns the placed facts in the RULER format as in the recall task of Table 3, with a constant cache.

Table R15. RULER-format needles at 124M, exact match, means over the training runs. Rows marked + add the component to the rewritten set.

design, PG19 124M, W=256cache positions beyond the windowexact match (%)
Sliding window00.0
Rewritten set (G=4)428.5
+ output gate412.8
+ sink slot (ARM)526.3
Accumulating ICM (R=4)1226.7

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Paper Revision. We add an output-gate ablation, continued training at 1.5B and 7B with RULER-format needles and LongBench, a comparison with an external memory over the evicted cache, every training run with paired intervals, two schematics, and a rewrite of the main text for readability, in a nine-page revision in Times.

We thank the three reviewers and have uploaded the revised PDF. The main text is rewritten throughout, and the changes are therefore listed below instead of colored. Table and figure numbers below refer to the revised PDF. In the replies, bare table numbers follow the submission, pointers to the revised PDF give its page, and Tables R1 to R15 are in the replies. The shared concerns are a pretrained LLM and long-context benchmarks (V6VB W2 and Q3, fvp3 W2 and Q1), seeds and intervals (rokC W3 and Q4, V6VB W3), and the scale of the audit (fvp3 W3 and Q2). The changes are:

  • Font and layout: Times, nine pages.
  • Readability (fvp3 W1, V6VB W5): following the reviewers' comments, we have rewritten the abstract, the introduction and Sections 2 to 7. Each result is stated with its reason, every term is defined before its first use, and setup details and secondary diagnostics have moved to the appendix with pointers. The results and their numbers are unchanged, and Table 15 (page 24) adds the 1.1B model.
  • Figure 2 (page 3): a schematic of the drop and swap interventions and the verdict rule of the slot audit.
  • Figure 5 (page 7): a schematic of ARM with the zero-valued sink slot inside the softmax and the update of the rewritten set.
  • Table 1 (page 5): the audits at 124M, 340M, 0.8B and 1.1B. The sink takes two of the four memory slots of Accumulating ICM up to 0.8B and three at 1.1B, none with the sink slot. The memory slot with the most attention stores no content that the model uses at every scale up to 1.1B.
  • Abstract and Section 5: we state the role of each component, the sink slot for the position of the sink in every training run and the rewritten set for the size of the cache. We remove the recall sentence of the abstract and the introduction that rested on two training runs.
  • Table 4 (page 8) and Table 37 (page 41): every recall cell is the mean over all training runs of the design, and the per-seed table lists every run.
  • Sections 6.2 and 6.3: Accumulating ICM, the rewritten set and ARM with four memory slots and without the output gate recall 51 to 53% of the placed facts, the accuracy of storing values without their keys. The rewritten set trades 14 points of long-range perplexity at 4k tokens, 568 to 582, for a constant cache that lowers the perplexity at 128k tokens from 123.2 to 78.0.
  • Section 3 and Table 8 (page 18): a content-free position is one whose content has a negligible effect on the loss of the audited sequences. Per-sequence swap effects do not cancel.
  • Section 5.2 and Appendix D (page 29): the default ARM carries the register set only as per-layer key-value states, and a query of an L-layer model therefore depends on raw tokens at most L-1 chunks back, 11 chunks at 124M.
  • Appendix E.3 (page 32): what the decoding step behind the 39x throughput includes. The per-chunk update adds about one percent of a step.
  • Tables 26 and 27 (page 36): the rewritten set without the sink slot and with the output gate at G=4 and G=16, with the gain of the gate split into two parts, the larger on a sliding window without memory slots.
  • Appendix F.4 and Table 33 (page 39): a swap of the register set by chunk distance. The default ARM uses the content of one chunk in the next two to three chunks, the fed-back variant for eight.
  • Appendix F.6 (page 40), Table 37 and Figure 16 (page 41): the recall of every training run of every design against the number of memory slots.
  • Appendix F.7 and Table 39 (page 42): paired sequence-level differences of the long-range loss with 95% bootstrap intervals. ARM (G=16) matches Accumulating ICM (R=128) within a paired interval of 0.04 in loss.
  • Figure 17 (page 43): the content stored in the memory slots and the long-range loss of every run, with and without the sink slot. The sink slot raises the content in every pair.
  • Appendix F.11 and Table 43 (page 44): RULER-format needles on held-out sequences. ARM (G=4) retrieves 26.3% of the needles beyond the window with 5 cache positions, the sliding window none.
  • Appendix F.12 and Table 44 (page 45): an external memory with selective retrieval over the evicted cache. At the budget of ARM it recalls 86% of the facts and ARM 52%, and it stores 130,816 positions at 128k tokens, ARM 5.
  • Appendix F.13 (page 45) and Tables 45 to 48 (pages 46 and 47): continued training of Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct under the protocol of Activation Beacon. At 1.5B the memory slots of every design pass the removal test and fail the swap test, and the four designs score within 2 F1 of one another on the single-document QA group of LongBench. At 7B, Accumulating ICM and ARM reach 10.8 and 10.9 in perplexity at 32k tokens and lie within 2.1 F1 of each other on every single-document QA task, with 5 cache positions instead of 64 for ARM.

We thank the reviewers for their constructive feedback, which has improved the clarity and substance of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.