- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DoubleTake, breaking down the usability of fast long-context weight memory into three issues: capacity, binding, and selectivity. It tackles these problems with multi-step unpinned updates and cross-position mixing, local convolutions on Q/K/V, and bidirectional change-sensitive writes. The authors tested these components on different tasks and got pretty good results.
Critical strengths
- The authors break down the problem in a fairly reasonable and clear way. Questions like 'how much can be stored,' 'whether the necessary associations are written,' and 'which signals to prioritize keeping' are indeed worth analyzing separately, which is more informative than just reporting the maximum input length.
- The controlled memory experiments, architecture-matched stock-memory comparisons, and remote masking experiments provide complementary evidence. The authors also clearly distinguish between architecture-matched controls and smaller external genome models.
Critical weaknesses
- The internal loop target given in this text lacks an error correction item, and the explanation of "repeated fitting to eliminate crosstalk" is not supported by the update rules. Eq. (2) Using In the linear case, (nabla_Wmathcal L=-VK^top) is independent of the current memory state. Therefore, as long as the same deterministic NS transformation is used at each step, the multi-step update after state normalization is exactly satisfied It is fully equivalent to a write with an increased learning rate. Proper association is amplified together with crosstalk; multiple steps alone do not improve the signal-to-noise ratio of the current block write; while signal strength changes relative to the old state.
- The strongest performance achieved by the model depends on additional inference gain and does not include a complete three-component method; existing ablation experiments are insufficient to support the main gain from a unified mechanism. At 131 kb, DT increases from 0.834 with no gain to 0.927 with gain 4; at 999 kb, from 0.722 to 0.856. The latter's 0.134 gain is quite significant, and the DT without gain is lower than 0.741 for stock memory at this length. Therefore, the best results close to 1 Mb cannot be attributed solely to capacity versus binding fixes; the authors need to explain the gain selection data, search scope, and whether they are fixed for all tasks and lengths, and further compare after applying the same gain search budget to stock memory.
- Downstream data partitioning is not clear enough, and it is unclear whether there is conflict with "zero-shot" claims. The paper does not clearly provide the number of boundaries for each set, chromosome or region divisions, or isolation methods for adjacent and overlapping windows. When the sequence context grows to several hundred kb or 1 Mb, adjacent samples may share a large number of sequences.
- The argument that "writing at the same location cannot form cross-position bindings". The derivation on page 4 of the article holds for unmixed, position-independent inputs, but the full DT/stock encoder already includes local convolution and sliding window attention before the memory module, so the (h_i) of subsequent layers may contain information about adjacent locations. Therefore, the representation constraint of isolated memory modules cannot be directly elevated to the universal non-expressibility of complete LaCT architectures.
- "Non-normalization" preserves the additive components of old parameters but does not guarantee readability of old associations. This is the difference between parameter retention and functional retention. Even in linear models, $W_m=W_0+\sum_{r=1}^{m}\Delta W_r, \qquad W_mq=W_0q+\sum_{r=1}^{m}\Delta W_rq.$ Subsequent writes will still interfere with the old queries. In nonlinear SwiGLU memory, multiple matrices jointly determine the output, and retaining the old additive term of each matrix does not guarantee that the old function value remains unchanged.
Questions for authors
Please kindly refer to the weaknesses.
5 comments
Table R2. TAD-boundary AUROC under random and chromosome-level probe splits (frozen encoder features, logistic probe of Table 11; distinct windows from the full held-out chromosomes; random split: 70/30, mean over 20 splits; chromosome split: probe trained on one held-out chromosome, tested on the other; "—": not evaluated).
| Encoder | 2k | 8k | 32k | 65k | 131k | 262k | 524k | 999k |
|---|---|---|---|---|---|---|---|---|
| Test windows overlapping a train window (random split) | 0% | 4% | 11% | 23% | 32% | 68% | 88% | 98% |
| Stock memory | ||||||||
| Random split | .663 | .685 | .706 | .749 | .742 | .752 | .750 | .741 |
| chr8 → chr9 | .700 | .717 | .697 | .765 | .732 | .789 | .753 | .721 |
| chr9 → chr8 | .666 | .689 | .688 | .728 | .729 | .773 | .717 | .711 |
| DT + gain 4 | ||||||||
| Random split | .633 | .691 | .744 | .766 | .765 | .791 | .766 | — |
| chr8 → chr9 | .593 | .686 | .747 | .768 | .753 | .788 | .751 | — |
| chr9 → chr8 | .607 | .678 | .739 | .755 | .777 | .783 | .740 | — |
Table R4. Annotated TAD boundaries per set.
| Set | Chromosomes | Boundaries |
|---|---|---|
| Probe train (chromosome split) | chr8 | 776 |
| Probe test (chromosome split) | chr9 | 697 |
| Held-out total | chr8, chr9 | 1,473 |
| Pretraining (MLM) | remaining 21 | 15,670 |
| Total | all 23 | 17,143 |
[W2] Inference gain
"The strongest performance achieved by the model depends on additional inference gain and does not include a complete three-component method …" The strongest configuration uses an inference gain (l. 334–335) and omits the change-point write (Table 5, l. 714–719); the three components are combined only in Selective DT (Sec. 4.4). We update the claim to the mechanisms tested in each experiment.
"At 131 kb, DT increases from 0.834 with no gain to 0.927 with gain 4; at 999 kb, from 0.722 to 0.856." The values are those of Table 11 (l. 984–985). With distinct windows sampled from the full held-out chromosomes, DT + gain 4 reaches 0.765 at 131 kb (Table R2); the final version reports the distinct-window protocol in Table 11, Table 2 and the abstract.
"… the DT without gain is lower than 0.741 for stock memory at this length." Table 11 reports 0.722 at 999 kb. We update the claim: the extrapolation claim (l. 074–076) is softened, and the gain is reported as a separate inference setting.
"… explain the gain selection data, search scope, and whether they are fixed for all tasks and lengths, and further compare after applying the same gain search budget to stock memory." We update the claim: the best results close to 1 Mb are not attributed to the capacity and binding fixes alone. The requested details:
- Search scope: gains 1, 2, 4 and 8.
- Selection data: the held-out chromosomes; for stock memory, gain 4 selected on chr8 gives 0.783 on chr9 at 131 kb, against 0.732 at gain 1.
- Fixed for all tasks and lengths: one fixed gain of 4 at inference for all lengths and tasks; the final version reports every result with and without the gain.
- Same gain for stock memory: the same gain raises stock memory from 0.742 to 0.800 at 131 kb (TAD) and lowers DT memory recall from 0.877 to 0.214 (binding task), so the gain is a readout effect, not a memory effect.
[W3] Data partitioning
"Downstream data partitioning is not clear enough, and it is unclear whether there is conflict with 'zero-shot' claims." "Zero-shot" refers only to the encoder, so the partitioning has no conflict with the claim: pretrained on human GRCh38 with chr8 and chr9 held out (l. 262–263), the encoder never sees mouse sequence and stays frozen; a linear probe is fit per task and species (l. 454–455). The final version states the definition in the abstract (l. 027–028) and Sec. 4.5, and describes the partitioning below.
"… the number of boundaries for each set, chromosome or region divisions, or isolation methods for adjacent and overlapping windows." All three are now reported. Boundaries per set: Table R4. Divisions: human chr8 and chr9 (l. 821–822); mouse mm10 chr1–5 (l. 454). Isolation: a chromosome-level split (train on chr8, test on chr9, and the reverse), where no training window shares sequence with a test window (Table R2).
"… adjacent samples may share a large number of sequences." Shown in Table R2: under the random split, 68% of test windows at 262 kb and 98% at 999 kb overlap a training window. The reported values sample evaluation windows from a tokenized validation file, in which neighbouring windows overlap; with distinct windows, DT + gain 4 reaches 0.765 at 131 kb (W2). With distinct windows, the chromosome-level split stays within 0.04 of the random split at every length, for both stock memory and DT + gain 4 (Table R2).
Table R3. Binding task, full blocks (MQAR recall vs. gap between the written pair and the query).
| Setup | Gap 16 | Gap 64 | Gap 256 | Gap 1024 |
|---|---|---|---|---|
| Attention only | .97 | .02 | .02 | .02 |
| Full stock block | .18 | .18 | .18 | .17 |
| DT memory only | .96 | .97 | .91 | .89 |
| Full DT block | .87 | .97 | .98 | .98 |
[W4] Same-position writes
"The derivation on page 4 of the article holds for unmixed, position-independent inputs, but the full DT/stock encoder already includes local convolution and sliding window attention before the memory module, so the () of subsequent layers may contain information about adjacent locations." The derivation (Sec. 3.2, Eq. 3; App. A, Eq. 7) concerns the isolated memory write, and does contain information about adjacent locations. Shown in the rebuttal, the adjacent-location information in does not form the binding in memory: the full stock block, with the local convolution and sliding window attention, recalls 0.17–0.18 at every gap from 16 to 1024, while the full DT block recalls 0.87–0.98 (Table R3).
"Therefore, the representation constraint of isolated memory modules cannot be directly elevated to the universal non-expressibility of complete LaCT architectures." We update the claim from non-expressibility to an empirical claim on complete blocks. Attention alone falls from 0.97 at gap 16 to 0.02 at gap 64, the full stock block stays at 0.17–0.18, and the full DT block keeps 0.98 at gap 1024 (Table R3). Prior work agrees: a single test-time regression layer with a short convolution solves the Multi-Query Associative Recall task (Wang et al., 2026).
[W5] Retention
"'Non-normalization' preserves the additive components of old parameters but does not guarantee readability of old associations. This is the difference between parameter retention and functional retention." We update the claim: the final version claims no readability guarantee (l. 180; App. A, l. 631–633) and adopts the terms parameter retention and functional retention, the distinction behind catastrophic interference (McCloskey & Cohen, 1989).
"Even in linear models, , . Subsequent writes will still interfere with the old queries." The old term is kept, and every later write adds to the readout. Renormalisation does not change the interference in the linear case: renormalisation rescales old signal and later interference by the same factor. Shown in Table R1: for one step, recall is 0.08 at both with and without renormalisation. We update the claim in App. A that renormalisation "erases" while non-normalization "dilutes" (l. 631–633).
"In nonlinear SwiGLU memory, multiple matrices jointly determine the output, and retaining the old additive term of each matrix does not guarantee that the old function value remains unchanged." The final version measures functional retention instead of claiming functional retention. Shown in the rebuttal: with five steps, renormalisation lowers recall from 1.00 to 0.31 at (Table R1); without renormalisation, DT memory still recalls 0.89 of written associations at a gap of 1024 (Table R3).
We thank the reviewer again. In the final version we will:
- revise the multi-step explanation and restore the linear-case equivalence (W1);
- report every result with and without the gain, and soften the extrapolation claim (W2);
- report boundary counts and the chromosome-level split (W3);
- state the binding claim empirically for complete blocks (W4);
- adopt parameter and functional retention as separate terms (W5).
References
- McCloskey, M., & Cohen, N. J. (1989). Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24, 109–165.
- Wang, K. A., Shi, J., & Fox, E. B. (2026). Test-time regression: A unifying framework for designing sequence models with associative memory. JMLR, 27(169), 1–41.
We thank the reviewer for the careful, technical reading. Below: each sentence of the review, with the manuscript location, the rebuttal result, or the updated claim.
Derivation R-A. The SwiGLU memory gradient depends on the state. DT's memory is fW(x)=W1(SiLU(W0x)⊙W2x) (l. 134–136). With ai=SiLU(W0ki), bi=W2ki and gi=W1⊤vi:
∇W1L=−∑ivi(ai⊙bi)⊤,∇W2L=−∑i(gi⊙ai)ki⊤,∇W0L=−∑i(gi⊙bi⊙SiLU′(W0ki))ki⊤.
Each gradient depends on matrices that change after every step, so W(S)=W(0)+(∑sηs)NS(VK⊤) holds only for a linear memory.
Table R1. Isolated capacity test (recall vs. stored pairs N; d=32; 3 seeds; best learning rate per rule from the same grid, single-step grid extended to 5×; rerun of Table 8).
[W1] Multi-step update
"The internal loop target given in this text lacks an error correction item, and the explanation of 'repeated fitting to eliminate crosstalk' is not supported by the update rules." Lmem (Eq. 2, l. 172) has no error-correction term. We update the claim: the explanation "repeated fitting to eliminate crosstalk" is removed from l. 062–063, l. 725–726 and l. 946, and the multi-step effect is reported only as measured in Table R1.
"In the linear case, ∇WL=−VK⊤ is independent of the current memory state … It is fully equivalent to a write with an increased learning rate." In the linear case, the equivalence is in the manuscript (App. A, Eq. 4, l. 612–616; l. 625). The sentence at l. 625 is truncated, and the final version restores the sentence with the reviewer's formula. DT's memory is the nonlinear SwiGLU network (l. 134–136), where the gradient depends on the current memory state (Derivation R-A). Shown in the rebuttal, Table R1 tests the equivalence directly: one step at the best learning rate, searched up to 5× larger, recalls 0.08 at N=16, while five steps recall 1.00. In DT's memory, the multi-step update is therefore not a write with an increased learning rate.
"Proper association is amplified together with crosstalk; multiple steps alone do not improve the signal-to-noise ratio of the current block write; while signal strength changes relative to the old state." In the linear case, Eq. 5 (l. 618–620) shows the same: signal and crosstalk scale together, so multiple steps alone do not improve the signal-to-noise ratio of the current block write; only the signal strength relative to the old state changes. In the linear case, the NS orthogonalisation (Eq. 6, l. 627) is the only source of crosstalk reduction. For the nonlinear SwiGLU memory, App. A reports the multi-step benefit as empirical (l. 633–635), and Table R1 measures the benefit.