acceptodds
Under review as a conference paper at ICLR 2027

Double-Take: Improving Capacity, Binding, and Selectivity in Long-Context within Bidirectional Memory

Abstract

Processing a long sequence does not guarantee that distant information remains usable. As context grows, a fixed-size memory must retain many associations, connect information across positions, and preserve relevant structure without being overwhelmed by everything it observes. These are distinct requirements: a memory may store more information without representing the relationships needed for retrieval, and it may transport those relationships without preserving the ones that matter. Thus, the challenge is not simply extending context length, but constructing a memory that remains usable as context grows. We study this problem in large-chunk test-time training (TTT) and identify two basic failures: single-step writes accumulate interference, limiting capacity, while selfassociative writes cannot form cross-position bindings. We introduce DoubleTake (DT), which repairs these failures with multi-step unpinned updates and cross-position mixing, and adds bidirectional change-sensitive writes for selective retention. Capacity alone leaves end-to-end recall at the 0.10 chance floor; adding binding raises it to 0.92. Trained only on 32 kb genomic contexts, DT reaches 0.927 AUROC at 131 kb and 0.856 near 1 Mb, while transferring zero-shot from human to mouse. Together, these results identify capacity, binding, and selectivity as three requirements for usable long-range memory

Primary area:
foundation or frontier models, including LLMs
Keywords:
Genomics, Bioinformatics, Machine Learning, Test-Time Training, On-Line Learning, In-context Adaptation, Deep Learning, Transformer, DNA modelling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cATyOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DoubleTake, breaking down the usability of fast long-context weight memory into three issues: capacity, binding, and selectivity. It tackles these problems with multi-step unpinned updates and cross-position mixing, local convolutions on Q/K/V, and bidirectional change-sensitive writes. The authors tested these components on different tasks and got pretty good results.

Critical strengths
  1. The authors break down the problem in a fairly reasonable and clear way. Questions like 'how much can be stored,' 'whether the necessary associations are written,' and 'which signals to prioritize keeping' are indeed worth analyzing separately, which is more informative than just reporting the maximum input length.
  2. The controlled memory experiments, architecture-matched stock-memory comparisons, and remote masking experiments provide complementary evidence. The authors also clearly distinguish between architecture-matched controls and smaller external genome models.
Critical weaknesses
  1. The internal loop target given in this text lacks an error correction item, and the explanation of "repeated fitting to eliminate crosstalk" is not supported by the update rules. Eq. (2) Using Lmem=−∑i⟨fW(ki),vi⟩\mathcal L_{\mathrm{mem}}=-\sum_i\langle f_W(k_i),v_i\rangle In the linear case, (nabla_Wmathcal L=-VK^top) is independent of the current memory state. Therefore, as long as the same deterministic NS transformation is used at each step, the multi-step update after state normalization is exactly satisfied W(S)=W(0)+(∑sηs)NS⁡(VK⊤)W^{(S)}=W^{(0)}+\Big(\sum_s\eta_s\Big)\operatorname{NS}(VK^\top) It is fully equivalent to a write with an increased learning rate. Proper association is amplified together with crosstalk; multiple steps alone do not improve the signal-to-noise ratio of the current block write; while signal strength changes relative to the old state.
  2. The strongest performance achieved by the model depends on additional inference gain and does not include a complete three-component method; existing ablation experiments are insufficient to support the main gain from a unified mechanism. At 131 kb, DT increases from 0.834 with no gain to 0.927 with gain 4; at 999 kb, from 0.722 to 0.856. The latter's 0.134 gain is quite significant, and the DT without gain is lower than 0.741 for stock memory at this length. Therefore, the best results close to 1 Mb cannot be attributed solely to capacity versus binding fixes; the authors need to explain the gain selection data, search scope, and whether they are fixed for all tasks and lengths, and further compare after applying the same gain search budget to stock memory.
  3. Downstream data partitioning is not clear enough, and it is unclear whether there is conflict with "zero-shot" claims. The paper does not clearly provide the number of boundaries for each set, chromosome or region divisions, or isolation methods for adjacent and overlapping windows. When the sequence context grows to several hundred kb or 1 Mb, adjacent samples may share a large number of sequences.
  4. The argument that "writing at the same location cannot form cross-position bindings". The derivation on page 4 of the article holds for unmixed, position-independent inputs, but the full DT/stock encoder already includes local convolution and sliding window attention before the memory module, so the (h_i) of subsequent layers may contain information about adjacent locations. Therefore, the representation constraint of isolated memory modules cannot be directly elevated to the universal non-expressibility of complete LaCT architectures.
  5. "Non-normalization" preserves the additive components of old parameters but does not guarantee readability of old associations. This is the difference between parameter retention and functional retention. Even in linear models, $W_m=W_0+\sum_{r=1}^{m}\Delta W_r, \qquad W_mq=W_0q+\sum_{r=1}^{m}\Delta W_rq.$ Subsequent writes will still interfere with the old queries. In nonlinear SwiGLU memory, multiple matrices jointly determine the output, and retaining the old additive term of each matrix does not guarantee that the old function value remains unchanged.
Questions for authors

Please kindly refer to the weaknesses.

5 comments

We thank the reviewer for the careful, technical reading. Below: each sentence of the review, with the manuscript location, the rebuttal result, or the updated claim.

Derivation R-A. The SwiGLU memory gradient depends on the state. DT's memory is fW(x)=W1(SiLU(W0x)⊙W2x)f_W(x) = W_1\big(\mathrm{SiLU}(W_0x) \odot W_2x\big) (l. 134–136). With ai=SiLU(W0ki)a_i = \mathrm{SiLU}(W_0k_i), bi=W2kib_i = W_2k_i and gi=W1⊤vig_i = W_1^\top v_i:

∇W1L=−∑ivi(ai⊙bi)⊤,∇W2L=−∑i(gi⊙ai)ki⊤,∇W0L=−∑i(gi⊙bi⊙SiLU′(W0ki))ki⊤.\nabla_{W_1}\mathcal{L} = -\sum_i v_i(a_i\odot b_i)^\top,\quad \nabla_{W_2}\mathcal{L} = -\sum_i (g_i\odot a_i)k_i^\top,\quad \nabla_{W_0}\mathcal{L} = -\sum_i \big(g_i\odot b_i\odot \mathrm{SiLU}'(W_0k_i)\big)k_i^\top.

Each gradient depends on matrices that change after every step, so W(S)=W(0)+(∑sηs) NS(VK⊤)W^{(S)} = W^{(0)} + (\sum_s \eta_s)\,\mathrm{NS}(VK^\top) holds only for a linear memory.

Table R1. Isolated capacity test (recall vs. stored pairs NN; d=32d = 32; 3 seeds; best learning rate per rule from the same grid, single-step grid extended to 5×; rerun of Table 8).

Write1248163264
1 step, renorm (stock)1.00.67.50.21.08.04.02
1 step, no renorm1.00.67.50.25.08.07.03
Delta rule, 1 step1.00.67.42.21.08.05.03
5 steps, renorm1.001.001.00.79.31.08.03
5 steps, no renorm (DT)1.001.001.001.001.00.92.71

[W1] Multi-step update

"The internal loop target given in this text lacks an error correction item, and the explanation of 'repeated fitting to eliminate crosstalk' is not supported by the update rules." Lmem\mathcal{L}_\text{mem} (Eq. 2, l. 172) has no error-correction term. We update the claim: the explanation "repeated fitting to eliminate crosstalk" is removed from l. 062–063, l. 725–726 and l. 946, and the multi-step effect is reported only as measured in Table R1.

"In the linear case, ∇WL=−VK⊤\nabla_W\mathcal{L} = -VK^\top is independent of the current memory state … It is fully equivalent to a write with an increased learning rate." In the linear case, the equivalence is in the manuscript (App. A, Eq. 4, l. 612–616; l. 625). The sentence at l. 625 is truncated, and the final version restores the sentence with the reviewer's formula. DT's memory is the nonlinear SwiGLU network (l. 134–136), where the gradient depends on the current memory state (Derivation R-A). Shown in the rebuttal, Table R1 tests the equivalence directly: one step at the best learning rate, searched up to 5× larger, recalls 0.08 at N=16N = 16, while five steps recall 1.00. In DT's memory, the multi-step update is therefore not a write with an increased learning rate.

"Proper association is amplified together with crosstalk; multiple steps alone do not improve the signal-to-noise ratio of the current block write; while signal strength changes relative to the old state." In the linear case, Eq. 5 (l. 618–620) shows the same: signal and crosstalk scale together, so multiple steps alone do not improve the signal-to-noise ratio of the current block write; only the signal strength relative to the old state changes. In the linear case, the NS orthogonalisation (Eq. 6, l. 627) is the only source of crosstalk reduction. For the nonlinear SwiGLU memory, App. A reports the multi-step benefit as empirical (l. 633–635), and Table R1 measures the benefit.

Table R2. TAD-boundary AUROC under random and chromosome-level probe splits (frozen encoder features, logistic probe of Table 11; distinct windows from the full held-out chromosomes; random split: 70/30, mean over 20 splits; chromosome split: probe trained on one held-out chromosome, tested on the other; "—": not evaluated).

Encoder2k8k32k65k131k262k524k999k
Test windows overlapping a train window (random split)0%4%11%23%32%68%88%98%
Stock memory
Random split.663.685.706.749.742.752.750.741
chr8 → chr9.700.717.697.765.732.789.753.721
chr9 → chr8.666.689.688.728.729.773.717.711
DT + gain 4
Random split.633.691.744.766.765.791.766—
chr8 → chr9.593.686.747.768.753.788.751—
chr9 → chr8.607.678.739.755.777.783.740—

Table R4. Annotated TAD boundaries per set.

SetChromosomesBoundaries
Probe train (chromosome split)chr8776
Probe test (chromosome split)chr9697
Held-out totalchr8, chr91,473
Pretraining (MLM)remaining 2115,670
Totalall 2317,143

[W2] Inference gain

"The strongest performance achieved by the model depends on additional inference gain and does not include a complete three-component method …" The strongest configuration uses an inference gain (l. 334–335) and omits the change-point write (Table 5, l. 714–719); the three components are combined only in Selective DT (Sec. 4.4). We update the claim to the mechanisms tested in each experiment.

"At 131 kb, DT increases from 0.834 with no gain to 0.927 with gain 4; at 999 kb, from 0.722 to 0.856." The values are those of Table 11 (l. 984–985). With distinct windows sampled from the full held-out chromosomes, DT + gain 4 reaches 0.765 at 131 kb (Table R2); the final version reports the distinct-window protocol in Table 11, Table 2 and the abstract.

"… the DT without gain is lower than 0.741 for stock memory at this length." Table 11 reports 0.722 at 999 kb. We update the claim: the extrapolation claim (l. 074–076) is softened, and the gain is reported as a separate inference setting.

"… explain the gain selection data, search scope, and whether they are fixed for all tasks and lengths, and further compare after applying the same gain search budget to stock memory." We update the claim: the best results close to 1 Mb are not attributed to the capacity and binding fixes alone. The requested details:

  • Search scope: gains 1, 2, 4 and 8.
  • Selection data: the held-out chromosomes; for stock memory, gain 4 selected on chr8 gives 0.783 on chr9 at 131 kb, against 0.732 at gain 1.
  • Fixed for all tasks and lengths: one fixed gain of 4 at inference for all lengths and tasks; the final version reports every result with and without the gain.
  • Same gain for stock memory: the same gain raises stock memory from 0.742 to 0.800 at 131 kb (TAD) and lowers DT memory recall from 0.877 to 0.214 (binding task), so the gain is a readout effect, not a memory effect.

[W3] Data partitioning

"Downstream data partitioning is not clear enough, and it is unclear whether there is conflict with 'zero-shot' claims." "Zero-shot" refers only to the encoder, so the partitioning has no conflict with the claim: pretrained on human GRCh38 with chr8 and chr9 held out (l. 262–263), the encoder never sees mouse sequence and stays frozen; a linear probe is fit per task and species (l. 454–455). The final version states the definition in the abstract (l. 027–028) and Sec. 4.5, and describes the partitioning below.

"… the number of boundaries for each set, chromosome or region divisions, or isolation methods for adjacent and overlapping windows." All three are now reported. Boundaries per set: Table R4. Divisions: human chr8 and chr9 (l. 821–822); mouse mm10 chr1–5 (l. 454). Isolation: a chromosome-level split (train on chr8, test on chr9, and the reverse), where no training window shares sequence with a test window (Table R2).

"… adjacent samples may share a large number of sequences." Shown in Table R2: under the random split, 68% of test windows at 262 kb and 98% at 999 kb overlap a training window. The reported values sample evaluation windows from a tokenized validation file, in which neighbouring windows overlap; with distinct windows, DT + gain 4 reaches 0.765 at 131 kb (W2). With distinct windows, the chromosome-level split stays within 0.04 of the random split at every length, for both stock memory and DT + gain 4 (Table R2).

Table R3. Binding task, full blocks (MQAR recall vs. gap between the written pair and the query).

SetupGap 16Gap 64Gap 256Gap 1024
Attention only.97.02.02.02
Full stock block.18.18.18.17
DT memory only.96.97.91.89
Full DT block.87.97.98.98

[W4] Same-position writes

"The derivation on page 4 of the article holds for unmixed, position-independent inputs, but the full DT/stock encoder already includes local convolution and sliding window attention before the memory module, so the (hih_i) of subsequent layers may contain information about adjacent locations." The derivation (Sec. 3.2, Eq. 3; App. A, Eq. 7) concerns the isolated memory write, and hih_i does contain information about adjacent locations. Shown in the rebuttal, the adjacent-location information in hih_i does not form the binding in memory: the full stock block, with the local convolution and sliding window attention, recalls 0.17–0.18 at every gap from 16 to 1024, while the full DT block recalls 0.87–0.98 (Table R3).

"Therefore, the representation constraint of isolated memory modules cannot be directly elevated to the universal non-expressibility of complete LaCT architectures." We update the claim from non-expressibility to an empirical claim on complete blocks. Attention alone falls from 0.97 at gap 16 to 0.02 at gap 64, the full stock block stays at 0.17–0.18, and the full DT block keeps 0.98 at gap 1024 (Table R3). Prior work agrees: a single test-time regression layer with a short convolution solves the Multi-Query Associative Recall task (Wang et al., 2026).

[W5] Retention

"'Non-normalization' preserves the additive components of old parameters but does not guarantee readability of old associations. This is the difference between parameter retention and functional retention." We update the claim: the final version claims no readability guarantee (l. 180; App. A, l. 631–633) and adopts the terms parameter retention and functional retention, the distinction behind catastrophic interference (McCloskey & Cohen, 1989).

"Even in linear models, Wm=W0+∑r=1mΔWrW_m = W_0 + \sum_{r=1}^m \Delta W_r, Wmq=W0q+∑r=1mΔWrqW_m q = W_0 q + \sum_{r=1}^m \Delta W_r q. Subsequent writes will still interfere with the old queries." The old term W0qW_0 q is kept, and every later write ΔWrq\Delta W_r q adds to the readout. Renormalisation does not change the interference in the linear case: renormalisation rescales old signal and later interference by the same factor. Shown in Table R1: for one step, recall is 0.08 at N=16N = 16 both with and without renormalisation. We update the claim in App. A that renormalisation "erases" W0W_0 while non-normalization "dilutes" W0W_0 (l. 631–633).

"In nonlinear SwiGLU memory, multiple matrices jointly determine the output, and retaining the old additive term of each matrix does not guarantee that the old function value remains unchanged." The final version measures functional retention instead of claiming functional retention. Shown in the rebuttal: with five steps, renormalisation lowers recall from 1.00 to 0.31 at N=16N = 16 (Table R1); without renormalisation, DT memory still recalls 0.89 of written associations at a gap of 1024 (Table R3).

We thank the reviewer again. In the final version we will:

  • revise the multi-step explanation and restore the linear-case equivalence (W1);
  • report every result with and without the gain, and soften the extrapolation claim (W2);
  • report boundary counts and the chromosome-level split (W3);
  • state the binding claim empirically for complete blocks (W4);
  • adopt parameter and functional retention as separate terms (W5).

References

  • McCloskey, M., & Cohen, N. J. (1989). Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24, 109–165.
  • Wang, K. A., Shi, J., & Fox, E. B. (2026). Test-time regression: A unifying framework for designing sequence models with associative memory. JMLR, 27(169), 1–41.
Reviewer_T8G6OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies how to improve the ability of test-time training (TTT) architectures to retain and retrieve information from long sequences. The authors argue that existing large-chunk TTT memory mechanisms suffer from three limitations: limited associative capacity due to interference between memory writes, difficulty representing cross-position associations, and insufficient selectivity in retaining important information.

To address these issues, the authors propose Double-Take (DT), which combines three modifications to the TTT memory mechanism: (1) multiple inner-loop updates without weight renormalization to improve memory capacity, (2) local convolutional mixing of query/key/value representations to enable cross-position binding, and (3) bidirectional change-point writes to promote selective retention.

The paper evaluates these mechanisms through synthetic associative-recall experiments and long-context genomic modeling tasks. The results show improvements in associative recall and genomic TAD-boundary prediction, including performance at sequence lengths substantially exceeding the pretraining context. Additional experiments investigate selective retention and cross-species generalization.

Critical strengths
  1. Interesting and relevant problem. The paper addresses an important challenge in long-context sequence modeling: the ability to process long sequences does not necessarily imply that distant information can be effectively retained and retrieved. The proposed distinction between memory capacity, binding, and selectivity provides an interesting perspective on this problem.

  2. Promising empirical results. The paper demonstrates substantial improvements in associative recall and long-context genomic prediction. In particular, the proposed method achieves an AUROC of 0.927 on TAD-boundary classification at 131 kb despite being pretrained on only 32 kb sequences, and maintains reasonable performance at nearly 1 Mb. The cross-species experiments also provide encouraging evidence of generalization.

  3. Useful controlled experiments. The synthetic associative-recall experiments attempt to isolate the limitations of existing fast-weight memory mechanisms. The observation that improving capacity alone does not improve cross-position recall, whereas introducing local mixing increases recall from 0.10 to 0.92, is particularly interesting.

Critical weaknesses
  1. Unclear problem formulation and insufficiently defined concepts. One of my main concerns is that the paper does not clearly formulate the problem it aims to solve. Although capacity, binding, and selectivity are presented as three central properties of long-context memory, their precise meanings and relationships to the downstream learning objectives are not sufficiently established. For example, it is unclear whether binding refers to representing local cross-position relationships or retrieving arbitrary long-range associations. Similarly, selectivity is motivated as preferentially retaining important information, but the connection between the proposed change-point mechanism and task-relevant information is not adequately justified. These ambiguities make the technical contributions difficult to identify and evaluate.

  2. Insufficient justification and unclear novelty of the proposed mechanisms. The proposed method combines multi-step memory updates, local convolutional mixing, and bidirectional change-point writes. However, it is not sufficiently clear which aspects of these modifications are technically novel or why they are necessary. For example, the capacity analysis primarily concerns interference in linear associative memory and does not directly explain the benefit of multiple updates in the nonlinear model. Similarly, the argument that same-position writes cannot represent cross-position associations does not fully account for the contextual representations already produced by convolution and local attention in the complete architecture. A more careful explanation of the limitations of existing methods and how each proposed mechanism addresses them would strengthen the paper.

Questions for authors
  1. Problem formulation and definitions. Could the authors provide more precise definitions of capacity, binding, and selectivity, and explain how these properties relate to the downstream genomic prediction tasks? In particular, what distinguishes successful cross-position binding from ordinary contextual feature mixing?

  2. Technical contributions. What is the main technical novelty of each proposed component relative to existing fast-weight memory and sequence-mixing methods? Could the authors clarify why multiple nonlinear inner updates are beneficial beyond update normalization and the removal of weight renormalization?

3 comments

We thank the reviewer for the careful reading. The TAD AUROC of 0.927 highlighted by the reviewer is updated by the distinct-window ablation in our general comment (0.765 at 131 kb); the controlled experiments are unchanged. Below: the problem formulation and definitions, then each sentence of the review.

The problem the paper aims to solve. The memory is a fast-weight state WW. Each chunk mm is first read with the state carried from earlier chunks, Om=fWm−1(Qm)O_m = f_{W_{m-1}}(Q_m), and then written into WW (Eq. 1–2; l. 243–246). Information from beyond the ±1,536 nt local path (Table 4) can reach a prediction only through WW. The problem is therefore: distant information must be retained and retrieved through a fixed-size memory written once per chunk.

Definitions.

  • Capacity: how many key–value associations written in one chunk can still be recalled, measured as recall vs. the number of stored pairs NN (Fig. 2a, Table 8).
  • Binding: the write stores a relationship between two different positions i≠ji \neq j, i.e. cross terms vikj⊤v_i k_j^\top (Eq. 3, Eq. 7), so a query at one position retrieves content from the other; measured as cross-position recall (Fig. 2b, Table 9) and recall over distance (Fig. 2c, Table 10).
  • Selectivity: a change between left and right context is written more strongly than uniform context, measured as Ssel=Δboundary/Δrandom>1S_\text{sel} = \Delta_\text{boundary}/\Delta_\text{random} > 1 (Eq. 8, Fig. 4c–d, Tables 13–14).

[W1] Problem formulation and definitions

"One of my main concerns is that the paper does not clearly formulate the problem it aims to solve." The manuscript states the problem as three questions (l. 045–048, l. 099–105). The formulation above states the problem precisely, and we add the formulation to Sec. 3.

"Although capacity, binding, and selectivity are presented as three central properties of long-context memory, their precise meanings and relationships to the downstream learning objectives are not sufficiently established." The definitions above give the meaning and the measured quantity of each property; every measurement is in the manuscript (Tables 8–10, 13–14). Relation to the downstream tasks: a long genomic window holds more associations than one write can store (capacity); regulatory partners lie at different positions (binding); TAD boundaries and structural variants are changes in context (selectivity).

"For example, it is unclear whether binding refers to representing local cross-position relationships or retrieving arbitrary long-range associations." Binding refers to representing local cross-position relationships, which the memory then retrieves at long range. The width-3 convolution places cross terms viki−r⊤v_i k_{i-r}^\top, r≤2r \le 2, into the write (Eq. 3, l. 191–197), and the persistent state carries the cross terms to later chunks (l. 199–202: local mixing provides the binding, the persistent memory provides the transport). Binding does not refer to arbitrary long-range associations: one write binds only positions within the convolution width. Fig. 2c measures the long-range retrieval: as the separation grows, DT recall changes only from 0.859 to 0.827, while attention-only recall falls from 0.142 to 0.019 (Table 10).

"Similarly, selectivity is motivated as preferentially retaining important information, but the connection between the proposed change-point mechanism and task-relevant information is not adequately justified." We update the claim: "important information" (l. 059–060, l. 067–068) becomes the selectivity definition above. The change-point target meets the definition by construction, because yi=μiL−μiRy_i = \mu^L_i - \mu^R_i cancels when the two sides are similar (Sec. 3.3, l. 212–215). Changing only the write target raises SselS_\text{sel} from 1.06 to 1.15 on controlled boundaries and from 1.01 to 1.09 on gnomAD-SV, and raises P(rare > common) from 0.42 to 0.67 (Sec. 4.4, l. 421–425; Tables 13–14). The Fig. 4c–d comparisons are not matched on composition, so we describe selectivity as a preference for contextual change, not for function.

"These ambiguities make the technical contributions difficult to identify and evaluate." The definitions above and the component list in W2 map each technical contribution to one property, one mechanism and one result.

[W2] Novelty and justification of the mechanisms

"The proposed method combines multi-step memory updates, local convolutional mixing, and bidirectional change-point writes." The three mechanisms are the three changes to LaCT listed in Sec. 3 (l. 124–126).

"However, it is not sufficiently clear which aspects of these modifications are technically novel or why they are necessary." Novelty and necessity of each modification:

  • Multi-step memory updates. Novel: several updates with free retention, instead of LaCT's single renormalised write (Zhang et al., 2025) or the delta rule (Schlag et al., 2021; Yang et al., 2024). Necessary for capacity: at 16 associations, recall is 0.09 with the stock write and 0.99 with DT (Table 8).
  • Local convolutional mixing. The convolution itself is not technically novel (H3, Fu et al., 2023; Hyena, Poli et al., 2023; Based, Arora et al., 2024b; Titans, Behrouz et al., 2026). Novel: the diagnosis that a self-associative write has no cross-position term (App. A, Eq. 7). Necessary for binding: with the capacity repair alone, cross-position recall stays at chance (0.10), and with mixing recall reaches 0.92 (Table 9).
  • Bidirectional change-point writes. Novel as a write target. Necessary for selectivity: with the associative write, SselS_\text{sel} stays near 1 (0.99–1.07, Table 12), and with the change-point write SselS_\text{sel} rises to 1.15 (Table 13).

"For example, the capacity analysis primarily concerns interference in linear associative memory and does not directly explain the benefit of multiple updates in the nonlinear model." In linear associative memory, multiple updates bring no benefit: the gradient −VK⊤-VK^\top does not depend on the state, so SS steps equal one step of size SηS\eta. App. A states the equivalence at l. 625; the sentence is truncated, and the final version restores the sentence and adds the equivalence to Sec. 3.1. In the nonlinear model the benefit is empirical, as App. A states (l. 633–635): in the SwiGLU memory the gradient depends on the current state, so the steps are not equivalent. Table 8 separates the factors at 16 associations: all rows use the same Newton–Schulz normalisation; five updates with renormalisation give 0.34, and five updates without renormalisation give 0.99. One update without renormalisation gives 0.08 (Table R1 in our response to Reviewers cATy and zYQk), so neither factor alone reaches DT.

"Similarly, the argument that same-position writes cannot represent cross-position associations does not fully account for the contextual representations already produced by convolution and local attention in the complete architecture." We update the claim to the isolated memory write and to associations beyond the local reach. Convolution and local attention produce cross-position information only within ±1,536 nt (Table 4; l. 690–691) and recompute the information for every query, whereas the convolution on the memory's inputs writes the cross term into WW, which carries the term to later chunks (l. 199–202). In complete blocks, the full stock block recalls 0.17–0.18 at every gap from 16 to 1024, while the full DT block recalls 0.87–0.98 (Table R3 in our response to Reviewers cATy and zYQk). Prior work also finds that local mixing helps where attention already mixes tokens (Canon layers; Allen-Zhu, 2025).

"A more careful explanation of the limitations of existing methods and how each proposed mechanism addresses them would strengthen the paper." The component list above gives the limitation each mechanism addresses and the supporting result; we add the list and the definitions to Sec. 3.

[Q1] "…more precise definitions of capacity, binding, and selectivity… what distinguishes successful cross-position binding from ordinary contextual feature mixing?" Answered in W1 (definitions and relation to the downstream tasks) and W2 (fourth answer): successful binding persists in memory beyond the ±1,536 nt local reach, while contextual feature mixing does not (Table 10; Table R3).

[Q2] "…main technical novelty of each proposed component… why multiple nonlinear inner updates are beneficial beyond update normalization and the removal of weight renormalization?" Answered in W2 (second and third answers).

Thank you again for your detailed review. As the discussion closes on 18 November, we answered each one: a precise problem formulation with definitions of capacity, binding and selectivity, the novelty and necessity of each mechanism, the factors behind the multi-step benefit, and a new full-block test showing that binding differs from ordinary contextual mixing.

The other reviewers also recognized the value of the work. Reviewer zYQk found that "separating capacity, binding, and selectivity is a useful frame, and the synthetic suite isolates each in turn rather than reporting an aggregate." Reviewer cATy noted that "the authors break down the problem in a fairly reasonable and clear way", and after our response wrote that the responses "have addressed many of my concerns" and raised the score from 2 to 3.

If our responses address your concerns, we would be grateful if you would consider adjusting your score.

Reviewer_zYQkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Double-Take (DT) modifies the LaCT large-chunk test-time memory framework through three main interventions: five Newton–Schulz inner updates with no per-chunk weight renormalization ("free retention"), a width-3 depthwise causal convolution on Q/K/V streams prior to the write, and a bidirectional change-point write target (left-minus-right 512-nt context mean), framed as repairs to capacity, binding, and selectivity. DT is integrated into an 87M-parameter single-nucleotide encoder pretrained via masked language modeling (MLM) on 32 kb GRCh38 windows. Synthetic recall reaches 0.93 at 32 associations, and cross-position recall rises from a 0.10 chance baseline to 0.92 with mixing. Frozen-encoder TAD AUROC rises from 0.847 at 32 kb to 0.927 at 131 kb and 0.856 near 1 Mb; frozen DT achieves 0.728 on DNALongBench eQTL; and human-to-mouse transfer is evaluated zero-shot.

Critical strengths
  1. Separating capacity, binding, and selectivity is a useful frame, and the synthetic suite isolates each in turn rather than reporting an aggregate.
  2. Stock, span, and distal controls share architecture, pretraining, and downstream protocol, and public models are explicitly labeled external references. The ±1.5 kb attention-only receptive field gives a clean local/memory boundary.
Critical weaknesses
  1. The TAD task is not shown to measure long-range memory. It separates boundary-centred windows from windows ≥100 kb away, two extremes differing in CTCF, GC, gene density and N fraction, using an undescribed probe split over heavily overlapping windows. Both the leakage and the composition shortcut grow with context, which is the reported trend (Xi & Beer 2018; Schreiber et al. 2020), the reported AUROC of 0.927 cannot then be definitively attributed to memory rather than regional characteristics or memorized loci.
  2. The results rely on a 4× gain on the memory branch, applied only at inference and never explained. Without it, DT falls to 0.722 at 999 kb, below the stock baseline’s 0.741, so the central extrapolation claim rests entirely on this setting. The gain also costs accuracy at 2 and 8 kb, so it trades short-context performance for long, a trade-off the paper does not explore.
  3. The comparisons do not cleanly isolate the proposed mechanisms. As Sec. 4.2 and Table 5 indicate, all variants share one pretrained encoder with only the write rule swapped, so the DT-vs-stock gap measures robustness to that swap rather than the benefit of training with the rule. Table 5 also omits the change-point write from the main configuration, leaving the three mechanisms untested together.
  4. Capacity and binding are weakly established. Short convolutions before the write are standard, and the defect they repair is derived for the isolated memory, not the full block, where the convolution and attention already mix neighbours. The capacity ablation also varies two things at once and gives no evidence the stock baseline was tuned as carefully.
  5. The selectivity result may follow from the definition of the write. A left-minus-right contrast responds to any shift in composition, and boundaries are compositionally distinctive, so the memory would favour them regardless of their function. The comparison needs random regions matched to boundaries on compositional contrast and error bars to demonstrate selectivity for functional structure.
  6. HyenaDNA and Caduceus are the only long-context models evaluated, I would include top performing models like AlphaGenome, Evo2, etc.
Questions for authors
  1. How were the probe’s train and test examples split, and do the results hold under a chromosome-level split?
  2. What AUROC do GC/k-mer, CTCF-density and one-hot CNN baselines reach at each context length?
  3. How was the 4× memory-branch gain chosen, on what data, and was it used across all experiments?
  4. Were the memory variants pretrained separately from scratch, or were they swapped post-hoc, and was the change-point write active during pretraining for each experiment?
  5. Does the full stock block, complete with its internal convolution and sliding-window attention, actually fail the binding task? Furthermore, how do a 1-step-without-renormalization write and a delta-rule write score on the isolated capacity test?
  6. What do composition-matched perturbations score on the Figure 4c-d comparisons, and what are the sample sizes (n) and confidence intervals for those measurements?
  7. How does frozen DT compare with AlphaGenome or Evo 2 on the eQTL and TAD tasks?
5 comments

We thank the reviewer for the careful review. Following the reviewer's questions on the TAD task, we add three ablations: distinct-window sampling, a chromosome-level split, and composition, CTCF-density and one-hot CNN baselines. Tables R1–R3 are shared with our response to Reviewer cATy. Each question points to the weakness answer that covers the question.

Table R2. TAD-boundary AUROC under random and chromosome-level probe splits (frozen encoder features, logistic probe of Table 11; distinct windows from the full held-out chromosomes; random split: 70/30, mean over 20 splits; chromosome split: probe trained on one held-out chromosome, tested on the other; "—": not evaluated).

Encoder2k8k32k65k131k262k524k999k
Test windows overlapping a train window (random split)0%4%11%23%32%68%88%98%
Stock memory
Random split.663.685.706.749.742.752.750.741
chr8 → chr9.700.717.697.765.732.789.753.721
chr9 → chr8.666.689.688.728.729.773.717.711
DT + gain 4
Random split.633.691.744.766.765.791.766—
chr8 → chr9.593.686.747.768.753.788.751—
chr9 → chr8.607.678.739.755.777.783.740—

Table R5. DT vs. baselines (TAD AUROC; windows and probe of Table R2, random split; 512 = probe readout window; 3,072 = ±1,536 nt attention-only receptive field).

Model2k8k32k65k131k262k524k
DT + gain 4.633.691.744.766.765.791.766
GC/k-mer, 512.685.673.738.687.665.694.725
GC/k-mer, 3,072.636.738.732.764.707.769.741
CTCF density.661.636.777.813.860.780.760
One-hot CNN.660.760.830.850.860.820.860

[W1] TAD task

"The TAD task is not shown to measure long-range memory." We update the claim: the TAD task alone does not establish long-range memory of boundary structure. Shown in the rebuttal: from 65 kb on, DT exceeds GC/k-mer composition within the 512-position probe window, and from 131 kb on, within the 3,072-position receptive field (0.765 vs. 0.665 and 0.707 at 131 kb; Table R5), so DT's features carry information from beyond the local receptive field.

"It separates boundary-centred windows from windows ≥100 kb away, two extremes differing in CTCF, GC, gene density and N fraction, using an undescribed probe split over heavily overlapping windows." The two extremes are defined in the manuscript (App. B.7, l. 825–827) and differ in composition and CTCF density (Table R5). The probe split, now described in App. B.7: a logistic classifier on frozen features, trained and tested on a random 70/30 partition of boundary and control windows from the held-out chromosomes. Shown in the rebuttal: a chromosome-level split, where the probe is trained on one held-out chromosome and tested on the other, so no test window shares sequence with a training window (Table R2).

"Both the leakage and the composition shortcut grow with context, which is the reported trend (Xi & Beer 2018; Schreiber et al. 2020), the reported AUROC of 0.927 cannot then be definitively attributed to memory rather than regional characteristics or memorized loci." Shown in the rebuttal, three ablations:

  • Distinct windows. The reported values sample evaluation windows from a tokenized validation file, in which neighbouring windows overlap. With distinct windows sampled from the full held-out chromosomes, DT + gain 4 reaches 0.765 at 131 kb; the final version reports the distinct-window protocol in Table 11, Table 2 and the abstract.
  • Chromosome-level split. With distinct windows, the chromosome-level split stays within 0.04 of the random split at every length, for both stock memory and DT + gain 4 (Table R2).
  • Composition baselines. CTCF density rises from 0.661 at 2 kb to 0.860 at 131 kb, and the one-hot CNN matches or exceeds DT at every length, while composition within the 512-position probe window stays below DT from 8 kb on (Table R5).

Memorized loci are excluded in the manuscript: chr8 and chr9 are held out from pretraining (l. 262–263). We update the claim: DT's features carry information from beyond the local receptive field, but the TAD task does not separate boundary structure from regional characteristics.

Table R6. Gain sweep (TAD AUROC, protocol of Table R2).

GainDT, 32kDT, 131kStock, 32kStock, 131k
1 (none).696.727.706.742
2.731.740.740.753
4.744.765.742.800
8.678.688.707.722

[W2] Inference gain

"The results rely on a 4× gain on the memory branch, applied only at inference and never explained." The final version explains the gain in Sec. 4.2. Shown in the rebuttal:

  • Choice: a sweep over gains 1, 2, 4 and 8 on the TAD task; gain 4 is the peak for both DT and stock memory (Table R6).
  • Data: the held-out chromosomes, under the logistic-probe protocol of the main results. For stock memory, gain 4 selected on chr8 gives 0.783 on chr9 at 131 kb, against 0.732 at gain 1.
  • Use: one fixed gain of 4 at inference for all lengths and tasks.
  • Effect: the gain is not specific to DT: gain 4 raises stock memory more than DT at 131 kb (+0.058 vs. +0.038; Table R6).

The final version reports every result with and without the gain.

"Without it, DT falls to 0.722 at 999 kb, below the stock baseline's 0.741, so the central extrapolation claim rests entirely on this setting." The values are those of Table 11. The gain is not specific to DT (W2), so we update the claim: the extrapolation claim (l. 074–076, Sec. 4.2) is softened, and the gain is reported as a separate inference setting.

"The gain also costs accuracy at 2 and 8 kb, so it trades short-context performance for long, a trade-off the paper does not explore." Table 11 shows the cost (0.693 vs. 0.723 at 2 kb; 0.760 vs. 0.771 at 8 kb). Shown in the rebuttal, Table R6 explores the trade-off: gain 4 helps both DT and stock memory at 32 kb and 131 kb, and gain 8 lowers both below gain 1, so the trade-off belongs to the readout scale, not to DT.

[W3] Isolation of the mechanisms

"The comparisons do not cleanly isolate the proposed mechanisms." The TAD sweep does not isolate the mechanisms; the controlled experiments do: capacity (Table R1, a full 2 × 2 of steps × renormalisation), binding (Table 9; Table R3) and selectivity (Tables 12–13, where only the write target changes).

"As Sec. 4.2 and Table 5 indicate, all variants share one pretrained encoder with only the write rule swapped, so the DT-vs-stock gap measures robustness to that swap rather than the benefit of training with the rule." We update the wording: Sec. 4.2 ("keeps our encoder fixed but replaces the stock memory", l. 333–334) and App. B (l. 660–662) describe a swap, but the variants are trained as follows. All variants share a base encoder pretrained with masked-language modeling, and each variant is specialized by continued pretraining under its own write rule: stock memory is pretrained from scratch; DT continues from the stock checkpoint under DT's configuration (five inner steps, free retention, width-3 convolution) with a fresh optimizer. The DT-vs-stock gap therefore compares training with the rule, not robustness to an inference-time swap. The comparison also includes a difference in training steps, since DT continues from the stock checkpoint; the final version reports the step counts in App. B.3.

"Table 5 also omits the change-point write from the main configuration, leaving the three mechanisms untested together." The manuscript tests the three mechanisms together in Selective DT (Table 5, l. 719), the configuration evaluated in Fig. 4c–d (Sec. 4.4). The main DT encoder is pretrained with the capacity and binding repairs, so the TAD results do not test the three mechanisms together; the final version states the division in Sec. 4.2.

Table R1. Isolated capacity test (recall vs. stored pairs NN; d=32d = 32; 3 seeds; best learning rate per rule from the same grid, single-step grid extended to 5×; rerun of Table 8).

Write1248163264
1 step, renorm (stock)1.00.67.50.21.08.04.02
1 step, no renorm1.00.67.50.25.08.07.03
Delta rule, 1 step1.00.67.42.21.08.05.03
5 steps, renorm1.001.001.00.79.31.08.03
5 steps, no renorm (DT)1.001.001.001.001.00.92.71

Table R3. Binding task, full blocks (MQAR recall vs. gap between the written pair and the query).

SetupGap 16Gap 64Gap 256Gap 1024
Attention only.97.02.02.02
Full stock block.18.18.18.17
DT memory only.96.97.91.89
Full DT block.87.97.98.98

[W4] Capacity and binding

"Capacity and binding are weakly established." The manuscript establishes each with a controlled test that changes one factor: capacity in Table 8, and binding in Table 9 (recall 0.10 → 0.92 once mixing is added). The rebuttal adds the full capacity ablation (Table R1) and the full-block binding test (Table R3).

"Short convolutions before the write are standard, and the defect they repair is derived for the isolated memory, not the full block, where the convolution and attention already mix neighbours." The convolution is not claimed as new (H3, Fu et al., 2023; Hyena, Poli et al., 2023; Based, Arora et al., 2024b); the claim is the diagnosis in App. A, Eq. 7. We update the claim to the isolated write and to relations beyond the ±1,536 nt reach of the local convolution and attention (Table 4). Shown in the rebuttal: in the full block, the full stock block recalls 0.17–0.18 at every gap from 16 to 1024, while the full DT block recalls 0.87–0.98 (Table R3).

"The capacity ablation also varies two things at once and gives no evidence the stock baseline was tuned as carefully." Shown in the rebuttal: Table R1 adds the missing cell (one step, no renormalisation), giving a full 2 × 2. At 16 associations, more steps alone give 0.08 → 0.31, removing renormalisation alone gives 0.08 → 0.08, and both together give 1.00. A delta-rule write also stays at 0.08. Every rule, the stock baseline included, uses the best learning rate from the same grid, over 3 seeds.

[W5] Selectivity

"The selectivity result may follow from the definition of the write." The manuscript defines the write to emphasise left–right change (Sec. 3.3, l. 212–215). Changing only the write target produces a directional preference: SselS_\text{sel} rises from 1.06 (associative write) to 1.15 (change-point write) on controlled boundaries and from 1.01 to 1.09 on gnomAD-SV (Table 13).

"A left-minus-right contrast responds to any shift in composition, and boundaries are compositionally distinctive, so the memory would favour them regardless of their function." The rare-vs-common comparison in the manuscript is stratified by allele frequency, a proxy for functional constraint, not by distance from a boundary, and P(rare > common) rises from 0.42 to 0.67 (Table 14); the preference is therefore not explained by the boundary-vs-far contrast alone. We update the claim: the final version describes selectivity as a preference for contextual change.

"The comparison needs random regions matched to boundaries on compositional contrast and error bars to demonstrate selectivity for functional structure." Composition-matched random regions were not run, so we update the claim: the final version does not claim selectivity for functional structure and states the limitation in App. B.10 (l. 899–902). Fig. 4a uses n = 100 boundaries (App. B.9, l. 872).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.