acceptodds
Under review as a conference paper at ICLR 2027

Stability of deep test-time-training memories : divergence and silent collapse

Abstract

Test-time-training (TTT) memories, introduced by Titans and ATLAS, store the past in the weights of a small neural network that keeps learning as the model runs. Each input triggers a gradient step on the memory's own prediction error. The memory updates itself on the fly at a fixed cost. This made long-term memory practical at scale and started a fast-growing line of work. The mechanism is delicate. The memory is a network continuously rewritten by the data it must retain, and its update is repeatedly reported to be unstable. No condition is known for when the update stays bounded. None is known for whether the trained memory stays healthy. For a memory that is itself a deep nonlinear network, neither has been established. We study both questions on two independent platforms and find two mechanisms. First, divergence. An update changes the network so that the next update is larger, and the state blows up. We give a condition that a practitioner can check and that prevents it: bound the size of each write, and keep a positive forgetting rate. Orthogonalising the write direction without bounding its size does not prevent it. Second, collapse. The memory stays numerically stable but loses its diversity. Nearly every write ends up pointing the same way. Collapse is learned during training, and the prediction loss barely moves while it happens. We give the tools to catch both failures: the stability condition above, and a paired retention probe with a rank diagnostic that read memory diversity directly. Whether collapse costs accuracy at scale is open. What we establish is that the prediction loss will not tell you it happened.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time training, linear attention, associative memory, deep memory modules, training stability, rank collapse, online gradient updates, memory evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_UXw2OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This is a paper which studies the stability of test-time-training memories, explaining some conditions which provide diagnostics of whether (a) the updates to the memory are divergent, and (b) the memory is collapsing to a 1d subspace. Unfortunately the work is extremely difficult to read and I find it hard to suggest acceptance.

Critical strengths

This is a very interesting thing to study: stability of RNNs is something that has a long history in the literature, and the non-linear generalization that is important for TTT memories of the type discussed here is really of intrinsic interest. Unfortunately I do not feel that this problem is well addressed in this work.

Critical weaknesses

The primary issue is simply that the information-density of the work and the poor exposition make it extremely hard to read it and to understand what has been done. It seems clear this was mostly LLM written. I usually take the stance that there is nothing wrong with using an LLM to draft the work, but when doing this one must carefully study and edit the output text to make sure it is suitable for a human reader. I do not believe this was done to the extent needed in this case.

A few specific examples for concreteness:

  1. Proposition 1 on line 215 derives a stability condition. One wants to compare this stability condition to the previous literature, in particular that of Tang et. al 2026, which did a similar analysis. Yet this comparison is actually done way back on line 118, before Proposition 1 has even appeared in the text or any of the symbols on line 118 have been defined. It is very hard to follow the logical flow.
  2. Are L R, \eta_{-}, \alpha_{+} defined before they are used on line 244? I can't seem to find them.
  3. On line 255, the word "self-referential" is used to describe the instability, as a discovery -- but isn't every non-linear instability of this sort? For example if one performed a more standard thing, e.g. forced an an RNN with a nonlinear activation outside of its stability region, would the same effect happen? (More to the point, is this section fundamentally different from things that happen for an RNN? It might be -- but precisely how? etc.)
  4. On line 257, there is the line "This places the published family." I really don't understand what this sentence means at all.

This continues, though I will stop giving examples.

After several re-readings I do believe I understood most of the main points of the work, but this required considerable effort and I am not 100% sure.

Questions for authors

I really would like to suggest that the authors restructure the exposition of the results to make it easier to follow and digest the scientific results.

Some suggestions are:

  1. At the moment the logical flow is difficult to follow: as usual for mostly LLM writing, the text is extremely information-dense, and minutiae appear to be given the same emphasis as important and novel results. In my opinion this has to be carefully rewritten with significant human input.
  2. It is important to carefully contrast the results found with the earlier literature; this has to be done in a way that puts the results in a proper context (e.g. by first reviewing any pertinent RNN-stability literature on the same problem, then explaining how the nonlinearities here and the novel form of the update change the problem, etc., comparing the specific formulas found with the closest specific results in the earlier literature, etc.) Again I believe this requires significant human input.
  3. Finally, I suggest a more explicit separation between theory and experiment: i.e. at the moment theoretical arguments such as those around line 366 are scattered within the experimental section. It would be cleaner if the theory section provided a clear hypothesis which the experimental section cleanly tested.

I look forward to reading an improved version but I should stress that really significant rewriting would be necessary for me to improve my score. I do believe that the scientific problem is very interesting and encourage the authors to work on the exposition as needed.

1 comment
AuthorsOpenReview ↗

Response to Reviewer UXw2 : rewrite plan and the RNN question

We thank the reviewer for the precise and detailed feedback. We are very glad that you find the topic interesting. We are also sincerely sorry that you found the paper too dense and not clear enough. We are preparing a revision that improves the clarity without radically changing the content and conclusions. We will upload it in the coming days. The structure changes, and we add new measurements. We will do our best to be clear and rigorous, and to change the structure so that the important elements are easier to extract.

As you suggest, the main text will keep only three results:

  • the loop between the state of the memory and its curvature, and the bound that cuts it;
  • the collapse of the memory toward one direction during training, and its cost in storage capacity;
  • the two diagnostics that detect these failures.

The rest moves to the appendix. The theory section will state hypotheses, and each experiment will test one of them.

The RNN question

In an RNN, the state is the vector h_t that summarises the sequence read so far, and the transition is h_{t+1} = σ(W h_t + U x_t). The weights W are fixed during the forward pass. With σ = tanh, the state stays bounded no matter what. If the network is forced outside its stability region, it oscillates or becomes chaotic, but its forward pass does not blow up. What blows up are the gradients of backpropagation (Pascanu et al., 2013). With σ linear or ReLU, the state can grow, but at most exponentially, at a rate set by W.

In the memory we study, the state is the weight matrix W_t of a small two-layer network. The transition is a gradient step on the inner loss: W_{t+1} = (1 − α) W_t − θ ∇L(W_t). For two layers and a quadratic loss, this gradient grows like the cube of the weights (it is exactly cubic when σ is linear). The Jacobian of the transition is (1 − α) I − θ H(W_t), where H is the curvature of the inner loss, and this curvature grows like ‖W_t‖². The local growth rate is therefore set by the state itself, not by fixed weights. Two consequences follow. The fixed point is stable only within a finite basin. Outside the basin, the growth is faster than exponential, and the state becomes infinite within a single forward pass. For example, gradient descent on x⁴/4 reads x ← x(1 − θx²). The origin is stable, the basin is |x| < √(2/θ), and far outside the basin log|x| roughly triples at each step.

Part of this analysis is classical. The local stability region of our Proposition 1 is that of LMS with momentum and leakage. Exceeding a local threshold is not divergence (Pascanu et al., 2013; Cohen et al., 2021). What is specific to this memory is that the curvature depends on the state. A threshold on the step θ therefore holds only at the state where it was computed. When the state grows, the curvature grows, and the same step becomes too large. No step fixed in advance can therefore guarantee the whole trajectory.

The closest comparison is the LMS filter and its normalised version (Nagumo and Noda, 1967). LMS learns a weight vector by gradient descent on an input x. Its step is stable as long as it stays below 2/‖x‖². This threshold falls as the input grows. Normalised LMS therefore divides the step by ‖x‖², and the threshold no longer depends on the input. Our memory has two layers, and each layer has its own input. The first layer receives the key k. Its norm ‖k‖ is fixed by key normalisation, as in normalised LMS. The second layer receives h = σ(W₁ k). Its norm grows with the state, and nothing normalises it. Bounding the size of each write plays that role for the second layer. This is the content of Lemma 1. It is also why orthogonalisation alone is not enough: it does not bound the size. The rewritten version will place our results relative to this literature before presenting them.

The examples

We also thank you for the precise examples. The comparison with Tang et al. now comes after Proposition 1: their condition λ_max(H) < 2 is the special case η = α = 0 of ours. The symbols L, R, η₋ and α₊ belonged to Corollary 2; the rule is now stated without them, its step margin leaves the main text, and the Corollary stays in the appendix with its definitions. The word "self-referential" is removed. The sentence "This places the published family" meant that this explains why ATLAS trains stably while the raw Titans-style rule diverges above a step threshold. It is rewritten that way.

Reviewer_4encOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

the paper studies two phenomena in TTT: divergence and collapse. Both occur silently, since the prediction loss stays flat and doesn't' indicate a problem. The first happens during inner loop and the latter during the outer loop training. By analysing a 2-layer network thr authors hypothesize through a quadratic analysis that divergence is a result of lack of proper bound and forgetting rate. The two layer network weights, unlike in the linear case, become part of the curvature creating a vicious cycle leading to exploding state. Collapse on the other hand happens since either the outer loop brings the momentum to near zero (video model) or the forgetting gate goes high (enwiki). The first is fixed with a proper bound, and for the second they show collapse is avoidable but no bulletproof fix is proposed. Two analysis tools are built: a memory probe, and the rank measure (pr) the study is conducted with two datasets: enwiki8 (with titan open source implementation) and EgoLife (with a small scale model the authors implement)

Critical strengths

(+) the insight that the weights holding the memory are also playing a direct role in the memory loss curvature is useful to understanding that bounding the update is critical and cannot be natrually achieved with only using a fixed step size.

(+) limitations are honest and transparent

(+) the causal intervention helps identify the momentum as the cause behind writing diversity (in the video model). Also, that collapse can't be fixed in hindsight through component transplanting or reviving momentum is also interesting.

Critical weaknesses

(-) my main concern is that despite the interesting measures of collapse and divergence, nothing proposed here performs significantly better in terms of prediction performance of a memory-free transformer. While recall improved with memory it was tied between collapse and collapse free variants. the one improvement shown on enwik8 with the authors' non-collapsed model isn't analysed. can this be explained with memory? i believe not, based on the reported results in section E. If i'm correct, can the authors run a controlled experiment to identify where the gap comes from?

(-) scope is limited to small scale models, often non-converged ones.

(-) Psi_c is shown to prevent collapse but why is that is never discussed. It couldn't be the bound since both arms are bounded. The causal intervention also doesn't explains it.

(-) the retention probe is a cool metric but to my understanding based on the reported results it doesn't flag collapse, which is the criticism of the authors against the prediction loss.

(-) the authors claim the condition is something a practitioner could utilize. but in practice it is demonstrated that actual models break the condition but do not fail. The curvature is much smaller than what the bound assumes which may lead to suppressing the memory too much. It seems the authors themselves did not follow the suggested practice for training the models (see section C).

(-) presentation is good in abstract and conclusion. everywhere else, and in method mostly, it is packed, hard to follow and fragmented. For instance, testbed is not defined in the main text, psi acts in two roles -- bounding and coupling, arm and chimera are terms never defined, etc.

Questions for authors

see weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4enc (1/2): enwik8 gap, scale

We sincerely thank the reviewer for this careful reading, appendices included. You brought to light ambiguities and shortcomings on our part. We reply below. We ran the controlled experiment requested in your first point. A rewrite is in progress; we describe it at the end.

1. Prediction and the enwik8 gap

These are two valid concerns, and we ran the controlled experiment you suggest.

A correction first. In the submitted version, the numbers of Appendix E were not comparable with each other, for two reasons. First, they came from the validations run during training. Each covers a single text sequence, which makes the numbers very noisy. Second, the models did not share the same learning rate (10⁻⁴ for the ψc model, 2 × 10⁻⁴ for the memory-free model).

We therefore redid the evaluation properly. We compare three models, trained with the same rate of 2 × 10⁻⁴, with three seeds each:

  • the reference model (the community implementation, whose memory collapses);
  • the same model, retrained with the ψc rule;
  • a transformer without memory.

All are evaluated on the same 64 held-out sequences of 512 characters. Result: the ψc model does not beat the memory-free transformer. It is slightly worse.

The experiment. Recall first how the memory works. At the start of each sequence, it starts from weights learned during training. It then modifies these weights at each write, along the sequence. The rest of the network reads its output.

For each seed, we take the reference model and the ψc model. We then test two manipulations, on the same sequences:

  • writes off: the memory keeps its initial weights for the whole sequence;
  • read off: the output of the memory is replaced by zero.

Loss in nats/char, seeds 0 / 1 / 2:

ManipulationReference modelψc model
None2.52 / 2.36 / 2.391.43 / 1.53 / 1.35
Writes offsame, within 0.004same, within 0.002
Read off3.13 / 3.21 / 2.621.44 / 1.67 / 1.41

The memory-free transformer scores 1.33 on all three seeds.

What we conclude. Turning the writes off changes neither model. So the gap does not come from what the memory writes at test time. Turning the read off, however, degrades both models. So the network does use the memory, but as a fixed layer learned during training. This settles the question you raised: the gap comes from the weights learned during training, not from what the memory writes at test time. The ψc rule removes most of this gap, but brings no gain over the memory-free transformer. We will say so plainly in the revision and replace the numbers of Appendix E.

The aim of our work is not to build a model that beats a transformer. It is to study the divergence and the collapse of TTT memories. On the associative recall task, the two rules tie. The difference appears when we measure what the memory can still store after a long stream (point 4).

2. Scale and convergence

We agree that the models are small. We do not aim at the state of the art. We study two failure modes, and we analyse them from a theoretical angle. Small models let us train many seeds and examine every write.

Two things on convergence. First, divergence is not a late-training effect. Above the step threshold, our runs without a bound on the writes diverge within the first training steps (at step 8 in Fig. 2a). The blow-up itself happens while reading a single sequence of 128 tokens, before the next update of the model. The memory writes at every token. Within about ten chunks of 8 tokens, its weights gain about five orders of magnitude (Fig. 2b), then become infinite.

Second, collapse does not go away with longer training. We trained two models with the unit rule up to 30,000 steps. Their memory is collapsed from step 4,000 on and stays so to the end. The ψc model is not immune either. The number of directions its memory uses decreases slowly late in training. We also added seeds. The measurement of collapse during training (Fig. 4a) goes from 2 to 8 seeds per rule. Under the unit rule, the memory collapses on all 8 seeds. Under ψc, it stays diverse on 6 seeds out of 8

AuthorsOpenReview ↗

Response to Reviewer 4enc (2/2): ψc, retention probe, stability condition, presentation

3. Why ψc protects

This is a fair point; the submitted version did not answer this question.

What we observe. Each write follows a decayed sum of the recent gradients (the momentum); η is the learned decay. With the unit rule, training drives η to zero. Its median value falls below 10⁻³ within 6,000 steps and keeps falling. With ψc, η stays above 0.97. Our causal intervention shows that η makes the difference. We take, from the healthy model, the part of the network that computes η. We graft it onto the collapsed model. Its writes then become diverse again. But this intervention does not say why training switches η off with unit and not with ψc. Both rules bound the writes in the same way, so the bound explains nothing, as you note.

What we tested. We tested one hypothesis. Under ψc, the size of a write grows with the size of the sum; switching η off would therefore shrink the writes, and training would lose from it. Under unit, the size of a write does not depend on the size of the sum, and nothing of the kind holds η up. We therefore retrained the ψc model with this link between η and the write size cut out of the gradient, on 8 seeds. η stays active on all 8 seeds, with a median value of 0.99, as for the normal ψc model. Rank and memory gain are unchanged. So the hypothesis does not hold. We do not yet know why ψc keeps η. We will state it as an open question in the revision.

4. The retention probe

You are right, and our contribution 3 was badly phrased. The retention probe does not flag collapse. Collapsed models keep a gain of +0.38 to +0.53 on it, against about +0.55 for healthy models. On 8 seeds the two ranges overlap. It is the rank measure that flags collapse. The retention probe checks that the model reads its memory. It also flags a memory that diverges or stops writing.

We measured the cost of collapse differently. We pass a stream of 100 video clips through the trained model. The model no longer changes; only the memory is written along the stream. We then write random pairs into the memory, with the learned gates, and query each key. Each pair maps the key of one item to the value of another, independent item. So no learned rule can guess the value from the key. We then subtract the score of a memory that received other pairs, which isolates the content specific to each memory.

From 4,000 training steps on, the unit rule stores about three times fewer pairs, at every checkpoint (8 seeds per rule, p ≤ 0.002). On a fresh memory, with no prior stream, the unit rule stores at least as much. The cost of collapse therefore appears on long streams. We will add this measurement in the revision and rephrase contribution 3.

5. The stability condition

The paper contains two different results, and the revision separates them.

The design rule (Lemma 1). Bound the size of each write, and keep the forgetting rate above a positive floor. Then the weights of the memory stay bounded at every step, whatever the gradients. This result holds for the actual memory, with no approximation. It requires no small write step. So the rule does not restrain the memory.

The numerical threshold (Corollary 2). This is a different result. It gives a maximal value of the write step. To obtain it, we replace the memory by a quadratic approximation, and we bound its curvature in the worst case. This bound is very conservative. It exceeds the measured curvature by a factor of 10 to 100. The step it gives is therefore 10 to 100 times too small, and exceeding it does not imply divergence. Our models exceed it without diverging. We agree that this threshold is not a practical recipe.

Our own models. It is true that we do not apply the rule ourselves. On our testbed, the forgetting rate and the write step are learned freely, with no floor and no cap. In practice the writes stay small; nothing enforces it. The forgetting rate sometimes falls very close to zero.

What changes. In the revision, "a condition that a practitioner can check" becomes "a design rule, enforced by construction". The step margin of Corollary 2 leaves the main text; the corollary stays in the appendix as a conservative diagnostic. We also state which models apply the rule, and which only in practice.

6. Presentation

We agree. We are rewriting the paper to make it more readable, without radically changing the content and conclusions. The structure changes, and we add new measurements. The testbed, that is, our small video prediction model on EgoLife and its data, is now described in the main text, in Section 3. A table lists the write rules, with one symbol per role. Thus ψ no longer denotes both the bound and the coupling. We also replace our home-made vocabulary: an "arm" becomes "the model trained with rule X", a "chimera" becomes "a hybrid model" (a component of one model grafted onto another). The rewritten version will be online in the coming days, with a list of changes.

I thank the authors for their detailed response, especially for running the enwik8 experiment I requested and toning down the results statements. I find the divergence analysis useful in that it points to capping the write size, rather than orthogonalisation, as what prevents divergence. That being said, a main concern remains: collapsed and healthy memories are compared only on tasks where memory isn’t shown to help to begin with. on associative recall, where it does help, both unit and their usq collapse. As Reviewer 7L3E also notes, collapse may simply reflect the model bypassing a module it doesn’t need, so given that the cost of collapse rests only on a synthetic storage probe, I am maintaining my score of 2. A will add though, that if the other reviewers feel strongly about the divergence analysis being useful enough on its own to justify acceptance, i will not oppose.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7L3E : does collapse matter, seeds, larger chunks

We thank the reviewer for this summary and for two precise questions. We answer them with measurements. We also state what we could not establish.

1. Does collapse matter? (W1, Q1)

Your reading is plausible on our testbed. The memory does not help the training loss, and collapse may be the model learning to bypass it. What we measured is that when collapse happens, the memory loses storage capacity, and the training loss does not show it.

We measured it with the following test. We replay a stream of 100 clips through the trained model. We then write random pairs into the memory, with its learned gates, and query each key. Since the pairs are random, only the memory can retain them. We subtract the score of a memory that received other pairs, so as to count only what was written. From 4,000 training steps on, the collapsed model stores about three times fewer pairs than the healthy one. On a fresh memory, with no prior stream, the collapsed model stores at least as much. The gap therefore appears only after a long stream, and it concerns the content specific to each memory. It matters for the intended use: a model that keeps a prediction objective and can, at the same time, retain information over the long term. Storage capacity measures this second function.

On Q1. We measured the gates of the associative recall models of Appendix F. Their state does collapse, in the sense of the paper: its participation ratio falls from 5 to 24 at the start to 1 to 9 at the end of a sequence. This holds for both rules. The momentum gate, however, stays alive (medians 0.46 and 0.73). Forgetting is strong (medians 0.58 and 0.39 per write). The state therefore keeps mostly its last few writes. On this task, forgetting collapses the state, not momentum. Forcing momentum back on is therefore not the right lever there. This is consistent with the tie between the two rules. Likewise, on the community implementation, pinning momentum for all of training does not prevent collapse (two seeds per pinned value).

On this storage task, the memory is indispensable, and collapse costs a factor of three. What we do not have is a task learned by the model itself, where it needs its memory to succeed, and where collapse costs it accuracy. We will clarify this point in the revision.

2. Scale and seeds (W2)

We agree on scale. We do not aim at a large model. We study a failure mode on models small enough to train many seeds and to replay every write.

Convergence does not change the two phenomena. Divergence happens within a single forward pass, early in training. Collapse persists as training continues: at 30,000 steps, the model trained with the unit rule stays collapsed from step 4,000 on (two seeds).

We raised the seed counts of the claims you point to.

ClaimBeforeNow
Rank collapse during training (Fig. 4a)28 per rule
enwik8 controls (community implementation)1 to 33
Pinned momentum on the community implementation12 per value
Divergence ladder (Fig. 2a)23 (5 at the boundary rung)

Claims that remain at one or two seeds are flagged as such.

3. Larger model, realistic chunks (Q2)

We did not test this scale, and the question is interesting. Two facts bear on it. First, collapse does not need small chunks. The community implementation freezes the memory weights over 128 tokens. Its state still collapses: the participation ratio falls from 63.7 to 1 to 3. This holds on 12 of 12 blocks, over three seeds. Second, collapse does not need the momentum gate everywhere. On our testbed, collapse goes through it. However, we pinned momentum on the community implementation, and the model still collapsed. On associative recall, momentum stays alive while the state concentrates. If the implementation sums a chunk of 2,048 tokens into a single write, that write can already have a high rank, with or without momentum. We would then not expect the momentum route, but another route remains possible. Testing this needs a larger model; we list it as future work.

AuthorsOpenReview ↗

Revised PDF : what changed

We thank the three reviewers. Their reports converged on two points: the paper was hard to read, and it did not show whether collapse matters. We have uploaded a revised PDF that intends to address both.

What is new in substance. Section 5.3 and Figure 5 add a storage test after a long stream, on 8 seeds per rule: a collapsed memory stores three times fewer key-value pairs than a memory whose state stayed diverse, and the difference appears only after a long stream. Figure 3 extends the collapse result from 2 to 8 seeds per rule. Appendix A measures the learned gates token by token and gives a windowed variant of Lemma 1 that needs no per-token forgetting floor. Appendix C tests, and rejects, one hypothesis on why training keeps the momentum under the usq rule.

What is corrected. Some statements of the first version were inexact or overstated. In particular, the rank-capped recall result ("0.47 against 0.19") is withdrawn, the enwik8 evaluation is redone at equal learning rate, and the community retraining is now described as slowing collapse without preventing it.

What changed in form. The text is entirely rewritten. Section 3 states three hypotheses that Section 5 tests one by one; Section 4 describes the experimental protocol. Figures and appendices are renumbered.

Summary of changes.

  • Section 5.3, Figure 5: storage test after a long stream, 8 seeds per rule, with fresh-memory and reset-buffer controls (new).
  • Figure 3: collapse during training, 8 seeds per rule instead of 2.
  • Appendix A: Lemma 2 stated; windowed variant of Lemma 1; Table 3 of the measured gates.
  • Appendix C: hypothesis on the momentum under usq, tested on 16 seeds and rejected; community retraining described as slowing, not preventing, collapse.
  • Appendix D: enwik8 re-evaluated at equal learning rate; the gap between models comes from the trained weights, not from the test-time writes.
  • Withdrawn: the rank-capped recall result, the "ψc, n = 1" point of the retention figure, Nazari's bound appendix, the chunk-factor appendix.
  • Figure 2: the green curves replayed with their training rule (the conclusion holds); the unit rule added.
  • Numbering: Figures 2, 3, 4a and 5 of the first version are now Figures 1, 2, 3 and 4; Appendices E, F, G are now D, E, F; Corollary 2 is Corollary 1; the "ψc rule" of our responses is the usq rule of Table 1.
  • Where the reviewers' main points are addressed: 4enc, Section 6 and Appendix A (the α floor), Appendix C (why usq keeps its momentum), Appendix B (the retention probe); 7L3E, Sections 5.3 and 6 (whether collapse matters, chunk sizes); UXw2, the whole text, Sections 2.2 and 3 (writing and structure).

Update (15 Nov): the PDF has been re-uploaded. Sections 3 and 4 are swapped so that the theory now precedes the experimental protocol, and Section 2.1 states up front what is taken from Titans. The results are unchanged.

Dear Reviewers,

The authors have responded three days ago and haven't received any feedback since then. Please, read the responses, say whether your concerns were addressed and when appropriate raise your score.

Thank you for the timely response. My concern is mostly addressed by the more seed counts provided by the authors in the rebuttal. Also, the new storage test shows that collapsed memories keep about three times fewer pairs than uncollapsed ones, after a long stream. The first point that remains open is a learned task where collapse costs accuracy. The second one is larger model with realistic chunk sizes. I maintain my positive score of 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.