acceptodds
Under review as a conference paper at ICLR 2027

Toaster: Nonlinear Memory from Cascaded Linear Memories and Target Propagation

Abstract

Recurrent memories let language models process long sequences with a fixed-size state. Their design must balance the richness of the stored representation with the cost of updating it. Linear memories such as DeltaNet and KDA store one matrix and take one delta-rule step per token. A chunk of such steps has an exact WY form, so they train fast. On the other hand, deep memories such as TTT and Titans store a small MLP, which can hold maps that no matrix can. However, their update has no WY form. Toaster bridges the two with one fact: a gradient step on a linear layer is a delta-rule write. It builds a two-layer memory from two linear memories with an activation between them. Target propagation gives the hidden memory its values by projecting the output error back through the output memory. Given a chunk's cross-layer signals, each layer can then run on an unmodified delta-rule kernel. Our default runs this kernel on the output layer and updates the hidden layer once per chunk. Running it on both layers gives the best model in our ablation. At matched backbones and training-token budgets, Toaster has lower loss than KDA at all five tested sizes (0.2B–1.6B non-embedding parameters) on both C4 and PG19. Its C4 perplexity is also slightly lower than the attention baseline at every size. At the largest backbone, its mean accuracy on 13 three-shot tasks is , compared with for KDA. In length extrapolation, its loss stays below KDA's up to 20K tokens ( the training length). On the 145M backbone, at a hidden width matched to Titans' memory, Toaster takes at most approximately KDA's training-step time and is – faster than the tested Titans configurations at 32K context.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Online Optimization, Sequence Modeling, Attention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pfJ4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TOASTER, a novel recurrent memory which aims to combine the expressivity of nonlinear memories (such as TTT and Titans) with the fast parallel training of linear memories (like DeltaNet and KDA). The latter can rely on a WY representation of the delta-rule corrections to compute chunkwise updates efficiently, by inverting a low-tri linear system, but nonlinear memories don’t generally allow for this, precisely due to the nonlinearity, which forces instead sequential and/or mini-batch updates. TOASTER addresses this by approximating the gradient propagations through the MLP they represent, following a strategy rooted in target propagation. The approximation computes these gradients from a frozen state at the beginning of each chunk, which allows one to reuse classical WY-form-based kernels for approximately recovering the required state updates. Pretrained on C4 at five scales (up to 1.6B), TOASTER is reported to reach lower loss than KDA at every size and slightly lower C4 perplexity than attention, at a training cost within 1.33x of KDA's.

Critical strengths
  • The core observation that target propagation allows to recycle delta-rule kernels is a nice catch
  • The empirical study is quite extensive (with some caveats, see below)
  • The paper is unusually candid about its own limitations (with more caveats, see below)
Critical weaknesses
  • W1: The overall claim that “TOASTER resolves an apparent conflict between two families of recurrent memory” is overstated: the core approach relies on an approximation (a single Jacobi step of the nonlinear system defining the delta correction), whose impact is not properly measured
  • W2: The paper's own ablations suggest that target propagation, its core idea, contributes very little; most of the gain over KDA comes from elsewhere.
  • W3: The comparison with linear memories and attention is dirtied by confounders: gains are generally small, tend to shrink with scale and vanish when actually matching compute, no multiple-seed results are reported, and linear baselines employ hidden states of 8x different size.
Questions for authors

W1: Impact of approximation The main motivation behind this paper is to overcome the mini-batch updates forced by nonlinear memories, as they force sequentiality less one loses accuracy (L73-76). However, I do not believe TOASTER really escapes this problem: it merely side-steps it, and it’s unclear with what effect. At its core, TOASTER still relies on a nonlinear recurrence: as such, a gradient step on it can then be represented via a delta-rule update only formally, since the actual “value” (z-g) depends on the state itself, which evolves along the chunk. Computing a chunk of such steps exactly still requires solving a nonlinear system. TOASTER side-steps this by freezing the cross-layer signals at the chunk-initial states, which turns each layer’s update into an ordinary delta-rule recurrence that the WY kernel can solves efficiently. But this is only an approximation of the true update: the real non-linear system governing it is effectively being solved in an approximate fashion, via a single Jacobi/Picard iteration, initialised with the states at the start of each chunk (notice this is similar in spirit to using “stale state updates” for TTT, although the procedure is different). Now, this is not at all an issue per-se, provided that the approximation is a valid one. At the very least, however, I would’ve expected that: (i) this was clearly recognised and discussed (this is mentioned in passing in L242, but not properly discussed), and more openly stated in the paper, adequately tuning down the claim that TOASTER resolves the nonlinearity issue of TTT; (ii) the effect of this approximation was actually measured, by checking error wrt the real solution recovered sequentially: is one Jacobi iteration enough? Moreover (iii) with this approximation, the chunk-size CC becomes an inherent design choice for the model, which has the potential to affect the state evolution: the approximation is exact for C=1C=1 (as you correctly point out), but progressively grows with CC, consequently drifting from the “true” state evolution, and an investigation of this effect would also be relevant (since it creates a potential mismatch between training and decoding configurations). Notice this is not the case for KDA: varying CC does not vary the final StSt (up to rounding-off errors). Finally, (iv) the impact of this approximation may compound even worse for deeper memories (which are not studied in the paper), which again might hinder the generalisability of the method.

W2: What drives the gains Another concern comes from the ablations in Tab12: if I’m understanding them correctly, they suggest that target propagation (the main strategy implemented by the paper) actually plays a minor role in boosting performance. In fact, even with γ=0\gamma=0 (for which the method collapses to a delta-rule memory over a learned nonlinear feature map, L1408-1410), TOASTER reaches 15.71 agains a 15.69 for the default. In other words, the effect from the TTT-style update accounts only for ~0.02 of the 0.31 PPL gain over KDA (which I would expect to be within seed-to-seed variation, although such variation is not reported). The reported WY run confounds the effects of γ\gamma and the different update strategy. A WY-kernel run with γ=0\gamma=0 would then be useful to properly attribute this gain to the error signal rather than to the update rule. Similarly, using only identity as activation (whose read is a linear map of rank at most dkd_k, like in KDA), already recovers 0.20 of the 0.31 gain. You attribute part of this to the bias, the learned initial states and the gate scales (L1404-1406), and KDA was not retuned with TOASTER's gate scales (L482). So it is unclear how much of the gain over KDA is due to the cascade itself. Finally, since each coordinate of the hidden correction is bounded by about 1.13γ∼0.0111.13\gamma\sim 0.011 (L278-279), I wonder how far StSt actually moves from its learned initial state over a sequence. Did you measure this? If it barely moves, TOASTER is essentially KDA with a learned nonlinear expansion of the keys, and the natural baseline would be KDA with such a feature map of the same width.

W3 Baselines comparison The main concern in this sense is that the comparison against KDA is not on equal grounds: while you’re matching parameters (although it’s unclear how closely, given what you write in L1112), hidden state sizes of TOASTER vs KDA vary by a factor of 8x8x (L1100). Considering how hidden state size directly relates to an RNN’s capacity, this is not a fair comparison. Moreover, gains tend to shrink with model size, and TOASTER has the smallest fitted exponent in Tab6. The fact that in Fig2 (mid) you’re not considering the sequence mixer in your FLOPs calculation (and probably parameter counts, again for L1112), makes me wonder how I should really interpret the results there. Also downstream and retrieval evidence is not conclusive, 6/13 tasks are at chance (which is however expected at this scale), and results on retrieval are not particularly convincing given my comment on the KDA state size considered. To your credit, you are explicit about some of these limitations, but I would’ve adopted a more sober tone throughout the paper, then.

Minor:

  • L14: “A chunk of such steps has an exact WY form” as it’s written, this sentence doesn’t make sense: expand
  • L18: “Target propagation gives the hidden memory its values” it gives them an approximation (in general)
  • L19-20: this too is really hard to parse
  • L35: “pays for it exactness” Exactness of what?
  • L38: “because the state is bounded … how cheaply it can be trained”: how does the fact that the state is bounded impacts how cheaply it can be trained? Cheap training is a property one seeks regardless
  • L46: ”Products of such factors admit the compact WY representation” this is a bit better than L14…
  • L49: “their memory is limited to linear mappings and hence its capacity is limited” Arguably, capacity has more to do with the boundedness of the state size. I’d say expressivity is limited by the linearity constraint
  • L87: “whatever residual is presented as the value” this too reads odd. A delta rule removes what was previously stored in the “slot” of St associated with kt, and replaces it with vt.
  • L90: “so such a layer can be trained by an unmodified delta-rule kernel” not true in general: the Jacobian of the loss can mess up the structure of the delta update computation
  • Tab1 is rather redundant wrt Fig1
  • L111: “we describe three design choices...”: please provide a clearer high-level description of them
  • L146: “and the gates are passed alongside the keys” explain
  • L154: “the capacity is set by the width of the network rather than by dkdvd_k d_v” true, but the width of the network is still related to the size of the hidden state, as that defines the weights of the MLP.
  • L165: “recovering the exact WY form layer by layer at the price of computing the signals that pass between layers once per chunk” make this clearer: the real price you’re paying is that you’re approximating this signal
  • Prop1 is really a tautology, so should be presented as a remark rather than an actual proof. In words, it reads “If we manage to get the target exactly right, so that it matches the actual gradient, then updating the way we do is equivalent to performing backprop and TTT”. The crux of it all, is that in general (C>1) this exact target is not available.
  • L242: “The approximations are that … sums its C writes as in Eq. (3)” Precisely, that’s the point: you’re applying an approximation, because the true gradient (depending on S_{t-1}) is not available. Your method is then not equivalent to TTT: it’s an approximation that makes its application faster
  • L262: “Linear memories are bias-free outer products, and a bias adds nothing to them” I honestly don’t understand what this means
3 comments
AuthorsOpenReview ↗

Response to Reviewer pfJ4 (1/3): W1, the approximation

Thank you for your detailed and constructive feedback. In response to your comments, we have revised the paper and added additional experiments to address your questions. Thanks to your feedback, we believe these changes have strengthened the manuscript. Please find our detailed responses below, in three parts (W1; W2; W3 and minor points). Tables R1 and R2 are in the general response, and table numbers refer to the revised paper.

\quad

W1: The overall claim that "TOASTER resolves an apparent conflict between two families of recurrent memory" is overstated: the core approach relies on an approximation (a single Jacobi step of the nonlinear system defining the delta correction), whose impact is not properly measured

We agree with this characterization. Within a chunk, the exact per-token recurrence couples the two layers nonlinearly. TOASTER evaluates the cross-layer signals (hidden keys and targets) once, at the chunk-start states. Given these signals, the output layer is advanced by an exact linear-memory kernel; the hidden layer takes a summed chunk-start step by default (as in Eq. 3), or the kernel in the both-layers variant. Across layers this is one Jacobi/Picard sweep, exact only for C=1C = 1. The submission said this in Section 3.3 and Appendix D, but too briefly, and several sentences in the main text claimed more.

(i) Framing. We revised the abstract, introduction and conclusion. The paper no longer says that TOASTER "resolves" the conflict. It says that a layer's update becomes executable by an exact kernel once the cross-layer signals are given, and that approximating these signals once per chunk is the price. Section 3.3 now discusses this approximation directly, and Proposition 1 is presented as a remark.

(ii) Is one sweep enough? We measured it directly on a 145M model with the default kernel on both layers, where C′=1C' = 1 is the exact coupled per-token recurrence that the chunked algorithm approximates (Table R3, top). Evaluating the C=32C = 32-trained model across decoding chunk sizes C′∈1,4,8,16,32,64C' \in \\{1, 4, 8, 16, 32, 64\\}, C4 perplexity moves smoothly from 15.97 at C′=1C' = 1 to 15.77 at the training chunk size C′=32C' = 32 (22.73 to 22.33 on PG19) and 15.79 at C′=64C' = 64—staying within 0.20 of C′=1C' = 1 across the full range and within 0.02 across C′∈[16,64]C' \in [16, 64]—while the hidden-memory states of the first block differ from C′=1C' = 1 by only 1.7% in relative Frobenius norm after 4K tokens. One sweep is enough in practice.

(iii) Chunk size as a design choice. You are right that, unlike in KDA, CC is part of the model, as it is in Titans, LaCT or TNT. One clarification: there is no mismatch between training and decoding as long as decoding uses the training value of CC. Decoding runs the same chunked recurrence, and it can be executed token by token with the same results up to rounding, because each token only needs the chunk-start states and the within-chunk kernel recurrence. The paper now states this explicitly. To see how the choice of CC affects quality, we also trained 145M TOASTER across C∈8,16,32,64,128,256C \in \\{8, 16, 32, 64, 128, 256\\} (Table R3, bottom): smaller chunks refresh the cross-layer targets more frequently and improve C4 perplexity to 15.59 (C=8C = 8) and 15.56 (C=16C = 16) vs. 15.69 at default C=32C = 32, C∈[32,128]C \in [32, 128] stays at 15.67–15.69, and even C=256C = 256 reaches 15.74 (0.26–0.44 below KDA's 16.00 across the entire 32× range, with C=32C = 32 balancing throughput and quality).

Table R3. Effect of the chunk size (C4 perplexity).

Evaluated with chunk size C′C' (trained with C=32C = 32)148163264
Default kernel on both layers, 145M15.9715.9015.8415.7915.7715.79
Trained with chunk size CC (default, 145M)8163264128256
C4 ppl15.5915.5615.6915.6715.6915.74

(iv) Deeper memories. We agree that the approximation is applied once per hidden layer and may compound with depth. We have not trained deeper cascades and make no claim about them; the limitations say this directly.

AuthorsOpenReview ↗

Response to Reviewer pfJ4 (2/3): W2, what drives the gains

\quad

W2: The paper's own ablations suggest that target propagation, its core idea, contributes very little; most of the gain over KDA comes from elsewhere.

Your comment on Table 12 (Table 13 in the revision) pinpointed an important nuance. As noted in our original paper, setting γ=0\gamma = 0 (setting the inner error learning rate to 00 at 145M: 15.71 vs. 15.69) does not make S(1)S^{(1)} truly static, because the input-dependent weight-decay gate αt(1)\alpha^{(1)}_t still updates S(1)S^{(1)} at every chunk. Within rebuttal compute limits, we ran a new 145M ablation that completely freezes S(1)S^{(1)} (St(1)≡S0(1)S^{(1)}_t \equiv S^{(1)}_0, 0% drift) and compared it directly with our Table 13 results.

Completely frozen S(1)S^{(1)} vs. active S(1)S^{(1)} at 145M. At the exact same architecture and 8×8\times state size (Table R1 in the general response), completely freezing S(1)S^{(1)} yields 15.86 C4 perplexity, compared with 16.00 for KDA, 15.71 when γ=0\gamma = 0 (weight decay still active on S(1)S^{(1)}), and 15.69 with full target propagation. Thus a fixed nonlinear feature map accounts for 0.14 of TOASTER's 0.31 gain over KDA, whereas adapting S(1)S^{(1)} in context contributes 0.17—more than half of the total improvement—with input-dependent decay reaching 15.71 and the cross-layer error signal taking it to 15.69.

How far does S(1)S^{(1)} move? Over a 4K sequence, S(1)S^{(1)} actively changes from its learned initial Frobenius norm by 4.7% at 145M and 3.4% at 1.2B under 32-step chunk-averaged updates, and by 31.5% (up to 68.3% per head) with the default kernel on both layers (the model of Table R3), while S(2)S^{(2)} changes by 53–60%. Freezing S(1)S^{(1)} at its initial state at inference (St(1)≡S0(1)S^{(1)}_t \equiv S^{(1)}_0) already degrades this model's perplexity by +0.39+0.39 on C4 (15.77→16.1615.77 \to 16.16) and +0.94+0.94 on PG19 (22.33→23.2822.33 \to 23.28); under 32-step chunk averaging S(2)S^{(2)} largely absorbs inference-only freezing (+0.001+0.001/+0.003+0.003 on C4/PG19), whereas retraining from scratch with S(1)S^{(1)} completely frozen degrades 145M C4 perplexity by +0.17+0.17 (15.69→15.8615.69 \to 15.86; Table R1).

KDA with a feature map. A KDA with an added feature map that expands its keys is no longer KDA: it is a different memory with a larger state. Because the γ=0\gamma = 0 row in Table 13 (15.71) still adapted S(1)S^{(1)} via input-dependent weight decay, the exact implementation of a delta-rule memory over a fixed learned nonlinear feature map is the completely frozen S(1)S^{(1)} row in Table R1 (15.86), cleanly separating static feature-map capacity (0.14 PPL) from in-context S(1)S^{(1)} adaptation (0.17 PPL).

Identity row and gate scales. You are right that the identity row differs from KDA in the bias, the learned initial state and the gate scales, so its 0.20 cannot be attributed to the cascade alone; we reworded this. KDA uses its own gate scales, and TOASTER's smaller scales were tuned for the cascade. We did not retune KDA with them, as the limitations state.

The main benefit of TOASTER is that a wide nonlinear memory becomes trainable with a linear-memory kernel, at 1.25–1.48× KDA's step time. Target propagation is what allows the hidden layer to learn in context within this construction, and its measured contribution is the one in Table R1, which the revised paper reports in Section 4 and Appendix I (Table 14).

AuthorsOpenReview ↗

Response to Reviewer pfJ4 (3/3): W3, baselines; minor points

\quad

W3: The comparison with linear memories and attention is dirtied by confounders: gains are generally small, tend to shrink with scale and vanish when actually matching compute, no multiple-seed results are reported, and linear baselines employ hidden states of 8x different size.

State size. TOASTER holds 8× the state of KDA. We see this as what the architecture provides rather than as a confound. KDA itself has no state expansion: its state is tied to the head dimension, and enlarging the memory of a recurrent model is not trivial. Comparing the two at the same backbone is in the same spirit as comparing a matrix-state memory such as Gated DeltaNet with a vector-state recurrent network such as an LSTM: one does not ask the LSTM to carry a matrix-sized state, because the larger state is part of what the newer architecture contributes. That said, we ran the ablation (Table R2 in the general response): giving KDA 2×, 4× and 8× the total state by trading heads for wider heads at fixed H⋅dk=D=1024H \cdot d_k = D = 1024 (8 / 4 / 2 heads of width 128 / 256 / 512, keeping all D×DD \times D Q/K/V/gate/O projection parameter counts unchanged) yields 15.96 / 15.84 / 15.79, remaining 0.06–0.10 above TOASTER's 15.90, 15.76 and 15.69 at every matched state size at 145M. Enlarging the state closes about two-thirds of the 16.00-to-15.69 gap by collapsing 16 heads down to two wide heads, whereas TOASTER scales dhd_h with 16 heads of width 64 fixed.

Parameters and compute. KDA and TOASTER are closely matched in parameters: they share the backbone at every size (Table 6) and differ only in a small number of mixer parameters. Figure 2 uses the standard 6ND estimate, which leaves out the operations of every sequence mixer. Table 8 counts them. Against attention, whose quadratic term is the largest mixer cost at 4K, TOASTER's gain rises (1.22 at 145M and 1.06 at 1.2B on C4). Against KDA, the two are at parity (0.99–1.03): at our sizes, the gain over KDA is about what the extra kernel arithmetic would buy. The paper says so in Section 4 and in the limitations.

Scale. You are right that the gap to KDA narrows with size (0.020 to 0.006 nats on C4) and that TOASTER has the smallest fitted exponent. The gap is roughly stable over our two largest sizes, and TOASTER is better than KDA per parameter and per training token in all ten settings. In the setting in which recurrent memories are deployed, a 3:1 hybrid with global attention at 1.2B, TOASTER improves on the KDA hybrid by 0.04 (C4) and 0.24 (PG19) perplexity, and on attention by 0.18 and 0.44 (Table 3). Compared with Titans, the other nonlinear memory, TOASTER trains 2.7–5.2× faster at 32K and reaches lower perplexity at 1.2B.

Seeds. All runs are single runs: we could not afford to repeat the ladder, and the limitations say so. We rest the evidence on the consistency of the ordering. TOASTER is below KDA in all ten settings of the ladder, and again in the new hybrid runs at 1.2B. On the 20K evaluation, the per-window standard error of the difference between TOASTER and KDA is about 0.001 nats, against differences of 0.008–0.014 nats (Table 12). This is evaluation noise, not seed-to-seed variance, which we cannot estimate.

Downstream and retrieval. We agree that this evidence is not conclusive, and the limitations say so. In particular, we have not run the retrieval tests against a KDA with wider heads, so that comparison remains open.

\quad

Minor: L14, L18, L19-20, L35, L38, L46, L49, L87, L90, Tab1, L111, L146, L154, L165, Prop1, L242, L262

Thank you for the careful reading. We adopted all of these points. Briefly: L14 now says what has a WY form (the product of a chunk's steps), as L46 already did; L19–20 now say that the cross-layer signals, which include the targets of L18, are approximated once per chunk; L35 and L38 are reworded; L49 and L154 use "expressivity" and relate the width to the state size; L87 and L90 state that the delta rule replaces what is stored under a key, and that the kernel applies only once the error signal is given; L111 is split into short sentences that name the three design choices; L146 explains how the gates enter the kernel; L165 and L242 name the approximation as the price; Proposition 1 is a remark; and L262 now says that a constant input coordinate matters in the cascade because it shifts the pre-activations of the hidden layer. Table 1 will move to the appendix in the final version; we left it in place for now so that the table numbers stay stable during the discussion.

We hope the new measurements and the more sober presentation address your concerns. We would be glad to clarify anything further.

Reviewer_GjVWOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to combine the expressive capacity of nonlinear recurrent memories with the efficient chunkwise training of linear memories. It proposes Toaster, which cascades two linear memories with an activation function and uses target propagation to construct hidden-layer targets, allowing each layer to use a WY-form update conditioned on the chunk’s cross-layer signals. Across five model sizes, Toaster achieves lower language-modeling loss than KDA on C4 and PG19 and improves average accuracy on 13 downstream tasks at the largest scale. In a separate timing configuration with memory width matched to Titans, Toaster takes approximately 1.33× KDA’s training-step time and is 2.8–5.5× faster than the tested Titans configurations at 32K context.

Critical strengths
  • This paper introduces a nonlinear memory mechanism whose constituent linear memories admit WY-form updates, enabling efficient pretraining. Experimental results show better downstream performance than KDA. In the timing experiments with a narrower memory, its training-step time is only approximately 1.3× that of KDA.
Critical weaknesses
  • The use of two linear memories connected by a nonlinearity has precedents in ABC [1], GSA [2], and the recent Raven [3], which use related two-state memory structures with softmax-based reading. GSA explicitly uses two linear-attention layers connected by softmax and supports chunkwise computation. Although Appendix B distinguishes Toaster’s coupled updates through target propagation, the empirical contribution of this coupling remains unclear: removing the hidden-layer error signal changes perplexity only from 15.69 to 15.71 in Table 12.
Questions for authors
  1. Why is Titans not included in the downstream performance experiments? Although the paper compares perplexity and training speed against Titans, a downstream comparison under matched training budgets would help assess the gap between these two MLP-based memory mechanisms.
  2. The ablations in Table 4 are too limited. Since the hidden-state width dhd_h, analogous to the number of memory slots in GSA, is important for retrieval performance, could the authors provide more detailed performance and speed evaluations—both training and inference—across a wider range of dhd_h? Could the authors also evaluate more activation functions, particularly softmax, to clarify the contribution of the chosen activation?
  3. Appendix C describes the extension to more memory layers. It would be interesting to evaluate how performance and computational cost scale with the number of layers in the memory module.

[1] Hao Peng et al. ABC: Attention with Bounded-memory Control. ACL, 2022.

[2] Yu Zhang et al. Gated Slot Attention for Efficient Linear-Time Sequence Modeling. NeurIPS, 2024.

[3] Arshia Afzal et al. Raven: High-Recall Sequence Modeling with Sparse Memory Routing. arXiv, 2026.

6 comments
AuthorsOpenReview ↗

Response to Reviewer GjVW (1/2): relation to ABC, GSA and Raven; Titans downstream

Thank you for your detailed and constructive feedback. In response to your comments, we have revised the paper and added additional experiments to address your questions. Thanks to your feedback, we believe these changes have strengthened the manuscript. Please find our detailed responses below. Table R1 is in the general response, and table numbers refer to the revised paper.

\quad

The use of two linear memories connected by a nonlinearity has precedents in ABC [1], GSA [2], and the recent Raven [3] ... the empirical contribution of this coupling remains unclear: removing the hidden-layer error signal changes perplexity only from 15.69 to 15.71 in Table 12.

Thank you for the pointer to Raven, which we now discuss together with ABC and GSA in the related work. We agree that these are the closest precedents for placing a nonlinearity between two linear memories. The difference lies in how the first memory is written. In ABC, GSA and Raven, the memories are written from the input (through gating in GSA and a learned sparse router in Raven) and read through a softmax over slots; no error signal from the output reaches the first memory. In TOASTER the two writes are coupled: the output memory is keyed by the hidden activations, and the hidden memory is written toward a target obtained by projecting the output error back through the output memory. With the exact target, the two writes together are one gradient step on the composite two-layer memory (Section 3.2).

You are right that the submission did not show clearly enough how S(1)S^{(1)} adapts in context. Over 4K tokens, TOASTER's first layer S(1)S^{(1)} actively evolves from its initial state by 4.7% at 145M and 3.4% at 1.2B. Moreover, as noted in our original paper, setting γ=0\gamma = 0 (setting the inner error learning rate to 00 in Table 12 / Table 13 in the revision: 15.71 vs. 15.69) does not make S(1)S^{(1)} truly static, because input-dependent weight decay αt(1)\alpha^{(1)}_t still updates S(1)S^{(1)} at every chunk. In a new 145M ablation that completely freezes S(1)S^{(1)} (St(1)≡S0(1)S^{(1)}_t \equiv S^{(1)}_0; Table R1), C4 perplexity rises to 15.86 (vs. 16.00 for KDA, 15.71 for γ=0\gamma = 0, and 15.69 for full target propagation): static nonlinear capacity gives 0.14 of the 0.31 gain over KDA, while in-context S(1)S^{(1)} adaptation contributes 0.17 (more than half of the total gain).

\quad

Why is Titans not included in the downstream performance experiments? ... a downstream comparison under matched training budgets would help assess the gap between these two MLP-based memory mechanisms.

In the submission, Titans was trained only at 145M (Figure 5), because it is the most expensive of the models to train: at its best chunk size, C = 16, a training step takes 1.7× as long as TOASTER's (Table 9). We evaluate downstream tasks only at 1.2B, because we do not consider the three-shot accuracies of 145M models a reliable indicator (even at 1.2B, six of the 13 tasks are at or near chance). We have now trained Titans at 1.2B under the same training budget, with hidden width 64 and chunk size 32 (Appendix G.4), a setting in which its training step is comparable to TOASTER's (Table 9). We trained it with all layers replaced and in a 3:1 hybrid with global attention (one attention layer after every three mixer layers), and added both to the downstream comparison (Table 3):

1.2BC4 pplPG19 ppl13-task avg. (%)
All layers replaced: KDA10.0013.3750.52
All layers replaced: Titans10.0913.4850.04
All layers replaced: TOASTER9.9413.2951.84
3:1 hybrid: KDA9.8212.9950.85
3:1 hybrid: Titans9.9013.0850.55
3:1 hybrid: TOASTER9.7812.7552.11

TOASTER is ahead of Titans in both settings, in perplexity and in downstream accuracy. With all layers replaced, Titans is 2–6 points below TOASTER on ARC-Easy, HellaSwag and PIQA. As for all models of this size, several downstream tasks are near chance, so we regard the perplexities as the more reliable part of this comparison.

AuthorsOpenReview ↗

Response to Reviewer GjVW (2/2): hidden width, activation, and depth

\quad

The ablations in Table 4 are too limited. ... could the authors provide more detailed performance and speed evaluations—both training and inference—across a wider range of (dhd_h)? Could the authors also evaluate more activation functions, particularly softmax ...?

Table 4 of the submission (Table 5 in the revision) is only a summary: the full ablation (Table 13, Appendix G.9) covers four widths and five activations. We extended it as you suggested, with a wider memory and a softmax activation.

Table R4. Hidden width and activation (145M, C4 perplexity; KDA is at 16.00).

Hidden width dhd_h64128192256 (default)512
State per head (relative to KDA)2×4×6×8×16×
C4 ppl15.9015.7615.7215.6915.62
ActivationidentitytanhSiLUleaky ReLUGELU (default)softmax
C4 ppl15.8015.8215.7915.7115.6918.19

Perplexity improves steadily with the width, reaching 15.62 at dh=512d_h = 512. With a softmax between the two memories, as in GSA, the perplexity is 18.19, 2.50 above GELU. Our reading is that a softmax restricts the hidden activations to a distribution over the hidden units, which suits slot selection but leaves the output memory less room than an unconstrained activation.

For speed, Table 9 gives training-step times at dh=64d_h = 64 and dh=256d_h = 256: 1,023 and 1,223 ms at a 4K context, against 905 ms for KDA. Across context lengths from 2K to 32K, the four times wider memory takes 7–20% longer per step. Inference runs the same chunked recurrence as training, so its cost scales in the same way; we have not timed it separately. We also have not measured retrieval as a function of the width: the needle tests are run on the 1.2B models, for which we trained a single width.

\quad

Appendix C describes the extension to more memory layers. It would be interesting to evaluate how performance and computational cost scale with the number of layers in the memory module.

We agree that this is an interesting direction. In terms of cost, a cascade of N layers needs at most N kernel calls and O(N) dense products per chunk (Appendix C). In terms of quality, we have not trained deeper cascades. Each extra hidden layer also adds one more level of the chunk-start approximation, and a proper study has to revisit the step sizes and the stability of every layer, which we could not complete during the rebuttal. We therefore leave it to future work, and the limitations say so.

We hope these additions address your concerns, and we would be glad to answer further questions.

Thank you for the additional experiments and the clarifications regarding the chunk-start approximation. After carefully considering the Reviewer pfJ4’s concerns and the authors’ responses, I believe that the paper’s central contribution requires a substantial reframing. The method does not provide exact chunkwise parallelization of the coupled nonlinear memory dynamics. Instead, it approximates the cross-layer signals using the chunk-initial states, allowing the resulting conditional, per-layer updates to use existing linear-memory kernels. Although the revision now acknowledges this distinction, I believe it should consistently guide the motivation, technical claims, and conclusions throughout the paper. The contribution should be presented as an approximate update scheme for nonlinear memory, with its benefits and limitations evaluated accordingly. The new comparison with KDA at matched recurrent-state sizes also substantially reduces the empirical advantage. At 145M, increasing KDA’s state to match the default TOASTER configuration reduces the C4 perplexity gap from 0.31 to 0.10. While a residual improvement remains, this result weakens the attribution of the original gain to the proposed mechanism. I appreciate the authors’ efforts, and the new results partially address my concerns. Nevertheless, I believe that a substantial revision of the paper’s overall framing and contribution claims is still needed. I am therefore lowering my rating to 2.

Reviewer_Y6UkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TOASTER, a recurrent memory architecture that seeks to combine the efficiency of linear-memory models such as KDA with the expressiveness of deep test-time-training memories such as Titans. The key insight is that a gradient step on a linear layer can be expressed as a delta-rule memory write. Using target propagation, TOASTER constructs a nonlinear two-layer memory from two linear memories connected through an activation function, allowing each layer to retain the efficient WY-parallelizable update structure. Across five model scales (0.2B-1.6B parameters), TOASTER consistently outperforms KDA on language modeling benchmarks, slightly surpasses attention on C4 perplexity, improves length extrapolation to 20K tokens, and achieves significantly higher throughput than Titans while maintaining competitive quality.

Critical strengths
  • Introduces a novel and elegant theoretical idea that bridges linear recurrent memories and deep test-time-training memories through target propagation, providing a principled way to obtain nonlinear memory while preserving efficient linear-memory updates.
  • Demonstrates strong empirical consistency, outperforming KDA across all tested model sizes on both C4 and PG19 and maintaining advantages in long-context evaluation.
  • Provides a compelling efficiency-quality tradeoff, achieving much better runtime than Titans while retaining many of the benefits of nonlinear memory architectures.
  • Evaluates across multiple dimensions including language modeling, downstream reasoning tasks, length extrapolation, retrieval, throughput, and ablations, producing a relatively comprehensive experimental picture. The ablations provide useful insights into the contributions of memory width, nonlinearity, target propagation, and update mechanisms.
Critical weaknesses
  • The reported gains over KDA and attention are generally quite small, especially at larger scales where perplexity differences often become marginal despite the significantly larger memory state and computational overhead.
  • TOASTER requires roughly 7-9× more memory-kernel operations and about 8× larger recurrent state than KDA, raising concerns about whether the empirical gains justify the additional complexity and resource consumption.
  • Downstream task improvements are modest, with an average gain of roughly one point, and several benchmark tasks remain at or near chance performance for all models, making it difficult to draw strong conclusions about downstream reasoning benefits.
  • The study stops at 1.6B parameters and 4K training context, making it unclear whether the observed advantages persist at the scales where modern frontier language models operate. Is it possible to do some experiments at 3B scale?
  • The proposed architecture shows improved average perplexity but still lags behind attention on several retrieval-oriented settings such as challenging needle-in-a-haystack tasks, highlighting potential memory-capacity limitations.
Questions for authors
  • TOASTER increases recurrent state size by roughly 8× relative to KDA. How much of the observed improvement comes from increased memory capacity versus the proposed target-propagation mechanism itself?
  • The gains become quite small at larger scales. Do the authors expect the improvement over KDA to persist, disappear, or grow at significantly larger model sizes (e.g., 10B+ parameters)?
  • Have the authors evaluated TOASTER against more recent test-time-training approaches such as E²-TTT, MesaNet, Modular TTT, or other emerging nonlinear memory methods under comparable settings?
  • How sensitive is performance to hidden-state width when comparisons are made at matched recurrent state size and computational budget rather than matched backbone parameters?
  • The paper demonstrates improved language-modeling loss, but downstream gains are relatively modest. Do the authors believe the architectural improvements primarily improve memorization rather than reasoning?
  • Given the weaker performance on some needle-in-a-haystack retrieval tasks compared to attention, what are the key remaining bottlenecks preventing recurrent memories from matching explicit token storage mechanisms?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Y6Uk (1/3): size of the gains, scale, and downstream evidence

Thank you for your detailed and constructive feedback. In response to your comments, we have revised the paper and added additional experiments to address your questions. Thanks to your feedback, we believe these changes have strengthened the manuscript. Please find our detailed responses below, in three parts. We group related points. Tables R1 and R2 are in the general response, and table numbers refer to the revised paper.

\quad

The reported gains over KDA and attention are generally quite small, especially at larger scales ...

The study stops at 1.6B parameters and 4K training context ... Is it possible to do some experiments at 3B scale?

... Do the authors expect the improvement over KDA to persist, disappear, or grow at significantly larger model sizes (e.g., 10B+ parameters)?

Size of the gain. We would argue that the gain is not small at this scale, where the loss–compute curve is very flat: on our ladder, 0.01 nats corresponds to about 9% of training compute (Appendix G.2). TOASTER's gain over KDA is 0.006–0.020 nats on C4, so at matched backbones and training tokens KDA would need 6–16% more training compute to reach the same loss (Table 8). The gain is also consistent: TOASTER is below KDA in all ten settings of the ladder, and again in the new hybrid setting at 1.2B, by 0.04 (C4) and 0.24 (PG19) perplexity (Table 3). Relative to attention, TOASTER is on par in-domain at every size, and its hybrid improves on attention on both corpora (9.78 vs. 9.96 on C4, 12.75 vs. 13.19 on PG19).

Scale. We agree that 3B and beyond is the real test. Unfortunately, the largest model of our ladder (1.6B non-embedding parameters) is the largest we can afford; the limitations note that the study stops there. We expect the improvement over KDA to persist. TOASTER is better than KDA at every size; the gap narrows from 145M to 850M and is roughly stable over the two largest sizes (0.0067 → 0.0059 nats on C4, 0.0063 → 0.0064 on PG19), reaching 9.94 vs. 10.00 for KDA and 10.09 for Titans at 1.2B (and 9.78 vs. 9.82 and 9.90 in the 3:1 hybrids). We also kept the hidden width fixed (dh=256d_h = 256) while the backbone grew eightfold, so the advantage should hold up best if the memory grows with the model. This is untested, and we hope that groups with larger budgets will try it. All models are trained at a 4K context, and the advantage over KDA is maintained when evaluating at 20K (Figure 3).

\quad

Downstream task improvements are modest ... several benchmark tasks remain at or near chance performance for all models ...

... Do the authors believe the architectural improvements primarily improve memorization rather than reasoning?

We agree that the downstream evidence is modest and noisy, and we say so in the limitations. This is common for models of this size that are pretrained on a single web corpus and evaluated three-shot without fine-tuning: several tasks stay near chance for every architecture, and the differences on the rest are a few points at most. We could not afford to train far beyond this budget, so at this stage we regard validation perplexity as the more reliable indicator. We report downstream accuracy as a sanity check, and the picture is consistent: TOASTER has the best average with all layers replaced (51.84%, against 50.69% for attention, 50.52% for KDA and 50.04% for Titans), and its hybrid has the best average of the four hybrids (52.11%; Table 3).

For the same reason, we do not think these results can tell memorization from reasoning. The suite scores multiple-choice answers to short prompts by likelihood, and the tasks that need multi-step reasoning (e.g., MathQA) are at chance for every model. A one-point difference in the average should not be read as evidence about reasoning in either direction. Whether the better language-modeling loss turns into better reasoning needs larger models and dedicated benchmarks, which we leave to future work.

AuthorsOpenReview ↗

Response to Reviewer Y6Uk (2/3): cost, target propagation, and state size

\quad

TOASTER requires roughly 7-9× more memory-kernel operations and about 8× larger recurrent state than KDA ...

... How much of the observed improvement comes from increased memory capacity versus the proposed target-propagation mechanism itself?

How sensitive is performance to hidden-state width when comparisons are made at matched recurrent state size and computational budget ...?

Cost. The 7–9× figure applies to the memory kernel alone, which is a small part of the model: about 1% of a block's operations for KDA and 8% for TOASTER at 1.2B, and less as models get wider (Appendix F). In wall-clock time, our unoptimized implementation takes 1.25–1.48× KDA's training step at 145M (Table 9). The state is 8× KDA's, but it does not grow with the context and is smaller than attention's key–value cache beyond about 260 tokens.

Capacity or target propagation? Over a 4K context, TOASTER's first layer S(1)S^{(1)} actively evolves from its initial state by 4.7% at 145M and 3.4% at 1.2B (and by 31.5% when the default kernel advances both layers). As noted in our original paper, setting γ=0\gamma = 0 (setting the inner error learning rate to 00 in Table 13: 15.71 vs. 15.69 at 145M) does not make S(1)S^{(1)} truly static, because input-dependent weight decay αt(1)\alpha^{(1)}_t still updates S(1)S^{(1)} in context. Within rebuttal compute limits, we added a new 145M ablation that completely freezes S(1)S^{(1)} (St(1)≡S0(1)S^{(1)}_t \equiv S^{(1)}_0; Table R1): at the exact same 8×8\times state size, completely freezing S(1)S^{(1)} gives 15.86 C4 perplexity, compared with 16.00 for KDA, 15.71 for γ=0\gamma = 0 (weight decay active on S(1)S^{(1)}), and 15.69 for full target propagation. Thus a static nonlinear feature map accounts for 0.14 of TOASTER's 0.31 gain over KDA, while adapting S(1)S^{(1)} in context accounts for the remaining 0.17 (more than half of the total gain). We view the wide nonlinear memory and target propagation as two parts of one method, keeping training affordable via a linear-memory kernel.

Matched state size. We see the larger state as a feature of the architecture rather than a confound. KDA itself has no state expansion: its state is tied to the head dimension, and a KDA with an added feature map that expands the keys would no longer be KDA. Enlarging the memory of a recurrent model is not trivial, and it is what TOASTER contributes. Comparing the two at the same backbone is in the same spirit as comparing a matrix-state memory such as Gated DeltaNet with a vector-state recurrent network such as an LSTM: one does not ask the LSTM to carry a matrix-sized state, because the larger state is part of what the newer architecture contributes. That said, we ran the ablation (Table R2): giving KDA 2×, 4× and 8× the total state by trading heads for wider heads at fixed H⋅dk=D=1024H \cdot d_k = D = 1024 (8 / 4 / 2 heads of width 128 / 256 / 512, keeping all D×DD \times D Q/K/V/gate/O projection parameter counts unchanged) yields 15.96 / 15.84 / 15.79, remaining 0.06–0.10 above TOASTER's 15.90, 15.76 and 15.69 at every matched state size at 145M. Enlarging the state closes about two-thirds of the 16.00-to-15.69 gap, at the cost of collapsing 16 heads down to two wide heads, whereas TOASTER scales an internal width dhd_h with 16 heads of width 64 unchanged. Within TOASTER, perplexity improves steadily with the hidden width, down to 15.62 at dh=512d_h = 512, while replacing GELU with a GSA-style softmax raises C4 perplexity from 15.69 to 18.19 by restricting the hidden activations to a simplex (Table 13 in the revised paper).

Matched compute. At a matched computational budget, with the mixers' own operations counted, TOASTER and KDA are at parity at our sizes (ratio 0.99–1.03, Table 8). The advantage that the paper reports is per parameter and per training token; Section 4 and the limitations state this.

AuthorsOpenReview ↗

Response to Reviewer Y6Uk (3/3): retrieval and recent TTT methods

\quad

... still lags behind attention on several retrieval-oriented settings such as challenging needle-in-a-haystack tasks ...

... what are the key remaining bottlenecks preventing recurrent memories from matching explicit token storage mechanisms?

Thank you for this question. We do not claim that TOASTER matches global attention at exact retrieval. A fixed-size state has to compress the context, whereas attention keeps every token, so exact recall of high-entropy content from far back (the UUIDs of S-NIAH-3) is limited by capacity and interference, however the state is updated. This gap is well documented for recurrent and linear-attention models [1–4], and it is why production models such as Kimi Linear and Qwen3-Next interleave a few attention layers with their linear memory layers. Within this class, TOASTER retains more than KDA when the needle is a number (63% vs. 37% on S-NIAH-2 at 4K) and is on par on UUIDs (27% vs. 28%). Closing the gap completely is an open and important direction.

In practice, this limitation is handled as for linear memories, by interleaving a few global attention layers. We trained such 3:1 hybrids at 1.2B, with one global attention layer after every three mixer layers (Table 3). The TOASTER hybrid is below attention on both corpora: 9.78 vs. 9.96 on C4, and 12.75 vs. 13.19 on PG19, the long-document corpus on which TOASTER alone trails attention. It is also below the KDA, Titans and sliding-window hybrids. On S-NIAH-1 / 2 / 3 at 4K (Table 4's protocol), the 1.2B hybrids score 99.7 / 89.3 / 77.3% (TOASTER), 84.7 / 81.7 / 59.7% (Titans), 89.3 / 78.7 / 56.3% (sliding-window) and 98.7 / 73.3 / 44.0% (KDA): adding one attention layer per four puts TOASTER ahead on all three 4K tasks and lifts its 4K UUID recall (S-NIAH-3) from 27% to 77.3%.

\quad

Have the authors evaluated TOASTER against more recent test-time-training approaches such as E²-TTT, MesaNet, Modular TTT ...?

Our empirical comparison with nonlinear test-time-training memories is against Titans, at 145M (Figure 5) and now also at 1.2B (Table 3): TOASTER is below Titans by 0.15 (C4) and 0.19 (PG19) perplexity with all layers replaced, and by 0.12 and 0.33 in the hybrid. E²-TTT and Modular TTT are concurrent arXiv preprints (August 2026), and we could not reproduce them faithfully within the rebuttal period. MesaNet is a linear memory: it solves its in-context regression to optimality, and its read-out is linear like KDA's. Our linear baseline is KDA, the memory TOASTER is built on. Appendix B discusses E²-TTT and Modular TTT and lists MesaNet among the linear memories. In short, E²-TTT derives a closed-form chunk transition for an MLP memory in which every gradient is still evaluated at the chunk-start weights, whereas TOASTER makes each layer a linear memory, so that the delta-rule kernel can apply the writes of a chunk in sequence (for the output layer by default, and for both layers in the kernel variant). Modular TTT reports that deeper fast-weight networks tend to hurt because activations grow too large, which is the instability that our bounded error signal is designed to prevent. We would be glad to add empirical comparisons once reference implementations are available.

We hope these answers and the new results address your concerns, and we would be glad to discuss further.

[1] Jelassi et al., Repeat After Me, ICML 2024.

[2] Arora et al., Zoology, ICLR 2024.

[3] Arora et al., Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff, ICML 2024.

[4] Wen et al., RNNs are not Transformers (Yet), ICLR 2025.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: new experiments and summary of changes

We thank all three reviewers for their careful and constructive reviews. This comment summarizes the new results; the individual responses give the details. Table and figure numbers refer to the revised paper, in which Tables 2–12 of the submission have become Tables 3–13.

1. What does target propagation contribute? All three reviews ask whether first-layer adaptation matters or whether TOASTER reduces to KDA over a fixed nonlinear feature map. Over a 4K context, TOASTER's first layer S(1)S^{(1)} actively evolves by 4.7% at 145M and 3.4% at 1.2B. As noted in our original paper, setting γ=0\gamma = 0 (setting the inner error learning rate to 00 in Table 13: 15.71 vs. 15.69 at 145M) is not a truly static state, because input-dependent weight decay αt(1)\alpha^{(1)}_t still updates S(1)S^{(1)} at every chunk. Within rebuttal compute limits, we ran a new 145M ablation that completely freezes S(1)S^{(1)} (St(1)≡S0(1)S^{(1)}_t \equiv S^{(1)}_0, 0% drift) and compared it directly with our Table 13 results.

Table R1. Completely frozen S(1)S^{(1)} vs. in-context first-layer adaptation at 145M.

Model (145M)S(1)S^{(1)} changes?C4 ppl
KDA—16.00
TOASTER, completely frozen S(1)S^{(1)}No15.86
TOASTER, γ=0\gamma = 0 (Table 13; weight decay active on S(1)S^{(1)})Yes15.71
TOASTER, full target propagation (default)Yes15.69

All TOASTER rows share the exact same architecture and 8×8\times state size: a fixed nonlinear feature map (frozen S(1)S^{(1)}: 15.86) gives 0.14 of the 0.31 gain over KDA (16.00), while adapting S(1)S^{(1)} in context contributes 0.17 (more than half of the total gain), reaching 15.71 via input-dependent decay and 15.69 with full target propagation.

2. Titans and hybrid models at 1.2B (Table 3). We added Titans at 1.2B (hidden width 64, chunk size 32) and a 3:1 hybrid setting, in which one global attention layer follows every three mixer layers. With all layers replaced, TOASTER is below Titans by 0.15 (C4) and 0.19 (PG19) perplexity and has the higher downstream average (51.84% vs. 50.04%). In the hybrid, TOASTER reaches 9.78 on C4 and 12.75 on PG19. This is below attention (9.96 and 13.19) and below the KDA (9.82 and 12.99), Titans (9.90 and 13.08) and sliding-window (9.91 and 13.19) hybrids, and its downstream average is the best of the four hybrids (52.11%).

3. The chunk-start approximation. We varied decoding chunk size C′C' against the exact per-token recurrence (C′=1C' = 1) and trained 145M models across C∈8,16,32,64,128,256C \in \\{8, 16, 32, 64, 128, 256\\} (Table 15 in the revised paper). Varying C′C' from 11 to 3232 moves C4 perplexity smoothly from 15.97 to 15.77 (15.79 at C′=64C' = 64). Across training CC, finer chunks improve C4 perplexity to 15.59 (C=8C = 8) and 15.56 (C=16C = 16) vs. 15.69 at C=32C = 32, C∈[32,128]C \in [32, 128] stays at 15.67–15.69, and C=256C = 256 reaches 15.74 (vs. KDA's 16.00).

4. State size. KDA's state is tied to its head dimension: it can only be enlarged by trading heads for wider heads, and nothing sets it independently of the key and value widths. Setting the size of the memory independently is not trivial, and it is what TOASTER contributes. As an ablation, we still gave KDA a larger state in this way, at a fixed backbone: trading heads down to 8, 4 and 2 heads of width 128, 256 and 512 at fixed H⋅dk=D=1024H \cdot d_k = D = 1024 (keeping all D×DD \times D Q/K/V/gate/O projection parameter counts unchanged) lowers 145M C4 perplexity from 16.00 to 15.96, 15.84 and 15.79, closing about two-thirds of the gap while remaining 0.06–0.10 above TOASTER (15.90, 15.76, 15.69) at every matched state size without sacrificing TOASTER's 16 heads of width 64.

Table R2. KDA with a larger state (145M, C4 perplexity).

Total state (relative to default KDA)1×2×4×8×
KDA (16 / 8 / 4 / 2 heads of dimension 64 / 128 / 256 / 512)16.0015.9615.8415.79
TOASTER (16 heads, dhd_h = 64 / 128 / 256)–15.9015.7615.69

5. Hidden width and activation. Table 13 of the revised paper now includes dh=512d_h = 512 (15.62 vs. 15.69 at dh=256d_h = 256) and a GSA-style softmax activation (18.19 vs. 15.69 for GELU).

Changes in the revised paper. The abstract, introduction and conclusion now state that the cross-layer signals are approximated once per chunk, and no longer say that TOASTER "resolves" the conflict between the two families. Section 3.3 discusses this approximation directly. Titans at 1.2B and the hybrids are in Table 3, the new ablations are in Appendix I (Tables 14–17) and Table 13, and Raven is discussed in the related work. Changes are marked in blue.

Area_Chair_5DVYOpenReview ↗

Reminder to reviwers to read author's responses

Dear Reviewers,

Thank you for your efforts in providing thorough reviews.

As the discussion period is underway, I would like to kindly remind you to read the authors' responses to your reviews and provide follow-up comments where appropriate or leave a comment. In particular, please consider whether the responses address your questions and concerns, and update your assessment if the clarifications or additional information affect your initial evaluation.

Your engagement in this discussion is important for ensuring a fair and thorough review process.

Thank you again for your time and contributions!

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.