- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TOASTER, a novel recurrent memory which aims to combine the expressivity of nonlinear memories (such as TTT and Titans) with the fast parallel training of linear memories (like DeltaNet and KDA). The latter can rely on a WY representation of the delta-rule corrections to compute chunkwise updates efficiently, by inverting a low-tri linear system, but nonlinear memories don’t generally allow for this, precisely due to the nonlinearity, which forces instead sequential and/or mini-batch updates. TOASTER addresses this by approximating the gradient propagations through the MLP they represent, following a strategy rooted in target propagation. The approximation computes these gradients from a frozen state at the beginning of each chunk, which allows one to reuse classical WY-form-based kernels for approximately recovering the required state updates. Pretrained on C4 at five scales (up to 1.6B), TOASTER is reported to reach lower loss than KDA at every size and slightly lower C4 perplexity than attention, at a training cost within 1.33x of KDA's.
Critical strengths
- The core observation that target propagation allows to recycle delta-rule kernels is a nice catch
- The empirical study is quite extensive (with some caveats, see below)
- The paper is unusually candid about its own limitations (with more caveats, see below)
Critical weaknesses
- W1: The overall claim that “TOASTER resolves an apparent conflict between two families of recurrent memory” is overstated: the core approach relies on an approximation (a single Jacobi step of the nonlinear system defining the delta correction), whose impact is not properly measured
- W2: The paper's own ablations suggest that target propagation, its core idea, contributes very little; most of the gain over KDA comes from elsewhere.
- W3: The comparison with linear memories and attention is dirtied by confounders: gains are generally small, tend to shrink with scale and vanish when actually matching compute, no multiple-seed results are reported, and linear baselines employ hidden states of 8x different size.
Questions for authors
W1: Impact of approximation The main motivation behind this paper is to overcome the mini-batch updates forced by nonlinear memories, as they force sequentiality less one loses accuracy (L73-76). However, I do not believe TOASTER really escapes this problem: it merely side-steps it, and it’s unclear with what effect. At its core, TOASTER still relies on a nonlinear recurrence: as such, a gradient step on it can then be represented via a delta-rule update only formally, since the actual “value” (z-g) depends on the state itself, which evolves along the chunk. Computing a chunk of such steps exactly still requires solving a nonlinear system. TOASTER side-steps this by freezing the cross-layer signals at the chunk-initial states, which turns each layer’s update into an ordinary delta-rule recurrence that the WY kernel can solves efficiently. But this is only an approximation of the true update: the real non-linear system governing it is effectively being solved in an approximate fashion, via a single Jacobi/Picard iteration, initialised with the states at the start of each chunk (notice this is similar in spirit to using “stale state updates” for TTT, although the procedure is different). Now, this is not at all an issue per-se, provided that the approximation is a valid one. At the very least, however, I would’ve expected that: (i) this was clearly recognised and discussed (this is mentioned in passing in L242, but not properly discussed), and more openly stated in the paper, adequately tuning down the claim that TOASTER resolves the nonlinearity issue of TTT; (ii) the effect of this approximation was actually measured, by checking error wrt the real solution recovered sequentially: is one Jacobi iteration enough? Moreover (iii) with this approximation, the chunk-size becomes an inherent design choice for the model, which has the potential to affect the state evolution: the approximation is exact for (as you correctly point out), but progressively grows with , consequently drifting from the “true” state evolution, and an investigation of this effect would also be relevant (since it creates a potential mismatch between training and decoding configurations). Notice this is not the case for KDA: varying does not vary the final (up to rounding-off errors). Finally, (iv) the impact of this approximation may compound even worse for deeper memories (which are not studied in the paper), which again might hinder the generalisability of the method.
W2: What drives the gains Another concern comes from the ablations in Tab12: if I’m understanding them correctly, they suggest that target propagation (the main strategy implemented by the paper) actually plays a minor role in boosting performance. In fact, even with (for which the method collapses to a delta-rule memory over a learned nonlinear feature map, L1408-1410), TOASTER reaches 15.71 agains a 15.69 for the default. In other words, the effect from the TTT-style update accounts only for ~0.02 of the 0.31 PPL gain over KDA (which I would expect to be within seed-to-seed variation, although such variation is not reported). The reported WY run confounds the effects of and the different update strategy. A WY-kernel run with would then be useful to properly attribute this gain to the error signal rather than to the update rule. Similarly, using only identity as activation (whose read is a linear map of rank at most , like in KDA), already recovers 0.20 of the 0.31 gain. You attribute part of this to the bias, the learned initial states and the gate scales (L1404-1406), and KDA was not retuned with TOASTER's gate scales (L482). So it is unclear how much of the gain over KDA is due to the cascade itself. Finally, since each coordinate of the hidden correction is bounded by about (L278-279), I wonder how far actually moves from its learned initial state over a sequence. Did you measure this? If it barely moves, TOASTER is essentially KDA with a learned nonlinear expansion of the keys, and the natural baseline would be KDA with such a feature map of the same width.
W3 Baselines comparison The main concern in this sense is that the comparison against KDA is not on equal grounds: while you’re matching parameters (although it’s unclear how closely, given what you write in L1112), hidden state sizes of TOASTER vs KDA vary by a factor of (L1100). Considering how hidden state size directly relates to an RNN’s capacity, this is not a fair comparison. Moreover, gains tend to shrink with model size, and TOASTER has the smallest fitted exponent in Tab6. The fact that in Fig2 (mid) you’re not considering the sequence mixer in your FLOPs calculation (and probably parameter counts, again for L1112), makes me wonder how I should really interpret the results there. Also downstream and retrieval evidence is not conclusive, 6/13 tasks are at chance (which is however expected at this scale), and results on retrieval are not particularly convincing given my comment on the KDA state size considered. To your credit, you are explicit about some of these limitations, but I would’ve adopted a more sober tone throughout the paper, then.
Minor:
- L14: “A chunk of such steps has an exact WY form” as it’s written, this sentence doesn’t make sense: expand
- L18: “Target propagation gives the hidden memory its values” it gives them an approximation (in general)
- L19-20: this too is really hard to parse
- L35: “pays for it exactness” Exactness of what?
- L38: “because the state is bounded … how cheaply it can be trained”: how does the fact that the state is bounded impacts how cheaply it can be trained? Cheap training is a property one seeks regardless
- L46: ”Products of such factors admit the compact WY representation” this is a bit better than L14…
- L49: “their memory is limited to linear mappings and hence its capacity is limited” Arguably, capacity has more to do with the boundedness of the state size. I’d say expressivity is limited by the linearity constraint
- L87: “whatever residual is presented as the value” this too reads odd. A delta rule removes what was previously stored in the “slot” of St associated with kt, and replaces it with vt.
- L90: “so such a layer can be trained by an unmodified delta-rule kernel” not true in general: the Jacobian of the loss can mess up the structure of the delta update computation
- Tab1 is rather redundant wrt Fig1
- L111: “we describe three design choices...”: please provide a clearer high-level description of them
- L146: “and the gates are passed alongside the keys” explain
- L154: “the capacity is set by the width of the network rather than by ” true, but the width of the network is still related to the size of the hidden state, as that defines the weights of the MLP.
- L165: “recovering the exact WY form layer by layer at the price of computing the signals that pass between layers once per chunk” make this clearer: the real price you’re paying is that you’re approximating this signal
- Prop1 is really a tautology, so should be presented as a remark rather than an actual proof. In words, it reads “If we manage to get the target exactly right, so that it matches the actual gradient, then updating the way we do is equivalent to performing backprop and TTT”. The crux of it all, is that in general (C>1) this exact target is not available.
- L242: “The approximations are that … sums its C writes as in Eq. (3)” Precisely, that’s the point: you’re applying an approximation, because the true gradient (depending on S_{t-1}) is not available. Your method is then not equivalent to TTT: it’s an approximation that makes its application faster
- L262: “Linear memories are bias-free outer products, and a bias adds nothing to them” I honestly don’t understand what this means
3 comments
Response to Reviewer pfJ4 (2/3): W2, what drives the gains
W2: The paper's own ablations suggest that target propagation, its core idea, contributes very little; most of the gain over KDA comes from elsewhere.
Your comment on Table 12 (Table 13 in the revision) pinpointed an important nuance. As noted in our original paper, setting (setting the inner error learning rate to at 145M: 15.71 vs. 15.69) does not make truly static, because the input-dependent weight-decay gate still updates at every chunk. Within rebuttal compute limits, we ran a new 145M ablation that completely freezes (, 0% drift) and compared it directly with our Table 13 results.
Completely frozen vs. active at 145M. At the exact same architecture and state size (Table R1 in the general response), completely freezing yields 15.86 C4 perplexity, compared with 16.00 for KDA, 15.71 when (weight decay still active on ), and 15.69 with full target propagation. Thus a fixed nonlinear feature map accounts for 0.14 of TOASTER's 0.31 gain over KDA, whereas adapting in context contributes 0.17—more than half of the total improvement—with input-dependent decay reaching 15.71 and the cross-layer error signal taking it to 15.69.
How far does move? Over a 4K sequence, actively changes from its learned initial Frobenius norm by 4.7% at 145M and 3.4% at 1.2B under 32-step chunk-averaged updates, and by 31.5% (up to 68.3% per head) with the default kernel on both layers (the model of Table R3), while changes by 53–60%. Freezing at its initial state at inference () already degrades this model's perplexity by on C4 () and on PG19 (); under 32-step chunk averaging largely absorbs inference-only freezing (/ on C4/PG19), whereas retraining from scratch with completely frozen degrades 145M C4 perplexity by (; Table R1).
KDA with a feature map. A KDA with an added feature map that expands its keys is no longer KDA: it is a different memory with a larger state. Because the row in Table 13 (15.71) still adapted via input-dependent weight decay, the exact implementation of a delta-rule memory over a fixed learned nonlinear feature map is the completely frozen row in Table R1 (15.86), cleanly separating static feature-map capacity (0.14 PPL) from in-context adaptation (0.17 PPL).
Identity row and gate scales. You are right that the identity row differs from KDA in the bias, the learned initial state and the gate scales, so its 0.20 cannot be attributed to the cascade alone; we reworded this. KDA uses its own gate scales, and TOASTER's smaller scales were tuned for the cascade. We did not retune KDA with them, as the limitations state.
The main benefit of TOASTER is that a wide nonlinear memory becomes trainable with a linear-memory kernel, at 1.25–1.48× KDA's step time. Target propagation is what allows the hidden layer to learn in context within this construction, and its measured contribution is the one in Table R1, which the revised paper reports in Section 4 and Appendix I (Table 14).
Response to Reviewer pfJ4 (3/3): W3, baselines; minor points
W3: The comparison with linear memories and attention is dirtied by confounders: gains are generally small, tend to shrink with scale and vanish when actually matching compute, no multiple-seed results are reported, and linear baselines employ hidden states of 8x different size.
State size. TOASTER holds 8× the state of KDA. We see this as what the architecture provides rather than as a confound. KDA itself has no state expansion: its state is tied to the head dimension, and enlarging the memory of a recurrent model is not trivial. Comparing the two at the same backbone is in the same spirit as comparing a matrix-state memory such as Gated DeltaNet with a vector-state recurrent network such as an LSTM: one does not ask the LSTM to carry a matrix-sized state, because the larger state is part of what the newer architecture contributes. That said, we ran the ablation (Table R2 in the general response): giving KDA 2×, 4× and 8× the total state by trading heads for wider heads at fixed (8 / 4 / 2 heads of width 128 / 256 / 512, keeping all Q/K/V/gate/O projection parameter counts unchanged) yields 15.96 / 15.84 / 15.79, remaining 0.06–0.10 above TOASTER's 15.90, 15.76 and 15.69 at every matched state size at 145M. Enlarging the state closes about two-thirds of the 16.00-to-15.69 gap by collapsing 16 heads down to two wide heads, whereas TOASTER scales with 16 heads of width 64 fixed.
Parameters and compute. KDA and TOASTER are closely matched in parameters: they share the backbone at every size (Table 6) and differ only in a small number of mixer parameters. Figure 2 uses the standard 6ND estimate, which leaves out the operations of every sequence mixer. Table 8 counts them. Against attention, whose quadratic term is the largest mixer cost at 4K, TOASTER's gain rises (1.22 at 145M and 1.06 at 1.2B on C4). Against KDA, the two are at parity (0.99–1.03): at our sizes, the gain over KDA is about what the extra kernel arithmetic would buy. The paper says so in Section 4 and in the limitations.
Scale. You are right that the gap to KDA narrows with size (0.020 to 0.006 nats on C4) and that TOASTER has the smallest fitted exponent. The gap is roughly stable over our two largest sizes, and TOASTER is better than KDA per parameter and per training token in all ten settings. In the setting in which recurrent memories are deployed, a 3:1 hybrid with global attention at 1.2B, TOASTER improves on the KDA hybrid by 0.04 (C4) and 0.24 (PG19) perplexity, and on attention by 0.18 and 0.44 (Table 3). Compared with Titans, the other nonlinear memory, TOASTER trains 2.7–5.2× faster at 32K and reaches lower perplexity at 1.2B.
Seeds. All runs are single runs: we could not afford to repeat the ladder, and the limitations say so. We rest the evidence on the consistency of the ordering. TOASTER is below KDA in all ten settings of the ladder, and again in the new hybrid runs at 1.2B. On the 20K evaluation, the per-window standard error of the difference between TOASTER and KDA is about 0.001 nats, against differences of 0.008–0.014 nats (Table 12). This is evaluation noise, not seed-to-seed variance, which we cannot estimate.
Downstream and retrieval. We agree that this evidence is not conclusive, and the limitations say so. In particular, we have not run the retrieval tests against a KDA with wider heads, so that comparison remains open.
Minor: L14, L18, L19-20, L35, L38, L46, L49, L87, L90, Tab1, L111, L146, L154, L165, Prop1, L242, L262
Thank you for the careful reading. We adopted all of these points. Briefly: L14 now says what has a WY form (the product of a chunk's steps), as L46 already did; L19–20 now say that the cross-layer signals, which include the targets of L18, are approximated once per chunk; L35 and L38 are reworded; L49 and L154 use "expressivity" and relate the width to the state size; L87 and L90 state that the delta rule replaces what is stored under a key, and that the kernel applies only once the error signal is given; L111 is split into short sentences that name the three design choices; L146 explains how the gates enter the kernel; L165 and L242 name the approximation as the price; Proposition 1 is a remark; and L262 now says that a constant input coordinate matters in the cascade because it shifts the pre-activations of the hidden layer. Table 1 will move to the appendix in the final version; we left it in place for now so that the table numbers stay stable during the discussion.
We hope the new measurements and the more sober presentation address your concerns. We would be glad to clarify anything further.
Response to Reviewer pfJ4 (1/3): W1, the approximation
Thank you for your detailed and constructive feedback. In response to your comments, we have revised the paper and added additional experiments to address your questions. Thanks to your feedback, we believe these changes have strengthened the manuscript. Please find our detailed responses below, in three parts (W1; W2; W3 and minor points). Tables R1 and R2 are in the general response, and table numbers refer to the revised paper.
We agree with this characterization. Within a chunk, the exact per-token recurrence couples the two layers nonlinearly. TOASTER evaluates the cross-layer signals (hidden keys and targets) once, at the chunk-start states. Given these signals, the output layer is advanced by an exact linear-memory kernel; the hidden layer takes a summed chunk-start step by default (as in Eq. 3), or the kernel in the both-layers variant. Across layers this is one Jacobi/Picard sweep, exact only for C=1. The submission said this in Section 3.3 and Appendix D, but too briefly, and several sentences in the main text claimed more.
(i) Framing. We revised the abstract, introduction and conclusion. The paper no longer says that TOASTER "resolves" the conflict. It says that a layer's update becomes executable by an exact kernel once the cross-layer signals are given, and that approximating these signals once per chunk is the price. Section 3.3 now discusses this approximation directly, and Proposition 1 is presented as a remark.
(ii) Is one sweep enough? We measured it directly on a 145M model with the default kernel on both layers, where C′=1 is the exact coupled per-token recurrence that the chunked algorithm approximates (Table R3, top). Evaluating the C=32-trained model across decoding chunk sizes C′∈1,4,8,16,32,64, C4 perplexity moves smoothly from 15.97 at C′=1 to 15.77 at the training chunk size C′=32 (22.73 to 22.33 on PG19) and 15.79 at C′=64—staying within 0.20 of C′=1 across the full range and within 0.02 across C′∈[16,64]—while the hidden-memory states of the first block differ from C′=1 by only 1.7% in relative Frobenius norm after 4K tokens. One sweep is enough in practice.
(iii) Chunk size as a design choice. You are right that, unlike in KDA, C is part of the model, as it is in Titans, LaCT or TNT. One clarification: there is no mismatch between training and decoding as long as decoding uses the training value of C. Decoding runs the same chunked recurrence, and it can be executed token by token with the same results up to rounding, because each token only needs the chunk-start states and the within-chunk kernel recurrence. The paper now states this explicitly. To see how the choice of C affects quality, we also trained 145M TOASTER across C∈8,16,32,64,128,256 (Table R3, bottom): smaller chunks refresh the cross-layer targets more frequently and improve C4 perplexity to 15.59 (C=8) and 15.56 (C=16) vs. 15.69 at default C=32, C∈[32,128] stays at 15.67–15.69, and even C=256 reaches 15.74 (0.26–0.44 below KDA's 16.00 across the entire 32× range, with C=32 balancing throughput and quality).
Table R3. Effect of the chunk size (C4 perplexity).
(iv) Deeper memories. We agree that the approximation is applied once per hidden layer and may compound with depth. We have not trained deeper cascades and make no claim about them; the limitations say this directly.