acceptodds
Under review as a conference paper at ICLR 2027

Fast Weights in Fast Weights: Towards Bridging Parallel and Sequential Test-Time Learning

Abstract

Test-time training (TTT) layers store context in the weights of a small inner network that is trained by gradient descent as the sequence is read. For hardware efficiency, practical TTT evaluates every gradient in a chunk at the chunk-entrance weights, discarding the effect of earlier within-chunk updates on later gradients. What this mini-batch approximation loses, and which part is worth recovering, is not well understood. Using block-level backward error analysis, we show that the leading-order gap between sequential and mini-batch TTT is the causal interaction , which is also the leading perturbation of the inner loop's modified vector field. For a two-layer fast-weight MLP, splits into four layer-wise interactions with closed forms. This motivates , a fast-weights-in-fast-weights design that keeps the first layer as a chunk-level fast weight and makes the output layer a token-level memory, while retaining chunk parallelism. With features fixed, the output layer follows the delta rule, which a chunkwise WY solve computes exactly. The influence of earlier first-layer updates reduces to feature transport, a causal linear-attention pass that shifts the key and query features before the solve. For small step sizes, we prove that matches the sequential output-layer update through second order, leaving the two first-layer interactions as the entire second-order residual. From 340M to 1B parameters, improves on the chunk-parallel baseline LaCT in perplexity and average zero-shot accuracy at every scale. On long-context tasks, it improves the average score by 8.5 points over LaCT and by 3.8 points over an ablation without feature transport.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test time training, large language models, efficient transformer, test time adaptation, online learning, implicit bias

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_trM6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the discrepancy between sequential and chunk-parallel test-time training (TTT) for fast-weight networks. Using block-level backward error analysis, the authors identify the leading-order causal interaction omitted by mini-batch updates. For two-layer fast-weight MLPs, they decompose this interaction into four layer-wise terms and propose FF-TTT, which combines a chunkwise WY-based solve for the output-layer delta-rule recurrence with a first-order feature-transport mechanism that accounts for the influence of earlier first-layer updates. The proposed method retains chunk-level processing of the first layer while recovering key causal interactions in the output layer. Experiments at 340M, 760M, and 1B parameters show improvements over LaCT in perplexity and average zero-shot accuracy.

Critical strengths

1. Principled analysis of the gap between sequential and chunk-parallel TTT. The paper uses block-level backward error analysis to characterize the leading-order causal interactions omitted by chunk-parallel updates, decomposing the discrepancy into four layer-wise terms for a two-layer fast-weight MLP. It provides a mathematical explanation for the limitations of existing parallel approximations and motivates the proposed method.

2. Promising empirical validation across model scales. Experiments at 340M, 760M, and 1B parameters show improvements over LaCT in perplexity and average zero-shot accuracy. These results strengthen the case that the analysis leads to practical benefits, rather than being only a theoretical contribution.

Critical weaknesses

1. Limited scope of the theoretical analysis. The backward error analysis and its decomposition of omitted causal interactions are developed for a two-layer fast-weight MLP under gradient descent. It is therefore unclear how broadly the analysis generalizes to deeper fast-weight networks, alternative update rules, or other architectures. The authors acknowledge this limitation but the paper does not discuss how the analysis might generalize or what obstacles exist. This limits the strength of the paper’s theoretical claims and makes it difficult to assess whether the proposed design principles extend beyond the specific setting studied.

2. The practical benefits are not fully justified relative to the additional computational cost. FF-TTT introduces feature transport to recover interactions omitted by chunk-parallel updates, but this mechanism adds computational overhead that may become significant at larger chunk sizes. Although the reported language-modeling and long-context results are promising, a more comprehensive characterization of the accuracy–throughput–memory trade-off across chunk sizes would strengthen the paper. In particular, sensitivity analyses across chunk sizes and hardware configurations would help establish whether the performance gains consistently justify the additional overhead in practical deployments.

Questions for authors

Clarification request 1: The theoretical analysis focuses on a two-layer fast-weight MLP. Can the authors clarify which parts of the backward error analysis and the resulting four-term decomposition remain valid for deeper fast-weight networks or alternative update rules? This would materially affect my assessment of the generality of the paper’s theoretical contribution.

Clarification request 2: Can the authors quantify the end-to-end throughput and memory overhead of FF-TTT relative to LaCT across the evaluated model sizes and chunk sizes, particularly where feature transport becomes computationally expensive? This would help determine whether the reported quality improvements translate into a compelling practical trade-off.

3 comments
AuthorsOpenReview ↗

Response to reviewer trM6 (Part 1/2)

Thank you for your rigorous feedback. We provided detailed clarifications to address all the questions you raised.


W and Q: Limited scope of the theoretical analysis. The theoretical analysis focuses on a two-layer fast-weight MLP. Can the authors clarify which parts of the backward error analysis and the resulting four-term decomposition remain valid for deeper fast-weight networks or alternative update rules?

A: We have added a new appendix E with detailed derivations and proofs extending our framework to (1) fast-weight networks of arbitrary depth and (2) gradient descent with momentum. Our results show that the WY-plus-transport framework is not specific to two-layer MLPs or plain gradient descent. The key principle remains unchanged: the last linear layer serves as the memory, whose sequential dynamics are solved exactly, while the updates to preceding layers are captured through feature transport.

(1) Generalization to deeper fast-weight networks.

Our analysis extends to an LL-layer fast-weight MLP, where the last layer plays the role of W2W_2 and all previous layers jointly play the role of W1W_1.

  • Causal interaction: The decomposition generalizes from four terms to L2L^2 layer-wise interactions. In particular, ALLA_{LL} captures the memory interaction, while ALβA_{L\beta}, for β<L\beta < L, captures the effects of hidden-layer updates on the memory.

  • Memory interaction: The last linear layer still follows the delta rule when its input features are fixed. Therefore, the WY triangular solve applies without modification, regardless of network depth.

  • Feature transport: The effects of preceding-layer updates are propagated recursively through the network:

    ΔUj=DjUs(Φj⊤Φj)+WjΔΦj,\Delta U_j = D_j U_s(\Phi_j^\top \Phi_j) + W_j \Delta \Phi_j,

    ΔΦj+1=Jj⊙ΔUj.\Delta \Phi_{j+1} = J_j \odot \Delta U_j.

    This requires one causal linear-attention computation per hidden layer, and reduces to our original transport formula when L=2L=2.

Importantly, the resulting update still matches the sequential last-layer update through second order. The proof only requires smooth hidden-layer feature mappings and a linear last layer, covering the SwiGLU architecture used in our experiments.

(2) Generalization to gradient descent with momentum.

We further show that the framework extends beyond standard gradient descent. With momentum updates mt=μmt−1+gtm_t = \mu m_{t-1} + g_t, the parameter displacement depends on weighted historical gradients:

Wt−1=W0−η∑i<twt−igi,W_{t-1} = W_0 - \eta \sum_{i<t} w_{t-i} g_i,

where wn=∑r=0n−1μrw_n = \sum_{r=0}^{n-1} \mu^r, assuming zero initial momentum.

This leads to a momentum-weighted causal mask, defined as:

Uμ(A)[i,t]=wt−iA[i,t],for i<t.U_\mu(A)[i,t] = w_{t-i} A[i,t], \quad \text{for } i < t.

The same decomposition and feature-transport structure remain applicable with the corresponding momentum weights. Moreover, the last-layer sequential dynamics can still be solved exactly through a triangular system:

R(I+ηUμ(Φ⊤Φ))=WLΦ−V.R(I + \eta U_\mu(\Phi^\top \Phi)) = W_L \Phi - V.

Thus, the mathematical structure of the WY solver and feature transport is preserved. In blockwise computation, the weighted mask retains the same matrix dimensions, while chunkwise execution additionally requires maintaining a momentum state across chunks.

(3) Scope and practical implications.

These extensions demonstrate that the two-layer structure is not essential to our framework. The principle is to separate exact last-layer memory updates from the feature drift induced by previous layers. Extending to deeper networks introduces additional feature-transport computations but does not require redesigning the WY memory solver. Similarly, momentum can be incorporated through weighted causal interactions and an additional state. More complex optimizers such as Muon, which involve nonlinear transformations of the gradient, remain an interesting direction for future work.

AuthorsOpenReview ↗

Response to reviewer trM6 (Part 2/2)

W and Q: Can the authors quantify the end-to-end throughput and memory overhead of FF-TTT relative to LaCT across the evaluated model sizes and chunk sizes, particularly where feature transport becomes computationally expensive? This would help determine whether the reported quality improvements translate into a compelling practical trade-off.

A: We conducted additional end-to-end throughput experiments comparing our optimized Triton kernels for FF-TTT against the officially released Triton kernels of LaCT at the 1B scale. We use DHSA Long-Data-Collections with a maximum sequence length of 32K and fix the SWA window size at 2048 for all configurations.

ChunkLaCT (tok/s)FF-TTT (tok/s)FF-TTT / LaCT
4K24,07622,7805.38% slower
8K26,21724,7465.61% slower
16K27,92926,2146.15% slower

The results show that larger chunk sizes improve throughput for both methods, while FF-TTT incurs only a modest overhead of 5.4–6.2% relative to LaCT's official implementation. This overhead remains relatively stable as chunk size increases, suggesting that feature transport scales efficiently in practice. Together with the improved long-context performance, these results demonstrate a favorable quality–efficiency trade-off.

Thank you for the detailed response and revisions. I will increase my Rating from 2 to 3.

Reviewer_z2gmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3(+1)
Read review
Paper summary

This paper studies what TTT loses when all gradients in a chunk are computed at the chunk-entrance weights, as in LaCT. Using backward error analysis, the authors show that the leading gap from sequential updates is a second-order causal interaction term, which splits into four layer-wise parts for a two-layer fast-weight MLP. Based on this, FF-TTT keeps the first layer at the chunk level, updates the second layer token by token with the delta rule through a WY solve, and corrects the key and query features with a causal linear-attention pass, called feature transport. For small step sizes, its second-layer update matches the sequential one through second order. Built on the LaCT architecture at 340M to 1B parameters, FF-TTT improves average zero-shot accuracy over LaCT by 0.5 to 0.6 points. On five long-context tasks, it improves the average score by 8.5 points over LaCT and by 3.8 points over the variant without feature transport. Its throughput is about 5.5% lower and it uses about 14% more memory.

Critical strengths

The paper‘s design comes directly from the analysis and first works out what mini-batching loses inside a chunk. To second order, this is a causal interaction term, which the authors split into four parts for a two-layer MLP. Two of these parts act on the second layer and can be recovered without giving up chunk parallelism, one through the WY solve and one through feature transport. FF-TTT is built from these two pieces. I like this design because it follows from the analysis, and the paper is also clear about what it does not recover, the first-layer terms A11 and A12. The derivations are careful. Each result comes with its assumptions and error order, and the authors say openly that the guarantees only hold for small step sizes.

Critical weaknesses

1.It is unclear where the gains over LaCT come from. FF-TTT keeps LaCT’s outer architecture, but it changes several things inside the fast-weight module at once. W2 is now updated token by token with the delta rule, feature transport is added, the inner loss is changed to a reconstruction loss, a new initialization is used, and the head configuration differs at 760M and 1B. The zero-shot gain over LaCT is small, 0.5 to 0.6 points on average, and Table 1 has no ablation at all. The only ablation in the paper, LaCT-DeltaNet without feature transport, appears only in Table 2. That table does not say which model size it uses, covers only 5 of the LongBench tasks with no reason given for the choice, and includes only 4 models. It also evaluates beyond 8K tokens, while training uses 4K sequences. Feature transport also adds about 11 GiB of memory (from 82.5 to 93.5 GiB). With this evidence, I cannot tell whether the improvement comes from the proposed mechanism, which is the main contribution of the paper.

2.The theory does not cover the setting used in the experiments. The second-order result needs bη to be small, but the experiments use b = 2048 with normalized features, so bη2 is large, as the authors themselves point out in Remark A.6. Their own large-step analysis in Appendix A.3 says that in this regime, the neglected error is of the same order as the correction that feature transport adds. Figure 1 points the same way. The step-size sweep stops at λ ≈ 30, while the real setting is λ = 2048. At native step sizes, transport helps a lot only in layer 0, where it closes 56% to 65% of the gap, and much less in the other layers shown (9% to 19%). I also could not find any comparison of model quality against sequential or small-chunk TTT, even though getting closer to sequential TTT is the motivation of the paper. The authors themselves note that Figure 1 does not show better training loss. So the link between the theory and the reported gains is missing.

Questions for authors

1.Could you add component-wise ablations to Table 1? For example: LaCT with only the loss changed to the reconstruction loss; LaCT-DeltaNet (without feature transport); FF-TTT without the cyclic-permutation initialization; and FF-TTT with the same head configuration as LaCT at 760M and 1B. Also, does the LaCT baseline keep the Muon inner-loop update used in the original LaCT? 2.Could you train FF-TTT and LaCT-DeltaNet at a few smaller chunk sizes (e.g., b = 64 and 256) and compare model quality? This would test whether feature transport gives a larger gain in settings where the theory applies better. 3.Which model size is used in Table 2? The five tasks and the three length buckets match LongBench-E, but they only cover its code-completion and few-shot learning categories. Why are the QA, summarization and synthetic retrieval tasks not included? Could you report results on all LongBench-E tasks and on NIAH, and add Transformer and Gated DeltaNet? 4.Minor: Figure 1, the only figure in the paper, is hard to read. The four panels have no labels, the font is very small, and the caption mixes the approximation results with the efficiency results. The paper does not say which model size and checkpoint are used, and “median output gap” and “gap closed” are not defined. Could you improve this in the revision?

2 comments
AuthorsOpenReview ↗

Response to reviewer z2gm (Part 1/2)

Thank you for your detailed feedback. We provided detailed clarifications to address all the questions you raised.


W and Q: It is unclear where the gains over LaCT come from. FF-TTT keeps LaCT’s outer architecture, but it changes several things inside the fast-weight module at once. Could you add component-wise ablations to Table 1? For example: LaCT with only the loss changed to the reconstruction loss; LaCT-DeltaNet (without feature transport); FF-TTT without the cyclic-permutation initialization; and FF-TTT with the same head configuration as LaCT at 760M and 1B. Could you add component-wise ablations to Table 1? For example: LaCT with only the loss changed to the reconstruction loss; LaCT-DeltaNet (without feature transport); FF-TTT without the cyclic-permutation initialization; and FF-TTT with the same head configuration as LaCT at 760M and 1B. Also, does the LaCT baseline keep the Muon inner-loop update used in the original LaCT?

A: We have expanded our ablation experiments and revised the paper to better clarify the sources of improvement over LaCT.

Clarification on reconstruction loss and head configurations. Our LaCT baseline uses the official Muon-based implementation. We attempted to isolate the effect of the reconstruction loss by modifying LaCT's kernel to replace its original dot-product objective with an MSE reconstruction objective. However, this modification introduced substantial register pressure and register spilling for head dimensions larger than 256, making it impractical to evaluate the modified kernel under LaCT's original large-state configuration.

Meanwhile, our FF-TTT implementation builds upon the DeltaNet kernel, which constrains the key dimension to at most 256. Consequently, we could not directly evaluate FF-TTT under LaCT's original head configuration.

Additional Matched Results. To address the concern about architectural differences, we have added a new baseline, LaCT-m, which uses the same number of heads and per-head dimensions as FF-TTT. This enables a more controlled comparison by eliminating differences in head configurations, helping to better isolate the improvements introduced by our method.

ScaleModelWiki. PPL ↓LMB. PPL ↓ARC-c ↑Hella. ↑PIQA ↑Wino. ↑LMB. Acc. ↑BoolQ ↑OBQA ↑SciQ ↑Avg. ↑
760MLaCT-m w/ Dot.21.2518.3131.3346.9568.7955.3741.2559.5534.6079.7052.19
760MLaCT-m w/ MSE.21.3818.5231.4747.8168.8854.9040.5259.7034.7079.4052.17
760MFF-TTT w/o FT21.0818.2532.4547.9768.3355.0341.2860.1035.5079.9052.57
760MFF-TTT20.7717.7732.5148.1968.8256.1241.7960.0636.0080.1052.95
1BLaCT-m w/ Dot.20.9219.6831.4649.9770.4452.5241.9360.5937.6080.7053.15
1BLaCT-m w/ MSE.20.6919.7532.2349.3370.1652.8242.6158.7637.5081.1053.06
1BFF-TTT w/o FT19.4017.9433.7650.5670.6655.8242.0460.0539.5081.7054.26
1BFF-TTT19.0916.8534.8752.6871.9556.6744.2661.9938.6081.0055.25

The gains do not stem from head configuration or inner loss, as matching either brings negligible improvement to LaCT. With both controlled, the exact block solve improves performance by 0.40/1.20 points at 760M/1B, while feature transport adds another 0.38/0.99 points, consistently reducing perplexity at both scales.

Clarification on initialization methods. Before conducting large-scale experiments, we performed preliminary ablations comparing low-rank initialization with and without permutation. We trained 760M-parameter models for 4K steps on FineWeb-Edu and evaluated per-token loss on a held-out validation set across different context lengths.

Method0–1K2K–4K4K–8K
Low-rank2.6122.5812.482
Low-rank w/ permutation2.6082.5232.428

Permutation consistently reduces validation loss across all context lengths, motivating its adoption in our final architecture.

AuthorsOpenReview ↗

Response to reviewer z2gm (Part 2/2)

Q: Additional LongBench-e and NIAH experiments. Which model size is used in Table 2?

A: We conducted additional long-context evaluations with 1B-scale models on LongBench-E and NIAH. All models were trained on approximately 58B tokens from DHSA Long-Data-Collections, a length-bucketed version of Together AI's Long-Data-Collections. We selected this dataset because it combines long-form text with instruction-following examples, providing a suitable training mixture for long-context retrieval and summarization tasks. We used 1B model in our Table 2.

ModelLongBench-E 0–4kLongBench-E 4–8kLongBench-E 8k+LongBench-E OverallNIAH-1 4K (%)NIAH-1 8K (%)NIAH-1 16K (%)NIAH-1 32K (%)
FF-TTT18.2417.8515.9817.36100.0100.0100.099.2
FF-TTT w/o FT15.7016.2313.6115.18100.0100.0100.094.4
DeltaNet15.3514.7413.7814.62100.0100.0100.097.5
LaCT15.3013.189.6112.7095.788.484.580.8
SWA-204816.7510.305.5710.8789.871.212.50.0

FF-TTT consistently achieves the best performance across all LongBench-E context lengths and NIAH settings, improving the LongBench-E overall score from 12.70 (LaCT) to 17.36 and achieving 99.2% accuracy on NIAH at 32K. Removing feature transport reduces the overall LongBench-E score to 15.18 and NIAH-32K accuracy to 94.4%, further demonstrating its contribution to long-context modeling. We have incorporated these results and additional experimental details into the revised paper's appendix.


W: .The theory does not cover the setting used in the experiments.

A: Clarification on feature transport at native step sizes.

We thank the reviewer for the rigorous feedback. We acknowledge that Proposition 4.1 provides a small-step guarantee, while the experimental setting (b=2048b=2048 with normalized features) operates in a large-step regime.

First, the large-step analysis does not imply that feature transport becomes ineffective. Although the neglected higher-order error can be of the same order as the correction, this does not mean they have the same magnitude. The effectiveness of transport depends on how much it reduces the discrepancy from sequential updates, which we measure empirically through the gap closure 1−ET2/ET11-E_{\rm T2}/E_{\rm T1}. The third panel of Figure 1 evaluates this directly at the native setting (b=2048b=2048), showing that transport still closes a meaningful portion of the gap, although the improvement varies across layers. We will explicitly mark the native operating point and clarify these metrics in the revised figure.

Second, we conducted additional experiments with smaller chunk sizes using a 340M-parameter model to further examine the connection between the theoretical approximation and model quality.

ChunkPPL w/o FT ↓PPL w/ FT ↓Gap closed: Dolma (%)Gap closed: PG19 (%)
204814.213.765.02 / 12.32 / 9.1257.41 / 12.41 / 13.83
25612.812.484.13 / 71.79 / 54.2482.94 / 69.65 / 64.44
6412.311.998.50 / 87.39 / 71.4090.70 / 88.33 / 74.20

The results reveal two consistent trends: (1) smaller chunks lead to substantially higher gap closure, consistent with the small-step analysis; and (2) feature transport consistently reduces perplexity across all chunk sizes, including the native setting of 2048. These findings provide empirical evidence linking improved approximation of sequential updates to better language modeling performance, even beyond the regime covered by our theoretical guarantee.


Q: .Minor: Figure 1, the only figure in the paper, is hard to read. The four panels have no labels, the font is very small, and the caption mixes the approximation results with the efficiency results. The paper does not say which model size and checkpoint are used, and “median output gap” and “gap closed” are not defined. Could you improve this in the revision?

A: Thank you for the suggestion. We have redrawn Figure 1 with larger fonts, added labels (a)–(d) to distinguish the four panels, and revised the caption to provide clearer explanations of the experimental settings and results.

Specifically, panels (a) and (b) examine the validity range of our small-step theoretical approximation, while panel (c) directly evaluates the effectiveness of feature transport at the native step size, demonstrating its practical benefits beyond the small-step regime. We have also clarified the definitions of median output gap and gap closed and added details on the model size and checkpoint used in the experiments.

Reviewer_Yq4bOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies TTT (test-time training) layers, where fast weights are updated online by gradient descent on the test sequence. In practice TTT is trained in blocks (mini-batch updates), which removes the causal interaction between tokens inside a block. The authors use backward error analysis (BEA) to identify exactly what mini-batching drops: to leading order, a causal interaction term that sums, over each token in the block, the Hessian of that token's loss applied to the accumulated prefix gradient. They show this term is the leading perturbation of the inner-loop dynamics, not just a Taylor remainder.

For a two-layer MLP fast-weight network, they split this interaction into four layer-wise parts. The two acting on the output layer are structured. The first is the memory interaction: with features held fixed, the sequential update is exactly the delta rule, and a WY triangular solve reproduces it exactly. The second is the feature-transport interaction: earlier updates to the first layer change the features that later output-layer writes consume. The key observation is that this does not require the full sequential first-layer trajectory; it is enough to transport the prefix effect into feature space, which is exactly causal linear attention. This yields FF-TTT: the first layer stays chunk-level, the second layer becomes a token-level memory, and a feature-transport correction is applied before the WY solve. They prove the combined update matches the sequential output-layer endpoint through second order, and the excluded first-layer terms form an explicit residual. Experiments at 340M, 760M, and 1B parameters show small but consistent gains over LaCT in perplexity and zero-shot accuracy, and larger gains on long-memory tasks (about 3.8 points over LaCT-DeltaNet, the ablation without transport).

Critical strengths
  1. The theoretical framing is coherent and the derivation is rigorous. BEA is an appropriate tool, and the paper establishes a well-defined leading-order object and demonstrates that it is the leading perturbation of the modified vector field, not merely a Taylor remainder.
  2. The decomposition is substantive rather than cosmetic. The memory interaction is the fixed-feature delta rule and is handled exactly by the WY solve. The feature-transport interaction is a feature shift and is handled by causal linear attention. Both are computable in parallel. The paper also states explicitly why the first layer cannot be resummed analogously: the joint recurrence is bilinear and admits no compact triangular factorization.
  3. The decision to keep the first layer at chunk granularity is a meaningful design insight. Recovering the feature-transport interaction requires only the prefix effect transported into feature space, not the sequential first-layer trajectory.
  4. The ablation is well chosen. LaCT-DeltaNet (WY solve without transport) is the appropriate control, and the gap of approximately 3.77 points on long-memory tasks substantially exceeds the zero-shot gap of 0.5 to 0.6 points. This supports the claim that the improvement concerns long-range memory rather than general language-modeling noise.
  5. The initialization scheme is well motivated. The cyclic-shift permutation prevents sign collapse in the nonnegative orthant. The modification is minor in cost, and the rationale is clearly presented.
  6. The long-memory evaluation is well targeted. LCC, RepoBench, TREC, TriviaQA, and SAMSum across 0-4K, 4-8K, and 8K+ buckets directly assess the capability the method is intended to improve.
Critical weaknesses
  1. The improvements over LaCT are small, and the claim of competitiveness with Transformer is overstated. The average zero-shot deltas of 0.51, 0.49, and 0.60 points at 340M, 760M, and 1B are within run-to-run variation for most of the reported benchmarks, and no seeds, error bars, or variance estimates are reported. Table 1 also shows Transformer outperforming FF-TTT on several columns at 1B (HellaSwag, WinoGrande, BoolQ), and FF-TTT ranks first or second on most, but not all, columns. The abstract-level claim should be moderated accordingly.
  2. The throughput and memory comparison is not like-for-like. FF-TTT achieves 48.5k tokens/s/GPU versus 51.3k for LaCT, but memory is 93.5 versus 81.8 GiB, a memory premium of approximately 14 percent. The TTT-MLP figures derive from a separate JAX synthetic benchmark, whereas the LaCT figures derive from completed PyTorch training runs. The paper notes this caveat but still presents the values in a single figure. Either the frameworks should be matched or the cross-framework comparison should be removed.
Questions for authors
  1. What are the empirical magnitudes of the two excluded first-layer residuals relative to the two recovered output-layer interactions in a trained 340M model? If the cross-layer residual is non-negligible, does it account for the trust-region rolloff observed in Fig. 1?
  2. LaCT-DeltaNet isolates feature transport. Can the authors additionally ablate query transport separately from key transport? Which of the two accounts for the observed gain?
  3. Why is Gated DeltaNet omitted from the 1B block of Table 1? Can a sliding-window Transformer be added to Table 2 as an absolute reference for the long-memory tasks?
  4. The limitations section states that deeper MLPs and Muon or momentum inner optimizers would require their own transport rules. How substantial is this extension in practice? Is the WY-plus-transport template reusable, or is the two-layer structure essential?
2 comments
AuthorsOpenReview ↗

Response to reviewer Yq4b (Part 1/2)

Thank you for recognizing the coherence and rigor of our theoretical derivation, the substantive nature of our decomposition, the soundness of our architectural design, and the well-targeted evaluation of long-term memory capabilities. We provided detailed clarifications to address all the questions you raised.


W: The improvements over LaCT are small, and the claim of competitiveness with Transformer is overstated. The abstract-level claim should be moderated accordingly.

A: We have revised our claims regarding the comparison with Transformers and included new long-memory results below, along with additional general language modeling experiments in the revised paper. Our new results consistently demonstrate improvements over the straightforward approach of increasing the chunk size in TTT.

W: The throughput and memory comparison is not like-for-like. FF-TTT achieves 48.5k tokens/s/GPU versus 51.3k for LaCT, but memory is 93.5 versus 81.8 GiB, a memory premium of approximately 14 percent. The TTT-MLP figures derive from a separate JAX synthetic benchmark, whereas the LaCT figures derive from completed PyTorch training runs.

A :We thank the reviewer for this point and agree that the JAX-based TTT-MLP numbers are not directly comparable to the PyTorch ones.

(1) FF-TTT vs. LaCT is like-for-like. Both are measured in PyTorch from completed training runs on the same hardware and configuration. In this setting, FF-TTT has ~5.5% lower throughput (48.5k vs. 51.3k tokens/s/GPU) and ~14% higher memory (93.5 vs. 81.8 GiB). We acknowledge this overhead and will state this trade-off explicitly in the revision.

(2) TTT-MLP is cross-framework. The only public TTT-MLP implementation is in JAX, and we followed the protocol of LaCT in reporting it as a reference. In the revision, we now state explicitly in the main text and in the figure caption that these numbers come from a different framework, from synthetic-step benchmarks, and from a different memory metric, and are therefore not a like-for-like comparison. If the reviewer still finds this presentation misleading, we are happy to remove the cross-framework numbers from the figure.


Q: Why is Gated DeltaNet omitted from the 1B block of Table 1? Can a sliding-window Transformer be added to Table 2 as an absolute reference for the long-memory tasks?

A: We have incorporated SWA and conducted additional long-context experiments with 1B-scale models. As shown in the table below, our method consistently improves performance on both real-world and synthetic long-context benchmarks, including LongBench-e and NIAH.

For training, we use 58B tokens from DHSA Long-Data-Collections, a length-bucketed version of Together AI's Long-Data-Collections. We choose this dataset as it contains both long-form text and instruction-following examples, providing a suitable training mixture for evaluating long-context capabilities on tasks such as information retrieval and summarization.

ModelLongBench-E 0–4kLongBench-E 4–8kLongBench-E 8k+LongBench-E OverallNIAH-1 4K (%)NIAH-1 8K (%)NIAH-1 16K (%)NIAH-1 32K (%)
FF-TTT18.2417.8515.9817.36100.0100.0100.099.2
FF-TTT w/o FT15.7016.2313.6115.18100.0100.0100.094.4
DeltaNet15.3514.7413.7814.62100.0100.0100.097.5
LaCT15.3013.189.6112.7095.788.484.580.8
SWA-204816.7510.305.5710.8789.871.212.50.0

We revised the paper and included detailed results in the appendix. The GDN experiments are underway, and we will incorporate results in the paper once they are complete.


Q: LaCT-DeltaNet isolates feature transport. Can the authors additionally ablate query transport separately from key transport? Which of the two accounts for the observed gain?

A: Thank you for the suggestion. Before conducting large-scale experiments, we performed preliminary ablations to investigate the contribution of query transport. Specifically, we trained 760M-parameter models for 4K steps on FineWeb-Edu and evaluated the per-token validation loss across different context lengths.

As shown in the table below, adding query transport to the key-transport baseline consistently improves performance at longer context lengths, with increasingly pronounced gains as the context length grows. In particular, query transport reduces the validation loss from 2.580 to 2.523 at 2K–4K and from 2.570 to 2.428 at 4K–8K, while maintaining comparable performance at shorter lengths.

Model0–1K2K–4K4K–8K
FF-TTT w/o QT2.6082.5802.570
FF-TTT2.6082.5232.428

These preliminary results motivated us to use query transport in the final architecture, as it provides substantial additional benefits for long-context modeling.

AuthorsOpenReview ↗

Response to reviewer Yq4b (Part 2/2)

Q: The limitations section states that deeper MLPs and Muon or momentum inner optimizers would require their own transport rules. How substantial is this extension in practice? Is the WY-plus-transport template reusable, or is the two-layer structure essential?

A: We thank the reviewer for this question. We have added a new appendix E with detailed derivations and proofs extending our framework to (1) fast-weight networks of arbitrary depth and (2) gradient descent with momentum. Our results show that the WY-plus-transport framework is not specific to two-layer MLPs or plain gradient descent. The key principle remains unchanged: the last linear layer serves as the memory, whose sequential dynamics are solved exactly, while the updates to preceding layers are captured through feature transport.

(1) Generalization to arbitrary-depth fast-weight networks.

Our analysis extends naturally to an LL-layer fast-weight MLP, where the last layer plays the role of W2W_2 and all preceding layers jointly play the role of W1W_1.

  • Causal interaction: The decomposition generalizes from four terms to L2L^2 layer-wise interactions. In particular, ALLA_{LL} captures the memory interaction, while ALβA_{L\beta}, for β<L\beta < L, captures the effects of hidden-layer updates on the memory.

  • Memory interaction: The last linear layer still follows the delta rule when its input features are fixed. Therefore, the WY triangular solve applies without modification, regardless of network depth.

  • Feature transport: The effects of preceding-layer updates are propagated recursively through the network:

    ΔUj=DjUs(Φj⊤Φj)+WjΔΦj,\Delta U_j = D_j U_s(\Phi_j^\top \Phi_j) + W_j \Delta \Phi_j,

    ΔΦj+1=Jj⊙ΔUj.\Delta \Phi_{j+1} = J_j \odot \Delta U_j.

    This requires one causal linear-attention computation per hidden layer, and reduces to our original transport formula when L=2L=2.

Importantly, the resulting update still matches the sequential last-layer update through second order. The proof only requires smooth hidden-layer feature mappings and a linear last layer, covering the SwiGLU architecture used in our experiments.

(2) Generalization to gradient descent with momentum.

We further show that the framework extends beyond standard gradient descent. With momentum updates mt=μmt−1+gtm_t = \mu m_{t-1} + g_t, the parameter displacement depends on weighted historical gradients:

Wt−1=W0−η∑i<twt−igi,W_{t-1} = W_0 - \eta \sum_{i<t} w_{t-i} g_i,

where wn=∑r=0n−1μrw_n = \sum_{r=0}^{n-1} \mu^r, assuming zero initial momentum.

This leads to a momentum-weighted causal mask, defined as:

Uμ(A)[i,t]=wt−iA[i,t],for i<t.U_\mu(A)[i,t] = w_{t-i} A[i,t], \quad \text{for } i < t.

The same decomposition and feature-transport structure remain applicable with the corresponding momentum weights. Moreover, the last-layer sequential dynamics can still be solved exactly through a triangular system:

R(I+ηUμ(Φ⊤Φ))=WLΦ−V.R(I + \eta U_\mu(\Phi^\top \Phi)) = W_L \Phi - V.

Thus, the mathematical structure of the WY solver and feature transport is preserved. In blockwise computation, the weighted mask retains the same matrix dimensions, while chunkwise execution additionally requires maintaining a momentum state across chunks.

(3) Scope and practical implications.

These extensions demonstrate that the two-layer structure is not essential to our framework. The reusable principle is to separate exact last-layer memory updates from the feature drift induced by preceding layers. Extending to deeper networks introduces additional feature-transport computations but does not require redesigning the WY memory solver. Similarly, momentum can be incorporated through weighted causal interactions and an additional state. More complex optimizers such as Muon, which involve nonlinear transformations of the gradient, remain an interesting direction for future work.

Comments on the paper

1 comment

Thank you for the detailed response and the new experiments. The LaCT-m baseline with matched heads, the comparison of the two losses and the ablation without feature transport make it fairly clear that the gains come mainly from the block solve and feature transport. The full LongBench-E and NIAH results also show a clear improvement on long context. The smaller-chunk experiment is helpful as well. I still have some reservations. The effect of the initialization is not ablated at full scale, and the long-context comparison does not include Transformer or Gated DeltaNet. I hope the revision will include these results and state that the theory only covers small step sizes. I am raising my score to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.