- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a block version of a 2nd order online learning algorithm for a linear model under a specific form of loss, which extends formulations commonly used in test-time training of language models. The paper establishes differentiability, justifying its use for end-to-end training, and performs controlled experiments to isolate the effects of the proposed method.
Critical strengths
- Interesting theoretical result that informs the design of a new adaptive version of an RLS algorithm.
- Informative experiments that support the theory and attribute the predicted and observed effects, e.g. forward vs backward rules
Critical weaknesses
- Actual performance on the motivating use case of test-time training is weak. Perplexity is very similar to the baseline while throughput is significantly reduced
- Throughput increases are shown against weak baselines, i.e. a sequential high-precision implementation
- Minor:
- Results in the abstract are unattributed or ambiguous. The MQAR benchmark could be named up front, and the baseline method for the 9.4 point increase could be described briefly. The numbers for MSE are also absolute figures that are difficult to interpret without further context of the synthetic regression task, so mentioning the setting and reporting a relative improvement may be a bit more informative.
- The theory for the applicable piecewise smooth setting is in the appendices instead of the main paper.
Questions for authors
- How tight is the bound in unnumbered inline inequality from Section 3.5 in practice?
3 comments
Thanks to the authors for the responses. They are useful for better contextualizing the paper and its results.
For the section 3.5 bound, from your description, it sounds like the matrices in the additional numerical experiment are synthetic, instead of coming from actual use cases. Could the authors confirm this understanding? If my understanding is correct, then this experiment would not answer the question. If I misunderstood, then being explicit about how realistic the matrices are would be advised.
As Reviewer aaDw pointed out, the sequential implementation in PyTorch is not meaningful. What optimization and validation was performed on the custom GPU implementations, considering this paper isn't primarily GPU kernel optimization work? Comparing against existing implementations may have been preferable, so that this would not need additional justification.
Thanks for the follow up!
-
On the Section 3.5 bound, our original controlled-conditioning experiments used synthetic matrices. As requested, we also evaluated the bound along FlashRLS trajectories using MNIST, digits, and breast-cancer features with logistic updates, fp32 recurrent state, an fp64 core, and chunk sizes 16, 64, and 128. The ratio of true direction-solve error to the residual bound has medians of 0.66–0.98 across settings, with a minimum of 0.21. The triangular systems are generally well-conditioned on these trajectories (median condition number 1–3, maximum 23). These show tightness on real-data trajectories and complement the controlled synthetic conditioning tests.
-
We did look for already existing implementations, but to our knowledge, there isn't a maintained public GPU implementation of QRD-RLS that supports our setup: batched sequential updates, intermediate reads at every observation, and a backward pass. The closest work we found was Andrew and Dingle (2014), which implements GPU QR updating, but not this complete online-learning workload and their code isn't publicly available. Libraries like cuSOLVER and MAGMA provide optimized QR and Cholesky factorizations but applying these naively looks more like this:
Compared to FlashRLS speedup- torch.linalg.cholesky, refactor every row (GPU) 365× FlashRLS time torch.linalg.qr, refactor every row (GPU) 2,110× FlashRLS time So the GPU based implementations we have are significantly more optimized than this.
Repeatedly refactorizing the matrix introduces a lot of unnecessary overhead work. Structure-aware QR updates avoid that cost, but require sequential factor modifications and intermediate solves (this is the method we went for, it gave the best speed optimized baselines.) For response-dependent updates, the local weights are also determined in order, so we cannot precompute a weighted block and factorize it. In terms of hyper optimizing the updates: Even an optimized factor-based implementation must either perform these dependent factor updates and reads in order, or move the dependencies into a smaller interaction system and batch the expensive matrix ops afterward. The "hyperoptimized" baseline would be the construction we have developed in FlashRLS.
Response to reviewer GC6c
Thanks for the thoughtful review! We agree that FlashRLS is not yet throughput-competitive with optimized learned-memory layers, but we think the results show a more interesting accuracy–throughput tradeoff than perplexity alone suggests.
In our 50M-parameter Gated KalmaNet experiment trained on 1B SlimPajama tokens, exact reads recover 6.4 ± 1.2 MQAR points under weaker regularization, while maintaining comparable perplexity and reducing read error from 1.6×10⁻² to 5×10⁻⁵. More broadly, weaker regularization preserves low-variance feature directions that can carry useful associative information, but makes the system harder to solve accurately. Gated KalmaNet (Peng et al., CVPR 2026) identifies this tradeoff, and MesaNet (von Oswald et al., ICLR 2026) demonstrates the broader value of regression-based learned memories. Our near-duplicate-key experiment illustrates the benefit: at κ≈10⁶, direct reads achieve 98% recall versus 22% for CG-30. This suggests that accurate direct solves could enable learned memories to retain finer distinctions without relying on strong regularization, although throughput remains a limitation.
We also want to highlight Criteo, where the adaptive formulation is tested on 45.8M real-world examples. FlashRLS achieves better logloss than Adagrad and FTRL (0.4630 vs. 0.4698 and 0.4716), and higher AUC (0.7885 vs. 0.7810 and 0.7795), while matching sequential online Newton at 38.8× lower runtime. This demonstrates practical benefits beyond language-model test-time training.
Regarding the speedup baselines, Appendix K.4 identifies the tokenwise PyTorch/autograd implementation. We also compare custom GPU Householder, Givens, and pivoted QR implementations in Section 4.3 and Appendix N, reporting 3.9–61.2× speedups across the tested configurations.
For the Section 3.5 bound, we conducted a small additional numerical experiment using fp32 triangular solves against fp64 references, across chunk sizes C = 16–128 and incoming covariance condition numbers 1–10⁶. For response-adaptive logistic Laplace at C = 64, the median ratio of actual reconstruction error to its residual bound was 0.957–0.993 across five seeds per setting. Thus, the bound is quite tight in the tested regimes.
Finally, we agree that the abstract could identify MQAR and the stopped-curvature baseline more explicitly, and describe the Huber result as a roughly 33% MSE reduction under outlier contamination. We will also move the key piecewise-smooth result from Appendix G.6 into the main text when given the additional space.