acceptodds
Under review as a conference paper at ICLR 2027

Depth is Degree: Test-Time-Training Updates Are Polynomial Filters, and Learned Step Sizes Only Choose the Coefficients

Abstract

We replace the learned constants of test-time-training updates — step sizes in TTT, momenta in Titans, conjugate-gradient coefficients in MesaNet, the single step of linear attention — and of unrolled restoration solvers with a polynomial head that computes them from the operator: it starts at the textbook solver, costs the same operator applications, and learns no solver constant. The head rests on one fact: each such update is a solver for a quadratic in-context loss, and we prove it is a polynomial filter in its operator whose degree is the number of operator applications, so learned step sizes and momenta choose a filter but cannot enlarge the class; the head computes the class-optimal one. The regime is one protocol on one GPU (42M and 125M parameters), where the ordering does not change with size. On multi-query associative recall the head ties learned step sizes, momentum, and conjugate gradient at the ten-seed median (86/57 at loads 32/64); the tie is the claim, and the head is the only rule with no failed seed. Inside language models it is the lowest-perplexity update at degree four, by 1.8 over learned step sizes at 125M; on in-context regression with a dominant nuisance mode — a high-pass target — the exact solve and adaptive conjugate gradient fail at 2.7× the head's error, which is the class optimum. The theorem is also a design rule: capacity enters only through the nonlinearities and re-estimated operators between solver steps — attention is one — and the solver constants choose within a class the head spans directly, so they can be computed rather than learned. In one unrolled solver network whose operator is re-estimated in every layer (image restoration) the bounded head ties the textbook solver with no rate to choose and no failed seed, while learned step sizes collapse at three of four solver rates and at the fourth land above the exact solve of their own system — a gain the theorem assigns to the re-estimated operator, not to the constants; the “each layer is an iteration” reading holds at initialization, and training replaces it with a measured spectral response. The rule is one line: learn the operator, compute the solver.

Primary area:
learning on time series and dynamical systems
Keywords:
test-time training, linear RNNs, state-space models, linear attention, algorithm unrolling, polynomial filters

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_zmr4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies test-time training updates and unrolled linear solvers through the lens of polynomial filtering. Its central observation is that a fixed-operator, constant-coefficient Krylov iteration computes p(A)bp(A)b, where the number of operator applications bounds the polynomial degree. Based on this view, the paper argues that learned step sizes and momentum only select coefficients within a restricted polynomial class. It proposes replacing such updates with a directly parameterized polynomial head whose coefficients can be computed via a small least-squares problem.

Critical strengths
  1. The paper offers a perspective connecting Krylov methods, test-time training, linear attention, and unrolled optimization.
Critical weaknesses
  1. The presentation must be improved. The manuscript is unusually compressed and slogan-driven. Different concepts are frequently packed into the same sentences and paragraphs, making the paper extremely difficult to understand. The paper reads like machine-compressed research language.

  2. The technical novelty appears substantially more limited than the claim suggests. The basic connection between finite-step Krylov iterations and operator polynomials, Chebyshev/Bernstein parameterizations, and the optimality properties of CG are classical. The main contribution seems to be a unifying reinterpretation of these facts in the context of TTT and unrolled networks, together with a direct polynomial-head implementation. This is potentially useful, but does not appear commensurate with the paper’s broad theoretical framing.

  3. Conclusions are overgeneralized. The main results concern fixed-operator, constant-coefficient linear Krylov updates, whereas practical TTT and sequence models may contain input-dependent coefficients, nonlinearities, dynamic operators, or gates. The resulting claims therefore do not justify broad prescriptions such as “learn the operator, compute the solver,” nor the stronger suggestion that learned solver constants are generally unnecessary.

  4. The empirical evidence is too limited to support the claims. The experiments cover many domains but are individually limited, e.g., toy-level language-model studies. Furthermore, comparisons that do not always isolate the learned operator from the solver parameterization. Some reported results show ties or occasional advantages for tuned learned constants.

Questions for authors

Please refer to the weakness.

1 comment

We thank the reviewer for the careful reading, and for pressing on novelty and on whether the comparisons separate the operator from the solver.

Novelty (W2). We agree that "a fixed-constant Krylov iteration is a polynomial in the operator", the Chebyshev and Bernstein parameterizations, and the optimality of CG are classical; Section 2 says so. The parts we believe are not:

  1. The reachable set (Proposition 1, Corollary 1). Learned step sizes reach only residual polynomials with real roots. On a prescribed detail band at degree d=20d=20 (Table 6) the computed head reaches the 6060 dB cap; learned step sizes stop at 17.617.6 dB, steps with momenta at 26.426.4 dB, and the nested unrolled parameterization at 41.741.7 dB. The nested gap grows with the budget: 0.6/7.6/18.3/32.40.6 / 7.6 / 18.3 / 32.4 dB at d=6/10/20/40d = 6 / 10 / 20 / 40. The closest prior work we know is Takabe and Wadayama (2020, arXiv:2001.05142 and arXiv:2010.13335), who explain learned step sizes of unfolded gradient descent on convex quadratics by Chebyshev steps. Theirs is the convergence-only, gradient-descent case. We missed it, and a future version will cite it.
  2. The uniform statement across TTT, Titans, MesaNet-style CG, linear attention and unrolled solvers (Theorem 1), which is what lets one head replace the constants of all of them.
  3. Localization (Theorem 2). Capacity enters only through the nonlinearities and the re-estimated operators between solver steps, never through the solver constants.
  4. The dilemma (Proposition 4). Fixed-constant rules sit on one curve; only input-dependent coefficients leave it, by 48×48\times at degree 88 for condition number 100100 (E6).

Overgeneralization (W3). Conceded, as in the general response. Gated, input-dependent and nonlinear updates are outside the theorems; Theorem 2 says only where they enter. The claim we can defend is: within the fixed-operator, constant-coefficient regime the head computes what the constants learn, with no learned constant and no failed seed.

Comparisons that do not isolate the operator from the solver (W4). Correct, and this is the right question to ask of Table 2. Each arm trains its own operator learner, so the arms are compared at their own operators. The learned-constant arm at its best rate (30.2030.20 dB) lands above the exact solve of the system it nominally solves (30.1130.11 dB), which cannot be a solver effect; Theorem 2(ii) assigns it to the re-estimated operator. The clean comparison holds the operator fixed, and E4 and E6 are that comparison: there, every fixed-constant arm is at or below the class optimum, which the head computes.

Empirical scale (W4). The language-model results are at 42M and 125M parameters with one seed each. The ordering head << learned step sizes << linear attention (in perplexity) is the same at both sizes; momentum and CG were run at 42M only. The ten-seed medians are on MQAR. We make no claim beyond that regime.

Reviewer_mD1pOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper starts from an interesting observation that certain test-time training (TTT) updates can be expressed as polynomial filters of a fixed linear operator. It investigates whether directly computing the polynomial coefficients is an effective alternative to learning solver parameters such as step sizes and momentum. The paper approaches this question from both theoretical and empirical perspectives. On the theoretical side, the "optimality dilemma" offers an insightful perspective: accurately solving the underlying linear system may not be optimal for downstream tasks, and learned updates may instead serve as task-specific spectral filters. On the empirical side, the results are more mixed, revealing both the potential benefits and practical limitations of the proposed approach.

Critical strengths
  1. The polynomial-filter interpretation of iterative updates and the "optimality dilemma" provide useful insights into the relationship between iterative optimization and downstream task performance.

  2. Directly computing polynomial coefficients is an interesting alternative to learning solver constants, with potential advantages in optimization stability and reduced hyperparameter tuning.

Critical weaknesses
  1. Restricted theoretical scope and unclear generalizability. The main theoretical characterization relies on linear updates with fixed operators, typically derived from quadratic objectives. However, modern TTT architectures often incorporate nonlinear memory updates, input-dependent parameters, and dynamically changing operators. Although Theorem 2 discusses nonlinearities and re-estimated operators, it primarily characterizes where these mechanisms fall outside the fixed-polynomial formulation. It remains unclear how broadly the proposed approach applies to realistic architectures and how much of its practical advantage depends on the quadratic setting.

  2. Mixed empirical results. The proposed method often matches existing approaches rather than consistently outperforming them. On multi-query associative recall, the polynomial head achieves comparable median accuracy to learned step sizes, momentum, and classical conjugate gradient, with its primary advantage being stability across random seeds. In image restoration, the bounded head performs comparably to the textbook solver, while learned constants with a tuned learning rate achieve higher PSNR. The language-model experiments show promising improvements but are limited to 42M and 125M parameters, with only one reported seed. Overall, the evidence for a general advantage over learned solver parameters remains limited.

  3. Limited theoretical novelty. Much of the theoretical development builds on classical Krylov-subspace methods, conjugate-gradient optimality, and polynomial approximation theory. While connecting these results to learned test-time updates is interesting, the extent of the new theoretical contribution beyond applying established results to this setting could be clarified.

  4. Presentation. The writing is highly compressed and difficult to follow, with frequent slogans and sweeping claims that are not always adequately qualified by the theoretical assumptions or empirical evidence. The rapid transitions between numerical linear algebra, TTT architectures, graph filtering, and image restoration further obscure the central arguments. More explicit explanations of the relationships between these settings would substantially improve readability.

Questions for authors

1. Dependence on linearity and quadratic objectives.

How much does the proposed approach depend on the assumption of linear updates derived from quadratic objectives? The polynomial characterization is established for fixed-operator linear Krylov iterations, whereas modern TTT architectures often employ nonlinear memory updates, input-dependent step sizes, and operators that change between iterations.

Although Theorem 2 discusses nonlinearities and re-estimated operators, it is unclear whether the proposed polynomial head retains its theoretical or empirical advantages in these more general settings. Could the authors clarify which aspects of the approach extend beyond quadratic objectives, and whether the method can be integrated into realistic nonlinear TTT architectures without substantially restricting their expressive capacity?

2. Numerical stability of directly computed coefficients.

Directly computing polynomial coefficients may introduce numerical conditioning issues, particularly at higher polynomial degrees or for ill-conditioned operators. The authors propose Chebyshev and bounded polynomial parameterizations to mitigate these problems, but the experiments also report instability in the unconstrained least-squares head.

How does the numerical stability of the proposed approach scale with polynomial degree, operator condition number, and model size? In particular, is there an inherent trade-off between the expressivity and theoretical optimality of the unconstrained polynomial head and the numerical stability of bounded parameterizations? Could the authors provide a more systematic analysis of this trade-off, including the additional computational cost of solving for the coefficients?

1 comment

We thank the reviewer for the constructive questions on scope and on numerical stability.

Q1, dependence on linearity and quadratic objectives. The head applies wherever an update is a polynomial in a fixed operator: TTT-Linear, Titans' momentum, linear attention, MesaNet's CG at fixed coefficients, and unrolled quadratic solvers. It does not apply to nonlinear memory updates or input-dependent step sizes, and we do not claim that it does. Theorem 2 is the boundary: the solver constants are inside the class; the nonlinearities and re-estimated operators are outside it, and that is where the capacity of a realistic layer lives. Integrating the head into such a layer means replacing only its solver constants and leaving the gates, nonlinearities and operator re-estimation untouched. The restoration network of E2 is an example: its operator is re-estimated in every block.

Q2, numerical stability of computed coefficients. The trade-off the reviewer describes is real and measured.

  • The unconstrained least-squares head is exact on fixed-operator blocks (E6, E7). On a network that re-estimates its operator in every block, the per-step fit follows the spectra of the training crops and amplifies out-of-range modes on full-size images. Its three denoising seeds score 18.5/30.0/20.918.5 / 30.0 / 20.9 dB on the test set (Table 7).
  • The bounded Bernstein head confines the response to [0,1][0,1] whatever the spectrum (Proposition 5). Its three denoising seeds are 29.57/29.38/29.4329.57 / 29.38 / 29.43 dB.
  • Cost: one spectral upper bound per operator (a trace or a few power iterations) and one linear solve of size d+1d+1; no extra memory.

A systematic sweep over degree and condition number for the language-model case is not in the paper, and we do not claim one.

Mixed empirical results (W2). On recall and on restoration the targets are low-pass, and the theorem predicts a tie between fixed-constant rules. The tables show that tie. In deblurring (Table 7, seed 0) the exact solve, textbook CG, learned constants at the shared rate and the bounded head score 28.77/28.83/28.57/28.2928.77 / 28.83 / 28.57 / 28.29 dB. So the head has no accuracy advantage there, and we do not claim one. What separates the rules is reliability. Two further deblurring seeds finished after the submission deadline and are not in the submitted PDF; we report them here for completeness:

armseed 0seed 1seed 2
bounded head, 10×10\times rate28.4728.4728.1828.1828.2828.28
learned constants, 100×100\times rate25.7125.7128.6428.6427.6827.68
least-squares head27.7227.7228.3328.3311.9211.92

An accuracy gap appears only where the target is outside the restricted classes, on a fixed operator (Table 6): on the deblurring response learned step sizes are 1.81.8 dB and the nested unrolled parameterization 1.41.4 dB below the class optimum, and on a detail band 4242 dB and 1818 dB below.

Presentation (W4). Conceded; see the general response.

Reviewer_QcUqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a linear solver for test-time-training (TTT) layers, under fixed operator and quadratic loss assumptions. It shows that these updates produce polynomial filters whose degree is controlled by the number of operator applications. Motivated by this characterization, it proposes a direct polynomial head whose coefficients are fitted through a small least-squares solve, alongside a bounded variant for stability.

Validation includes synthetic in-context regression, multi-query associative recall, language models, and image restoration. The authors report comparable recall accuracy with improved stability across seeds in the main setting.

Critical strengths
  1. Connecting several solver-based updates through polynomial-filter representation under fixed-operator assumptions provides a useful framework for understanding what additional solver steps and learned scalar coefficients can change. The direct polynomial head computes its coefficients through least squares for the specified quadratic fitting problem, making the analysis practically relevant and offering a way to reduce reliance on gradient-based tuning of solver constants.

  2. The experiments cover several tasks, including synthetic regression, associative recall, language modeling, and image restoration, using common architectures and matched operator-application budgets within comparisons. In the main recall setting, the method achieves comparable performance with improved stability across seeds.

Critical weaknesses

Although the experiments cover several task families and demonstrate improved stability in the main recall setting, the evidence for a broader practical advantage remains limited. The language-model results use a single seed and dataset, and the largest model omits momentum and CG baselines that are competitive at the smaller scale.

In denoising, the computed head is unstable across seeds, while the successful bounded variant remains less accurate than the tuned learned baseline. These limitations make it difficult to assess how reliably the proposed approach improves upon well-tuned alternatives, weakening the empirical support for the paper’s general recommendation to replace learned solver constants.

Questions for authors
  1. Additional evidence, if available: How well do the accuracy and stability benefits extend beyond the main recall setting? Do you have further results on practical tasks that strengthen confidence in their reproducibility?

  2. Clarification of practical benefit: When compared with well-tuned existing methods, what is the main advantage of the proposed approach- better accuracy, greater stability, or reduced tuning effort? Which results most strongly support that advantage?

1 comment

We thank the reviewer for the review, and for asking where the head helps beyond recall.

Q1, beyond the main recall setting.

  • Restoration. The fixed-constant rules tie in deblurring as they do in denoising, which is what Theorem 1 predicts: exact solve 28.7728.77, textbook CG 28.8328.83, learned constants at the shared rate 28.5728.57, bounded head 28.2928.29 dB (Table 7, seed 0). The head's advantage there is its spread. Over three seeds the bounded head scores 28.47/28.18/28.2828.47 / 28.18 / 28.28 dB, learned constants at 100×100\times the shared rate 25.71/28.64/27.6825.71 / 28.64 / 27.68 dB, and the least-squares head 27.72/28.33/11.9227.72 / 28.33 / 11.92 dB. Seeds 1 and 2 finished after the submission deadline and are not in the submitted PDF.
  • Language models. At 42M parameters and degree 44 the four update rules are within 1.41.4 perplexity; the head (72.0772.07) sits between momentum (71.9171.91) and textbook CG (72.3272.32) and ahead of learned step sizes (73.2673.26). At 125M, among the three arms we could afford, it is lowest: 48.1248.12 against 49.9249.92 for learned step sizes and 52.7452.74 for linear attention (Table 10). One seed each, which we state as a limitation. The abstract's phrase "the lowest-perplexity update at degree four" holds at 125M and not at 42M, and should have said so.
  • Where accuracy separates. Only where the target lies outside the restricted classes: the prescribed band responses of E4 (Table 6), and in-context regression with a dominant nuisance mode, where the exact solve and adaptive CG are at 2.7×2.7\times the head's error.

Q2, the main advantage. Reliability and less tuning, not accuracy, except in the cases above. The head has no solver learning rate to search and no failed seed in ten on MQAR, against three of ten for learned step sizes and five of ten for linear attention (Figure 2, Table 11). In E2 (denoising) the learned-constant arm falls 2.52.5 dB or more below the head at three of the four solver rates we tried (Table 7).

Momentum and CG at 125M. Not run. The 125M comparison is head versus learned step sizes versus linear attention, and a future version will say so in the table.

Comments on the paper

1 comment

We thank the reviewers for their time. We are not uploading a revision in this discussion period; this reply is for the record.

Two points are conceded in full.

  1. The paper is too compressed. The abstract carries too many claims in one paragraph, slogans stand in for arguments, and the moves between numerical linear algebra, sequence layers and image restoration are not explained.
  2. The scope is narrower than the prose suggests. Every theorem is about a fixed operator and constant coefficients. The sentence "learn the operator, compute the solver" should have been stated as a design rule for that regime, not as a prescription for gated or input-dependent layers, which the theorems do not cover.

A future version will state the regime as its first premise, drop the slogan, and write each result at the length it needs.

The individual replies answer the technical points: what is new beyond the classical Krylov facts, what the restoration rows show once the operator is held fixed, and why the least-squares head fails on some seeds while the bounded head does not.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.