- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies test-time training updates and unrolled linear solvers through the lens of polynomial filtering. Its central observation is that a fixed-operator, constant-coefficient Krylov iteration computes , where the number of operator applications bounds the polynomial degree. Based on this view, the paper argues that learned step sizes and momentum only select coefficients within a restricted polynomial class. It proposes replacing such updates with a directly parameterized polynomial head whose coefficients can be computed via a small least-squares problem.
Critical strengths
- The paper offers a perspective connecting Krylov methods, test-time training, linear attention, and unrolled optimization.
Critical weaknesses
-
The presentation must be improved. The manuscript is unusually compressed and slogan-driven. Different concepts are frequently packed into the same sentences and paragraphs, making the paper extremely difficult to understand. The paper reads like machine-compressed research language.
-
The technical novelty appears substantially more limited than the claim suggests. The basic connection between finite-step Krylov iterations and operator polynomials, Chebyshev/Bernstein parameterizations, and the optimality properties of CG are classical. The main contribution seems to be a unifying reinterpretation of these facts in the context of TTT and unrolled networks, together with a direct polynomial-head implementation. This is potentially useful, but does not appear commensurate with the paper’s broad theoretical framing.
-
Conclusions are overgeneralized. The main results concern fixed-operator, constant-coefficient linear Krylov updates, whereas practical TTT and sequence models may contain input-dependent coefficients, nonlinearities, dynamic operators, or gates. The resulting claims therefore do not justify broad prescriptions such as “learn the operator, compute the solver,” nor the stronger suggestion that learned solver constants are generally unnecessary.
-
The empirical evidence is too limited to support the claims. The experiments cover many domains but are individually limited, e.g., toy-level language-model studies. Furthermore, comparisons that do not always isolate the learned operator from the solver parameterization. Some reported results show ties or occasional advantages for tuned learned constants.
Questions for authors
Please refer to the weakness.
We thank the reviewer for the careful reading, and for pressing on novelty and on whether the comparisons separate the operator from the solver.
Novelty (W2). We agree that "a fixed-constant Krylov iteration is a polynomial in the operator", the Chebyshev and Bernstein parameterizations, and the optimality of CG are classical; Section 2 says so. The parts we believe are not:
Overgeneralization (W3). Conceded, as in the general response. Gated, input-dependent and nonlinear updates are outside the theorems; Theorem 2 says only where they enter. The claim we can defend is: within the fixed-operator, constant-coefficient regime the head computes what the constants learn, with no learned constant and no failed seed.
Comparisons that do not isolate the operator from the solver (W4). Correct, and this is the right question to ask of Table 2. Each arm trains its own operator learner, so the arms are compared at their own operators. The learned-constant arm at its best rate (30.20 dB) lands above the exact solve of the system it nominally solves (30.11 dB), which cannot be a solver effect; Theorem 2(ii) assigns it to the re-estimated operator. The clean comparison holds the operator fixed, and E4 and E6 are that comparison: there, every fixed-constant arm is at or below the class optimum, which the head computes.
Empirical scale (W4). The language-model results are at 42M and 125M parameters with one seed each. The ordering head < learned step sizes < linear attention (in perplexity) is the same at both sizes; momentum and CG were run at 42M only. The ten-seed medians are on MQAR. We make no claim beyond that regime.