- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper address a fundamental problem in learning when we need to choose between reusing learning rate schedule or tune it. Because one can think of training steps as a regularizer the authors use that to build a restriction on the parameters adjustments. They show we misjudge a model ability to adjust and compare optimal to practical updates of weights. That in practice allow them to predict if there is a better scheduler.
Critical strengths
I find this results novel and it goes beyond theoretical prediction as the authors also showed it works in practice. The problem itself is important and can assist in optimizing model optimization cost.
Critical weaknesses
The models used here, both theoretical and experimental, are too simple and only relevant for linear filters. That limitation reduce the usefulness of this approach dramatically but keeps the insights interesting.
Questions for authors
How would you tackle or approximate non-linear filters for the same question?
2 comments
Response to Reviewer KRTK (Part 2/2):
- Experimental details: All conditions share the same hidden initialization, nominal head learning rate , and fixed Digits split (1,347 training and 450 held-out images). For each of 24 paired initializations, we fit the log–log slope of the RMS prediction change between and over across all held-out images and 10 output coordinates (brackets report the IQR across initializations). The rate Jacobian is evaluated at , and rank-two counts apply the descriptive threshold .
3. Can the trajectory approximation select useful schedules for nonlinear models?
Effective within a local trust region (5%–15% radius) on short trajectories, while low-dimensional direct validation dominates on long trajectories: The second-order trajectory surrogate accurately matches direct validation within a local schedule-change radius (5%–15%) on short trajectories and outperforms first-order Taylor selection, whereas at large radii (60%) or over 100–300 ResNet-18 updates—where exact unrolled differentiation hits memory limits and finite-difference curvature becomes step-sensitive—direct validation over our low-dimensional schedule menu is both more accurate and faster.
- Short trajectories across trust-region radii (shifted/blurred Digits, 64 configurations): We evaluated updates with 180 calibration images and 417 disjoint test images (after an exploratory step stabilizing raw source rates using only training-loss decreases; all test cross-entropy [CE] changes below are relative to that stabilized reference):
Table 2a. Digits trajectory-surrogate schedule selection across perturbation radii (64 configurations).
| Radius | 2nd-order match with 7-cand. | 1st-order match with 7-cand. | 7-cand. validation CE | Uniform-scaling CE | 2nd-order surrogate CE |
|---|---|---|---|---|---|
| 5% radius | 64/64 (100%) | 52/64 (81.3%) | -0.0293 | -0.0283 | -0.0293 |
| 15% radius | 58/64 (90.6%) | 41/64 (64.1%) | -0.0418 | -0.0365 | -0.0396 |
| 30% radius | 39/64 (60.9%) | 28/64 (43.8%) | -0.0502 | -0.0441 | -0.0114 |
| 60% radius | 18/64 (28.1%) | 19/64 (29.7%) | -0.0551 | -0.0489 | +0.1404 |
At a 5% radius, second-order selection matched seven-candidate validation in 64/64 configurations, reducing mean test CE by 0.0293 (improving test CE in 60 configurations, tying in 1, and worsening in 3). Including the Hessian term substantially improves agreement over first-order Taylor at 5%–15% radii, whereas by 60% radius higher-order trajectory nonlinearity dominates (+0.1404 vs. -0.0551), cleanly delineating the local trust region.
- Longer trajectories and finite-difference (FD) diagnostics (full-network ResNet-18 on Office-Home, 18 configurations): Across three transfers, , and three seeds, we evaluated full-network schedule selection. Exact unrolled automatic differentiation exceeded memory at updates on a 96 GiB GPU, and our prespecified central-FD fallback exhibited severe step-size sensitivity over long horizons: at , a single FD curvature entry shifted from 67.0 to 1.59 to -0.99 across FD step sizes .
Table 2b. Office-Home full-network ResNet-18 selectors at 5% and 60% schedule-change radii.
| Selector | Mean CE vs. reuse (5% radius) | Mean CE vs. reuse (60% radius) | Target tuning time (s) |
|---|---|---|---|
| Uniform-scaling validation (2-cand.) | -0.0148 | -0.0549 | 39.4 |
| Seven-candidate validation | -0.0162 | -0.0614 | 89.2 |
| First-order FD | -0.0139 | -0.0228 | 101.7 |
| Second-order FD | -0.0115 | +0.0064 | 101.7 |
Negative CE differences indicate improvement over reuse, and tuning times are reconstructed from measured stages under a no-cache deployment accounting. While FD selectors achieve modest gains at a 5% radius (-0.0139 and -0.0115), FD curvature noise degrades second-order FD at a 60% radius (+0.0064). In contrast, direct validation over our two-candidate uniform-scaling menu captures nearly the full gain of seven-candidate validation (-0.0549 vs. -0.0614) in just 39.4 s.
4. Presentation and readability improvements.
Finally, we appreciate your feedback on readability: in the revised manuscript, we will substantially streamline the Abstract, Introduction, and Figure 1 caption by opening with a concrete -step gradient-descent workflow and two-step example before introducing operator notation.
Response to Reviewer KRTK (Part 1/2):
Thank you for recognizing the novelty and practical relevance of our work, and for your insightful question on extending our framework to nonlinear models. Below, we address concerns step by step.
1. How can we tackle or approximate nonlinear models for this question?
Thank you for raising this central question, which bridges our exact spectral-filter theory and practical deep-learning workflows. Two complementary levels of approximation: We can tackle nonlinear models at two distinct levels: (i) applying our exact finite-step filter operations to a quadratic head on frozen nonlinear representations (where preservation, reuse, and retuning execute directly), or (ii) using a second-order trajectory-differentiation surrogate around schedule reuse when updating all parameters end-to-end.
Level 1: Frozen nonlinear representations (Office-Home Benchmark, p=513): A pretrained nonlinear backbone provides fixed features, and a quadratic head is trained for K gradient updates, retaining our exact finite-step spectral-filter representation F0=I−∏k=1K(I−α0,kS0). Across three Office-Home transfers (
Real_WorldtoArt,Clipart, andProduct), K∈{100,300}, and three seeds (18 configurations), reuse and calibration-selected retuning consistently outperform preservation:Table 1a. Office-Home quadratic heads on frozen ResNet-18 features (p=513).
At a 60% radius, uniform-scaling and 7-candidate validation selected identical uniform rate increases across all 18 configurations. These natural transfers demonstrate how the four operations execute in high dimensions, while direct application of our local theorem additionally requires its source-optimality and local task-path assumptions.
Level 2: Full-trajectory approximation for end-to-end training: Let θK(q) denote the network parameters after K updates under a low-dimensional schedule perturbation q around reuse, with target calibration loss L(q)=Lcal(θK(q)). We approximate this post-training loss via the second-order Taylor surrogate:
L(q)=L(0)+g⊤q+21q⊤Hq,g=∇qL(0),H=∇q2L(0),
where g and H propagate through the entire K-step trajectory. This surrogate selects among a fixed menu of schedules using target calibration labels: it addresses local schedule selection around reuse, whereas our paper's Γs criterion addresses the population-risk comparison between optimal retuning and preservation.
2. How does updating the hidden-layer parameters change the output geometry?
Continuous transition from singular quadratic to full-rank first-order geometry: Updating hidden-layer parameters progressively removes the equal-rate Jacobian degeneracy of the fixed-feature model, shifting the two-step split response from second order (slope 2.000) to first order (slope 0.962) and making the rate Jacobian numerically rank two in 24/24 initializations.
To quantify this transition, we compared matched 64-unit
tanhnetworks on Digits (K=2 full-batch gradient updates with squared loss and a zero-initialized head) across varying ratios of hidden-to-head learning rates ηhid/ηhead∈{0,0.01,0.1,1.0}:Table 1b. Equal-step output geometry across feature-learning strength in 64-unit
tanhnetworks (K=2).