acceptodds
Under review as a conference paper at ICLR 2027

When Does Learning-Rate Retuning Help? Attainable Output Geometry Under a Fixed Update Budget

Abstract

Learning-rate schedules are often tuned for one training setting and reused when models are adapted to new data under fixed update budgets. A finite-step optimization trajectory can itself act as a statistical regularizer. We study when retuning the schedule after a task change improves on preserving the nominal finite-step filter. We characterize the leading retuned-minus-preserved risk through the statistical distance between the oracle drift and the one-sided attainable output set. For distinct interior steps and a simple spectrum in dimensions, the attainable correction space has dimension . Some spectral corrections are therefore inaccessible when , whereas the unrestricted commuting optimum becomes locally attainable when . At equal steps, step splitting generates first-order output corrections missed by the parameter Jacobian, where indexes the local task change. For two steps, this singular geometry is the endpoint of a near-equal transition governed by , where is the nominal half-gap. These results explain why unrestricted spectral retuning can overestimate attainable corrections while local Jacobian sensitivity can underestimate them. Synthetic experiments support the theoretical predictions, while real-data diagnostics distinguish preservation, schedule reuse, and retuning.

Primary area:
optimization
Keywords:
Learning-Rate Schedules, Optimization, Implicit Regularization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KRTKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper address a fundamental problem in learning when we need to choose between reusing learning rate schedule or tune it. Because one can think of training steps as a regularizer the authors use that to build a restriction on the parameters adjustments. They show we misjudge a model ability to adjust and compare optimal to practical updates of weights. That in practice allow them to predict if there is a better scheduler.

Critical strengths

I find this results novel and it goes beyond theoretical prediction as the authors also showed it works in practice. The problem itself is important and can assist in optimizing model optimization cost.

Critical weaknesses

The models used here, both theoretical and experimental, are too simple and only relevant for linear filters. That limitation reduce the usefulness of this approach dramatically but keeps the insights interesting.

Questions for authors

How would you tackle or approximate non-linear filters for the same question?

2 comments
AuthorsOpenReview ↗

Response to Reviewer KRTK (Part 1/2):

Thank you for recognizing the novelty and practical relevance of our work, and for your insightful question on extending our framework to nonlinear models. Below, we address concerns step by step.

1. How can we tackle or approximate nonlinear models for this question?

Thank you for raising this central question, which bridges our exact spectral-filter theory and practical deep-learning workflows. Two complementary levels of approximation: We can tackle nonlinear models at two distinct levels: (i) applying our exact finite-step filter operations to a quadratic head on frozen nonlinear representations (where preservation, reuse, and retuning execute directly), or (ii) using a second-order trajectory-differentiation surrogate around schedule reuse when updating all parameters end-to-end.

  • Level 1: Frozen nonlinear representations (Office-Home Benchmark, p=513p=513): A pretrained nonlinear backbone provides fixed features, and a quadratic head is trained for KK gradient updates, retaining our exact finite-step spectral-filter representation F0=I−∏k=1K(I−α0,kS0)F_0 = I - \prod_{k=1}^K(I-\alpha_{0,k}S_0). Across three Office-Home transfers (Real_World to Art, Clipart, and Product), K∈{100,300}K\in\{100,300\}, and three seeds (18 configurations), reuse and calibration-selected retuning consistently outperform preservation:

    Table 1a. Office-Home quadratic heads on frozen ResNet-18 features (p=513p=513).

    StrategyMean test MSEBeats preservationBeats reuse
    Retain source parameters (F0z0F_0 z_0)0.013850/180/18
    Preserve source filter (F0ztF_0 z_t)0.01193--3/18
    Reuse source schedule (K=100/K=300K=100 / K=300)0.0116115/18 (6/9, 9/9)--
    Retune schedule (5% radius, 7-cand.)0.0115317/1818/18
    Retune schedule (60% radius, uniform / 7-cand.)0.0112618/1818/18

    At a 60% radius, uniform-scaling and 7-candidate validation selected identical uniform rate increases across all 18 configurations. These natural transfers demonstrate how the four operations execute in high dimensions, while direct application of our local theorem additionally requires its source-optimality and local task-path assumptions.

  • Level 2: Full-trajectory approximation for end-to-end training: Let θK(q)\theta_K(q) denote the network parameters after KK updates under a low-dimensional schedule perturbation qq around reuse, with target calibration loss L(q)=Lcal(θK(q))L(q) = L_{\mathrm{cal}}(\theta_K(q)). We approximate this post-training loss via the second-order Taylor surrogate:

    L^(q)=L(0)+g⊤q+12q⊤Hq,g=∇qL(0),H=∇q2L(0),\widehat{L}(q) = L(0) + g^\top q + \tfrac{1}{2} q^\top H q, \qquad g = \nabla_q L(0), \quad H = \nabla_q^2 L(0),

    where gg and HH propagate through the entire KK-step trajectory. This surrogate selects among a fixed menu of schedules using target calibration labels: it addresses local schedule selection around reuse, whereas our paper's Γs\Gamma_s criterion addresses the population-risk comparison between optimal retuning and preservation.

2. How does updating the hidden-layer parameters change the output geometry?

Continuous transition from singular quadratic to full-rank first-order geometry: Updating hidden-layer parameters progressively removes the equal-rate Jacobian degeneracy of the fixed-feature model, shifting the two-step split response from second order (slope 2.000) to first order (slope 0.962) and making the rate Jacobian numerically rank two in 24/24 initializations.

To quantify this transition, we compared matched 64-unit tanh networks on Digits (K=2K=2 full-batch gradient updates with squared loss and a zero-initialized head) across varying ratios of hidden-to-head learning rates ηhid/ηhead∈{0,0.01,0.1,1.0}\eta_{\mathrm{hid}}/\eta_{\mathrm{head}} \in \{0, 0.01, 0.1, 1.0\}:

Table 1b. Equal-step output geometry across feature-learning strength in 64-unit tanh networks (K=2K=2).

Feature-learning regime (ηhid/ηhead\eta_{\mathrm{hid}}/\eta_{\mathrm{head}})Split-response slope, median [IQR]Rate-Jacobian σ2/σ1\sigma_2/\sigma_1, min–maxNumerically rank-two Jacobian
Frozen features (ηhid/ηhead=0\eta_{\mathrm{hid}}/\eta_{\mathrm{head}} = 0)2.000 [2.000, 2.000]2.53×10−17–2.40×10−162.53 \times 10^{-17}\text{–}2.40 \times 10^{-16}0/24
Weak feature updates (ηhid/ηhead=0.01\eta_{\mathrm{hid}}/\eta_{\mathrm{head}} = 0.01)1.842 [1.815, 1.868]5.12×10−5–2.08×10−45.12 \times 10^{-5}\text{–}2.08 \times 10^{-4}24/24
Moderate feature updates (ηhid/ηhead=0.10\eta_{\mathrm{hid}}/\eta_{\mathrm{head}} = 0.10)1.285 [1.240, 1.332]5.18×10−4–2.01×10−35.18 \times 10^{-4}\text{–}2.01 \times 10^{-3}24/24
Joint updates (ηhid/ηhead=1.00\eta_{\mathrm{hid}}/\eta_{\mathrm{head}} = 1.00)0.962 [0.958, 0.971]0.00538–0.020330.00538\text{–}0.0203324/24
AuthorsOpenReview ↗

Response to Reviewer KRTK (Part 2/2):

  • Experimental details: All conditions share the same hidden initialization, nominal head learning rate aa, and fixed Digits split (1,347 training and 450 held-out images). For each of 24 paired initializations, we fit the log–log slope of the RMS prediction change between (a+h,a−h)(a+h,a-h) and (a,a)(a,a) over h/a∈{2−8,…,2−12}h/a \in \{2^{-8},\ldots,2^{-12}\} across all held-out images and 10 output coordinates (brackets report the IQR across initializations). The rate Jacobian is evaluated at (a,a)(a,a), and rank-two counts apply the descriptive threshold σ2/σ1>10−8\sigma_2/\sigma_1 > 10^{-8}.

3. Can the trajectory approximation select useful schedules for nonlinear models?

Effective within a local trust region (5%–15% radius) on short trajectories, while low-dimensional direct validation dominates on long trajectories: The second-order trajectory surrogate accurately matches direct validation within a local schedule-change radius (5%–15%) on short trajectories and outperforms first-order Taylor selection, whereas at large radii (60%) or over 100–300 ResNet-18 updates—where exact unrolled differentiation hits memory limits and finite-difference curvature becomes step-sensitive—direct validation over our low-dimensional schedule menu is both more accurate and faster.

  • Short trajectories across trust-region radii (shifted/blurred Digits, 64 configurations): We evaluated K∈{4,12}K\in\{4,12\} updates with 180 calibration images and 417 disjoint test images (after an exploratory step stabilizing raw source rates using only training-loss decreases; all test cross-entropy [CE] changes below are relative to that stabilized reference):

Table 2a. Digits trajectory-surrogate schedule selection across perturbation radii (64 configurations).

Radius2nd-order match with 7-cand.1st-order match with 7-cand.7-cand. validation Δ\DeltaCEUniform-scaling Δ\DeltaCE2nd-order surrogate Δ\DeltaCE
5% radius64/64 (100%)52/64 (81.3%)-0.0293-0.0283-0.0293
15% radius58/64 (90.6%)41/64 (64.1%)-0.0418-0.0365-0.0396
30% radius39/64 (60.9%)28/64 (43.8%)-0.0502-0.0441-0.0114
60% radius18/64 (28.1%)19/64 (29.7%)-0.0551-0.0489+0.1404

At a 5% radius, second-order selection matched seven-candidate validation in 64/64 configurations, reducing mean test CE by 0.0293 (improving test CE in 60 configurations, tying in 1, and worsening in 3). Including the Hessian term HH substantially improves agreement over first-order Taylor at 5%–15% radii, whereas by 60% radius higher-order trajectory nonlinearity dominates (+0.1404 vs. -0.0551), cleanly delineating the local trust region.

  • Longer trajectories and finite-difference (FD) diagnostics (full-network ResNet-18 on Office-Home, 18 configurations): Across three transfers, K∈{100,300}K\in\{100,300\}, and three seeds, we evaluated full-network schedule selection. Exact unrolled automatic differentiation exceeded memory at K=30K=30 updates on a 96 GiB GPU, and our prespecified central-FD fallback exhibited severe step-size sensitivity over long horizons: at K=30K=30, a single FD curvature entry shifted from 67.0 to 1.59 to -0.99 across FD step sizes {0.005,0.01,0.02}\{0.005, 0.01, 0.02\}.

Table 2b. Office-Home full-network ResNet-18 selectors at 5% and 60% schedule-change radii.

SelectorMean Δ\DeltaCE vs. reuse (5% radius)Mean Δ\DeltaCE vs. reuse (60% radius)Target tuning time (s)
Uniform-scaling validation (2-cand.)-0.0148-0.054939.4
Seven-candidate validation-0.0162-0.061489.2
First-order FD-0.0139-0.0228101.7
Second-order FD-0.0115+0.0064101.7

Negative CE differences indicate improvement over reuse, and tuning times are reconstructed from measured stages under a no-cache deployment accounting. While FD selectors achieve modest gains at a 5% radius (-0.0139 and -0.0115), FD curvature noise degrades second-order FD at a 60% radius (+0.0064). In contrast, direct validation over our two-candidate uniform-scaling menu captures nearly the full gain of seven-candidate validation (-0.0549 vs. -0.0614) in just 39.4 s.

4. Presentation and readability improvements.

Finally, we appreciate your feedback on readability: in the revised manuscript, we will substantially streamline the Abstract, Introduction, and Figure 1 caption by opening with a concrete KK-step gradient-descent workflow and two-step example before introducing operator notation.

Reviewer_2EnDOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Paper is about fixed-budget retuning. Although I don't understand what that means (it's not common ML terminology, it might be about tuning the learning rate schedule when a model needs to be trained on a new task). As I understand, paper proves certain results about finite-step optimization for linear models in arbitrary dimensions. Simulation experiments on synthetic gaussian and a real bike-sharing dataset.

Critical strengths
  • Notation and theorems look rigorous, and the paper's extension of finite-step optimization to higher dimensional problems will be useful for the optimization community.
Critical weaknesses
  • Paper is extremely hard to read. I read the abstract and intro several times, and still don't understand what ML problem they are trying to solve. Can authors please rewrite the paper?
  • The results might be interesting, but the paper is unreadable and reeks of heavy AI generated content. I can't even understand related work. Line 292 reads "A decision reversal missed by the Jacobian." - why does a research paper needs to have click-bait paragraph titles.
  • What to read from Figure 1? The axes are not explained in clear words.
  • Only 1 real-world dataset of bike-sharing, and how that fits to the theory or used in this paper is not explained.
Questions for authors
  • What is fixed-budget retuning? What architecture / optimization or LR schedule algorithm is the paper about?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 2EnD (Part 1/2):

We sincerely thank you for your candid feedback. Because this is a theoretical paper involving many definitions and derivations, we got too caught up in abstract operator notation and failed to explain the basic machine-learning setup, intuition, and figures clearly in plain language. We appreciate your patience with the abstract notation, and we are grateful for the opportunity to clarify these points. Below, we provide clear, plain-language explanations for every point you raised, followed by substantial new experiments that expand our empirical evaluation well beyond a single dataset.

Q1 & W1: What fixed-budget retuning is, and what model, optimizer, and LR schedule we study.

We thank you for prompting us to state the concrete machine-learning setup explicitly, up front, in the Abstract and Introduction. Re-selecting KK scalar learning rates of KK-step gradient descent on a shifted quadratic task: Fixed-budget retuning simply means choosing KK new scalar learning rates α=(α1,…,αK)\alpha = (\alpha_1, \dots, \alpha_K) for KK-step gradient descent when adapting a linear predictor to a shifted target task, rather than reusing the old source learning rates or applying the saved source linear map.

  • Concrete machine-learning setup:

    • Model / Architecture: Linear predictors trained with squared loss—specifically, standard linear regression or a linear/quadratic prediction head trained atop frozen pretrained neural-network features.

    • Optimizer & LR Schedule: Full-batch gradient descent initialized at zero and run for a fixed budget of exactly KK updates, assigning one scalar learning rate αk\alpha_k to each update step k=1,…,Kk=1,\dots,K (hyperparameter search and preprocessing costs are tracked separately from the KK execution updates).

    • ML Problem & Four Executable Operations: On a quadratic objective with data curvature matrix SS, running KK gradient updates defines a linear map Fα(S)=I−∏k=1K(I−αkS)F_\alpha(S) = I - \prod_{k=1}^K (I - \alpha_k S) from the sufficient statistic zz to predictions. When the task shifts from source (S0,z0S_0, z_0) to target (St,ztS_t, z_t), we compare four concrete operations:

      OperationOutput predictorPlain-language meaning
      Retain source parametersFα0(S0)z0F_{\alpha_0}(S_0) z_0Keep the source predictor as-is; no target fitting
      Preserve source filterFα0(S0)ztF_{\alpha_0}(S_0) z_tApply saved source linear map to target regression statistic ztz_t
      Reuse source scheduleFα0(St)ztF_{\alpha_0}(S_t) z_tRun the old learning rates α0\alpha_0 on the target curvature StS_t
      Retune scheduleFα(St)ztF_\alpha(S_t) z_tSelect KK new learning rates α\alpha for the KK target updates
  • Intuitive two-step (K=2K=2) scalar example: Why does this lead to a nontrivial theoretical question? Along a curvature direction λ\lambda, running two equal learning rates (a,a)(a, a) produces the scalar gain 1−(1−aλ)21 - (1 - a\lambda)^2, whereas splitting the two rates into (a+h,a−h)(a + h, a - h) adds h2λ2h^2 \lambda^2. Although the first derivative with respect to hh at h=0h = 0 is zero, choosing hh on the scale ∣r∣\sqrt{\vert{}r\vert{}} produces an output change of order ∣r∣\vert{}r\vert{}—matching the scale of a task shift of size rr. Consequently, a standard first-order Jacobian analysis of the learning rates misses beneficial schedule adjustments at equal steps.

W1 & W2: Rewriting the prose, Related Work, and paragraph headings.

We sincerely thank you for identifying the dense theoretical presentation phrasing as readability problems, which we address with a rewrite below. Polished rewrite of the Introduction, Related Work, and headings: We substantially reorganized the Abstract, Introduction, Related Work, and section headings in plain, direct prose. We strip out ornate phrasing throughout the manuscript and replace headings such as Line 292 ("A decision reversal missed by the Jacobian") with neutral descriptive titles ("Risk comparison for two equal steps").

  • Reorganizing Related Work around specific prior results: We restructure Section 2 so each paragraph directly states which prior mathematical or algorithmic result we build on and what our finite-step analysis adds—covering classical tangent-cone risk projection (Chernoff, 1954; Drton, 2009), singular parameterizations (Levin et al., 2025), and polynomial spectral filters in linear networks (Kohn et al., 2022, 2024).
AuthorsOpenReview ↗

Response to Reviewer 2EnD (Part 2/2):

W3: What to read from Figure 1 and plain-language axis definitions.

Visual verification of our risk formula and where two simplified approximations fail: Figure 1 delivers two concrete messages: panels (a, b) verify that our theoretical formula accurately predicts whether retuning beats preservation across synthetic quadratic tasks, and panel (c) shows concrete tasks where two simplified approximations recommend the wrong action.

  • Panels (a, b) — Theory vs. exact numerical optimization:
    • Each point: Represents one synthetic quadratic problem and task-shift direction.
    • Horizontal axis: Our theoretical leading risk-gap coefficient Γs\Gamma_s.
    • Vertical axis: The directly computed numerical optimum of (retuned risk−preserved risk)/r2(\text{retuned risk} - \text{preserved risk}) / r^2 at a small task shift rr.
    • How to read it: Negative values mean retuning achieves lower test risk than preservation; points lying on the 45-degree diagonal confirm that our closed-form theory matches numerical optimization.
  • Panel (c) — Why simplified benchmarks fail:
    • Horizontal axis: The risk-gap coefficient predicted by a simplified approximation (either the unrestricted commuting filter benchmark or the first-order learning-rate Jacobian).
    • Vertical axis: Our exact KK-step schedule coefficient.
    • How to read it: Points in the upper-left quadrant are tasks where the unrestricted commuting benchmark wrongly recommends retuning when preservation is actually better (Γcomm<0<Γs\Gamma_{\mathrm{comm}} < 0 < \Gamma_s); points in the lower-right quadrant are equal-step tasks where the first-order Jacobian misses a beneficial retuning direction (Γs<0<ΓJac\Gamma_s < 0 < \Gamma_{\mathrm{Jac}}).

W4: How Bike Sharing fits the theory, and new experiments on Office-Home (ResNet-18), Digits, and high-dimensional tasks.

Clarified role of Bike Sharing plus substantial new experiments across ResNet-18 transfers (p=513p=513), finite-sample Digits prediction, and p=1024p=1024 checks: Bike Sharing is a linear regression diagnostic used to illustrate how the executable strategies behave in practice and what target data each requires, whereas our controlled synthetic and Digits experiments directly test the local risk formulas and finite-sample decision criterion Γ^s\widehat{\Gamma}_s. To expand our empirical evaluation well beyond a single dataset, we added three sets of experiments:

  • 1. Clarifying Bike Sharing and protocol sensitivity (Section 5 & Appendix I.5, I.8): On Bike Sharing (K=2K=2 linear regression across 12 target months after a year shift), preservation applies the saved source filter F0F_0 to a target training regression solution ztz_t, reuse runs the source rates on target curvature StS_t, and retuning selects rates on target validation data. Across the preprocessing protocols in Appendix I.5/I.8, retuning achieves lower test MSE than preservation in 9/12 months under the original protocol (median relative gap 17.0%) and in 8/12 months after domain centering with a shared target intercept (median relative gap 1.83%, matching reuse in 9/12 months). We will state explicitly at the start of Section 5 that Bike Sharing is an operational strategy comparison rather than a direct test of Γs\Gamma_s.

  • 2. New deep-learning transfers on Office-Home with pretrained ResNet-18 (36 configurations): Across three transfers (Real_World to Art, Clipart, and Product), K∈{100,300}K \in \{100, 300\} updates, and three seeds, we evaluated both a frozen quadratic head (p=513p=513, where our linear filter operations execute exactly) :

    Table 1. Office-Home transfers using ImageNet-pretrained ResNet-18 (K∈{100,300}K \in \{100, 300\}).

    Setting & StrategyMean test lossBeats preservationBeats reuse
    Frozen head : Retain parameters (F0z0F_0 z_0)0.013850/180/18
    Frozen head : Preserve filter (F0ztF_0 z_t)0.01193--3/18
    Frozen head : Reuse schedule (Fα0(St)ztF_{\alpha_0}(S_t)z_t)0.0116115/18--
    Frozen head : Retune schedule (60% radius)0.0112618/1818/18
  • 3. Direct finite-sample prediction of Γs\Gamma_s on Digits (p=8,K=2p=8, K=2) and high-dimensional checks (p≤1024p \le 1024): To test whether Γs\Gamma_s can be estimated from finite data before observing target evaluation risks, we added a controlled Digits experiment (1,347 source images, 450 disjoint evaluation images). As calibration batch size increases across n∈{512,2048,8192}n \in \{512, 2048, 8192\}, our schedule-specific estimator Γ^s\widehat{\Gamma}_s matches the held-out optimal retune-versus-preserve decision in 78.9%, 92.2%, and 99.2% of repetitions (and 75.4%, 90.6%, 98.4% when the nominal filter F^0\widehat{F}_0 is also learned from source data), compared with 50.0% for the simplified benchmark.

Thank you for engaging during the rebuttal period. I acknowledge On the positive, the PDF is a major rewrite of the submitted on the manuscript. On the other hand, it is not true to the submission and still calls for better writing.

Writing. The black text still has issues. E.g. in the abstract itself, not sure what the phrase "unrestricted commuting optimum" or sentences "O(p|r|) step splitting generates first-order output corrections missed by the parameter Jacobian, where r indexes the local task change. For two steps, this singular geometry is the endpoint of a near-equal transition governed by δ2r /|r|, where δr is the nominal half-gap." mean.

An abstract should be accessible to a broad ML audience, but what does "step splitting generates first-order output corrections missed..." and "endpoint of a near-equal transition governed by ..." and "nominal half-gap" mean?

The blue text also has issues. A lot of the experiment section is still hard to read with nouns and verbs that are not accessible to an ML audience. Experiments also point to the appendix, which is unparseable. What are even Tables 4 and 5, are they necessary in the appendix? The new appendices J and K also have these issues that are clearly AI slop writing e.g.

In contrast, direct validation over our two-candidate uniform-scaling menu captures nearly the full test CE reduction of seven-candidate validation (−0.0549 versus −0.0614) in just 39.4seconds, providing an efficient and numerically robust schedule selection rule for long deep-network trajectories.

The blue text rewrites are in active voice, I recommend rewriting the black text in active voice as well.

Problem setup. I now understand the problem setup better. You want to train a model for K training steps and want to use a different scalar learning rate for each training step. That is clear now, though I feel it's not realistic? Usually there's a constant LR or a smooth and continuous schedule for training a model. That's why I think the theory and experiments are very detached.

I'd suggest rewriting the paper after digesting your results at length. I'm keeping my score to 1.

Reviewer_h4BJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper examines when retuning a finite-step learning-rate schedule after a task change is preferable to preserving the original spectral filter, given a fixed number of update steps. Rather than analyzing parameter changes directly, the authors characterize the local risk difference through the distance between the oracle drift and the one-sided set of reachable output corrections.

For a simple spectrum and pairwise distinct interior step sizes, the paper shows that the reachable correction space has dimension min⁡(K,p)\min(K,p). This identifies adjustments that an unrestricted spectral filter could make but a K-step schedule cannot. For equal step sizes, the authors show that splitting the steps by O(∣r∣)O(\sqrt{|r|}) yields first-order output directions missed by the ordinary parameter Jacobian. They also characterize the transition between singular and regular behavior for nearly equal two-step schedules. The results are examined through low-dimensional Gaussian experiments and a Bike Sharing comparison of preservation, schedule reuse, and validation-selected retuning.

Critical strengths
  1. The paper offers a meaningful characterization of finite-step retuning in output space. The min⁡(K,p)\min(K,p) result makes the reachability constraint imposed by a fixed update budget explicit, while the equal-step-size analysis identifies directions that a conventional Jacobian analysis misses. These results clarify why unrestricted spectral adjustment and first-order parameter linearization can both misrepresent the benefits of retuning. I consider this the paper's strongest contribution.

  2. The theoretical development is careful and supported by useful checks. The appendix addresses localization, alternative representations of the nominal schedule, one-sided reachability, finite-perturbation errors, and the transition near equal step sizes. The appendix provides explicit risk coefficients and sign-reversal constructions that make the claims concrete and inspectable. The experiments also report deviations from the local predictions at larger perturbations, bootstrap coverage shortfalls, and sensitivity to the real-data evaluation protocol. This transparency helps distinguish the proved local results from their empirical limitations.

Critical weaknesses
  1. The connection to practical learning-rate retuning remains limited. The geometric results rely on local task changes, fixed dimensionality and update budget, response-independent linear spectral filters, population-level quantities, and an optimality condition at the nominal point. The numerical checks are restricted to p≤5p \leq 5 and K≤3K \leq 3. Moreover, applying the fixed nominal filter F0F_0 to target-domain inputs, as in the paper's preservation baseline, is not generally equivalent to retaining source-model parameters. The theory is valuable within its stated setting, but its implications for high-dimensional, nonlinear training workflows remain uncertain.

  2. The main decision criterion is not directly validated with finite-sample or real-data estimates. The theoretical comparison depends on the algorithm-specific attainable-risk coefficient Γs\Gamma_s. The synthetic experiments assess the local expansion using known population quantities, whereas the appendix's finite-sample analysis primarily concerns Γcomm\Gamma_{\mathrm{comm}}, a commuting benchmark that provides only one-sided evidence for the algorithm-specific decision in general. The Bike Sharing experiment illustrates differences among preservation, schedule reuse, and retuning, but does not test whether Γs\Gamma_s can be estimated from data to predict when retuning will help. I therefore view the experiments as validation of the local geometric analysis rather than evidence for a practical decision rule.

Questions for authors
  1. How should the preservation baseline be interpreted in a realistic training workflow? In what settings can a fixed F0F_0 be constructed and executed directly, and how does this differ from retaining pretrained parameters or reusing the original learning-rate schedule?

  2. Can the algorithm-specific attainable-risk coefficient Γs\Gamma_s be estimated from finite samples and used to predict whether preservation or retuning will perform better, without selecting the strategy using observed target-task risk? If the current experiments do not test this, could the authors clarify that the Bike Sharing results are illustrative rather than a direct validation of the decision criterion?

  3. Which results are intended as the paper's principal theorem-level novelty relative to prior work on tangent-cone projections, singular parameterizations, and polynomial spectral filters? In particular, is the main advance the general risk-projection formulation or the characterization of finite-step reachability and degeneracy at equal step sizes?

5 comments
AuthorsOpenReview ↗

Response to Reviewer h4BJ (Part 1/3):

Thank you for your careful assessment and for recognizing the finite-step reachability and equal-rate results as the paper's strongest contributions. Below, we address each of your questions step by step.

Q1 & W1: How is preservation executed, and how does it differ from retaining parameters or reusing the schedule?

Thank you for this insightful question, which goes to the heart of how our operator-level formulation connects to practical adaptation workflows.

Executable in fixed-feature quadratic models: Preservation is directly executable whenever adaptation acts in fixed feature coordinates with a quadratic objective (such as linear regression or quadratic heads on frozen nonlinear representations) by applying the stored source filter F0F_0 to the target statistic ztz_t, whereas retaining parameters freezes the source predictor F0z0F_0 z_0 without target fitting and reusing the schedule runs the source learning rates on the target curvature Fα0(St)ztF_{\alpha_0}(S_t)z_t.

  • Operator definitions and cost accounting: Storing the source filter

    F_0 = I - \prod_{k=1}^{K}(I - \alpha_{0,k}S_0) \end{xalignat*}$$ and applying it in the shared feature coordinates defines four distinct operations: | Operation | Quadratic output | Mechanism | | --- | :---: | --- | | Retain source parameters | $F_0 z_0$ | Keep source predictor; no target fitting | | Preserve source filter | $F_0 z_t$ | Apply source map to target statistic | | Reuse source schedule | $F_{\alpha_0}(S_t) z_t$ | Run source rates on target curvature | | Retune schedule | $F_\alpha(S_t) z_t$ | Choose new rates in $K$-step family | In our least-squares implementation, $z_t$ is a regression solution computed from target training data. Preservation therefore uses target labels and a linear solve, whereas retaining source parameters requires no target fitting. Throughout the paper, the budget $K$ counts only final gradient updates, with preprocessing and schedule-search costs accounted for separately.
  • Generalization to higher pp and KK across budgets and radii (Office-Home Benchmark, p=513p=513): On an ImageNet-pretrained ResNet-18 with a quadratic head trained on frozen features (p=513p=513), preservation applies F0F_0 directly in the shared feature coordinates across three transfers (Real_World to Art, Clipart, and Product), K∈{100,300}K\in\{100,300\}, and three seeds (18 configurations total):

    Table 1a. Office-Home Benchmark quadratic heads on frozen ResNet-18 features (p=513p=513).

    StrategyMean test MSEBeats preservationBeats reuse
    Retain parameters (F0z0F_0 z_0)0.013850/180/18
    Preservation (F0ztF_0 z_t)0.01193--3/18
    Schedule reuse (K=100/K=300K=100 / K=300)0.0116115/18 (6/9, 9/9)--
    Retuning (5% radius, 7-cand.)0.0115317/1818/18
    Retuning (60% radius, uniform / 7-cand.)0.0112618/1818/18

    At a 60% radius, uniform-scaling (2 candidates) and 7-candidate validation selected identical uniform rate increases across all 18 configurations, both achieving 0.01126 mean test MSE. This demonstrates how all four operations execute and scale at higher pp and KK (see Part 3, Table 3a for synthetic population-risk checks up to p=1024p=1024).

  • What changes when backbone features are updated (K=2K=2 tanh networks): To quantify how end-to-end feature learning alters our linear operator assumptions, we compared matched 64-unit tanh networks on Digits (1,347 training and 450 held-out images across 24 random initializations, K=2K=2) with either frozen hidden features or both layers trainable:

    Table 1b. Equal-step output geometry in matched 64-unit tanh networks (K=2K=2).

    MetricFrozen featuresBoth layers trained
    Split-response slope, median [IQR]2.000 [2.000, 2.000]0.962 [0.958, 0.971]
    Rate-Jacobian σ2/σ1\sigma_2/\sigma_1, min–max2.53×10−17–2.40×10−162.53\times 10^{-17}\text{–}2.40\times 10^{-16}0.00538–0.020330.00538\text{–}0.02033
    Numerically rank-two Jacobian0/2424/24

    Here, the log-log slope measures RMS prediction change between (a+h,a−h)(a+h,a-h) and (a,a)(a,a) across h/a∈{2−8,…,2−12}h/a \in \{2^{-8},\dots,2^{-12}\}, and rank-two counts use σ2/σ1>10−8\sigma_2/\sigma_1 > 10^{-8}. Frozen features exactly recover the quadratic split response and rank-one Jacobian predicted by our linear theory, whereas training both layers makes the response first-order and the rate Jacobian rank-two in all 24/24 initializations.

AuthorsOpenReview ↗

Response to Reviewer h4BJ (Part 2/3):

Q2 & W2: Can the algorithm-specific coefficient be estimated before observing candidate target-task risks?

Thank you for highlighting this crucial distinction between verifying a population formula and testing a data-driven decision rule. Validated via finite-sample calibration: Yes, in a controlled local setting satisfying nominal optimality, a predictor can estimate the schedule-specific coefficient Γs\Gamma_s from finite calibration samples and choose between preservation and retuning without access to candidate target-task evaluation risks, holding both for an exact nominal filter F0F_0 and for a source-estimated filter F^0\widehat{F}_0.

  • Controlled Digits finite-sample prediction (p=8,K=2p=8, K=2): We use eight-dimensional PCA signals with a constructed observation channel that ensures nominal optimality. Curvature and its task derivative are known, while the oracle drift is withheld from the estimator: two labelled batches at nearby task perturbations estimate that drift, and an unlabelled nominal batch estimates the risk metric over the actual two-step reachable set.

    Table 2. Agreement with held-out optimal retune-versus-preserve decisions at r=0.01r=0.01.

    Predictorn=512n = 512n=2,048n = 2,048n=8,192n = 8,192
    Distinct-step Γ^s\widehat{\Gamma}_s (known F0F_0)78.9%92.2%99.2%
    Distinct-step Γ^s\widehat{\Gamma}_s (learned F^0\widehat{F}_0, nsrc=nn_{\mathrm{src}}=n)75.4%90.6%98.4%
    Equal-step cone Γ^2,+\widehat{\Gamma}_{2,+} (known F0F_0)76.2%91.0%98.8%
    Commuting benchmark Γ^comm\widehat{\Gamma}_{\mathrm{comm}}50.0%50.0%50.0%
    Equal-step Jacobian Γ^Jac\widehat{\Gamma}_{\mathrm{Jac}}53.5%51.2%50.0%
  • Experimental protocol : Four balanced controls comprise two preservation-favorable and two retuning-favorable cases (256 calibration repetitions per control), designed so that the two preservation-favorable distinct-step controls satisfy the commuting reversal Γcomm<0<Γs\Gamma_{\mathrm{comm}} < 0 < \Gamma_s and the two retuning-favorable equal-step controls satisfy the Jacobian reversal Γ2,+<0<ΓJac\Gamma_{2,+} < 0 < \Gamma_{\mathrm{Jac}}. Each repetition draws two labelled batches and one unlabelled nominal batch of size nn by resampling a fixed pool of 1,347 source images with fresh observation noise (2n2n labelled and nn unlabelled observations per repetition; nn does not count distinct images), and evaluates on 450 disjoint images (where local risk differences are ≈10−5\approx 10^{-5} in squared-error units). Consistent with Proposition B.2 and Eq. (H.14), replacing F0F_0 with a source-estimated reference F^0\widehat{F}_0 incurs only a modest small-sample penalty at n=512n=512 (75.4% vs. 79.0%) and reaches 98.4% at n=8,192n=8,192.

  • Natural-data scope and finite-sample rate selection: Bike Sharing and Office-Home Benchmarks compare executable strategies using held-out evaluation rather than testing Γs\Gamma_s directly. In a separate cross-year Beijing sensor check across 1,320 ordered station triples at a 2% blend, Γ^s\widehat{\Gamma}_s agreed with the evaluation-year empirical-oracle decision in 90.2% of cases versus 99.5% for always predicting retuning, because nominal optimality is not preserved across years.

AuthorsOpenReview ↗

Response to Reviewer h4BJ (Part 3/3):

Q3: Which results are the principal theorem-level contributions?

We appreciate this constructive question, which helps us sharpen the positioning of our theory relative to existing literature. Finite-step reachability and singular equal-rate geometry: Our principal theorem-level contribution is the characterization of the complete one-sided attainable output sets for the positive-step product family as the task operator changes (Theorems 4.2–4.3 and Propositions 4.1–4.2), whereas Theorem 4.1 provides the general quadratic risk-projection framework onto which the oracle drift is projected.

  • Distinct interior rates (Theorem 4.2 and Proposition 4.1): Under a simple spectrum, the reachable correction space has dimension min⁡(K,p)\min(K,p), and our restriction-cost formula quantifies the nonnegative risk cost of directions available to the unrestricted commuting benchmark but inaccessible to the KK-step schedule (which depends on task drift and can vanish even when K<pK<p).
  • Equal rates (Theorem 4.3): Step splitting on the scale O(∣r∣)O(\sqrt{|r|}) generates an additional one-sided output ray captured by the complete attainable set but omitted by the ordinary parameter Jacobian.
  • Nearly equal two-step schedules (Proposition 4.2): The scaling ratio δr2/∣r∣\delta_r^2 / |r| governs the continuous transition between the equal-rate singular geometry and the distinct-rate regular regime.

W1: Do the numerical checks extend beyond p≤5p\leq5 and K≤3K\leq3?

Thank you for pointing out the limited dimensional range of the initial synthetic sweep. Verified up to p=1024p=1024, K=10K=10, and under rotating curvature and noncommuting weights: our expanded synthetic population-risk checks with known moments confirm that the local coefficient's predicted sign and reachability threshold hold accurately across high dimensions, longer schedules, rotating eigenbases, and noncommuting metrics.

Table 3a. Higher-dimensional and multi-step checks of the local risk prediction.

GeometryDimensions and budgetRadius ∣r∣\lvert r\rvertSign agreement
Diagonal curvaturep=1024p=1024, K∈{2,4,8}K\in\{2,4,8\}10−610^{-6}1,532/1,536 (0 false, 4 inconcl.)
Diagonal curvature (moderate ∣r∣\lvert r\rvert)p=1024p=1024, K∈{2,4,8}K\in\{2,4,8\}10−310^{-3}1,530/1,536 (99.6% agreement)
Rotating curvature (K=2K=2)p∈{16,64,128}p\in\{16,64,128\}, K=2K=20.0020.0022,304/2,304 (exact 2-step opt.)
Rotating curvature (K>2K>2)p∈{16,64,128}p\in\{16,64,128\}, K∈{4,8}K\in\{4,8\}10−510^{-5}1,530/1,536 (0 false, 6 inconcl.)
Noncommuting metric (WS0≠S0WWS_0 \neq S_0W)p∈{16,64,128}p\in\{16,64,128\}, K=2K=210−310^{-3}768/768 (slope 1.000)
  • Numerical bracketing and K→pK \to p reachability transition: For K>2K>2, we bracket the optimum using a lower bound from a relaxed filter family and an upper bound from feasible schedules (at p=1024,∣r∣=10−6p=1024, |r|=10^{-6}, agreement is 512/512 at K=2K=2, 511/512 at K=4K=4, and 509/512 at K=8K=8, with all 4 inconclusive cases having bracket intervals straddling zero and 0 false signs). Furthermore, varying K∈{2,4,6,8,10}K \in \{2, 4, 6, 8, 10\} at fixed dimension p=8p=8 directly confirms the min⁡(K,p)\min(K,p) phase transition in Theorem 4.2:

Table 3b. Reachability phase transition across K/pK/p at fixed dimension (p=8p=8).

Budget KKRank min⁡(K,p)\min(K,p)Normalized restriction costCommuting reversal rate
K=2K = 2 (K/p=0.25K/p = 0.25)20.76225.0%
K=4K = 4 (K/p=0.50K/p = 0.50)40.28410.9%
K=6K = 6 (K/p=0.75K/p = 0.75)60.0583.1%
K=8K = 8 (K/p=1.00K/p = 1.00)8 (Full)<10−15< 10^{-15}0.0%
K=10K = 10 (K/p=1.25K/p = 1.25)8 (Full)<10−15< 10^{-15}0.0%

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.