acceptodds
Under review as a conference paper at ICLR 2027

Optimal Training-Time Scaling in Gradual Adaptation

Abstract

In gradual adaptation, how should the training time on each task change as the number of intermediate tasks increases? We study this question for overparameterized linear regression tasks that change smoothly and share a zero-loss solution. With tasks and training time on each, the final learning progress converges to a continuum curve when . The limiting progress is for small and for large , so both very short and very long training produce little progress. It follows that optimal per-task training times scale as , equivalently . Experiments on gradually rotated MNIST and a natural Yearbook time shift are consistent with less per-task training as the path is divided more finely.

Primary area:
learning theory
Keywords:
gradual adaptation, continual learning, training-time scaling, gradient flow, curriculum learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_g4y4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2(−1)
Soundness
2(−1)
Read review
Paper summary

The paper studies the optimal local training time sNs_N for training NN tasks in overparameterized linear regression. By adopting the effective training time τ=NsN\tau=Ns_N, the main theorem shows that learning grows as Θ(τ)\Theta(\tau) for small τ\tau and drops as Θ(τ−1)\Theta(\tau^{-1}) for large τ\tau. Consequently, it shows that the optimal local training time sNs_N is Θ(N−1)\Theta(N^{-1}). The prediction was tested on rotated MNIST and temporal Yearbook data.

Critical strengths
  • The paper is well-written.
  • Building on the setting of previous work on gradual adaptation in overparameterized linear regression, the paper provides a novel theoretical explanation for the optimal local training time sNs_N for training NN tasks.
Critical weaknesses
  • Experiment design

The experiments involve classification tasks, which contradicts the theoretical setting. In overparameterized linear regression, the optimal solution closest to the initial point is obtained by projection onto the null space (which is the basis of the analysis in the paper). However, in classification, the optimal solution diverges to infinity as the loss decreases to zero. Therefore, to test whether the predicted training-time trend persists in a real-world scenario, the task should be a regression task.

  • Practical implication

The prediction from the theory seems to have weak practical implications. To be specific, Theorem 2 shows that NsN∗=Θ(1)N s_N^* = \Theta(1). If this holds in practice, it can help estimate sN∗s_N^* for arbitrary NN if �0��0� is known.

However, the experiments on Rotated MNIST and Yearbook illustrate that NsN∗=Θ(1)N s_N^* = \Theta(1) does not hold. In that case, the practical implication of the theorem becomes “There exists sN∗<∞s_N^* < \infty,” which seems obvious.

Questions for authors
  • Why does constant rank guarantees that V(t)V(t) varies smoothly with tt? (line 162)
7 comments
AuthorsOpenReview ↗

Training-time guidance and smoothness of the nullspace

Thank you for asking how closely the theory applies to the experiments and whether the scaling can guide tuning. We have substantially extended both the theory and the experiments. For smooth nonlinear networks trained on squared loss, the new theorem allows any smooth evaluation loss, including cross-entropy, and gives conditions under which at least one of the two task orders exhibits 1/N1/N scaling.

Across two datasets, we varied the model (full MLP or linear classifier on frozen features), training loss (squared loss or cross-entropy), and evaluation criterion (held-out loss or accuracy), and the tuning rule you suggested predicted the optimal per-task update count to within 8% in all 30 small-learning-rate settings. The plateau was less apparent in the original Rotated MNIST experiments because the task counts were too small and the learning rate too large to observe the regime where the theory predicts a near-constant total budget.

1. Experiment design. Our experiments were designed primarily to test how far the theoretical prediction carries beyond the assumptions under which we proved it. We agree that the original experiments were too limited to make a convincing case for that extrapolation. We have since extended both the theory and the experiments, bringing the two closer together.

For nonlinear networks trained by gradient flow on squared loss and evaluated with any smooth loss, one of the two task orders is guaranteed to exhibit 1/N1/N scaling, even without a shared zero-loss solution. This holds when the task path is nonsingular and a small training budget improves the evaluation loss starting from the better endpoint. Cross-entropy can itself serve as the evaluation loss in this theorem. As you point out, using cross-entropy as the training loss on separable data leads to different long-time dynamics. Those experiments therefore test a further extrapolation. Theorem and proof sketch

We tested the prediction across models, training losses, learning rates, and evaluation criteria on Rotated MNIST and Rotated Fashion-MNIST. With squared-loss MLP training and held-out squared loss used to select the update count, the fitted exponents ranged from 0.95 to 1.01 at the three smallest learning rates. The exponents also stayed close to one when we trained the MLP with cross-entropy: across both datasets, they ranged from 0.97 to 1.02 when selecting by held-out cross-entropy or accuracy. Experimental results

2. Practical implication. We directly tested your suggestion of using the optimum at one task count to predict it at another. We tuned the updates per task at 64 tasks and divided by eight to predict the optimum at 512. Across all 30 settings at the three smallest learning rates, the prediction was within 8% of the separately fitted optimum, with a median error of 4%. This gives a quantitative way to choose the per-task budget when increasing the number of intermediate tasks along the same path. Tuning results

Smoothness of the nullspace (line 162). Here we use both the smoothness of H(t)H(t) and its constant rank. Since H(t)H(t) is positive semidefinite and continuous on a compact interval, constant rank gives a uniform lower bound μ>0\mu > 0 on its positive eigenvalues. By the spectral theorem and the residue theorem, the orthogonal projector onto V(t)=ker⁡H(t)V(t)=\ker H(t) is

P(t)=12πi∮∣z∣=μ/2(zI−H(t))−1,dz.P(t)=\frac{1}{2\pi i}\oint_{|z|=\mu/2}(zI-H(t))^{-1}\\,dz.

Indeed, on an eigenspace with eigenvalue λ\lambda, the resolvent acts by multiplication by (z−λ)−1(z-\lambda)^{-1}. The normalized contour integral is therefore one for the zero eigenspace and zero for each positive eigenspace. The same contour works for every tt. Along it, zI−H(t)zI-H(t) is invertible, so its inverse inherits the C3C^3 dependence of H(t)H(t) on tt. Differentiating under the integral shows that P(t)P(t) is C3C^3, which is the sense in which the nullspace varies smoothly.

1. Assumption on adjusting the number of tasks (NN)

The paper assumes that the number of intermediate tasks (NN) along the same adaptation path can be adjusted. Clarifying in which practical scenarios of gradual domain adaptation the task count can be varied while maintaining the same path, along with citing prior works that adopt a similar assumption of selecting NN (as in this paper), would greatly clarify the paper's practical validity.


2. Practical implications

It seems my initial question regarding "practical implications" has not been fully addressed. The additional experiments show that, at a small learning rate, the optimal total budget (T⋆=Nηk⋆T^\star = N\eta k^\star) approaches a plateau with respect to NN, and that the budget tuned at N=64N = 64 can approximate the optimal number of updates for N=512N = 512. However, applying this approach requires first estimating the optimal budget at a baseline NN. Moreover, since the paper does not discuss which η\eta can yield the optimal loss, it also does not guarantee that retraining in this manner will achieve a lower loss. I still wonder under what actual circumstances one would retrain by changing the number of tasks along the same adaptation path, and how useful reusing the existing budget is in such cases.

Furthermore, the fact that T⋆=Nηk⋆T^\star = N\eta k^\star remains constant is most pronounced at small learning rates and is not guaranteed at large learning rates (as observed in the Yearbook experiments). Given that most model training employs large learning rates where gradient flow does not hold, could you clarify the practical implications and value of the paper's analysis and the prediction of k⋆k^\star?


3. Newly added theorem require substantial effort regarding the representation.

The statement "It follows that training for T⋆/NT^\star/N on each task is asymptotically optimal for a fixed total training time T⋆>0T^\star > 0" is misleading. It is difficult to follow because key terms and concepts are not adequately explained, such as "exact-fit transport," what the "two task orders" refer to in "at least one of the two task orders along the path," and what the "path" itself represents. Furthermore, it is unclear why forward and reverse orders must be considered to extend the linear regression results, or what demonstrating a counterexample implies from this perspective.


4. The revised PDF must comply with all ICLR policies, including format.

Please check your submitted paper.

AuthorsOpenReview ↗

Questions 1 and 2

Our starting point was a question about continual learning. When an agent keeps acquiring new experience, how should it allocate its training time? Suppose NN equally sized batches arrive in sequence. Making one pass over all data collected so far after each arrival requires processing N(N+1)/2N(N+1)/2 batches in total. Making one pass over each incoming batch takes NN batches in total. This difference in cost motivated us to consider sequential training and ask how long to train on the current batch before moving on.

We originally considered an embodied agent learning from continuous interaction. Over a fixed period, an agent that collects a batch at regular intervals and then trains on it faces two related choices: how often to sample and how long to train each time. A policy update also changes the states the agent subsequently visits. Training longer can therefore change both what the agent learns from its current experience and what experience it acquires next. A single study can hardly disentangle all these effects.

To isolate the relationship between the sampling interval and training time, we fix a smoothly varying path of task distributions. Sampling more frequently then adds intermediate tasks along the same path. In our linear regression model, a shared zero-loss solution also rules out conflict between task objectives. Even with compatible tasks, fitting each task nearly to convergence along a finely sampled path can make the model closely follow the moving solution space. The residual changes direction from one task to the next while the distance to the common solution set barely decreases. Training less on each task can therefore bring the model closer to a common solution by the end of the path. Explaining this mechanism has value in its own right, independently of how precisely it predicts the best hyperparameters for a particular system.

We treat learning progress as a function of the sampling interval and the training time per stage, and optimize the latter for each choice of the former. Let the path span a fixed duration HH, with sampling interval Δt=H/N\Delta t=H/N and training time ss per task. In our linear model, G(Δt,s)G(\Delta t,s) measures the reduction in squared distance to the common solution set over the entire path. For each Δt\Delta t, we seek the training time s⋆(Δt)s^\star(\Delta t) that maximizes this gain. Under our assumptions, s⋆(Δt)=Θ(Δt)s^\star(\Delta t)=\Theta(\Delta t) as the sampling interval tends to zero. Thus, sampling more densely calls for a correspondingly shorter training time at each stage.

The analogy we have in mind is how scaling laws in deep learning are used for specific tasks. A law relating loss to training data size may admit a proof only under restricted assumptions, yet suggest a prediction that can be tested more broadly. Its range and accuracy on a particular task still have to be measured. Theory can also change which experiments we think to run. Without this analysis, we might not have expected that keeping the total amount of training approximately constant would be a promising first choice when adding tasks along the same path.

Your question about the learning rate illustrates how the precision of this prediction can vary. Across all 30 small-learning-rate settings on Rotated MNIST and Rotated Fashion-MNIST, dividing the updates per task tuned at 64 tasks by eight predicted the fitted optimum at 512 tasks to within 8%. At larger learning rates, the curves can depart from the near-1/N1/N shape seen at small learning rates, and the optima may vary more across experiments. Even so, the larger-learning-rate results and the original Yearbook experiment agree with the qualitative prediction: adding intermediate tasks along the same path generally favors fewer updates per task.

Although our motivation was abstract, prior work has directly varied the discretization of a fixed distribution path. Wang et al. varied the number of domains along a fixed 0° to 60° rotation and found that, at fixed sample size per domain, target error first fell and then rose. He et al. compared inserting one to four domains along a given feature interpolation path. Our question is how long to train at each stage when that choice changes. Wang et al., Fig. 3, He et al., Algorithm 1 and Tables 3–6

In robotic pick-and-place, Dai et al. divided expert demonstrations into 1, 3, 5, or 8 stages, trained from the stages nearest the goal back toward the start, and compared environment steps to convergence. Here the expert demonstration trajectory defines the curriculum: each segment determines where an episode starts, and the current policy determines what happens next. Even with a single demonstration, the number of stages is a practical training choice. Dai et al., §5.1, Fig. 2

Reviewer_HGRiOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper theoretically investigates optimal training-time allocation as the number of intermediate tasks increases in gradual adaptation. Under some assumptions, this paper derives an optimal per-task training-time scaling theory. Experiments on synthetic and real-world datasets provide empirical support for the theoretical findings.

Critical strengths
  1. The paper provides a theoretical analysis of optimal training-time allocation in gradual adaptation, offering valuable insights into its optimization.
  2. The theoretical findings have potential practical significance.
  3. Numerical experiments provide some empirical support for the theoretical findings.
Critical weaknesses
  1. Strong theoretical assumptions. For example, the analysis assumes that all tasks share a zero-loss solution, which may not hold in real-world scenarios.
  2. Lack of a practical optimization method. Although the paper provides the theoretical analysis of optimal training-time scaling, it does not give a concrete optimization method based on the theoretical findings, limiting its practical applicability.
  3. Limited experimental validation. Further experiments demonstrating clear performance improvements on more challenging and realistic downstream tasks would provide stronger empirical support for the theoretical findings.
  4. The colors in Figure 2 are difficult to distinguish. A more distinct color palette would improve readability.
Questions for authors

I would consider raising my score if the authors could more clearly demonstrate the practical value of the theoretical findings through more realistic experiments or practical hyperparameter optimization methods.

4 comments
AuthorsOpenReview ↗

The shared-solution assumption and practical guidance for tuning

Thank you for asking how far the scaling result extends and whether it can guide tuning. We have substantially extended both the theory and the experiments. The new theorem allows smooth nonlinear networks and tasks without a shared zero-loss solution, and gives conditions under which at least one of the two task orders exhibits 1/N1/N scaling.

Across two datasets, we varied the model (full MLP or linear classifier on frozen features), training loss (squared loss or cross-entropy), and evaluation criterion (held-out loss or accuracy), and the scaling predicted the optimal per-task update count to within 8% in all 30 small-learning-rate settings. The plateau was less apparent in the original Rotated MNIST experiments because the task counts were too small and the learning rate too large to observe the regime where the theory predicts a near-constant total budget.

1. For nonlinear networks and any smooth evaluation loss, one of the two task orders is guaranteed to exhibit 1/N1/N scaling, even without a shared zero-loss solution. Consider training a smooth network with squared loss along the same task path in two orders, forward and reverse. If the path is nonsingular and a small training budget improves the evaluation loss when starting from the better of the two endpoints, then the optimal per-task training time in that order satisfies sN⋆=Θ(1/N)s_N^\star=\Theta(1/N). Theorem and proof sketch

2–3. The scaling gives a concrete tuning rule, and we tested how accurately it predicts the optimal update count. Tune the updates per task at one task count, then scale that budget inversely with the number of tasks along the same path. Going from 64 to 512 tasks, for example, gives a prediction of one eighth of the update count tuned at 64 tasks. On Rotated MNIST and Rotated Fashion-MNIST, this prediction was within 8% of the optimum fitted separately at 512 tasks in all 30 small-learning-rate settings, with a median error of 4%. The settings included a fully trained MLP and a linear classifier on frozen features, squared loss and cross-entropy for training, and held-out loss and accuracy for selecting the optimum. Full experimental comparison

4. Figure 2. Thank you for pointing this out. We will revise the colors to make the curves easier to distinguish.

AuthorsOpenReview ↗

Revised manuscript uploaded

We uploaded the revised manuscript on November 15. In response to points 1 and 3, we have substantially extended both the theory and the experiments. For point 2, we have clarified the practical tuning guidance provided by the analysis. For point 4, we have reorganized the figures and tables to improve readability. We would appreciate your assessment of these revisions.

Thanks for the authors' response and the substantial revisions, which have addressed most of my concerns. However, the practical value of the theoretical findings remains insufficiently demonstrated, particularly in more challenging and realistic downstream tasks. Overall, I appreciate the improvements in both the theoretical analysis and practical applicability, and I will raise my score.

Thank you for taking the time to reassess the revised manuscript. We agree that the practical value of the scaling law on more challenging downstream tasks remains an open question. Our resources limited the range of tasks we could test in this study, and we see evaluation on those tasks as an important direction for future work.

Comments on the paper

8 comments
AuthorsOpenReview ↗

Response to pDJo on average path risk and the nonlinear experiments

Thank you for identifying the contribution and scope of our paper so precisely. Since our initial response, we have substantially extended both the theory and the experiments, narrowing the gap you identified. By extending Kato transport to curved manifolds of exact-fit solutions, we show that under the conditions below, at least one of the two task orders exhibits 1/N1/N scaling for smooth nonlinear networks evaluated by any smooth loss, including average population squared loss.

Across two datasets, we varied the model (full MLP or linear classifier on frozen features), training loss (squared loss or cross-entropy), and evaluation criterion (held-out loss or accuracy), and the scaling predicted the optimal per-task update count to within 8% in all 30 small-learning-rate settings. The plateau was less apparent in the original Rotated MNIST experiments because the task counts were too small and the learning rate too large to observe the regime where the theory predicts a near-constant total budget.

1. We can now answer your question about average population risk without extending the two-dimensional rotation argument in Appendix A.6. For a smooth nonlinear network trained by gradient flow on squared loss, consider training along the same task path in forward and reverse order. If the exact-fit path is nonsingular and a small budget improves the evaluation loss from the better of the two endpoints, at least one of the two orders is guaranteed to exhibit sN⋆=Θ(1/N)s_N^\star=\Theta(1/N). This applies to any smooth evaluation loss satisfying the improvement condition, including average population squared loss. The tasks need not share a zero-loss solution. Theorem and proof sketch

The counterexample shows why the choice of order matters. We constructed a smooth rank-one path in two dimensions satisfying the original assumptions of Theorem 2. In the prescribed order, the distance-optimal total training time stays bounded, while every total training time minimizing average population squared loss diverges as NN grows. A finite budget shortens the residual, but longer training turns it toward a direction with sufficiently lower prediction loss to outweigh the difference in length. We give the full proof in a separate comment below.

2. The scaling was accurate enough to guide tuning on Rotated MNIST and Rotated Fashion-MNIST. In the experiment closest to the new theorem, we trained an MLP with squared loss and selected the update count by held-out squared loss averaged along the path. The fitted exponents were 0.95–1.01 across both datasets at the three smallest learning rates. Across all 30 configurations tested at those rates, dividing the per-task budget tuned at 64 tasks by eight predicted the optimum at 512 tasks within 8% in every case, with a median error of 4%. This establishes a practical use for the scaling in these settings. How much moving-nullspace transport contributes remains open because the sweeps did not measure curvature or track the residual along the training trajectory. Experimental results

Scope of the evidence. The theory guarantees the scaling for at least one order of every forward–reverse pair satisfying its assumptions. The counterexample shows that a prescribed order can fail even in linear regression. On Rotated MNIST and Rotated Fashion-MNIST, the agreement is quantitative and extends to settings beyond the theorem, including cross-entropy training and selection by accuracy. On Yearbook, adding tasks generally favored fewer updates per task, so the theory still gives a useful direction for tuning. The optimal total budget varied more, however, and the accuracy of the 1/N1/N tuning rule there remains uncertain. We do not yet know what accounts for this difference. Taken together, the results explain why inverse scaling can arise across a broad class of task paths and show where it is already accurate enough to guide practice.

AuthorsOpenReview ↗

Tuning the per-task update budget across task counts

On Rotated MNIST and Rotated Fashion-MNIST, the 1/N1/N scaling predicted by our theory was accurate enough to guide tuning across the models, training losses, and evaluation criteria we tested at small learning rates. We tuned the number of updates per task at 64 tasks, then divided it by eight to predict the optimum at 512 tasks. Across all 30 settings, the predicted update count was within 8% of the optimum fitted separately at 512 tasks, with a median error of 4%.

We trained the full MLP with squared loss and cross-entropy on both datasets. On Rotated MNIST, we also trained a linear classifier on frozen MLP features. Whether we selected the update count by held-out loss or accuracy, the fitted exponents at the three smallest learning rates ranged from 0.95 to 1.01. The plateau is apparent in the total budget: as the number of tasks along the same rotation path increased from 40 to 512 (a factor of 12.8), the optimal total number of updates increased by only 5% in the median across the 30 settings.

Table 1. Scaling of the optimal number of updates per task at each learning rate. Entries give the fitted exponent bb in k⋆∝N−bk^\star \propto N^{-b} over N=40N=40 to 512512, with 95% bootstrap intervals across five seeds. An exponent of 1 means that doubling the number of tasks halves the optimal number of updates per task.

ModelDatasetTraining lossHeld-out criterion for selecting k⋆k^\starη = 0.10.050.0250.01250.006250.0031250.00156250.00078125
Full MLPRotated MNISTCross-entropyLowest cross-entropy/////0.98 [0.97, 0.99]0.98 [0.97, 0.98]0.98 [0.97, 0.98]
Full MLPRotated MNISTCross-entropyHighest accuracy/////0.98 [0.97, 0.99]0.97 [0.97, 0.99]0.97 [0.97, 0.98]
Full MLPRotated MNISTSquared lossLowest squared loss///0.97 [0.90, 1.03]0.96 [0.93, 1.01]0.95 [0.85, 0.99]0.97 [0.84, 0.98]0.96 [0.91, 0.97]
Full MLPRotated MNISTSquared lossHighest accuracy////0.97 [0.96, 0.99]0.99 [0.97, 1.00]0.98 [0.98, 0.99]0.98 [0.97, 0.99]
Full MLPRotated Fashion-MNISTCross-entropyLowest cross-entropy/////0.99 [0.97, 1.02]1.00 [0.98, 1.01]0.97 [0.96, 0.98]
Full MLPRotated Fashion-MNISTCross-entropyHighest accuracy/////0.99 [0.97, 1.02]0.98 [0.96, 0.99]0.98 [0.96, 0.99]
Full MLPRotated Fashion-MNISTSquared lossLowest squared loss///1.05 [0.08, 1.06]1.01 [0.48, 1.16]1.01 [0.72, 1.31]1.01 [0.91, 1.39]1.00 [0.91, 1.40]
Full MLPRotated Fashion-MNISTSquared lossHighest accuracy////1.00 [0.98, 1.01]0.98 [0.96, 0.99]0.98 [0.97, 0.99]0.99 [0.97, 0.99]
Linear on frozen featuresRotated MNISTCross-entropyLowest cross-entropy///0.99 [0.97, 1.00]0.98 [0.98, 0.99]0.98 [0.98, 0.99]0.98 [0.98, 0.99]0.98 [0.98, 0.99]
Linear on frozen featuresRotated MNISTCross-entropyHighest accuracy///0.99 [0.97, 1.01]0.98 [0.98, 0.99]0.98 [0.97, 0.99]0.98 [0.98, 0.99]0.99 [0.98, 0.99]

Each criterion was evaluated on held-out images and averaged over 13 fixed rotation angles. A slash (/) indicates that the optimum reached the lower limit of one update per task for at least one task count in the fitting range.

The original Rotated MNIST experiment used too few tasks to reach the near-constant total budget predicted by the theory. Even at small learning rates, the new sweep recovers the original exponent of about 0.84 over 4–32 tasks. Extending the sweep to 40–512 tasks reveals the plateau. Smaller learning rates make it possible to observe this regime because, at the original learning rate, the optimum in our sweep already reaches one update per task at 32 tasks.

AuthorsOpenReview ↗

Optimal per-task training time scales as 1/N in at least one direction for nonlinear networks with smooth evaluation losses

Theorem (at least one direction of a task path exhibits 1/N1/N scaling). Consider a smooth nonlinear network trained by gradient flow on squared loss along a fixed, smooth task path. Start at a parameter aa that exactly fits the first task. Discretize the path uniformly into NN tasks and train for time ss on each, giving a total budget T=NsT=Ns. In both directions, evaluate the final parameter using the same continuous loss RR, with lower values preferred. Assume the gradient flows exist and the following two conditions hold.

  1. The path has no singular points. Starting from aa, the parameters remain bounded along the path that tracks the exact-fit solutions in the normal direction, and the Jacobian of the network outputs with respect to the parameters has full row rank throughout.
  2. A small training budget improves the evaluation loss. Starting from the endpoint with lower evaluation loss and traversing the entire path, some fixed small total budget yields a limiting evaluation loss strictly below the starting value as NN grows. If the endpoints have equal loss, improvement in either direction suffices.

Then, in the direction specified by the second condition, any optimal per-task training time satisfies sN⋆=Θ(1/N)s_N^\star=\Theta(1/N).

Proof sketch. The first step extends Kato transport from the linear setting to the manifold of exact-fit solutions. Write F(t,w)F(t,w) for the network outputs on task tt, y(t)y(t) for the targets, and J=DwF(t,w)J=D_wF(t,w) for the Jacobian. At an exact fit, the Hessian of the squared loss is J⊤JJ^\top J. Full row rank gives contraction in the normal directions, while tangential drift is of second order in the distance to the manifold. Thus, as N→∞N\to\infty and Ns→∞Ns\to\infty, gradient flow tracks the moving manifold in the normal direction. The limiting parameter path satisfies

w˙=J†(y′(t)−∂tF(t,w)).\dot w=J^\dagger\bigl(y'(t)-\partial_t F(t,w)\bigr).

Let b=w(1)b=w(1) be its endpoint, with w(0)=aw(0)=a. Long-budget training on the reverse path returns to aa. Replacing tt by 1−t1-t leaves the Jacobian unchanged at the corresponding task and parameter, while reversing the sign of the right-hand side. Hence w(1−t)w(1-t) solves the reverse equation with initial value bb. Uniqueness gives the reverse trajectory and its endpoint aa.

This return lets us start at the endpoint with lower evaluation loss. If R(a)≤R(b)R(a)\le R(b), take the forward direction: the zero-budget loss is R(a)R(a) and the long-budget limit is R(b)R(b). If R(b)<R(a)R(b)<R(a), take the reverse direction: these values are R(b)R(b) and R(a)R(a), respectively. In either case, the fixed budget in the second condition beats both extremes.

If the optimal total budget NsN⋆Ns_N^\star tended to zero or infinity along a subsequence, its evaluation loss would converge to the corresponding endpoint value. The fixed budget would then achieve strictly lower loss, contradicting optimality. Thus NsN⋆Ns_N^\star is bounded above and bounded away from zero independently of NN, giving sN⋆=Θ(1/N)s_N^\star=\Theta(1/N).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.