- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 2(−1)
Read review
Paper summary
The paper studies the optimal local training time for training tasks in overparameterized linear regression. By adopting the effective training time , the main theorem shows that learning grows as for small and drops as for large . Consequently, it shows that the optimal local training time is . The prediction was tested on rotated MNIST and temporal Yearbook data.
Critical strengths
- The paper is well-written.
- Building on the setting of previous work on gradual adaptation in overparameterized linear regression, the paper provides a novel theoretical explanation for the optimal local training time for training tasks.
Critical weaknesses
- Experiment design
The experiments involve classification tasks, which contradicts the theoretical setting. In overparameterized linear regression, the optimal solution closest to the initial point is obtained by projection onto the null space (which is the basis of the analysis in the paper). However, in classification, the optimal solution diverges to infinity as the loss decreases to zero. Therefore, to test whether the predicted training-time trend persists in a real-world scenario, the task should be a regression task.
- Practical implication
The prediction from the theory seems to have weak practical implications. To be specific, Theorem 2 shows that . If this holds in practice, it can help estimate for arbitrary if is known.
However, the experiments on Rotated MNIST and Yearbook illustrate that does not hold. In that case, the practical implication of the theorem becomes “There exists ,” which seems obvious.
Questions for authors
- Why does constant rank guarantees that varies smoothly with ? (line 162)
7 comments
1. Assumption on adjusting the number of tasks ()
The paper assumes that the number of intermediate tasks () along the same adaptation path can be adjusted. Clarifying in which practical scenarios of gradual domain adaptation the task count can be varied while maintaining the same path, along with citing prior works that adopt a similar assumption of selecting (as in this paper), would greatly clarify the paper's practical validity.
2. Practical implications
It seems my initial question regarding "practical implications" has not been fully addressed. The additional experiments show that, at a small learning rate, the optimal total budget () approaches a plateau with respect to , and that the budget tuned at can approximate the optimal number of updates for . However, applying this approach requires first estimating the optimal budget at a baseline . Moreover, since the paper does not discuss which can yield the optimal loss, it also does not guarantee that retraining in this manner will achieve a lower loss. I still wonder under what actual circumstances one would retrain by changing the number of tasks along the same adaptation path, and how useful reusing the existing budget is in such cases.
Furthermore, the fact that remains constant is most pronounced at small learning rates and is not guaranteed at large learning rates (as observed in the Yearbook experiments). Given that most model training employs large learning rates where gradient flow does not hold, could you clarify the practical implications and value of the paper's analysis and the prediction of ?
3. Newly added theorem require substantial effort regarding the representation.
The statement "It follows that training for on each task is asymptotically optimal for a fixed total training time " is misleading. It is difficult to follow because key terms and concepts are not adequately explained, such as "exact-fit transport," what the "two task orders" refer to in "at least one of the two task orders along the path," and what the "path" itself represents. Furthermore, it is unclear why forward and reverse orders must be considered to extend the linear regression results, or what demonstrating a counterexample implies from this perspective.
4. The revised PDF must comply with all ICLR policies, including format.
Please check your submitted paper.
Questions 1 and 2
Our starting point was a question about continual learning. When an agent keeps acquiring new experience, how should it allocate its training time? Suppose equally sized batches arrive in sequence. Making one pass over all data collected so far after each arrival requires processing batches in total. Making one pass over each incoming batch takes batches in total. This difference in cost motivated us to consider sequential training and ask how long to train on the current batch before moving on.
We originally considered an embodied agent learning from continuous interaction. Over a fixed period, an agent that collects a batch at regular intervals and then trains on it faces two related choices: how often to sample and how long to train each time. A policy update also changes the states the agent subsequently visits. Training longer can therefore change both what the agent learns from its current experience and what experience it acquires next. A single study can hardly disentangle all these effects.
To isolate the relationship between the sampling interval and training time, we fix a smoothly varying path of task distributions. Sampling more frequently then adds intermediate tasks along the same path. In our linear regression model, a shared zero-loss solution also rules out conflict between task objectives. Even with compatible tasks, fitting each task nearly to convergence along a finely sampled path can make the model closely follow the moving solution space. The residual changes direction from one task to the next while the distance to the common solution set barely decreases. Training less on each task can therefore bring the model closer to a common solution by the end of the path. Explaining this mechanism has value in its own right, independently of how precisely it predicts the best hyperparameters for a particular system.
We treat learning progress as a function of the sampling interval and the training time per stage, and optimize the latter for each choice of the former. Let the path span a fixed duration , with sampling interval and training time per task. In our linear model, measures the reduction in squared distance to the common solution set over the entire path. For each , we seek the training time that maximizes this gain. Under our assumptions, as the sampling interval tends to zero. Thus, sampling more densely calls for a correspondingly shorter training time at each stage.
The analogy we have in mind is how scaling laws in deep learning are used for specific tasks. A law relating loss to training data size may admit a proof only under restricted assumptions, yet suggest a prediction that can be tested more broadly. Its range and accuracy on a particular task still have to be measured. Theory can also change which experiments we think to run. Without this analysis, we might not have expected that keeping the total amount of training approximately constant would be a promising first choice when adding tasks along the same path.
Your question about the learning rate illustrates how the precision of this prediction can vary. Across all 30 small-learning-rate settings on Rotated MNIST and Rotated Fashion-MNIST, dividing the updates per task tuned at 64 tasks by eight predicted the fitted optimum at 512 tasks to within 8%. At larger learning rates, the curves can depart from the near- shape seen at small learning rates, and the optima may vary more across experiments. Even so, the larger-learning-rate results and the original Yearbook experiment agree with the qualitative prediction: adding intermediate tasks along the same path generally favors fewer updates per task.
Although our motivation was abstract, prior work has directly varied the discretization of a fixed distribution path. Wang et al. varied the number of domains along a fixed 0° to 60° rotation and found that, at fixed sample size per domain, target error first fell and then rose. He et al. compared inserting one to four domains along a given feature interpolation path. Our question is how long to train at each stage when that choice changes. Wang et al., Fig. 3, He et al., Algorithm 1 and Tables 3–6
In robotic pick-and-place, Dai et al. divided expert demonstrations into 1, 3, 5, or 8 stages, trained from the stages nearest the goal back toward the start, and compared environment steps to convergence. Here the expert demonstration trajectory defines the curriculum: each segment determines where an episode starts, and the current policy determines what happens next. Even with a single demonstration, the number of stages is a practical training choice. Dai et al., §5.1, Fig. 2
Training-time guidance and smoothness of the nullspace
Thank you for asking how closely the theory applies to the experiments and whether the scaling can guide tuning. We have substantially extended both the theory and the experiments. For smooth nonlinear networks trained on squared loss, the new theorem allows any smooth evaluation loss, including cross-entropy, and gives conditions under which at least one of the two task orders exhibits 1/N scaling.
Across two datasets, we varied the model (full MLP or linear classifier on frozen features), training loss (squared loss or cross-entropy), and evaluation criterion (held-out loss or accuracy), and the tuning rule you suggested predicted the optimal per-task update count to within 8% in all 30 small-learning-rate settings. The plateau was less apparent in the original Rotated MNIST experiments because the task counts were too small and the learning rate too large to observe the regime where the theory predicts a near-constant total budget.
1. Experiment design. Our experiments were designed primarily to test how far the theoretical prediction carries beyond the assumptions under which we proved it. We agree that the original experiments were too limited to make a convincing case for that extrapolation. We have since extended both the theory and the experiments, bringing the two closer together.
For nonlinear networks trained by gradient flow on squared loss and evaluated with any smooth loss, one of the two task orders is guaranteed to exhibit 1/N scaling, even without a shared zero-loss solution. This holds when the task path is nonsingular and a small training budget improves the evaluation loss starting from the better endpoint. Cross-entropy can itself serve as the evaluation loss in this theorem. As you point out, using cross-entropy as the training loss on separable data leads to different long-time dynamics. Those experiments therefore test a further extrapolation. Theorem and proof sketch
We tested the prediction across models, training losses, learning rates, and evaluation criteria on Rotated MNIST and Rotated Fashion-MNIST. With squared-loss MLP training and held-out squared loss used to select the update count, the fitted exponents ranged from 0.95 to 1.01 at the three smallest learning rates. The exponents also stayed close to one when we trained the MLP with cross-entropy: across both datasets, they ranged from 0.97 to 1.02 when selecting by held-out cross-entropy or accuracy. Experimental results
2. Practical implication. We directly tested your suggestion of using the optimum at one task count to predict it at another. We tuned the updates per task at 64 tasks and divided by eight to predict the optimum at 512. Across all 30 settings at the three smallest learning rates, the prediction was within 8% of the separately fitted optimum, with a median error of 4%. This gives a quantitative way to choose the per-task budget when increasing the number of intermediate tasks along the same path. Tuning results
Smoothness of the nullspace (line 162). Here we use both the smoothness of H(t) and its constant rank. Since H(t) is positive semidefinite and continuous on a compact interval, constant rank gives a uniform lower bound μ>0 on its positive eigenvalues. By the spectral theorem and the residue theorem, the orthogonal projector onto V(t)=kerH(t) is
P(t)=2πi1∮∣z∣=μ/2(zI−H(t))−1,dz.
Indeed, on an eigenspace with eigenvalue λ, the resolvent acts by multiplication by (z−λ)−1. The normalized contour integral is therefore one for the zero eigenspace and zero for each positive eigenspace. The same contour works for every t. Along it, zI−H(t) is invertible, so its inverse inherits the C3 dependence of H(t) on t. Differentiating under the integral shows that P(t) is C3, which is the sense in which the nullspace varies smoothly.