- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of training logistic regression model using full-batch GD with large initialization scale and focuses on understanding the intermediate fixed-step training trajectory. The authors discover a geometric approximation of this intermediate behavior and construct an ideal path consisting of two stages, namely, negative margin correction and minimum margin growth. They proved that, after dividing the weights by the initialization scale and setting different time scaling for different stages, the actual GD trajectory converges uniformly to their ideal path, as the initialization scale tends to infinity. They further find applications in the peak evaluation loss and the cumulative training loss. They empirically verify their theory.
Critical strengths
The paper connects the actual GD trajectory to a clear and computable geometric approximation path. Theorems and their proofs seem to be mathematically correct. I particularly value their Theorem 1, which gives a clean and quantitative uniform convergence bound. The paper also contains experiments to justify their theory.
Critical weaknesses
I'm not convinced of the motivation of this work. What I understand are: (i) A pre-trained model might have large weights and thus the continuing training task can be think of training with a large initialization; (ii) Different initialization scales can induce different intermediate training trajectories while the asymptotic behavior could be similar; (iii) Then, this paper focuses on deriving a geometric description of a two-stage training behavior. However, I have the following concerns: How should we use this knowledge of the ideal path? Would it challenge the existing early-stopping or regularization criteria? (See also my question below.) Overall, after reading the paper, it makes to me that the main purpose of studying the ideal path is to understand GD better for logistic regression, rather than to facilitate optimization or generalization.
Furthermore, I found the article not sufficiently readable. (i) Some notations appear early in the introduction without clarification. (E.g., what is the so-called "limiting normalized initial position" and what is on page 1?) (ii) In Section 2, there are confusing notions and statements without explanation. (E.g., in Section 2.1, why do "we observe "? In section 2.2, what are the so-called "limiting velocity" and the "path derivative"? What is the used in line 151 as well as later in Equations (4), (5) and (6)?) I believe the paper needs to be well polished for acceptance.
Questions for authors
-
The current work studies logistic regression using full-batch GD. Is it possible to generalize your theory to other ML/AI models (e.g. neural networks)?
-
While the authors find applications of their ideal path theory in predicting the peak evaluation loss and the cumulative training loss, how can we use such applications to improve optimization or generalization? Is there any novel early-stopping or regularization algorithm that can provably improve generalization in logistic regression given the ideal geometric path?
Regarding the two questions above, I would like to know whether the theory in this paper is broadly important and useful.
4 comments
(2/3) Response to Reviewer ZYSu
2. Early stopping, regularization, and generalization
Question: Do these predictions lead to a new early-stopping or regularization algorithm with provable improvements in optimization or generalization?
Response: We do not currently establish such an algorithm or a population-generalization improvement. The present contribution is a theory of training dynamics with quantitative predictions. In particular, the evaluation features in our peak-loss result are fixed: the result predicts their loss along training, rather than a population generalization gap. Likewise, a cumulative training-loss formula is not itself a generalization guarantee.
The predictions nevertheless identify quantities that a future intervention could use. The path can locate portions of the trajectory where continued target training harms specified evaluation margins, while the first-passage and peak-time results connect geometric events to actual GD stopping indices. Turning these observations into a stopping or regularization procedure with a generalization guarantee would require additional analysis of validation-based selection, regularized dynamics, or distributional assumptions. Our current example does not invalidate validation-based early stopping; rather, it shows why reasoning only from the initial and final models can miss intermediate behavior that trajectory-level evaluation would reveal.
The fixed evaluation-set loss and its hinge limit are defined in Section 4.2, equation (14) (p. 8, lines 386–393). The linear-threshold stopping result is Corollary 1 (p. 7, lines 335–345), and the unique second-stage peak-time result is Corollary 2 (p. 7, lines 346–361).
3. Scope beyond logistic regression
Question: Can the theory be generalized to other models, including neural networks?
Response: A direct setting covered by the analysis is an unregularized logistic output layer trained on fixed features, including a frozen learned representation, provided the signed features are strictly separable. Jointly training a neural network is a substantive extension: the feature geometry then evolves, and our proof relies on a fixed geometry and convex subgradient flows. We therefore do not claim that the present ideal path describes general neural-network fine-tuning.
To strengthen the evidence within the stated model, we now transfer actual logistic-GD checkpoints from a source digits task to a target task, with all weights transferred unchanged. Across nine source-confidence thresholds and ten random seeds, the mean uniform target-path error decreases from to . This supports the trajectory prediction for naturally obtained initializations on fixed image features. It is not an experiment with jointly learned neural features, and the near-zero endpoint recovery phenomenon is demonstrated separately by the analytic and controlled synthetic examples.
We see the fixed-representation setting as a tractable first step toward a broader question: whether related trajectory geometry persists when representations themselves evolve during adaptation or post-training. Establishing that requires new analysis rather than a direct application of the current theorem.
The revised experiment is reported in Section 5.3 and Figure 4(c) (p. 10, lines 501–527), with the full protocol, all nine thresholds, and scope checks in Appendix E.4 (pp. 33–34, lines 1744–1835). The fixed-feature logistic dynamics and large-initialization family are specified in Section 2.1 (p. 3, lines 124–149).
(3/3) Response to Reviewer ZYSu
4. Notation and readability
Question/Suggestion: Please clarify , the normalization , the point , and the meaning of the limiting velocity and path derivative.
Response: Thank you for identifying these specific obstacles. We revised the opening of Section 2.2 to introduce the rescaled point and derive its velocity before giving the geometric rules.
- : For with , it is the limiting position of . It may have negative target margins and need not be a target separator.
- : Division by the fixed initialization scale keeps the starting position finite while retaining changes in both norm and direction. Dividing by the evolving norm would retain direction alone.
- : This is a point in rescaled parameter space, corresponding to the unscaled parameter .
- Velocity: With , the normalized update divided by is displacement per unit rescaled time. Away from zero margins, its limit is , which defines along the ideal trajectory. At a change of segment, the geometric rule determines the outgoing motion; the path remains continuous although its velocity can change.
The initialization and normalization are specified in Section 2.1 (p. 3, lines 136–149). The definition of and explicit displacement-per-unit-time derivation are in Section 2.2 (p. 3, lines 153–161); the boundary rule and explanation of the stage transition follow on p. 4, lines 162–180. Section 2.4 states continuity, finite velocity changes, and the segment-by-segment construction of the complete path (p. 5, lines 231–244).
We especially appreciate the reviewer’s emphasis on the broader importance of the theory. For us, it is important that an initial research spark be developed into a rigorous piece of work that can be clearly communicated, scrutinized, and discussed by the community, and the reviewer’s questions have helped us sharpen that transition. We believe the most interesting next questions are not merely additional formulas for logistic regression, but whether trajectory-level geometry can support principled decisions about monitoring, stopping, regularization, and stability in richer learning systems. We hope the revision makes clearer both what the present theory establishes and where the genuinely open questions begin. We would be very interested in continuing this discussion with researchers working on optimization, learning dynamics, and post-training, as we believe this viewpoint may open several new directions for analysis.
I thank the authors for your detailed and careful rebuttal. I value the mathematical analysis presented in the paper as well as the additional experiments you did in the rebuttal. However, since the scope of the paper is limited, I can only raise my rating to 2. I expect the authors to extend the theory to more general scenarios and to develop corresponding intervention algorithms in future works.
(1/3) Response to Reviewer ZYSu
We thank the reviewer for the careful review and for recognizing the value of the computable geometric characterization and the quantitative uniform convergence bound in Theorem 1. Your central question, i.e., what this geometric knowledge is ultimately useful for, also helped us recognize that the paper needed a clearer bridge from the mathematical characterization to learning and adaptation. As discussed in our overall response, our original motivation came from asking when soft output-layer decisions begin to exhibit hard-limit geometry as the logit scale grows. The ideal path arose from discovering that the hard-limit correction dynamics stop while the soft logistic dynamics continue on a slower time scale. We now make both this motivation and the predictive role of the path more explicit.
We have added finite-scale and iteration-time results, replaced the digits initialization with actual pretrained checkpoints, and clarified the notation and derivation in Section 2. We address the motivation, practical scope, and presentation below.
1. Motivation and use of the ideal path
Question: How should the ideal-path characterization be used, and why is understanding the intermediate trajectory important?
Response: The hard/soft distinction helps explain why the intermediate trajectory is not already captured by the usual endpoint picture. The scaled logistic loss approaches a polyhedral loss whose correction flow stops when all margins are nonnegative. Soft logistic GD nevertheless continues on a slower clock. We identify this continuation through minimum-margin geometry and connect both stages in a computable ideal path, with quantitative control of fixed-step GD on each fixed path window. This describes intermediate behavior that the eventual maximum-margin direction alone does not determine.
The path then enables several concrete predictions. For a fixed evaluation set, the leading peak evaluation loss can be computed from finitely many ideal-path vertices. Our analytic construction shows that this peak can be R/25+o(R) even when the losses at both ends of the specified training window tend to zero. Thus small endpoint losses can conceal substantial intermediate deterioration. The cumulative-loss results quantify the different contributions of the two stages, while the iteration-time results connect geometric positions to the number of GD updates needed to reach them.
More specifically, for a linear threshold first strictly crossed in the second stage, Corollary 1 shows that its actual first-passage index TR satisfies
\qquad b(x)=\min_i z_i^\top x,$$ where $x_*$ is the first crossing point on the ideal path. Corollary 2 further shows, under a unique second-stage ideal maximizer, that the actual evaluation-loss peak has the corresponding leading iteration scale. These results distinguish a behavior that is geometrically present along the path from one that is reached within a given amount of training. The statement is asymptotic: it predicts the leading scale and does not guarantee affordability under a fixed budget. We therefore view the present theory as **predictive rather than prescriptive**. Within the model, it can tell us where an evaluation loss becomes unfavorable, how severe the leading excursion is, and on what iteration scale it occurs. Those are quantities one would need before designing targeted evaluation, checkpointing, stopping, regularization, or data-reweighting interventions. We do not claim that the present paper already provides such an intervention algorithm. $\Rightarrow$ The revised abstract now states this diagnostic interpretation explicitly (p. 1, lines 29–32). The contribution summary separates geometry, iteration time, path functionals, and pretrained-checkpoint experiments in Section 1.1 (p. 2, lines 64–84). Proposition 3 and Corollaries 1–2 are in Section 4.1 (p. 7, lines 324–361), and the peak-loss prediction and finite-vertex characterization are in Section 4.2 (pp. 7–8, lines 365–418).