acceptodds
Under review as a conference paper at ICLR 2027

Ideal Paths: Geometric Approximation of Logistic Gradient Descent Trajectories at Large Initialization

Abstract

Modern training on a new task often starts from a previously trained model rather than from scratch, raising the question of how this initialization affects the subsequent training trajectory. Classical implicit-bias results characterize the direction selected by prolonged training, but this direction alone does not provide information regarding the intermediate behavior. We address this question through a geometric approximation of full-batch logistic gradient descent (GD) trajectories on strictly linearly separable data, with large initialization of scale motivated by prior training. From any limiting normalized initial position, we use minimum-norm projection rules to construct a unique continuous *ideal path* consisting of finitely many linear segments. The path has two stages: negative-margin correction followed by minimum-margin growth. We prove that, after an explicit two-stage time reparameterization, the fixed-step GD trajectory divided by converges uniformly to this path on every fixed parameter interval as . Further, our quantitative error bounds account for initialization perturbations and the transition between stages. This approximation provides asymptotic formulas for peak evaluation loss and cumulative training loss. In particular, peak evaluation loss can grow linearly in even when both endpoint losses tend to zero. The cumulative losses in the correction and margin-growth stages, normalized by and , respectively, converge to explicit limits. Experiments on controlled geometries and fixed image features complement our theoretical results.

Primary area:
learning theory
Keywords:
implicit bias, gradient descent, training dynamics, geometric approximation, large initialization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZYSuOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the problem of training logistic regression model using full-batch GD with large initialization scale and focuses on understanding the intermediate fixed-step training trajectory. The authors discover a geometric approximation of this intermediate behavior and construct an ideal path consisting of two stages, namely, negative margin correction and minimum margin growth. They proved that, after dividing the weights by the initialization scale and setting different time scaling for different stages, the actual GD trajectory converges uniformly to their ideal path, as the initialization scale tends to infinity. They further find applications in the peak evaluation loss and the cumulative training loss. They empirically verify their theory.

Critical strengths

The paper connects the actual GD trajectory to a clear and computable geometric approximation path. Theorems and their proofs seem to be mathematically correct. I particularly value their Theorem 1, which gives a clean and quantitative uniform convergence bound. The paper also contains experiments to justify their theory.

Critical weaknesses

I'm not convinced of the motivation of this work. What I understand are: (i) A pre-trained model might have large weights and thus the continuing training task can be think of training with a large initialization; (ii) Different initialization scales can induce different intermediate training trajectories while the asymptotic behavior could be similar; (iii) Then, this paper focuses on deriving a geometric description of a two-stage training behavior. However, I have the following concerns: How should we use this knowledge of the ideal path? Would it challenge the existing early-stopping or regularization criteria? (See also my question below.) Overall, after reading the paper, it makes to me that the main purpose of studying the ideal path is to understand GD better for logistic regression, rather than to facilitate optimization or generalization.

Furthermore, I found the article not sufficiently readable. (i) Some notations appear early in the introduction without clarification. (E.g., what is the so-called "limiting normalized initial position" u0u_0 and what is eRe_R on page 1?) (ii) In Section 2, there are confusing notions and statements without explanation. (E.g., in Section 2.1, why do "we observe wk/Rw_{k}/R"? In section 2.2, what are the so-called "limiting velocity" and the "path derivative"? What is the xx used in line 151 as well as later in Equations (4), (5) and (6)?) I believe the paper needs to be well polished for acceptance.

Questions for authors
  1. The current work studies logistic regression using full-batch GD. Is it possible to generalize your theory to other ML/AI models (e.g. neural networks)?

  2. While the authors find applications of their ideal path theory in predicting the peak evaluation loss and the cumulative training loss, how can we use such applications to improve optimization or generalization? Is there any novel early-stopping or regularization algorithm that can provably improve generalization in logistic regression given the ideal geometric path?

Regarding the two questions above, I would like to know whether the theory in this paper is broadly important and useful.

4 comments
AuthorsOpenReview ↗

(1/3) Response to Reviewer ZYSu

We thank the reviewer for the careful review and for recognizing the value of the computable geometric characterization and the quantitative uniform convergence bound in Theorem 1. Your central question, i.e., what this geometric knowledge is ultimately useful for, also helped us recognize that the paper needed a clearer bridge from the mathematical characterization to learning and adaptation. As discussed in our overall response, our original motivation came from asking when soft output-layer decisions begin to exhibit hard-limit geometry as the logit scale grows. The ideal path arose from discovering that the hard-limit correction dynamics stop while the soft logistic dynamics continue on a slower time scale. We now make both this motivation and the predictive role of the path more explicit.

We have added finite-scale and iteration-time results, replaced the digits initialization with actual pretrained checkpoints, and clarified the notation and derivation in Section 2. We address the motivation, practical scope, and presentation below.

 

1. Motivation and use of the ideal path

Question: How should the ideal-path characterization be used, and why is understanding the intermediate trajectory important?

Response: The hard/soft distinction helps explain why the intermediate trajectory is not already captured by the usual endpoint picture. The scaled logistic loss approaches a polyhedral loss whose correction flow stops when all margins are nonnegative. Soft logistic GD nevertheless continues on a slower clock. We identify this continuation through minimum-margin geometry and connect both stages in a computable ideal path, with quantitative control of fixed-step GD on each fixed path window. This describes intermediate behavior that the eventual maximum-margin direction alone does not determine.

The path then enables several concrete predictions. For a fixed evaluation set, the leading peak evaluation loss can be computed from finitely many ideal-path vertices. Our analytic construction shows that this peak can be R/25+o(R)R/25+o(R) even when the losses at both ends of the specified training window tend to zero. Thus small endpoint losses can conceal substantial intermediate deterioration. The cumulative-loss results quantify the different contributions of the two stages, while the iteration-time results connect geometric positions to the number of GD updates needed to reach them.

More specifically, for a linear threshold first strictly crossed in the second stage, Corollary 1 shows that its actual first-passage index TRT_R satisfies

\qquad b(x)=\min_i z_i^\top x,$$ where $x_*$ is the first crossing point on the ideal path. Corollary 2 further shows, under a unique second-stage ideal maximizer, that the actual evaluation-loss peak has the corresponding leading iteration scale. These results distinguish a behavior that is geometrically present along the path from one that is reached within a given amount of training. The statement is asymptotic: it predicts the leading scale and does not guarantee affordability under a fixed budget. We therefore view the present theory as **predictive rather than prescriptive**. Within the model, it can tell us where an evaluation loss becomes unfavorable, how severe the leading excursion is, and on what iteration scale it occurs. Those are quantities one would need before designing targeted evaluation, checkpointing, stopping, regularization, or data-reweighting interventions. We do not claim that the present paper already provides such an intervention algorithm. $\Rightarrow$ The revised abstract now states this diagnostic interpretation explicitly (p. 1, lines 29–32). The contribution summary separates geometry, iteration time, path functionals, and pretrained-checkpoint experiments in Section 1.1 (p. 2, lines 64–84). Proposition 3 and Corollaries 1–2 are in Section 4.1 (p. 7, lines 324–361), and the peak-loss prediction and finite-vertex characterization are in Section 4.2 (pp. 7–8, lines 365–418).
AuthorsOpenReview ↗

(2/3) Response to Reviewer ZYSu

2. Early stopping, regularization, and generalization

Question: Do these predictions lead to a new early-stopping or regularization algorithm with provable improvements in optimization or generalization?

Response: We do not currently establish such an algorithm or a population-generalization improvement. The present contribution is a theory of training dynamics with quantitative predictions. In particular, the evaluation features in our peak-loss result are fixed: the result predicts their loss along training, rather than a population generalization gap. Likewise, a cumulative training-loss formula is not itself a generalization guarantee.

The predictions nevertheless identify quantities that a future intervention could use. The path can locate portions of the trajectory where continued target training harms specified evaluation margins, while the first-passage and peak-time results connect geometric events to actual GD stopping indices. Turning these observations into a stopping or regularization procedure with a generalization guarantee would require additional analysis of validation-based selection, regularized dynamics, or distributional assumptions. Our current example does not invalidate validation-based early stopping; rather, it shows why reasoning only from the initial and final models can miss intermediate behavior that trajectory-level evaluation would reveal.

⇒\Rightarrow The fixed evaluation-set loss LevL_{\mathrm{ev}} and its hinge limit HevH_{\mathrm{ev}} are defined in Section 4.2, equation (14) (p. 8, lines 386–393). The linear-threshold stopping result is Corollary 1 (p. 7, lines 335–345), and the unique second-stage peak-time result is Corollary 2 (p. 7, lines 346–361).

 

3. Scope beyond logistic regression

Question: Can the theory be generalized to other models, including neural networks?

Response: A direct setting covered by the analysis is an unregularized logistic output layer trained on fixed features, including a frozen learned representation, provided the signed features are strictly separable. Jointly training a neural network is a substantive extension: the feature geometry then evolves, and our proof relies on a fixed geometry and convex subgradient flows. We therefore do not claim that the present ideal path describes general neural-network fine-tuning.

To strengthen the evidence within the stated model, we now transfer actual logistic-GD checkpoints from a source digits task to a target task, with all weights transferred unchanged. Across nine source-confidence thresholds and ten random seeds, the mean uniform target-path error decreases from 0.118±0.0150.118\pm0.015 to 0.018±0.0010.018\pm0.001. This supports the trajectory prediction for naturally obtained initializations on fixed image features. It is not an experiment with jointly learned neural features, and the near-zero endpoint recovery phenomenon is demonstrated separately by the analytic and controlled synthetic examples.

We see the fixed-representation setting as a tractable first step toward a broader question: whether related trajectory geometry persists when representations themselves evolve during adaptation or post-training. Establishing that requires new analysis rather than a direct application of the current theorem.

⇒\Rightarrow The revised experiment is reported in Section 5.3 and Figure 4(c) (p. 10, lines 501–527), with the full protocol, all nine thresholds, and scope checks in Appendix E.4 (pp. 33–34, lines 1744–1835). The fixed-feature logistic dynamics and large-initialization family are specified in Section 2.1 (p. 3, lines 124–149).

AuthorsOpenReview ↗

(3/3) Response to Reviewer ZYSu

4. Notation and readability

Question/Suggestion: Please clarify u0u_0, the normalization wk/Rw_k/R, the point xx, and the meaning of the limiting velocity and path derivative.

Response: Thank you for identifying these specific obstacles. We revised the opening of Section 2.2 to introduce the rescaled point and derive its velocity before giving the geometric rules.

  • u0u_0: For w0=Ru0+eRw_0=Ru_0+e_R with ∥eR∥/R→0\|e_R\|/R\to0, it is the limiting position of w0/Rw_0/R. It may have negative target margins and need not be a target separator.
  • wk/Rw_k/R: Division by the fixed initialization scale keeps the starting position finite while retaining changes in both norm and direction. Dividing by the evolving norm would retain direction alone.
  • xx: This is a point in rescaled parameter space, corresponding to the unscaled parameter w=Rxw=Rx.
  • Velocity: With τ=ηk/R\tau=\eta k/R, the normalized update divided by η/R\eta/R is displacement per unit rescaled time. Away from zero margins, its limit is ∑i:zi⊤x<0pizi\sum_{i:z_i^\top x<0}p_i z_i, which defines y˙(τ)\dot y(\tau) along the ideal trajectory. At a change of segment, the geometric rule determines the outgoing motion; the path remains continuous although its velocity can change.

⇒\Rightarrow The initialization and normalization are specified in Section 2.1 (p. 3, lines 136–149). The definition of xx and explicit displacement-per-unit-time derivation are in Section 2.2 (p. 3, lines 153–161); the boundary rule and explanation of the stage transition follow on p. 4, lines 162–180. Section 2.4 states continuity, finite velocity changes, and the segment-by-segment construction of the complete path (p. 5, lines 231–244).

 


We especially appreciate the reviewer’s emphasis on the broader importance of the theory. For us, it is important that an initial research spark be developed into a rigorous piece of work that can be clearly communicated, scrutinized, and discussed by the community, and the reviewer’s questions have helped us sharpen that transition. We believe the most interesting next questions are not merely additional formulas for logistic regression, but whether trajectory-level geometry can support principled decisions about monitoring, stopping, regularization, and stability in richer learning systems. We hope the revision makes clearer both what the present theory establishes and where the genuinely open questions begin. We would be very interested in continuing this discussion with researchers working on optimization, learning dynamics, and post-training, as we believe this viewpoint may open several new directions for analysis.

I thank the authors for your detailed and careful rebuttal. I value the mathematical analysis presented in the paper as well as the additional experiments you did in the rebuttal. However, since the scope of the paper is limited, I can only raise my rating to 2. I expect the authors to extend the theory to more general scenarios and to develop corresponding intervention algorithms in future works.

Reviewer_A6eEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies full-batch gradient descent for logistic regression on strictly linearly separable data under large initialization. The main contribution is an “ideal path” that geometrically approximates the normalized GD trajectory. The path consists of two stages: (1) negative-margin correction and (2) minimum-margin growth. Experiments on synthetic geometries and fixed image features support the theoretical predictions.

Critical strengths
  1. Clean geometric characterization. The two-stage decomposition provides an intuitive interpretation of logistic GD in the large-initialization limit: first correcting negative margins and subsequently increasing the minimum margin.
  2. Interesting implications for transient behavior. The observation that evaluation loss can temporarily grow as O(R), despite being small at both endpoints, illustrates why studying the entire optimization trajectory can reveal behavior missed by endpoint analysis.
Critical weaknesses

Main Weakness: Practical Relevance of the Large-Initialization Regime. My primary concern is the practical motivation for the asymptotic regime R→∞R→\infty. The paper motivates large initialization through reuse of parameters obtained after prolonged training on a previous separable task. More specifically, this motivation relies on unregularized logistic GD, for which the parameter norm diverges while its direction approaches a maximum-margin separator. I am not convinced that this setting adequately represents typical modern transfer-learning or fine-tuning pipelines.

  1. In practical training it is common to stop training based on a finite training budget or validation performance, rather than continuing optimization until the parameter norm becomes very large.
  2. Modern network training (even within the logistic-regression setting) commonly employs mechanisms intended to control optimization and parameter scale, such as regularization and weight decay. These mechanisms fundamentally change the motivation based on indefinitely increasing parameter norms.
  3. Very large parameter scales are not necessarily something practitioners actively seek in deep networks. Extreme parameter or activation scales can interact unfavorably with optimization and numerical stability, and modern architectures and training recipes contain mechanisms to control such behavior.

Therefore, it is unclear whether the limiting regime studied here corresponds to the parameter scales encountered in realistic pretrained neural networks.

Questions for authors

How large are parameter norms in realistic pretrained/fine-tuned models relative to the scale required for the ideal-path approximation to become accurate? Can the authors provide an empirical characterization rather than relying primarily on the R→∞R\to\infty limit?

What happens when the preceding task is trained with weight decay or early stopping? Does an analogous ideal-path characterization exist when the initialization has a bounded norm?

Can the authors demonstrate the predicted two-stage geometry and transient evaluation-loss behavior in an end-to-end neural-network transfer-learning experiment using naturally obtained pretrained weights, without artificially scaling the initialization?

4 comments
AuthorsOpenReview ↗

(1/3) Response to Reviewer A6eE

We thank the reviewer for recognizing the clean two-stage geometry, the implications for transient evaluation loss, and the clarity and soundness of our analysis. Your questions helped us distinguish the mathematical regime we study from the broader motivation of adapting a previously learned model to a new task. As discussed in our overall response, our interest in large initialization arose from a soft-to-hard output-layer question: increasing logit scale makes soft predictions approach a hard decision away from ties, which led us to ask how the corresponding training dynamics behave.

In the revised manuscript, we have added an explicit finite-scale guarantee and replaced the scaled maximum-margin digits initialization with actual pretrained checkpoints. We respond to each of the question in detail below.

 

1. How large must the initialization be?

Concern: Does the large-initialization regime correspond to finite pretrained models, and can its accuracy be checked beyond the asymptotic limit?

Response: We agree that norm divergence under unregularized logistic training is not a general description of modern pretraining. Our result concerns fixed-feature, separable logistic regression, and we do not infer from it a universal parameter-norm threshold for neural networks. The relevant scale depends on feature normalization and target geometry, as well as the step size and path window.

To make the finite-scale content explicit, we added Proposition 2 in the main text and Proposition 5 with explicit constants in Appendix F. Their contribution is an explicit, direction-uniform finite-scale guarantee; this is not a claim of a sharper asymptotic rate than Theorem 1. For unregularized target GD with 0<ηΛ<20<\eta\Lambda<2, every fixed window [0,T][0,T] and every R≥1R\ge1 satisfy

\le K\frac{\lVert e_R\rVert}{R} +\frac{A_T}{\sqrt R} +\frac BR.$$ The constants are explicit and conservative, and can be chosen uniformly over unit initial directions. For an actual checkpoint, take $R=\|w_0\|$ and $u=w_0/R$, so $e_R=0$. The bound then compares the normalized target trajectory with the ideal path starting from that checkpoint's own direction, including the transition between stages. It does not require the checkpoint to be close to a source maximum-margin direction. There is also a direct connection to confidence on fixed pretraining features. Writing $C_{\mathrm{pre}}=\min_i\sigma((z_i^{\mathrm{pre}})^\top w_0)$ gives $$R\ge \frac{\operatorname{logit}(C_{\mathrm{pre}})} {\max_i\|z_i^{\mathrm{pre}}\|}.$$ For fixed source and target features, if a sequence of checkpoints attains $C_{\mathrm{pre}}\to1$, the normalized target-path error relative to each checkpoint's own ideal path tends to zero, even if its direction changes. This is a conditional statement about attained confidence, not an assumption that every pretraining procedure drives confidence to one. The guarantee concerns normalized parameter trajectories; it does not itself establish vanishing error in raw logits or predicted probabilities or improved generalization. $\Rightarrow$ Proposition 2 and the confidence interpretation are in Section 3.2 (p. 6, lines 305–323). Appendix F.1 sets up the finite-scale problem (p. 35, lines 1864–1889), and Proposition 5 states the explicit complete-path bound and direction-uniform constants (p. 36, lines 1915–1929). The actual-checkpoint argument and its scope are in Appendix F.6 (pp. 39–40, lines 2097–2128). The finite-checkpoint experiment below independently measures approximation accuracy at naturally obtained scales; the conservative bound does not by itself certify the reported error magnitudes.
AuthorsOpenReview ↗

(2/3) Response to Reviewer A6eE

2. Early stopping, weight decay, and bounded checkpoints

Concern: What changes if the preceding task uses early stopping or weight decay? Does the characterization apply to bounded-norm initializations?

Response: The checkpoint version of the finite-scale bound does not prescribe how the source checkpoint was produced. If early stopping or source-task regularization yields a nonzero checkpoint, we may set R=∥w0∥R=\|w_0\| and u=w0/Ru=w_0/R and compare subsequent unregularized target GD with the ideal path associated with that checkpoint direction. Appendix F.6 makes this source-algorithm independence explicit. However, keeping the checkpoint norm bounded does not, by itself, produce the asymptotic vanishing-error conclusion: the finite-scale bound remains a finite approximation guarantee, and we do not claim an exact piecewise-linear description at arbitrary small norms.

It is important to distinguish regularization during source training from regularization during target training. Source regularization changes the checkpoint to which the finite-scale comparison is applied. Target weight decay changes the update dynamics themselves and is outside the present theorem; we have not established an analogous ideal-path result for those regularized target dynamics. Thus the large-scale limit is used here to isolate and quantify a geometric mechanism, not to recommend increasing parameter norms in practice.

⇒\Rightarrow The source-algorithm independence of the checkpoint interpretation is stated in Appendix F.6 (pp. 39–40, lines 2097–2121). The finite-scale theorem itself studies full-batch, unregularized target GD with 0<ηΛ<20<\eta\Lambda<2 (Appendix F.1, p. 35, lines 1877–1882). The manuscript does not claim that early stopping or fixed source weight decay produces diverging norms, nor does it analyze target-side weight decay.

AuthorsOpenReview ↗

(3/3) Response to Reviewer A6eE

3. Experiments with naturally obtained pretrained weights

Concern: Can the two-stage geometry and transient evaluation-loss behavior be demonstrated using actual pretrained weights, including in neural-network transfer learning?

Response: We have replaced the previous digits initialization with weights obtained from actual source-task training. We pretrain logistic classifiers on all 357 images of digits 3/8, with fixed pixel features and a trainable bias, from ten random initializations. For each seed, we select the first GD iterate reaching each of nine prescribed minimum correct-label probability thresholds between 60% and 99.8%. All 65 parameters are transferred unchanged to the fixed 32-image 5/6 target task, without rescaling or interpolation. Checkpoint selection uses only source data and is fixed before target training.

All 90 target runs cover both stages on the common window [0,0.60][0,0.60]. Each is compared with the ideal path constructed from its actual checkpoint direction, independently of the target GD trajectory. Selected results from Table 6 are:

Source confidence thresholdMean checkpoint norm RRUniform normalized target-path error
60%8.6060.118±0.0150.118\pm0.015
90%14.4210.070±0.0070.070\pm0.007
99%25.6580.025±0.0040.025\pm0.004
99.8%33.7340.018±0.0010.018\pm0.001

Errors are full 65-dimensional suprema over the window, reported as mean ±\pm sample SD across ten seeds, with no alignment of actual and ideal entry times. They demonstrate increasingly accurate trajectory approximation at finite, naturally obtained checkpoint scales in this fixed-feature setting; they are not estimates of the numerical tightness of Proposition 2.

We have not performed the requested experiment with a neural network whose features are also trained. The new digits experiment tests the two-stage geometry using actual source-to-target training of logistic models. The transient loss peak with near-zero endpoint losses is demonstrated separately by the analytic construction and controlled synthetic experiment. We therefore do not claim that the digits experiment establishes the recovery phenomenon or generalizes the theorem to feature-learning networks.

⇒\Rightarrow The new experiment is in Section 5.3 and Figure 4(c) (p. 10, lines 501–527). Appendix E.4 gives the protocol and actual checkpoint-selection rule (p. 33, lines 1744–1799), the complete results and common-direction comparison (p. 34, lines 1801–1823), and verification and scope checks (p. 34, lines 1825–1835). The analytic transient-loss construction is in Section 4.2 and Figure 2 (p. 8, lines 394–418). The controlled synthetic functional experiment is described in Section 5.2 (p. 10, lines 486–499), with Figure 4(a,b) summarized at lines 522–527.

 


We appreciate the reviewer inspiring us to separate the mathematical large-scale regime from claims about realistic pretrained systems. We think the finite-scale checkpoint viewpoint, the distinction between source and target regularization, and the question of whether similar geometry persists with learned representations are important directions in their own right. More broadly, we hope this work opens a useful discussion about when soft-to-hard geometric limits are informative for real adaptation dynamics.

Dear Reviewer A6eE,

We would appreciate it if you could take some time to look through our responses, and we hope your concerns have been well addressed.

Thank you again for all the constructive feedback, and we would be very grateful if you could consider revisiting your assessment of our work!

Best,

The authors

Reviewer_XbzWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies full-batch logistic GD on strictly linearly separable data when the initialization scale becomes large. Its main contribution is a finite, piecewise-linear ideal path describing the normalized training trajectory, rather than only its eventual direction. The path combines negative-margin correction with minimum-margin growth, using minimum-norm geometric rules to resolve boundary cases and tied margins.

The authors prove uniform convergence on fixed intervals of a two-stage time parameterization, with quantitative bounds accounting for initialization perturbations and stage-transition errors. They derive predictions for evaluation loss peaks and cumulative training loss, including examples where intermediate eval loss grows linearly with initialization scale despite vanishing endpoint losses. Validation of the conclusion combines theory, controlled geometries, several random separable problems, and experiments which support the results.

Critical strengths
  • Theorem 1 controls the entire two-stage trajectory including its most delicate transition which is interesting since it approximates normalized fixed-step GD with a finite piecewise-linear path. There is explicit control of both the position and timing of entry into the nonnegative-margin region, rather than aligning the two trajectories. The minimum-norm rules also handle zero margins and tied minimum margins without generic position assumptions. The improved error bound under the nonexpansive-step condition further strengthens this argument

  • the authors construct an evaluation-loss peak of R/25+o(R)R/25 + o(R) while both endpoint losses vanish with the limiting maximum computable from finitely many path vertices. Corollary 2 additionally identifies different cumulative-loss scales before and after correction. These are supported by the experiments/simulations: in the four-sample experiment, the normalized peak at R=80R=80 is larger versus the predicted, and both cumulative-loss coefficients are close to their predicted limits.

  • the paper is also readable and the geometric examples help to make the main construction understandable

Critical weaknesses
  • the digit experiment validates the approximation more strongly than the motivating phenomenon of temporary forgetting followed by recovery. The experiment constructs initialization by directly computing and scaling the source task’s maximum-margin direction, rather than reusing weights obtained from an actual source-task training run. Moreover the authors report in the appendix an ideal normalized endpoint evaluation loss only slightly below the peak. It appears this window does not demonstrate the near-zero endpoint recovery established in the synthetic examples. The larger digit tasks validate entry locations only. These distinctions limit the broader empirical support for the more general transfer-learning motivation

  • Equation (11) defines second-stage time through accumulated Q(wk)Q(w_k), and the functional results stop at the corresponding path-defined index NRN_R. It seems to me the peak prediction and the post-entry cumulative-loss limit do not directly specify when the relevant behavior occurs in normal iteration time. This matters practically: the authors reports random-instance runs requiring up to approximately 1 billion updates. The paper would benefit from a sharper distinction between predicting the geometric trajectory and predicting behavior within an affordable update budget.

Questions for authors
  • Does the current analysis yield a trajectory independent estimate of the iteration count needed to reach a specified second-stage vertex or eval loss peak, ie: a leading-order estimate in log iteration time? If not, please verify that the peak and cumulative-loss predictions concern the path-defined stopping rule rather than a fixed iteration cost

  • Appendix E.3 rejects random candidates when ideal-path construction fails, in addition to applying the rank, weight, and entry-time filters. How many candidates were rejected specifically because construction failed, and what caused those failures? This information would help determine whether the retained instances provide representative evidence for the numerical reliability of the construction.

5 comments
AuthorsOpenReview ↗

(1/3) Response to Reviewer XbzW

We thank the reviewer for the careful review and for recognizing the value of our full two-stage trajectory analysis, its treatment of the stage transition and tied margins, and its predictions for evaluation-loss peaks and cumulative training loss. As discussed in our overall response, we view the ideal path as separating three related questions: what intermediate behavior the geometry predicts, when that behavior is reached in ordinary GD iterations, and how reliably the geometric construction describes finite pretrained checkpoints. Your comments directly helped us sharpen the latter two points.

Your review identifies three distinctions that deserve clearer treatment: actual pretrained weights versus scaled maximum-margin directions, geometric progress versus iteration cost, and successful path construction versus selection of the random instances. We have added an experiment using actual pretrained checkpoints and new iteration-time results, and we report the requested sampling audit below.

 

1. Actual pretrained initializations and the scope of the digits experiment

Concern: The original digits initialization is constructed from the source maximum-margin direction, and the experiment does not demonstrate temporary forgetting followed by recovery to near-zero endpoint losses.

Response: These are two separate limitations, and we address them separately. We have replaced the original digits experiment with sequential source- and target-task training using actual pretrained weights. We pretrain logistic classifiers on all 357 images of digits 3/8, using fixed pixel features and a trainable bias. For each of 10 random initializations, we select the first actual GD iterate reaching each of nine minimum correct-label probability thresholds, from 60% to 99.8%. All 65 checkpoint parameters are then transferred unchanged to the fixed 32-image 5/6 target task. The selection rule uses only source data and involves no rescaling or replacement of the checkpoint direction.

For each checkpoint, let R=∥w0∥R=\|w_0\| and u=w0/Ru=w_0/R. We compare the normalized target trajectory P^\widehat P with the ideal path P(⋅;u)P(\cdot;u) constructed from that checkpoint's direction and the target data. All 90 runs complete the common window [0,0.60][0,0.60] and cover both stages. The error

\|\widehat P(r)-P(r;u)\|_2$$ uses the full 65-dimensional parameter vector, including the bias, without aligning actual and ideal entry times. Selected results are: | Source confidence threshold | Mean checkpoint norm $R$ | $E^{\mathrm{ckpt}}$ (mean $\pm$ sample SD) | | --- | ---: | ---: | | 60% | 8.606 | $0.118\pm0.015$ | | 90% | 14.421 | $0.070\pm0.007$ | | 99% | 25.658 | $0.025\pm0.004$ | | 99.8% | 33.734 | $0.018\pm0.001$ | These results support the trajectory approximation for actual pretrained checkpoints in the fixed-feature setting. Using the common source maximum-margin direction instead gives a larger error of $0.202\pm0.004$ at the 99.8% threshold, compared with $0.018\pm0.001$ for the checkpoint's own direction. This illustrates why the actual direction matters at finite pretraining levels. The new digits experiment does **not** establish near-zero endpoint recovery. That phenomenon is established by the analytic construction and tested separately in the controlled synthetic experiment. We have kept these empirical claims distinct: the digits experiment tests trajectory approximation with naturally obtained pretrained weights, while the controlled examples test the predicted loss peak and recovery. $\Rightarrow$ The replacement experiment appears in Section 5.3 and Figure 4(c) (p. 10, lines 501–527). Appendix E.4 gives the data and optimization protocol and actual checkpoint-selection rule (p. 33, lines 1744–1799), the complete nine-threshold results and common-direction comparison (p. 34, lines 1801–1823), and verification and scope checks (p. 34, lines 1825–1835). The separate controlled peak-and-recovery experiment is in Section 5.2 (p. 10, lines 486–499), while the analytic peak-and-recovery construction is in Section 4.2 and Figure 2 (p. 8, lines 394–418).
AuthorsOpenReview ↗

(2/3) Response to Reviewer XbzW

2. Iteration-time estimates and the stopping rule

Question: Can the analysis predict the iteration count needed to reach a specified second-stage vertex or evaluation-loss peak independently of the actual GD trajectory? How does this relate to a feasible update budget?

Response: We have added results that connect the geometric path directly to ordinary GD iteration time. Let

\qquad K_R(r)=\min\{k\ge0:r_{R,k}\ge r\}.$$ Proposition 3 proves, under the assumptions of Theorem 1, that for every fixed $\tau_*<r_0<r_1<\infty$, $$\sup_{r\in[r_0,r_1]} \left|\frac{\log(1+\eta K_R(r))}{R}-b(P(r))\right| \longrightarrow0.$$ Thus, for a specified second-stage vertex $x_v=P(r_v)$, $$K_R(r_v)=\eta^{-1}\exp\{Rb(x_v)+o(R)\}.$$ The exponent is determined by the ideal path and training geometry, without simulating GD or accumulating its $Q(w_k)$. Although $K_R$ itself is defined through the actual path clock, Corollary 1 also provides a stopping event that does not use that clock: if the ideal path first reaches a linear threshold $h^\top x=c$ at $r_*>\tau_*$ and strictly crosses it immediately afterward, then, for $x_*=P(r_*)$, $$T_R=\inf\{k\ge0:h^\top w_k\ge Rc\}, \qquad \frac{w_{T_R}}R\longrightarrow x_*, \qquad \frac{\log(1+\eta T_R)}R\longrightarrow b(x_*).$$ The revised manuscript now also states the peak-specific consequence directly. If $H_{\mathrm{ev}}(P(r))$ has a unique maximizer $r_p\in(\tau_*,S]$ on $[0,S]$, and $k_R^{\mathrm{peak}}$ is the first actual maximizer of $L_{\mathrm{ev}}(w_k)$ over $0\le k\le K_R(S)$, then Corollary 2 gives $$r_{R,k_R^{\mathrm{peak}}}\longrightarrow r_p, \qquad \frac{\log(1+\eta k_R^{\mathrm{peak}})}R \longrightarrow b(P(r_p)),$$ or equivalently $$k_R^{\mathrm{peak}} =\eta^{-1}\exp\{Rb(P(r_p))+o(R)\}.$$ This directly links the ideal peak to the leading iteration scale of the actual GD peak when the unique maximizer lies in the second stage. The controlled four-sample experiment in Section 5.2 has a correction-stage peak instead; there the first-stage path clock is linear in the raw iteration index rather than exponential. These results also clarify the computational limitation emphasized in the review: a fixed positive-margin point can require exponentially many updates as $R$ grows. The peak and cumulative-loss limits in Section 4.2 remain stated on the path-defined window $N_R=K_R(S)$. The new clock predicts the leading iteration scale associated with such a window, but it does **not** guarantee that the window is reachable within an affordable fixed update budget. $\Rightarrow$ Proposition 3, Corollary 1, and the new peak-time Corollary 2 are in Section 4.1 (p. 7, lines 324–361). Appendix G proves Proposition 3 and Corollary 1 (pp. 40–42, lines 2131–2266); the short argument for Corollary 2 is given directly in the main text (p. 7, lines 346–361). Section 4.2 defines the path-based stopping window and path-functional framework immediately afterward (pp. 7–8, lines 365–393).
AuthorsOpenReview ↗

(3/3) Response to Reviewer XbzW

3. Random-sample rejection and construction reliability

Question: How many random candidates were rejected because ideal-path construction failed, and what caused those failures?

Response: The replay of the original sampling procedure found zero construction failures: all 56 candidates reaching the construction step were successfully constructed.

The original script stored accepted instances without logging rejection counts. We therefore replayed the original seed, generation procedure, filtering order, and construction settings. The 40 retained instances reproduce the archived data, weights, initialization directions, and step sizes to floating-point precision. The breakdown is:

OutcomeNumber of candidates
Rejected: rank below 30
Rejected: minimum sample weight below 0.0533
Rejected: all initial margins nonnegative5
Rejected: ideal-path construction failed0
Rejected: ideal entry time outside [0.05,3][0.05,3]16
Accepted40
Total94

Of the 16 entry-time exclusions, one was below 0.05 and 15 were above 3. Thus there are no observed construction failures in this replay to attribute to numerical causes.

The filters use the sampled geometry, weights, and ideal entry time before GD is run; no candidate is selected by its GD approximation error. All 200 runs on the retained 40 instances enter the reported random-family comparison. These results document successful construction for every candidate reaching that step in this replay; the retained family is still conditioned on the stated filters, so this is not an unfiltered or worst-case reliability guarantee.

⇒\Rightarrow The sampling and rejection procedure itself is specified in Appendix E.3 (pp. 31–32, lines 1665–1678). The replay counts above are an additional audit supplied in this response; they are not used to define or select the reported approximation errors.

 


We greatly appreciate these questions, which helped us sharpen what we now see as an important distinction between geometric progress and ordinary iteration time. We hope the revised results make that distinction precise, and we would welcome further discussion with researchers interested in optimization dynamics, trajectory-level diagnostics, and the computational implications of such geometric limits.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to the Review Team on: Motivation, Contributions, and Broader Significance

We sincerely thank the reviewers for their careful reading and thoughtful feedback. Their questions have helped us clarify the motivation, scope, and significance of the work. Since several reviews raise broader questions about the paper's motivation and contribution, we use this opportunity to explain what led us here, what the geometric characterization enables, and its relevance to learning and adaptation.

 


How this work began. This work grew out of our study of output-layer decision rules, particularly softmax and hardmax. A question that had occupied us for some time was how the soft-to-hard limit is reflected not just in predictions, but in training dynamics. For a linear output layer, scaling its logits is equivalent, at the prediction level, to lowering the softmax temperature; away from ties, the probabilities approach a hard decision. This led us to study training from large initialization in binary logistic regression. We found that the limiting geometry captures only part of the dynamics. The scaled loss approaches a polyhedral function whose flow corrects negative margins and stops once all training margins are nonnegative. Yet the soft logistic dynamics continue on a slower time scale, governed by the samples with the smallest margins. Connecting these two descriptions led us to the two-stage ideal path.

What the geometric approximation enables. The ideal path is a finite, piecewise-linear object determined by the training data, sample weights, and normalized initialization. Under strict separability, we prove that the normalized trajectory of unregularized, fixed-step, full-batch GD converges uniformly to this path as the initialization scale grows, on every fixed window of the two-stage time parameter, including across the stage transition. This makes intermediate training behavior accessible beyond the eventual parameter direction. In particular, the limiting normalized evaluation-loss peak can be computed from finitely many path vertices, and the analysis yields distinct cumulative-loss scales for the two stages. The iteration-time results add the leading update scale for reaching specified second-stage positions, linear first-passage events, and, under a unique second-stage maximizer, the actual evaluation-loss peak. Together, these results connect where the trajectory goes, what losses it encounters, and how long the progress takes.

Why we believe this matters for learning and adaptation. One concrete insight is that nearly hard predictions on the training data need not mean that evaluation performance has stabilized. In our model, residual soft gradients continue to move the parameters after negative-margin correction, substantially changing evaluation margins. Our construction shows that two checkpoints with small evaluation losses can be separated by a large transient loss. The iteration clock adds a complementary insight: even when the geometry permits later recovery, reaching that part of the path can require exponentially many updates. Whether deterioration is transient and how costly recovery is are therefore separate questions.

These distinctions matter when deciding whether to stop or continue adaptation. The path identifies portions of the trajectory with unfavorable evaluation margins, while the clock describes their asymptotic iteration scale. The active-set geometry also identifies which training samples govern each segment. We see these quantities as a basis for more targeted evaluation and checkpointing, and eventually stopping or regularization strategies that avoid undesirable excursions. The present paper establishes these predictions, rather than a provable generalization improvement or validated stabilization method.

The most direct connection beyond our experiments is a binary logistic output head trained on frozen representations under our separability and GD assumptions. This provides a concrete setting for studying adaptation from a confident pretrained model. More broadly, we hope related geometry can help anticipate and mitigate intermediate performance degradation during neural-network or LLM post-training. Extending the analysis to changing representations and other update rules is necessary before drawing such conclusions for full-model training.

 


We believe this viewpoint opens questions beyond the present paper, particularly how trajectory geometry extends to richer models and might eventually inform principled interventions during adaptation. We hope this perspective contributes a useful addition to the conference and broader discussions on learning dynamics and post-training. We sincerely appreciate the opportunity to exchange ideas with the community.

AuthorsOpenReview ↗

Summary of Key Revisions

We summarize the main revisions below and then respond to each reviewer in detail.

  1. Explicit finite-scale guarantees. We added a bound with explicit constants, uniform over unit checkpoint directions, together with a conditional connection between attained pretraining confidence and target-path approximation for fixed tasks. This allows each checkpoint to be compared with its own ideal path without requiring convergence to a common source maximum-margin direction (Section 3.2, p. 6, lines 305–323; Proposition 5 in Appendix F.1, p. 36, lines 1915–1929; Appendix F.6, pp. 39–40, lines 2097–2128).

  2. Iteration-time predictions. We added the leading logarithmic iteration count along the second stage, a first-passage result for a strictly crossed linear threshold, and a corollary characterizing the iteration time of a unique second-stage evaluation-loss peak (Section 4.1, p. 7, lines 324–361). Proofs of Proposition 3 and Corollary 1 appear in Appendix G (pp. 40–42, lines 2131–2266); the peak-time argument is given directly after Corollary 2 in the main text.

  3. Actual pretrained checkpoints. We replaced the scaled maximum-margin digits initialization with weights obtained from source-task logistic training and transferred unchanged to the target task. Across nine confidence thresholds and ten seeds, the mean uniform path error relative to each checkpoint's own ideal path decreases from 0.1180.118 to 0.0180.018 as the threshold increases. This experiment tests trajectory approximation on fixed image features; the transient peak with small endpoint losses is demonstrated separately by the analytic and controlled synthetic examples (Section 5.3 and Figure 4(c), p. 10, lines 501–527; Appendix E.4, pp. 33–34, lines 1744–1835).

  4. Clearer construction and notation. We now explicitly define the point in rescaled parameter space and derive the limiting velocity from displacement per unit rescaled time before introducing the geometric rules (Section 2.2, pp. 3–4, lines 153–180). Section 2.4 then makes the continuity and velocity changes of the complete path explicit (p. 5, lines 231–244).

  5. Clearer practical interpretation. We now state explicitly in the abstract that the path and iteration-time characterization provide a diagnostic of intermediate adaptation behavior that endpoint evaluations can miss (p. 1, lines 29–32). The revised contribution summary also separates the geometric characterization, iteration-time results, path functionals, and pretrained-checkpoint experiments (Section 1.1, p. 2, lines 64–84).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.