acceptodds
Under review as a conference paper at ICLR 2027

Loss-Level Connectivity Across Optimization Trajectories

Abstract

The training data and optimization process jointly shape how deep neural networks navigate their loss landscapes. This interaction motivates studying not only the geometry of optimization, but also how that geometry relates to the model’s evolving behavior on the training data. Yet, existing studies have largely focused on loss-landscape geometry around converged solutions, where learning from the training data is largely complete. Mode connectivity exemplifies this line of work by asking whether independently trained solutions can be connected through paths of low loss. This perspective naturally raises a question during training: how is this geometry organized while the model is still learning the structure of the training data? However, extending this perspective to intermediate training stages is less straightforward, because the objective itself changes continuously as the network continues to learn from the data. To separate this changing objective value from the geometry being examined, we study connectivity between independently trained states at matched objective levels, which we refer to as loss-level connectivity. In the settings we examine, matched-loss states can be connected by fitted nonlinear paths that approximately preserve the endpoint-defined objective level across multiple stages of training. We then analyze example-wise residuals along these paths. These residuals distinguish controlled subsets of training examples even after matching their endpoint losses, and vary systematically under label-corruption and training interventions. These findings show that path-based measurements can contain information that is not captured by scalar endpoint loss alone.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Loss Landscape, Training Dynamics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RhdAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether loss-level connectivity holds during the optimization process.

The authors select a pair of models during the optimization process and use a Bezier curve to connect the two models. Then check whether the Bezier curve has a similar loss to the terminal points.

Critical strengths

I would like to use the term Low-Loss Region (LLR) to describe the collection of all models that could achieve loss values smaller than a threshold LtL_{t}.

(1) Loss-Level Connectivity is a generalized form of mode connectivity. Mode connectivity states that LLR(LminL_{min}) is probably a connected region, with LminL_{min} being the minimal loss value. The authors generalize this concept and state that the boundary of LLR(L1L_1) is also probably a connected region, with L1>LminL_1>L_{min}. This generalized form of mode connectivity might be an important step for the mode connectivity field.

(2) The authors provide very comprehensive quantification/measurement of the loss-level connectivity and example-wise analysis.

Critical weaknesses

(1) I don't see the value of using loss-level connectivity as a tool to investigate neural networks. In other words, what information does LLC provide that cannot be obtained from standard mode-connectivity analysis? Since this paper introduces LLC, I think the authors should clarify it.

The authors do provide lots of empirical observations in Section 3. However, I do not find a concrete conclusion in it.

(2) About the Bezier curve.

In 2018, there were two mode-connectivity works: AutoNEB [1] and FGE [2]. In AutoNEB, Figure 5 shows that ResNet20 on CIFAR-100 exhibits a saddle loss (a loss barrier in the middle of the path). However, the FGE method does not show this loss barrier at that time. A more recent paper (LLPF [3]) points out that FGE has a bug in the FGE implementation, causing the obtained modes to be highly correlated in weight space, see Appendix C of [3]. Since Figure 2, the ResNet18@CIFAR100 panel does not show a saddle loss either. I hope the authors could measure the cosine similarity of their terminal points and reveal their source code.

(If I recall correctly, the first step of AutoNEB is to optimize the center point of the linear interpolation, which is identical to the first step of FGE. FGE uses this step to find "Control Point C" [line 153].)

Minor remark:

  • Please explain that "corrupted dataset = fully randomized-label dataset". Corrupted dataset can also indicate that you shift the label by 1, etc. I figured this out only after reading Figure 7 panel d.

  • Do you explain AUROC on its first appearance?

  • Section 2.3, I suggest changing "Parameter Redundancy" to "parameter-space symmetry". Parameter redundancy can be misunderstood to mean "overparameterized neural networks".

[1] AutoNEB: https://arxiv.org/abs/1803.00885

[2] FGE: https://arxiv.org/abs/1802.10026

[3] LLPF: https://arxiv.org/abs/2505.02604

Questions for authors

(1) Please measure the layer-wise cosine similarity of your terminal points. I would also like to request the source code. (I am aware of Section 2.3, you can measure layer-wise cosine similarity and say they are 0 - fully uncorrelated- so connectivity cannot come from redundant symmetries)

(2) Many figures lack variance indications. For example, Figure 5. Is it possible to obtain completely different results with repeated runs? Please report the variance across repeated experiments.

During the rebuttal, I will modify my score based on Weakness 1, Question 1 and Question 2.

Minor personal question:

  • I am wondering if it is possible to demonstrate the existence of loss-connectivity from mode connectivity?

Consider two loss values: L1L_1 and L2L_2. Given L1<L2L_1<L_2, we have LLR(L1)⊂LLR(L2)LLR(L_1) \subset LLR(L_2). If L1L_1 is the final loss after training and LLR(L1)LLR(L_1) is connected, then is it possible to show that LLR(L2)LLR(L_2) is automatically connected?

Consider two models AA and BB during the optimization process. You can just follow the training trajectory from AA to the first trained model. Then use mode connectivity to arrive at the second trained model. Finally, follow the training trajectory to arrive BB. This forms a subset loss-connectivity because the loss of this trajectory is usually smaller than L2L_2. Can we directly induce that loss-connectivity at L2L_2 from the subset loss-connectivity?

  • Does loss/mode connectivity still hold when the corruption ratio=1? Figure 7 (d) seems to provide some information, but it is too vague.
5 comments
AuthorsOpenReview ↗

Response 1

We appreciate the reviewer for taking the time to carefully review our paper. We deeply appreciate the constructive comments. In this response, we aim to carefully address the feedback provided by the reviewer.

W1. What LLC reveals beyond low-loss connectivity.

A low-loss connection establishes that two states can be joined. LLC asks which learned predictions remain intact between them at nearly the same objective value. It also asks how this changes during learning. The newly added Sec. 3.6 relates prediction retention to learning time and explains the role of the preserved objective.

Also, the states are learned independently, not inherited from one initialization. Our endpoints start from independent random weights and share no checkpoint. In fact, our selected endpoints have weight cosine similarity of −0.003 to +0.011 (Table 17). Yet we recover approximately level-preserving paths before convergence. These paths do not rely on a route through lower-loss final solutions.

Our aim is to understand how the objective, data, and model shape the loss landscape. At each stage of training, keeping the current objective nearly fixed does not require keeping every prediction the models have learned so far. Loss increases on some examples can be balanced by decreases in other example losses or the norm penalty. The residuals capture these uneven changes. By repeating this comparison as training progresses, we examine how the predictions retained between independent states change as the models learn from the data, without simply moving to a lower objective value.

The main findings are as follows.

1. On clean data, earlier-learned predictions are better retained.

We examine examples correctly classified by both endpoints and check whether their predictions remain correct at the curve midpoint. We define learning recency (epochs from the first consistently correct prediction to the endpoint). Consistently correct means correct at every later evaluated checkpoint. The results for ResNet-18/CIFAR-100 without label corruption (across three trajectory pairs) are shown below. (Most recent is the group learned closest to the endpoints. Oldest is the group learned longest before the endpoints.) Each percentage is the fraction of examples in that group whose midpoint prediction differs from the training label, multiplied by 100.

Objective levelMost recent predictions changed (%)Oldest predictions changed (%)
2.8732213
2.047307
1.930363

2. During memorization, the level-preserving connection reaches states that memorize less and generalize better at nearly the same objective value.

We compare clean and corrupted examples with similar endpoint losses. Both endpoints predict the training label for each example. Each percentage is the fraction of examples in that group whose midpoint prediction differs from the training label, multiplied by 100. A prediction change is not necessarily a correction to the original class. The results for ResNet-18/CIFAR-100 with 20% corruption (across three pairs for memorization stage and two pairs for converged stage) are shown below.

StageObjective levelMatched clean predictions changed (%)Memorized noisy predictions changed (%)
Memorization2.54920.975.4
Memorization2.12310.750.4
Memorization1.9197.236.2
Converged0.3490.10.1

Clean predictions are better retained during memorization. Moreover, about 45% of changed memorized predictions return to their original class at each of these three levels. We estimate this from the Returned/Abandoned percentages in Table 13. One possible interpretation for this is that class-related structure is more consistently retained between the independent states than their memorization of individual corrupted labels. We do not directly measure shared representations.

Examining only converged models would miss this contrast during memorization. Both change rates fall to about 0.1% at convergence. The labels still remain corrupted. Residual-based corruption detection still has AUROC 0.81 ± 0.03 (Appendix D.2.1, paragraph below Table 12). Thus, informative loss differences persist without substantial prediction changes.

AuthorsOpenReview ↗

Response 2 (Cont.)

3. Changing the objective changes the result on the same endpoints.

We keep the endpoints, curve family, and fitting budget fixed. We compare three objectives. The first preserves cross-entropy plus the ℓ₂ penalty (our regularized-level objective). The second preserves cross-entropy alone. The third minimizes the regularized objective. Each entry gives the percentage of corrupted examples memorized by both endpoints that the midpoint predicts as their original class. We report mean ± standard deviation over three pairs (Tables 13 and 14).

Endpoint objective levelPreserve regularized levelPreserve CE-only levelMinimize regularized loss
2.54934.2 ± 1.011.9 ± 0.70.0 ± 0.1
2.12323.0 ± 2.48.2 ± 2.50.2 ± 0.2
1.91916.4 ± 2.94.8 ± 0.20.1 ± 0.1

The decomposition explains how fitting to corrupted labels can weaken while the total objective stays nearly fixed. At the three memorization levels shown above, the regularized-level midpoint has a lower norm penalty than the endpoint mean on average across pairs. Increased cross-entropy, mainly on corrupted examples, approximately offsets this decrease (Sec. 3.6, Table 12). These uneven loss increases appear as larger midpoint residuals on corrupted examples. Cross-entropy uses the training labels, so returning to the original class can increase the loss against a corrupted label.

At these same levels, the regularized-level midpoint improves test accuracy by 3.2–3.6 percentage points over the endpoint mean. Loss-minimizing fitting lowers both objective terms and largely retains the memorized labels. Its midpoint test accuracy is 1.9–3.1 percentage points below the endpoint mean. Both accuracy ranges summarize the three-pair means at these levels (Table 14).

In this comparison, the level-preserving connection reaches states that memorize less and generalize better while keeping the objective nearly unchanged. The endpoints stay the same, but changing the fitting objective changes the outcome. These controls support the role of the preserved objective, not a complete causal account of which examples change.

Together, these results show what is retained between independent states at nearly the same objective level during training. They relate this retention to the data and the learning process.

W2/Q1. Endpoint independence and the Bézier construction and source code.

Table 17 directly checks flattened-weight and layer-wise cosine similarity. The summary below gives ranges across the listed settings and stages. Flattened similarity uses convolutional and linear weight tensors.

Endpoint constructionFlattened-weight cosineMedian layer-wise cosine
Independently trained endpoints−0.003 to +0.011−0.001 to +0.010
Independent random initializations−0.001 to +0.001−0.000 to +0.001
Children of a shared checkpoint+0.23 to +0.56+0.40 to +0.72

Our endpoints are not children of a common checkpoint. They are also not highly aligned in these measurements. Low cosine alone does not exclude parameter symmetries. Sec. 2.3 separately shows prediction and logit changes along the path. Thus, the path is not simply a function-preserving transformation.

We initialize the Bézier control point at the linear midpoint. We optimize only that point. Both trained endpoints remain fixed (Appendix A.3). Tables 18–20 report actual path deviations.

Our code is available anonymously at https://anonymous.4open.science/r/LLC-code-B7B5/ following the Author Guideline.

W3. Terminology.

Thank you. Sec. 3.1 now states that a fraction p of the training labels is replaced by a different class chosen uniformly at random, so p = 1 is the fully randomized-label case of Sec. 3.5 rather than a deterministic label shift. Sec. 2.3 uses “parameter-space symmetry” instead of “parameter redundancy.” In Sec. 3.2, we expand AUROC as “area under the receiver operating characteristic curve.”

Q2. Variability across repeated runs.

Thank you. The revised version now shows the mean and one standard deviation over three trajectory pairs.

AuthorsOpenReview ↗

Response 3 (Cont.)

Q3. Level versus sublevel connectivity and randomized labels.

A route through lower-loss solutions gives a sublevel connection. It does not necessarily give a connection at the original level. For example, let L(u,v)=(u2+v2−1)2.L(u,v)=(u^2+v^2-1)^2. The minima form a connected unit circle. For 0 < c < 1, the sublevel set is a connected annulus. But the level set consists of two separate circles. Their squared radii are (1−c)(1-\sqrt{c}) and (1+c)(1+\sqrt{c}). Points on different circles can connect through the minima. But they cannot connect while keeping loss c.

For fully randomized labels, we compare how closely the fitted paths stay at their endpoint-defined level. The results below are mean ± standard deviation over three pairs. The first row uses one pair instead. Clean and randomized-label levels are close but not identical (Table 21).

Clean-label levelClean-label DmaxRandom-label levelRandom-label Dmax
3.0220.0363.0400.042
2.0000.038 ± 0.0092.0230.065 ± 0.038
1.8130.041 ± 0.0051.7810.043 ± 0.014
1.6380.035 ± 0.0091.6500.162 ± 0.096
1.3430.039 ± 0.0061.3190.083 ± 0.047
1.0640.040 ± 0.0101.0960.077 ± 0.009

Some random-label paths remain close to the level while the other examined levels generally have larger and more variable deviations.

Reviewer_AJrQOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a novel notion of mode connectivity, namely loss-level connectivity. Loss-level connectivity discusses the landscape before converging to a local minimizer, hence the notion can be applied to any time in training. The authors first show that two points in the training trajectory can be connected by a constant-loss curve that is quadratic bezier (whereas linear interpolation fails). Then they demonstrate that the models retrieved by different optimizers can also be connected. At last, they use label-corruption experiments as a demonstration of how data can affect this new notion of connectivity.

Critical strengths

A new notion of mode connectivity is conceptually interesting, and it can shed new light on the loss landscape. It is interesting to see that the same-loss sets in a neural network training problem is connected, and models from a different optimizer is connected.

Critical weaknesses

I didn't understand why the label-corruption experiments are trying to convey. I believe they want to demonstrate the effect of data in mode connectivity, but the motivation or justification of such experiment needs more work.

Also some reference missing; [1] discusses cross-optimizer connectivity for the first time, at least in the pretraining setup I believe.

[1] Zhang, Fangzhao, et al. "Optimizer-Induced Mode Connectivity: From AdamW to Muon." arXiv preprint arXiv:2605.09991 (2026).

Questions for authors

See critical weaknesses. So the justification of the experiments is what I need for the score to go up.

3 comments
AuthorsOpenReview ↗

Response 1

We appreciate the reviewer for taking the time to carefully review our paper. We deeply appreciate the constructive comments. In this response, we aim to carefully address the feedback provided by the reviewer.

W1. Motivation for label corruption and the missing reference.

We clarified the motivation in Sec. 3 and the conclusions in the newly added Sec. 3.6.

We use label corruption to examine how loss-level connections change as the model learns the data’s class structure and memorizes labels that conflict with it. Changing the corruption ratio changes the labels that the models must learn. Separately, we keep the dataset fixed and change how strongly corrupted examples contribute during training, while using the same curve-fitting objective. We examine how clean and corrupted examples behave along the connections as training progresses. Together, these experiments ask how loss-level geometry reflects both the training data and how the models learn from it.

Separately, the clean-label analysis shows that recently learned predictions are less consistently retained than earlier-learned predictions (Table 16). The noisy-label analysis compares clean fitting and memorized labels. It asks how they are retained between states that already agree (Table 15). The aim is to understand learning through connectivity. It is not merely to show that noise affects the loss.

We also added Zhang et al. [1] to Sec. 4.3. We acknowledge its optimizer-induced and cross-optimizer connectivity results. Our SGDM–Adam experiment asks whether intermediate states can be connected while preserving their matched objective level.

Reference

[1] F. Zhang et al., “Optimizer-Induced Mode Connectivity: From AdamW to Muon,” arXiv:2605.09991, 2026.

Thank you for the clarification. I believe my initial assessment was fair and I remain my score.

Thank you for your time and constructive feedback. Your comments helped us clarify the motivation of our work.

Reviewer_yfPsOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies connections between independently trained checkpoints with similar training loss values. Rather than minimizing loss along a connecting Bezier curve, it fits the curve to preserve the loss level of the endpoints, including at intermediate training stages.

The more interesting finding is at the sample level: clean and synthetically corrupted examples can have similar losses at both endpoint models but different loss patterns along the connecting curve. The authors study this through endpoint-loss matching and residual normalization.

Critical strengths

1- The sample-level finding is interesting. Matching examples on their losses at both endpoints makes the comparison more informative than simply observing that corrupted examples have higher losses. The remaining separation, including after residual normalization, suggests that the path contains information beyond those two scalar measurements.

2 - Section 3 examines the finding from several angles. The corruption-ratio experiments, fixed-corruption training intervention, and reproducibility analyses provide useful checks. I appreciate that the authors acknowledge the limits of causal interpretations of these results.

Critical weaknesses

1 - The contribution is not sufficiently distinguished from relevant prior work. In particular, "Lottery Tickets on a Data Diet" (NeurIPS 2022), already study per-example interpolation loss barriers and use an LMC score to rank and select training examples. This is a closely related prior work missed in the discussion here that is missing. Work such as "Mechanistic Mode Connectivity" (ICML 2023) and "Linear Connectivity Reveals Generalization Strategies" (ICLR 2023) also matters for interpreting connectivity through model behavior. The nonlinear, matched-loss formulation may add something, but the paper does not establish that distinction convincingly.

2 - The significance of the additional signal remains unclear. Fitting a curve to a prescribed loss level appears to be a relatively modest adaptation of existing MC procedures. The stronger contribution therefore needs to come from the sample-level analysis. However, the paper does not establish what this reveals beyond simpler measurements such as ensemble disagreement or training-dynamics statistics. Nor does it establish a mechanism explaining the residual patterns or demonstrate a benefit from using them to clean or re-weight data. I do not require all these outcomes, but I need a stronger case for why the observation matters. The concentration of the central analysis on ResNet-18/CIFAR-100 is another limitation.

Questions for authors

Novelty clarification: How does the sample-level contribution differ from the per-example barriers and LMC scores in Paul et al. (2022), beyond replacing linear interpolation with a fitted level-preserving curve?

Significance clarification: Do your existing analyses establish that the residual descriptors provide information beyond ensemble disagreement, variability of per-example losses across independently trained models, or ordinary training-dynamics statistics? What evidence specifically supports the importance of the connecting geometry?

Method clarification: Section 2 describes fitting deviations from the interpolated endpoint loss, whereas Appendix A.3 uses a constant target equal to the mean endpoint loss. Which objective is actually optimized, and how does the endpoint mismatch affect the reported deviations?

5 comments
AuthorsOpenReview ↗

Response 1

We appreciate the reviewer for taking the time to carefully review our paper.

W1/Q1. Related work and the sample-level contribution.

We have added Paul et al. [1], Lubana et al. [2], and Juneja et al. [3] to Sec. 4.3. We also implemented the per-example barrier of [1] in Table 10.

The difference is not only the curve. It is also which models we compare and what we ask about them. Paul et al. [1] rank examples using linear barriers between two children from shared initialization (or checkpoint). However, our two models instead start from independent random weights. They are trained separately. We compare intermediate checkpoints with similar objective values. We then ask which predictions already learned by both endpoints remain intact along a connection that approximately preserves their objective level.

Lubana et al. [2] and Juneja et al. [3] connect model behavior to mechanistic similarity and generalization strategies. However, our specific focus is example-wise behavior between independently learned intermediate states. We ask how this behavior relates to learning history.

We first select examples with similar losses at both endpoints. The results below measure how well each method distinguishes corrupted from clean examples. They report corruption-detection AUROC on ResNet-18/CIFAR-100 with 20% corruption. The shared-parent comparison starts both children from an epoch-3 checkpoint. It reports a 95% bootstrap interval. The other two columns use independently initialized endpoints. They report mean ± standard deviation across three pairs (Table 10).

LevelShared-parent barrierIndependent linear pathIndependent LLC
2.1230.823 [0.817, 0.829]0.574 ± 0.0080.824 ± 0.007
1.9190.835 [0.829, 0.841]0.580 ± 0.0060.776 ± 0.011
1.0340.845 [0.840, 0.851]0.599 ± 0.0100.671 ± 0.007
0.8220.855 [0.849, 0.860]0.606 ± 0.0100.702 ± 0.018
0.7360.860 [0.854, 0.865]0.601 ± 0.0050.744 ± 0.009

The shared-parent barrier remains strong. We therefore do not claim uniformly better ranking. On the same independently initialized endpoints, however, LLC provides stronger separation than linear interpolation at every reported level.

W2/Q2. What LLC reveals about learning.

Two independently trained models can agree on an example. Yet that prediction may not be preserved between them even when the connection keeps their objective value nearly unchanged. The newly added Sec. 3.6 relates prediction retention to learning time and explains the role of the preserved objective.

We also compare independent learning outcomes, not branches of one initialization. Our models start from independent random weights and share no checkpoint as mentioned above. Thus, the analysis is not restricted to a shared starting point or final solutions.

Our explanation for our findings is that, at each stage of training, keeping the current objective nearly fixed does not require keeping every prediction the models have learned so far. Along the connection, losses can increase on some examples while decreases in other example losses or the norm penalty balance the total. The residuals capture these uneven changes. By repeating this comparison as training progresses, we examine how the predictions retained between independent states change as the models learn from the data. Below, we relate these changes to learning history and test the role of the preserved objective.

The main findings are as follows.

1. On clean data, earlier-learned predictions are better retained along the curve.

We examine examples correctly classified by both endpoints and check whether their predictions remain correct at the curve midpoint. We define learning recency (epochs from the first consistently correct prediction to the endpoint). Consistently correct means correct at every later evaluated checkpoint. The results for ResNet-18/CIFAR-100 without label corruption (across three trajectory pairs) are shown below. (Most recent is the group learned closest to the endpoints. Oldest is the group learned longest before the endpoints.) Each percentage is the fraction of examples in that group whose midpoint prediction differs from the training label, multiplied by 100.

Objective levelMost recent predictions changed (%)Oldest predictions changed (%)
2.8732213
2.047307
1.930363

The difference is not when the models become correct. It is whether their already-learned predictions remain correct between them. Arpit et al. [4] found that networks tend to learn simple patterns first. That observation concerns when predictions are acquired. Here, both models have already acquired the predictions, and there is no label noise. Yet recently learned predictions are lost more often at the midpoint of their level-preserving connection.

AuthorsOpenReview ↗

Response 2 (Cont.)

2. During memorization, the level-preserving connection reaches states that memorize less and generalize better at nearly the same objective value.

We compare clean and corrupted examples with similar endpoint losses. Both endpoints predict the training label for each example. Each percentage is the fraction of examples in that group whose midpoint prediction differs from the training label, multiplied by 100. A prediction change is not necessarily a correction to the original class. The results for ResNet-18/CIFAR-100 with 20% corruption (across three pairs for memorization stage and two pairs for converged stage) are shown below.

StageObjective levelMatched clean predictions changed (%)Memorized noisy predictions changed (%)
Memorization2.54920.975.4
Memorization2.12310.750.4
Memorization1.9197.236.2
Converged0.3490.10.1

Clean predictions are better retained during memorization. Moreover, about 45% of changed memorized predictions return to their original class at each of these three levels. We estimate this from the Returned/Abandoned percentages in Table 13. One possible interpretation for this is that class-related structure is more consistently retained between the independent states than their memorization of individual corrupted labels. We do not directly measure shared representations.

Looking only at converged endpoints would miss this contrast during memorization. At convergence, predictions rarely change. Yet residuals still detect corruption with AUROC 0.81 ± 0.03 (Appendix D.2.1, paragraph below Table 12). Prediction changes and informative loss differences are therefore distinct.

3. The level constraint matters even on the same endpoints.

We keep the endpoints, curve family, and fitting budget fixed. We compare three objectives for curve fitting. The first preserves cross-entropy plus the ℓ₂ penalty (our regularized-level objective). The second preserves cross-entropy alone. The third minimizes the regularized objective. Each entry below gives actual corrections to the original class. These are percentages of all corrupted examples memorized by both endpoints. We report mean ± standard deviation over three pairs in the same setting (Tables 13 and 14).

Endpoint objective levelPreserve regularized levelPreserve CE-only levelMinimize regularized loss
2.54934.2 ± 1.011.9 ± 0.70.0 ± 0.1
2.12323.0 ± 2.48.2 ± 2.50.2 ± 0.2
1.91916.4 ± 2.94.8 ± 0.20.1 ± 0.1

The decomposition explains how fitting to corrupted labels can weaken while the total objective stays nearly fixed. At the three memorization levels shown above, the regularized-level midpoint has a lower norm penalty than the endpoint mean on average across pairs. Increased cross-entropy, mainly on corrupted examples, approximately offsets this decrease (Sec. 3.6, Table 12). These uneven loss increases appear as larger midpoint residuals on corrupted examples. Cross-entropy uses the training labels, so returning to the original class can increase the loss against a corrupted label.

At these same levels, the regularized-level midpoint improves test accuracy by 3.2–3.6 percentage points over the endpoint mean. Loss-minimizing fitting lowers both objective terms and largely retains the memorized labels. Its midpoint test accuracy is 1.9–3.1 percentage points below the endpoint mean. Both accuracy ranges summarize the three-pair means at these levels (Table 14).

In this comparison, the level-preserving connection reaches states that memorize less and generalize better while keeping the objective nearly unchanged. The endpoints stay the same, but changing the fitting objective changes the outcome. These controls support the role of the preserved objective, not a complete causal account of which examples change.

AuthorsOpenReview ↗

Response 3 (Cont.)

4. LLC complements endpoint and ensemble summaries, but not the stronger history baseline.

We compare the following feature sets (Appendix D.1).

Endpoint features are the two cross-entropies, their absolute difference, prediction disagreement, Jensen–Shannon divergence, the two margins, and logit cosine similarity. Ensemble features use five independently trained models, with one checkpoint per model closest to the same objective level. They are the mean and variance of cross-entropy, prediction disagreement (one minus the largest vote share), entropy of the mean prediction, and class-averaged logit variance. History features use 26 features from each endpoint run, computed only up to its endpoint epoch. Each run contributes the forgetting count, mean margin, mean and variance of cross-entropy, first-learned epoch, a never-learned indicator, the ℓ₂ distances between predicted probabilities and the one-hot label at epochs 6 and 9, and an 18-point loss trajectory. All label-based features use the training labels.

Each arrow gives mean corruption-detection AUROC before and after adding LLC residuals. These results use three pairs and endpoint-loss-matched examples in the same setting (Table 11).

Objective levelEndpoint features → + LLCFive-model features → + LLCHistory features → + LLC
2.1230.518 → 0.8720.657 → 0.8730.983 → 0.983
1.9190.517 → 0.8330.656 → 0.8340.985 → 0.985
1.0340.548 → 0.6550.682 → 0.7260.985 → 0.984
0.8220.569 → 0.6830.723 → 0.7640.984 → 0.984
0.7360.577 → 0.7320.737 → 0.7950.983 → 0.983

The endpoint/ensemble gains also hold without matching. However, we want to note that our aim is to understand how data and optimization jointly shape the loss landscape. It is not to outperform history diagnostics. Residual separation extends to CIFAR-10, human-noise CIFAR-10N/100N, ResNet-50, and ViT-Tiny. These results use three pairs per setting (Table 22). This supports the broader residual signal.

Q3. Clarification.

Thank you for pointing out our mistake. The main text misstated the fitting objective. The control point is optimized toward the mean endpoint loss. The deviations are measured against the interpolated endpoint loss, so that the small difference between the two endpoint losses is not counted as a deviation of the path. Sec. 2.1 now states both. As a check, Table 18 also reports Dmax against the mean endpoint loss. The reported three-pair means differ by at most 0.013, and the value against the interpolated loss is never the smaller one.

References

[1] M. Paul et al., “Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks,” NeurIPS, 2022.

[2] E. S. Lubana et al., “Mechanistic Mode Connectivity,” ICML, 2023.

[3] J. Juneja et al., “Linear Connectivity Reveals Generalization Strategies,” ICLR, 2023.

[4] D. Arpit et al., “A Closer Look at Memorization in Deep Networks,” ICML, 2017.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.