- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates whether loss-level connectivity holds during the optimization process.
The authors select a pair of models during the optimization process and use a Bezier curve to connect the two models. Then check whether the Bezier curve has a similar loss to the terminal points.
Critical strengths
I would like to use the term Low-Loss Region (LLR) to describe the collection of all models that could achieve loss values smaller than a threshold .
(1) Loss-Level Connectivity is a generalized form of mode connectivity. Mode connectivity states that LLR() is probably a connected region, with being the minimal loss value. The authors generalize this concept and state that the boundary of LLR() is also probably a connected region, with . This generalized form of mode connectivity might be an important step for the mode connectivity field.
(2) The authors provide very comprehensive quantification/measurement of the loss-level connectivity and example-wise analysis.
Critical weaknesses
(1) I don't see the value of using loss-level connectivity as a tool to investigate neural networks. In other words, what information does LLC provide that cannot be obtained from standard mode-connectivity analysis? Since this paper introduces LLC, I think the authors should clarify it.
The authors do provide lots of empirical observations in Section 3. However, I do not find a concrete conclusion in it.
(2) About the Bezier curve.
In 2018, there were two mode-connectivity works: AutoNEB [1] and FGE [2]. In AutoNEB, Figure 5 shows that ResNet20 on CIFAR-100 exhibits a saddle loss (a loss barrier in the middle of the path). However, the FGE method does not show this loss barrier at that time. A more recent paper (LLPF [3]) points out that FGE has a bug in the FGE implementation, causing the obtained modes to be highly correlated in weight space, see Appendix C of [3]. Since Figure 2, the ResNet18@CIFAR100 panel does not show a saddle loss either. I hope the authors could measure the cosine similarity of their terminal points and reveal their source code.
(If I recall correctly, the first step of AutoNEB is to optimize the center point of the linear interpolation, which is identical to the first step of FGE. FGE uses this step to find "Control Point C" [line 153].)
Minor remark:
-
Please explain that "corrupted dataset = fully randomized-label dataset". Corrupted dataset can also indicate that you shift the label by 1, etc. I figured this out only after reading Figure 7 panel d.
-
Do you explain AUROC on its first appearance?
-
Section 2.3, I suggest changing "Parameter Redundancy" to "parameter-space symmetry". Parameter redundancy can be misunderstood to mean "overparameterized neural networks".
[1] AutoNEB: https://arxiv.org/abs/1803.00885
[2] FGE: https://arxiv.org/abs/1802.10026
[3] LLPF: https://arxiv.org/abs/2505.02604
Questions for authors
(1) Please measure the layer-wise cosine similarity of your terminal points. I would also like to request the source code. (I am aware of Section 2.3, you can measure layer-wise cosine similarity and say they are 0 - fully uncorrelated- so connectivity cannot come from redundant symmetries)
(2) Many figures lack variance indications. For example, Figure 5. Is it possible to obtain completely different results with repeated runs? Please report the variance across repeated experiments.
During the rebuttal, I will modify my score based on Weakness 1, Question 1 and Question 2.
Minor personal question:
- I am wondering if it is possible to demonstrate the existence of loss-connectivity from mode connectivity?
Consider two loss values: and . Given , we have . If is the final loss after training and is connected, then is it possible to show that is automatically connected?
Consider two models and during the optimization process. You can just follow the training trajectory from to the first trained model. Then use mode connectivity to arrive at the second trained model. Finally, follow the training trajectory to arrive . This forms a subset loss-connectivity because the loss of this trajectory is usually smaller than . Can we directly induce that loss-connectivity at from the subset loss-connectivity?
- Does loss/mode connectivity still hold when the corruption ratio=1? Figure 7 (d) seems to provide some information, but it is too vague.
5 comments
Response 2 (Cont.)
3. Changing the objective changes the result on the same endpoints.
We keep the endpoints, curve family, and fitting budget fixed. We compare three objectives. The first preserves cross-entropy plus the ℓ₂ penalty (our regularized-level objective). The second preserves cross-entropy alone. The third minimizes the regularized objective. Each entry gives the percentage of corrupted examples memorized by both endpoints that the midpoint predicts as their original class. We report mean ± standard deviation over three pairs (Tables 13 and 14).
| Endpoint objective level | Preserve regularized level | Preserve CE-only level | Minimize regularized loss |
|---|---|---|---|
| 2.549 | 34.2 ± 1.0 | 11.9 ± 0.7 | 0.0 ± 0.1 |
| 2.123 | 23.0 ± 2.4 | 8.2 ± 2.5 | 0.2 ± 0.2 |
| 1.919 | 16.4 ± 2.9 | 4.8 ± 0.2 | 0.1 ± 0.1 |
The decomposition explains how fitting to corrupted labels can weaken while the total objective stays nearly fixed. At the three memorization levels shown above, the regularized-level midpoint has a lower norm penalty than the endpoint mean on average across pairs. Increased cross-entropy, mainly on corrupted examples, approximately offsets this decrease (Sec. 3.6, Table 12). These uneven loss increases appear as larger midpoint residuals on corrupted examples. Cross-entropy uses the training labels, so returning to the original class can increase the loss against a corrupted label.
At these same levels, the regularized-level midpoint improves test accuracy by 3.2–3.6 percentage points over the endpoint mean. Loss-minimizing fitting lowers both objective terms and largely retains the memorized labels. Its midpoint test accuracy is 1.9–3.1 percentage points below the endpoint mean. Both accuracy ranges summarize the three-pair means at these levels (Table 14).
In this comparison, the level-preserving connection reaches states that memorize less and generalize better while keeping the objective nearly unchanged. The endpoints stay the same, but changing the fitting objective changes the outcome. These controls support the role of the preserved objective, not a complete causal account of which examples change.
Together, these results show what is retained between independent states at nearly the same objective level during training. They relate this retention to the data and the learning process.
W2/Q1. Endpoint independence and the Bézier construction and source code.
Table 17 directly checks flattened-weight and layer-wise cosine similarity. The summary below gives ranges across the listed settings and stages. Flattened similarity uses convolutional and linear weight tensors.
| Endpoint construction | Flattened-weight cosine | Median layer-wise cosine |
|---|---|---|
| Independently trained endpoints | −0.003 to +0.011 | −0.001 to +0.010 |
| Independent random initializations | −0.001 to +0.001 | −0.000 to +0.001 |
| Children of a shared checkpoint | +0.23 to +0.56 | +0.40 to +0.72 |
Our endpoints are not children of a common checkpoint. They are also not highly aligned in these measurements. Low cosine alone does not exclude parameter symmetries. Sec. 2.3 separately shows prediction and logit changes along the path. Thus, the path is not simply a function-preserving transformation.
We initialize the Bézier control point at the linear midpoint. We optimize only that point. Both trained endpoints remain fixed (Appendix A.3). Tables 18–20 report actual path deviations.
Our code is available anonymously at https://anonymous.4open.science/r/LLC-code-B7B5/ following the Author Guideline.
W3. Terminology.
Thank you. Sec. 3.1 now states that a fraction p of the training labels is replaced by a different class chosen uniformly at random, so p = 1 is the fully randomized-label case of Sec. 3.5 rather than a deterministic label shift. Sec. 2.3 uses “parameter-space symmetry” instead of “parameter redundancy.” In Sec. 3.2, we expand AUROC as “area under the receiver operating characteristic curve.”
Q2. Variability across repeated runs.
Thank you. The revised version now shows the mean and one standard deviation over three trajectory pairs.
Response 3 (Cont.)
Q3. Level versus sublevel connectivity and randomized labels.
A route through lower-loss solutions gives a sublevel connection. It does not necessarily give a connection at the original level. For example, let The minima form a connected unit circle. For 0 < c < 1, the sublevel set is a connected annulus. But the level set consists of two separate circles. Their squared radii are and . Points on different circles can connect through the minima. But they cannot connect while keeping loss c.
For fully randomized labels, we compare how closely the fitted paths stay at their endpoint-defined level. The results below are mean ± standard deviation over three pairs. The first row uses one pair instead. Clean and randomized-label levels are close but not identical (Table 21).
| Clean-label level | Clean-label Dmax | Random-label level | Random-label Dmax |
|---|---|---|---|
| 3.022 | 0.036 | 3.040 | 0.042 |
| 2.000 | 0.038 ± 0.009 | 2.023 | 0.065 ± 0.038 |
| 1.813 | 0.041 ± 0.005 | 1.781 | 0.043 ± 0.014 |
| 1.638 | 0.035 ± 0.009 | 1.650 | 0.162 ± 0.096 |
| 1.343 | 0.039 ± 0.006 | 1.319 | 0.083 ± 0.047 |
| 1.064 | 0.040 ± 0.010 | 1.096 | 0.077 ± 0.009 |
Some random-label paths remain close to the level while the other examined levels generally have larger and more variable deviations.
Response 1
We appreciate the reviewer for taking the time to carefully review our paper. We deeply appreciate the constructive comments. In this response, we aim to carefully address the feedback provided by the reviewer.
A low-loss connection establishes that two states can be joined. LLC asks which learned predictions remain intact between them at nearly the same objective value. It also asks how this changes during learning. The newly added Sec. 3.6 relates prediction retention to learning time and explains the role of the preserved objective.
Also, the states are learned independently, not inherited from one initialization. Our endpoints start from independent random weights and share no checkpoint. In fact, our selected endpoints have weight cosine similarity of −0.003 to +0.011 (Table 17). Yet we recover approximately level-preserving paths before convergence. These paths do not rely on a route through lower-loss final solutions.
Our aim is to understand how the objective, data, and model shape the loss landscape. At each stage of training, keeping the current objective nearly fixed does not require keeping every prediction the models have learned so far. Loss increases on some examples can be balanced by decreases in other example losses or the norm penalty. The residuals capture these uneven changes. By repeating this comparison as training progresses, we examine how the predictions retained between independent states change as the models learn from the data, without simply moving to a lower objective value.
The main findings are as follows.
1. On clean data, earlier-learned predictions are better retained.
We examine examples correctly classified by both endpoints and check whether their predictions remain correct at the curve midpoint. We define learning recency (epochs from the first consistently correct prediction to the endpoint). Consistently correct means correct at every later evaluated checkpoint. The results for ResNet-18/CIFAR-100 without label corruption (across three trajectory pairs) are shown below. (Most recent is the group learned closest to the endpoints. Oldest is the group learned longest before the endpoints.) Each percentage is the fraction of examples in that group whose midpoint prediction differs from the training label, multiplied by 100.
2. During memorization, the level-preserving connection reaches states that memorize less and generalize better at nearly the same objective value.
We compare clean and corrupted examples with similar endpoint losses. Both endpoints predict the training label for each example. Each percentage is the fraction of examples in that group whose midpoint prediction differs from the training label, multiplied by 100. A prediction change is not necessarily a correction to the original class. The results for ResNet-18/CIFAR-100 with 20% corruption (across three pairs for memorization stage and two pairs for converged stage) are shown below.
Clean predictions are better retained during memorization. Moreover, about 45% of changed memorized predictions return to their original class at each of these three levels. We estimate this from the Returned/Abandoned percentages in Table 13. One possible interpretation for this is that class-related structure is more consistently retained between the independent states than their memorization of individual corrupted labels. We do not directly measure shared representations.
Examining only converged models would miss this contrast during memorization. Both change rates fall to about 0.1% at convergence. The labels still remain corrupted. Residual-based corruption detection still has AUROC 0.81 ± 0.03 (Appendix D.2.1, paragraph below Table 12). Thus, informative loss differences persist without substantial prediction changes.