- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper develops a theoretical framework showing that Bézier curves between two trained models can provide better multi-task learning and unlearning performance compared to linear interpolation.
Critical strengths
A theoretical explanation is indeed what the current mode-connectivity (between independently trained models) literature needs. Any account that goes beyond generic overparameterization arguments would make a valuable contribution to the mode-connectivity field.
Critical weaknesses
(1) The scope of this paper is "shared-initialization cross-task near-linear mode connectivity" rather than "mode connectivity across independently trained models".
The author claims that "mode connectivity refers to the phenomenon that independently trained neural networks..." (in the abstract). However, their theoretical explanation is built upon "models are trained from a common random initialization". Mathematically, if two models are independently obtained, then the layer-wise cosine similarity of each layer's weights should be zero. However, if you start training from the same initialization, layer-wise cosine similarity is not zero [2].
Empirical results are also obtained with an identical pretrained model (pretrained ViT-tiny). This is not mode connectivity, either. This falls within the scope of linear mode connectivity, see [1]: "linear mode connectivity has been consistently observed in model spawning", "model spawning: A network is randomly initialized, trained for a small number of epochs, and then spawned into two copies which continue to be independently trained using different SGD randomnesses"
(2) The authors' results are trivial in the Linear Mode Connectivity (LMC) context.
LMC works already show that linear interpolation has low-loss values between two permuted/spawned modes [1]. Given this context, the authors further show that the Bezier curve is better than linear interpolation. One easy assumption is that the Bezier curve bends the linear interpolation a little bit towards the "low-loss" direction.
Figure 1, the midpoint of the Bezier curve is obtained with training on the full dataset (union of two subsets, Line 404), while the left and right terminal points are obtained with a partial dataset. This naturally causes "Bezier paths substantially outperform direct linear interpolation across all partitions" (Line 391) because the Bezier path is trained on the full dataset.
Questions for authors
I am not sure I understand Equation 11. Is this a standard approach to unlearning? The objective is unbounded and can be minimized by driving towards .
8 comments
Response to Reviewer's comments (Weaknesss Part)
Next we address the weaknesses pointed out by the reviewer.
- The scope of this paper is "shared-initialization cross-task near-linear mode connectivity" rather than "mode connectivity across independently trained models".
Answer: We thank the reviewer for highlighting this distinction. Our theoretical results concern models trained on different tasks from a common initialization under the approximation. The ViT experiments likewise use task-specific models fine-tuned from a shared pretrained checkpoint. We agree that the phrase "independently trained'' can obscure this scope, and we will revise the abstract and introduction to describe the shared initialization explicitly.
This setting is closely related to the shared-initialization and model-spawning settings discussed in [1,2]. In particular, [1] describes linear mode connectivity as a form of mode connectivity, and [2] studies it by training copies of a common model with different SGD randomness. The connectivity properties we study concern the loss along the connecting path and does not require the endpoint weights to have zero layer-wise cosine similarity. Shared initialization is naturally available when a common pretrained model is fine-tuned for different tasks, making this setting directly relevant to combining task-specific models.
Within this setting, our contribution is an explicit analysis of linear and optimized quadratic Bézier paths under different learning objectives. For example, in multi-task learning, and fit and , respectively, but can have nonzero losses on the other task. Shared initialization therefore does not by itself establish that their interpolation performs well on both tasks. Theorem 3.1 and Corollary 3.2 characterize the pointwise and cumulative multi-task losses in terms of these cross-task residuals, identifying when and by how much optimizing the Bézier control improves upon linear interpolation. Our unlearning analysis similarly identifies conditions under which models on the connecting paths more closely match the retain-only reference in terms of the task losses.
These results complement the same-task linear connectivity studied in [1,2] by quantifying how optimized nonlinear paths balance multiple or competing objectives. This provides a theoretical basis for using path optimization in multi-task learning and unlearning within the shared-initialization setting. We will emphasize this contribution while making the scope of our assumptions clearer.
- The results are trivial in the Linear Mode Connectivity (LMC) context.
Answer: We thank the reviewer for raising this point. The same-task LMC results in [1] establish low-loss connections between solutions evaluated on a common task. In our multi-task setting, each endpoint fits its own task but can have nonzero loss on the other task. The question is therefore how the connecting path balances these different objectives, and how optimizing the path changes its multi-task performance.
We agree that the inclusion of the linear path in the quadratic family implies a weak cumulative-loss inequality under exact optimization. Our contribution is the quantitative characterization of this improvement. In the -linearized model, Corollary 3.2 gives the exact relation
=\frac{3}{8} \int_0^1 \ell_+^{\mathrm{NTK}}(\theta_I(\alpha)) d\alpha,$$ with strict improvement whenever at least one cross-task residual is nonzero. Theorem 3.1 further establishes that the optimized Bézier path has no greater multi-task loss at every $\alpha\in[1/5,4/5]$. These results specify both the magnitude of the cumulative improvement and an entire interval of pointwise improvement, neither of which follows from the inclusion of the linear path alone. For finite-width networks, the comparison additionally requires the predicted improvement to exceed the stated linearization errors. Regarding Figure 1, we indeed optimize the Bézier control point using the union of the two task datasets while keeping the endpoints fixed. This reuses the original training data and requires additional data access and optimization compared with constructing the linear path. We agree that this additional training contributes to the observed improvement. Figure 1 demonstrates the performance of the resulting family of models, including improvements over broad portions of the path, but does not by itself separate the benefit of path optimization from that of further training on the joint dataset. [1] https://arxiv.org/abs/2307.08286 [2] https://arxiv.org/abs/1912.05671- Mode Connectivity vs Linear Mode Connectivity
I believe we have reached agreement that the paper is more closely related to the Linear Mode Connectivity (LMC) literature than to the Mode Connectivity literature. I would appreciate it if the authors could revise the abstract and, if feasible, the title, since “mode connectivity” currently appears in the title. However, changing the title may be difficult at this stage.
There is one point I would like to clarify. You state:
"the connectivity properties we study concern the loss along the connecting path and do not require the endpoint weights to have zero layer-wise cosine similarity."
Connecting models whose corresponding layers have zero cosine similarity is generally more difficult. When the cosine similarity is nonzero (between 0 and 1), the problem is easier. This is also why model permutation is required in LMC. Therefore, the statement that your method “does not require the endpoint weights to have zero layer-wise cosine similarity” is somewhat imprecise.
- Thank you for the clarification. I agree that your theoretical contribution provides new insights into model merging. However, this contribution is not sufficiently reflected or demonstrated in the empirical evaluation.
Response to Reviewer's comments (Questions Part)
We thank the reviewer for the detailed review, in particular for identifying the need to state the theoretical scope more precisely and to clarify the optimization objective. We have uploaded a revised version of the manuscript which incorporates reviewers' comments.
We shall first directly answer the reviewer's questions. Due to limited space, we shall address the weaknesses pointed out by the reviewer in a later comment.
Answer: We thank the reviewer for raising this concern. The loss in Equation 11 can indeed be unbounded below without regularization. In our original analysis, the Bézier control point is obtained by minimizing the regularized cumulative unlearning loss specified in Section 4.1. This objective includes the penalty ∥θ−θ(0)∥22, and we explicitly assume that the resulting objective is strictly convex.
In the NTK-linearized model, the Hessian of this control objective is
I+\frac{2}{15} \left( \lambda\Phi_1^\top\Phi_1-\Phi_2^\top\Phi_2 \right) \right],$$ where $\Phi_1$ and $\Phi_2$ are the retain-task and forget-task Jacobians at the common initialization. The factor $2/15$ follows from integrating the squared coefficient of the control point: $\int_0^1[2\alpha(1-\alpha)]^2\,d\alpha=2/15$. The stated strict-convexity assumption is therefore equivalent to requiring the matrix in brackets to be positive definite. Under this condition, the regularized control objective is bounded below and has a unique finite minimizer. We recognize that the connection between Equation 11 and the regularized objective used to obtain the control point could be clearer. We will clarify the role of regularization when introducing the unlearning loss and state the positive-definiteness condition explicitly in Section 4.1.