- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how cross-group curvature affects the joint optimization of shared and task-specific parameters, as well as experts and routers in mixture-of-experts models. It analyzes conditions under which accounting for these interactions improves local convergence and accelerates escape from coupled saddles. Based on this motivation, it proposes CoPEN, which augments SOAP-style tensor-wise updates with a periodically refreshed, low-dimensional joint-curvature correction. Controlled experiments and Taskonomy evaluations with ViT and V-MoE report improved optimization progress, lower validation losses, and reduced time to selected loss targets compared with the evaluated baselines.
Critical strengths
- The paper clearly explains how cross-group curvature can create optimization difficulties beyond task-gradient conflicts.
- CoPEN incorporates low-dimensional coupling corrections without forming the full Hessian, which is practical.
- The empirical results appear to be competitive.
Critical weaknesses
- The motivation overlaps with ConsMTL[1] in MTL, while the correction framework closely relates to two-level K-FAC. Please highlight the specific novelty of this paper, and compare with these two baselines.
- The analysis uses true Hessian coupling, whereas the practical algorithm employs a positive-semidefinite surrogate, which forms a gap.
- The experiments do not clearly isolate gains from cross-group terms versus other curvature adjustments.
Reference:
[1] Towards Consistent Multi-Task Learning: Unlocking the Potential of Task-Specific Parameters. CVPR 2025.
Questions for authors
Please refer to the listed Weaknesses.
2 comments
Response to Reviewer Rurz (2/2)
1. How accurately does CoPEN approximate coupling under the chosen grouping?
For both metrics, the reference curvature is measured by finite differences along CoPEN's parameter directions.
- Cross-group curvature alignment: cosine similarity between the reference cross-group curvature and CoPEN's estimate. Closer to 1 means more similar coupling patterns.
- Coupling error reduction (%): the reduction in normalized coupling discrepancy relative to omitting coupling, measured as with (spectral norm). Here and are the normalized reference and CoPEN couplings, measured along base-update directions. For a curvature matrix , the normalized off-diagonal blocks are , with diagonal blocks set to zero. Removing all coupling gives , or 0% reduction. Exact agreement gives 100%. Higher is better.
| Diagnostic | ViT | MoE |
|---|---|---|
| Cross-group curvature alignment | 0.97 | 0.99 |
| Coupling error reduction (%) ↑ | 81 | 87 |
| Maximum contraction-factor ratio | 0.508 | 0.422 |
To examine whether practical CoPEN satisfies Theorem 1’s approximation requirement, we assess CoPEN’s estimated coupling using the measured curvature. Following Theorem 1, we compare the coupling-aware contraction factor with the optimal block-diagonal factor to isolate the effect of cross-group coupling. The maximum ratios are 0.508 for ViT and 0.422 for MoE. Since both maxima are below 1, holds in every measurement where Theorem 1 applies, supporting the theorem’s local contraction comparison in these diagnostics.
At the evaluated checkpoints, CoPEN reduces normalized coupling discrepancy by 81% in ViT and 87% in MoE relative to omitting coupling. Coupling results are averaged over three seeds. The MoE measurements use the same grouping as the Block-diag. comparison.
2. Which cross-group interactions matter for optimization?
In MoE, expert and router parameters form separate groups within each routed layer, and task heads form another group. Off-diagonal blocks of capture expert–router, expert–head, router–head, and cross-layer coupling.
A small MoE retains the full model's learned expert–router structure, with eight experts and two routers as separate groups. For each pair, we hold other parameters fixed and compare update directions computed with and without its cross-group curvature. The table reports , where each is the best loss decrease along that direction predicted by the same damped local quadratic model (Appendix D, Table 7):
| Group pair | Reduction in predicted loss decrease |
|---|---|
| Same layer, expert–router | 59% |
| Same layer, expert–expert | 54% |
| Across layers, expert–router | 24% |
| Across layers, expert–expert | 18% |
Values are averaged over six seeds. The largest reductions in predicted decrease occur when within-layer expert–router and expert–expert coupling is removed, supporting the importance of these interactions in preconditioning. In this small MoE, retaining coupling also reduces mean test loss from 6.897 to 5.650 () versus Block-diag. (Appendix D, Table 4).
W3. Does a matched Block-diag. ablation on ViT and MoE isolate the gains from cross-group terms? Yes. The control sets only the off-diagonal blocks of to zero. It retains the same basis-construction and diagonal-block estimation rules, damping, mixing weight, refresh schedule, and training settings.
Validation losses are averaged over three seeds. Updates and training time are reported through each method's final validation checkpoint.
| Metric | ViT CoPEN | ViT Block-diag. | MoE CoPEN | MoE Block-diag. |
|---|---|---|---|---|
| Validation loss ↓ | 0.14306 | 0.14531 | 0.14344 | 0.14401 |
| Updates ↓ | 16,896 | 21,504 | 12,288 | 16,896 |
| Training time (h) ↓ | 1.53 | 1.95 | 2.29 | 3.15 |
At their final validation checkpoints, CoPEN has lower losses on both backbones with 21–27% less training time. The matched control isolates the contribution of cross-group terms while retaining the same within-group correction.
Response to Reviewer Rurz (1/2)
We thank the reviewer for the insightful feedback. We will incorporate the clarifications and comparisons discussed below into the revised paper to address all the points raised.
W1. What is novel relative to ConsMTL and two-level K-FAC?
1. CoPEN avoids separate per-task gradient computation and storage. ConsMTL (Qin et al., CVPR 2025) computes and retains separate task gradients for aggregation, incurring additional time and memory costs.
2. CoPEN preconditions structural coupling shaped by task roles and input-dependent participation. We estimate how updates to different parameter groups jointly affect task outputs and use this curvature to coordinate their updates.
3. Our theory identifies a limitation of even exact within-group preconditioning. Cross-group coupling can still create nearly flat joint directions or saddles despite positive-definite group blocks. Theorem 1 gives conditions for faster local convergence than block-diagonal updates with the best scalar learning rate. Theorem 2 shows that Hessian-aligned coupling strengthens unstable-mode growth while keeping positive-curvature modes stable. These results explain how retaining coupling improves optimization dynamics, distinct from ConsMTL's gradient-alignment derivation (Sec. 3.2, Proposition 3.1) and adaptive two-level K-FAC's natural-gradient approximation-error analysis (Koroko et al., Sec. 3.3).
On Taskonomy ViT and MoE, CoPEN achieves the lowest three-seed mean validation loss and converges in fewer updates than ConsMTL and two-level K-FAC. Both baselines follow the paper's training and hyperparameter-search protocols.
W2. How is the gap between the theoretical analysis and the practical PSD correction addressed? We agree that the gap between the theoretical analysis and the practical PSD correction needs to be addressed.
CoPEN does not explicitly encode negative curvature in its PSD metric, but Theorem 2 also uses a positive-definite preconditioner. Its normalized cross-group block κC/ρ has norm κ<1. Negative curvature enters the dynamics through the loss gradient, which drives an existing perturbation along the unstable direction. The preconditioner Mκ gives this direction relatively greater weight while controlling updates along stable directions, thereby accelerating escape.
Motivated by this mechanism, CoPEN uses the projected PSD curvature K=Q⊤J⊤WJQ to avoid the cost of full-Hessian computation. Its damped inverse in Eq. (16) attenuates directions with larger eigenvalues of K more strongly. When these align with stable Hessian modes and unstable modes lie in lower-curvature directions of K within the represented subspace, the correction can suppress stable components more than unstable ones and support faster escape.
We provide two sets of empirical evidence in the next comment for this theory–practice connection: how accurately CoPEN approximates coupling and how retaining that coupling benefits optimization.