acceptodds
Under review as a conference paper at ICLR 2027

The Coupling Is the Lever: Cross-Group Geometry for Joint Optimization

Abstract

Multi-task models contain heterogeneous parameter groups, including shared and task-specific components as well as experts and routers in mixture-of-experts architectures. Even when individual groups are well conditioned, optimization can remain slow or unstable because important directions may emerge only from their interaction. We show that these hidden joint modes arise from cross-group curvature that is missed when parameter groups are preconditioned independently. Near local minima, they create slow directions, while near saddles, individually stable groups can become unstable jointly. For a fixed multi-task objective, we characterize when accounting for this joint geometry improves local convergence and accelerates saddle escape. We then develop CoPEN (Coupling-aware Preconditioning for Efficient Network Optimization), which augments tensor-wise preconditioning with a low-dimensional model of cross-group curvature without forming the full Hessian. CoPEN applies to both shared-task-specific architectures and input-dependent expert-router interactions in mixture-of-experts models. Controlled and model-scale experiments show improved optimization progress and validation performance over the evaluated optimizer and multi-task baselines.

Primary area:
optimization
Keywords:
Multi-task learning, Second-order optimization, Preconditioning, cross-group curvature

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RurzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how cross-group curvature affects the joint optimization of shared and task-specific parameters, as well as experts and routers in mixture-of-experts models. It analyzes conditions under which accounting for these interactions improves local convergence and accelerates escape from coupled saddles. Based on this motivation, it proposes CoPEN, which augments SOAP-style tensor-wise updates with a periodically refreshed, low-dimensional joint-curvature correction. Controlled experiments and Taskonomy evaluations with ViT and V-MoE report improved optimization progress, lower validation losses, and reduced time to selected loss targets compared with the evaluated baselines.

Critical strengths
  1. The paper clearly explains how cross-group curvature can create optimization difficulties beyond task-gradient conflicts.
  2. CoPEN incorporates low-dimensional coupling corrections without forming the full Hessian, which is practical.
  3. The empirical results appear to be competitive.
Critical weaknesses
  1. The motivation overlaps with ConsMTL[1] in MTL, while the correction framework closely relates to two-level K-FAC. Please highlight the specific novelty of this paper, and compare with these two baselines.
  2. The analysis uses true Hessian coupling, whereas the practical algorithm employs a positive-semidefinite surrogate, which forms a gap.
  3. The experiments do not clearly isolate gains from cross-group terms versus other curvature adjustments.

Reference:

[1] Towards Consistent Multi-Task Learning: Unlocking the Potential of Task-Specific Parameters. CVPR 2025.

Questions for authors

Please refer to the listed Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Rurz (1/2)

We thank the reviewer for the insightful feedback. We will incorporate the clarifications and comparisons discussed below into the revised paper to address all the points raised.

W1. What is novel relative to ConsMTL and two-level K-FAC?

1. CoPEN avoids separate per-task gradient computation and storage. ConsMTL (Qin et al., CVPR 2025) computes and retains separate task gradients for aggregation, incurring additional time and memory costs.

2. CoPEN preconditions structural coupling shaped by task roles and input-dependent participation. We estimate how updates to different parameter groups jointly affect task outputs and use this curvature to coordinate their updates.

  • ConsMTL: Addresses shared–head dependence through gradient alignment. CoPEN instead applies a joint-curvature correction to the parameter update.
  • Two-level K-FAC: The original coarse correction uses one constant basis vector per layer (Tselepidis et al., 2020, Eqs. (5)–(6)). CoPEN uses update directions for distinct structural groups to capture expert–router and group–head coupling. Because one coarse vector covers each layer, two-level K-FAC cannot represent coupling between experts and routers within the same layer. This interaction has the largest effect in our small-MoE analysis (59%, W2), and two-level K-FAC trails CoPEN more on MoE than on ViT (0.14709 vs. 0.14344, compared with 0.14396 vs. 0.14306).

3. Our theory identifies a limitation of even exact within-group preconditioning. Cross-group coupling can still create nearly flat joint directions or saddles despite positive-definite group blocks. Theorem 1 gives conditions for faster local convergence than block-diagonal updates with the best scalar learning rate. Theorem 2 shows that Hessian-aligned coupling strengthens unstable-mode growth while keeping positive-curvature modes stable. These results explain how retaining coupling improves optimization dynamics, distinct from ConsMTL's gradient-alignment derivation (Sec. 3.2, Proposition 3.1) and adaptive two-level K-FAC's natural-gradient approximation-error analysis (Koroko et al., Sec. 3.3).

On Taskonomy ViT and MoE, CoPEN achieves the lowest three-seed mean validation loss and converges in fewer updates than ConsMTL and two-level K-FAC. Both baselines follow the paper's training and hyperparameter-search protocols.

MethodViT full validation loss ↓ViT convergence step ↓ViT s/step ↓ViT time to convergence (h) ↓MoE full validation loss ↓MoE convergence step ↓MoE s/step ↓MoE time to convergence (h) ↓
AdamW0.1444246,0800.1551.980.1460628,1600.2481.94
One-level K-FAC0.2599710,2400.9222.620.347279,2162.4306.22
ConsMTL + AdamW0.1792832,7680.9458.600.1783712,8001.6185.75
Two-level K-FAC0.1439631,2320.9538.270.1470931,7442.48221.89
CoPEN0.1430616,8960.3271.530.1434412,2880.6722.29

W2. How is the gap between the theoretical analysis and the practical PSD correction addressed? We agree that the gap between the theoretical analysis and the practical PSD correction needs to be addressed.

CoPEN does not explicitly encode negative curvature in its PSD metric, but Theorem 2 also uses a positive-definite preconditioner. Its normalized cross-group block κC/ρ\kappa C/\rho has norm κ<1\kappa<1. Negative curvature enters the dynamics through the loss gradient, which drives an existing perturbation along the unstable direction. The preconditioner MκM_\kappa gives this direction relatively greater weight while controlling updates along stable directions, thereby accelerating escape.

Motivated by this mechanism, CoPEN uses the projected PSD curvature K=Q⊤J⊤WJQK=Q^\top J^\top WJQ to avoid the cost of full-Hessian computation. Its damped inverse in Eq. (16) attenuates directions with larger eigenvalues of KK more strongly. When these align with stable Hessian modes and unstable modes lie in lower-curvature directions of KK within the represented subspace, the correction can suppress stable components more than unstable ones and support faster escape.

We provide two sets of empirical evidence in the next comment for this theory–practice connection: how accurately CoPEN approximates coupling and how retaining that coupling benefits optimization.

AuthorsOpenReview ↗

Response to Reviewer Rurz (2/2)

1. How accurately does CoPEN approximate coupling under the chosen grouping?

For both metrics, the reference curvature is measured by finite differences along CoPEN's parameter directions.

  • Cross-group curvature alignment: cosine similarity between the reference cross-group curvature and CoPEN's estimate. Closer to 1 means more similar coupling patterns.
  • Coupling error reduction (%): the reduction in normalized coupling discrepancy relative to omitting coupling, measured as 100(1−e)100(1-e) with e=∥C^−C∥2/∥C∥2e=\Vert\widehat C-C\Vert_2/\Vert C\Vert_2 (spectral norm). Here CC and C^\widehat C are the normalized reference and CoPEN couplings, measured along base-update directions. For a curvature matrix XX, the normalized off-diagonal blocks are Xii−1/2XijXjj−1/2X_{ii}^{-1/2}X_{ij}X_{jj}^{-1/2}, with diagonal blocks set to zero. Removing all coupling gives e=1e=1, or 0% reduction. Exact agreement gives 100%. Higher is better.
DiagnosticViTMoE
Cross-group curvature alignment0.970.99
Coupling error reduction (%) ↑8187
Maximum contraction-factor ratio ε/ρ\varepsilon/\rho0.5080.422

To examine whether practical CoPEN satisfies Theorem 1’s approximation requirement, we assess CoPEN’s estimated coupling using the measured curvature. Following Theorem 1, we compare the coupling-aware contraction factor ε\varepsilon with the optimal block-diagonal factor ρ\rho to isolate the effect of cross-group coupling. The maximum ratios are 0.508 for ViT and 0.422 for MoE. Since both maxima are below 1, ε<ρ\varepsilon<\rho holds in every measurement where Theorem 1 applies, supporting the theorem’s local contraction comparison in these diagnostics.

At the evaluated checkpoints, CoPEN reduces normalized coupling discrepancy by 81% in ViT and 87% in MoE relative to omitting coupling. Coupling results are averaged over three seeds. The MoE measurements use the same grouping as the Block-diag. comparison.

2. Which cross-group interactions matter for optimization?

In MoE, expert and router parameters form separate groups within each routed layer, and task heads form another group. Off-diagonal blocks of KtK_t capture expert–router, expert–head, router–head, and cross-layer coupling.

A small MoE retains the full model's learned expert–router structure, with eight experts and two routers as separate groups. For each pair, we hold other parameters fixed and compare update directions computed with and without its cross-group curvature. The table reports 100(1−Δblock/Δjoint)100(1-\Delta_{\mathrm{block}}/\Delta_{\mathrm{joint}}), where each Δ\Delta is the best loss decrease along that direction predicted by the same damped local quadratic model (Appendix D, Table 7):

Group pairReduction in predicted loss decrease
Same layer, expert–router59%
Same layer, expert–expert54%
Across layers, expert–router24%
Across layers, expert–expert18%

Values are averaged over six seeds. The largest reductions in predicted decrease occur when within-layer expert–router and expert–expert coupling is removed, supporting the importance of these interactions in preconditioning. In this small MoE, retaining coupling also reduces mean test loss from 6.897 to 5.650 (×10−3\times10^{-3}) versus Block-diag. (Appendix D, Table 4).

W3. Does a matched Block-diag. ablation on ViT and MoE isolate the gains from cross-group terms? Yes. The control sets only the off-diagonal blocks of KtK_t to zero. It retains the same basis-construction and diagonal-block estimation rules, damping, mixing weight, refresh schedule, and training settings.

Validation losses are averaged over three seeds. Updates and training time are reported through each method's final validation checkpoint.

MetricViT CoPENViT Block-diag.MoE CoPENMoE Block-diag.
Validation loss ↓0.143060.145310.143440.14401
Updates ↓16,89621,50412,28816,896
Training time (h) ↓1.531.952.293.15

At their final validation checkpoints, CoPEN has lower losses on both backbones with 21–27% less training time. The matched control isolates the contribution of cross-group terms while retaining the same within-group correction.

Reviewer_JDuQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates cross-group parameter coupling as a source of optimization difficulty in multi-task learning and Mixture-of-Experts (MoE) models. The authors show that off-diagonal Hessian blocks can create poorly conditioned joint directions near local minima and negative-curvature directions near saddle points, even when individual parameter groups are well conditioned. They provide theoretical conditions under which coupling-aware preconditioning improves local convergence and accelerates saddle escape compared with block-diagonal preconditioning.

Motivated by this analysis, the authors propose CoPEN, which augments SOAP-style tensor-wise preconditioning with a low-dimensional cross-group curvature correction without constructing the full Hessian. The method is evaluated on controlled toy problems and ten Taskonomy tasks using pretrained ViT-B/16 and V-MoE-B/16 models with full-parameter fine-tuning. The experiments report improved validation performance over the evaluated optimizer and multi-task baselines, along with reduced wall-clock time-to-target compared with AdamW despite higher per-step computational costs.

Critical strengths
  1. The paper identifies cross-group curvature as an optimization bottleneck overlooked by independent group-wise preconditioning. The theoretical analysis characterizes how such coupling can create slow joint directions near local minima and unstable directions near coupled saddles. This provides a meaningful perspective on parameter-space optimization that complements existing methods focused on task-gradient coordination.

  2. CoPEN incorporates cross-group curvature through a low-dimensional correction without constructing the full Hessian. Controlled experiments with matched block-diagonal baselines provide evidence for the benefit of retaining cross-group interactions. On Taskonomy, CoPEN achieves the best mean validation loss among the evaluated methods on both ViT-B/16 and V-MoE-B/16, while reducing wall-clock time-to-target by 16–47% compared with AdamW. These results demonstrate promising practical benefits despite the additional per-step computational cost.

Critical weaknesses
  1. The full-model experiments lack a matched block-diagonal CoPEN baseline, making it difficult to isolate the benefit of cross-group interactions from within-group low-dimensional curvature correction. Moreover, the model-scale evaluation is limited to full fine-tuning of two relatively small pretrained vision models on Taskonomy, without larger-scale or model-scale from-scratch training experiments. The main results report only three-seed averages without variability estimates, and primarily rely on validation losses rather than standard task-specific evaluation metrics. These limitations leave the independent contribution, robustness, scalability, and downstream benefits of CoPEN insufficiently established.

  2. Theorems 1 and 2 analyze idealized preconditioners derived from exact Hessian blocks, whereas practical CoPEN combines SOAP updates with a low-dimensional positive-semidefinite (PSD) curvature approximation. In particular, Theorem 2 relies on cross-group information from an indefinite Hessian (ρ>1\rho > 1), while the practical PSD approximation does not explicitly represent negative Hessian curvature. The paper does not establish whether the practical update in Eq. (16) reproduces the unstable-mode amplification predicted by Theorem 2, or whether it satisfies the approximation condition required by Theorem 1. Consequently, the theoretical results motivate coupling-aware preconditioning but do not directly establish the claimed optimization mechanisms for CoPEN

Questions for authors

Can the authors provide a matched block-diagonal CoPEN ablation on ViT and MoE, removing only the off-diagonal blocks of the projected curvature matrix, to demonstrate that the reported improvements arise specifically from cross-group coupling?

6 comments
AuthorsOpenReview ↗

Response to Reviewer JDuQ (1/2)

We thank the reviewer for the thoughtful comments and constructive questions. We will address the points discussed below in the revised paper.

W1, Q. Does a matched Block-diag. ablation on ViT and MoE isolate the gains from cross-group terms? Yes. The control sets only the off-diagonal blocks of KtK_t to zero. It retains the same basis-construction and diagonal-block estimation rules, damping, mixing weight, refresh schedule, and training settings.

Validation losses are averaged over three seeds. Updates are counted through each method's final validation checkpoint.

MetricViT CoPENViT Block-diag.MoE CoPENMoE Block-diag.
Validation loss ↓0.143060.145310.143440.14401
Updates ↓16,89621,50412,28816,896

At their final validation checkpoints, CoPEN has lower losses on both backbones with 21–27% fewer updates. The matched control isolates the contribution of cross-group terms while retaining the same within-group correction.

W1. Does the evaluation cover larger models? We extend the evaluation to the pretrained OLMoE-1B-7B language model [1], fine-tuned on five task groups from the Tülu 3 SFT mixture [2]: math, code, chat, multilingual, and instruction following (IF). We use the paper's hyperparameter-search protocol and report averages over three seeds.

The table reports final validation loss for each task and the average across the five tasks. The last two columns show CoPEN's updates to reach each baseline's final average loss and the percentage of updates saved.

OptimizerMath ↓Code ↓Chat ↓Multilingual ↓IF ↓Average ↓CoPEN updates ↓Fewer updates
CoPEN0.33820.87561.30251.15161.15410.96442––
Muon0.33860.87681.30451.15111.15640.96547≈18,4945.3%
NorMuon0.33860.87681.30461.15111.15650.96551≈18,4565.5%
Trion0.34150.88001.31321.18181.16010.97530≈15,36921.3%
AdamW0.34250.88091.31541.18721.16040.97728≈14,47625.9%
SOAP0.34450.88661.32611.17501.17460.98135≈12,60435.4%

CoPEN achieves the lowest average validation loss and reaches each baseline's final loss with fewer updates. Across optimizers, validation-loss standard deviations are at most 0.00011, well below CoPEN's improvement over the strongest baseline.

[1] Niklas Muennighoff et al. OLMoE: Open Mixture-of-Experts Language Models. ICLR 2025.

[2] Nathan Lambert et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124, 2024.

W1. Can you report variability across seeds? The table below reports validation loss as mean ± standard deviation across three seeds for SOAP and CoPEN.

ModelSOAPCoPEN
ViT0.14579 ± 0.000280.14306 ± 0.00022
MoE0.14624 ± 0.000120.14344 ± 0.00042

We will report standard deviations for all baselines in the revised paper.

W1. Do the validation-loss gains translate to standard task metrics? We evaluated depth, surface-normal, and segmentation metrics on the full validation split using the selected checkpoints. For brevity, the table shows CoPEN and the best of the six evaluated baselines for each metric. We will include the full comparison in the revised paper.

ViT CoPENViT best baselineMoE CoPENMoE best baseline
Depth (Euclidean) abs. rel. ↓0.14650.1508 (PCGrad)0.14680.1532 (NorMuon)
Depth (z-buffer) abs. rel. ↓0.14990.1545 (AdamW)0.14990.1572 (SOAP)
Normal mean angular error (°) ↓15.4415.53 (NorMuon)15.5915.63 (NorMuon)
Segmentation mIoU (%) ↑49.248.3 (NorMuon)49.848.4 (NorMuon)

These results show that lower validation loss also translates into improved task-specific metrics. For the remaining six tasks, we predict continuous target maps, so MAE or MSE directly measures the error between predicted and target maps.

AuthorsOpenReview ↗

Response to Reviewer JDuQ (2/2)

W2. How does the practical CoPEN update connect to the conditions and mechanisms in Theorems 1–2? We agree that the connection between Theorems 1–2 and the practical update needs clearer evidence.

CoPEN does not explicitly encode negative curvature in its PSD metric, but Theorem 2 also uses a positive-definite preconditioner. Its normalized cross-group block κC/ρ\kappa C/\rho has norm κ<1\kappa<1. Negative curvature enters the dynamics through the loss gradient, which drives an existing perturbation along the unstable direction. The preconditioner MκM_\kappa gives this direction relatively greater weight while controlling updates along stable directions, thereby accelerating escape.

Motivated by this mechanism, CoPEN uses the projected PSD curvature K=Q⊤J⊤WJQK=Q^\top J^\top WJQ to avoid the cost of full-Hessian computation. Its damped inverse in Eq. (16) attenuates directions with larger eigenvalues of KK more strongly. When these align with stable Hessian modes and unstable modes lie in lower-curvature directions of KK within the represented subspace, the correction can suppress stable components more than unstable ones and support faster escape.

We provide two sets of empirical evidence below for this theory–practice connection: how accurately CoPEN approximates coupling and how retaining that coupling benefits optimization.

1. How accurately does CoPEN approximate coupling under the chosen grouping?

For both metrics, the reference curvature is measured by finite differences along CoPEN's parameter directions.

  • Cross-group curvature alignment: cosine similarity between the reference cross-group curvature and CoPEN's estimate. Closer to 1 means more similar coupling patterns.
  • Coupling error reduction (%): the reduction in normalized coupling discrepancy relative to omitting coupling, measured as 100(1−e)100(1-e) with e=∥C^−C∥2/∥C∥2e=\Vert\widehat C-C\Vert_2/\Vert C\Vert_2 (spectral norm). Here CC and C^\widehat C are the normalized reference and CoPEN couplings, measured along base-update directions. For a curvature matrix XX, the normalized off-diagonal blocks are Xii−1/2XijXjj−1/2X_{ii}^{-1/2}X_{ij}X_{jj}^{-1/2}, with diagonal blocks set to zero. Removing all coupling gives e=1e=1, or 0% reduction. Exact agreement gives 100%. Higher is better.
DiagnosticViTMoE
Cross-group curvature alignment0.970.99
Coupling error reduction (%) ↑8187
Maximum contraction-factor ratio ε/ρ\varepsilon/\rho0.5080.422

To address the reviewer’s question about whether practical CoPEN satisfies Theorem 1’s approximation requirement, we assess CoPEN’s estimated coupling using the measured curvature. Following Theorem 1, we compare the coupling-aware contraction factor ε\varepsilon with the optimal block-diagonal factor ρ\rho to isolate the effect of cross-group coupling. The maximum ratios are 0.508 for ViT and 0.422 for MoE. Since both maxima are below 1, ε<ρ\varepsilon<\rho holds in every measurement where Theorem 1 applies, supporting the theorem’s local contraction comparison in these diagnostics.

The alignments support the preservation of Hessian-coupling patterns. At the evaluated checkpoints, CoPEN reduces normalized coupling discrepancy by 81% in ViT and 87% in MoE relative to omitting coupling. Coupling results are averaged over three seeds. The MoE measurements use the same grouping as the Block-diag. comparison. Figure 1 separately provides empirical evidence that the practical correction escapes earlier than the matched Block-diag. update.

2. Which cross-group interactions matter for optimization?

In MoE, expert and router parameters form separate groups within each routed layer, and task heads form another group. Off-diagonal blocks of KtK_t capture expert–router, expert–head, router–head, and cross-layer coupling.

A small MoE retains the full model's learned expert–router structure, with eight experts and two routers as separate groups. For each pair, we hold other parameters fixed and compare update directions computed with and without its cross-group curvature. The table reports 100(1−Δblock/Δjoint)100(1-\Delta_{\mathrm{block}}/\Delta_{\mathrm{joint}}), where each Δ\Delta is the best loss decrease along that direction predicted by the same damped local quadratic model (Appendix D, Table 7):

Group pairReduction in predicted loss decrease
Same layer, expert–router59%
Same layer, expert–expert54%
Across layers, expert–router24%
Across layers, expert–expert18%

Values are averaged over six seeds. The largest reductions in predicted decrease occur when within-layer expert–router and expert–expert coupling is removed, supporting the importance of these interactions in preconditioning. In this small MoE, retaining coupling also reduces mean test loss from 6.897 to 5.650 (×10−3\times10^{-3}) versus Block-diag. (Appendix D, Table 4).

Thank you for the additional experiments and detailed rebuttal, which substantially address my earlier concerns. My remaining concern is that the newly added OLMoE-1B-7B experiments report only validation losses across five task categories, without corresponding downstream performance metrics. Could the authors provide task-specific evaluation results to demonstrate that the improvements in validation loss translate into meaningful downstream performance gains? If these results confirm the effectiveness of CoPEN, I would be willing to raise my rating to 3 (Weak Acceptance).

Reviewer_t5VZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how cross-group curvature can create poorly conditioned or unstable joint optimization directions even when each parameter group is well conditioned in isolation. It analyzes local convergence near minima and unstable-mode amplification near coupled saddles. The practical method, CoPEN, combines SOAP-style tensor-wise updates with a low-dimensional correction based on a projected positive-semidefinite output-curvature metric. Controlled experiments compare retained versus removed cross-group terms, and Taskonomy experiments with ViT-B/16 and V-MoE-B/16 report improved validation losses and fewer updates to target losses. The additional per-step cost is also reported, with lower training time to the evaluated targets than AdamW.

Critical strengths
  1. The paper isolates a useful optimization issue beyond task-gradient conflict. The normalized coupling matrix makes clear why accounting for curvature within each group can still leave difficult joint modes. The local analyses explain both slow convergence and the existence of saddles with positive-definite diagonal blocks, under explicitly stated assumptions.

  2. The experiments connect controlled comparisons with model-scale outcomes and cost. The matched Block-diag. controls in the toys help isolate the contribution of cross-group terms. The Taskonomy evaluation covers dense and routed backbones, uses a common hyperparameter-search budget, and reports validation loss, per-task results, per-step cost, memory, and time to target. The reported benefit is therefore supported by more than iteration counts alone.

Critical weaknesses
  1. The saddle analysis does not directly characterize the practical correction. Theorem 2 concerns a preconditioner built from the true Hessian's normalized cross-group coupling. In contrast, Eqs. (15)–(16) use G=JTWJG = J^T WJ with positive-semidefinite WW, projected to K=QTGQK = Q^T GQ. Appendix B.4 correctly states that this metric represents joint output sensitivity rather than detecting negative Hessian curvature. However, the main text still presents Figure 1's earlier escape as closely supporting the theorem and the practical method together. Appendix D.1.2 specifies the toy loss but leaves the precise preconditioner used there insufficiently explicit. The theoretical result can motivate the method, but the current account does not establish which part of that mechanism the implemented update realizes.

  2. The model-scale evidence does not isolate cross-group terms as directly as the toy controls do. Comparisons against SOAP support the usefulness of the complete correction, but the correction also changes scaling within the selected subspace. A model-scale control retaining the same bases, diagonal projected blocks, damping, and refresh schedule while removing only cross-group entries would distinguish these effects. This also depends on the selected grouping: Appendix C and Table 8 list layer-wise and separate expert/router variants, but do not identify the grouping used for the main MoE row. When experts and routers share a structural group, their interactions are within a projected diagonal block, so the claimed cross-group mechanism must be interpreted at that grouping level.

Questions for authors
  1. What exact preconditioner and update produce Figure 1: the Hessian-based construction from Theorem 2, the practical rule in Eq. (16), or another construction? Please state what the observed escape establishes about the practical method, given the distinction acknowledged in Appendix B.4.

  2. Which MoE grouping is used in Tables 1–2, and how does it represent the expert–router interactions emphasized in the motivation? Could you include a focused model-scale Block-diag. control that removes only off-diagonal projected group blocks while retaining the remaining correction settings?

3 comments
AuthorsOpenReview ↗

Response to Reviewer t5VZ (1/2)

We thank the reviewer for the insightful comments and questions. We will revise the paper to address all the points raised.

W1. How does the practical PSD correction support the saddle-escape mechanism in Theorem 2? We agree that the link between Theorem 2’s saddle-escape mechanism and the practical correction needs clearer explanation.

CoPEN does not explicitly encode negative curvature in its PSD metric, but Theorem 2 also uses a positive-definite preconditioner. Its normalized cross-group block κC/ρ\kappa C/\rho has norm κ<1\kappa<1. Negative curvature enters the dynamics through the loss gradient, which drives an existing perturbation along the unstable direction. The preconditioner MκM_\kappa gives this direction relatively greater weight while controlling updates along stable directions, thereby accelerating escape.

Motivated by this mechanism, CoPEN uses the projected PSD curvature K=Q⊤J⊤WJQK=Q^\top J^\top WJQ to avoid the cost of full-Hessian computation. Its damped inverse in Eq. (16) attenuates directions with larger eigenvalues of KK more strongly. When these align with stable Hessian modes and unstable modes lie in lower-curvature directions of KK within the represented subspace, the correction can suppress stable components more than unstable ones and support faster escape.

We provide two sets of empirical evidence below for this theory–practice connection: how accurately CoPEN approximates coupling and how retaining that coupling benefits optimization.

1. How accurately does CoPEN approximate coupling under the chosen grouping?

For both metrics, the reference curvature is measured by finite differences along CoPEN's parameter directions.

  • Cross-group curvature alignment: cosine similarity between the reference cross-group curvature and CoPEN's estimate. Closer to 1 means more similar coupling patterns.
  • Coupling error reduction (%): the reduction in normalized coupling discrepancy relative to omitting coupling, measured as 100(1−e)100(1-e) with e=∥C^−C∥2/∥C∥2e=\Vert\widehat C-C\Vert_2/\Vert C\Vert_2 (spectral norm). Here CC and C^\widehat C are the normalized reference and CoPEN couplings, measured along base-update directions. For a curvature matrix XX, the normalized off-diagonal blocks are Xii−1/2XijXjj−1/2X_{ii}^{-1/2}X_{ij}X_{jj}^{-1/2}, with diagonal blocks set to zero. Removing all coupling gives e=1e=1, or 0% reduction. Exact agreement gives 100%. Higher is better.
DiagnosticViTMoE
Cross-group curvature alignment0.970.99
Coupling error reduction (%) ↑8187

The alignments support the preservation of Hessian-coupling patterns. At the evaluated checkpoints, CoPEN reduces normalized coupling discrepancy by 81% in ViT and 87% in MoE relative to omitting coupling. Coupling results are averaged over three seeds. The MoE measurements use the same grouping as the Block-diag. comparison.

2. Which cross-group interactions matter for optimization? A small MoE with learned experts and routers retains the full model's expert–router structure. For expert–router and expert–expert pairs, removing cross-group curvature reduces the loss decrease predicted by a local quadratic model (Table 7).

W1, Q1. Which preconditioner and update are used in Figure 1? Figure 1 applies the same CoPEN preconditioning construction used in the ViT and MoE experiments:

\delta=b+\gamma Q\bigl[\lambda_c(K+\lambda_c I)^{-1}-I\bigr]Q^\top b.$$ In this two-coordinate example, each coordinate forms a group ($Q=I$). With $b=-\nabla L$ and $\gamma=1$, Eq. (16) gives the preconditioned direction $-(K+\lambda_c I)^{-1}\nabla L$. Block-diag. removes only the off-diagonal entries of $K$ and uses the same damping. Figure 1 separately provides empirical evidence that the practical correction escapes earlier than the matched Block-diag. update. We will specify this setup in Appendix D.1.2, the main text, and the caption.
AuthorsOpenReview ↗

Response to Reviewer t5VZ (2/2)

W2, Q2. Does a matched Block-diag. ablation on ViT and MoE isolate the gains from cross-group terms? Yes. The control sets only the off-diagonal blocks of KtK_t to zero. It retains the same basis-construction and diagonal-block estimation rules, damping, mixing weight, refresh schedule, and training settings.

Validation losses are averaged over three seeds. Updates and training time are reported through each method's final validation checkpoint.

MetricViT CoPENViT Block-diag.MoE CoPENMoE Block-diag.
Validation loss ↓0.143060.145310.143440.14401
Updates ↓16,89621,50412,28816,896
Training time (h) ↓1.531.952.293.15

At their final validation checkpoints, CoPEN has lower losses on both backbones with 21–27% less training time. The matched control isolates the contribution of cross-group terms while retaining the same within-group correction.

W2, Q2. Which MoE grouping is used in Tables 1–2, and how does it represent expert–router interactions? In MoE, expert and router parameters form separate groups within each routed layer, and task heads form another group. Off-diagonal blocks of KtK_t capture expert–router, expert–head, router–head, and cross-layer coupling.

A small MoE retains the full model's learned expert–router structure, with eight experts and two routers as separate groups. For each pair, we hold other parameters fixed and compare update directions computed with and without its cross-group curvature. The table reports 100(1−Δblock/Δjoint)100(1-\Delta_{\mathrm{block}}/\Delta_{\mathrm{joint}}), where each Δ\Delta is the best loss decrease along that direction predicted by the same damped local quadratic model (Appendix D, Table 7):

Group pairReduction in predicted loss decrease
Same layer, expert–router59%
Same layer, expert–expert54%
Across layers, expert–router24%
Across layers, expert–expert18%

Values are averaged over six seeds. The largest reductions in predicted decrease occur when within-layer expert–router and expert–expert coupling is removed, supporting the importance of these interactions in preconditioning. In this small MoE, retaining coupling also reduces mean test loss from 6.897 to 5.650 (×10−3\times10^{-3}) versus Block-diag. (Appendix D, Table 4).

Now that we have posted our responses and uploaded the revised manuscript, we would be grateful if you could take a look at them when convenient. We would appreciate your feedback on whether our response addresses your concerns and would be happy to clarify any remaining points. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.