acceptodds
Under review as a conference paper at ICLR 2027

Rotation-Based Gradient Balancing: Reconciling Alignment and Fidelity in Multi-Task Learning

Abstract

Multi-task learning (MTL) enables knowledge sharing across tasks but often suffers from gradient conflicts. Gradient manipulation methods reduce these conflicts by directly altering task gradient angles to eliminate opposing directions. However, over-prioritizing geometric alignment may introduce substantial deviations from the original gradient directions, diluting valuable task-specific information and degrading performance. To overcome these limitations, we propose the Rotation-Based Gradient Balancing (RGB) method. RGB enables task gradients to be finely adjusted toward a consensus direction, holistically reducing global conflict while penalizing excessive deviation from the original task-gradient directions. Our method consistently achieves competitive results across various MTL benchmarks ranging from 3 to 40 tasks. Empirically, ablation studies show that simply increasing the degree of rotation does not consistently improve task performance, reinforcing the importance of controlling gradient modification when balancing alignment and fidelity.

Primary area:
optimization
Keywords:
Multi-task Learning, Optimization, Deep Learning, Gradient Conflicts, Pareto Optimality

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_63xuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper focuses on resolving gradient conflicts in multi-task learning (MTL) with shared parameters. In this setting, task gradients can point in conflicting directions, such that an update that benefits one task may hurt another.

The paper proposes Rotation-Based Gradient Balancing (RGB), which modifies task gradients before combining them into a shared-model update. Existing approaches that modify task gradients to improve alignment may overcorrect, potentially destroying useful task-specific information. RGB attempts to balance these competing objectives.

Rather than merely reweighting task gradients or projecting out conflicting components, RGB rotates each normalized gradient toward a common reference direction. The rotation angles are task-specific and jointly optimized using an alignment objective with a penalty for deviating from the original gradients. The experiments evaluate RGB on four benchmarks with 3–40 tasks and compare it with existing approaches.

Critical strengths
  • The paper makes a useful distinction between reducing gradient conflict and improving multi-task optimization.

  • The fidelity penalty is important because it discourages the optimization from pursuing gradient alignment without accounting for deviations from the original task gradients.

  • Extensive theoretical analysis is appreciated.

Critical weaknesses
  • Gradient manipulation is already explored by existing approaches. The paper could better establish the methodological contribution of its rotation-based formulation and alignment–fidelity objective relative to prior methods.

  • Except for CelebA, the benchmarks contain only 3–11 tasks. Moreover, CelebA uses the same input images for its 40 binary-attribute tasks. How well does RGB scale to substantially larger and more heterogeneous task sets? What is the computational cost of the gradient operations and rotation optimization as the number of tasks increases?

  • RGB normalizes task gradients before combining them, which removes their relative magnitude information. How does it behave when tasks have substantially different loss scales, convergence rates, or degrees of difficulty? How does it compare with methods that explicitly account for task magnitudes or loss scales?

  • Existing approaches, including the proposed method, generally assume that all tasks are learned together in a single MTL model. However, this may not scale well to large task sets, and prior studies have shown that grouping related tasks can improve MTL performance. Does the paper compare RGB with task-grouping approaches, particularly recent methods that explicitly identify groups of tasks, rather than applying gradient modification to all tasks jointly?

Questions for authors

See weakness.

3 comments

We thank the reviewer for the constructive feedback and comments. We address your concerns and questions in the following.

Weaknesses:

1. Methodological contribution relative to prior approaches.

Thank you for the suggestion. We agree that gradient manipulation is an established approach to multi-task learning. RGB’s core methodological contribution lies in its task-specific rotation parameterization and the joint optimization of these angles through an alignment–fidelity objective.

To make the distinction from prior gradient manipulation methods easier to identify, we have refined the synthesis following the comparison with prior methods in the Introduction, making RGB’s joint control of rotation magnitude and fidelity explicit. Our angular sweeps and inner-budget ablations provide empirical support for this design: blindly increasing modification does not consistently improve performance, and the preferred amount varies across datasets.

2. Scaling to larger and more heterogeneous task sets.

Our evaluation focuses on optimization within a shared multi-task model across benchmarks containing 3–40 tasks. CelebA evaluates RGB with 40 jointly learned attributes, while the other benchmarks cover different task types and application domains. We agree that CelebA does not by itself establish effectiveness on substantially larger, heterogeneous task collections. Evaluating such collections is beyond the experimental scope of this paper; our empirical claims concern the benchmark settings evaluated here.

To address computational scalability as the number of tasks increases, we have included a compact runtime comparison in the main text (Section 4.3) using the configurations associated with the reported performance results. Let TT denote the number of tasks, DD the shared-parameter dimension, and αsteps\alpha_{\text{steps}} the inner-optimization budget. RGB computes task-specific gradients through task-wise backward passes, constructs the Gram matrices once per rotation optimization at a cost of O(T2D)\mathcal{O}(T^2D), and performs αsteps\alpha_{\text{steps}} kernelized inner iterations at a total cost of O(αstepsT2)\mathcal{O}(\alpha_{\text{steps}}T^2). Thus, excluding task-gradient computation, the rotation procedure costs O(T2D+αstepsT2).\mathcal{O}(T^2D+\alpha_{\text{steps}}T^2). The pairwise computations scale quadratically with task count, whereas the inner-optimization cost grows linearly with αsteps\alpha_{\text{steps}}. Increasing the inner budget does not require additional network backward passes or repeated Gram-matrix construction. Kernelization therefore removes repeated parameter-dimensional operations from the inner loop, while retaining the task-count dependence of pairwise alignment. These measurements and complexity estimates characterize computational scaling on the evaluated benchmarks. Effectiveness on substantially larger, heterogeneous task collections remains future work.

3. Gradient normalization and task-scale sensitivity.

RGB normalizes task gradients to prevent larger gradient magnitudes from dominating the shared directional update. Such differences can arise from loss units and scaling, as well as task-specific optimization dynamics. Normalization ensures the rotation objective operates on task-gradient directions, allowing RGB to optimize alignment and fidelity without directly weighting tasks by their original gradient norms.

To evaluate sensitivity to loss scaling, we conducted controlled Cityscapes experiments in which the training loss of depth estimation was multiplied by s∈0.1,10s\in\\{0.1,10\\}, while the other task losses remained unchanged. We kept each method’s configuration fixed across scaling factors and evaluated using the original task metrics and STL references. We compare RGB with UW and GradNorm to assess its behavior alongside alternative task-balancing mechanisms that explicitly account for task magnitudes and loss scales. The table below reports results averaged over three seeds.

MethodDepth-loss multiplier ssmIoU [%] ↑L1 [px] ↓MSE ↓ΔM\Delta M\\% ↓
UW0.157.1410.100.4010.47
UW1057.5510.130.355.23
GradNorm0.152.1210.170.4417.36
GradNorm1054.4010.070.356.71
RGB (αsteps=30\alpha_{steps}=30)0.166.0210.030.32−1.95
RGB (αsteps=30\alpha_{steps}=30)1066.0510.030.32−1.96

RGB achieves the best per-task metrics and ΔM\Delta M\\% among these methods at both scaling factors. Its ΔM\Delta M\\% remains consistent across the two settings, while UW and GradNorm compared with UW and GradNorm exhibit substantial gaps between the two multipliers. These results support RGB’s robustness to the tested depth-loss rescaling. They assess sensitivity to numerical loss scales, rather than explicit adaptation to task difficulty or relative convergence rates.

Although RGB normalizes the individual rotated directions, their aggregate is not constrained to unit norm: rotation changes both the shared direction and its magnitude through directional agreement and cancellation. This differs from adjusting task weights using gradient magnitudes or loss information. The aggregate magnitude reflects agreement among the rotated directions; it does not restore the original task-gradient magnitudes.

The outer optimizer (Adam in our experiments) subsequently converts this aggregate into a parameter update, incorporating first- and second-moment estimates to adapt the step coordinate-wise. RGB’s inner optimization controls task-direction geometry, while the outer optimizer determines the resulting parameter update.

4. Comparison with task-grouping approaches.

Thank you for highlighting this direction. Our main tables already include an empirical comparison with GO4Align [1] on NYUv2 and QM9. To make this comparison more explicit, we have expanded the Introduction’s discussion to distinguish two uses of task grouping: selecting task subsets to train in separate models, and organizing task updates within a single shared model.

The latter is particularly relevant to RGB’s optimization setting. GO4Align dynamically groups tasks and uses risk-guided group indicators to coordinate optimization. Selective Task Group Updates [2] also operate within a shared model, using task affinities to form groups and update them sequentially. RGB instead jointly optimizes task-specific rotation angles through an alignment–fidelity objective, without explicitly partitioning the tasks.

We have now formally discussed both methods in the revised Introduction, while distinguishing methodological comparison from empirical evaluation. Our quantitative comparison includes GO4Align. We have not evaluated Selective Task Group Updates under our protocols; its published NYUD-v2 results use a different task configuration and architecture, so they cannot be directly incorporated into our performance tables.

We also agree that larger benchmarks would be valuable for investigating how joint learning behaves as task diversity and task count increase. At substantially larger scales, selecting which tasks share parameters becomes increasingly relevant. RGB could potentially operate within selected task groups, combining task-group selection with alignment–fidelity control during optimization. Evaluating this combination is a useful direction for future work.


Reference:
[1] GO4Align: Group Optimization for Multi-Task Alignment
[2] Selective Task Group Updates for Multi-Task Optimization

Reviewer_63xuOpenReview ↗

Response by Reviewer 63xu

Thanks to the authors for addressing my concerns and providing additional results. I will keep my score unchanged.

Reviewer_6BmPOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The submission introduces Rotation-Based Gradient Balancing (RGB), a gradient manipulation optimization strategy tailored for multi-task learning settings.

Critical strengths
  • The paper presents a clear research motivation, effectively emphasizing the core concepts of gradient alignment and fidelity.
  • The empirical evaluation demonstrates state-of-the-art results across all evaluated benchmark datasets.
Critical weaknesses
  • A runtime and computational efficiency comparison should be incorporated into the main text. Although gradient-driven multi-task learning methods often boost performance, they frequently incur significant training time overheads as task counts grow. A full efficiency analysis in the primary paper is necessary.

  • Relying on overall multi-task performance to identify optimal parameters (especially alpha steps) risks compromising test set generalization. Evaluation results on the test set are currently absent and should be provided.

Questions for authors

please check the weaknesses.

1 comment

We thank the reviewer for the constructive feedback and comments. We agree that computational efficiency and the evaluation protocol should be presented more clearly, and we have updated the manuscript accordingly.

Weaknesses:

1. Runtime and computational efficiency.

Our current efficiency analysis appears in Appendix R. We have added a compact comparison to the main text in Section 4.3, reporting training time per epoch and relative overhead for RGB and the compared gradient-based MTL methods across the evaluated benchmarks.

For RGB, we measured runtime using the dataset-specific inner-iteration budgets employed in Tables 1–2. This reports the computational cost associated with the reported accuracy results. Within each benchmark, timing comparisons will use the same hardware, architecture, batch size, and measurement procedure. Below is a runtime comparison of RGB and other gradient-based MTL methods across four benchmarks, using LS (equal weighting) as the reference baseline:

MethodNYUv2: s/epoch (×LS)Cityscapes: s/epoch (×LS)QM9: s/epoch (×LS)CelebA: s/epoch (×LS)
LS78 (1.00×)222 (1.00×)108 (1.00×)156 (1.00×)
MGDA336 (4.31×)246 (1.11×)384 (3.56×)1212 (7.77×)
PCGrad354 (4.54×)258 (1.16×)372 (3.44×)1794 (11.50×)
GradDrop132 (1.69×)252 (1.14×)258 (2.39×)1002 (6.42×)
CAGrad372 (4.77×)246 (1.11×)318 (2.94×)1014 (6.50×)
IMTL-G138 (1.77×)246 (1.11×)324 (3.00×)840 (5.38×)
Nash-MTL378 (4.85×)282 (1.27×)618 (5.72×)1710 (10.96×)
RGB420 (5.38×)288 (1.30×)270 (2.50×)936 (6.00×)

The main-text comparison will make the performance–runtime trade-off directly visible, with Appendix R providing the supporting implementation and memory details.

2. Evaluation protocol and hyperparameter selection.

Thank you for raising this critical point regarding test-set generalization. We realize our wording in Appendix J caused confusion, and we wish to clarify. The results reported in Tables 1 and 2 are the final evaluation results, not training-set performance. We have revised Appendix J to eliminate this ambiguity.

Our experiments strictly follow the standard evaluation protocols of the official benchmark codebases. For QM9 and CelebA, the optimal model checkpoint is saved based purely on validation set performance, and the final ΔM\Delta M\\% reported in our main tables is evaluated on the completely held-out test set. For NYUv2 and Cityscapes, which do not provide a separate test set in the standard MTL literature, we follow the exact protocol of all prior baselines by selecting and reporting on the official evaluation splits. We have clarified the evaluation and model-selection procedures in the paper.

To address the concern that per-dataset tuning of the αsteps\alpha_{\text{steps}} hyperparameter might compromise generalization, we evaluated RGB under a fixed budget of αsteps=5\alpha_{\text{steps}}=5 across all four datasets (requiring zero hyperparameter sweeps).

DatasetTuned αsteps\alpha_{\text{steps}}Tuned ΔM\Delta M\\% ↓Tuned MR ↓Fixed αsteps\alpha_{\text{steps}}Fixed ΔM\Delta M\\% ↓Fixed MR ↓
NYUv210−8.612.115-7.622.89
Cityscapes30-1.703.005-0.604.67
QM9412.41.64521.01.91
CelebA100-1.493.2351.247.78

Even when locking the budget to a strict default of 5 steps without any per-dataset tuning, RGB continues to achieve the best aggregate performance (lowest ΔM\Delta M\\% and Mean Rank) on NYUv2, Cityscapes, and QM9 among all evaluated baselines.

While CelebA requires a larger budget due to its massive task count diluting the gradient updates, RGB’s strong aggregate performance on the other three benchmarks is fundamentally robust out-of-the-box. We have added Table 3 (Appendix J) to the revised manuscript, which explicitly separates the tuned and fixed-budget results so readers can accurately assess the performance trade-offs.

Reviewer_mrS1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that gradient-manipulation MTL methods (PCGrad-style projections, GradVac, RotoGrad) over-emphasize geometric alignment and can erase task-specific directional information. It introduces a scalar cost–benefit view of rotation magnitude, the Multi-Task Equilibrium Relationship (MER; Eqs. 2–4, Fig. 1), to motivate controlled modification rather than aggressive deconfliction.

RGB (Sec. 3, Alg. 1) (i) normalizes task gradients, (ii) maintains an EMA consensus d_t, (iii) rotates each bar g_i in the plane of bar g_i and an orthogonal correction w_i aligned with d_t, and (iv) chooses task-specific angles α∈[0,π/2]^T by minimizing a conflict-plus-fidelity objective (Eq. 5). A kernelized T×T form (App. H) avoids materializing T×D rotated gradients in the inner loop. Under Lipschitz smoothness, Robbins–Monro stepsizes, and a quantitative common-descent assumption (Assumption E.1), the outer update's Pareto-stationarity residual ρ(θ_t) vanishes (Theorem E.2).

Experiments: NYUv2 (Table 1), Cityscapes / CelebA / QM9 (Table 2) via LibMTL, Align-MTL, and FAMO/Nash-MTL pipelines. RGB reports competitive or best Δm% and mean rank for T from 3 to 40. Ablations (Apps. J–O) show U-shaped sensitivity to α_steps and that more rotation is not uniformly better. Code is promised upon publication.

Critical strengths
  1. Competitive, protocol-aligned results across 3–40 tasks. NYUv2 (Table 1): Δm%=−8.61, MR =2.11, ahead of ConsMTL (−6.72) and PIVRG (−6.50), with large gains on segmentation/depth. Table 2: best reported Δm%/MR on Cityscapes, CelebA (40 tasks), and QM9 under the authors' aligned codebases. With kernelized efficiency (App. R) and stderr reporting (App. P), this is the main reason the paper clears the acceptance bar for me.

  2. Alignment–fidelity control is empirically supported. Simply increasing rotation / inner budget does not help monotonically (App. I; Figs. 6–8; App. J), and the limitations section states this honestly (Sec. 5). That framing matters more than another pure projection heuristic and motivates the joint conflict+fidelity objective (Eq. 5).

Critical weaknesses
  1. Theory packaging overshoots what is proved and optimized. MER is a motivational scalar model with arbitrary functional forms; it does not derive Eq. 5's weights, the EMA construction, or chosen α_steps. Convergence (Theorem E.2) hinges on Assumption E.1, which is not established for the finite-inner-loop, EMA-referenced practical update. Introduction contribution bullet 2 risks reading as stronger theory than the appendices deliver. It is unclear that MER + Pareto claims are necessary for, or tightly coupled to, the empirical gains.

  2. Heavy per-dataset α_steps tuning. The key knob is a tuned operating point (4 / 10 / 30 / 100 across QM9 / NYUv2 / Cityscapes / CelebA). Sensitivity is acknowledged, but main tables still present RGB as consistently superior without showing that competing gradient manipulators got an analogous inner budget, or that a single default works. For a method whose thesis is controlling modification magnitude, dataset-specific inner budgets are a first-order limitation for significance and reproducibility until automatic stopping or a robust default is shown.

Questions for authors
  1. MER → Eq. 5 (assessment-changing). Can you derive or ablate a link from MER's marginal balance to Eq. 5's conflict/fidelity coefficients? If MER is only motivational, would you demote it from a primary "contribution" and center claims on Eq. 5 + the empirical trade-off?

  2. Assumption E.1 in practice. Do you monitor min_i⟨g_i,v⟩ / ρ(θ)^2 during training? On which benchmarks/iterations does Assumption E.1 hold, and what happens to ρ(θ_t) when the finite-α_steps solution violates the descent certificate in Theorem D.2?

  3. Kernel vs Original (Table 12). If the kernel method is mathematically equivalent, why does NYUv2 Δm% differ by ~2.7 points at α_steps=50? Please name the exact numerical differences (stabilizer e, post-rotation renormalization, ∇_α L, etc.).

  4. Fairness of α_steps tuning. If each baseline got a comparable search over an analogous inner budget, or if RGB used one fixed α_steps across all four datasets, how do Tables 1–2 rankings change? A single default or validation-selected-but-frozen setting would change my confidence in the "consistently competitive" claim.

4 comments

We thank the reviewer for the constructive feedback and comments. We address your concerns and questions in the following.

Weaknesses:

1. Theory packaging overshoots what is proved and optimized.

We agree that the paper should distinguish more clearly between the motivating model, the implemented alignment–fidelity objective, and the assumptions behind the convergence analysis. MER is a conceptual account of two competing effects, which are gradient alignment and deviation from the original task directions. It is not a fitted model. We did not use it to derive the coefficients in Eq. 5 or the inner-iteration budget. ​We have explicitly stated this in the revised manuscript and updated contribution bullet 2 to center on the empirical alignment–fidelity trade-off, framing MER strictly as the motivating framework. We have also clarified that Theorem E.2 establishes convergence of the deterministic update under its assumptions, rather than guaranteeing convergence of every practical RGB training trajectory. Q1 and Q2 below take up the link between the objective and the model, and report diagnostics that check whether the descent condition holds in practice.

2. Heavy per-dataset α_steps tuning.

We acknowledge that to achieve the best performance of RGB currently requires some tuning of αsteps\alpha_{\text{steps}}. The values used in Tables 1–2 (4, 10, 30 and 100 for QM9, NYUv2, Cityscapes and CelebA) are dataset-specific operating points. Although Sec. 5 identifies this sensitivity as a limitation, we agree that its implications should also be clear alongside the main results. In the revised manuscript, we have explicitly separated the results at tuned budget from those evaluated under one shared setting. Specifically, we have added Table 3 in Appendix J, which directly compares the dataset-specific configurations against a single fixed budget (αsteps=5\alpha_{\text{steps}}=5) across all four datasets. This ensures the impact of tuning is immediately visible next to the headline numbers. As discussed in Sec. 5, developing stopping criteria that determine the inner optimization budget is an important direction for future work. We further address this robustness concern and discuss these fixed-budget results below in Q4.

Questions:

1. MER → Eq. 5 (assessment-changing).

MER expresses the principle that additional alignment should be balanced against deviation from task-specific gradient directions. As we have clarified in Section 2.2 of the revised manuscript, the logarithmic benefit and exponential cost are an analytical illustration of diminishing marginal benefits and increasing marginal costs. They are not empirical fits, and their parameters are illustrative. We did not use them to derive the coefficients in Eq. 5 or inner-iteration budget.

Eq. 5 implements this principle through a geometric objective that jointly rewards agreement among transformed gradients and penalizes deviation from their original directions. However, the scalar marginal-balance condition does not uniquely determine these coefficients. We therefore do not claim that optimizing Eq. 5 exactly solves the scalar MER model.

The evidence for the trade-off itself is experimental. In the angular sweeps and inner-budget ablations, greater gradient modification does not consistently improve performance, and the preferred amount varies across datasets. Appendix L provides complementary evidence by examining the association between performance and distance to an empirical MER reference.

To prevent overclaiming, we updated the paper to present MER strictly as an illustrative, motivating framework, rather than a standalone theoretical contribution. We have revised the primary contribution statement in the Introduction to center squarely on the empirical alignment–fidelity trade-off and the geometric objective, which is precisely what the experiments support.

2. Assumption E.1 in practice.

We appreciate your insightful suggestion to check whether the descent condition holds in practice. Based on your feedback, we conducted the requested diagnostic experiments and have added this analysis as Appendix S in the revised manuscript. We have instrumented the finite-inner-loop RGB implementation to monitor ht=min⁡i⟨gi,vt⟩ρt2,ρt2=min⁡pi≥0∑ipi=1∥∑ipigi∥2.h_t = \frac{\min_i \langle g_i, v_t \rangle}{\rho_t^2}, \qquad \rho_t^2 = \min_{\substack{p_i \ge 0 \\\\ \sum_i p_i = 1}} \left\lVert \sum_i p_i g_i \right\rVert^2.

Both use the raw minibatch gradients of the shared parameters and the RGB direction vtv_t, taken before any outer-optimizer transformation.

Here ρt\rho_t is the minimum norm of a convex combination of task gradients, and ρt2\rho_t^2 is its square. A zero residual means Pareto stationarity for the measured gradients. The ratio hth_t compares the weakest task’s alignment with the squared residual. If ht>0h_t>0, −vt-v_t is a strict common descent direction for the measured task gradients. If ht<0h_t<0, at least one task is adversely aligned with the proposed update.

For Theorem D.2, we also record the minimum geometric margin mt=min⁡i{⟨ri,vt⟩−∥ri−gˉi∥∥vt∥}.m_t = \min_i \left\lbrace \langle r_i, v_t \rangle - \lVert r_i - \bar{g}_i \rVert \lVert v_t \rVert \right\rbrace.

It tells us whether the theorem’s alignment–fidelity bound certifies descent for every task. If mt>0m_t>0, strict common descent is guaranteed. If mt≤0m_t\le0, the certificate says nothing, and descent may still hold, so we check the original task inner products separately.

We ran one run per benchmark at the dataset-specific inner budgets below. Sampling starts at iteration 1, then every 100 iterations for Cityscapes, NYUv2 and CelebA, and every 1000 for QM9. Every recorded residual estimate passed the numerical optimality check.

DiagnosticCityscapesNYUv2QM9CelebA
αsteps\alpha_{\text{steps}}30104100
Last sampled iteration22200794002770009500
Number of samples22379527896
Sampled common-descent condition satisfied223/223 (100%)642/795 (80.75%)64/278 (23.02%)1/96 (1.04%)
Theorem D.2 certificate satisfied223/223 (100%)628/795 (78.99%)64/278 (23.02%)1/96 (1.04%)
Median hth_t2.2261.127−76.720−80.059
ρt2\rho_t^2: first sample1.2186240.3835316.75121×10−36.75121\times10^{-3}0.272449
ρt2\rho_t^2: last sample0.0136240.0224402.99974×10−42.99974\times10^{-4}6.50467×10−66.50467\times10^{-6}

Cityscapes is the clean case. Across all recorded iterations hth_t stays between 0.548 and 4.332, so the sampled inequality holds with κ=0.54\kappa=0.54 throughout, and the minimum geometric margin is 0.172393. NYUv2 shows why the certificate and the descent condition differ. In 14 of its samples the certificate fails while common descent still holds.

QM9 (11 tasks) and CelebA (40 tasks) pass far less often than the three-task benchmarks. That is plausible, since demanding positive alignment for every task at once gets harder as tasks are added. But these runs differ in task geometry, data and optimization settings too, so we cannot attribute the drop to task count alone.

Despite these violations, the residual exhibits an overall decreasing trend. The table gives the median ρt2\rho_t^2 over three consecutive, roughly equal groups of recorded samples.

BenchmarkEarly thirdMiddle thirdLate third
Cityscapes0.0815860.0299040.015261
NYUv20.1273390.0664530.038713
QM90.0014200.000072800.00003933
CelebA0.0076830.0039620.00005769

To see what happens after a certificate violation, we compared each violating sample with the next recorded one.

BenchmarkViolations with a subsequent sampleSubsequent ρt2\rho_t^2 decreaseSubsequent ρt2\rho_t^2 increase
NYUv216771 (42.51%)96 (57.49%)
QM9214105 (49.07%)109 (50.93%)
CelebA9450 (53.19%)44 (46.81%)

Certificate violations are followed by both increases and decreases in the sampled residual. So violations do not stop the residual from shrinking overall. These comparisons span several updates and different minibatches, so they say little about any immediate causal effect.

We have incorporated these findings into the revised manuscript. Specifically, we have added plots of hth_t, ρt\rho_t, and average task loss against sampled training iterations to the Appendix S (Figures 16 & 17), with hth_t shown separately and a zero reference line marking the common-descent boundary. These plots distinguish task-wise descent behavior, stationarity-residual evolution, and overall learning progress.

We have also explicitly stated in the text that Theorem E.2 is conditional on Assumption E.1, and that finite inner optimization does not guarantee E.1. The diagnostics describe what the practical update does. They do not replace the assumptions of the deterministic theorem.

3. Kernel vs Original (Table 12).

Thank you for catching this. We apologize for the unclear presentation. The common budget of αsteps=50\alpha_{\text{steps}}=50 in the efficiency paragraph applied only to the runtime and memory measurements. It did not apply to the ΔM\Delta M\\% results. On checking, the original RGB ΔM\Delta M\\% on NYUv2 mistakenly reported αsteps=5\alpha_{\text{steps}}=5, while kernel RGB used αsteps=10\alpha_{\text{steps}}=10. The two were compared at different inner budgets, so the ΔM\Delta M\\% gap cannot be attributed to kernelization alone.

To provide an exact one-to-one comparison, we have rerun the original RGB on NYUv2 using the matched budget of αsteps=10\alpha_{\text{steps}}=10 over three seeds. As shown in our updated manuscript (now is Table 13), Original RGB achieves ΔM\Delta M\\%=-7.13, compared with −8.61-8.61 for Kernel RGB under the same inner-iteration budget.

To ensure no further misunderstanding, we have completely revised this section in Appendix Q. Table 13 now reports performance (ΔM\Delta M\\%), training time, and memory usage strictly using the exact same matched inner-rotation budget for both implementations within each dataset. The reported computational cost now directly corresponds to the evaluated performance configuration.

From the NYUv2 training run, we saved 10 snapshots, at iterations 1, 100 and 1000, then every 10000 from 10000 to 70000. Each snapshot stores the original task gradients, the normalized gradients, the rotation bases, the EMA direction and the inner-optimization settings. We replayed both formulations on the same saved inputs in float32, with 10 inner SGD steps, an inner learning rate of 0.1 and the same stabilizer ϵ=10−8\epsilon=10^{-8}.

We compared the objective and its angle gradients at three common angle vectors per snapshot, and compared the returned angles and shared update after inner optimization:

QuantityMaximum difference: implemented kernel vs. original
Absolute objective difference1.16×10−41.16\times10^{-4}
Maximum coordinate difference in ∇αL\nabla_{\boldsymbol\alpha}\mathcal L3.54×10−53.54\times10^{-5}
Maximum returned-angle difference1.80×10−61.80\times10^{-6} rad
Relative shared-update difference, ∥vK−vO∥2/∥vO∥2\lVert v_{\mathrm{K}} - v_{\mathrm{O}} \rVert_2 / \lVert v_{\mathrm{O}} \rVert_21.42×10−61.42\times10^{-6}

The audit identifies a specific implementation difference. The original fidelity term directly computes 14T∑i∥ri−gˉi∥2,\frac{1}{4T}\sum_i\|r_i-\bar g_i\|^2, while the implemented kernel fidelity uses 12T∑i(1−ri⊤gˉi).\frac{1}{2T}\sum_i(1-r_i^\top\bar g_i).

The two are identical for unit-norm vectors. With stabilized normalization and finite precision, the vectors in the inner loop are only approximately unit, so the equivalence holds approximately. With matched inner-iteration budgets, the remaining NYU Δm%\Delta m\% gap is 1.481.48 percentage points. Our numerical audit shows that, on the evaluated snapshots, the maximum relative difference between the RGB shared update vectors is 1.42×10−61.42\times10^{-6}. Although the measured differences are small under matched inputs, small changes in successive updates can accumulate or be amplified during training, potentially producing different optimization trajectories. We have revised the paragraph to separate the exact Gram-matrix expansion from the unit-norm fidelity approximation we implement.

Both replays renormalize after rotation, ri←normalize⁡(ri,ϵ)r_i\leftarrow\operatorname{normalize}(r_i,\epsilon), before averaging the rotated gradients. The shared-update differences above include that step, so a mismatch in post-rotation normalization does not explain them.

4. Fairness of α_steps tuning.

We thank the reviewer for asking for a fixed-budget comparison. We evaluated RGB with αsteps=5\alpha_{\text{steps}}=5 on all four benchmarks, keeping the remaining experimental settings unchanged. Mean rank (MR) follows the metrics and ranking convention of Tables 1–2. Tuned and fixed-budget RGB are each evaluated against the same baseline results, with fixed-budget RGB replacing tuned RGB in the latter comparison.

DatasetTuned αsteps\alpha_{\text{steps}}Tuned ΔM\Delta M\\% ↓Tuned MR ↓Fixed αsteps\alpha_{\text{steps}}Fixed ΔM\Delta M\\% ↓Fixed MR ↓
NYUv210−8.612.115-7.622.89
Cityscapes30-1.703.005-0.604.67
QM9412.41.64521.01.91
CelebA100-1.493.2351.247.78

Under the shared budget, MR increases from 2.11 to 2.89 on NYUv2, from 3.00 to 4.67 on Cityscapes, from 1.64 to 1.91 on QM9, and from 3.23 to 7.78 on CelebA. NYUv2 and Cityscapes retain negative ΔM\Delta M\\%, while QM9 retains a low mean rank despite its increase in ΔM\Delta M\\%. Most importantly, RGB retains the strongest aggregate performance (best overall ΔM\Delta M\\% and Mean Rank) on these three datasets when compared to the baselines in Tables 1–2. CelebA exhibits the only major deterioration in mean rank relative to the tuned settings.

We also measured how far the rotations actually go under this budget, as the average returned angle:

DatasetAverage rotation angle at αsteps=5\alpha_{\text{steps}}=5 (rad)
NYUv20.008380
Cityscapes0.007809
QM90.001443
CelebA0.000891

The same iteration count gives very different amounts of rotation, so a shared budget is not a shared amount of gradient modification.

One likely reason is the task count. The objective averages over tasks, so each per-angle SGD gradient carries a 1/T1/T factor. With a fixed inner learning rate and comparable gradient geometry, more tasks means less angular movement in the same number of steps, and a larger budget would let the solve make more progress on the alignment–fidelity objective. The measured angles are consistent with this. This provides a mechanistic explanation for the observed budget dependence; the specific choice of 100 iterations is empirical.

By explicitly reporting these fixed-budget results alongside the tuned ones in Appendix J (Table 3) of our revised manuscript, the dependence on per-dataset budgets is now fully transparent to the reader.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Evidence

We sincerely thank the Area Chair and all three reviewers for their constructive feedback. We have uploaded a revised manuscript and provided detailed responses to the individual reviews.

For ease of reference, we briefly summarize the main revisions below.

  1. Methodological clarification and theoretical scope: We have clarified the distinction between RGB's implemented alignment–fidelity objective and MER's role as a first order motivating framework. We have also refined the convergence statements to reflect their explicit assumptions and added empirical diagnostics of common descent and the Pareto-stationarity residual (Appendix S). The observed residual trends provide useful empirical evidence for general convergence to pareto stationary.

  2. Hyperparameter sensitivity and evaluation protocols: We have added a fixed-budget comparison using α_steps = 5 across all four benchmarks (Appendix J, Table 3), alongside the original dataset-specific configurations. RGB retains competitive aggregate performance on NYUv2, Cityscapes, and QM9 under this fixed setting, while CelebA exhibits greater sensitivity. We have also clarified the benchmark evaluation and checkpoint-selection protocols.

  3. Computational efficiency and implementation verification: We have added runtime comparisons to Section 4.3, reporting the computational overhead associated with the evaluated configurations. We have also corrected the original-versus-kernel comparison to use matched inner-iteration budgets and conducted a numerical implementation audit (Appendix Q).

  4. Robustness and comparisons with related approaches: We have added controlled loss-scale experiments on Cityscapes, where RGB achieves Δm% values of −1.95 and −1.96 under depth-loss multipliers of 0.1 and 10, respectively. We have also clarified the distinctions between RGB and existing gradient manipulation and task-grouping approaches, including GO4Align, while acknowledging the limits of the evaluated task-scale range. We hope this concise overview helps locate the relevant revisions and additional evidence. Further experimental details, methodological clarifications, and limitations are documented in our reviewer-specific responses and revised manuscript.

We appreciate the Area Chair's and reviewers' time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.