- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper focuses on resolving gradient conflicts in multi-task learning (MTL) with shared parameters. In this setting, task gradients can point in conflicting directions, such that an update that benefits one task may hurt another.
The paper proposes Rotation-Based Gradient Balancing (RGB), which modifies task gradients before combining them into a shared-model update. Existing approaches that modify task gradients to improve alignment may overcorrect, potentially destroying useful task-specific information. RGB attempts to balance these competing objectives.
Rather than merely reweighting task gradients or projecting out conflicting components, RGB rotates each normalized gradient toward a common reference direction. The rotation angles are task-specific and jointly optimized using an alignment objective with a penalty for deviating from the original gradients. The experiments evaluate RGB on four benchmarks with 3–40 tasks and compare it with existing approaches.
Critical strengths
-
The paper makes a useful distinction between reducing gradient conflict and improving multi-task optimization.
-
The fidelity penalty is important because it discourages the optimization from pursuing gradient alignment without accounting for deviations from the original task gradients.
-
Extensive theoretical analysis is appreciated.
Critical weaknesses
-
Gradient manipulation is already explored by existing approaches. The paper could better establish the methodological contribution of its rotation-based formulation and alignment–fidelity objective relative to prior methods.
-
Except for CelebA, the benchmarks contain only 3–11 tasks. Moreover, CelebA uses the same input images for its 40 binary-attribute tasks. How well does RGB scale to substantially larger and more heterogeneous task sets? What is the computational cost of the gradient operations and rotation optimization as the number of tasks increases?
-
RGB normalizes task gradients before combining them, which removes their relative magnitude information. How does it behave when tasks have substantially different loss scales, convergence rates, or degrees of difficulty? How does it compare with methods that explicitly account for task magnitudes or loss scales?
-
Existing approaches, including the proposed method, generally assume that all tasks are learned together in a single MTL model. However, this may not scale well to large task sets, and prior studies have shown that grouping related tasks can improve MTL performance. Does the paper compare RGB with task-grouping approaches, particularly recent methods that explicitly identify groups of tasks, rather than applying gradient modification to all tasks jointly?
Questions for authors
See weakness.
3 comments
3. Gradient normalization and task-scale sensitivity.
RGB normalizes task gradients to prevent larger gradient magnitudes from dominating the shared directional update. Such differences can arise from loss units and scaling, as well as task-specific optimization dynamics. Normalization ensures the rotation objective operates on task-gradient directions, allowing RGB to optimize alignment and fidelity without directly weighting tasks by their original gradient norms.
To evaluate sensitivity to loss scaling, we conducted controlled Cityscapes experiments in which the training loss of depth estimation was multiplied by , while the other task losses remained unchanged. We kept each method’s configuration fixed across scaling factors and evaluated using the original task metrics and STL references. We compare RGB with UW and GradNorm to assess its behavior alongside alternative task-balancing mechanisms that explicitly account for task magnitudes and loss scales. The table below reports results averaged over three seeds.
| Method | Depth-loss multiplier | mIoU [%] ↑ | L1 [px] ↓ | MSE ↓ | ↓ |
|---|---|---|---|---|---|
| UW | 0.1 | 57.14 | 10.10 | 0.40 | 10.47 |
| UW | 10 | 57.55 | 10.13 | 0.35 | 5.23 |
| GradNorm | 0.1 | 52.12 | 10.17 | 0.44 | 17.36 |
| GradNorm | 10 | 54.40 | 10.07 | 0.35 | 6.71 |
| RGB () | 0.1 | 66.02 | 10.03 | 0.32 | −1.95 |
| RGB () | 10 | 66.05 | 10.03 | 0.32 | −1.96 |
RGB achieves the best per-task metrics and among these methods at both scaling factors. Its remains consistent across the two settings, while UW and GradNorm compared with UW and GradNorm exhibit substantial gaps between the two multipliers. These results support RGB’s robustness to the tested depth-loss rescaling. They assess sensitivity to numerical loss scales, rather than explicit adaptation to task difficulty or relative convergence rates.
Although RGB normalizes the individual rotated directions, their aggregate is not constrained to unit norm: rotation changes both the shared direction and its magnitude through directional agreement and cancellation. This differs from adjusting task weights using gradient magnitudes or loss information. The aggregate magnitude reflects agreement among the rotated directions; it does not restore the original task-gradient magnitudes.
The outer optimizer (Adam in our experiments) subsequently converts this aggregate into a parameter update, incorporating first- and second-moment estimates to adapt the step coordinate-wise. RGB’s inner optimization controls task-direction geometry, while the outer optimizer determines the resulting parameter update.
4. Comparison with task-grouping approaches.
Thank you for highlighting this direction. Our main tables already include an empirical comparison with GO4Align [1] on NYUv2 and QM9. To make this comparison more explicit, we have expanded the Introduction’s discussion to distinguish two uses of task grouping: selecting task subsets to train in separate models, and organizing task updates within a single shared model.
The latter is particularly relevant to RGB’s optimization setting. GO4Align dynamically groups tasks and uses risk-guided group indicators to coordinate optimization. Selective Task Group Updates [2] also operate within a shared model, using task affinities to form groups and update them sequentially. RGB instead jointly optimizes task-specific rotation angles through an alignment–fidelity objective, without explicitly partitioning the tasks.
We have now formally discussed both methods in the revised Introduction, while distinguishing methodological comparison from empirical evaluation. Our quantitative comparison includes GO4Align. We have not evaluated Selective Task Group Updates under our protocols; its published NYUD-v2 results use a different task configuration and architecture, so they cannot be directly incorporated into our performance tables.
We also agree that larger benchmarks would be valuable for investigating how joint learning behaves as task diversity and task count increase. At substantially larger scales, selecting which tasks share parameters becomes increasingly relevant. RGB could potentially operate within selected task groups, combining task-group selection with alignment–fidelity control during optimization. Evaluating this combination is a useful direction for future work.
Reference:
[1] GO4Align: Group Optimization for Multi-Task Alignment
[2] Selective Task Group Updates for Multi-Task Optimization
Response by Reviewer 63xu
Thanks to the authors for addressing my concerns and providing additional results. I will keep my score unchanged.
We thank the reviewer for the constructive feedback and comments. We address your concerns and questions in the following.
Weaknesses:
1. Methodological contribution relative to prior approaches.
Thank you for the suggestion. We agree that gradient manipulation is an established approach to multi-task learning. RGB’s core methodological contribution lies in its task-specific rotation parameterization and the joint optimization of these angles through an alignment–fidelity objective.
To make the distinction from prior gradient manipulation methods easier to identify, we have refined the synthesis following the comparison with prior methods in the Introduction, making RGB’s joint control of rotation magnitude and fidelity explicit. Our angular sweeps and inner-budget ablations provide empirical support for this design: blindly increasing modification does not consistently improve performance, and the preferred amount varies across datasets.
2. Scaling to larger and more heterogeneous task sets.
Our evaluation focuses on optimization within a shared multi-task model across benchmarks containing 3–40 tasks. CelebA evaluates RGB with 40 jointly learned attributes, while the other benchmarks cover different task types and application domains. We agree that CelebA does not by itself establish effectiveness on substantially larger, heterogeneous task collections. Evaluating such collections is beyond the experimental scope of this paper; our empirical claims concern the benchmark settings evaluated here.
To address computational scalability as the number of tasks increases, we have included a compact runtime comparison in the main text (Section 4.3) using the configurations associated with the reported performance results. Let T denote the number of tasks, D the shared-parameter dimension, and αsteps the inner-optimization budget. RGB computes task-specific gradients through task-wise backward passes, constructs the Gram matrices once per rotation optimization at a cost of O(T2D), and performs αsteps kernelized inner iterations at a total cost of O(αstepsT2). Thus, excluding task-gradient computation, the rotation procedure costs O(T2D+αstepsT2). The pairwise computations scale quadratically with task count, whereas the inner-optimization cost grows linearly with αsteps. Increasing the inner budget does not require additional network backward passes or repeated Gram-matrix construction. Kernelization therefore removes repeated parameter-dimensional operations from the inner loop, while retaining the task-count dependence of pairwise alignment. These measurements and complexity estimates characterize computational scaling on the evaluated benchmarks. Effectiveness on substantially larger, heterogeneous task collections remains future work.