- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ROWA, an adaptive task-weighting framework for multi-task learning that avoids computing individual task gradients for weight selection. It uses changes in task losses to approximate gradient-based progress measurements and updates task weights through mirror descent, aiming to extend low-cost weighting to different alpha-fairness criteria. The framework also incorporates smooth weighted-Chebyshev scalarization to express task preferences and explore different trade-offs. The paper presents a convergence analysis toward Pareto stationarity and evaluates the method on CelebA, QM9, NYUv2, and Cityscapes, covering two to forty tasks. Experiments compare predictive performance and training efficiency against both loss-weighting and gradient-based methods. The reported results show competitive performance, particularly on QM9, and lower computational costs than several gradient-based baselines, although performance gains are not consistent across all benchmarks and metrics.
Critical strengths
The experimental evaluation is reasonably comprehensive, covering four standard benchmarks with different task types and two to forty tasks. Comparisons against numerous loss-weighting and gradient-based baselines, together with training-time measurements, provide useful evidence about the method's performance and efficiency.
Critical weaknesses
-
The incremental contribution is not sufficiently isolated. Low-cost adaptive weighting, alpha-fairness, and smooth preference-based scalarization have been studied in FAMO, FairGrad, and smooth Tchebycheff scalarization, respectively. ROWA appears primarily to integrate and extend these ideas, but the ablations mainly vary alpha and tau. They do not isolate the contribution of the preference module or establish the benefit of the proposed weight update over simpler alternatives, such as smooth weighted-Chebyshev weighting alone. Consequently, the source and magnitude of the method's additional benefit remain unclear.
-
The empirical advantages are uneven and lack sufficient statistical support. In Table 1, ROWA improves the reported delta-m percentage metric over LDC-MTL by only 0.32 percentage points on CelebA and 0.30 on Cityscapes, while its mean task rankings are worse. On NYUv2, it underperforms FAMO and FairGrad. Although the QM9 results are more encouraging, the main comparison tables report averages over three runs without standard deviations or confidence intervals, making it difficult to determine whether the smaller improvements are robust.
Questions for authors
- Can you provide controlled comparisons removing the preference module and using smooth weighted-Chebyshev weighting alone, under matched training and tuning budgets, to isolate the benefit of the proposed weight update?
- Can you report variability across runs for ROWA and the strongest baselines, particularly for the small gains on CelebA and Cityscapes?
4 comments
Response to Weakness 2
We fully understand the reviewer's concerns. We acknowledge that ROWA does not achieve the best performance on both MR and across all benchmarks, which is challenging for MTL methods in general. Different methods have different priorities, and our ROWA focuses more on achieving general α-fair task weighting at a low computational cost.
Following common reporting practice in the MTL works, we report the variability Mean ± SD (Standard Deviation) for ROWA and the loss-oriented baselines as follows: (On NYUv2, FairGrad achieves the best performance but does not report the SD, so it is omitted from this table.)
| Method | CelebA | Cityscapes | NYU-v2 | QM9 (batch 120) | QM9 (batch 60) |
|---|---|---|---|---|---|
| FAMO | −1.00 ± 0.09 | 1.53 ± 1.76 | -4.72 ± 1.05 | 64.77 ± 1.23 | 35.80 ± 2.42 |
| Go4Align | 1.21 ± 0.19 | 2.23 ± 2.78 | -4.25 ± 0.73 | 139.43 ± 15.77 | 124.67 ± 11.80 |
| LDC-MTL | -1.67 ± 0.11 | 0.19 ± 1.07 | -2.90 ± 1.11 | 55.56 ± 6.21 | 46.13 ± 3.70 |
| ROWA | -1.99 ± 0.29 | -0.11 ± 0.41 | -4.21 ± 0.26 | 51.99 ± 4.64 | 32.45 ± 3.72 |
As can be seen, ROWA's SDs are generally comparable to those of the baselines, and are the lowest on NYUv2 and Cityscapes.
Response to Q1
Removing the preference module corresponds to setting (Table 1-3), while using smooth WC alone corresponds to the original STCH method [1]. To ensure a fair comparison, we use [1]'s provided results on NYU-v2 and QM9, which can be founded in [1]'s Table 2 and 7. We calculated and MR using their reported results and obtained the following table.
NYU-v2:
| Method | ↓ | MR ↓ | mIoU ↑ | Pix Acc ↑ | Abs Err ↓ | Rel Err ↓ | Normal Mean ↓ | Normal Median ↓ | Within 11.25° ↑ | Within 22.5° ↑ | Within 30° ↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| STCH | -1.35 | 5.44 | 41.35 | 66.07 | 0.4965 | 0.2010 | 26.55 | 21.81 | 24.84 | 51.39 | 64.86 |
| ROWA | -4.21 | 5.78 | 37.26 | 63.87 | 0.5300 | 0.2135 | 24.96 | 19.52 | 29.46 | 56.70 | 69.12 |
| ROWA (rerun) | -4.64 | 5.22 | 37.86 | 64.22 | 0.5385 | 0.2194 | 24.81 | 19.25 | 29.94 | 57.27 | 69.50 |
(In the rebuttal stage, we rerun ROWA on NYU-v2 under 3 random seeds, and obtain a better overall performance -4.64 and MR value 5.22. After re-computing MR, STCH's MR remains 5.44. We report these additional results in the rebuttal to further demonstrate ROWA's competitive performance, while retaining the original experimental results in the manuscript for consistency.)
QM9:
| Method | ↓ | MR ↓ | ↓ | ↓ | ↓ | ↓ | ↓ | ZPVE ↓ | ↓ | U ↓ | H ↓ | G ↓ | ↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| STCH | 56.9 | 5.64 | 0.166 | 0.260 | 94.48 | 101.20 | 1.85 | 4.88 | 58.34 | 58.68 | 58.70 | 58.27 | 0.104 |
| ROWA | 52.0 | 4.95 | 0.145 | 0.251 | 100.49 | 89.81 | 1.72 | 5.17 | 64.34 | 64.79 | 64.82 | 63.65 | 0.087 |
It can be observed that, ROWA outperforms STCH on the overall performance by a large margin.
[1] Lin, X., Zhang, X., Yang, Z., Liu, F., Wang, Z., Zhang, Q.: Smooth tchebycheff scalarization for multi-objective optimization. arXiv:2402.19078 (2024).
Response to Q2
The variability is reported in the response to the weakness 2.
Regrarding the small gains on Cityscapes and CelebA, ROWA achieves a better mean performance than LDC-MTL on both datastes, with lowest variability across runs on Cityscapes, and only a narrow overlap in the mean ± SD intervals ([-1.78,-1.70]) with LDC-MTL on CelebA.
I appreciate the author’s response and their candid acknowledgment that, while ROWA excels in efficiency, it does not achieve optimal performance. The rebuttal addressed my primary concerns, and I have raised my rating accordingly.
Thank you very much for your encouraging feedback!
Response to Weakness 1
Thank you for raising these questions, which enable us to further clarify our novelty and the benefits of our task weighting strategy.
Our novelty is threefold
We introduce a weighted utility problem argmaxd∈Bϵ∑i=1Kci1−α(gi(θt)⊤d)1−α, \mboxs.t. gi(θt)⊤d≥0, which helps specify a new preference-guided target ξc(θt)1/α⊙wt−1/α within the online weight adaptation framework. The design of c(θt) is inspired by the smoothed weighted-Chebyshev (WC) scalarization, but ROWA is not a simply extension of it. Besides, as explained in line 243-246 of the original manuscript, if α is set 0 in our weighted utility problem, the smoothed WC based gradient d=∑i=1Kci(θt)gi(θt) can be recovered. It implies that our framework is more general.
We adopt a novel analysis framework so that we can establish an O(1/T) non-asymptotic convergence guarantee toward Pareto stationarity in Theorem 1. Moreover, our theoretical analysis also covers the limiting case (α→+∞), corresponding to the min-max setting considered by FAMO. While, FAMO characterizes the stationary points of the continuous time limit in Section 3.4; FairGrad establishes the asymptotic convergence to the Pareto stationary point in Theorem 7.3.
Benefits of the weight update
➡️ Continue to the next post.