acceptodds
Under review as a conference paper at ICLR 2027

ROWA: A Low-Complexity Adaptive Weight Framework for Multi-Task Learning

Abstract

Gradient manipulation methods in Multi-Task Learning (MTL) are effective for balancing competing objectives, yet they often suffer from severe computational bottlenecks. Recent -aware utility optimization approaches unify various task-level fairness criteria, but require expensive task-specific backward passes at each training step, where denotes the number of tasks. To overcome this scalability barrier, we propose ROWA (GeneRic Online Weight Adaptation), which is derived from an online optimization formulation incorporating the loss decrease and -aware targets. ROWA is implementation-friendly governed by a single mirror descent step at each iteration, and completely obviates the need for any task-specific gradient computation during task weighting. Remarkably, ROWA can be generalized to arbitrary -configurations and readily incorporate the Weighted-Chebyshev strategy to enable exploration of user-preferred regions on the Pareto front. We establish an iteration complexity for the convergence of ROWA to Pareto stationarity in the nonconvex setting. Extensive experiments on four benchmarks with two to 40 tasks show that ROWA outperforms a wide range of strong baselines at negligible computational overhead. Ablation studies further confirm its flexibility across fairness criteria and Pareto-front preferences.

Primary area:
optimization
Keywords:
Multi-task learning, Fairness optimization, Pareto Front Navigation, Mirror Descent

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_iYWJOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ROWA, an adaptive task-weighting framework for multi-task learning that avoids computing individual task gradients for weight selection. It uses changes in task losses to approximate gradient-based progress measurements and updates task weights through mirror descent, aiming to extend low-cost weighting to different alpha-fairness criteria. The framework also incorporates smooth weighted-Chebyshev scalarization to express task preferences and explore different trade-offs. The paper presents a convergence analysis toward Pareto stationarity and evaluates the method on CelebA, QM9, NYUv2, and Cityscapes, covering two to forty tasks. Experiments compare predictive performance and training efficiency against both loss-weighting and gradient-based methods. The reported results show competitive performance, particularly on QM9, and lower computational costs than several gradient-based baselines, although performance gains are not consistent across all benchmarks and metrics.

Critical strengths

The experimental evaluation is reasonably comprehensive, covering four standard benchmarks with different task types and two to forty tasks. Comparisons against numerous loss-weighting and gradient-based baselines, together with training-time measurements, provide useful evidence about the method's performance and efficiency.

Critical weaknesses
  1. The incremental contribution is not sufficiently isolated. Low-cost adaptive weighting, alpha-fairness, and smooth preference-based scalarization have been studied in FAMO, FairGrad, and smooth Tchebycheff scalarization, respectively. ROWA appears primarily to integrate and extend these ideas, but the ablations mainly vary alpha and tau. They do not isolate the contribution of the preference module or establish the benefit of the proposed weight update over simpler alternatives, such as smooth weighted-Chebyshev weighting alone. Consequently, the source and magnitude of the method's additional benefit remain unclear.

  2. The empirical advantages are uneven and lack sufficient statistical support. In Table 1, ROWA improves the reported delta-m percentage metric over LDC-MTL by only 0.32 percentage points on CelebA and 0.30 on Cityscapes, while its mean task rankings are worse. On NYUv2, it underperforms FAMO and FairGrad. Although the QM9 results are more encouraging, the main comparison tables report averages over three runs without standard deviations or confidence intervals, making it difficult to determine whether the smaller improvements are robust.

Questions for authors
  1. Can you provide controlled comparisons removing the preference module and using smooth weighted-Chebyshev weighting alone, under matched training and tuning budgets, to isolate the benefit of the proposed weight update?
  2. Can you report variability across runs for ROWA and the strongest baselines, particularly for the small gains on CelebA and Cityscapes?
4 comments

Response to Weakness 1

Thank you for raising these questions, which enable us to further clarify our novelty and the benefits of our task weighting strategy.

Our novelty is threefold

  1. We develop a new online optimization formulation to accommodate general α-fairness criteria. As explained in lines 167–174 of the original manuscript, directly approximating the Gt⊤GtwG_t^\top G_tw term with loss history changes (denoted as Δli\Delta l_i here) breaks the algebraic balance between the two sides of (2) (Δli=ξw−1/α\Delta l_i=\xi w^{-1/\alpha}), preventing the direct ww-solving procedure used in FairGrad. To address this, we introduces the regret-minimization problem (4) and the mirror descent (5). While, FAMO is developed based on MGDA, where the gradient Jt⊤JtzJ_t^\top J_tz used for the zz-update can be more directly approximated using Δli\Delta l_i. Moreover, ROWA differs substantially from FAMO in both its ww-weight adaptation and model parameter θ\theta-update mechanisms:
MethodWeight adaptationθ\theta-update (Adam omitted)
FAMOUpdate ξi,t+1=ξi,t−β(∇zi,t⋅(log⁡li,t−log⁡li,t+1)+γξi,t)\xi_{i,t+1}=\xi_{i,t}-\beta(\nabla z_{i,t}\cdot(\log l_{i,t}-\log l_{i,t+1})+\gamma\xi_{i,t}), then project to the simplex via zt=Softmax⁡(ξt)z_t=\operatorname{Softmax}(\xi_t)θt+1=θt−α∑i=1kct⋅(zi,t∇log⁡li,t\theta_{t+1}=\theta_t-\alpha\sum\limits_{i=1}^{k}c_t\cdot(z_{i,t}\nabla\log l_{i,t})
ROWAUpdate wi,tw_{i,t} using mirror descent with direction hi,t=ui,tut.abs().mean()−vi,tvt.mean()+γlog⁡wi,th_{i,t}=\frac{u_{i,t}}{u_t.abs().mean()}-\frac{v_{i,t}}{v_t.mean()}+\gamma\log w_{i,t}, where ui,t=li,t(log⁡li,t−log⁡li,t+1)u_{i,t}=l_{i,t}(\log l_{i,t}-\log l_{i,t+1}) and vi,t=ci,t1/αwi,t−1/αv_{i,t}=c_{i,t}^{1/\alpha}w_{i,t}^{-1/\alpha}θt+1=θt−η∑i=1kwi,t∇li,t\theta_{t+1}=\theta_t-\eta\sum\limits_{i=1}^{k}w_{i,t}\nabla l_{i,t}
  1. We introduce a weighted utility problem arg⁡max⁡d∈Bϵ∑i=1Kci(gi(θt)⊤d)1−α1−α, \mboxs.t. gi(θt)⊤d≥0\arg\max_{d\in B_{\epsilon}}\sum_{i=1}^{K} c_{i}\frac{(g_i(\theta_t)^\top d)^{1-\alpha}}{1-\alpha},\ \mbox{s.t.}\ g_i(\theta_t)^\top d\geq0, which helps specify a new preference-guided target ξc(θt)1/α⊙wt−1/α\xi c(\theta_t)^{1/\alpha}\odot w_t^{-1/\alpha} within the online weight adaptation framework. The design of c(θt)c(\theta_t) is inspired by the smoothed weighted-Chebyshev (WC) scalarization, but ROWA is not a simply extension of it. Besides, as explained in line 243-246 of the original manuscript, if α\alpha is set 00 in our weighted utility problem, the smoothed WC based gradient d=∑i=1Kci(θt)gi(θt)d=\sum_{i=1}^K c_i(\theta_t)g_i(\theta_t) can be recovered. It implies that our framework is more general.

  2. We adopt a novel analysis framework so that we can establish an O(1/T)O(1/\sqrt{T}) non-asymptotic convergence guarantee toward Pareto stationarity in Theorem 1. Moreover, our theoretical analysis also covers the limiting case (α→+∞\alpha \to +\infty), corresponding to the min-max setting considered by FAMO. While, FAMO characterizes the stationary points of the continuous time limit in Section 3.4; FairGrad establishes the asymptotic convergence to the Pareto stationary point in Theorem 7.3.

Benefits of the weight update

  1. All performance evaluations in Table 1-3 are conducted without the preference module (i.e., by setting ci(θ)=1c_i(\theta)=1 as explained in Appendix line 801-806 of the original manuscript.). Table 1-3 show that ROWA outperform other baselines on CelebA, Cityscapes and QM9 benchmarks. It highlights the benefits of ROWA's α\alpha-fair task weighting adaptation alone. For greater clarity, we have revised Appendix Section B.1 accordingly.
  2. In Sections 5.2(2), we further incorporate user preferences by varying ci(θ)c_i(\theta) through τ\tau and the preference vector r\mathbf{r}. Figure 7 shows that, under equal preferences, ROWA still achieves comparable or better performance than the strongest baselines on CelebA, QM9, and Cityscapes under certain values of τ\tau, i.e., -1.83, 55.59, 0.19 respectively. Figure 7 aims to show that introducing the preference module allows ROWA to maintain competitive task performance while enabling preference-guided optimization.

➡️ Continue to the next post.

Response to Weakness 2

We fully understand the reviewer's concerns. We acknowledge that ROWA does not achieve the best performance on both MR and Δm%\Delta m\% across all benchmarks, which is challenging for MTL methods in general. Different methods have different priorities, and our ROWA focuses more on achieving general α-fair task weighting at a low computational cost.

Following common reporting practice in the MTL works, we report the variability Mean ± SD (Standard Deviation) for ROWA and the loss-oriented baselines as follows: (On NYUv2, FairGrad achieves the best performance but does not report the SD, so it is omitted from this table.)

MethodCelebACityscapesNYU-v2QM9 (batch 120)QM9 (batch 60)
FAMO−1.00 ± 0.091.53 ± 1.76-4.72 ± 1.0564.77 ± 1.2335.80 ± 2.42
Go4Align1.21 ± 0.192.23 ± 2.78-4.25 ± 0.73139.43 ± 15.77124.67 ± 11.80
LDC-MTL-1.67 ± 0.110.19 ± 1.07-2.90 ± 1.1155.56 ± 6.2146.13 ± 3.70
ROWA-1.99 ± 0.29-0.11 ± 0.41-4.21 ± 0.2651.99 ± 4.6432.45 ± 3.72

As can be seen, ROWA's SDs are generally comparable to those of the baselines, and are the lowest on NYUv2 and Cityscapes.

Response to Q1

Removing the preference module corresponds to setting ci(θ)=1c_i(\theta)=1 (Table 1-3), while using smooth WC alone corresponds to the original STCH method [1]. To ensure a fair comparison, we use [1]'s provided results on NYU-v2 and QM9, which can be founded in [1]'s Table 2 and 7. We calculated Δm%\Delta m\% and MR using their reported results and obtained the following table.

NYU-v2:

MethodΔm%\Delta m\% ↓MR ↓mIoU ↑Pix Acc ↑Abs Err ↓Rel Err ↓Normal Mean ↓Normal Median ↓Within 11.25° ↑Within 22.5° ↑Within 30° ↑
STCH-1.355.4441.3566.070.49650.201026.5521.8124.8451.3964.86
ROWA-4.215.7837.2663.870.53000.213524.9619.5229.4656.7069.12
ROWA (rerun)-4.645.2237.8664.220.53850.219424.8119.2529.9457.2769.50

(In the rebuttal stage, we rerun ROWA on NYU-v2 under 3 random seeds, and obtain a better overall performance -4.64 and MR value 5.22. After re-computing MR, STCH's MR remains 5.44. We report these additional results in the rebuttal to further demonstrate ROWA's competitive performance, while retaining the original experimental results in the manuscript for consistency.)

QM9:

MethodΔm%\Delta m\% ↓MR ↓μ\mu ↓α\alpha ↓ϵHOMO\epsilon_{HOMO} ↓ϵLUMO\epsilon_{LUMO} ↓⟨R2⟩\langle R^2\rangle ↓ZPVE ↓U0U_0 ↓U ↓H ↓G ↓cvc_v ↓
STCH56.95.640.1660.26094.48101.201.854.8858.3458.6858.7058.270.104
ROWA52.04.950.1450.251100.4989.811.725.1764.3464.7964.8263.650.087

It can be observed that, ROWA outperforms STCH on the overall performance Δm%\Delta m\% by a large margin.

[1] Lin, X., Zhang, X., Yang, Z., Liu, F., Wang, Z., Zhang, Q.: Smooth tchebycheff scalarization for multi-objective optimization. arXiv:2402.19078 (2024).

Response to Q2

The variability is reported in the response to the weakness 2.

Regrarding the small gains on Cityscapes and CelebA, ROWA achieves a better mean performance than LDC-MTL on both datastes, with lowest variability across runs on Cityscapes, and only a narrow overlap in the mean ± SD intervals ([-1.78,-1.70]) with LDC-MTL on CelebA.

I appreciate the author’s response and their candid acknowledgment that, while ROWA excels in efficiency, it does not achieve optimal performance. The rebuttal addressed my primary concerns, and I have raised my rating accordingly.

Thank you very much for your encouraging feedback!

Reviewer_ZuXpOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a computationally efficient α\alpha-aware multi-task learning (MTL) method. The paper shows that at every gradient iteration when one has to find a new set of task weightings, one can solve an online learning problem efficiently with mirror descent with O(1)\mathcal{O}(1) gradient complexity instead of the typical O(K)\mathcal{O}(K) complexity that gradient-based MTL methods usually incur. The paper then establishes a non-asymptotic O(1T)\mathcal{O}(\frac{1}{\sqrt{T}}) convergence of this algorithm to Pareto stationary-points under mild conditions in the non-convex setting. The paper then empirically shows that their algorithm performs either competitively with or outperforms leading MTL baselines on standard supervised learning MTL benchmarks, while being consistently significantly efficient than gradient-based methods in terms of training time.

Critical strengths
  • I find the regret minimization formulation to solve the problem induced by the gram matrix to be quite novel and important. This significantly removes the computational burden that gradient-based MTL methods incur.
  • On the theoretical front, my understanding is that the paper uses milder assumptions for proving convergence compared to existing MTL methods like Nash-MTL (which require additional gradient independence assumptions).
Critical weaknesses
  • All results are over only 3 random seeds. While I understand this unfortunately somehow has become standard practice across the MTL literature for NYUv2/QM9/Cityscapes/CelebA, such few seeds make it a bit hard to be completely convinced of the trends/claims. However, I am willing to discount this weakness as I understand several gradient based methods take excessive runtimes that can make collecting results for more seeds a bit restrictive.
  • I feel 11 task QM9 would be a better benchmark for the ablation on α\alpha rather than cityscapes/NYUv2 (2/3 tasks only), because of its greater diversity of loss scales, especially because the paper claims on line 472 that "Increasing α\alpha can generally improve the Δ\Delta \\% of lower-loss tasks".
Questions for authors

The paper was overall clear and I have only few minor questions.

  • What is the value for α\alpha for ROWA for Tables 1-3?
  • Is there a specific reason why QM9 could not be used for ablation over α\alpha?
2 comments

Thank you for your encouraging review and your valuable suggestions, which helped us further strengthen our work.

Response to Weakness 1

We understand the reviewer's concern. To provide more robust empirical evidence, we reran ROWA with 8 different random seeds and report the Mean ± SD across these runs below. As can be seen, the performance of ROWA is still competitive.

MethodCelebANYU-v2CityscapesQM9
ROWA-1.81 ± 0.33−4.28 ± 0.57-0.11± 1.454.905 ± 4.142

Response to Weakness 2

Our α\alpha-ablation experiments follow the experimental protocol of FairGrad (Ban & Ji, 2024, Section 5.4), which investigates different α\alpha settings on NYU-v2 and Cityscapes. We selected them to maintain consistency with prior α\alpha-fairness evaluations, and demonstrate that our O(1)O(1) approach still preserves the influence of α\alpha on training results.

Response to Q1

We set α=2\alpha=2 in Table 1-3.

Response to Q2

We sincerely thank the reviewer for this insightful question. We think QM9 is not an ideal benchmark for this purpose.

To improve the overall performance Δm%\Delta m\% on QM9, most MTL methods (including Nash-MTL, FairGrad, FAMO, MGDA and our ROWA) set scale_y=True in their implementations, which standardizes the ground-truth labels of each task as yi′=yi−μiσiy_i'=\frac{y_i-\mu_i}{\sigma_i}, where μi\mu_i and σi\sigma_i denote the mean and standard deviation of the training labels for task ii, respectively. The training losses are then computed using these standardized labels. When reporting Table 9, the MAEs are converted back to their original scales.

Here we report an example of the training losses that directly interact with α\alpha during the training process.

"Real" Training Loss Example:

μαϵHOMO\epsilon_{HOMO}ϵLUMO\epsilon_{LUMO}⟨R²⟩ZPVEU₀UHGCᵥ
Initial batch1.4179670.8246010.9406291.1066010.8265440.9603180.9413120.9716560.9284130.9614921.004575
Epoch 0 average0.6033240.2386430.6669410.4137910.1345000.0703980.1041070.1032750.1006550.1066810.213818

It can be observed from the above table that

"bigger loss" tasks become: μ\mu α\alpha ϵHOMO\epsilon_{HOMO} ϵLUMO\epsilon_{LUMO} cvc_v

"lower loss" tasks become: <R2R^2> ZPVE U0U_0 U H G

which is different from the MSE's scale reported in Table 9.

As can be seen in the following table, the trend generally matches our claims that "Increasing α\alpha can generally improve the Δ%\Delta\% of lower-loss tasks".

Methodμ\mu ↓α\alpha ↓ϵHOMO\epsilon_{HOMO} ↓ϵLUMO\epsilon_{LUMO} ↓⟨R2⟩\langle R^2\rangle ↓ZPVE ↓U0U_0 ↓U ↓H ↓G ↓cvc_v ↓MR ↓Δm%\Delta m\% ↓
ROWA (α=1\alpha=1)0.1180.26485.4785.882.5157.2585.6986.2886.3984.050.0962.9177.9
ROWA (α=2\alpha=2)0.1450.251100.4989.811.7205.1764.3464.7964.8263.650.0872.2752.0
ROWA (α=5\alpha=5)0.2720.296136.60114.721.4524.9964.0064.4064.4463.720.0932.4576.5
ROWA (α=10\alpha=10)0.4390.367170.03150.221.3844.8463.4063.8563.8963.180.1042.36113.3

Since it would require substantial additional words and figures in the main paper to clarify what is the "real" lower-loss tasks and how varying α\alpha affects them, we only use NYU-v2 and Cityscapes.

Thank you for your response, I have no more questions.

Reviewer_AUFfOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the computational cost of multi-task learning. Existing α-aware utility optimization approaches typically require task-specific gradient computation at each training step, incurring additional computational overhead from backpropagation. To address the computational bottleneck of these optimization methods, ROWA constructs low-cost feedback from the changes in losses before and after parameter updates, avoiding separate gradient computation for each task. Through theoretical derivation, ROWA reduces the number of backward passes required per training step from O(K) to O(1), while supporting user-specified task preferences. The experiments cover four benchmarks, CelebA, QM9, Cityscapes, and NYU-v2, with 2–40 tasks. The results show that ROWA is competitive on several aggregate metrics and has clear runtime advantages over methods that compute task-specific gradients across different benchmarks. The ablation studies further show that adjusting user preferences changes the performance profile across tasks, varying α affects the performance trade-offs among tasks, and the optimal value of the smoothing parameter \tau also varies across datasets. The authors analyze the optimal values of different hyperparameters. Overall, the paper proposes a valuable approach to task weighting in multi-task learning and provides theoretical analysis and experimental validation. However, some issues concerning theoretical boundary cases, the scope of applicability, and certain experimental conclusions require further clarification.

Critical strengths

1.A valuable method for addressing the computational bottleneck in multi-task optimization In multi-task learning, as the number of tasks increases and the size of the shared network increases, computing each task's gradient separately to determine its weight can incur substantial computational overhead. Through theoretical analysis and derivation, ROWA uses a weighted sum of task losses to update the model, thereby avoiding task-specific backward passes during weight updates while retaining support for user-specified preferences. The paper also provides convergence analysis to explain the theoretical basis of this design, which has considerable research value. 2.Experimental results validate the feasibility and applicability of the method The experiments cover four benchmarks, CelebA, QM9, Cityscapes, and NYU-v2, with 2–40 tasks. The results show that the method achieves leading results on the main metrics of several benchmarks, although not all of them (Tables 1–3). ROWA also requires less training time than other MTL optimization methods, measured either per epoch or by the wall-clock time needed to reach the same target performance (Figures 4 and 6). These results support the method's efficiency–performance trade-off in settings with many tasks and provide direct evidence of its practical value.

Critical weaknesses

The construction and comparison protocol of Figure 5 are not clearly explained and do not provide sufficient support for ROWA's more comprehensive and balanced Pareto front coverage. Based on the lower subfigure of Figure 5, the authors conclude that ROWA achieves more comprehensive and balanced coverage across all tasks. However, the results labeled “ours” appear to have been normalized using ROWA's own results as the reference, so the resulting shape cannot serve as direct evidence of balanced performance across tasks. The paper also does not clearly disclose how the figure was plotted or explain whether the data for the other baselines were processed in a comparable way. Furthermore, considering the upper subfigure of Figure 5 together with Table 12, increasing the preference weights for U_0, U, and G does not yield the lowest MAE on these tasks, and some performance profiles on these tasks are similar. The authors do not provide further explanation of this. Therefore, Figure 5 currently provides more direct support for the observation that adjusting preferences in ROWA changes task performance, but it does not sufficiently support the conclusion that ROWA is superior to most baselines in Pareto front exploration and comprehensive, balanced coverage. This issue weakens the persuasiveness of the paper's contribution concerning user preference control.

Questions for authors
  1. What are the specific aggregation method and baseline comparison settings used in Figure 5? Please clarify the metric transformations in both the upper and lower subfigures, especially which configurations the data for “ours” and the different baselines in the lower subfigure come from and whether they are comparable. Please also explain how this comparison supports superior Pareto front coverage.
  2. The objective values in Figures 1 and 2 of the main text include zero or negative losses, whereas Algorithm 1 uses log⁡L\log L. Please explain this. Also, please check the remainder notation o(a3)o(a^3) after retaining the second-order term in Remark 1; it should be O(a3)O(a^3).
  3. The main text states that the framework can cover different or even arbitrary α settings, but Eq. (1) requires an additional explanation of its definition at α=1\alpha=1, and α=0\alpha=0 cannot be directly substituted into Eq. (2) or Algorithm 1. Please clarify these boundary cases. Please also make the dimensions of GG and the expression GwGw consistent in line 142 of the main text.
  4. In Table 1, CelebA and Cityscapes show some improvement in Δm%\Delta_m\% over the strongest baseline, whereas ROWA does not outperform the strongest baseline in terms of MR. The wording “surpassing prior approaches by a large margin” in lines 389–391 of the main text is not sufficiently supported.
  5. Table 1 reports only Δₘ% for CelebA. Please explain why MR is omitted and provide the missing results if possible.
  6. In Figure 6, the runtime on Cityscapes and NYU-v2 is still 1.30 and 1.22 times that of LS, respectively, showing a clear difference in overhead from the QM9 and CelebA datasets. Please explain the main source of this difference. The claim of “negligible computational overhead” in the abstract should also be qualified.
3 comments

Response to Weakness

We sincerely thank the reviewer for pointing these out, which helped us improve the clarity of our paper. In the main paper, we have updated the explanation of Figure 5, as shown in the highlighted Lines 453 - 469 in the updated manuscript. We also added an entire Appendix Section B.2 in the updated manuscript, to explain these issues in details.

How Figure 5 is plotted

  1. Data processing. We evaluate ROWA under 11 preference settings, assigning weight 0.80.8 to one task and 0.020.02 to each remaining task, as mentioned in Lines 452-453. For each setting and seed, we select the epoch with the lowest test loss on the preferred task and extract all task losses at that epoch. We average these losses over three seeds to obtain Lˉp,j\bar L_{p,j} for preference setting pp and task jj. As shown in Table 12, ROWA allows explicit control of task preferences, yielding 11 performance vectors which corresponds to MAE values (test loss Lˉp,j\bar L_{p,j}) across 11 lines. Whereas, performance result of each other baseline in Table 9 is generated without a preference control, corresponding to 1 line. We construct ROWA's task-wise envelope as ej=min⁡pLˉp,je_j=\min_p \bar{L}_{p,j}.

  2. Curve construction. For all curves in the upper sub-figure of Figure 5, the value on axis jj for task jj is ej/Lˉp,je_j/\bar{L}_{p,j} . This makes the envelope equal to 1 on each of the 11 axes. For all curves in the lower sub-figure of Figure 5, the value on axis jj is ej/Lm,je_j/L_{m,j}, where Lm,jL_{m,j} is the corresponding baseline mm's loss on task jj, obtained from Table 9. Thus, the "Ours" curve uses the envelope and equals 1 on every axis, while the baseline curves use the same reference and can exceed 1 if their losses are lower.

Clarifying the discussion of Figure 5

  1. Unlike Figures 2 and 9 in the toy examples, which clearly plotted the preference-guided convergent points on the two-dimensional Pareto fronts, Figure 5 does not explicitly show the Pareto front curvature. Since directly visualizing an 11-dimensional Pareto front is challenging, we use a radar chart in Figure 5 to illustrate how adjusting user preferences enables ROWA to explore diverse task-performance trade-offs.

  2. For the "comprehensive and balanced coverage" concern of the lower figure, we agree that using ROWA's task-wise performance envelope as the normalization reference makes its results appear uniformly balanced by construction. Our original interpretation overstated the "more balance" claim.

Next we would like to introduce another experiment to further explain how ROWA achieves a "more comprehensive" (than other baselines) Pareto front coverage, by evaluating the HyperVolumn (HV) metric of all methods.

Clarifying Pareto front coverage via HV

For our method, HV is the volume of the union of the boxes formed between each loss vector and a common reference point, with overlapping regions counted only once. Each preference-free baseline is represented by one reported loss vector in Table 9, so its HV reduces to a single box volume. In contrast, ROWA has 11 loss vectors as reported in Table 12, so its HV is calculated as the volume of the union of the 11 corresponding boxes.

A larger HV indicates that these solutions jointly dominate a larger volume relative to the common reference point, which implies a broader Pareto front coverage. Our goal is to examine whether ROWA's multiple preference-guided solutions collectively cover a broader region of the objective space. The result is reported as below. It can be seen that our ROWA's solution set achieves the highest HV among all baselines.

MethodRaw-lossHV↑Radar-scoreHV↑Rank↓
Ours2.206e+114.572e-051
MGDA4.454e+092.972e-0810
PCGrad4.026e+092.581e-0811
CAGrad4.686e+093.487e-089
Nash-MTL1.589e+112.368e-054
FairGrad1.692e+112.843e-053
LS2.583e+072.799e-1113
SI8.532e+097.925e-076
RLW3.138e+051.503e-1314
DWA3.651e+074.172e-1112
UW1.046e+081.877e-098
Go4Align2.452e+095.883e-087
FAMO7.129e+105.161e-065
LDC-MTL7.043e+102.854e-052

(We report HV in both the raw-loss space (Raw-loss HV) and the normalized inverse-loss space used in Figure 5 (Radar-score HV). For the latter, we use sk=ek/Lks_k=e_k/L_k and the consistently transformed reference point qk=ek/rkq_k=e_k/r_k, where eke_k is the minimum loss across ROWA's 11 solutions for task kk.)

➡️ Continue to the next post.

Preferences on U0U_0, UU, HH and GG

The similar performances on these tasks are due to the strong correlations among them.

To see this, we examined the Pearson correlations among the ground-truth labels of QM9 tasks. Each sample of QM9 datasets owns 11 ground-truth labels used for 11 tasks. For example, given NN samples, we construct two label vectors for tasks U0U_0 and UU:yU0=(y1,U0,y2,U0,..,yN,U0),yU=(y1,U,y2,U,..,yN,U).y_{U_0}=(y_{1,U_0},y_{2,U_0},..,y_{N,U_0}),y_{U}=(y_{1,U},y_{2,U},..,y_{N,U}). We then compute their Pearson correlation coefficient as $r_{U_0,U}= \frac{\sum_{n=1}^{N}(y_{n,U_0}-\bar y_{U_0})(y_{n,U}-\bar y_U)} {\sqrt{\sum_{n=1}^{N}(y_{n,U_0}-\bar y_{U_0})^2} \sqrt{\sum_{n=1}^{N}(y_{n,U}-\bar y_U)^2}}.$ We repeat this calculation for all task pairs. Correlations closer to 1 indicate that the labels follow nearly the same pattern of variation. When trained with a shared network, these tasks may benefit from the same learned features. Thus, emphasizing one task may also improve the others, leading to similar performance profiles under different preference settings.

As shown in the following table, all pairwise correlations between U0U_0, UU, HH and GG exceed 0.9997, indicating extremely strong relationships among these four tasks.

Task PairPearson rr
(U0,U)(U_0,U)0.99999535
(U0,H)(U_0,H)0.99999216
(U0,G)(U_0,G)0.99985764
(U,H)(U,H)0.99999890
(U,G)(U,G)0.99982210
(H,G)(H,G)0.99979570

(Their similarity are expected, due to their closely related thermodynamic definitions and shared electronic-energy contribution. The close physical relationships suggest that they may involve weaker trade-offs and benefit from shared representations.)

In contrast, other task pairs do not have such high similarity. To see this, we also examined the correlations among other 7 tasks and found that they are generally weaker.

TaskMost Correlated TaskPearson rr
μ\muLUMO-0.395400
α\alphaGG-0.840287
HOMOα\alpha0.241924
LUMOZPVE0.651245
⟨R2⟩\langle R^2\rangleCvC_v0.693518
ZPVEHH-0.967417
CvC_vU0U_0-0.770228

These two tables may help explain why higher preference weights have more distinct effects on individual tasks, whereas the four highly correlated tasks exhibit similar performance patterns.

Response to Q1

please see the reponse to the weakness 1.

Response to Q2

Thanks for your question. For Figures 1 and 2, we use a numerical treatment log⁡(li−lmin⁡+ϵ)\log(l_i-l_{\min}+\epsilon) where lminl_{min} is set -2 and -20 for two tasks respectively, and ϵ\epsilon is set 1e-12 to avoid log⁡0\log0. This implementation is similar to that used in the FAMO repository. We have corrected the notation O(a3)O(a^3) in Remark 1 of the revised manuscript. Thank you for catching this typo.

Response to Q3

Thank you for raising this question. We'd like to further clarify the special cases of α\alpha as follows:

When α=1\alpha=1, (1) recovers the Nash-MTL (Navon et al., 2022) objective in the limit, i.e., arg⁡max⁡d∈Bϵ∑ilog⁡(gi⊤d)\arg\max_{d\in B_\epsilon}\sum_i\log(g_i^\top d), as mentioned it in the Introduction (lines 43-44 of the original manuscript). We have clarified the special objectives corresponding to α=0,1,+∞\alpha=0,1,+\infty in the revised manuscript after giving (1). Besides, we add sentences to clarify that (2) applies only when α≠0\alpha\neq0, while for α=0\alpha=0 we have w=1\mathbf{w}=\textbf{1} corresponding to LS. We have corrected the typo: Gt=(g1(θt)⊤,⋯ ,gK(θt)⊤)⊤∈Rd×K\mathbf{G}_t=(\mathbf{g}_1(\theta_t)^\top,\cdots,\mathbf{g}_K(\theta_t)^\top)^\top\in\mathbb{R}^{d\times K}.

Response to Q4

We have revised the wording “surpassing prior approaches by a large margin” in line 389-391 of the original manuscript to "outperforming the strongest baseline LDC-MTL by 0.32 and 0.30 percentage points, respectively", in the revised manuscript.

Response to Q5

MR for CelebA was previously omitted because we could not find the F1 scores for all 40 tasks reported by the other methods in Table 1. We got them in the rebuttal stage and now add it in the revised manuscript and show it in the following table. It can be observed that ROWA achieves the lowest MR. We have added the MR values in Table 1.

ROWAMGDAPCGradCAGradNash-MTLFairGradLSSIRLWDWAUWFAMOGo4AlignLDC-MTL
MR ↓2.9513.139.008.507.507.508.7010.207.839.288.104.554.453.33

➡️ Continue to the next post.

Response to Q6

Due to the usage of the log⁡li(θt+1)\log l_i(\theta_{t+1}) in the weight adaptation, ROWA introduces a second forward pass. We provide the following table, to show that the additional overhead mainly comes from the second forward pass.

BenchmarksBath sizeAdditional forward time (ms/batch)Weight update time (ms)
Cityscapes (2-task)8200.40.5
NYU-v2 (3-task)261.580.48
QM9 (11-task)12013.440.47
CelebA (40-task)25614.441.21

Besides, the additional forward pass takes longer on Cityscapes and NYUv2 than on QM9 and CelebA, leading to bigger 1.3x and 1.22x. It implies that the training time is influenced by different model architecture and input resolution, rather than task number alone. We have revised the "negligible" in the abstract to "low".

Comments on the paper

1 comment

Dear Reviewers,

Thank you so much for your time and efforts! As the discussion period ends soon, we would appreciate it if you could review our responses and additional experimental results. We're more than happy to clarify anything further.

Best regards,
The Authors of Submission 28375

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.