- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ERCo, a MTL framework that addresses the mismatch between training losses and task-specific evaluation metrics. The task coordination is formulated as a bilevel problem, where the upper level optimizes the task weights via aggregated evaulation surrogates, while the lower level optimizes the model parameters using weighted task losses. To efficiently solve the problem, ERCo maintains two models: a primary model guided by both evaluation surrogate and training losses, and an auxiliary response smodel optimized using only training losses. The differences in task losses between the two models are used as appriximation of gradient alignment between individual task losses and evaluation surrogates, which is then used to update task weights. Therefore, it avoids computing and storing the per-task gradients. The experiments on supervised and reinforcement learning MT benchmarks demonstrate the effectiveness of the proposed method.
Critical strengths
The strengths are summarized below:
- The proposed response-based coordination is clear and intuitive.
- The emprirical study is extensive, covering both supervised and reinforcement learning benchmarks.
- The ablation studies provide useful evidence that response-based coordination offers additional benefits.
Critical weaknesses
The weaknesses are summarized below:
- The novelty is limited. It appears to combine several existing ideas, inclusing evaluation performance-aware task coordination PIVRG [1], and bilevel inspired formulation LDC-MTL [2].
- Appendix B.2 states that the response model and optimizer states are synchronized at the beginning of each epoch. However, the supplementary codes show that the synchronization is performed at each mini batch.
- ERCo avoids storing individual gradients for all tasks. But it also maintains an additional model and optimizer state and performs two backward passes, additional forward evaluations, and model/optimizer-state synchronization at every mini-batch in the released implementation. The reported 1.04× peak-memory overhead on CelebA in Figure 2 is plausible maybe due to the activation memory domindates the model state memory, as the CNN model used here is small. It does not necessarily imply the similar efficiency on larger models. Also for the 1.24x training-time overhead. It may be vague to interpret without separating GPU computation, data loading, and synchronization costs.
- Regarding the ablation, I would like to see a comparison with LDC-MTL augmented with the evaluation surrogate , to clearly separate the benefits of such guidance from the bilevel-style formulation.
[1] Revisiting Fairness in Multitask Learning: A Performance-Driven Approach for Variance Reduction. https://cvpr.thecvf.com/virtual/2025/poster/35104.
[2] LDC-MTL: Balancing Multi-Task Learning through Scalable Loss Discrepancy Control. https://arxiv.org/abs/2502.08585.
Questions for authors
See above weaknesses.
2 comments
Author Response
W4: LDC-MTL with evaluation guidance
Following your suggestion, we augmented LDC-MTL's model update with the same evaluation objective , while retaining its original loss-discrepancy objective and task-weight update rule. We selected the guidance coefficient through a grid search and averaged the results over three random seeds.
| Method | CelebA | NYU-v2 |
|---|---|---|
| LDC-MTL | −1.08 | −4.75 |
| LDC-MTL+ | −1.52 | −6.61 |
| ERCo | −4.78 | −8.97 |
The detailed NYU-v2 results are:
| Metric | LDC-MTL+ | ERCo |
|---|---|---|
| mIoU | 40.864 | 41.098 |
| Pixel accuracy | 66.212 | 67.012 |
| Depth AbsErr | 0.53040 | 0.54998 |
| Depth RelErr | 0.21770 | 0.22487 |
| Normal mean | 25.720 | 24.762 |
| Normal median | 18.716 | 17.573 |
| Within | 31.333 | 33.765 |
| Within | 58.169 | 60.621 |
| Within | 69.401 | 71.339 |
Adding evaluation guidance improves LDC-MTL's aggregate performance on both benchmarks. ERCo achieves further improvements of 3.26 and 2.37 percentage points in on CelebA and NYU-v2, respectively. On NYU-v2, LDC-MTL+ achieves lower depth errors, while ERCo performs better on both segmentation metrics and all five surface-normal criteria.
Together with the Q+LS and Q+FAMO controls in Table 4, these results support the additional benefit of response-based coordination beyond directly incorporating evaluation guidance into existing task-weighting methods.
We have included the aggregate comparison in Section 4.4 (Table 4, p. 9) and the detailed results in Appendix D.4 (Table 10, p. 22).
Author Response
Thank you for recognizing ERCo's response-based coordination mechanism and broad empirical evaluation. We address your concerns below. Revisions are highlighted in blue in the updated manuscript.
W1: Contribution beyond PIVRG and LDC-MTL
ERCo builds on performance-aware task coordination and bilevel optimization. Its main methodological contribution is a paired-response mechanism that translates evaluation guidance into task-priority updates to address the mismatch between training losses and evaluation criteria. We clarify its distinctions from PIVRG and LDC-MTL below.
PIVRG. PIVRG uses measured task performance to guide the aggregation of individual task-loss gradients, aiming to reduce performance imbalance. ERCo instead constructs a differentiable objective from evaluation criteria and incorporates its gradient into the primary-model update. It then compares the evaluation-guided model with a loss-only response model to obtain task-wise priority feedback from post-update loss differences, without explicitly computing individual task-parameter gradients.
LDC-MTL. LDC-MTL uses an upper-level objective to control discrepancies among task losses. ERCo instead uses normalized evaluation surrogates at the upper level and weighted standard losses at the lower level. Their coordination signals also differ: LDC-MTL compares losses across tasks, whereas ERCo compares each task's loss between evaluation-guided and loss-only model responses. Eq. (6) provides a local interpretation of these response differences in terms of task-loss gradient alignment with the evaluation gradient.
These distinctions concern both the coordination objective and the priority-update mechanism. They are particularly relevant when training losses and evaluation criteria emphasize different errors, as with absolute and relative depth errors. On Cityscapes, ERCo achieves a relative depth error of 17.79, compared with 26.79 for LDC-MTL and 27.87 for PIVRG. The evaluation-augmented LDC-MTL control in W4 further assesses the benefit of response-based coordination under the same evaluation guidance. We have expanded these comparisons in Appendix F (pp. 27–28).
W2: Synchronization and implementation
Thank you for identifying the inconsistency between the released code and the manuscript. The originally submitted code used iteration-wise synchronization, whereas the reported experiments used epoch-wise synchronization of both model parameters and optimizer states, as described in Appendix B.2. We apologize for this discrepancy and have corrected the code to match the reported experiments.
We also compared the two synchronization schedules on CelebA across five evaluation-guidance strengths. The table reports Δm (mean ± standard error).
Epoch-wise synchronization achieves lower mean Δm across all five tested settings. We have included this comparison in Appendix D.7 (p. 23).
W3: Training time and memory
The reported 1.04× peak-memory and 1.24× training-time ratios were measured on CelebA with 40 tasks using epoch-wise synchronization. ERCo maintains an additional model and optimizer state and performs an additional aggregated backward pass and two no-gradient forward evaluations per mini-batch. These costs depend on model size and architecture, so the measured overhead may not extend to larger models.
ERCo avoids explicitly computing and storing per-task parameter gradients, using a constant number of aggregated backward passes with respect to the number of tasks. This distinguishes it from methods that explicitly manipulate individual task gradients, although it does not guarantee lower computational costs in general.
To clarify the runtime comparison, Appendix D.5 reports end-to-end training times on an NVIDIA A100, including model updates, forward evaluations, data loading, and synchronization. ERCo takes 50.042 seconds per epoch, compared with 40.382 seconds for linear scalarization and 754.683–944.430 seconds for the measured gradient-oriented baselines. These measurements demonstrate ERCo's computational efficiency in the tested CelebA setting, but do not establish similar overhead on larger architectures or isolate the costs of individual operations.