acceptodds
Under review as a conference paper at ICLR 2027

ERCo: Evaluation-Guided Response Coordination for Multi-Task Learning

Abstract

Multi-task learning (MTL) exploits shared representations to learn multiple tasks jointly, but most MTL optimizers coordinate tasks through training losses or their gradients rather than the criteria used for final evaluation. Since losses and evaluation criteria can emphasize different error structures, improving or balancing task losses does not necessarily yield favorable evaluation trade-offs. To bridge this gap, we propose ERCo, an *Evaluation-Guided Response Coordination* framework that separates evaluation guidance from loss-based learning. ERCo aggregates normalized differentiable evaluation surrogates at the upper level while retaining dynamically weighted standard task losses at the lower level. At each iteration, an evaluation-guided primary model and a loss-only response model are updated under the same task weights. Their task-wise loss differences provide feedback for adapting task priorities without explicitly storing per-task parameter gradients. Extensive experiments demonstrate strong aggregate performance across diverse MTL benchmarks, with measured training efficiency comparable to simple loss-weighting baselines.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Multi-Task Learning, Loss-Metric Mismatch, Differentiable Surrogates, Task Weighting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fguTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ERCo, a MTL framework that addresses the mismatch between training losses and task-specific evaluation metrics. The task coordination is formulated as a bilevel problem, where the upper level optimizes the task weights via aggregated evaulation surrogates, while the lower level optimizes the model parameters using weighted task losses. To efficiently solve the problem, ERCo maintains two models: a primary model guided by both evaluation surrogate and training losses, and an auxiliary response smodel optimized using only training losses. The differences in task losses between the two models are used as appriximation of gradient alignment between individual task losses and evaluation surrogates, which is then used to update task weights. Therefore, it avoids computing and storing the per-task gradients. The experiments on supervised and reinforcement learning MT benchmarks demonstrate the effectiveness of the proposed method.

Critical strengths

The strengths are summarized below:

  1. The proposed response-based coordination is clear and intuitive.
  2. The emprirical study is extensive, covering both supervised and reinforcement learning benchmarks.
  3. The ablation studies provide useful evidence that response-based coordination offers additional benefits.
Critical weaknesses

The weaknesses are summarized below:

  1. The novelty is limited. It appears to combine several existing ideas, inclusing evaluation performance-aware task coordination PIVRG [1], and bilevel inspired formulation LDC-MTL [2].
  2. Appendix B.2 states that the response model and optimizer states are synchronized at the beginning of each epoch. However, the supplementary codes show that the synchronization is performed at each mini batch.
  3. ERCo avoids storing individual gradients for all tasks. But it also maintains an additional model and optimizer state and performs two backward passes, additional forward evaluations, and model/optimizer-state synchronization at every mini-batch in the released implementation. The reported 1.04× peak-memory overhead on CelebA in Figure 2 is plausible maybe due to the activation memory domindates the model state memory, as the CNN model used here is small. It does not necessarily imply the similar efficiency on larger models. Also for the 1.24x training-time overhead. It may be vague to interpret without separating GPU computation, data loading, and synchronization costs.
  4. Regarding the ablation, I would like to see a comparison with LDC-MTL augmented with the evaluation surrogate QQ, to clearly separate the benefits of such guidance from the bilevel-style formulation.

[1] Revisiting Fairness in Multitask Learning: A Performance-Driven Approach for Variance Reduction. https://cvpr.thecvf.com/virtual/2025/poster/35104.

[2] LDC-MTL: Balancing Multi-Task Learning through Scalable Loss Discrepancy Control. https://arxiv.org/abs/2502.08585.

Questions for authors

See above weaknesses.

2 comments
AuthorsOpenReview ↗

Author Response

Thank you for recognizing ERCo's response-based coordination mechanism and broad empirical evaluation. We address your concerns below. Revisions are highlighted in blue in the updated manuscript.

W1: Contribution beyond PIVRG and LDC-MTL

ERCo builds on performance-aware task coordination and bilevel optimization. Its main methodological contribution is a paired-response mechanism that translates evaluation guidance into task-priority updates to address the mismatch between training losses and evaluation criteria. We clarify its distinctions from PIVRG and LDC-MTL below.

PIVRG. PIVRG uses measured task performance to guide the aggregation of individual task-loss gradients, aiming to reduce performance imbalance. ERCo instead constructs a differentiable objective from evaluation criteria and incorporates its gradient into the primary-model update. It then compares the evaluation-guided model with a loss-only response model to obtain task-wise priority feedback from post-update loss differences, without explicitly computing individual task-parameter gradients.

LDC-MTL. LDC-MTL uses an upper-level objective to control discrepancies among task losses. ERCo instead uses normalized evaluation surrogates at the upper level and weighted standard losses at the lower level. Their coordination signals also differ: LDC-MTL compares losses across tasks, whereas ERCo compares each task's loss between evaluation-guided and loss-only model responses. Eq. (6) provides a local interpretation of these response differences in terms of task-loss gradient alignment with the evaluation gradient.

These distinctions concern both the coordination objective and the priority-update mechanism. They are particularly relevant when training losses and evaluation criteria emphasize different errors, as with absolute and relative depth errors. On Cityscapes, ERCo achieves a relative depth error of 17.79, compared with 26.79 for LDC-MTL and 27.87 for PIVRG. The evaluation-augmented LDC-MTL control in W4 further assesses the benefit of response-based coordination under the same evaluation guidance. We have expanded these comparisons in Appendix F (pp. 27–28).

W2: Synchronization and implementation

Thank you for identifying the inconsistency between the released code and the manuscript. The originally submitted code used iteration-wise synchronization, whereas the reported experiments used epoch-wise synchronization of both model parameters and optimizer states, as described in Appendix B.2. We apologize for this discrepancy and have corrected the code to match the reported experiments.

We also compared the two synchronization schedules on CelebA across five evaluation-guidance strengths. The table reports Δm\Delta m% (mean ± standard error).

ccEpoch-wise synchronizationIteration-wise synchronization
10−410^{-4}−4.766 ± 0.143−4.446 ± 0.056
10−310^{-3}−4.684 ± 0.134−4.377 ± 0.015
10−210^{-2}−4.890 ± 0.128−4.658 ± 0.062
10−110^{-1}−4.779 ± 0.101−4.504 ± 0.368
11−3.845 ± 0.181−2.510 ± 0.158

Epoch-wise synchronization achieves lower mean Δm\Delta m% across all five tested settings. We have included this comparison in Appendix D.7 (p. 23).

W3: Training time and memory

The reported 1.04× peak-memory and 1.24× training-time ratios were measured on CelebA with 40 tasks using epoch-wise synchronization. ERCo maintains an additional model and optimizer state and performs an additional aggregated backward pass and two no-gradient forward evaluations per mini-batch. These costs depend on model size and architecture, so the measured overhead may not extend to larger models.

ERCo avoids explicitly computing and storing per-task parameter gradients, using a constant number of aggregated backward passes with respect to the number of tasks. This distinguishes it from methods that explicitly manipulate individual task gradients, although it does not guarantee lower computational costs in general.

To clarify the runtime comparison, Appendix D.5 reports end-to-end training times on an NVIDIA A100, including model updates, forward evaluations, data loading, and synchronization. ERCo takes 50.042 seconds per epoch, compared with 40.382 seconds for linear scalarization and 754.683–944.430 seconds for the measured gradient-oriented baselines. These measurements demonstrate ERCo's computational efficiency in the tested CelebA setting, but do not establish similar overhead on larger architectures or isolate the costs of individual operations.

AuthorsOpenReview ↗

Author Response

W4: LDC-MTL with evaluation guidance

Following your suggestion, we augmented LDC-MTL's model update with the same evaluation objective QτQ_\tau, while retaining its original loss-discrepancy objective and task-weight update rule. We selected the guidance coefficient cc through a grid search and averaged the results over three random seeds.

MethodCelebA Δm\Delta m%\downarrowNYU-v2 Δm\Delta m%\downarrow
LDC-MTL−1.08−4.75
LDC-MTL+QτQ_\tau−1.52−6.61
ERCo−4.78−8.97

The detailed NYU-v2 results are:

MetricLDC-MTL+QτQ_\tauERCo
mIoU ↑\uparrow40.86441.098
Pixel accuracy ↑\uparrow66.21267.012
Depth AbsErr ↓\downarrow0.530400.54998
Depth RelErr ↓\downarrow0.217700.22487
Normal mean ↓\downarrow25.72024.762
Normal median ↓\downarrow18.71617.573
Within 11.25∘11.25^\circ ↑\uparrow31.33333.765
Within 22.5∘22.5^\circ ↑\uparrow58.16960.621
Within 30∘30^\circ ↑\uparrow69.40171.339

Adding evaluation guidance improves LDC-MTL's aggregate performance on both benchmarks. ERCo achieves further improvements of 3.26 and 2.37 percentage points in Δm\Delta m% on CelebA and NYU-v2, respectively. On NYU-v2, LDC-MTL+QτQ_\tau achieves lower depth errors, while ERCo performs better on both segmentation metrics and all five surface-normal criteria.

Together with the Q+LS and Q+FAMO controls in Table 4, these results support the additional benefit of response-based coordination beyond directly incorporating evaluation guidance into existing task-weighting methods.

We have included the aggregate comparison in Section 4.4 (Table 4, p. 9) and the detailed results in Appendix D.4 (Table 10, p. 22).

Reviewer_pqLROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ERCo (Evaluation-Guided Response Coordination), an MTL framework that incorporates differentiable evaluation surrogates into training to address loss–metric mismatch. ERCo maintains an evaluation-guided primary model and a loss-only response model, and uses their task-wise loss discrepancies to adapt task-priority weights without explicitly storing per-task gradients. The method is evaluated on CelebA, NYU-v2, Cityscapes, QM9, and Meta-World MT10, showing strong aggregate performance across diverse supervised and reinforcement-learning settings.

Critical strengths

S1. Well-motivated evaluation-aware perspective for MTL

The paper addresses a practically relevant limitation of conventional MTL optimization: standard task losses are convenient for training but may not align well with the criteria used for final evaluation. The proposed framework provides a clear way to incorporate differentiable surrogates of task-specific evaluation criteria while retaining standard task losses for stable model optimization. This separation between loss-based learning and evaluation-aware guidance is intuitive.

S2. Broad empirical evaluation across diverse MTL settings

The empirical study covers a diverse set of supervised and reinforcement-learning benchmarks, including CelebA, NYU-v2, Cityscapes, QM9, and Meta-World MT10. These benchmarks span many-task classification, dense prediction, molecular regression, and multi-task control, providing reasonably broad evidence that the proposed framework is not tied to a single task type or evaluation metric.

Critical weaknesses

W1. Limited Alignment Between Eq. (6) and the Practical Synchronization Scheme

Equation (6) provides a first-order interpretation when the primary and response models start from the same anchor. In the implementation, however, the two models are synchronized only at the beginning of each epoch and then evolve independently within the epoch. As a result, the response discrepancy measured later in the epoch may reflect not only the local interaction between the task-loss and evaluation-guidance directions, but also the accumulated difference between the two model states.

W2. Limited Isolation of the MTL-Specific Coordination Benefit

The paper shows that adding evaluation guidance QτQ_\tau to multi-task training improves performance, and that response-based priority feedback provides further gains over alternative weighting schemes. However, loss–metric mismatch is not specific to MTL, and the current experiments do not isolate how much of the improvement comes from the proposed task coordination mechanism versus the generic benefit of directly optimizing QτQ_\tau. A useful control would be to evaluate each task in a single-task setting with the same evaluation-guided objective, e.g., STL+QτSTL+Q_\tau, and compare it against standard STL and ERCo. If STL+QτSTL+Q_\tau already accounts for most of the gain, the improvement may primarily stem from metric-aware optimization; conversely, a substantial additional gain from ERCo would more directly support the value of evaluation-guided task coordination.

Questions for authors

Q1. Could the authors clarify whether the local interpretation in Eq. (6) can be extended to the case where the two branches start from nearby but different model states? For example, under smoothness assumptions, can the deviation from the gradient-alignment term be bounded in terms of ∣y−θ|y-\theta? Empirically reporting the within-epoch branch drift or the correlation between Δi\Delta_i and the direct gradient-alignment signal would also help justify the epoch-level synchronization scheme.

Q2. Could the authors provide or discuss a single-task control in which each task is trained with the same evaluation-guided objective, e.g., STL+QτSTL+Q_\tau? Such a comparison would help disentangle the generic benefit of metric-aware optimization from the additional benefit of the proposed multi-task coordination mechanism. In particular, how much of ERCo’s improvement remains beyond what can already be achieved by applying QτQ_\tau in a single-task setting?

3 comments
AuthorsOpenReview ↗

Author Response

Thank you for recognizing the motivation and empirical evaluation of ERCo. We address your concerns and questions below. Revisions are highlighted in blue in the updated manuscript.

W1/Q1: Local interpretation and epoch-wise synchronization

We agree that within-epoch response discrepancies reflect both local gradient interactions and accumulated branch differences. Eq. (6) considers paired gradient-descent steps from a common starting point. We extend this interpretation to distinct starting points under fixed task weights and LL-smooth task losses. Let r=∥y−θ∥2r=\|y-\theta\|_2 denote the distance between the two models. Theorem 1 establishes

\leq \bigl(\|\nabla L_i(y)\|_2+Lr\bigr)r+O(\eta^2).$$ The bound quantifies the additional approximation error due to branch separation. Synchronization sets $r=0$, recovering the common-starting-point approximation. We have added Theorem 1 in Section 3.3 (p. 5), with its proof and a bounded-gradient corollary in Appendix E.1–E.2 (p. 25). In practice, ERCo synchronizes both model parameters and Adam optimizer states at the beginning of each epoch. The two branches then evolve separately, and their measured task-loss differences determine priority updates. To examine the resulting branch drift, we recorded the relative parameter distance $r_{\mathrm{rel}}=\|y-\theta\|_2/\|y\|_2$ during an additional NYU-v2 run. | Measurement point | Absolute distance | Relative distance | | --- | --- | --- | | After synchronization | 0 | 0% | | Iteration 100 | 2.469 | 0.918% | | Iteration 200 | 3.264 | 1.214% | | Iteration 300 | 3.863 | 1.436% | | Iteration 398 (epoch end) | 4.436 | 1.649% | | After next synchronization | 0 | 0% | The end-of-epoch relative distances at epochs 1, 20, 100, 101, 120, and 200 were 6.548%, 3.567%, 3.204%, 1.679%, 1.649%, and 1.582%, respectively. These measurements illustrate how branch drift accumulates within an epoch and is reset by synchronization. We also compared epoch-wise and iteration-wise synchronization on CelebA across five guidance strengths. Epoch-wise synchronization achieved lower mean $\Delta m%$ in all tested settings (Appendix D.7, Table 14). The theoretical bound applies to gradient-descent responses, whereas the experiments use Adam. It therefore provides a local interpretation rather than a guarantee for the full training dynamics. We have reported the synchronization and drift diagnostics in Appendix D.7 (pp. 23–24). ### W2/Q2: Evaluation-guided single-task control Following your suggestion, we trained each NYU-v2 task independently using $L_i+\rho_n Q_i$, where $Q_i$ contains only the evaluation criteria for task $i$. We used the same guidance setting as ERCo ($c=0.01$) and repeated the experiments over three random seeds. | Metric | STL | STL+$Q$ | ERCo | | --- | --- | --- | --- | | mIoU $\uparrow$ | 38.30 | 40.80 | **41.10** | | Pixel accuracy $\uparrow$ | 63.76 | 65.48 | **67.01** | | Depth AbsErr $\downarrow$ | 0.6754 | 0.6325 | **0.5500** | | Depth RelErr $\downarrow$ | 0.2780 | 0.2558 | **0.2249** | | Normal mean $\downarrow$ | 25.01 | 25.34 | **24.76** | | Normal median $\downarrow$ | 19.21 | 17.59 | **17.57** | | Within $11.25^\circ$ $\uparrow$ | 30.14 | **34.06** | 33.76 | | Within $22.5^\circ$ $\uparrow$ | 57.20 | 60.37 | **60.62** | | Within $30^\circ$ $\uparrow$ | 69.15 | 70.57 | **71.34** | | $\Delta m% \downarrow$ | 0.00 | −5.69 | **−8.97** | STL+$Q$ improves aggregate performance over standard STL, demonstrating the benefit of evaluation guidance in single-task learning. ERCo achieves a further improvement of 3.28 percentage points in $\Delta m%$, with approximately 13.0% and 12.1% lower absolute and relative depth errors, respectively. It also improves both segmentation metrics, although STL+$Q$ performs slightly better at the $11.25^\circ$ normal threshold. Since STL+$Q$ trains tasks independently, this comparison reflects both parameter sharing and task coordination. To better isolate the coordination mechanism, we also compare ERCo with shared-model baselines using the same evaluation objective. As shown in Table 4, Q+LS and Q+FAMO achieve $\Delta m%$ values of −6.62 and −7.64, respectively, compared with −8.97 for ERCo. These results provide complementary evidence that response-based coordination offers benefits beyond directly adding evaluation guidance to conventional MTL weighting methods. We have added the single-task control to Section 4.4 (Table 4, p. 9), with detailed results in Appendix D.4 (Table 10, p. 22).

Thank you to the authors for the detailed rebuttal and additional clarifications. I have carefully reviewed the responses and revisions, and I believe my main concerns have been adequately addressed. Since my initial assessment was already positive, I will maintain my original score.

AuthorsOpenReview ↗

Author Response

Thank you for taking the time to review our response and the revisions. We’re glad that the additional clarifications addressed your main concerns, and we appreciate your constructive feedback.

Reviewer_jjwdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ERCo, an evaluation-guided response coordination framework for multi-task learning. ERCo combines differentiable evaluation surrogates with dynamically weighted task losses. It uses task-wise loss differences between a primary model and a response model to update task priorities. Experiments on five MTL benchmarks show strong aggregate performance.

Critical strengths
  1. Evaluation-guided task coordination provides a useful perspective on the loss–metric mismatch in MTL.

  2. The paper includes extensive comparisons across supervised MTL and multi-task reinforcement learning, with ablation and efficiency analyses.

Critical weaknesses
  1. Limited novelty. ERCo builds on existing evaluation-aware task weighting and bilevel optimization methods. Its additional contribution over Auto-λ, LDC-MTL, and MEHA remains unclear.

  2. Limited evaluation analysis. ERCo relies on differentiable evaluation surrogates and criterion-specific calibration. The effect of surrogate quality and calibration scales on task coordination is not sufficiently analyzed.

  3. Limited theoretical justification. Eq.6 provides a local first-order interpretation of response discrepancies. However, its connection to the actual task-priority updates with periodic synchronization remains unclear.

Questions for authors
  1. What are the main differences between ERCo and existing evaluation-aware bilevel MTL methods?

  2. How sensitive is ERCo to the choice of evaluation surrogates and calibration scales?

  3. How well does the local interpretation in Equation (6) explain the task-priority updates during training?

  4. What is the additional benefit of response-based task coordination over directly incorporating evaluation guidance into existing loss-weighting methods, beyond the performance gains in Table 4?

4 comments
AuthorsOpenReview ↗

Author Response

Thank you for recognizing the evaluation-aware perspective and broad empirical evaluation. We address your concerns and questions below. Revisions are highlighted in blue in the updated manuscript.

W1/Q1: Contribution beyond existing bilevel methods

ERCo addresses the loss–metric mismatch in multi-task learning by using differentiable evaluation surrogates at the upper level while retaining weighted task losses at the lower level. Its main methodological contribution is a response-based coordination mechanism that translates evaluation guidance into task-priority updates.

We clarify this contribution by comparing ERCo with three closely related methods.

Auto-λ. Auto-λ learns task weights through a validation-loss meta-objective, using a virtual training step and a finite-difference hypergradient approximation. ERCo instead incorporates evaluation guidance directly into the primary-model update and adapts task priorities using task-loss differences between evaluation-guided and loss-only model responses. The priority update holds both model states fixed, avoiding differentiation through the training steps.

LDC-MTL. LDC-MTL uses an upper-level objective to control discrepancies among task losses. ERCo instead defines its upper-level objective using evaluation criteria and updates priorities by comparing each task's loss across two model responses. Thus, LDC-MTL compares losses across tasks, whereas ERCo compares the same task's loss between evaluation-guided and loss-only learning. Equation (6) provides a local interpretation of these response differences in terms of task-loss gradient alignment with the evaluation gradient.

MEHA. ERCo builds on MEHA's auxiliary-response approach but uses the auxiliary model differently. MEHA tracks a proximal lower-level solution to approximate gradients of a Moreau-envelope penalty. ERCo instead maintains a loss-only reference model without proximal coupling, periodically synchronizes it with the primary model, and uses their task-wise loss differences to update priorities.

In summary, ERCo combines evaluation-guided model updates with response-based task coordination, allowing evaluation information to influence subsequent task-loss weighting. The comparisons using the same QτQ_{\tau} in Table 4 provide empirical evidence of the additional benefit over directly incorporating evaluation guidance into existing weighting methods. We have clarified these distinctions in Appendix F (pp. 27–28) and the coordination mechanism in Section 3.3 (pp. 5–6).

W2/Q2: Surrogate quality and calibration

Surrogate sensitivity on CelebA. We examined the sensitivity of ERCo to the soft F1 surrogate's sharpness across all 40 attributes. Specifically, we replaced the probabilities in the soft counts with pni(T)=sigmoid⁡(ani/T)p_{ni}^{(T)}=\operatorname{sigmoid}(a_{ni}/T), where ania_{ni} is the attribute logit. We compared the original setting (T=1T=1) with sharper (T=0.5T=0.5) and smoother (T=2T=2) surrogates, keeping c=0.1c=0.1 fixed.

F1-surrogate temperatureΔm\Delta m%\downarrow
T=1T=1−4.779±0.101-4.779\pm0.101
T=0.5T=0.5−4.748±0.151-4.748\pm0.151
T=2T=2−4.058±0.145-4.058\pm0.145

The sharper surrogate performs comparably to the original, while the smoother surrogate reduces performance. Both alternatives still outperform the compared baselines, indicating that ERCo remains effective across the tested surrogate temperatures.

Fixed calibration on QM9. QM9 contains 11 regression tasks with substantially different target scales. We use the training-target mean absolute deviation (MAD) as the fixed calibration scale sis_i. To examine its influence, we compare the final MAEs of two representative properties, ⟨R2⟩\langle R^2\rangle and HOMO, relative to their single-task learning (STL) references and calibration scales. The results are from a single training run.

PropertyMAE / STL MAEMAE / Calibration Scale
⟨R2⟩\langle R^2\rangle3.6780.00912
HOMO1.0150.13992

Although the MAE of ⟨R2⟩\langle R^2\rangle decreases from 35.64 to 1.85 during training, it remains substantially higher than its STL reference (0.503). However, its calibration-normalized error is much smaller than that of HOMO. This discrepancy illustrates how fixed calibration scales may underemphasize tasks with substantial remaining performance gaps relative to STL. ERCo nevertheless achieves the second-best aggregate performance on QM9. Investigating adaptive calibration is a direction for future work.

These analyses have been added to Section 4.6 (p. 10) and Appendix D.6 (p. 23).

AuthorsOpenReview ↗

Author Response

W3/Q3: Local interpretation and epoch-wise feedback

Equation (6) provides a local interpretation of response discrepancies for paired gradient-descent steps from a common starting point. We extend this result to distinct starting points. With fixed task weights and LL-smooth task losses, let r=∥y−θ∥2r=\|y-\theta\|_2 denote the distance between the two models. For the paired gradient-descent updates, Theorem 1 establishes

\leq \bigl(\|\nabla L_i(y)\|_2+Lr\bigr)r+O(\eta^2).$$ The bound quantifies the additional approximation error due to branch separation. Synchronization sets $r=0$, recovering the common-starting-point approximation. To examine the connection with task-priority updates, we recorded response discrepancies $\Delta_i$, gradient alignment $a_i=\langle\nabla L_i,\nabla Q_\tau\rangle$, and priority-update directions on the first training batch after synchronization at selected epochs of NYU-v2. Entries are ordered as segmentation, depth, and surface normal. Arrows indicate the logit-update directions determined by Eq. (10). | Epoch | Response discrepancies $\Delta_i$ | Gradient alignment $a_i$ | Priority updates | | ----- | --------------------------------- | ------------------------- | ---------------------------------- | | 1 | $(+0.0113,+0.0053,+0.0047)$ | $(-0.6026,4.2873,6.5429)$ | $(\downarrow,\uparrow,\uparrow)$ | | 20 | $(-0.0494,-0.0035,-0.0088)$ | $(0.7955,0.5768,0.1253)$ | $(\uparrow,\downarrow,\downarrow)$ | | 100 | $(-0.0462,-0.0081,-0.0031)$ | $(0.9965,0.2343,0.1051)$ | $(\uparrow,\uparrow,\downarrow)$ | | 101 | $(-0.0563,-0.0069,-0.0014)$ | $(2.6028,0.3210,0.0351)$ | $(\uparrow,\uparrow,\downarrow)$ | | 120 | $(-0.0234,-0.0172,-0.0044)$ | $(3.5159,0.8644,0.8556)$ | $(\uparrow,\uparrow,\downarrow)$ | | 200 | $(-0.0112,-0.0127,-0.0005)$ | $(0.4474,1.5776,0.0415)$ | $(\uparrow,\uparrow,\downarrow)$ | Across these observations, larger gradient-alignment values generally correspond to smaller response discrepancies, which favor higher task priorities relative to the weighted average. These results support the local interpretation that response-based coordination tends to favor task-loss directions aligned with evaluation improvement. The theoretical bound applies to gradient-descent responses, while the experiments use Adam and epoch-wise synchronization. Thus, the diagnostic illustrates the connection immediately after synchronization rather than establishing it throughout training. We have added Theorem 1 in Section 3.3 (p. 5), its proof and corollary in Appendix E.1–E.2 (p. 25), and synchronization and response diagnostics in Appendix D.7 (pp. 23–24). ### Q4: Additional benefit of response-based coordination Directly adding $Q_\tau$ to an existing weighting method provides evaluation guidance for model learning but retains the original task-weighting rule. ERCo additionally uses evaluation guidance to adapt task priorities, encouraging subsequent weighted-loss updates to align with the evaluation objective. **Why response differences are useful.** Consider a common starting point $y$, and let $g_i=\nabla L_i(y)$, $g_w=\sum_i\sigma_i(x)g_i$, and $g_Q=\nabla Q_\tau(y)$. An evaluation-guided gradient-descent step gives $$L_i(y^+)-L_i(y) =-\eta\langle g_i,g_w\rangle -\eta\rho\langle g_i,g_Q\rangle+O(\eta^2).$$ The loss change combines the effects of standard loss-based learning and evaluation guidance. ERCo separates their first-order contributions by comparing the evaluation-guided model with a loss-only response updated from the same starting point under identical task weights. By Eq. (6), $$\Delta_i=-\eta\rho\langle g_i,g_Q\rangle+O(\eta^2).$$ Thus, response differences provide task-wise feedback on local alignment with the evaluation gradient, without explicitly computing individual task gradients. **Evaluation-oriented task weighting.** Define $$A(x)=\left\langle\sum_i\sigma_i(x)g_i,g_Q\right\rangle.$$ For a small weighted-loss step, $$Q_\tau\left(y-\alpha\sum_i\sigma_i(x)g_i\right) =Q_\tau(y)-\alpha A(x)+O(\alpha^2).$$ Hence, larger $A(x)$ corresponds to greater first-order improvement in the evaluation objective. Combining Eqs. (6) and (9), with the model states held fixed during the priority update, gives $$d_x=\nabla_x\sum_i\sigma_i(x)\Delta_i =-\eta\rho\nabla_x A(x)+O(\eta^2).$$ The resulting priority-descent direction therefore approximately increases $A(x)$ for sufficiently small steps, favoring task-loss gradients aligned with evaluation improvement. This explains the additional role of response-based coordination: evaluation guidance influences not only the current model update but also the task weights used in subsequent learning. We have added this explanation to Section 3.3 (p. 6), with a detailed analysis in Appendix E.3 (p. 26).

Thank you for the response. Given the limited novelty compared to prior work and the lack of substantial theoretical contributions, I maintain my rating.

AuthorsOpenReview ↗

Author Response

Thank you for your follow-up. We understand your concerns about novelty and theoretical contributions.

Our work aims to address the loss–metric mismatch in MTL through a distinct evaluation-guided formulation and paired-response coordination mechanism. We have clarified how ERCo differs from Auto-λ\lambda, LDC-MTL, and MEHA in both its formulation and update mechanism (Appendix F). We have also provided additional experiments comparing methods under the same evaluation guidance to support the benefit of our coordination mechanism.

Regarding the theoretical contribution, we have added a branch-drift error bound and an analysis connecting response feedback to evaluation-oriented task weighting (Theorem 1 and Appendix E).

We would appreciate it if you could clarify which specific aspects of novelty or theoretical analysis remain insufficient. This would help us better understand and address your concerns.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Thank you to all reviewers for the constructive feedback. We have revised the manuscript, with changes highlighted in blue. We summarize the main additions below.

1. Methodological contribution

ERCo addresses the loss–metric mismatch in MTL through an evaluation-guided formulation and a paired-response coordination mechanism. It uses differentiable evaluation surrogates to guide the primary model while retaining standard task losses for learning. Under the same task weights, a loss-only response model provides a reference, and their task-wise loss differences guide subsequent priority updates.

Unlike performance-informed gradient aggregation in PIVRG or cross-task loss-discrepancy control in LDC-MTL, ERCo translates evaluation guidance into task-priority feedback through paired model responses. It also differs from Auto-λ\lambda's validation-loss hypergradients and MEHA's proximal response tracking. Appendix F compares their formulations and update mechanisms in detail.

2. Additional theoretical analysis

We have extended the common-anchor interpretation in Eq. (6) to distinct branch starting points. Under fixed task weights and LL-smooth task losses, Theorem 1 establishes

\leq \bigl(\|\nabla L_i(y)\|_2+Lr\bigr)r+C_i\eta^2,$$ where $r=\|y-\theta\|_2$ measures branch separation. This bound **quantifies the additional approximation error caused by branch drift** and recovers the common-anchor approximation after synchronization. Appendix E further connects response feedback to evaluation-oriented task weighting under local gradient-descent assumptions. Appendix D.7 provides synchronization and response diagnostics. ### 3. Additional experiments on evaluation guidance and coordination We have added two reviewer-requested controls. STL+$Q$ trains each task independently with its own evaluation guidance, while LDC-MTL+$Q$ adds the same evaluation objective to LDC-MTL without changing its task-weighting rule. The NYU-v2 aggregate results, together with existing shared-model controls, are: | Method | NYU-v2 $\Delta m\%$ ↓ | |---|---:| | STL+$Q$ | −5.69 | | $Q$+LS | −6.62 | | $Q$+FAMO | −7.64 | | LDC-MTL+$Q$ | −6.61 | | **ERCo** | **−8.97** | STL+$Q$ confirms that evaluation guidance also benefits independent task learning. The shared-model comparisons provide complementary evidence for **the additional aggregate benefit of response-based coordination** under the same evaluation guidance. On CelebA, ERCo achieves −4.78 compared with −1.52 for LDC-MTL+$Q$. Section 4.4 and Appendix D.4 provide detailed results. ### 4. Surrogate sensitivity, calibration, and efficiency We have examined soft-F1 surrogate sharpness on CelebA and analyzed the influence of calibration scales on QM9. These experiments characterize the sensitivity of evaluation guidance and the relationship between calibrated errors and STL-relative performance gaps (Section 4.6 and Appendix D.6). ERCo **avoids explicitly computing and storing individual task-parameter gradients** and uses a constant number of aggregated backward passes with respect to task count. On the 40-task CelebA benchmark, it requires 1.24\$\times\$ the runtime and 1.04\$\times\$ the peak memory of LS (Appendix D.5). Across the reported baselines, ERCo achieves the best aggregate performance on CelebA, NYU-v2, and Cityscapes, second-best on QM9, and joint-best mean success on MT10. These results, together with the additional controls and analysis, support the practical value of evaluation-guided response coordination for MTL. We thank the reviewers again for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.