- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies multi-task learning when related targets converge at different rates, so continued training may help one target while overfitting another. It proposes an MoE controller that uses recent validation improvement to adjust task loss weights and expert-specific train–validation gaps to penalize routing toward potentially overfitting experts. A spectral optimal-stopping analysis motivates the approach, and experiments on tropical cyclone forecasting, molecular property prediction, and an Ising-model benchmark report improved joint prediction.
Critical strengths
This paper has a creative idea to use learning speed as a proxy for how to weigh losses and routing decisions for MOEs. The intuition is that learning speed can be used to weigh the total loss. Its proposed controller is simple to integrate into an MoE, and experiments across three scientific domains provide evidence of improved joint prediction.
Critical weaknesses
The critical weaknesses of the paper are that
- Stop grads do not eliminate statistical dependence between val data and the learned weight. The eval statistics are used as a controller for the training objective. This means that the evals are no longer true validation data. This is ok if the held out data is never used, but this could also be interpreted as a change in the data distribution weights.
- The ablations do not clearly establish whether the gains come from the proposed routing mechanism rather than simpler validation-driven task weighting or the additional supervision supplied by validation feedback.
- Theory is underspecified: A power-law feature spectrum alone does not determine optimal stopping. In general target coefficients, noise, sample size, and the learning setup can also matter.
- Equation (5) evaluates each expert separately through the task head, whereas Eq. (8) predicts from a weighted mixture of expert features. An expert that performs poorly alone can still contribute complementary information to the mixture.
Questions for authors
- If the task is currently improving, why does that imply it needs higher weight? Arguably tasks that have not made progress and are stalled may need more weight?
- Have you compared training without the MOE routing change and just with reweighting the loss? It is unclear if this is the primary phenomenon driving the gains.
4 comments
Additional Responses to Reviewer 4dBb
A4. Clarification of Expert-Level Feedback and Mixture Prediction
Thank you for this insightful comment. We have clarified the distinction between expert-level feedback and mixture prediction in Section 3.
Equation (5) measures each expert's generalization gap through its individual prediction. This gap serves as an auxiliary routing penalty and does not directly determine the expert's contribution to the final prediction. Importantly, the penalty reflects the difference between training and validation errors, not the absolute prediction error of an individual expert.
Equation (7) combines this penalty with learned input-dependent expert affinities. Experts with larger normalized gaps receive lower relative routing weights, while the softmax allocation preserves their participation in the mixture. Equation (8) then combines the weighted expert features before the task head, allowing experts to contribute complementary information.
We have revised Section 3 to clarify these distinct roles. Expert-level feedback regulates the allocation of capacity according to generalization gaps, while the learned router and mixture prediction preserve input-dependent expert interactions.
A5. Rationale for Generalization-Guided Task Allocation
Thank you for this insightful question. We have clarified the rationale for task weighting in Section 3.
The central principle of TAER-MoE is to allocate training resources according to observed generalization gains. Sustained validation improvement provides evidence that additional optimization remains beneficial. Increasing the weight of such tasks directs more training pressure toward learning processes that continue to improve generalization.
A stalled task does not necessarily benefit from greater training pressure. Stagnation may indicate insufficient optimization or that the task has approached its optimal generalization stage. In the latter case, increasing its weight may reinforce noise and lead to overfitting. Our spectral analysis explains why tasks can reach their optimal stages at different times, depending on the feature spectrum, target coefficients, noise, and sample size. Training resources should therefore follow demonstrated generalization gains, not stagnation alone.
TAER-MoE implements this principle through smoothed validation improvement and entropy-regularized task allocation. According to Eq. (4), tasks showing sustained improvement receive higher relative weights. Stalled tasks receive lower relative weights when other tasks continue improving, while the positive lower bound maintains their training. If all tasks stagnate, allocation gradually returns toward uniform weights. If a stalled task resumes improving, its weight increases accordingly.
Expert-level routing extends this principle to capacity allocation within each task. Generalization-gap penalties reduce the relative allocation to routes showing larger train–validation discrepancies while preserving input-dependent expert affinities.
We have clarified these principles in the revised Section 3. The controller prioritizes observed generalization gains while maintaining continued optimization across all tasks.
Response to Rebuttal
I thank the authors for their detailed responses to my questions. They clarified that the final held out validation data is never used when determining mixture weights, verified with an ablation study that their algorithm is stronger than merely tuning the weights of the data, improved the optimal stopping criteria. In light of these improvements, I will increase my score a bit.
Appreciation and Request for Further Reconsideration
Thank you for your positive feedback and willingness to reconsider your assessment. We sincerely appreciate your recognition of the additional experiments and theoretical clarifications. We hope these revisions have adequately addressed your concerns and would be grateful if you could consider updating your rating to a weak acceptance.
Response to Reviewer 4dBb
A1. Clarification of Validation Feedback and Data Independence
Thank you for this comment. We apologize for the lack of clarity in the original presentation. We have revised Section 3 to clarify the data partitioning, controller updates, and information boundaries.
The validation set provides feedback for task weighting, expert routing, and checkpoint selection. We acknowledge that this introduces statistical dependence between validation data and the learned model. However, the independent test set remains strictly held out. Test labels are never used for parameter optimization, controller updates, or model selection.
Within each training interval, task weights and routing corrections remain fixed. Their gradients with respect to model parameters satisfy
\qquad \nabla_\theta\operatorname{stopgrad} \left(\widetilde G_k^{(i)}(c)\right)=0.$$ Consequently, the gradient of the training objective is $$\begin{aligned} \nabla_\theta\mathcal L_{\mathrm{TAER\text{-}MoE}} ={}& \sum_{i=1}^{N_{\mathrm{task}}} \alpha_i(c) \mathbb E_{\mathcal D_{\mathrm{train}}} \left[ \nabla_\theta \ell_i\left(\hat y^{(i)}(x),y^{(i)}\right) \right] \\ &+ \beta_{\mathrm{load}}\nabla_\theta\mathcal L_{\mathrm{load}} + \beta_{\mathrm{reg}}\nabla_\theta\mathcal L_{\mathrm{reg}}. \end{aligned}$$ The corrected routing remains differentiable with respect to the learned router and expert parameters. Validation feedback influences optimization through fixed task weights and routing penalties between checkpoints. Stop-gradient defines this backward computation and does not eliminate statistical dependence on validation data. The controller changes the relative contributions of tasks and expert routes to optimization. The underlying data splits and evaluation distribution remain unchanged. The selected model and controller state are frozen before evaluation on the independent test set. We have clarified this protocol in the revised manuscript and plan to publicly release the complete code and dataset to support reproducibility. ### A2. Ablation Study on Task Weighting and Expert-Level Routing Thank you for this comment. We have conducted additional controlled ablation experiments on tropical cyclone forecasting to isolate the contributions of task weighting and expert-level routing. All configurations use the same expert pool, data splits, training protocol, and five identical random seeds. The validation set is shared across all methods, with no additional validation samples introduced. We have also included the new ablation results in the revised manuscript. The results are summarized below. | Configuration | ADE (km) ↓ | Pressure (hPa) ↓ | Wind (m/s) ↓ | |---|---|---|---| | Routing correction only | 94.95 ± 0.22 | 5.60 ± 0.06 | 3.93 ± 0.03 | | Task weighting only | 92.15 ± 0.21 | 5.38 ± 0.08 | 3.73 ± 0.02 | | **TAER-MoE** | **56.88 ± 1.30** | **5.27 ± 0.03** | **3.67 ± 0.03** | The full TAER-MoE achieves the lowest errors across all three prediction targets. Compared with task weighting alone, incorporating expert-level routing reduces ADE by 38.3%, pressure MAE by 2.0%, and wind MAE by 1.6%. These results demonstrate that task weighting alone does not account for the observed improvements. Expert-level routing provides additional benefits by adjusting capacity allocation according to expert-task generalization gaps. Both mechanisms use the same validation data, and their combination improves joint prediction across targets with different convergence stages. The controlled comparison confirms that the additional gains arise from incorporating expert-level routing into the task-weighted MoE framework. ### A3. Theoretical Foundation of Optimal Stopping Thank you for this valuable comment. We have refined Theorem 1 in Section 3.1 to establish a more precise relationship between spectral properties and optimal training stages. The revised analysis incorporates the feature spectrum, target coefficients, noise variance, and sample size under a fixed-design gradient-flow setting. The optimal stopping time is explicitly derived as $$T_i^{\star}=\frac{1}{\lambda_{r_i}}\log\left(1+\frac{n\lambda_{r_i}(c_{r_i}^{(i,e,c)})^2}{\sigma_i^2}\right).$$ This result characterizes how spectral properties, target structure, and statistical uncertainty jointly determine the optimal training stage. We have further specified the conditions for the stage-ratio scaling and established the corresponding excess-risk bound under an explicit curvature condition. We have also reorganized Section 3 to clarify the theoretical analysis and its connection to the proposed method. These revisions strengthen the theoretical foundation of asynchronous convergence and provide a clearer motivation for adaptive task weighting and expert allocation.