acceptodds
Under review as a conference paper at ICLR 2027

Learning at Different Speeds: Equilibrium Routing for AI-Driven Science

Abstract

AI for Science increasingly relies on single models to predict multiple observables of the same physical system. However, shared physics does not guarantee synchronized learning: one target may reach its best validation stage early and overfit under continued training, while another still requires optimization. We term this failure mode asynchronous convergence. We introduce Task-Aware Equilibrium Routing Mixture-of-Experts (TAER-MoE), a convergence-aware MoE framework that routes by generalization. TAER-MoE does not treat large loss or high expert affinity as sufficient evidence for assigning more capacity. Instead, it estimates task-level learning potential from validation descent and route-level overfitting from expert-task generalization gaps. A spectral bias–variance perspective links asynchronous convergence to task-dependent physical spectra and different optimal stopping times. Guided by this perspective, TAER-MoE uses an entropy-regularized equilibrium router to suppress expert-task routes with large train–validation gaps and reallocate training pressure toward routes where additional optimization remains useful. Across tropical cyclone forecasting, QM9 molecular property prediction, and a transverse-field Ising model benchmark, TAER-MoE improves joint test prediction over single-task, shared, MoE-based, gradient-based, and ensemble baselines. These results provide empirical evidence that validation-aware routing can improve scientific multi-task learning when coupled observables converge asynchronously.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Scientific Multi-Task Learning, Mixture-of-Experts Routing, Asynchronous Convergence

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4dBbOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies multi-task learning when related targets converge at different rates, so continued training may help one target while overfitting another. It proposes an MoE controller that uses recent validation improvement to adjust task loss weights and expert-specific train–validation gaps to penalize routing toward potentially overfitting experts. A spectral optimal-stopping analysis motivates the approach, and experiments on tropical cyclone forecasting, molecular property prediction, and an Ising-model benchmark report improved joint prediction.

Critical strengths

This paper has a creative idea to use learning speed as a proxy for how to weigh losses and routing decisions for MOEs. The intuition is that learning speed can be used to weigh the total loss. Its proposed controller is simple to integrate into an MoE, and experiments across three scientific domains provide evidence of improved joint prediction.

Critical weaknesses

The critical weaknesses of the paper are that

  1. Stop grads do not eliminate statistical dependence between val data and the learned weight. The eval statistics are used as a controller for the training objective. This means that the evals are no longer true validation data. This is ok if the held out data is never used, but this could also be interpreted as a change in the data distribution weights.
  2. The ablations do not clearly establish whether the gains come from the proposed routing mechanism rather than simpler validation-driven task weighting or the additional supervision supplied by validation feedback.
  3. Theory is underspecified: A power-law feature spectrum alone does not determine optimal stopping. In general target coefficients, noise, sample size, and the learning setup can also matter.
  4. Equation (5) evaluates each expert separately through the task head, whereas Eq. (8) predicts from a weighted mixture of expert features. An expert that performs poorly alone can still contribute complementary information to the mixture.
Questions for authors
  1. If the task is currently improving, why does that imply it needs higher weight? Arguably tasks that have not made progress and are stalled may need more weight?
  2. Have you compared training without the MOE routing change and just with reweighting the loss? It is unclear if this is the primary phenomenon driving the gains.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 4dBb

A1. Clarification of Validation Feedback and Data Independence

Thank you for this comment. We apologize for the lack of clarity in the original presentation. We have revised Section 3 to clarify the data partitioning, controller updates, and information boundaries.

The validation set provides feedback for task weighting, expert routing, and checkpoint selection. We acknowledge that this introduces statistical dependence between validation data and the learned model. However, the independent test set remains strictly held out. Test labels are never used for parameter optimization, controller updates, or model selection.

Within each training interval, task weights and routing corrections remain fixed. Their gradients with respect to model parameters satisfy

\qquad \nabla_\theta\operatorname{stopgrad} \left(\widetilde G_k^{(i)}(c)\right)=0.$$ Consequently, the gradient of the training objective is $$\begin{aligned} \nabla_\theta\mathcal L_{\mathrm{TAER\text{-}MoE}} ={}& \sum_{i=1}^{N_{\mathrm{task}}} \alpha_i(c) \mathbb E_{\mathcal D_{\mathrm{train}}} \left[ \nabla_\theta \ell_i\left(\hat y^{(i)}(x),y^{(i)}\right) \right] \\ &+ \beta_{\mathrm{load}}\nabla_\theta\mathcal L_{\mathrm{load}} + \beta_{\mathrm{reg}}\nabla_\theta\mathcal L_{\mathrm{reg}}. \end{aligned}$$ The corrected routing remains differentiable with respect to the learned router and expert parameters. Validation feedback influences optimization through fixed task weights and routing penalties between checkpoints. Stop-gradient defines this backward computation and does not eliminate statistical dependence on validation data. The controller changes the relative contributions of tasks and expert routes to optimization. The underlying data splits and evaluation distribution remain unchanged. The selected model and controller state are frozen before evaluation on the independent test set. We have clarified this protocol in the revised manuscript and plan to publicly release the complete code and dataset to support reproducibility. ### A2. Ablation Study on Task Weighting and Expert-Level Routing Thank you for this comment. We have conducted additional controlled ablation experiments on tropical cyclone forecasting to isolate the contributions of task weighting and expert-level routing. All configurations use the same expert pool, data splits, training protocol, and five identical random seeds. The validation set is shared across all methods, with no additional validation samples introduced. We have also included the new ablation results in the revised manuscript. The results are summarized below. | Configuration | ADE (km) ↓ | Pressure (hPa) ↓ | Wind (m/s) ↓ | |---|---|---|---| | Routing correction only | 94.95 ± 0.22 | 5.60 ± 0.06 | 3.93 ± 0.03 | | Task weighting only | 92.15 ± 0.21 | 5.38 ± 0.08 | 3.73 ± 0.02 | | **TAER-MoE** | **56.88 ± 1.30** | **5.27 ± 0.03** | **3.67 ± 0.03** | The full TAER-MoE achieves the lowest errors across all three prediction targets. Compared with task weighting alone, incorporating expert-level routing reduces ADE by 38.3%, pressure MAE by 2.0%, and wind MAE by 1.6%. These results demonstrate that task weighting alone does not account for the observed improvements. Expert-level routing provides additional benefits by adjusting capacity allocation according to expert-task generalization gaps. Both mechanisms use the same validation data, and their combination improves joint prediction across targets with different convergence stages. The controlled comparison confirms that the additional gains arise from incorporating expert-level routing into the task-weighted MoE framework. ### A3. Theoretical Foundation of Optimal Stopping Thank you for this valuable comment. We have refined Theorem 1 in Section 3.1 to establish a more precise relationship between spectral properties and optimal training stages. The revised analysis incorporates the feature spectrum, target coefficients, noise variance, and sample size under a fixed-design gradient-flow setting. The optimal stopping time is explicitly derived as $$T_i^{\star}=\frac{1}{\lambda_{r_i}}\log\left(1+\frac{n\lambda_{r_i}(c_{r_i}^{(i,e,c)})^2}{\sigma_i^2}\right).$$ This result characterizes how spectral properties, target structure, and statistical uncertainty jointly determine the optimal training stage. We have further specified the conditions for the stage-ratio scaling and established the corresponding excess-risk bound under an explicit curvature condition. We have also reorganized Section 3 to clarify the theoretical analysis and its connection to the proposed method. These revisions strengthen the theoretical foundation of asynchronous convergence and provide a clearer motivation for adaptive task weighting and expert allocation.
AuthorsOpenReview ↗

Additional Responses to Reviewer 4dBb

A4. Clarification of Expert-Level Feedback and Mixture Prediction

Thank you for this insightful comment. We have clarified the distinction between expert-level feedback and mixture prediction in Section 3.

Equation (5) measures each expert's generalization gap through its individual prediction. This gap serves as an auxiliary routing penalty and does not directly determine the expert's contribution to the final prediction. Importantly, the penalty reflects the difference between training and validation errors, not the absolute prediction error of an individual expert.

Equation (7) combines this penalty with learned input-dependent expert affinities. Experts with larger normalized gaps receive lower relative routing weights, while the softmax allocation preserves their participation in the mixture. Equation (8) then combines the weighted expert features before the task head, allowing experts to contribute complementary information.

We have revised Section 3 to clarify these distinct roles. Expert-level feedback regulates the allocation of capacity according to generalization gaps, while the learned router and mixture prediction preserve input-dependent expert interactions.

A5. Rationale for Generalization-Guided Task Allocation

Thank you for this insightful question. We have clarified the rationale for task weighting in Section 3.

The central principle of TAER-MoE is to allocate training resources according to observed generalization gains. Sustained validation improvement provides evidence that additional optimization remains beneficial. Increasing the weight of such tasks directs more training pressure toward learning processes that continue to improve generalization.

A stalled task does not necessarily benefit from greater training pressure. Stagnation may indicate insufficient optimization or that the task has approached its optimal generalization stage. In the latter case, increasing its weight may reinforce noise and lead to overfitting. Our spectral analysis explains why tasks can reach their optimal stages at different times, depending on the feature spectrum, target coefficients, noise, and sample size. Training resources should therefore follow demonstrated generalization gains, not stagnation alone.

TAER-MoE implements this principle through smoothed validation improvement and entropy-regularized task allocation. According to Eq. (4), tasks showing sustained improvement receive higher relative weights. Stalled tasks receive lower relative weights when other tasks continue improving, while the positive lower bound maintains their training. If all tasks stagnate, allocation gradually returns toward uniform weights. If a stalled task resumes improving, its weight increases accordingly.

Expert-level routing extends this principle to capacity allocation within each task. Generalization-gap penalties reduce the relative allocation to routes showing larger train–validation discrepancies while preserving input-dependent expert affinities.

We have clarified these principles in the revised Section 3. The controller prioritizes observed generalization gains while maintaining continued optimization across all tasks.

Reviewer_4dBbOpenReview ↗

Response to Rebuttal

I thank the authors for their detailed responses to my questions. They clarified that the final held out validation data is never used when determining mixture weights, verified with an ablation study that their algorithm is stronger than merely tuning the weights of the data, improved the optimal stopping criteria. In light of these improvements, I will increase my score a bit.

AuthorsOpenReview ↗

Appreciation and Request for Further Reconsideration

Thank you for your positive feedback and willingness to reconsider your assessment. We sincerely appreciate your recognition of the additional experiments and theoretical clarifications. We hope these revisions have adequately addressed your concerns and would be grateful if you could consider updating your rating to a weak acceptance.

Reviewer_uQa6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TAER-MoE, a task-aware mixture-of-experts framework for scientific multi-task learning under asynchronous convergence. It adjusts task-level training weights based on validation loss progress and modifies expert routing using expert-task generalization gaps. The method is evaluated on tropical cyclone forecasting, QM9 molecular property prediction, and the transverse-field Ising model. The results show improvements over several single-task, multi-task, MoE-based, gradient-based, and ensemble baselines.

Critical strengths

The paper evaluates the proposed method across three scientific tasks with different characteristics, providing evidence of its potential applicability beyond a single domain.

The reported improvements and ablation studies are promising and provide some support for the proposed validation-aware task weighting and expert routing mechanisms.

Critical weaknesses

Reproducibility concerns. Although the paper provides the main equations, the practical implementation of the training controller and routing corrections is not sufficiently transparent in the main text. The lack of an identified code repository or implementation available for inspection makes it difficult to independently verify the reported improvements.

Theoretical justification and empirical mechanism need clarification. The theoretical analysis relies on a local spectral approximation rather than a global guarantee. The connection between this analysis, the proposed validation-based proxies, and the observed performance gains could be more clearly established. Further evidence would help demonstrate that the improvements specifically arise from the proposed mechanism.

Questions for authors

Could the authors provide anonymized code or an implementation for inspection during the review process? Is there a plan to publicly release the code?

How do the authors verify that the proposed validation-based task weights and expert-task routing corrections reliably capture asynchronous convergence and account for the observed performance improvements?

I am open to revising my rating upward or downward depending on the authors’ rebuttal and how satisfactorily these concerns are addressed.

4 comments
AuthorsOpenReview ↗

Response to Reviewer uQa6

A1. Implementation transparency and reproducibility

Thank you for highlighting the importance of reproducibility. We have released an anonymized GitHub repository at https://github.com/anonymous-learning-ICLR/learning-at-different-speed, including the tropical cyclone forecasting implementation, training controller, expert routing corrections, and configuration files specifying the key hyperparameters. Detailed implementation settings are also provided in the supplementary material.

To facilitate independent verification, we provide pretrained checkpoints, a representative data subset, and evaluation and visualization scripts. These resources allow reviewers to reproduce the reported predictions, evaluate model performance, and inspect the correspondence between the implementation and the mathematical formulation. We plan to publicly release the full dataset and code upon acceptance.

A2. Theoretical justification and reliability of the feedback mechanism

Thank you for this helpful suggestion. We have revised Section 3 to clarify the connection between the local spectral analysis, the validation-based feedback signals, and the resulting allocation mechanisms.

First, the revised theoretical analysis establishes how task-dependent spectral properties, target coefficients, noise, and sample size determine optimal training stages. Theorem 1 provides an explicit optimal stopping time under a fixed-design spectral model. Equation (2) further establishes a lower bound on the excess prediction error associated with a shared training horizon under the stated curvature condition. These results provide a theoretical motivation for adaptive allocation under asynchronous convergence.

Second, we have established explicit optimization objectives for both feedback mechanisms. Smoothed validation descent estimates recent task progress and determines task weights through a constrained entropy-regularized objective. Expert-task generalization gaps provide routing penalties through a corresponding entropy-regularized allocation objective. Both updates follow directly from their optimality conditions. The revised formulation also specifies the normalization constraints, gradient detachment, and controller update sequence.

Finally, we conducted additional controlled ablations on tropical cyclone forecasting. Task weighting alone achieves an ADE of 92.15 ± 0.21 km, while routing correction alone achieves 94.95 ± 0.22 km. Their combination reduces ADE to 56.88 ± 1.30 km, with consistent improvements in pressure and wind prediction. These results support the complementary contributions of task-level and expert-level feedback.

The revised analysis provides a precise theoretical motivation under explicit assumptions, while the ablation experiments establish the empirical contribution of each mechanism.

A3. Empirical Validation of Asynchronous Convergence and Feedback Mechanisms

Thank you for this comment. We have conducted additional ablation experiments to verify the contributions of validation-based task weighting and expert-level routing.

Figure 2 demonstrates asynchronous convergence in tropical cyclone forecasting. Track prediction reaches its optimal stage earlier, while intensity prediction continues to improve with further training. Similar differences in convergence stages are observed between energy and entanglement entropy prediction in the quantum benchmark.

To assess the effectiveness of the proposed feedback mechanisms, we conducted controlled ablations using the same expert pool and experimental settings. Task weighting alone achieves an ADE of 92.15 ± 0.21 km, while routing correction alone achieves 94.95 ± 0.22 km. Their combination reduces ADE to 56.88 ± 1.30 km, with consistent improvements in pressure and wind prediction.

These results support the complementary roles of the two mechanisms. Task-level validation descent regulates training pressure, while expert-task generalization gaps guide capacity allocation. Both signals guide continuous allocation without imposing hard stopping decisions. Their combination improves joint prediction across targets with different convergence stages.

Thank you for the detailed response and for releasing the anonymized implementation. The additional ablation studies and theoretical clarifications are helpful.

However, after inspecting the released repository, I noticed a potential inconsistency in the implementation of the expert-level routing correction. Although the model defines a set_tsi_control() interface, I could not find a corresponding call to this function in the provided train.py. The training loop updates the TSI controller using validation metrics, but the resulting routing signals do not appear to be passed to the generator. Since the routing correction is bypassed when the control coefficient remains zero, could the authors clarify whether this is an omission in the released training script or whether a different implementation was used to obtain the reported routing ablation results?

Additionally, it would be helpful to clarify how the expert-task generalization gaps are computed in the actual implementation, as I could not identify this computation in the released code.

These clarifications are important for assessing the reproducibility and the actual contribution of the proposed routing mechanism.

AuthorsOpenReview ↗

Additional Responses to Reviewer uQa6

Thank you for carefully inspecting the repository and identifying these inconsistencies. We apologize for mistakenly uploading an earlier experimental version of the training code. We have checked the original experiment records and confirmed that the reported experiments, including the routing ablations, used the feedback implementation now provided in the updated repository. The error occurred when selecting the files for upload and did not affect the reported results.

The current training path uses set_feedback(). After collecting training and validation statistics, train.py calls control.update(...) followed by control.apply(model). This passes the task weights and expert-task penalties to the model. The penalties modify the routing logits before softmax, as specified in Equation (7). The current training path uses the feedback controller. The legacy set_tsi_control() interface remains in the inherited model for compatibility but is not used for these updates. We have also changed the logging label to [FEEDBACK-UPDATE] to avoid confusion.

The expert-task generalization gaps are explicitly computed in paper_pipeline.py and taer_feedback.py. For each expert and task, individual prediction losses are weighted by the routing probabilities under the preceding controller state. The code aggregates these statistics separately over the training and validation sets, computes the validation-minus-training difference, and applies the median/MAD normalization in Equations (5)–(6). Both passes use the same model parameters and preceding controller state in evaluation mode without gradients. The updated task weights and routing penalties are then applied together and saved with the checkpoint.

We have updated the anonymized repository accordingly. Thank you for helping us identify and correct this release error.

We hope these clarifications and revisions have addressed your concerns, and we would be grateful if you could reconsider your assessment of our work.

AuthorsOpenReview ↗

Kind Follow-up on the Author Response

Dear Reviewer,

Thank you again for your careful review and constructive feedback. As the author-visible discussion period draws to a close, we would be grateful if you could let us know whether any concerns remain.

If our responses and additional experiments have adequately addressed your main concerns, we would sincerely appreciate your reconsideration of the assessment and rating.

Thank you very much for your time and consideration.

Reviewer_VGBuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies a challenge in scientific multitask learning: different targets converge at different rates, so some may begin to overfit while others still benefit from further training. The authors propose TAER-MoE, which adjusts task weights based on validation improvements and refines routing using each expert’s training–validation error gap. The aim is to allocate more training to tasks and experts that can still improve generalization. Experiments cover typhoon forecasting, molecular property prediction, and quantum system prediction.

Critical strengths
  1. The central idea is interesting. Rather than focusing solely on which tasks are harder or which experts currently perform best, the method considers whether allocating further training to a task or expert remains beneficial. Applying this perspective to expert routing is a promising contribution.
  2. The training curves illustrate how different targets converge at different stages, and experiments across three scientific domains provide some support for the method’s applicability. The typhoon and quantum system experiments also report results across multiple random seeds, providing stronger empirical validation in these settings.
Critical weaknesses
  1. Although the idea is valuable, the additional benefit of expert-level routing over existing methods that adapt to learning progress remains unclear. The paper discusses related approaches that adjust task weights based on learning progress but does not directly compare against them. The molecular and quantum experiments primarily use Shared and MMoE as baselines. It is therefore difficult to determine whether the gains mainly come from dynamic task weighting or whether adapting expert allocation provides a substantial additional benefit.
  2. The method relies heavily on validation feedback, but the stability of these signals is not sufficiently established. A temporary plateau in validation performance does not necessarily mean that a task no longer benefits from training, and an expert’s training–validation error gap may depend on the number and composition of its samples. Given the paper’s emphasis on scientific settings with limited data, I am particularly concerned about whether such fine-grained feedback is reliable with small validation sets and whether the gains consistently carry over to an independent test set after repeated use of validation feedback.
Questions for authors
  1. Under matched model sizes, training budgets, and access to validation information, how much does TAER-MoE improve over methods that adjust task weights based on learning progress? What advantage does joint training offer over single-task models when each selects its own best validation checkpoint?
  2. With the same set of experts, how do task weighting alone, routing correction alone, and their combination compare? Could these results establish the independent contribution of expert-level feedback and clarify the discrepancy between the weight changes shown in Figure 6 and the description in the text?
1 comment
AuthorsOpenReview ↗

Response to Reviewer VGBu

A1. The additional contribution of expert routing remains unclear

Thank you for this valuable comment. We have conducted ablation experiments on tropical cyclone forecasting under identical experimental settings.

Task weighting alone achieves an ADE of 92.15 ± 0.21 km, a pressure MAE of 5.38 ± 0.08 hPa, and a wind MAE of 3.73 ± 0.02 m/s. Expert-level routing alone achieves 94.95 ± 0.22 km, 5.60 ± 0.06 hPa, and 3.93 ± 0.03 m/s, respectively. Combining both mechanisms reduces these errors to 56.88 ± 1.30 km, 5.27 ± 0.03 hPa, and 3.67 ± 0.03 m/s.

These results demonstrate the complementary contributions of both mechanisms. Task weighting regulates optimization across targets with different convergence stages, while expert-level routing adjusts capacity allocation within each task. Their combination improves joint generalization across all three prediction targets.

A2. Stability and Generalization of Validation Feedback

Thank you for this comment. We have clarified the role of validation feedback and conducted additional experiments on the quantum benchmark using five matched random seeds.

Task-level validation descent regulates training pressure, while expert-task generalization gaps identify routes with limited generalization. Both signals guide continuous allocation without imposing hard stopping decisions. Validation descent is smoothed using an exponential moving average, and task weights remain bounded away from zero. Expert-level gaps are normalized using robust statistics. Both feedback signals are updated periodically and remain fixed between checkpoints.

We evaluated the method against ExcessMTL and IGBv1 on an independent test set using the same five random seeds. TAER-MoE achieves an energy MAE of 0.001635 ± 0.000203, compared with 0.003585 ± 0.001112 and 0.003297 ± 0.000434, respectively. Its critical entropy MAE is 0.100529 ± 0.003983, compared with 0.176062 and 0.172540. Critical entropy degradation is reduced to 24.35 ± 4.85%, compared with 42.99% and 44.32%.

The consistent improvements across independent test metrics support the generalization and stability of validation-guided allocation. Test labels are excluded from model training, controller updates, and checkpoint selection.

A3. Performance Comparison and Benefits of Joint Training

Thank you for this comment. TAER-MoE is motivated by the physical coupling between scientific observables. Joint training enables shared representation learning, while asynchronous convergence creates different optimal training stages across tasks.

On the quantum benchmark, TAER-MoE reduces energy MAE by 58.8% and 10.1% relative to Shared and MMoE, respectively. Critical entropy MAE decreases by 9.8% and 12.4%. Compared with the learning-progress-aware methods ExcessMTL and IGBv1, TAER-MoE reduces energy MAE by 54.4% and 50.4%, respectively.

We trained separate single-task models for energy and entanglement entropy prediction. Each model independently selects its best validation checkpoint. Using the same five random seeds, the single-task models achieve an energy MAE of 0.001670 ± 0.000075 and a critical entropy MAE of 0.173630 ± 0.001426. TAER-MoE reduces these errors by 2.1% and 42.1%, respectively.

These results support the benefit of joint learning across physically coupled observables. TAER-MoE combines shared representation learning with adaptive task weighting and expert allocation, improving joint prediction across targets with different convergence stages.

A4. Ablation Study on Task Weighting and Expert-Level Routing

Thank you for this comment. We have conducted additional ablation experiments on tropical cyclone forecasting to evaluate the independent contributions of task weighting and expert-level routing. All configurations use the same expert pool and experimental settings.

The results are summarized below.

ConfigurationADE (km) ↓Pressure (hPa) ↓Wind (m/s) ↓
Routing correction only94.95 ± 0.225.60 ± 0.063.93 ± 0.03
Task weighting only92.15 ± 0.215.38 ± 0.083.73 ± 0.02
TAER-MoE56.88 ± 1.305.27 ± 0.033.67 ± 0.03

The full TAER-MoE achieves the lowest errors across all three targets. Compared with task weighting alone, incorporating expert-level routing reduces ADE by 38.3%, pressure MAE by 2.0%, and wind MAE by 1.6%.

These results demonstrate the complementary contributions of both mechanisms. Task weighting regulates optimization across targets with different convergence stages. Expert-level feedback adjusts capacity allocation within each task according to generalization gaps. Their combination improves joint prediction accuracy.

We also identified an inconsistency between the original Figure 6 and the updated controller formulation. We have removed Figure 6 and replaced the corresponding analysis with the new ablation results. This revision ensures consistency between the mathematical formulation and the experimental evidence.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Revisions

We thank the Area Chair and reviewers for their careful evaluation and constructive feedback. Their comments have helped us strengthen the theoretical clarity, empirical validation, and reproducibility of our work.

Our paper investigates asynchronous convergence in scientific multitask learning, where physically coupled observables reach their optimal generalization stages at different times. We propose TAER-MoE, a generalization-aware mixture-of-experts framework that allocates training resources according to observed validation progress and expert-task generalization gaps. A local spectral analysis motivates the learning-stage mismatch, while entropy-regularized allocation objectives define the task-weighting and expert-routing mechanisms. Experiments across tropical cyclone forecasting, quantum many-body prediction, and molecular property prediction demonstrate the effectiveness of this framework.

In response to the reviewers' concerns, we have made several substantive improvements.

Empirical validation. We added controlled ablations using the same expert pool, experimental settings, and five random seeds shared across all configurations. Combining task weighting and expert-level routing reduces tropical cyclone track ADE by 38.3% compared with task weighting alone. We also added ExcessMTL, IGBv1, and independently checkpointed single-task baselines to the quantum benchmark, using the same five random seeds across all methods. These experiments demonstrate the complementary contributions of both mechanisms and the benefits of joint learning.

Theoretical clarification. We reorganized Section 3 and refined the optimal-stopping analysis to explicitly incorporate spectral properties, target coefficients, noise, and sample size. We derived constrained entropy-regularized allocation objectives and clarified their correspondence with the implemented controller updates. We also specified the information boundaries between validation feedback, gradient computation, and independent test evaluation.

Reproducibility. We provided an implementation with training code, controller updates, routing corrections, pretrained checkpoints, evaluation scripts, and visualization tools. Following further reviewer inspection, we corrected an earlier code-upload error and clarified the implementation of expert-task generalization gaps. We verified that the reported experiments used the updated feedback implementation.

These revisions have strengthened the connection between our theoretical motivation, practical method, and empirical evidence. We appreciate the reviewers' engagement and hope the revised manuscript provides a clear basis for assessing the contribution of generalization-aware capacity allocation to scientific multitask learning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.