acceptodds
Under review as a conference paper at ICLR 2027

AMIYC: Add Me if You Can Conditional Adaptation Headroom and Recovery in Evolving MMoEs

Abstract

Multi-task mixture-of-experts (MMoE) models are commonly evaluated through routing behavior and end-to-end task quality, but neither establishes whether a trained state has exhausted the task-conditioned improvement available within its learned representation. We call this measurable opportunity conditional adaptation headroom. We introduce AMIYC, a framework that separates routing exposure from the task-visible expert state and distinguishes fresh adaptation opportunity, compatibility of earlier interventions, and recovery from a finite action bank. Across Amazon Reviews (Beauty, Grocery, Sports, and Toys), KuaiRand-Pure, and a production e-commerce ranking MMoE, bounded interventions reveal positive headroom on the tested cohorts, including near validation-selected checkpoints and when the same intervention coordinates participate in ordinary training. Alternative adapters show that the reference diagonal is not unique. As representations evolve, earlier operators may retain value, become harmful, or recover usefulness, while tested transport corrections do not consistently improve replay. Targeted Amazon studies further reveal reversals in checkpoint ordering after common-anchor fitting and associate a tested coupled adaptation schedule’s deficit predominantly with its base trajectory. Realizing this opportunity remains constrained: most final raw gain in the tested continuation is inherited from the historical anchor, affine calibration substantially reduces the apparent loss advantage, and useful ranking improvements remain selective. Finite-bank availability, action selection, and secondary-quality constraints further limit recovery. Fresh-seed confirmation reuses previously examined data. Together, these findings distinguish bounded, state-relative opportunity from compatibility and operational recoverability, without implying adaptation saturation or a universal remedy.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
mixture-of-experts, multi-task learning, model adaptation, conditional adaptation, transferability, optimization dynamics, representation dynamics, parameter-efficient adaptation, recommender systems

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_M2yNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the effectiveness of lightweight adaptation and compares it with base model training for Multi-task Mixture of Experts Settings. The paper tests multiple combinations of using adaptation during base MMoE model training, including, along with base model training, reusing the adaptation learned from previous checkpoints and not using an adapter at all. The paper also observes that the ranking among the adaptation methods reverses after separately adapting on the frozen MMoE after base model training. The paper also provides a method to predict whether an adaptation would be valuable compared to deferred adaptation.

Critical strengths

The idea of predicting headroom for adaptation and deciding between continual base model training and choosing to adapt is interesting. The paper provides comprehensive empirical analysis on multiple datasets and the chosen training setting.

Critical weaknesses
  • The confounding effect of optimization, including a higher frequency of parameter updates and the choice of learning rate and optimizer, is not well studied. This affects the validity of the conclusions made in the paper. Selecting checkpoints based on validation loss is understandable, but we need to make sure that the base model has sufficiently converged and that we can no longer obtain a lower validation loss. Please provide evidence for this. The frequency of parameter updates mentioned in the paper is too short.
    • This affects the conclusion regarding the availability of headroom as well as the dynamic performance change between different adaptation methods and rank reversal.
  • The paper has significant clarity issues, including excessive use of unnecessary jargon and not defining terminology and notations before use. A list of examples is provided below, but such issues are present throughout the paper. For a good paper, readers should not be required to guess the definition of terms and phrases used in almost every paragraph.
    • There are no citations for the statement in the first paragraph of the introduction. Please provide relevant citations. Also, please define or elaborate on these terms and phrases: "shared model", "accessible improvement", "retain accessible improvement near a validation-selected checkpoint", "earlier operator" (what is an operator?).
    • Please define or elaborate on these terms/phrases:
      • Lines 50- 53: "recovery resources", "accessible adaptation", "divergence from reuse" (reuse of what?), "recovery" (recovery of what?), "coordinates", "feedback into the base", "inherited information".
      • Lines 84-90: "task-visible", "the task-visible additive expert boundary", "prespecified bounded intervention", "each readout" (readout of what?), "fresh current-state fitting", "earlier operator", "intervention family, budget, and information access."
      • Lines 91-98: "tested cohorts", lines 93-95 ("It persists near high-quality ..."), "New-candidate",
      • Lines 98-105: "Native checkpoint", "deferred, crossed, and shadow controls associate a tested coupled schedule’s deficit predominantly with its base trajectory."
      • Lines 105-113: "state-relative adaptation value", "its construct validity", "opportunity from compatibility", "transport, and controlled trajectory comparisons", "selection, acquisition cost, calibration", "secondary-quality", "environments", "cross-domain measurements", " controlled trajectories"
      • Equations 1 and 2: many notations are not properly defined: e,u,g,ye,u,g,y.

Thus, the paper needs significant revision before it can be accepted to a high-impact venue such as ICLR.

Questions for authors

Please refer to the weakness section.

Reviewer_yf1NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces AMIYC (Add Me If You Can), an empirical framework for studying task-specific adaptation in evolving multi-task mixture-of-experts (MMoE) models. The paper distinguishes three concepts: adaptation opportunity, measured by fitting a bounded task-specific intervention at a frozen model state; compatibility, measured by replaying an intervention learned at an earlier model state; and recovery, measured by selecting useful interventions from a finite historical action bank. In the public models, routing, experts, and task towers are frozen during intervention fitting, while a small bounded scaling adjustment is optimized at the task-visible expert boundary. Experiments on Amazon Reviews, KuaiRand-Pure, and a production ranking MMoE show that fresh current-state fitting can reduce held-out task loss, while previously fitted operators may retain value or become harmful as the underlying model evolves. The paper further studies frozen-bank selection, trajectory-dependent adaptation, common-anchor refitting, calibration, and secondary ranking-quality constraints. The results suggest that current-state adaptation opportunity, historical compatibility, and operational recoverability should be treated as distinct quantities rather than conflated into a single notion of adaptation quality.

Critical strengths

The paper studies a concrete problem in evolving MMoE systems: a task-specific correction learned at one checkpoint can become ineffective after the shared model continues training. This is a practically relevant issue for multi-task systems that are continuously updated.

The proposed intervention is relatively simple and localized. During fitting, the router, experts, and task tower are kept frozen, while only a bounded low-dimensional scaling at the expert contribution is optimized, making it easier to isolate the effect of the added correction from changes to the original model.

The paper evaluates the proposed fitting and replay procedure on Amazon Reviews, KuaiRand-Pure, and a production ranking system. Fresh fitting consistently reduces the primary held-out loss in the tested settings, providing empirical evidence that the frozen models still admit additional optimization under the proposed intervention.

Critical weaknesses

I am not convinced that Fresh loss reduction demonstrates useful task-specific adaptation. In Table 4, two-parameter affine calibration gives larger BCE gains than refitting. After calibrating both models, refitting has only a small advantage on the core cohort and no clear advantage on fresh seeds. It also fails the ranking constraint in 3/5 fresh seeds. These results do not sufficiently support the stronger interpretation in the abstract that Fresh response reflects unused task-specific capability “within the learned representation.”

The practical value of the recovery policy remains unclear. On Amazon, the 75.3% capture rate is relative to the finite-bank oracle; Fresh is not evaluated on the same panels. On Kuai, the bank captures only 25% of Fresh value and the selector yields slightly negative gain. These results do not establish when bank-based recovery is a useful alternative to refitting.

Questions for authors

Can the authors demonstrate a reliable benefit over affine calibration on fresh seeds, in both BCE and ranking quality?

Does the checkpoint-ordering reversal persist after matched affine calibration? If this has not been tested, please qualify the claim in the abstract.

Can the authors compare the recovery policy with Fresh on the same Amazon panels, reporting both gains and total fitting and selection costs?

On Kuai, how much of the poor recovery comes from missing useful bank actions versus selection error? Under what conditions should the policy be expected to work?

Reviewer_EUL6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Recommendation apps like online shopping or short-video feeds often use one model that predicts a few things at once, like if you'll click buy or watch next. its built from a couple shared expert sub-networks called MMoE. The paper asks 3 questions about a trained model:

  • Opportunity or headroom: if we add a tiny per task knob that can turn an experts contribution up or down, does the model actually still improve?
  • Compatability: if the model keeps training do knob settings fitted earlier still help or do they start hurting?
  • Recovery: can a lookup table of stored knob settings plus a predictor choose a good one without refitting?

REsults: there is a tiny bit of improvement, and old settings sometimes help and sometimes hurt. But honestly most of the improvement can be obtained by simply rescaling the models final scores for calibration, and the gains are tiny.

Critical strengths
  • a usefull conceptual seperation that is honestly reported. distinguishing oppurtunity, compatability and recovery (selection from a finite bank) is a really clean framing for maintaining adapters on evolving models. the negative results like production plus 5 same replay loses value in 3/3 lineages while fresh improves are actually reported rather than hidden. this matters alot for practioners deciding whether to reuse or refit adapters.

  • careful within study role seperation and an explicit affine control. the fit select eval separation and the authors own affine calibration control in table 4 let readers see that the affect is mostly just calibration.

Critical weaknesses
  • the central construct is dominated by calibration and its super tiny. a simple affine map gains core and fresh which is basicly the full refit gain in the tables. after calibration the residual is core and with a confidence interval spanning zero for fresh. raw headroom is just a tiny fraction of the original loss. so the papers headline existance result really says very little about representation level adaptation oppurtunity.

  • also the stastical evidence is kinda weak and largely exploratory. there are only a few independant units but dependant counts are presented prominently like they are huge. there are loads of contrasts without any multiplicity control and almost all of the fresh cohort eval rows overlap previously examined data. the production setting is architecture witheld. the stated violation garantee isnt even met since the upper bound is way over the limit. there are also some totally inexplainable loss scales where the values jump around crazily between different tables.

Questions for authors
  • i just wanted some clarification on a few things. if calibration is applied before measuring headroom like if the no op includes the fit only affine map is fresh headroom actually positive in all 3 settings and with what CIs? also what are the loss units in table 1a anyway because why is kuais binary loss no op 8.606 in table 1a but 0.606 in table 2a and how exactly is the amazon average 21.07.

  • another clarification is which findings were preregistred and which were just derived after inspecting outcomes. can u provide a confirmitory replication on untouched data with a multiplicity correction across the main contrasts. also how do results change with full ranking metrics instead of 19 sampled negative ndcg@10 from krichene and rendle 2020.

  • one suggestion would be to compare against continued fine tuning of the task tower and against peft rank 1 or rank r ia3 at equal budget given that a rank 1 adapter already beats the referance diagonal in table 1c.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.