acceptodds
Under review as a conference paper at ICLR 2027

Reset or Reuse? Understanding How Optimizer Memory Shapes Continual Mixture-of-Experts Learning

Abstract

Continual fine-tuning of mixture-of-experts (MoE) models, a paradigm that sequentially adapts a pre-trained MoE model to a stream of tasks, has become a popular method in lifelong model deployment because it acquires new capabilities without retraining from scratch. However, standard sequential fine-tuning carries optimizer state across task boundaries and updates the full parameter space, so cross-task interference from inherited optimization history and from heterogeneous routed experts and always-active modules is neither measured nor controlled, while existing methods instead estimate parameter importance at substantial storage cost. To address this, we investigate the plasticity–forgetting dynamics of continual MoE fine-tuning through boundary-state interventions and role-structured updates, finding that interference degrades both what a new task acquires and what earlier tasks retain, which motivates separating what to discard from what to keep. Accordingly, we propose MoRE, a structured optimizer-update scheme for moment reset and reuse that clears the optimizer's fast state at task boundaries, reuses accumulated moment history to adapt the effective learning rate, and allocates subsequent updates across granularities, from role-level gates to per-coordinate modulation. On a six-task sequence from the TRACE continual learning benchmark, MoRE averages on OLMoE-Base, versus for sequential fine-tuning (SeqFT) and for the parameter-regularization baseline EWC; on Qwen, it reaches , versus for SeqFT and for EWC; it requires no Fisher estimation pass, and reduces storage for retained coordinate history by relative to EWC's Fisher-plus-anchor state at equal coverage.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Mixture-of-Experts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ixcFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how optimizer memory and parameter roles affect acquisition and retention during continual fine-tuning of fixed-capacity MoE models. It proposes MoRE (Moment Reset and Reuse), which separates task-local optimizer state from retained history and applies role gates and history-conditioned update scaling. Experiments on TRACE with OLMoE and Qwen, and on KuaiRand recommendation streams, report performance and efficiency gains. However, the benefits of resetting or carrying optimizer state depend on the learning regime, and higher final scores do not always indicate less forgetting.

Critical strengths

1.The research question and findings have practical relevance: cross-task optimizer state and heterogeneous parameter roles jointly affect task acquisition and retention.

2.The method is relatively simple to implement and requires no additional inference modules.

3.The experiments cover different architectures and different types of data change.

Critical weaknesses

1.The connection between the control signals and the central motivation of cross-task interference is insufficiently established. The paper attributes interference to inherited optimizer history and heterogeneous parameter roles, but primarily attenuates updates based on update magnitudes, experts’ task-wise update shares, and second-moment magnitudes. These statistics reflect parameter update activity rather than directly measuring conflicts between new and previous tasks. For example, a module with larger updates is not necessarily more prone to forgetting, and an expert whose updates are concentrated on one task is not necessarily more important to that task.

2.The core design primarily consists of update scaling, while the mechanistic explanation and conceptual depth remain limited. Task-local state reset and historical second-moment reuse have already been explored in ContinualAdam. The additional contribution here mainly lies in introducing role-, expert-, and coordinate-level controls. However, expressing history through a “shared-plus-task-specific decomposition” of second moments is largely an algebraic reformulation of denominator scaling. This does not, by itself, explain why the selected historical statistics and scaling rules improve continual learning.

3.The existing ablations do not sufficiently establish the necessity of the individual designs or clearly attribute the performance gains to specific components.

Questions for authors

1.Can AU, experts’ task-wise update shares, and second-moment magnitudes predict performance degradation on previous tasks? Could the authors provide direct evidence that these signals identify cross-task interference, rather than merely reflecting parameter update activity?

2.What additional benefits do MoRE’s role-, expert-, and coordinate-level controls provide over ContinualAdam or standard group-wise learning-rate control? Could component-wise ablations under matched experimental conditions, together with controls that match the actual update magnitudes, establish the necessity of each component and the retained history itself?

3.Using the same model and training settings, could the authors vary task differences, task order, and sequence length to clarify when optimizer state should be reset or reused, and assess the robustness of these conclusions?

3 comments

Dear Reviewer ixcF,

Thanks for the careful review. Thank you for recognizing the practical value of optimizer-state control and the simplicity of our proposed method. Your question about what the signals actually predict led us to add direct functional tests alongside the component comparisons. These results are now in the revised paper.

Q1 (W1). Do the signals predict old-task degradation?

The added tests connect history to old-task loss: expert scores predict local sensitivity most clearly at cm-to-ds, and coordinate-history placement improves preservation at a fixed update magnitude. We measure old-task loss after equal-norm single-expert updates and correlate it with the pre-update AU score and maximum task share (Appendix C.3):

BoundaryAU scoreMaximum task share
Py150 → cm-0.0625+0.1875
cm → ds+0.4615+0.5385
ds → 20Min0.0000-0.0625

Entries are within-layer rank correlations with signed old-task loss change. Experts and predictors are fixed before outcomes; four training minibatches are repeated interventions per expert. The analyzed counts are 24/22/24 experts, and the AU score uses unnormalized task energy. The association is strongest at cm-to-ds and weak at the other two boundaries.

For second-moment history, Table 9 tests allocation at the Qwen cm-to-ds boundary with the same model and gradient, matching the candidate-update norm within each role:

History allocationOld-task loss changeNew-task loss change
None+0.0109-0.8634
Uniform+0.0243-0.8065
Permuted+0.0019-0.7716
Measured-0.0041-0.5793

At equal update magnitude, measured history is the only allocation that reduces old-task loss, while all four allocations improve the new task. Thus the historical assignment changes preservation even when step size is controlled. Table 10 tests expert allocation at matched realized update norms across three boundaries and shows that the old/new-task balance varies with the transition.

Q2 (W2, W3). What do the additional components contribute?

MoRE improves final mean over ContinualAdam by 2.91 points; matched component experiments identify the contributions of role gates, expert allocation and coordinate history. Table 6 covers all six orders of C-STANCE, cm and ds, with two seeds and matched starts, data and budgets:

MethodFinal meanBWT
Carry48.81-2.13
ContinualAdam47.12-2.34
MoRE-R (ours)49.75-0.85
MoRE (ours)50.02-0.93

MoRE gains 2.91 points over ContinualAdam. Appendix C.1 presents the algorithms side by side.

Qwen's matched pre-cm comparison isolates role gates and expert allocation: gates raise mean 51.50→53.78 with AU off; unnormalized AU gives 55.08. Global/within-layer permutation and layer means give 54.34/54.30/54.21 (Table 7). The frozen modules and scaling strength are unchanged: permutation tests the assignment of historical coefficients, while layer means test expert-specific allocation within a layer. Task-normalizing those coefficients instead gives 52.13, versus 54.29 after permutation. These comparisons identify which allocation works in this setting.

Coordinate history improves Qwen BWT from -1.51 to -0.11, with means 54.90/54.27 (Table 8). The role-matched controls in Q1 hold update magnitude fixed and change its placement, directly testing the value of history beyond step-size reduction.

Q3. When should state be reset or reused?

Reset improves the final mean in both cm/ds directions, while local probes reveal different acquisition/retention preferences at each transition. We test all six three-task orders. At the common cm/ds endpoints, reset/carry means are 48.21/46.82 for cm→ds and 53.61/50.83 for ds→cm. Adding a C-STANCE prefix gives 52.22/50.84 and 55.92/53.92, respectively (Appendix C.2). These compare two- and three-task streams with their respective global cosine horizons.

A separate development-gradient probe ranks carry versus reset before either update is applied. It correctly orders 5/6 old-task and 4/6 new-task single-step comparisons. For cm→ds, reset improves both; for ds→cm, carry better preserves ds while reset improves cm more. These local preferences complement the complete-stream results and the 30-pair recommendation study.

Our main six-task order matches Yu et al. (ICML 2026). Spurious Forgetting (Zheng et al., ICLR 2025) follows TRACE's eight-task order, and Any-SSR (Tong et al., ICCV 2025) evaluates two published eight-task orders. Appendix C.2 reports all six permutations of C-STANCE, NumGLUE-cm and ds.

Thank you for your response. I would like to keep my score.

My concern is that, in the context of full-scale training, it remains unclear how much additional benefit the historical allocation strategy in MoRE provides compared to a baseline that uses grouped learning rates matched in terms of update magnitude. Furthermore, the performance gains are marginal; a difference of just a few tenths of a percentage point could easily be attributed to factors like the choice of random seed, or other variables.

Dear Reviewer ixcF,

Thank you for continuing the discussion. We would like to clarify what history contributes beyond learning-rate adjustment, and why the full-scale accuracy–cost result matters.

1. Matching update magnitude does not determine what an update preserves.

In your follow-up, you ask how much additional benefit MoRE's historical allocation provides during full-scale training over “a baseline that uses grouped learning rates matched in terms of update magnitude.” Adam itself already adapts coordinate-wise updates using exponential moving averages of gradients and their squares. This adaptation, or a task-wise layerwise learning-rate schedule, does not automatically protect earlier-task knowledge: a large update on a later task can still alter parameters important to previous tasks. Group-level rate multipliers are a coarser control; without cross-task history, grouping itself supplies no criterion for protecting earlier knowledge. Our Adam+SeqFT baseline already uses adaptive step sizes, yet still exhibits forgetting on both OLMoE checkpoints. MoRE combines task-boundary optimizer-state control with history-dependent expert allocation to balance new-task learning and earlier-task preservation.

The revised Table 9 directly controls update magnitude: at the Qwen cm-to-ds transition, we use the same model and gradient and match the update norm within each role.

History allocationEarlier-task loss changeCurrent-task loss change
None+0.0109−0.8634
Uniform+0.0243−0.8065
Permuted+0.0019−0.7716
Measured−0.0041−0.5793

Measured history alone improves both tasks in the norm-matched single-step test.

2. The improvements extend beyond a few tenths of a point and are accompanied by substantial efficiency gains.

ModelEWC final meanMoRE final meanEWC training timeMoRE training time
OLMoE-Base48.3949.5310 h 50 min4 h 12 min
Qwen1.5-MoE54.6054.9020 h 49 min15 h 31 min

MoRE reduces training time by 61% on OLMoE and 25% on Qwen while reaching or exceeding EWC's observed final scores. Qwen's peak memory also falls from 74.7 to 65.2 GB per GPU. OLMoE's 49.53 approaches the separate-model independent-fine-tuning reference of 50.05 while retaining one continual model.

You also describe the gains as “a difference of just a few tenths of a percentage point.” This does not characterize the full set of results. MoRE improves over standard SeqFT across both model families, including gains of 3.90 points on OLMoE-Base and 2.92 points on Qwen:

ModelSeqFTEWCMoREGain over SeqFT
OLMoE-Base45.6348.3949.53+3.90
OLMoE-Instruct47.9948.2048.96+0.97
Qwen1.5-MoE51.9854.6054.90+2.92

The OLMoE MoRE results average two seeds. The additional six-order, two-seed comparison gives MoRE 50.02 versus ContinualAdam 47.12, with MoRE ahead in ten of the twelve matched runs (Table 6). The evidence therefore spans model families, repeated runs and task orders.

Reviewer_rZfaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether Adam's optimizer state should be reset or reused when an MoE language model is fine-tuned on a sequence of tasks. MoRE resets the moments at each task boundary and then scales down later updates with per-role gates, without changing the architecture or replaying data. On OLMoE it also uses a per-expert scale built from past updates. On the TRACE benchmark with OLMoE and Qwen1.5-MoE, it beats sequential fine-tuning and is slightly above EWC.

Critical strengths
  1. Tested on more than one model: base and instruction-tuned OLMoE, and Qwen1.5-MoE with shared experts.
  2. A light intervention. It only touches the optimizer state and the update size, and adds no new modules or stored training data.
Critical weaknesses
  1. The paper motivates the setting by avoiding retraining from scratch, but it is unclear why the experiments need continual learning. All tasks and their data are available, so merging the per-task models in full parameter space, or replaying part of the earlier task data, are natural alternatives. On TRACE, the replay baseline uses a general instruction corpus (LIMA), and merging is not tried. Independent fine-tuning is used as the reference, but multi-task training on all tasks is the usual single-model upper bound and is not reported.
  2. Both backbones are older MoE models, and I am not sure the gains carry over to newer ones with finer-grained experts. All results use one task order. The paper finds that resetting the moments helps on TRACE with OLMoE while carrying them helps on a recommendation stream. It links this to whether the objective changes at a task boundary. How the choice depends on task similarity is not tested within one benchmark, for example by placing related or unrelated tasks next to each other.
Questions for authors
  1. How does MoRE compare with multi-task training and with merging the independently fine-tuned models?
  2. Does the comparison change if replay uses earlier task data, at the same ratio as the LIMA replay?
  3. Do the results hold for a newer MoE model and for other task orders?
3 comments

Dear Reviewer rZfa,

Thank you for the thoughtful review. We appreciate you characterizing MoRE as a light intervention. This reflects our design goal exactly. We ran the joint-training, replay and model-merging comparisons you suggested and marked the results in blue in the revision.

Q1 (W1). Multi-task training and model merging.

MoRE reaches 49.45, above joint training (48.51) and TIES-Merging (47.98), on the same six TRACE test sets (Table 12):

Single-model methodFinal mean
Uniform specialist merge38.76
TIES-Merging47.98
Joint training48.51
MoRE (unit gates, ours)49.45

The six specialists average 50.14 on their own tasks, but uniform averaging drops to 38.76: it weights each task's parameter change by 1/6, and opposing changes can cancel. TIES-Merging (Yadav et al., NeurIPS 2023) trims small changes and reconciles signs, recovering 9.22 points. MoRE exceeds this stronger merge by 1.47 points while learning sequentially without retaining separate task models.

Both merging methods reuse the same specialists; MoRE uses task-normalized AU coefficients (Appendix A.2). TIES uses the global top-20% threshold and scale 1, fixed before evaluation. Joint training uses the shuffled union of six 5,000-example task sets for three epochs. We added the TIES result and this explanation to Table 12 and Appendix C.4.

Q2 (W1). Replay of actual earlier tasks.

MoRE achieves a higher final mean and BWT than both replay baselines. At the same replay ratio, earlier-task TRACE replay also improves on LIMA replay (Table 12):

MethodFinal meanBWT
LIMA replay47.09-2.93
Earlier-task TRACE replay47.33-2.21
MoRE (unit gates, ours)49.45+0.46

Both replay baselines start from the base model and replace 1% of training examples. They use the same total presentations, seed, batch size and evaluator; TRACE replay draws from earlier tasks, while LIMA supplies general instruction data. MoRE uses no replay buffer and follows the shared-MeetingBank-prefix protocol specified in Table 12.

Q3-1 (W2). Transfer to a newer model.

On Qwen3-30B-A3B, MoRE improves final mean by 7.12 points over MoRE-R. Table 6 compares both methods with zero-shot evaluation on C-STANCE, cm and ds:

MethodFinal mean
Zero-shot (pretrained)39.31
MoRE-R (ours)57.73
MoRE (ours)64.85

All three models are evaluated on identical task inputs with Qwen3's native non-thinking chat template, the same short-answer instruction and strict exact-match scoring. The trained pair also shares initialization, data and budget. Table 6 gives the task-wise scores, and Appendix C.1 describes the protocol.

Q3-2 (W2). Robustness to task order.

MoRE has the highest average score over all six task permutations and two seeds, and exceeds ContinualAdam in 10/12 paired settings (Table 6):

MethodFinal meanBWT
Carry48.81-2.13
ContinualAdam47.12-2.34
MoRE-R (ours)49.75-0.85
MoRE (ours)50.02-0.93

Our main six-task order matches Yu et al. (ICML 2026). Spurious Forgetting (Zheng et al., ICLR 2025) follows TRACE's eight-task order, and Any-SSR (Tong et al., ICCV 2025) evaluates two published eight-task orders. Appendix C.2 reports all six permutations of C-STANCE, cm and ds, plus two- versus three-task controls.

The paired order comparisons swap the first two tasks while keeping the final task, per-task data, budgets and role rules fixed. A development-gradient probe correctly predicts which state policy produces lower old-task loss after one update in 5/6 directed transitions. Together, the complete streams and local probes test state choice across all six directed task transitions.

Thank you for the response. The new results address some of my concerns.

  1. The comparisons with joint training, TIES-Merging and replay of earlier task data answer my questions. One question: MoRE is above joint training, which is usually the upper bound for a single model. Could you explain why?
  2. The six task orders answer my question on task order.
  3. On Qwen3-30B-A3B, MoRE is compared only with MoRE-R and zero-shot. Without sequential fine-tuning or EWC on this model, I cannot tell whether the gains over the baselines carry over.

Dear Reviewer rZfa,

Thank you for taking another close look at the revised results. We are glad that the additional joint-training, merging, replay and task-order experiments have helped address your concerns.

The key distinction is how the updates are controlled. Continual-learning methods commonly modify optimization to preserve earlier knowledge: EWC penalizes changes to parameters that are important for previous tasks, while MoRE controls optimizer state at task boundaries and modulates expert updates using accumulated history. Our joint baseline trains on mixed data from all tasks, with a shared cosine learning-rate schedule and the same number of epochs as MoRE, without these additional continual-learning mechanisms. It provides oracle access to the data, but this alone does not guarantee optimal learning and retention.

An optimization upper bound would require an oracle that selects the best task-sequence-aware, element-wise learning-rate schedule for the task stream within a fixed training budget. Searching over schedules for billions of parameters across tasks and training steps is prohibitively expensive. Obtaining effective control of this kind at low cost is itself an important continual-learning problem; MoRE addresses it through structured, history-dependent update modulation. We therefore compare with strong, computationally feasible baselines, including standard joint training and EWC. Joint training remains a useful reference, and outperforming that particular procedure is compatible with its advantage in data access.

We are also running an additional short-budget SeqFT baseline on Qwen3-30B-A3B to provide the conventional sequential-training comparison you requested. Training and evaluating the 30B model takes substantial time even with multiple GPUs, and we appreciate your patience. We will update the comparison table with the verified result and its training budget as soon as the run and evaluation are complete.

(Updated) We have completed the additional Qwen3-30B-A3B comparisons with ContinualAdam and SeqFT (LoRA, rank-16 attention adapter), alongside reduced-budget SeqFT (13.6% steps) as a reference:

MethodC-STANCEcmdsFinal mean
Reduced SeqFT35.6080.2539.3851.74
Zero-shot (pretrained)53.8529.6334.4639.31
SeqFT (LoRA)50.9051.8538.4647.07
ContinualAdam50.6562.968.6240.74
MoRE-R (ours)51.1076.5445.5457.73
MoRE (ours)49.0582.7262.7764.85

Reduced SeqFT favors support labels on C-STANCE despite stronger arithmetic performance than LoRA. MoRE achieves the highest final mean among the compared methods, alongside changes in C-STANCE performance. For example, a passage says “relations between two countries will weaken,” while the claim says “they will remain strong”: the pretrained model predicts opposition, but MoRE predicts neutrality. Likewise, a post calls a medical drama “unsuitable for clinical learning,” while the claim calls it suitable; MoRE incorrectly predicts support. These examples indicate that subsequent cm training may weaken sensitivity to these passage–claim mismatches while improving arithmetic performance. We will include full-budget SeqFT results in a future revision.

Reviewer_RTd8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates how optimizer-state carryover and heterogeneous parameter roles affect continual fine-tuning of mixture-of-experts models. It proposes MoRE, which combines task-boundary resets of Adam’s local moments with structured update modulation through role gates, expert-level accumulated-update statistics, and an optional coordinate-level second-moment history. Evaluation covers a six-task TRACE sequence using OLMoE-Base, OLMoE-Instruct, and Qwen, together with a chronological KuaiRand recommendation stream. The reported TRACE results improve over sequential fine-tuning and achieve final average scores comparable to or above EWC, with lower measured training costs. The recommendation experiments additionally show that retaining optimizer moments can outperform resetting them under distribution drift, while history-based modulation improves retention. The main contribution is an optimizer-level framework for controlling adaptation across parameter granularities without expanding the model or requiring replay, supported by task trajectories, configuration comparisons, and reset/reuse studies.

Critical strengths
  1. The paper examines a consequential and comparatively underexplored implementation choice in continual MoE adaptation: which optimizer states should persist across task boundaries. Its separation of acquisition from retention is useful, and the contrast between TRACE and recommendation drift appropriately cautions against treating boundary resets as universally beneficial. In particular, the 30-seed paired recommendation study shares the first-period checkpoint and subsequent minibatch order across intervention arms, providing substantially stronger evidence than isolated final-score comparisons for the usefulness of history reuse in that setting.

  2. The evaluated approach offers a potentially useful performance–cost trade-off without expanding the inference architecture or requiring replay. The reported OLMoE configuration takes 4 hours 12 minutes versus 10 hours 50 minutes for EWC, while Qwen uses 65.2 GB versus 74.7 GB peak GPU memory per rank. These are meaningful practical benefits relative to the implemented EWC baseline.

Critical weaknesses
  1. The connection between the proposed algorithm and the evaluated configurations is insufficiently specified.

Section 4.2 and Appendix A.2 describe an online gating policy determined by training dynamics, with unit gates during warm-up and subsequent values given by Eq. (7), explicitly without task-specific lookup rules. However, Appendix A.1 and Table 3 specify task-dependent schedules, including gates fixed at zero throughout several Qwen task segments. These descriptions are not equivalent as written, and the manuscript does not explain which policy generated the main results or how the prescribed schedules follow from the dynamic policy. Numerical settings for the dynamic policy are also missing. Furthermore, Appendix A.3 refers to a coordinate-history variant in Table 9, but that table does not identify which configuration enables coordinate history.

This ambiguity is consequential: it prevents determining whether the results validate an automatic, transferable update-allocation mechanism or a separately configured set of task-dependent schedules. It also makes the contribution of historical information difficult to assess, particularly because the main Qwen configuration is described as role-only.

  1. The incremental methodological contribution and its empirical benefit over closely related, simpler alternatives are not yet convincingly established.

MoRE does preserve historical statistics and can indirectly protect previously learned parameters. However, accumulated parameter displacement and squared-gradient activity are proxies whose relationship to functional importance for old tasks is not sufficiently validated. The effective-rate identities explain the update rule but do not establish a retention advantage. The available configuration comparisons also change multiple components together, leaving unclear whether the gains arise from informative history allocation, boundary resets, or a favorable reduction in effective update size.

The issue with the baselines is their relevance and comparability, rather than simply their age. Although MoFO and CKA-NAS are included, ContinualAdam—the closest optimizer-state reuse method—is not faithfully evaluated: retaining a continuously evolving second moment in the m-only reset arm is different from its task-size-weighted historical second-moment aggregation. MoFO uses a different shared-prefix protocol, and CKA-NAS changes model capacity, replay, and training budget. These comparisons therefore do not isolate the proposed optimizer-level contribution.

The remaining margins are also insufficiently characterized statistically. MoRE exceeds EWC by 1.14, 0.76, and 0.30 average-score points on OLMoE-Base, OLMoE-Instruct, and Qwen, respectively. Most main-table baselines and all Qwen results are single-run, while the two-run MoRE standard deviations on OLMoE are 0.63 and 0.89 points. This does not prove that the gains are noise, but it leaves their robustness and sensitivity to tuning unresolved. On Qwen, MoRE also has worse BWT than EWC (-1.51 versus -0.70), so the final-score improvement does not establish better retention.

Questions for authors
  1. Which gating policy was used in the main experiments, and how were its configurations selected?

The main text and Eq. (7) describe a policy that adjusts gates automatically from training dynamics, with gates fixed at one during warm-up. Table 3 instead specifies task-dependent configurations, including gates fixed at zero throughout certain task segments. Please clarify which policy produced the results in Tables 1 and 9, and how the prescribed configurations relate to the dynamic policy. Do these settings require separate selection for each task? Appendix A.3 also refers to a coordinate-history variant that is not explicitly identified in Table 9. Please specify which results actually enable coordinate history.

  1. What do MoRE’s additional components contribute beyond existing optimizer-state reuse methods?

ContinualAdam already resets local moments while retaining historical second-moment information, whereas the paper’s m-only reset arm is not equivalent to that method. Could the authors provide a direct comparison under the same training conditions to establish MoRE’s incremental benefit? The MoFO comparison also uses a shared-prefix protocol that differs from the main-table protocol and therefore requires alignment for a direct comparison. If the principal advantage is comparable performance at lower cost, does this advantage remain against related approaches that also avoid a separate Fisher-estimation pass?

  1. Does the retained history guide old-task preservation, or does it primarily reduce update magnitude?

AU and second moments measure parameter displacement and gradient activity, respectively; neither directly establishes functional importance for previous tasks. What evidence shows that allocating updates according to these statistics is more effective than adjusting learning rates or freezing the same modules? Table 9 changes gate placement and expert scaling simultaneously, making their individual contributions difficult to identify. Please clarify the gains attributable to role gates, expert scaling, and coordinate history separately, and identify which results demonstrate the value of the historical information itself.

  1. Why does SeqFT exhibit substantially weaker new-task acquisition despite having no explicit retention constraints?

The difference is particularly pronounced on Qwen’s cm task: acquisition accuracy is approximately 43.2 for SeqFT, compared with 64.2 for both EWC and MoRE. Does this gap already exist before training on cm, or does it emerge during that task’s training? Please provide performance before task training, together with the evolution of current-task data loss and evaluation scores during training. The loss comparison should exclude EWC’s regularization term, and the optimizer-state boundary policies and learning-rate schedules should be specified for each method. For MoRE, how much of the acquisition improvement is explained by optimizer reset alone, and how much is contributed by the additional gates and history-based scaling?

  1. How should SeqFT’s positive BWT and MoRE’s higher final score but worse BWT be interpreted?

On Qwen, SeqFT achieves BWT of +0.23, whereas EWC and MoRE achieve -0.70 and -1.51, respectively. Each method’s BWT is measured relative to its own acquisition performance, so these values do not constitute a retention comparison from matched acquisition levels. Using per-task trajectories corresponding to the main-table configurations, please explain which tasks contribute to SeqFT’s positive BWT, whether their improvements mask deterioration on other tasks, and how MoRE’s acquisition advantage changes during subsequent training.

In particular, EWC and MoRE achieve the same acquisition accuracy on cm, but their final accuracies are approximately 61.7 and 59.3, respectively. This difference cannot be explained by unequal acquisition levels. Why do history-based scaling and role gating fail to provide better preservation on this task? Please clarify whether the current evidence primarily supports stronger acquisition, better retention, or a trade-off in which some retention is sacrificed for higher final performance and lower training cost.

1 comment

Dear Reviewer RTd8,

Thank you for your detailed review. We appreciate your positive evaluation of our method. We have added the comparisons and clarified the protocols in the paper as follows:

Q1 (W1). Which gates produced the main results?

The main results use MoRE's dynamic role schedules in Table 3: gates decay to a floor or hold it throughout a task. Task/role settings were selected through module probes and TRACE test feedback during development. The added training-feedback implementation fits one shared response to the known OLMoE dense-floor pattern, then freezes it for new-seed validation at saved task boundaries. All six floors match after 32 updates; three cm warmups select zero. Each run verifies application for 32 further updates. This validates floor reproduction; the main scores use Table 3's schedules. Coordinate history is off throughout Table 1 and on only in the final row of Table 8 (Appendix A).

Q2 (W2). What is added beyond ContinualAdam?

MoRE adds role and expert allocation to reset. ContinualAdam blends task-size-weighted second-moment history into Adam's denominator; Appendix C.1 compares the algorithms. Table 6 covers all six orders of C-STANCE, NumGLUE-cm and NumGLUE-ds, two seeds, and matched starts, data and budgets.

MethodFinal meanBWT
Carry48.81-2.13
ContinualAdam47.12-2.34
MoRE-R (ours)49.75-0.85
MoRE (ours)50.02-0.93

MoRE exceeds ContinualAdam by 2.91 points. Training time averages 4.98/5.37 h for MoRE/ContinualAdam (Appendix C.1). These timings include loading/saving, exclude evaluation and have unmatched concurrent load. MoFO's paired cost reference is MoRE⁻: 1.5× time and 40% more optimizer memory (Section 5).

Q3 (W2). Does history do more than shrink updates?

At the Qwen cm-to-ds boundary, measured coordinate history reduces earlier-task loss while shuffled history increases it at the same update magnitude (Table 9). Table 7 isolates gates and expert allocation from the same Qwen pre-cm state, data and learning rate:

ConfigurationMean (MB/Py/cm)Earlier mean (MB/Py)
MoRE-R, gates at 151.5049.48
Dynamic gates, AU off53.7851.66
MoRE (+ AU)55.0851.75
Global shuffle54.3451.88
Within-layer shuffle54.3051.83
Layer means54.2151.68

In this Qwen comparison, gates add 2.28 points; unnormalized AU adds 1.29 with the same modules frozen. Shuffling keeps the same frozen modules, coefficient values and scaling strength, but changes which experts receive them. Its lower final scores support history-guided allocation beyond module freezing; AU primarily improves cm learning. Task normalization changes this outcome: measured/shuffled scores are 52.13/54.29.

The Table 9 control matches update magnitude within each role, isolating placement from step size. Halving Qwen's learning rate lowers ds accuracy (71.38→65.85) without improving cm retention (Appendix C.3).

Q4 (W2). When does the cm learning gap arise?

In the matched-prefix experiment, the gap develops during cm training. All arms restore identical model/optimizer state at 0.00% accuracy, sharing data order and the cosine schedule; AU and coordinate-history scaling are off. Table 11 reports development cross-entropy and final test accuracy:

AblationBeforeStep 32Epoch 1Epoch 2Epoch 3cm acc. (%)
Carry2.79060.61651.48701.29911.853454.32
MoRE-R2.79060.94681.42521.72891.659755.56
+ dynamic gates2.79060.70651.95182.32342.541158.02

Reset adds 1.23 points and gates another 2.47 during this controlled continuation. Token-level loss and generated-answer accuracy capture different outcomes.

The original runs also show that low training loss does not ensure higher test accuracy:

MethodEpoch 1 lossEpoch 2Epoch 3cm accuracy (%)
SeqFT0.18240.00410.000843.21
EWC0.16610.00070.018164.20
MoRE0.1463<0.00005<0.0000564.20

MoRE starts at 3.70% before cm. Loss excludes EWC's penalty; the threshold denotes logging resolution. All use global cosine decay. SeqFT/EWC carry Adam moments and counters; MoRE resets them.

Q5 (W2). What explains the cm retention gap?

Qwen shows a trade-off: MoRE retains 59.26% on cm versus EWC's 61.73%, from equal 64.20% acquisition. This MoRE configuration uses reset and role gates, with expert and coordinate history off. Gates scale updates; EWC adds an anchor-restoring penalty.

MoRE scores higher on Py150, ds, 20Min and C-STANCE, with final mean 54.90 versus 54.60 and training time 15 h 31 min versus 20 h 49 min (Table 1). This is an observed single-run margin. SeqFT's positive BWT mainly reflects cm rising from 43.21 to 46.91, while MB, ds and 20Min decline.

Comments on the paper

1 comment

Dear Area Chair, Dear Reviewers,

Thank you for your time and thoughtful feedback. We have made every effort to address all questions through additional experiments, controlled comparisons, and clearer explanations. The new results and clarifications are incorporated into the revised manuscript, with changes marked in blue. We summarize the main issues and our responses below; the individual replies provide the supporting details.

1. Dynamic gating and the evaluated configurations (RTd8).

The main results use the dynamic role schedules in Table 3; Appendix A now states their selection procedure and the configuration of each history channel. The added training-feedback implementation fits a shared response to the OLMoE floor pattern, freezes it before new-seed boundary checks, and reproduces all six floors after 32 warmup updates, with subsequent application verified.

2. Benefit beyond optimizer-state reuse (RTd8, ixcF).

MoRE achieves a final mean of 50.02 versus 47.12 for ContinualAdam across all six orders of C-STANCE, cm and ds and two seeds (Table 6). Appendix C.1 presents the algorithms side by side, separates boundary reset from structured allocation, and reports training time and memory for both methods, which avoid a Fisher-estimation pass.

3. Contributions of the components and historical information (RTd8, ixcF).

The matched Qwen comparison isolates role gates and expert allocation: gates raise the mean from 51.50 to 53.78, and AU raises it to 55.08 (Table 7). At the same update norm within each role, measured coordinate history reduces earlier-task loss while permuted history increases it (Table 9). This directly demonstrates the value of historical assignment beyond reducing step size. The expert-level tests also characterize how predictive strength and the acquisition/retention balance vary across task transitions (Appendix C.3).

4. When acquisition gains arise and how retention changes (RTd8).

From identical pre-cm model and optimizer state, reset and role gates each improve acquisition during cm training. Table 11 provides pre-task performance, data-only training losses, development-loss trajectories and final test accuracy. For the original Qwen comparison, MoRE's advantage is higher final performance at lower cost, while EWC preserves cm better. SeqFT's positive BWT is driven mainly by cm improvement, while MB, ds and 20Min decline.

5. Task order, sequence length and newer architectures (ixcF, rZfa).

We added all six permutations of C-STANCE, cm and ds, two- versus three-task controls, and local carry/reset probes across the six directed task transitions (Appendix C.2). On Qwen3-30B-A3B, MoRE reaches 64.85 versus 57.73 for MoRE-R and 39.31 for zero-shot evaluation, using identical inputs and the same native chat template (Table 6).

6. Joint training, past-task replay and model merging (rZfa).

The new comparisons include joint training, replay of actual earlier-task data, and TIES-Merging (Yadav et al., NeurIPS 2023). Table 12 reports 49.45 for MoRE with unit role gates, alongside 48.51 for joint training, 47.33 for earlier-task replay and 47.98 for TIES. TIES substantially improves on uniform merging by resolving conflicting task updates; MoRE learns sequentially without retaining separate task models or a replay buffer.

These experiments also sharpen the mechanism we aim to understand: how optimizer memory controls both where updates are allocated and how much movement accumulates over a finite task. The sum of update lengths bounds parameter displacement and, under a smoothness assumption, bounds the change in an earlier-task loss. The matched-norm experiments show why update direction matters alongside this cumulative magnitude. For longer streams, our next step is to use warmup signals to set cumulative update budgets, extending the current per-step controls across an entire task.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.