- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how optimizer memory and parameter roles affect acquisition and retention during continual fine-tuning of fixed-capacity MoE models. It proposes MoRE (Moment Reset and Reuse), which separates task-local optimizer state from retained history and applies role gates and history-conditioned update scaling. Experiments on TRACE with OLMoE and Qwen, and on KuaiRand recommendation streams, report performance and efficiency gains. However, the benefits of resetting or carrying optimizer state depend on the learning regime, and higher final scores do not always indicate less forgetting.
Critical strengths
1.The research question and findings have practical relevance: cross-task optimizer state and heterogeneous parameter roles jointly affect task acquisition and retention.
2.The method is relatively simple to implement and requires no additional inference modules.
3.The experiments cover different architectures and different types of data change.
Critical weaknesses
1.The connection between the control signals and the central motivation of cross-task interference is insufficiently established. The paper attributes interference to inherited optimizer history and heterogeneous parameter roles, but primarily attenuates updates based on update magnitudes, experts’ task-wise update shares, and second-moment magnitudes. These statistics reflect parameter update activity rather than directly measuring conflicts between new and previous tasks. For example, a module with larger updates is not necessarily more prone to forgetting, and an expert whose updates are concentrated on one task is not necessarily more important to that task.
2.The core design primarily consists of update scaling, while the mechanistic explanation and conceptual depth remain limited. Task-local state reset and historical second-moment reuse have already been explored in ContinualAdam. The additional contribution here mainly lies in introducing role-, expert-, and coordinate-level controls. However, expressing history through a “shared-plus-task-specific decomposition” of second moments is largely an algebraic reformulation of denominator scaling. This does not, by itself, explain why the selected historical statistics and scaling rules improve continual learning.
3.The existing ablations do not sufficiently establish the necessity of the individual designs or clearly attribute the performance gains to specific components.
Questions for authors
1.Can AU, experts’ task-wise update shares, and second-moment magnitudes predict performance degradation on previous tasks? Could the authors provide direct evidence that these signals identify cross-task interference, rather than merely reflecting parameter update activity?
2.What additional benefits do MoRE’s role-, expert-, and coordinate-level controls provide over ContinualAdam or standard group-wise learning-rate control? Could component-wise ablations under matched experimental conditions, together with controls that match the actual update magnitudes, establish the necessity of each component and the retained history itself?
3.Using the same model and training settings, could the authors vary task differences, task order, and sequence length to clarify when optimizer state should be reset or reused, and assess the robustness of these conclusions?
3 comments
Thank you for your response. I would like to keep my score.
My concern is that, in the context of full-scale training, it remains unclear how much additional benefit the historical allocation strategy in MoRE provides compared to a baseline that uses grouped learning rates matched in terms of update magnitude. Furthermore, the performance gains are marginal; a difference of just a few tenths of a percentage point could easily be attributed to factors like the choice of random seed, or other variables.
Dear Reviewer ixcF,
Thank you for continuing the discussion. We would like to clarify what history contributes beyond learning-rate adjustment, and why the full-scale accuracy–cost result matters.
1. Matching update magnitude does not determine what an update preserves.
In your follow-up, you ask how much additional benefit MoRE's historical allocation provides during full-scale training over “a baseline that uses grouped learning rates matched in terms of update magnitude.” Adam itself already adapts coordinate-wise updates using exponential moving averages of gradients and their squares. This adaptation, or a task-wise layerwise learning-rate schedule, does not automatically protect earlier-task knowledge: a large update on a later task can still alter parameters important to previous tasks. Group-level rate multipliers are a coarser control; without cross-task history, grouping itself supplies no criterion for protecting earlier knowledge. Our Adam+SeqFT baseline already uses adaptive step sizes, yet still exhibits forgetting on both OLMoE checkpoints. MoRE combines task-boundary optimizer-state control with history-dependent expert allocation to balance new-task learning and earlier-task preservation.
The revised Table 9 directly controls update magnitude: at the Qwen cm-to-ds transition, we use the same model and gradient and match the update norm within each role.
| History allocation | Earlier-task loss change | Current-task loss change |
|---|---|---|
| None | +0.0109 | −0.8634 |
| Uniform | +0.0243 | −0.8065 |
| Permuted | +0.0019 | −0.7716 |
| Measured | −0.0041 | −0.5793 |
Measured history alone improves both tasks in the norm-matched single-step test.
2. The improvements extend beyond a few tenths of a point and are accompanied by substantial efficiency gains.
| Model | EWC final mean | MoRE final mean | EWC training time | MoRE training time |
|---|---|---|---|---|
| OLMoE-Base | 48.39 | 49.53 | 10 h 50 min | 4 h 12 min |
| Qwen1.5-MoE | 54.60 | 54.90 | 20 h 49 min | 15 h 31 min |
MoRE reduces training time by 61% on OLMoE and 25% on Qwen while reaching or exceeding EWC's observed final scores. Qwen's peak memory also falls from 74.7 to 65.2 GB per GPU. OLMoE's 49.53 approaches the separate-model independent-fine-tuning reference of 50.05 while retaining one continual model.
You also describe the gains as “a difference of just a few tenths of a percentage point.” This does not characterize the full set of results. MoRE improves over standard SeqFT across both model families, including gains of 3.90 points on OLMoE-Base and 2.92 points on Qwen:
| Model | SeqFT | EWC | MoRE | Gain over SeqFT |
|---|---|---|---|---|
| OLMoE-Base | 45.63 | 48.39 | 49.53 | +3.90 |
| OLMoE-Instruct | 47.99 | 48.20 | 48.96 | +0.97 |
| Qwen1.5-MoE | 51.98 | 54.60 | 54.90 | +2.92 |
The OLMoE MoRE results average two seeds. The additional six-order, two-seed comparison gives MoRE 50.02 versus ContinualAdam 47.12, with MoRE ahead in ten of the twelve matched runs (Table 6). The evidence therefore spans model families, repeated runs and task orders.
Dear Reviewer ixcF,
Thanks for the careful review. Thank you for recognizing the practical value of optimizer-state control and the simplicity of our proposed method. Your question about what the signals actually predict led us to add direct functional tests alongside the component comparisons. These results are now in the revised paper.
Q1 (W1). Do the signals predict old-task degradation?
The added tests connect history to old-task loss: expert scores predict local sensitivity most clearly at cm-to-ds, and coordinate-history placement improves preservation at a fixed update magnitude. We measure old-task loss after equal-norm single-expert updates and correlate it with the pre-update AU score and maximum task share (Appendix C.3):
Entries are within-layer rank correlations with signed old-task loss change. Experts and predictors are fixed before outcomes; four training minibatches are repeated interventions per expert. The analyzed counts are 24/22/24 experts, and the AU score uses unnormalized task energy. The association is strongest at cm-to-ds and weak at the other two boundaries.
For second-moment history, Table 9 tests allocation at the Qwen cm-to-ds boundary with the same model and gradient, matching the candidate-update norm within each role:
At equal update magnitude, measured history is the only allocation that reduces old-task loss, while all four allocations improve the new task. Thus the historical assignment changes preservation even when step size is controlled. Table 10 tests expert allocation at matched realized update norms across three boundaries and shows that the old/new-task balance varies with the transition.
Q2 (W2, W3). What do the additional components contribute?
MoRE improves final mean over ContinualAdam by 2.91 points; matched component experiments identify the contributions of role gates, expert allocation and coordinate history. Table 6 covers all six orders of C-STANCE, cm and ds, with two seeds and matched starts, data and budgets:
MoRE gains 2.91 points over ContinualAdam. Appendix C.1 presents the algorithms side by side.
Qwen's matched pre-cm comparison isolates role gates and expert allocation: gates raise mean 51.50→53.78 with AU off; unnormalized AU gives 55.08. Global/within-layer permutation and layer means give 54.34/54.30/54.21 (Table 7). The frozen modules and scaling strength are unchanged: permutation tests the assignment of historical coefficients, while layer means test expert-specific allocation within a layer. Task-normalizing those coefficients instead gives 52.13, versus 54.29 after permutation. These comparisons identify which allocation works in this setting.
Coordinate history improves Qwen BWT from -1.51 to -0.11, with means 54.90/54.27 (Table 8). The role-matched controls in Q1 hold update magnitude fixed and change its placement, directly testing the value of history beyond step-size reduction.
Q3. When should state be reset or reused?
Reset improves the final mean in both cm/ds directions, while local probes reveal different acquisition/retention preferences at each transition. We test all six three-task orders. At the common cm/ds endpoints, reset/carry means are 48.21/46.82 for cm→ds and 53.61/50.83 for ds→cm. Adding a C-STANCE prefix gives 52.22/50.84 and 55.92/53.92, respectively (Appendix C.2). These compare two- and three-task streams with their respective global cosine horizons.
A separate development-gradient probe ranks carry versus reset before either update is applied. It correctly orders 5/6 old-task and 4/6 new-task single-step comparisons. For cm→ds, reset improves both; for ds→cm, carry better preserves ds while reset improves cm more. These local preferences complement the complete-stream results and the 30-pair recommendation study.
Our main six-task order matches Yu et al. (ICML 2026). Spurious Forgetting (Zheng et al., ICLR 2025) follows TRACE's eight-task order, and Any-SSR (Tong et al., ICCV 2025) evaluates two published eight-task orders. Appendix C.2 reports all six permutations of C-STANCE, NumGLUE-cm and ds.