acceptodds
Withdrawn from ICLR 2027

MoEcraft: Trajectory-Based Sparse Upcycling of Instruction-Tuned Small Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We study sparse upcycling of instruction-tuned small language models into mixture-of-experts (MoE) models without returning to large-scale continual pre-training. MoEcraft reuses checkpoints from a single on-policy reinforcement-learning (RL) trajectory to initialize experts, followed by lightweight joint router-expert calibration. It combines all-linear low-rank adaptation, feed-forward adapters from eight checkpoints, and top-2 routing with calibration on 512 examples. In the canonical Qwen3-1.7B run, the chain scores 52.35 at the dense RL endpoint, 51.84 after sparse assembly, and 56.95 after calibration on Overall13, the mean over 13 benchmarks. Across three RL trajectories, calibration adds 3.60-5.11 points, and final performance averages 56.14 +/- 0.72 (mean +/- sample standard deviation). A single-run supervised fine-tuning trajectory control using the same sparse construction and calibration gains 2.50 points from assembly but loses 0.64 during calibration, driven by a summarization regression. The stage-wise pattern also recurs with a second RL optimizer and on Llama and Falcon models. The tested pipelines complete within one day per model on a single 96GB GPU. Under a fixed serving setup, the 1.7B-derived MoE achieves 1.54x/1.22x the output throughput of released dense Qwen3-4B at batch sizes 1/4 with 2.76B active parameters, but has 9.10B total parameters and a larger resident footprint (30 versus 20 GiB). These results characterize the utility and performance-footprint trade-offs of reusing RL trajectories for post-alignment sparse upcycling in the tested settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-experts, sparse upcycling, low-rank adaptation, expert routing, small language models, verifiable rewards

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_f39SOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

MoEcraft initializes sparse experts from checkpoints along an RL trajectory and jointly calibrates the router and expert adapters.

Critical strengths

The paper includes cross-family results, three trajectory seeds and measured serving costs.

Critical weaknesses

The presentation falls below an acceptable standard: Figures 1 and 5 compress essential labels and annotations into tiny text, while Tables 1 and 20 crowd multiple attributes and measurements into layouts that require excessive effort to decipher.

The authors use AI assistance with figure layout, but the resulting visuals need substantial redesign. More fundamentally, the experiments do not establish the value of trajectory harvesting: a single-terminal-checkpoint interpolation baseline reaches 55.92 versus 56.95 for the proposed method, without a paired multi-seed comparison establishing that advantage.

Most improvement occurs during additional RL calibration, yet no matched dense continuation baseline isolates the benefit of sparse conversion from simply training longer. None of the closest published adapter-MoE pipelines is reproduced. The “capacity-aware” rank rule is also unsupported by converted-model rank sweeps.

Reproducibility is particularly weak: training code and run-level configurations are withheld, the Best-8 selection criterion is unrecorded, and parts of the routing analysis lack underlying measurement metadata. Finally, the workstation-efficiency argument rests on a conversion requiring 92.9 GiB and a model with 9.10B resident parameters. The paper offers a complicated construction without convincing evidence that its distinctive components justify that complexity.

I think this paper shoulded be rejected.

Questions for authors

Please refer the weaknesses.

Reviewer_on9sOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents MoEcraft to upcycle instruction-tuned SLMs into sparse MoE models without continual pre-training CPTCPT by harvesting intermediate checkpoints from a single on-policy RL trajectory and jointly calibrating the router and LoRA adapters on 512 examples; however, the manuscript is marred by severe text misalignment and overlapping in the figures, and it represents an incremental combination of existing techniques with limited novelty over prior work such as UpIT.

Critical strengths
  1. The paper addresses a practical "Post-Post-Training" sparse upcycling setting on a single workstation GPU, avoiding the continual pre-training cost of hundreds or thousands of GPU-days required by traditional sparse upcycling.

  2. The authors provide extensive comparative evaluations spanning three model families—Qwen3 across 0.6B, 1.7B, and 4B scales, Llama-3.2 across 1B and 3B scales, and Falcon3 across 1B and 3B scales—alongside two reinforcement learning optimizers, SAPO and GRPO.

Critical weaknesses
  1. The functional complementarity and marginal utility of checkpoint-derived experts remain insufficiently established. The experts are extracted from successive checkpoints of the same RL run and may therefore exhibit substantial functional overlap. The reported analyses neither establish which historical checkpoints provide useful functionality beyond the terminal checkpoint nor quantify their individual marginal contributions. The closely related UpIT addresses expert differentiation by separately optimizing each expert and its routing vector on expert-specific data. However, the present paper does not sufficiently establish the functional complementarity of the resulting experts. This uncertainty weakens the empirical justification for the trajectory-harvesting design.

  2. The paper has substantial presentation issues. Figures 1 and 2 contain severely misaligned and overlapping text, impairing readability. Despite available space in the main text, several useful experiments and findings are relegated to the appendix, while the main text contains considerable repetition and content that adds little substantive information. In addition, the experimental comparisons should include dense baselines with comparable total parameter counts. For example, the MoE with approximately 9B total parameters should be compared with Qwen3-8B to assess the performance gains under a comparable total-parameter budget, especially given that even the smaller dense Qwen3-4B after SAPO already outperforms it at 61.40 versus 56.95.

Questions for authors
  1. Can you quantify the functional overlap and individual marginal contributions of the intermediate checkpoints, particularly relative to the terminal-adapter interpolation baseline?

  2. Unlike UpIT, which optimizes each expert and routing vector on expert-specific data, how does MoEcraft achieve comparable expert differentiation from a single trajectory, and how do the two methods compare empirically under the same budget?

  3. How does MoEcraft compare with a total-parameter-matched dense baseline such as Qwen3-8B, given that even the smaller SAPO-tuned dense Qwen3-4B already achieves a higher score of 61.40 versus 56.95 with a smaller memory footprint?

Reviewer_BPMiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to convert an instruction-tuned small language model into a sparse mixture-of-experts model without large-scale continual pretraining. MoEcraft reuses checkpoints from a single reinforcement-learning trajectory to initialize eight feed-forward experts, combines their attention updates into a shared module, and jointly calibrates the router and adapters using 512 training examples. Experiments on Qwen, Llama, and Falcon models show that assembly can initially reduce performance, while subsequent calibration improves it beyond the dense RL endpoint. Ablations examine expert initialization and calibration choices. Serving experiments also report higher throughput than Qwen3-4B under the tested conditions, but with greater memory use.

Critical strengths
  1. The paper addresses a practical problem and provides useful evidence about where the gains arise. It studies upcycling an already aligned model on a single workstation GPU and separately evaluates trajectory generation, expert assembly, and calibration. Results across three SAPO seeds, GRPO, and the Llama and Falcon families support the importance of calibration. The observed difference in expert utilization between SFT and RL trajectories also motivates further investigation of trajectory-based initialization.

  2. The resource measurements make the practical trade-offs assessable. Controlled serving experiments use vLLM with a fixed KV cache, and instrumented re-runs report per-stage time and memory. These measurements allow the performance gains to be weighed against the larger memory footprint. The paper’s explicit limitations and its framing as a trade-off, rather than a claim that smaller models outperform larger ones, strengthen the practical value of the study.

Critical weaknesses
  1. Missing implementation details and unresolved measurement questions limit independent verification of the results. Gating, router initialization, training-data composition, decontamination, and benchmark-specific scoring are insufficiently documented. Identical-copy MoEs score below their dense source, making it unclear whether the transition gap partly reflects implementation choices. Near-ceiling MBPP scores, below-chance GPQA results, and the identical task-group offsets in Table 17 also require clarification; the underlying inputs for Table 17 are not released. These gaps affect both the interpretation of the performance gains and the routing evidence used to explain them.

  2. The additional value of historical-trajectory reuse over simpler alternatives is not sufficiently established. Terminal-checkpoint interpolation already reaches 55.92 versus 56.95 for Sequential-8, but the comparison changes shared-attention initialization and lacks paired full-pipeline repetitions. The SFT comparison uses one unmatched endpoint, and many ablation differences are small relative to the reported cross-trajectory variation. A budget-matched dense continuation, checkpoint soup, and a relevant existing upcycling or adapter-MoE baseline are missing. Given the relationship to UpIT and BTX, these controls are important for establishing whether the historical trajectory itself provides a meaningful advantage.

Questions for authors
  1. How are the router and top-2 gate weights initialized and normalized, and why does the identical-copy MoE score 50.97 rather than reproduce the dense endpoint’s 52.35 before further training?

  2. Can you provide the data and scoring protocols needed to interpret the near-ceiling MBPP and below-chance GPQA results, together with the underlying routing measurements explaining the identical task-group offsets in Table 17?

  3. With shared-attention initialization and additional training budgets controlled, does Sequential-8 consistently outperform terminal-checkpoint interpolation and continued optimization of the dense endpoint across paired runs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.