acceptodds
Under review as a conference paper at ICLR 2027

FlexMoRE: A Flexible Mixture of Rank-heterogeneous Experts for Efficient Modular Large Language Models

Abstract

Recent advances in mixture-of-experts (MoE) models can compose independently trained, domain-specialized experts around a shared backbone, but storing every expert at full size limits practical deployment. We introduce FlexMoRE (Flexible Mixture of Rank-Heterogeneous Experts), a post-hoc approach that replaces dense experts with low-rank updates relative to a shared public base model. Starting from the FlexOlmo expert collection, we evaluate six experts and combined FlexMoRE models with , , and active experts over 15 ranks, , on 132 tasks from eight benchmarks. Rank–performance relationships are strongly task-dependent: reasoning-oriented benchmarks, particularly BBH, Math2, and Code4, benefit from increased rank, whereas GEN5 and several knowledge-oriented settings show weak or negative rank sensitivity. This variation motivates heterogeneous capacity allocation. Compared with homogeneous low-rank mixtures, heterogeneous FlexMoRE configurations retain stronger aggregate performance at substantially lower stored capacity. For example, the MC9 selected configuration uses 10.75B rather than 33.27B stored parameters (a 67.7% reduction) with an original Avg. of 40.75 versus 40.77 for FlexOlmo; performance is slightly lower when MC9 is excluded. FlexMoRE achieves higher aggregate scores than an AdapterSoup baseline, whose performance deteriorates at higher ranks, supporting separately addressable, conditionally routed experts over static averaging in the evaluated setting. FlexMoRE therefore provides a practical post-hoc route to storage-efficient composition of independently trained experts, while highlighting rank allocation as an open problem.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-experts models, language modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jqYbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents FlexMoRE, a post-hoc compression approach for independently trained domain experts in FlexOlmo. Each domain expert is represented as a low-rank weight update relative to a shared public model through truncated SVD. The method retains the existing routing mechanism and assigns expert-specific ranks based on benchmark performance. Experiments investigate rank sensitivity, heterogeneous capacity allocation, and the resulting trade-offs between model storage and downstream performance.

Critical strengths

The method provides a practical way to compress an existing expert collection without retraining the experts or tuning the router. The experiments demonstrate substantial storage savings and include actual GPU memory measurements. In the evaluated implementation, the MC9-selected configuration reduces resident GPU memory from 62.74 GiB to 20.94 GiB. This supports its practical value when pretrained experts are already available and additional training is undesirable.

Critical weaknesses
  1. The proposed pipeline relies on fully trained domain-specific FFNs before approximating their weight updates through truncated SVD. A natural alternative is to train domain-specific LoRA updates directly against the frozen public model. Although post-hoc compression has a clear advantage when expert checkpoints already exist and training data are unavailable, the paper does not establish whether this route offers a favorable quality–cost trade-off when developing a new expert collection.

  2. The compression baselines are insufficient to establish the competitiveness of the proposed approach. They do not determine whether this approach preserves more useful capability than other compression strategies at a comparable storage budget, such as expert-weight quantization or another low-rank approximation method。

Questions for authors
  1. FlexMoRE first relies on fully trained domain-specific FFNs and then approximates their updates through low-rank factorization. An alternative is to train domain-specific LoRA updates against the same frozen public model from the outset. I recognize that these approaches address different starting conditions. Post-hoc compression can reuse existing checkpoints without accessing domain training data, whereas direct LoRA training requires such access. Nevertheless, a controlled comparison would clarify when the proposed route is preferable.

  2. Could the authors include a relevant compression baseline, such as expert-weight quantization or another low-rank approximation method, while preserving the independently addressable experts and routing mechanism? Comparing downstream quality at similar storage budgets in bytes, together with measured GPU memory and inference throughput, would help establish whether post-hoc factorization offers a competitive practical trade-off.

7 comments

We thank the reviewer for their service to our work. We notice that the reviewer emphasize that our method is a practical implementation, that we validate with substantial storage savings, when pretrained experts are already available and additional training is undesirable. We address each concern below.

  1. Post-hoc extraction versus directly trained LoRA experts.

Post-hoc extraction versus directly trained LoRA experts. We agree that directly trained LoRA experts are a relevant alternative, and that a controlled comparison would clarify the quality–cost trade-off when both approaches are feasible. However, it would be computationally expensive to conduct such a comparison as systematic as our post-hoc analysis with varying ranks between 1 and 2048.

Post-hoc extraction also allows the composer to explore different retained ranks per expert without repeating expert training. This flexibility matters because our results show that rank sensitivity differs across experts and evaluation tasks (Sec. 5 - Rank Sensitivity Analysis). We further demonstrated that FlexMore outperforms the AdapterSoup baseline across the board in Table 2 and Figure 2c.

Moreover, our setting that starts from independently contributed dense expert checkpoints comes with conceptual advantages: A provider of an expert may be unable or unwilling to share the original training data with the party composing the mixture. The composer may therefore be unable to train replacement LoRA experts, whereas post-hoc extraction can compress the existing checkpoints without accessing those data or tuning the inherited router.

We do not claim that post-hoc extraction is preferable when training data and compute are available and agree that a controlled comparison – while expensive – would be valuable. We will clarify this distinction in the paper and mention the LoRA training as future work.

  1. Other compression baselines.

We agree that quantization is a relevant alternative and potentially complementary approach. A controlled comparison is not yet complete, and therefore we do not claim superiority over quantization or other compression methods. We are investigating this comparison and hope to share results during the open-discussion period.

An illustrative calculation provides context for the storage budget. Assuming BF16 for the public model, and treating the parameters beyond the public model as quantizable, matching the MC9-selected configuration corresponds to an average of approximately 2.1 bits per parameter for those additional parameters: (16 \times (10.75-7.30)/(33.27-7.30) \approx 2.13). This estimate uses rounded parameter counts and excludes quantization metadata and other overhead. It is not a measured quantization result and does not establish a quantized model’s quality, resident memory, or throughput.

Our experiments do demonstrate a memory–throughput trade-off for the existing expert collection: in the tested Hugging Face eager backend, the MC9-selected configuration reduces resident GPU memory from 62.74 to 20.94 GiB (−66.6%), while reaching 0.88× the dense baseline’s throughput. A matched-byte comparison that preserves the expert branches and router, and measures quality, memory, and throughput, would be valuable future work.


Support of calculation: The calculation asks: if we leave the public model in BF16, how many bits could we spend on each additional dense-model parameter while matching FlexMoRE's raw weight storage?

• 16: bits per BF16 parameter. • 33.27B: total stored parameters in the dense FlexOlmo mixture. • 10.75B: total stored parameters in the MC9-selected FlexMoRE mixture. • 7.30B: parameters in the public base model, assumed to remain in BF16

in both representations.

Subtracting the shared public model gives: • 33.27 - 7.30 = 25.97B additional parameters in the dense mixture. • 10.75 - 7.30 = 3.45B BF16-equivalent parameters available for the

compressed representation.

The available storage per additional dense parameter is therefore: 16 × (3.45 / 25.97) ≈ 2.13 bits per parameter.

The counts are rounded, and the additional parameters may include smallbnon-expert components such as router parameters. This is an illustrative raw-storage calculation, not a measured quantization result; it excludes metadata and implementation overhead.

AuthorsOpenReview ↗

Quantization Baseline

Following up on point 2 of our response, we have now completed additional quantization experiments. We thank the reviewer for suggesting this comparison.

We added expert-weight quantization baselines at 2 and 4 bits, with and without quantizing the public model, see results in the Table below. Four-bit quantization of the experts while retaining the public base in BF16 nearly preserves the dense baseline’s aggregate scores (36.44 vs. 36.46 with MC9; 32.31 vs. 32.32 without MC9), whereas 2-bit quantization causes substantial regressions, resulting in aggregate scores of 11.65 and 8.27. This establishes 4-bit quantization as a meaningful compression baseline while illustrating the quality cost of more aggressive quantization.

We further evaluate combining quantization with FlexMoRE’s low-rank representation. The hybrid configurations achieve aggregate scores within 0.03 percentage points of BF16 FlexMoRE: 39.13/34.91 for q4-no-base and 39.12/34.90 for q4-all, compared with 39.15/34.92 for FlexMoRE (with/without MC9). Their corresponding estimated storage is 16.54 GB and 10.32 GB, versus 21.49 GB for BF16 FlexMoRE. These results show that four-bit quantization nearly preserves the aggregate performance of the reconstructed low-rank model. The estimated storage savings for quantized A/B factors remain to be validated together with performance in a factorized implementation.

Table: Quantization and hybrid-storage configurations. All configurations use seven active experts per token (a7). GB denotes weight storage, including quantization scales, rather than measured inference memory. q2 and q4 denote 2-bit and 4-bit quantization. “no-base” retains the public model in BF16; “all” includes the public model in quantization. All rows are complete (132/132 tasks). Avg. w/ MC9 is the macro average across the eight benchmarks; Avg. w/o MC9 excludes MC9.

ConfigurationParamsGBMC9Gen5MMLUMMLU-ProAGIEvalBBHMath2Code4Avg. w/ MC9Avg. w/o MC9
Dense FlexOlmo-a7 BF1633.27B66.5465.4644.5650.8922.0037.3735.4625.4210.5636.4632.32
q2-all33.27B15.4135.330.0023.1211.7123.030.000.000.0011.658.27
q2-no-base33.27B22.7235.330.0023.1211.7123.030.000.000.0011.658.27
q4-all33.27B22.9965.0345.4050.9621.8237.2434.3124.8610.5436.2732.16
q4-no-base33.27B29.2165.3644.8951.0121.9137.5934.8625.2710.6336.4432.31
FlexMoRE MC9 BF1610.75B†21.49†68.7356.6953.8724.4541.0240.6525.841.9539.1534.92
Hybrid MC9 q4-no-base10.75B*16.54*68.7156.6053.6924.4540.9240.7526.151.7839.1334.91
Hybrid MC9 q4-all10.75B*10.32*68.6856.4153.7924.5341.0340.3326.281.9239.1234.90

* Hypothetical quantized A/B storage; scores use dense reconstruction then quantization.

† BF16 A/B storage; actual checkpoint contains dense reconstructed weights.

Thank you for your response and the additional experiments. However, I still have a few concerns:

  1. This clarification highlights a specific practical scenario: the method is useful when pre-trained expert models are available, but their original training data is inaccessible. Nevertheless, in cases where both approaches are feasible, it remains unclear how this method compares to training LoRA-based expert models directly in terms of cost. Consequently, I am not certain what the specific advantage of this approach is when building an entirely new set of expert models.

  2. The comparison regarding quantization is quite informative; however, the aggregated scores mask significant disparities in the preservation of model capabilities. While FlexMoRE improves the MMLU score from 50.89 to 53.87, it causes a sharp decline in the Code4 score—from 10.56 down to 1.95. In contrast, applying 4-bit quantization to the expert models preserves performance across both metrics much more effectively. Could the authors please explain this trade-off and clarify under what circumstances choosing FlexMoRE would be preferable to direct quantization?

Reviewer_58XMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents FlexMoRE, which compresses independently trained dense FlexOlmo experts after training. The original router and backbone remain unchanged. The paper evaluates homogeneous and expert-specific rank allocations across 6 domain experts, 3 activation settings, and 132 tasks grouped into eight benchmarks. The main result is a substantial storage reduction: the MC9-selected heterogeneous configuration uses 10.75B rather than 33.27B parameters.

Critical strengths

Simple and useful post-hoc procedure - requires no data, expert retraining, or router tuning. Large evaluation matrix - 6 experts,15 ranks, multiple activation settings and 132 tasks.

Critical weaknesses

No clean equal-budget, held-out test of rank heterogeneity. Rank selection is not cleanly separated from evaluation.

Questions for authors

At exactly 10.75B stored parameters, how does the selected tuple compare with the best uniform and random allocations? Do rank choices transfer to a different task suite or a new set of independently trained experts?

2 comments

We thanks the reviewer for recognizing that our simple method is useful and emphasize that it does not require data, retraining or router tuning. We address each concern below.

  1. Separation of selection and evaluation.

The heterogeneous allocations were fixed using either MC9 alone or the original six-benchmark average, before Math2 and Code4 were evaluated. For the MC9-selected configuration, we report the average over the seven remaining benchmarks separately. Math2 and Code4 were not used to select either heterogeneous allocation. We will correct the description of the Avg-selected configuration to make this distinction explicit.

  1. Comparison at similar storage budgets.

Our homogeneous sweep provides comparisons around the 10.75B budget. At a2, the MC9-selected heterogeneous configuration achieves 36.80 on the average excluding MC9, compared with 32.68 for homogeneous rank 512, while using fewer stored parameters: 10.75B versus 11.75B. This supports the usefulness of the selected allocation relative to that homogeneous reference. This is not an exactly equal-budget comparison, and we have not evaluated random heterogeneous allocations at 10.75B. We therefore cannot determine whether the selection rule outperforms random allocation at that budget. Since ranks are selected using individual-expert evaluations rather than joint mixture performance, interactions with the other experts and the inherited router may also affect the resulting mixture.

  1. Transfer.

The MC9-selected ranks are determined from individual-expert evaluations and then reused unchanged in the composed mixture at a2, a4 and a7. Evaluation on benchmarks outside MC9 provides an additional test beyond the selection benchmark, although it remains within the same evaluation suite. The observed expert losses show that transfer is not uniform across tasks. We have not evaluated a different task suite or a new independently trained expert collection. Establishing a rank-selection strategy that transfers reliably across collections is an important direction for future work: the effect of an expert’s rank may depend not only on the calibration tasks, but also on the other available experts and how the inherited router weights them.

Thanks to the authors for the reponse. It addressed some of my concerns. However it is still unclear whether the selected allocation offers consistent advantage under fixed storage budget. I intend to maintain the current rating for now.

Reviewer_JbZCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents FlexMore approach, which compresses FlexOlmo dense experts into low-rank updates using truncated SVD and expert-specific ranks. Through extensive experiments, FlexMore shows substantial memory savings with maintained aggregated performance in some settings and analyze the effects of using different ranks.

Critical strengths
  1. The provided empirical evaluations are quite extensive. Different aspects are investigated across different experts, ranks, and benchmakrs.
  2. The introduced approach shows practical potential with memory savings without additional training, while keeping performance in some settings.
Critical weaknesses
  1. My main concern is about the novelty of the method. The approach basically is considered combining existing techniques, such as low-rank extraction using truncated SVD and the existing architecture. Using different ranks according to different experts also seems more like hyperparameter tuning than a new method. Structurally, LoRA-based MoE is also not new either.
  2. Heterogeneous rank selection appears to be heuristic. The authors select the ranks simply based on the benchmark performance of individual experts. While the results show some benefits, it is unclear whether this heuristic leads to be effective for the final mixture, or how it compares with alternative rank selection under the matched memory budgets.
  3. I appreciate the authors' efforts on extensive experiments, but the significant drops on certain benchmarks (e.g., Code4 and Math2) are concerning. While they show aggregated performance is competitive, it is questioning how reliably the method preserves domain-specific capabilities. Also, the experiments are limited to FlexOlmo.
Questions for authors

Please see weaknesses section above.

3 comments

We thank the reviewer for taking the time to review our work. We especially note the reviewer's acknowledgement of our exstensive experimental setup - including aspects of different experts, ranks and benchmarks - and the practical impact of the approach.

  1. Novelty and relationship to existing methods.

Note that we do not claim algorithmic novelty for the truncated SVD itself. Instead, we view our main contribution as the generalization of FlexOlmo to accomodate experts-as-LoRA-adapters and then test this heterogeneous-rank architecture via experiments that convert independently-trained full-rank experts into a routable low-rank mixture post hoc, without retraining the experts or tuning the router - while keeping them separately addressable and conditionally routed.

Notably, this differs from LoRA-based MoE methods, which train adapters and routing jointly on pooled data, and from quantization or merging, which reduce precision or fuse experts without retaining addressability. Our results show that FlexMoRE yields superior performance and better scaling behavior than AdapterSoup in Table 2.

These results come also with practical implications for cases, where data can be used for training specialized experts, but cannot be pooled. Reducing expert storage makes such composition feasible under realistic memory budgets. For example, the MC9-selected FlexMoRE-a2 configuration reduces stored parameters by 67.7%, from 33.27B to 10.75B, without expert retraining or additional router tuning, while achieving an eight-benchmark average of 40.75 versus 40.77 for dense FlexOlmo-a2; excluding MC9, the averages are 36.80 versus 37.65.

  1. Heuristic rank selection.

2.a. Heterogeneous allocation: Heterogeneous allocation allows different domain experts to retain different ranks, rather than assigning one common rank to every expert. For each heterogeneous configuration, each domain expert retains its selected rank from Table 5 while we vary the number of active experts per token between 2, 4, and 7 (a2, a4, and a7). The MC9-selected FlexMoRE-a2 configuration achieves an eight-benchmark average of 40.75 versus 40.77 for dense FlexOlmo-a2, while reducing stored parameters from 33.27B to 10.75B—a 67.7% reduction. This demonstrates a useful storage–performance trade-off, although aggregate performance does not imply uniform retention across benchmarks.

2.b. Selection and evaluation: We evaluate each domain expert alongside the public expert in the individual two-expert configuration at candidate ranks 1, 2, 4, …, 2048. For MC9-selected mixtures, we choose each expert’s rank by its highest MC9 score. For All-selected mixtures, we choose it by its highest macro average across the original six benchmarks: MC9, GEN5, AGIEval, BBH, MMLU, and MMLU-Pro. Neither criterion jointly optimizes ranks in the composed model. The selected ranks remain unchanged across a2, a4, and a7. Math2 and Code4 were added afterward and were not used for either selection. For MC9-selected configurations, we also report the average over the seven benchmarks excluding MC9, none of which was used in that rank-selection criterion. We do not claim transfer to new experts or a new task suite.

2.c. Domain-specific behaviour: The aggregate scores do not imply uniform capability retention. These allocations were fixed before adding Math2 and Code4 to the evaluation. The resulting mixtures therefore provide evidence beyond the benchmarks used for allocation, although capability retention varies by benchmark.

Our full homogeneous rank sweeps provide evidence for the usefulness of heterogeneous allocation: for example, the MC9-selected heterogeneous a2 mixture achieves an eight-benchmark average of 40.75 with 10.75B parameters, versus 37.00 for homogeneous rank 512 with 11.75B parameters. However, we have not compared alternative heterogeneous allocations at the same storage budget.

  1. Domain-specific losses.

We agree that competitive aggregate performance does not imply preservation of every specialist capability. In particular, the Code4 losses in the selected mixtures are substantial. Our rank sweeps show that capacity-sensitive tasks can require larger retained ranks, making the allocation criterion consequential. Heterogeneous composition permits allocating more capacity to such experts, potentially retaining a dense expert where compression is unsuitable. However, we have not established that these alternatives recover the losses at the same storage budget.

  1. Architectural scope.

Our experiments deliberately focus on FlexOlmo’s independently trained expert architecture. The demonstrated contribution is post-hoc compression within this setting; transfer to other expert collections remains untested.

Thanks to the authors for the response. However, I find my concerns remain unresolved.

My novelty concern was not specifically about truncated SVD itself, but it was more about the methodological contribution of the overall approach beyond combining existing techniques. I find the authors mainly emphasize the practical benefits without sufficiently addressing this concern. Also, the claim that existing LoRA-based MoE methods jointly train experts and routers significantly overlooks substantial prior work on independently trained experts.

Second, explaining how the ranks are selected does not justify the effectiveness of the heuristic. As the authors acknowledge, I see comparisons with alternative allocations under matched budgets are still missing.

Finally, the substantial performance drops on certain benchmarks remain concerning, and the suggested remedies are not experimentally validated.

I don't think the rebuttal sufficiently addresses my concerns. I failed to find sufficient justification to raise the score above 2. Thus, I intend to maintain my original score.

We thank the reviewer for clarifying their perspective on novelty. However, we would argue that beyond showing how dense experts can be decomposed post hoc while preserving dynamic conditional routing without retraining, our main contribution and claim to novelty is an empirical characterization of training-free low-rank sensitivity across 132 tasks. Specifically, we find that compressibility is sharply domain-dependent: reasoning-heavy tasks (Math2, Code4, BBH) degrade rapidly under truncation, whereas knowledge-heavy benchmarks (GEN5, MC9) remain stable at low ranks. The Code4 and Math2 regressions thus present not only limitations but core results of our experiments.

Regarding rank allocation, the combinatorial search space spans nearly 3 million heterogeneous configurations, making exhaustive evaluation prohibitively expensive (requiring tens of millions of GPU hours). While optimal global allocation remains an open problem, our 15-rank homogeneous sweep demonstrates that heterogeneous allocation achieves substantially stronger aggregate retention at much fewer parameters than uniform ranks.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment

We thank the reviewers for their input and suggestions and for engaging in the discussion. We appreciated the opportunity to address their concerns through additional experiments and clarifications, and their recognition of FlexMoRE’s practical value.

We have added the additional comparisons requested by the reviewers requested comparisons regarding other compression methods. Specifically, we added experiments with 2-bit and 4-bit quantization, showing that the 4-bit configurations largely preserve aggregate performance, whereas 2-bit quantization causes substantial degradation. Crucially, quantization can be combined with the low-rank experts in FlexMoRE. The results displaying that even this hybrid method (quantization + low-ranking) does not degrade the mixture’s performance are included in the revised version of the manuscript.

The reviewers also requested comparison with directly trained LoRA experts. Here, we would like to emphasize our comparison with AdapterSoup that we already had in the initial submission (Fig. 2, rightmost panel, and Tab. 2). Our results there display much better scaling behavior of FlexMoRE, which hints at destructive interference happening at higher ranks in AdapterSoup, but not in FlexMoRE.

Regarding the rank selection, we clarified the original six-benchmark All-selection criterion and the evaluation excluding MC9. Our sweeps provide a concrete heterogeneous operating point: 40.75 with 10.75B parameters versus 37.00 with 11.75B for homogeneous rank 512. This supports the usefulness of heterogeneous allocation, while finding the best allocation at an exactly matched budget remains future work.

One reviewer also pointed out the decline of Math performance under MC9, which we attribute to the MC9 calibration set not including any math data. Regarding capability-specific losses, we clarified that our results indicate potential for improved rank allocations, while developing better allocation heuristics or systematic selection methods remains beyond the scope of this work.

Together, these results demonstrate the practical flexibility of compressing existing, independently trained experts without repeating expert training or accessing their original data, while making the capability trade-offs explicit.

EDIT: The new results are integrated into the paper and can be found in the revised PDF ("rebuttal version") of the submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.