- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents FlexMoRE, a post-hoc compression approach for independently trained domain experts in FlexOlmo. Each domain expert is represented as a low-rank weight update relative to a shared public model through truncated SVD. The method retains the existing routing mechanism and assigns expert-specific ranks based on benchmark performance. Experiments investigate rank sensitivity, heterogeneous capacity allocation, and the resulting trade-offs between model storage and downstream performance.
Critical strengths
The method provides a practical way to compress an existing expert collection without retraining the experts or tuning the router. The experiments demonstrate substantial storage savings and include actual GPU memory measurements. In the evaluated implementation, the MC9-selected configuration reduces resident GPU memory from 62.74 GiB to 20.94 GiB. This supports its practical value when pretrained experts are already available and additional training is undesirable.
Critical weaknesses
-
The proposed pipeline relies on fully trained domain-specific FFNs before approximating their weight updates through truncated SVD. A natural alternative is to train domain-specific LoRA updates directly against the frozen public model. Although post-hoc compression has a clear advantage when expert checkpoints already exist and training data are unavailable, the paper does not establish whether this route offers a favorable quality–cost trade-off when developing a new expert collection.
-
The compression baselines are insufficient to establish the competitiveness of the proposed approach. They do not determine whether this approach preserves more useful capability than other compression strategies at a comparable storage budget, such as expert-weight quantization or another low-rank approximation method。
Questions for authors
-
FlexMoRE first relies on fully trained domain-specific FFNs and then approximates their updates through low-rank factorization. An alternative is to train domain-specific LoRA updates against the same frozen public model from the outset. I recognize that these approaches address different starting conditions. Post-hoc compression can reuse existing checkpoints without accessing domain training data, whereas direct LoRA training requires such access. Nevertheless, a controlled comparison would clarify when the proposed route is preferable.
-
Could the authors include a relevant compression baseline, such as expert-weight quantization or another low-rank approximation method, while preserving the independently addressable experts and routing mechanism? Comparing downstream quality at similar storage budgets in bytes, together with measured GPU memory and inference throughput, would help establish whether post-hoc factorization offers a competitive practical trade-off.
7 comments
Quantization Baseline
Following up on point 2 of our response, we have now completed additional quantization experiments. We thank the reviewer for suggesting this comparison.
We added expert-weight quantization baselines at 2 and 4 bits, with and without quantizing the public model, see results in the Table below. Four-bit quantization of the experts while retaining the public base in BF16 nearly preserves the dense baseline’s aggregate scores (36.44 vs. 36.46 with MC9; 32.31 vs. 32.32 without MC9), whereas 2-bit quantization causes substantial regressions, resulting in aggregate scores of 11.65 and 8.27. This establishes 4-bit quantization as a meaningful compression baseline while illustrating the quality cost of more aggressive quantization.
We further evaluate combining quantization with FlexMoRE’s low-rank representation. The hybrid configurations achieve aggregate scores within 0.03 percentage points of BF16 FlexMoRE: 39.13/34.91 for q4-no-base and 39.12/34.90 for q4-all, compared with 39.15/34.92 for FlexMoRE (with/without MC9). Their corresponding estimated storage is 16.54 GB and 10.32 GB, versus 21.49 GB for BF16 FlexMoRE. These results show that four-bit quantization nearly preserves the aggregate performance of the reconstructed low-rank model. The estimated storage savings for quantized A/B factors remain to be validated together with performance in a factorized implementation.
Table: Quantization and hybrid-storage configurations. All configurations use seven active experts per token (a7). GB denotes weight storage, including quantization scales, rather than measured inference memory. q2 and q4 denote 2-bit and 4-bit quantization. “no-base” retains the public model in BF16; “all” includes the public model in quantization. All rows are complete (132/132 tasks). Avg. w/ MC9 is the macro average across the eight benchmarks; Avg. w/o MC9 excludes MC9.
| Configuration | Params | GB | MC9 | Gen5 | MMLU | MMLU-Pro | AGIEval | BBH | Math2 | Code4 | Avg. w/ MC9 | Avg. w/o MC9 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Dense FlexOlmo-a7 BF16 | 33.27B | 66.54 | 65.46 | 44.56 | 50.89 | 22.00 | 37.37 | 35.46 | 25.42 | 10.56 | 36.46 | 32.32 |
| q2-all | 33.27B | 15.41 | 35.33 | 0.00 | 23.12 | 11.71 | 23.03 | 0.00 | 0.00 | 0.00 | 11.65 | 8.27 |
| q2-no-base | 33.27B | 22.72 | 35.33 | 0.00 | 23.12 | 11.71 | 23.03 | 0.00 | 0.00 | 0.00 | 11.65 | 8.27 |
| q4-all | 33.27B | 22.99 | 65.03 | 45.40 | 50.96 | 21.82 | 37.24 | 34.31 | 24.86 | 10.54 | 36.27 | 32.16 |
| q4-no-base | 33.27B | 29.21 | 65.36 | 44.89 | 51.01 | 21.91 | 37.59 | 34.86 | 25.27 | 10.63 | 36.44 | 32.31 |
| FlexMoRE MC9 BF16 | 10.75B† | 21.49† | 68.73 | 56.69 | 53.87 | 24.45 | 41.02 | 40.65 | 25.84 | 1.95 | 39.15 | 34.92 |
| Hybrid MC9 q4-no-base | 10.75B* | 16.54* | 68.71 | 56.60 | 53.69 | 24.45 | 40.92 | 40.75 | 26.15 | 1.78 | 39.13 | 34.91 |
| Hybrid MC9 q4-all | 10.75B* | 10.32* | 68.68 | 56.41 | 53.79 | 24.53 | 41.03 | 40.33 | 26.28 | 1.92 | 39.12 | 34.90 |
* Hypothetical quantized A/B storage; scores use dense reconstruction then quantization.
† BF16 A/B storage; actual checkpoint contains dense reconstructed weights.
Thank you for your response and the additional experiments. However, I still have a few concerns:
-
This clarification highlights a specific practical scenario: the method is useful when pre-trained expert models are available, but their original training data is inaccessible. Nevertheless, in cases where both approaches are feasible, it remains unclear how this method compares to training LoRA-based expert models directly in terms of cost. Consequently, I am not certain what the specific advantage of this approach is when building an entirely new set of expert models.
-
The comparison regarding quantization is quite informative; however, the aggregated scores mask significant disparities in the preservation of model capabilities. While FlexMoRE improves the MMLU score from 50.89 to 53.87, it causes a sharp decline in the Code4 score—from 10.56 down to 1.95. In contrast, applying 4-bit quantization to the expert models preserves performance across both metrics much more effectively. Could the authors please explain this trade-off and clarify under what circumstances choosing FlexMoRE would be preferable to direct quantization?
We thank the reviewer for their service to our work. We notice that the reviewer emphasize that our method is a practical implementation, that we validate with substantial storage savings, when pretrained experts are already available and additional training is undesirable. We address each concern below.
Post-hoc extraction versus directly trained LoRA experts. We agree that directly trained LoRA experts are a relevant alternative, and that a controlled comparison would clarify the quality–cost trade-off when both approaches are feasible. However, it would be computationally expensive to conduct such a comparison as systematic as our post-hoc analysis with varying ranks between 1 and 2048.
Post-hoc extraction also allows the composer to explore different retained ranks per expert without repeating expert training. This flexibility matters because our results show that rank sensitivity differs across experts and evaluation tasks (Sec. 5 - Rank Sensitivity Analysis). We further demonstrated that FlexMore outperforms the AdapterSoup baseline across the board in Table 2 and Figure 2c.
Moreover, our setting that starts from independently contributed dense expert checkpoints comes with conceptual advantages: A provider of an expert may be unable or unwilling to share the original training data with the party composing the mixture. The composer may therefore be unable to train replacement LoRA experts, whereas post-hoc extraction can compress the existing checkpoints without accessing those data or tuning the inherited router.
We do not claim that post-hoc extraction is preferable when training data and compute are available and agree that a controlled comparison – while expensive – would be valuable. We will clarify this distinction in the paper and mention the LoRA training as future work.
We agree that quantization is a relevant alternative and potentially complementary approach. A controlled comparison is not yet complete, and therefore we do not claim superiority over quantization or other compression methods. We are investigating this comparison and hope to share results during the open-discussion period.
An illustrative calculation provides context for the storage budget. Assuming BF16 for the public model, and treating the parameters beyond the public model as quantizable, matching the MC9-selected configuration corresponds to an average of approximately 2.1 bits per parameter for those additional parameters: (16 \times (10.75-7.30)/(33.27-7.30) \approx 2.13). This estimate uses rounded parameter counts and excludes quantization metadata and other overhead. It is not a measured quantization result and does not establish a quantized model’s quality, resident memory, or throughput.
Our experiments do demonstrate a memory–throughput trade-off for the existing expert collection: in the tested Hugging Face eager backend, the MC9-selected configuration reduces resident GPU memory from 62.74 to 20.94 GiB (−66.6%), while reaching 0.88× the dense baseline’s throughput. A matched-byte comparison that preserves the expert branches and router, and measures quality, memory, and throughput, would be valuable future work.
Support of calculation: The calculation asks: if we leave the public model in BF16, how many bits could we spend on each additional dense-model parameter while matching FlexMoRE's raw weight storage?
• 16: bits per BF16 parameter. • 33.27B: total stored parameters in the dense FlexOlmo mixture. • 10.75B: total stored parameters in the MC9-selected FlexMoRE mixture. • 7.30B: parameters in the public base model, assumed to remain in BF16
in both representations.
Subtracting the shared public model gives: • 33.27 - 7.30 = 25.97B additional parameters in the dense mixture. • 10.75 - 7.30 = 3.45B BF16-equivalent parameters available for the
compressed representation.
The available storage per additional dense parameter is therefore: 16 × (3.45 / 25.97) ≈ 2.13 bits per parameter.
The counts are rounded, and the additional parameters may include smallbnon-expert components such as router parameters. This is an illustrative raw-storage calculation, not a measured quantization result; it excludes metadata and implementation overhead.