- Rating
- 3(−1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper formulates a novel post-merge quantization (PMQ) setting, where a low-bit model is deployed after merging multiple expert models. The authors observe that the merged model may already contain expert-relative merging deviations and therefore may not serve as a reliable reconstruction target for standard PTQ. To address this issue, they propose E-PMQ, which incorporates expert-guided targets into the layer-wise quantization objective. Experimental results across vision and language models demonstrate the effectiveness and efficiency of E-PMQ.
Critical strengths
(+) The paper is strongly motivated by the post-merge quantization setting, where source experts are reused during low-bit calibration of an already merged model. This formulation addresses a practically relevant gap between model merging and PTQ, which seems to be a clear contribution for me.
(+) The authors demonstrate the effectiveness of E-PMQ against alternative pipelines for obtaining a low-bit model from multiple experts (Tables 6 and 7). In particular, E-PMQ achieves better accuracy than the compared expert-aware alternatives while maintaining competitive calibration cost than some baselines. In addition, no extra inference-time module is required for E-PMQ.
(+) The experimental evaluation is extensive, covering diverse architectures across both vision and language domains.
Critical weaknesses
[Validity]
(-) As acknowledged by the authors in App. A, E-PMQ requires access to all source expert checkpoints during post-merge calibration. This requirement may limit its practical applicability when only the merged model is available, or when retaining and accessing a large number of source experts is costly.
(-) E-PMQ relies on calibration-derived activation statistics in Sec. 4.4, since both and depend on . It is therefore unclear how robust the method is when only a limited amount of per-expert calibration data is available.
[Experiments]
(-) The experiments are limited to weight-only quantization. Since post-merge quantization in practical settings may also involve activation quantization, it remains unclear whether E-PMQ is effective under weight-activation quantization settings.
[Presentation]
(-) Sec. 4.4 is difficult to follow without referring to the appendix. In particular, the derivation of and are not sufficiently explained in the main text. A brief derivation or more intuitive explanation would improve the readability of the method.
Questions for authors
Please see the weaknesses.
4 comments
Dear Reviewer,
I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.
Thank you for your time and effort in reviewing our paper.
Thank you for the detailed responses and additional experiments. The calibration-budget analysis and the expanded derivation adequately address my concerns regarding W2 and W4.
However, W1 and W3 remain unresolved. While sequential expert loading reduces GPU memory requirements, the method still requires access to all source expert checkpoints, limiting its practical applicability. In addition, the effectiveness of E-PMQ under weight-activation quantization remains unverified, as acknowledged by the authors.
I also appreciate the additional comparisons with stronger merged models. Nevertheless, after considering the authors' responses and the concerns raised by other reviewers, I believe these remaining limitations justify a more moderate rating.
Therefore, I have decided to lower my rating from 4 (Clear Acceptance) to 3 (Weak Acceptance).
Thank you again for your valuable feedback.
For W1, E-PMQ addresses the PMQ setting for developers who retain source checkpoints and need to produce a low-bit merged model. This access requirement limits its applicability but does not diminish its practical value within that setting. Model merging is already used in industrial model development:
- Cohere — Command A: combines capability experts through linear weight averaging after SFT and offline preference optimization (Sec. 3.4.3) [1].
- Alibaba’s Qwen team — Qwen3-Embedding: merges candidate models in the final training stage to improve overall performance [2].
- Preferred Networks / Preferred Elements — PLaMo 2: merges separately fine-tuned general-purpose and math/code models, and performs further merging after DPO (Secs. 3.3.2–3.3.3) [3].
- NVIDIA — Nemotron 3 Ultra: merges pretraining checkpoints and selects a merged checkpoint for subsequent long-context training (Sec. 2.4) [4].
- SK Telecom — A.X K2: averages the last six pretraining checkpoints to produce the released base model (Sec. 3.5) [5].
These examples cover both separately trained model merging and training-checkpoint averaging. PLaMo 2 and A.X K2 also document low-precision deployment, demonstrating a practical need to quantize models produced through merging. E-PMQ targets this need when source checkpoints are available. Further discussion of industrial workflows, checkpoint availability, and applications is provided in our response to Reviewer hBby, W3.
For W3, our work focuses on weight-only PMQ. We view joint weight–activation calibration as an extension of this setting that requires further design and evaluation. We will clarify this scope and investigate the extension in future work.
We have also provided detailed responses to the concerns raised by the other reviewers. We hope these responses address your remaining questions, and we would welcome the opportunity to clarify any unresolved concerns.
We kindly ask you to reconsider your score in light of these clarifications and the additional evidence. We would be happy to continue the discussion.
References
[1] Cohere. Command A: An Enterprise-Ready Large Language Model. Sec. 3.4.3. Technical report.
[2] Qwen Team. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. “Model Training.” Official report.
[3] Preferred Networks / Preferred Elements. PLaMo 2 Technical Report. Secs. 3.3.2–3.3.3 and 4.2.1. Technical report.
[4] NVIDIA. Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. Sec. 2.4. Technical report.
[5] SK Telecom. A.X K2 Technical Report. Secs. 3.5 and 6.1. Technical report.
Rebuttal by Authors Part 1/1
Overall Response
We sincerely appreciate your positive assessment of our paper. We are glad you found the post-merge quantization setting practically relevant and highlighted E-PMQ’s accuracy gains over expert-aware baselines, competitive calibration cost, and lack of extra inference modules across a broad evaluation in vision and language.
Our response
E-PMQ requires all source expert checkpoints and representative calibration inputs during quantization. When only the merged checkpoint is available, conventional PTQ remains applicable. Once quantization is complete, deployment requires only the resulting low-bit checkpoint.
To reduce GPU memory pressure during calibration, E-PMQ can load experts sequentially. In our eight-expert CLIP ViT-B/32 experiment, sequential loading reduced peak allocated GPU memory from 9.69 to 2.38 GiB, while increasing processing time from 53.84 to 76.65 s and using 6.36 GiB of temporary storage. The source checkpoints still need to be stored and read during calibration.
This trade-off is most relevant to teams that already have the source checkpoints when preparing a low-bit release. For example, PLaMo 2.0-31B merges separately fine-tuned checkpoints and then applies INT4 GPTQ [1]. In our own 4-bit Llama-3.1-8B experiment, using the source experts for calibration with E-PMQ raised the mean score across six benchmarks from 61.66 with direct GPTQ to 62.91 with E-PMQ.
Our response
E-PMQ retains its accuracy advantage over GPTQ at all three tested calibration budgets. Table 25 in Appendix J reports a calibration-budget sweep for eight-task CLIP ViT-B/32 with Task Arithmetic. Entries are mean accuracies (%).
Reducing the budget from 256 to 64 samples per task lowers E-PMQ accuracy by 1.38 points. It still exceeds GPTQ by 7.37 points at the same budget. At 128 samples per task, it is within 0.03 points of its result at 256.
Our response
Thank you for your thoughtful feedback. Our experiments focus on weight-only quantization. Quantizing activations changes the layer inputs and their statistics, introducing an additional source of error. We plan to explore whether E-PMQ can be extended to joint weight and activation quantization in future work.
Our response
Thank you for pointing this out. We have added the following derivation to Sec. 4.4 to show how the layer-wise objective leads to the effective curvature, the expert-guided right-hand side, and the low-bit solver.
Expand the layer-wise objective. Omitting the layer index, E-PMQ minimizes L(Q)=∑i=1K∥(Q−Wi)Xi∥F2+λ∥Q−Wm∥F2,Q∈Qb.
Here �0� contains the inputs to the current layer of the partially quantized merged model on expert �1�'s calibration samples. We evaluate the expert weight �2� on those same inputs to obtain the target �3�. The anchor �4� is the full-precision merged weight. With �5�, expanding the squares gives L(Q)=Tr(QHQ⊤)−2Tr(QR⊤)+C,H=∑iHi+λI,R=∑iWiHi+λWm,
where �6� is independent of �7�. The anchor contributes �8� to the effective curvature �9� and $λWm $ to the right-hand side �11�.
Find the continuous target. Temporarily relaxing the low-bit constraint and setting the gradient �12� to zero gives $(QH=R)$. For �13�,�14� is positive definite, so the continuous minimizer is �15�.
Recover the low-bit weights. Completing the square and restoring the low-bit constraint gives the equivalent problem minQ∈QbTr[(Q−Qcont)H(Q−Qcont)⊤]. We use curvature-aware sequential rounding to approximately solve this problem. The continuous solution $Qcont $provides the reconstruction target, and �17� determines how deviations from it are penalized.
We sincerely thank you again for your positive assessment of our paper. We are glad to answer any further question.
Changes in the revised PDF
• W4 — Sec. 4.4 (p. 5) adds the full derivation of the layer-wise optimization procedure.
References