acceptodds
Under review as a conference paper at ICLR 2027

E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoring

Abstract

Low-resource deployment constraints have made model quantization essential for deploying neural networks while preserving performance. Meanwhile, model merging has become an increasingly practical low-resource strategy for integrating multiple task- or domain-specialized experts into a single model without joint training or multi-model serving. Together, quantization and model merging enable an efficient low-resource deployment pipeline by integrating multiple experts into one low-bit model. We formulate this setting as Post-Merge Quantization (PMQ). We show that directly applying post-training quantization (PTQ) to a merged model is unreliable because two distinct deviations are coupled: the quantization deviation introduced by low-bit reconstruction and the expert-relative merging deviation inherited from model merging. To mitigate these deviations, we propose E-PMQ, an expert-guided PMQ framework that uses source expert weights to provide expert-guided output targets during layer-wise calibration, together with merged-weight anchoring to stabilize the calibration and preserve the integrated behavior of the merged model. Experiments on CLIP, FLAN-T5, and Llama show consistent gains over naive PMQ across vision and language, from 8-task CLIP-ViT-B/32 to 20-task CLIP-ViT-L/14 and up to 8B-scale LLMs.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
quantization, model merge, CLIP, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QZw2OpenReview ↗
Rating
3(−1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper formulates a novel post-merge quantization (PMQ) setting, where a low-bit model is deployed after merging multiple expert models. The authors observe that the merged model may already contain expert-relative merging deviations and therefore may not serve as a reliable reconstruction target for standard PTQ. To address this issue, they propose E-PMQ, which incorporates expert-guided targets into the layer-wise quantization objective. Experimental results across vision and language models demonstrate the effectiveness and efficiency of E-PMQ.

Critical strengths

(+) The paper is strongly motivated by the post-merge quantization setting, where source experts are reused during low-bit calibration of an already merged model. This formulation addresses a practically relevant gap between model merging and PTQ, which seems to be a clear contribution for me.

(+) The authors demonstrate the effectiveness of E-PMQ against alternative pipelines for obtaining a low-bit model from multiple experts (Tables 6 and 7). In particular, E-PMQ achieves better accuracy than the compared expert-aware alternatives while maintaining competitive calibration cost than some baselines. In addition, no extra inference-time module is required for E-PMQ.

(+) The experimental evaluation is extensive, covering diverse architectures across both vision and language domains.

Critical weaknesses

[Validity]
(-) As acknowledged by the authors in App. A, E-PMQ requires access to all source expert checkpoints during post-merge calibration. This requirement may limit its practical applicability when only the merged model is available, or when retaining and accessing a large number of source experts is costly.

(-) E-PMQ relies on calibration-derived activation statistics in Sec. 4.4, since both HE-PMQℓH_{\mathrm{E\text{-}PMQ}}^\ell and RE-PMQℓR_{\mathrm{E\text{-}PMQ}}^\ell depend on Hiℓ=Xiℓ(Xiℓ)⊤H_i^\ell=X_i^\ell(X_i^\ell)^\top. It is therefore unclear how robust the method is when only a limited amount of per-expert calibration data is available.

[Experiments]
(-) The experiments are limited to weight-only quantization. Since post-merge quantization in practical settings may also involve activation quantization, it remains unclear whether E-PMQ is effective under weight-activation quantization settings.

[Presentation]
(-) Sec. 4.4 is difficult to follow without referring to the appendix. In particular, the derivation of HE-PMQH_{\mathrm{E\text{-}PMQ}} and RE-PMQR_{\mathrm{E\text{-}PMQ}} are not sufficiently explained in the main text. A brief derivation or more intuitive explanation would improve the readability of the method.

Questions for authors

Please see the weaknesses.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors Part 1/1

Overall Response

We sincerely appreciate your positive assessment of our paper. We are glad you found the post-merge quantization setting practically relevant and highlighted E-PMQ’s accuracy gains over expert-aware baselines, competitive calibration cost, and lack of extra inference modules across a broad evaluation in vision and language.

W1. As acknowledged by the authors in App. A, E-PMQ requires access to all source expert checkpoints during post-merge calibration...

Our response

E-PMQ requires all source expert checkpoints and representative calibration inputs during quantization. When only the merged checkpoint is available, conventional PTQ remains applicable. Once quantization is complete, deployment requires only the resulting low-bit checkpoint.

To reduce GPU memory pressure during calibration, E-PMQ can load experts sequentially. In our eight-expert CLIP ViT-B/32 experiment, sequential loading reduced peak allocated GPU memory from 9.69 to 2.38 GiB, while increasing processing time from 53.84 to 76.65 s and using 6.36 GiB of temporary storage. The source checkpoints still need to be stored and read during calibration.

This trade-off is most relevant to teams that already have the source checkpoints when preparing a low-bit release. For example, PLaMo 2.0-31B merges separately fine-tuned checkpoints and then applies INT4 GPTQ [1]. In our own 4-bit Llama-3.1-8B experiment, using the source experts for calibration with E-PMQ raised the mean score across six benchmarks from 61.66 with direct GPTQ to 62.91 with E-PMQ.

W2. E-PMQ relies on calibration-derived activation statistics in Sec. 4.4...

Our response

E-PMQ retains its accuracy advantage over GPTQ at all three tested calibration budgets. Table 25 in Appendix J reports a calibration-budget sweep for eight-task CLIP ViT-B/32 with Task Arithmetic. Entries are mean accuracies (%).

Samples per taskGPTQE-PMQ
6464.8672.23
12864.7773.58
25665.0373.61

Reducing the budget from 256 to 64 samples per task lowers E-PMQ accuracy by 1.38 points. It still exceeds GPTQ by 7.37 points at the same budget. At 128 samples per task, it is within 0.03 points of its result at 256.

W3. It remains unclear whether E-PMQ is effective under weight-activation quantization settings.

Our response

Thank you for your thoughtful feedback. Our experiments focus on weight-only quantization. Quantizing activations changes the layer inputs and their statistics, introducing an additional source of error. We plan to explore whether E-PMQ can be extended to joint weight and activation quantization in future work.

W4. Sec. 4.4 is difficult to follow without referring to the appendix.

Our response

Thank you for pointing this out. We have added the following derivation to Sec. 4.4 to show how the layer-wise objective leads to the effective curvature, the expert-guided right-hand side, and the low-bit solver.

  1. Expand the layer-wise objective. Omitting the layer index, E-PMQ minimizes L(Q)=∑i=1K∥(Q−Wi)Xi∥F2+λ∥Q−Wm∥F2,Q∈Qb.\mathcal L(Q)=\sum_{i=1}^{K}\|(Q-W_i)X_i\|_F^2 +\lambda\|Q-W_m\|_F^2, \qquad Q\in\mathcal Q_b.

    Here �0��0� contains the inputs to the current layer of the partially quantized merged model on expert �1��1�'s calibration samples. We evaluate the expert weight �2��2� on those same inputs to obtain the target �3��3�. The anchor �4��4� is the full-precision merged weight. With �5��5�, expanding the squares gives L(Q)=Tr⁡(QHQ⊤)−2Tr⁡(QR⊤)+C,H=∑iHi+λI,R=∑iWiHi+λWm,\mathcal L(Q)=\operatorname{Tr}(QHQ^\top) -2\operatorname{Tr}(QR^\top)+C,\qquad H=\sum_i H_i+\lambda I, \qquad R=\sum_i W_iH_i+\lambda W_m,

    where �6��6� is independent of �7��7�. The anchor contributes �8��8� to the effective curvature �9��9� and $λWm\lambda W_m $ to the right-hand side �11��11�.

  2. Find the continuous target. Temporarily relaxing the low-bit constraint and setting the gradient �12��12� to zero gives $(QH=R)$. For �13�,�14��13�, �14� is positive definite, so the continuous minimizer is �15��15�.

  3. Recover the low-bit weights. Completing the square and restoring the low-bit constraint gives the equivalent problem min⁡Q∈QbTr⁡ ⁣[(Q−Qcont)H(Q−Qcont)⊤].\min_{Q\in\mathcal Q_b} \operatorname{Tr}\!\left[(Q-Q_{\mathrm{cont}})H(Q-Q_{\mathrm{cont}})^\top \right]. We use curvature-aware sequential rounding to approximately solve this problem. The continuous solution $QcontQ_{\mathrm{cont}} $provides the reconstruction target, and �17��17� determines how deviations from it are penalized.


We sincerely thank you again for your positive assessment of our paper. We are glad to answer any further question.


Changes in the revised PDF

• W4 — Sec. 4.4 (p. 5) adds the full derivation of the layer-wise optimization procedure.

References

  • [1] Preferred Networks. PLaMo 2 Technical Report. Technical report, 2025.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Thank you for the detailed responses and additional experiments. The calibration-budget analysis and the expanded derivation adequately address my concerns regarding W2 and W4.

However, W1 and W3 remain unresolved. While sequential expert loading reduces GPU memory requirements, the method still requires access to all source expert checkpoints, limiting its practical applicability. In addition, the effectiveness of E-PMQ under weight-activation quantization remains unverified, as acknowledged by the authors.

I also appreciate the additional comparisons with stronger merged models. Nevertheless, after considering the authors' responses and the concerns raised by other reviewers, I believe these remaining limitations justify a more moderate rating.

Therefore, I have decided to lower my rating from 4 (Clear Acceptance) to 3 (Weak Acceptance).

Thank you again for your valuable feedback.

For W1, E-PMQ addresses the PMQ setting for developers who retain source checkpoints and need to produce a low-bit merged model. This access requirement limits its applicability but does not diminish its practical value within that setting. Model merging is already used in industrial model development:

  • Cohere — Command A: combines capability experts through linear weight averaging after SFT and offline preference optimization (Sec. 3.4.3) [1].
  • Alibaba’s Qwen team — Qwen3-Embedding: merges candidate models in the final training stage to improve overall performance [2].
  • Preferred Networks / Preferred Elements — PLaMo 2: merges separately fine-tuned general-purpose and math/code models, and performs further merging after DPO (Secs. 3.3.2–3.3.3) [3].
  • NVIDIA — Nemotron 3 Ultra: merges pretraining checkpoints and selects a merged checkpoint for subsequent long-context training (Sec. 2.4) [4].
  • SK Telecom — A.X K2: averages the last six pretraining checkpoints to produce the released base model (Sec. 3.5) [5].

These examples cover both separately trained model merging and training-checkpoint averaging. PLaMo 2 and A.X K2 also document low-precision deployment, demonstrating a practical need to quantize models produced through merging. E-PMQ targets this need when source checkpoints are available. Further discussion of industrial workflows, checkpoint availability, and applications is provided in our response to Reviewer hBby, W3.

For W3, our work focuses on weight-only PMQ. We view joint weight–activation calibration as an extension of this setting that requires further design and evaluation. We will clarify this scope and investigate the extension in future work.

We have also provided detailed responses to the concerns raised by the other reviewers. We hope these responses address your remaining questions, and we would welcome the opportunity to clarify any unresolved concerns.


We kindly ask you to reconsider your score in light of these clarifications and the additional evidence. We would be happy to continue the discussion.


References

[1] Cohere. Command A: An Enterprise-Ready Large Language Model. Sec. 3.4.3. Technical report.

[2] Qwen Team. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. “Model Training.” Official report.

[3] Preferred Networks / Preferred Elements. PLaMo 2 Technical Report. Secs. 3.3.2–3.3.3 and 4.2.1. Technical report.

[4] NVIDIA. Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. Sec. 2.4. Technical report.

[5] SK Telecom. A.X K2 Technical Report. Secs. 3.5 and 6.1. Technical report.

Reviewer_i9a4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies post-merge quantization, where multiple specialized models are merged and subsequently compressed into a single low-bit model. E-PMQ replaces ordinary merged-model reconstruction with expert-guided layer-output matching and regularizes quantized weights toward the merged checkpoint. The resulting quadratic objective is optimized using GPTQ-style sequential rounding. Experiments cover CLIP, FLAN-T5, and Llama models, including comparisons with direct quantization, quantize-then-merge, and representation repair followed by quantization. The paper addresses a useful deployment setting and provides extensive supplementary analysis. However, stronger controls are needed to distinguish its contribution from regularized regression-based model merging followed by quantization.

Critical strengths
  1. Experiments are extensive. Evaluation spans vision and language, multiple merging methods, larger task suites, and models up to 8B parameters. Comparisons with HDRQ, Surgery+GPTQ, TSV, and TVQ strengthen the empirical study.

  2. Improvements in several settings. Improvements over direct GPTQ are large for weaker CLIP mergers. Results on WUDI, TSV, and Llama also help characterize settings where gains are smaller.

  3. Detailed calibration-cost analysis. The appendix reports sequential expert loading, temporary storage, calibration budgets, and complete-pipeline costs, making the resource trade-offs more transparent.

Critical weaknesses
  1. About experimental results:

1.1 Under eight-task CLIP-B/32 Task Arithmetic, E-PMQ improves average accuracy but reduces EuroSAT from 76.7% to 66.3% and GTSRB from 68.5% to 57.0%. In the 20-task CLIP-L/14 WUDI results, FER2013 decreases from 64.2% to 40.44%, while KMNIST falls from 33.3% to 17.02%. Such regressions matter for a method motivated by preserving integrated expert capabilities. Reporting worst-task retention, normalized expert-relative accuracy, and the number of improved versus degraded tasks would provide a more complete assessment. The paper should qualify claims that anchoring preserves integrated behavior.

1.2 Table 22 reports complete-pipeline E-PMQ times of 53.84 seconds with all-at-once loading and 76.65 seconds with sequential loading. Table 23 reports 172.2 seconds at 256 samples per task. These appear to concern closely related eight-task CLIP settings, so the configuration or measurement differences should be explained.

1.3 The full-precision GTSRB expert accuracy also differs between Table 1’s fine-tuned reference and Table 10’s HDRQ expert reference, at 98.9% and 90.35%, respectively. Please clarify whether different checkpoints or evaluation protocols are involved.

  1. About method:

2.1 E-PMQ uses expert information to improve the merged model during quantization. However, it is unclear whether similar gains could be achieved by first repairing the full-precision model using the same expert information, then applying GPTQ. A matched comparison would clarify the benefit of combining these steps.

2.2 Removing the anchor causes accuracy to collapse, but the anchor both keeps weights near the merged model and stabilizes numerical optimization. The collapse could therefore reflect numerical instability rather than excessive expert guidance. Please separate these two effects.

2.3 1. Some full-precision merged models perform poorly, leaving substantial room for repair. Please evaluate better-tuned merged models to show whether E-PMQ remains effective when the starting model is already strong.

  1. Minor issues:

3.1 How calibration samples are assigned to experts when domains overlap or task identities are unavailable?

3.2 Clarify whether the layer-wise residual comparison evaluates both methods on identical activations or on their respective trajectories.

3.3 Explicitly define the metric used for each GLUE task, especially CoLA and MRPC/QQP.

Questions for authors

See Weaknesses.

8 comments
AuthorsOpenReview ↗

Rebuttal by Authors Part 1/4

Overall Response

Thank you for your review and important questions! We are glad that you found our evaluation extensive, the comparisons informative about where E-PMQ offers consistent gains over baseline, and the calibration-cost analysis helpful for understanding the resource trade-offs.

W1.1 Reporting worst-task retention, normalized expert-relative accuracy, and the number of improved versus degraded tasks would provide a more complete assessment. The paper should qualify claims that anchoring preserves integrated behavior.

Our response

Across 336 task–configuration observations, E-PMQ reduces regressions relative to the full-precision merge from 279 with GPTQ to 62. Using the corresponding full-precision merged model as the reference, we count improvements, ties, and regressions and report the maximum accuracy drop (percentage points) and minimum retention (the smallest ratio of quantized to full-precision accuracy across tasks).

ScopeGPTQ improved/tied/degradedE-PMQ improved/tied/degraded
Tables 1–2, 184 task–configuration observations24/3/157153/1/30
Including appendix results, 336 observations53/4/279272/2/62

Across the 24 configurations, E-PMQ has a smaller maximum drop in 21, the same drop in one, and a larger drop in two. Its minimum retention is higher in 21, equal in one, and lower in two. Directly compared with GPTQ, E-PMQ improves 317 observations, ties five, and degrades 14. Ties are based on the reported precision.

For eight-task B/32 Task Arithmetic, E-PMQ improves six tasks and degrades two relative to full precision. Its largest drop is 11.50 points on GTSRB, equal to GPTQ's, and EuroSAT is 2.80 points below GPTQ. For 20-task L/14 WUDI [1], E-PMQ's largest drop is 23.76 points on FER2013 and its minimum retention is 51.11% on KMNIST, compared with 25.00 points and 40.21% for GPTQ. Both tasks still regress substantially from the full-precision merge.

We also computed the mean of per-task accuracy ratios relative to the standalone full-precision experts in Table 1. For eight-task B/32 with Simple Averaging, Task Arithmetic, TIES, and WUDI, the ratios are 70.92%, 71.61%, 76.63%, and 89.45% for GPTQ and 83.37%, 81.65%, 82.95%, and 91.00% for E-PMQ.

These task-level statistics are reported in Appendix E.4, including Table 20.

W1.2 Inconsistent Calibration-Time Measurements

Our response

The original calibration-budget timings (Table 23 in the submitted PDF) were obtained on an H20, while the complete-pipeline timings in Table 24 in the revised PDF were measured on an A800. The revised budget sweep is Table 25 in Appendix J; the following table reports its updated A800 sequential-loading measurements.

The table pairs the accuracy results from the original budget sweep with the updated A800 timings. The ratio is E-PMQ time divided by GPTQ time at the same budget. The 256-sample E-PMQ accuracy of 73.61% rounds to 73.6% as reported in Table 1.

Samples per taskGPTQ avg. accuracy (%)E-PMQ avg. accuracy (%)GPTQ time (s)Sequential E-PMQ time (s)Time ratio
6464.8672.2310.1521.002.07×
12864.7773.5818.5043.312.34×
25665.0373.6126.3576.652.91×

W1.3 Inconsistent GTSRB Expert Accuracy

Our response

We checked the experiment records and confirmed that both tables use the same full-precision GTSRB expert checkpoint and evaluation protocol. The correct accuracy is 98.88%, which rounds to 98.9% in Table 1. The 90.35% in Table 10 is a transcription error. In the HDRQ comparison [2], this correction changes the full-precision expert average from 89.25% to 90.32% and the GTSRB difference from −0.75 to −9.28 points. Using the reported HDRQ average of 87.49%, the average gap is approximately 2.83 points. Table 10 and its discussion in Appendix B have been updated.

AuthorsOpenReview ↗

Rebuttal by Authors Part 2/4

W2.1 E-PMQ uses expert information to improve the merged model during quantization. However, it is unclear whether similar gains could be achieved by first repairing the full-precision model using the same expert information, then applying GPTQ. A matched comparison would clarify the benefit of combining these steps.

Our response

To test whether expert-guided repair followed by GPTQ can achieve similar gains, we extend the two-stage baseline introduced in Section 5.5 and Table 7. Representation Surgery [3] first uses the source experts to reduce merge-induced representation errors in the full-precision model. GPTQ then quantizes the backbone while retaining the Surgery modules. We report the full-precision accuracy after Surgery to show how much the repair step recovers before quantization. The comparison covers four merging methods in the eight-task setting, as well as WUDI with 14 and 20 tasks. The table reports mean accuracy (%), with differences expressed in percentage points.

CLIP ViT-B/32 settingFP merge+ Surgery (FP)Surgery + GPTQE-PMQΔ (E-PMQ − Surgery + GPTQ)
8 tasks, Simple Averaging66.5076.2373.5975.10+1.51
8 tasks, Task Arithmetic68.0074.7171.5973.61+2.02
8 tasks, TIES72.2076.1172.8674.80+1.94
8 tasks, WUDI84.5085.0081.4082.40+1.00
14 tasks, WUDI78.7080.2075.1077.86+2.76
20 tasks, WUDI67.1071.6767.4471.70+4.26
  • End-to-end accuracy. Applying E-PMQ directly to the merged checkpoint yields higher mean accuracy than Surgery + GPTQ in all six settings, with gains of 1.00–4.26 percentage points.
  • Efficiency. In the eight-task Simple Averaging comparison (Table 7), E-PMQ takes 76.7 s versus 142.2 s for Surgery + GPTQ, about 46% less processing time. E-PMQ uses one calibration stage and requires no additional repair module at inference.

Appendix B, Table 11 reports the expanded comparison; the following residual analysis explains why expert-guided calibration can still help after repair.

W2.2 Removing the anchor causes accuracy to collapse, but the anchor both keeps weights near the merged model and stabilizes numerical optimization. The collapse could therefore reflect numerical instability rather than excessive expert guidance. Please separate these two effects.

Our response

Thank you for pointing this out. Removing the anchor in Eq. (12) removes both the diagonal loading λI\lambda I and the merged-weight term λWm\lambda W_m. To test the effect of the penalty center, we use the same loading rule for both penalties and vary only the center. We evaluate eight-task CLIP ViT-B/32 with Task Arithmetic and 4-bit weights.

SettingAnchor scale αEffective curvatureRight-hand sideMean accuracy (%)
No anchor0HR5.37
Zero-centered penalty0.1H + λIR10.21
Merged-weight-centered penalty0.1H + λIR + λWₘ74.09

With the same loading rule, centering the penalty at the merged weights improves mean accuracy by 63.88 percentage points over the zero-centered penalty. The zero-centered penalty improves accuracy by 4.84 points over the no-anchor baseline. Because it also changes the optimization objective, this gain cannot isolate the effect of numerical stabilization.

This control and its interpretation are included in Appendix H.

AuthorsOpenReview ↗

Rebuttal by Authors Part 3/4

W2.3 Some full-precision merged models perform poorly, leaving substantial room for repair. Please evaluate better-tuned merged models to show whether E-PMQ remains effective when the starting model is already strong.

Our response

E-PMQ remains effective when the starting model is already strong. We extend the CLIP ViT-B/32 WUDI [1] comparison to 8, 14, and 20 tasks. In each setting, we use Surgery [3] to repair the full-precision merge and apply GPTQ and E-PMQ at 4-bit to the same repaired checkpoint. The table reports mean task accuracy (%).

PipelineWeight precision8 tasks14 tasks20 tasks
WUDIFull precision84.5078.7067.10
WUDI + SurgeryFull precision85.0080.2071.67
WUDI + Surgery + GPTQW481.4075.1067.44
WUDI + Surgery + E-PMQW483.5280.1573.70

In the eight-task setting, Surgery raises an already strong WUDI merge from 84.50% to 85.00%. From this repaired checkpoint, E-PMQ achieves 83.52% at 4-bit, 2.12 percentage points above GPTQ. The gains from the same repaired checkpoint are 5.05 and 6.26 percentage points in the 14- and 20-task settings, respectively. This comparison is included in Table 11 in Appendix B.

W3.1 How are calibration samples assigned to experts when domains overlap or task identities are unavailable?

Our response

When the source dataset is known. Following Surgery’s task-wise calibration setup, we draw a small subset from each task’s training set when available and pair it with the corresponding source expert. To address the concern about overlapping domains, the pairing follows the dataset the sample came from, even when its content also resembles another task. Feature matching uses the inputs without class labels.

When task provenance is unavailable. The current method requires the expert-to-input pairing to be specified during calibration. An anonymous mixture without source information would require a separate assignment step.

External unlabeled inputs. Following SurgeryV2’s wild-data setting [4] (Sec. IX-C, Table VII), we evaluated CIFAR-100 [5] and ImageNet [6] as alternative calibration sources for eight-task CLIP ViT-B/32 with Task Arithmetic. For each external dataset, we randomly sampled a single set of calibration images and used that set for all experts, without class labels.

MethodCalibration inputsMean accuracy (%)
Full-precision Task Arithmetic—68.00
w/ E-PMQCIFAR-100, labels unused69.35
w/ E-PMQImageNet, labels unused71.45
w/ E-PMQTask-related inputs73.61

Using CIFAR-100 and ImageNet for calibration without class labels yields 69.35% and 71.45% mean accuracy, respectively, both above the 68.00% full-precision Task Arithmetic baseline. These results show that unlabeled external images can be used for calibration, while task-specific calibration data yields the highest accuracy at 73.61%.

The calibration protocol and external-data results are reported in Appendix K.

W3.2 Clarify whether the layer-wise residual comparison evaluates both methods on identical activations or on their respective trajectories.

Our response

We thank the reviewer for the reminder. Figure 3 uses the same calibration samples for GPTQ and E-PMQ. Each method is evaluated on its own quantized trajectory. At each layer, we take the inputs produced by that method's preceding quantized layers and evaluate both the current quantized layer and the corresponding source expert layer on those inputs. We will clarify this protocol in the caption and describe the result as a lower expert-relative residual along E-PMQ's quantized trajectory.

Reviewer_hBbyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes E-PMQ for quantizing a merged model (built from several task experts) to 4-bit. During GPTQ-style layer-wise calibration, it matches each source expert's layer outputs instead of the merged model's, while a weight-space penalty keeps the quantized weights close to the merged checkpoint. On CLIP, FLAN-T5, and Llama it beats RTN, GPTQ, and AWQ, and on CLIP and GLUE the 4-bit model even exceeds the full-precision merged model.

Critical strengths
  1. Clear, well-motivated setting. Merge-then-quantize is an interesting and novel setup, and the paper is the first to name it and show that the merged model is an imperfect reconstruction target for standard PTQ.

  2. Simple, solver-agnostic method with zero inference cost. The change is a modified per-layer quadratic that any GPTQ-style rounder can consume; experts are used only at calibration, and the deployed artifact is a single 4-bit model with no adapters or routing.

Critical weaknesses
  1. The main gains come from weak upstream mergers, not from better quantization. Simple Averaging, Task Arithmetic, and TIES lose much of the expert information. E-PMQ reads all K source experts during calibration, while GPTQ sees only the merged weights. Any method with this extra access will recover accuracy that GPTQ cannot. The comparison is therefore not between quantizers but between unequal inputs. "Exceeding full precision" only shows that the merger was poor. On strong mergers, the advantage nearly disappears. On 8-task ViT-B/32 WUDI, E-PMQ reaches 82.4 vs GPTQ 81.1, below the 84.5 full-precision model. On ViT-L/14 WUDI, it is 91.3 vs 90.5. On TSV, it is 81.07 vs 80.07, and Surgery+GPTQ already reaches 80.94. The Llama gains are about 1 point. The evaluation should focus on SOTA mergers, and the current evaluation which is mostly based on naive mergers, does not provide enough grounds to prove the effectiveness of their proposed method.

  2. The setting is narrow and does not match common practice. E-PMQ needs every source expert, per-task calibration data, and the merged checkpoint at quantization time. In practice, usually only the merged model is released. In that case, the method reduces to GPTQ, as Appendix A admits.

Questions for authors

None

8 comments
AuthorsOpenReview ↗

Rebuttal by Authors Part 1/3

Overall Response

Thank you for your review and important questions! We are glad to hear that you found our proposed PMQ setting clear, well-motivated and E-PMQ simple, efficient and no extra adapters. Below we address the concerns and questions raised in the review.

W1: The main gains come from weak upstream mergers, not from better quantization. Simple Averaging, Task Arithmetic, and TIES lose much of the expert information. E-PMQ reads all K source experts during calibration, while GPTQ sees only the merged weights. Any method with this extra access will recover accuracy that GPTQ cannot. The comparison is therefore not between quantizers but between unequal inputs. "Exceeding full precision" only shows that the merger was poor. On strong mergers, the advantage nearly disappears. On 8-task ViT-B/32 WUDI, E-PMQ reaches 82.4 vs GPTQ 81.1, below the 84.5 full-precision model. On ViT-L/14 WUDI, it is 91.3 vs 90.5. On TSV, it is 81.07 vs 80.07, and Surgery+GPTQ already reaches 80.94. The Llama gains are about 1 point. The evaluation should focus on SOTA mergers, and the current evaluation which is mostly based on naive mergers, does not provide enough grounds to prove the effectiveness of their proposed method.

Our response

Thank you for raising the concern that our gains may come from repairing a weak merge or from access to additional expert information. When both methods start from the same Surgery-repaired checkpoint, E-PMQ achieves higher 4-bit mean accuracy than GPTQ in all six settings, with gains of 2.12–7.05 percentage points, including all three WUDI configurations.

1. Comparison setup

We extend the two-stage baseline in Sec. 5.5 and Table 7. Representation Surgery [1] first uses the source experts to repair the full-precision merge. We then apply GPTQ or E-PMQ to the same backbone and retain the same Surgery modules in both cases. This comparison holds the starting checkpoint and repair step fixed, so it tests whether using expert targets during quantization can still help after repair. We also compare direct E-PMQ with Surgery + GPTQ to evaluate the two complete deployment pipelines.

CLIP ViT-B/32 settingFP merge+ Surgery (FP)Surgery + GPTQE-PMQΔ (E-PMQ − Surgery + GPTQ)Surgery + E-PMQ
8 tasks, Simple Averaging66.5076.2373.5975.10+1.5178.54
8 tasks, Task Arithmetic68.0074.7171.5973.61+2.0276.35
8 tasks, TIES72.2076.1172.8674.80+1.9479.91
8 tasks, WUDI84.5085.0081.4082.40+1.0083.52
14 tasks, WUDI78.7080.2075.1077.86+2.7680.15
20 tasks, WUDI67.1071.6767.4471.70+4.2673.70

2. Gains after repairing a strong merge

On eight-task WUDI [2], the full-precision model reaches 85.00% after Surgery. At 4-bit, E-PMQ reaches 83.52% and GPTQ reaches 81.40%, corresponding to drops of 1.48 and 3.60 percentage points from the same repaired model. E-PMQ also outperforms GPTQ by 5.05 and 6.26 points on 14- and 20-task WUDI, respectively.

3. Why expert guidance can help after repair

Consider one layer with fixed calibration inputs XiX_i. Let WsW_s be the full-precision backbone weight at the common quantization starting point, YiY_i the expert target, and QQ a candidate low-bit weight. The expert-relative residual is

QXi−Yi=(Q−Ws)Xi⏟Ai(Q)+WsXi−Yi⏟Bi.QX_i-Y_i = \underbrace{(Q-W_s)X_i}_{A_i(Q)} + \underbrace{W_sX_i-Y_i}_{B_i}.

GPTQ reconstructs the starting checkpoint by minimizing the output change Ai(Q)A_i(Q) in its layer-wise objective. E-PMQ instead evaluates that change against the remaining expert mismatch BiB_i. Expanding its objective gives LE-PMQ(Q)=∑i∥Ai(Q)∥F2+2∑i⟨Ai(Q),Bi⟩F+λ∥Q−Wm∥F2+C,C=∑i∥Bi∥F2,\mathcal L_{\mathrm{E\text{-}PMQ}}(Q) = \sum_i\|A_i(Q)\|_F^2 + 2\sum_i\langle A_i(Q),B_i\rangle_F + \lambda\|Q-W_m\|_F^2 + C, \qquad C=\sum_i\|B_i\|_F^2, where �6��6� is the anchor reference and �7��7� is constant in �8��8�. A candidate achieves a lower E-PMQ objective than the GPTQ solution when its reduction in the cross term outweighs any increase in checkpoint reconstruction error and anchor penalty. Thus, even after Surgery, a low-bit weight can improve expert reconstruction by partially offsetting the remaining mismatch. The analysis concerns the quantized backbone layer on fixed inputs; the retained Surgery modules also affect deployed outputs. The same-checkpoint experiments show higher downstream accuracy in the six settings tested.

AuthorsOpenReview ↗

Rebuttal by Authors Part 2/3

4. End-to-end accuracy and cost

Direct E-PMQ also outperforms Surgery + GPTQ by 1.00–4.26 percentage points across the six settings. In the eight-task Simple Averaging comparison in Table 7, direct E-PMQ takes 76.7 s versus 142.2 s for Surgery + GPTQ and requires no separate repair module at inference. The end-to-end results compare the two complete pipelines, whose calibration objectives and model structures differ. The expanded comparison and analysis are included in Appendix B, Table 11 and the following paragraph.

W2: The evaluation should focus on SOTA mergers, and the current evaluation which is mostly based on naive mergers, does not provide enough grounds to prove the effectiveness of their proposed method.

Our response

Thank you for asking us to clarify our choice of upstream mergers. We evaluate E-PMQ across merging methods because its usefulness depends on how well it quantizes models produced by different upstream workflows.

  1. Applicability across common merging methods. Simple Averaging and Task Arithmetic are broadly applicable because they combine compatible checkpoints through inexpensive weight-space operations. SK Telecom builds the A.X K2 base model by taking a weighted average of six late-training checkpoints [4], and NVIDIA uses weighted checkpoint merging to select the starting point for long-context training of Nemotron 3 Ultra [3]. In post-training, Preferred Networks uses post-SFT merging in PLaMo 2 [5], while Sakana AI's EvoLLM-JP combines language and reasoning capabilities through weight-space merging [10]. These workflows illustrate why evaluating E-PMQ on simple merges matters in practice.

  2. Performance with interference-aware mergers. We include TIES and WUDI to test whether the gains persist when the upstream method explicitly addresses task interference. WUDI [2] reduces interference without additional data, but its published procedure adds layer-wise optimization compared with direct weight averaging or task-vector addition. On WUDI, E-PMQ improves mean accuracy over direct GPTQ by 3.22, 8.44, and 8.75 percentage points on 14-task ViT-B/32 (74.64% to 77.86%), 20-task ViT-B/32 (63.26% to 71.70%), and 20-task ViT-L/14 (72.19% to 80.94%), respectively (Tables 16, 18, and 19). In the eight-task WUDI setting highlighted by the reviewer, the gain is smaller (82.4% vs. 81.1%), and both low-bit results are below the full-precision merge (84.5%). These results quantify the variation across the evaluated settings.

The added WUDI repair/quantization comparisons are reported in Appendix B, Table 11.

W3: The setting is narrow and does not match common practice. E-PMQ needs every source expert, per-task calibration data, and the merged checkpoint at quantization time. In practice, usually only the merged model is released. In that case, the method reduces to GPTQ, as Appendix A admits.

Our response

Figure 1 contrasts direct PTQ of a merged checkpoint with expert-guided calibration in the PMQ setting. When only the merged checkpoint is available, GPTQ remains applicable and calibrates the quantized model against the full-precision merge. When source experts are available, E-PMQ uses them to account for both merge-induced and quantization-induced deviations and achieves higher mean accuracy than direct GPTQ in our experiments. As more compatible checkpoints become available and model merging is used across a wider range of tasks, retaining specialist capabilities in low-bit merged models becomes increasingly important. The examples below illustrate this need through industrial deployment, open-weight checkpoints, and applications that combine complementary capabilities.

1. Frontier industrial models already combine merging and low-precision deployment. NVIDIA’s 550B Nemotron 3 Ultra selected a merged pretraining checkpoint for further training and later released a mixed-precision NVFP4 model, reducing model size from 1,121 GB to 352.3 GB [3]. SK Telecom’s 688B A.X K2 averaged its last six pretraining checkpoints and reduced minimum serving memory from 646 GB to 370 GB with NVFP4 PTQ [4]. Closer to our setting, Preferred Networks’ PLaMo 2.0-31B merged separately fine-tuned checkpoints during post-training and used INT4 GPTQ to reduce model size to 17 GB [5]. Teams preparing low-bit releases can retain the source weights used for merging. In our 4-bit Llama-3.1-8B experiment, using these weights improved the mean score across six benchmarks from 61.66 with GPTQ to 62.91 with E-PMQ.

AuthorsOpenReview ↗

Rebuttal by Authors Part 3/3

2. The open-weight ecosystem provides a pool of candidate checkpoints for PMQ. On November 9, 2026, Hugging Face listed 3.23 million public model repositories, including 45,379 tagged merge [6]. A separate model-card lineage study identified 12,799 merge-derived models in its June 2025 snapshot [7]. HuggingGraph reports 2,519 merged descendants of Llama-3.1-70B, 1,254 of Mistral-7B-v0.1, and 1,132 of Qwen2.5-7B [8]. Releases such as BioMistral and EvoLLM-JP document their constituent checkpoints, giving downstream teams a way to identify and retrieve source weights when those weights and representative task inputs are accessible [9] [10].

3. Model merging supports a range of specialized applications. BioMistral combines biomedical and general language weights for medical question answering [9]. ES-Merging combines molecule, protein, and cell specialists for biological interaction and enzyme prediction [11]. LED-Merging addresses conflicts between safety alignment and math or coding ability [12]. Merge and Conquer extends instruction following across languages [13], RETAIN adds robot manipulation skills while preserving generalist behavior [14], and ZipLoRA combines independently learned subjects and styles for image generation [15]. When compatible experts are available and the merged model must fit a tighter deployment budget, E-PMQ uses those experts during low-bit calibration to help retain their task capabilities.


We kindly ask you to reconsider your score in light of these answers. We are glad to answer any further question.


Changes in the revised PDF

• W1 and W2 — App. B, Table 11 (p. 16) adds six repair/quantization comparisons, including the same-checkpoint comparisons and the 8-, 14-, and 20-task WUDI results.

• W1 — App. B (pp. 16–17) adds the fixed-input residual decomposition and cross-term analysis explaining how expert guidance can help after repair.

References

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.