acceptodds
Under review as a conference paper at ICLR 2027

RCDQ: Reference-Guided LLM Quantization with Interleaved Continuous Compensation and Discrete Decisions

Abstract

Full-precision reference outputs are widely used to compensate for accumulated errors in low-bit quantization of large language models. Existing methods incorporate these references in different ways. Some apply continuous compensation to an entire layer before quantization. Others incorporate deviations from reference outputs into sequential weight updates or use reference-based reconstruction objectives to optimize quantization parameters and integer codes. A key challenge is that improvements in output reconstruction achieved through continuous compensation may not be preserved after discretization. To address this challenge, we propose RCDQ (Reference-Guided Continuous–Discrete Quantization), a two-stage group-wise quantization method that uses the same full-precision reference objective to guide both continuous compensation and quantization decisions. In stage one, before quantizing each weight group, we recompute its weight compensation using statistics describing the output discrepancy between the quantized and full-precision paths. These statistics are updated as groups are processed sequentially. Under the same full-precision reference, in stage two, we then evaluate fake-quantized candidates constructed from the compensated weights and select the best quantization scale multipliers to reduce the discrepancy between the candidate outputs and the full-precision reference outputs. Experiments across seven Llama and Mistral models show that RCDQ achieves the lowest W3 perplexity in all 14 WikiText-2 and C4 model–dataset settings, together with the highest or tied-highest six-task average and MMLU accuracy on six of seven models, respectively. RCDQ remains robust at more aggressive precisions, achieving the lowest W2A16 perplexity in 10 of 14 settings and the lowest W3A8/W4A8 perplexity in three of four settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training quantization, large language models, low-bit quantization, model compression, error compensation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_46SrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a novel quantization technique, RCDQ. It is a two-stage group-wise post-training quantization method that uses one full-precision reference objective for both continuous weight compensation, and discrete per-channel scale-multiplier selection.

Critical strengths

The study works on the continuous-to-discrete gap, in quantization, and backs it with mechanism analysis, and a proxy-fidelity study. The method also plugs into GPTQ with cached statistics and needs no extra Transformer passes.

Critical weaknesses

Quantization helps for larger models, The largest model is Mistral-Nemo (12B). The paper needs results on 70B models (Llama-2-70B, Llama-3-70B), other families (Qwen2.5/3, Gemma), MoE models (Mixtral, DeepSeek). Generative evaluations such as GSM8K, HumanEval, and long-context tasks would also help, since perplexity and multiple-choice accuracy can hide degradation. Weight-only results compare only against GPTQ-family methods. They omit AWQ, OmniQuant, LeanQuant, and SchurQuant, even though the paper cites several of these as closely related. Results use a single calibration seed (seed 0), with no variance reported.

Questions for authors

See weakness section

3 comments

Thank you for your review. Unless specified, following added experiments use W3A16/G128 and 128×2048 calibration tokens. Paired methods share calibration/evaluation data and task protocols. Downstream tasks use C4-calibrated checkpoints; WikiText-2/C4 PPL uses separately calibrated checkpoints. Accuracies are percentages; L=Llama.

1. 70B. Reloaded Llama-2-70B WikiText-2 PPL is 3.8712/3.8515/3.8409/3.8400/3.7378 for GPTQ/GPTAQ/QEP/Qronos/RCDQ: a 2.66% reduction over the best baseline. Quantization on one A100-80GB with CPU offloading takes 208.36 min, with peak GPU allocated memory 25.85 GiB and host RSS 254.03 GiB.

2. Qwen. On Qwen3-8B-Base, RCDQ achieves WikiText-2/C4 PPL of 7.5996/11.3984 and six-task/MMLU/GSM8K accuracy of 67.55/70.92/54.36. It ranks first among five methods on both PPL metrics, MMLU and GSM8K; MMLU/GSM8K improve by 0.72/0.68 points over the respective best baselines.

3. MoE. On DeepSeek-V2-Lite, RCDQ achieves WikiText-2/C4 PPL of 6.7953/9.8786 and six-task/MMLU/GSM8K accuracy of 65.91/50.43‾\underline{50.43}/22.14. It ranks first among five methods on both PPL metrics, six-task mean and GSM8K, and ranks second on MMLU. Routers retain FP precision; FP/quantized trajectories may select different experts. Expert down projections use token-aligned aggregate-output residuals as references; gate/up use GPTQ.

4. Generative evaluation. All seven original models are evaluated on 1,319 GSM8K questions (8-shot CoT, greedy, strict-match, 2048-token limit) and 164 HumanEval problems (0-shot, greedy pass@1, one sample/problem, 1024-token limit, official tests).

Each cell below is GSM8K/HumanEval. Best/second among the five listed methods are bold/underlined; ties share ranks.

ModelGPTQGPTAQQEPQronosRCDQ
L2-7B8.64‾\underline{8.64}/4.279.40/4.278.49/4.278.19/7.93‾\underline{7.93}7.96/9.15
L2-13B14.33/12.80‾\underline{12.80}14.03/14.0216.76/11.5915.09‾\underline{15.09}/10.9813.72/12.80‾\underline{12.80}
L3-8B18.20/10.98‾\underline{10.98}22.29/10.3723.96/10.3725.63‾\underline{25.63}/10.98‾\underline{10.98}28.35/14.02
L3.1-8B18.04/9.1524.79/10.98‾\underline{10.98}20.92/7.3227.29‾\underline{27.29}/10.3729.95/14.63
L3.2-3B9.02/6.71‾\underline{6.71}9.48/6.71‾\underline{6.71}9.17/6.71‾\underline{6.71}10.24/4.889.78‾\underline{9.78}/14.63
Mistral-7B-v0.318.04/9.7619.71/13.4120.92/15.85‾\underline{15.85}24.26‾\underline{24.26}/12.8024.41/17.07
Mistral-Nemo-12B34.42/0.6135.41/4.2733.74/4.2738.44‾\underline{38.44}/5.49‾\underline{5.49}38.59/6.71

RCDQ ranks first on 4/7 GSM8K models and second on one; on HumanEval it ranks first on 6/7 and tied second on the remaining model. On Llama-3.2-3B, HumanEval rises from the best baseline's 6.71% to 14.63% (11 to 24 solved problems).

5. Other PTQ baselines. Added OmniQuant follows its official weight-only training procedure and native asymmetric quantization; RCDQ uses symmetric quantization. The paired W3 results below show RCDQ/OmniQuant, with the better value bold.

ModelC4 PPLMMLUGSM8K
L2-7B7.5991/7.808442.9/39.207.96/9.86
L2-13B6.9255/7.023451.6/51.3713.72/16.76
L3-8B10.8318/12.635257.7/54.0928.35/22.74
L3.1-8B10.8968/12.543259.6/55.0729.95/25.93
L3.2-3B12.8612/14.832148.9/44.679.78/7.66
Mistral-7B-v0.38.5772/8.764157.7/54.4124.41/24.03
Mistral-Nemo-12B9.7987/10.149661.5/59.9238.59/30.86

RCDQ wins all seven C4 PPL and MMLU comparisons, and five GSM8K comparisons. MMLU uses 5-shot evaluation on all 57 subjects.

Original Table 5 already compares AWQ at W4A8: WikiText-2 PPL is 5.8309/7.0264 for AWQ versus 5.6638/6.5386 for RCDQ on L2-7B/L3-8B. Added W4A4 uses matched full Hadamard rotation and A4 settings: rotation-adapted AWQ gives 7.1145/9.7606 versus RCDQ's 6.0384/8.0511.

6. Calibration seeds. We evaluate five methods on two models with seeds 0–4 (50 configurations), sharing calibration samples per model/seed. Below are RCDQ's mean±sample SD (n−1 denominator); accuracy SD is in percentage points. Bold/underline mark best/second mean among five methods.

ModelC4 PPL6-task meanMMLUGSM8K
L2-7B7.5981±0.003464.85±0.2142.60±0.428.01±0.86‾\underline{8.01±0.86}
L3-8B10.8713±0.031467.74±0.2857.98±0.4626.11±2.20‾\underline{26.11±2.20}

Qronos C4 PPL is 7.8227±0.0072/11.8076±0.3044, respectively. RCDQ has the best mean and smallest SD on C4 PPL, six-task mean and MMLU for both models, and the lowest PPL in all 10 model×seed comparisons. The six-task mean averages 0-shot PIQA, ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande and BoolQ; MMLU is 5-shot and GSM8K 8-shot greedy strict-match.

We will add these results and full protocols to the revised paper.

Thank you for the detailed response and the additional experiments. The multi-seed results, generative evaluations (GSM8K/HumanEval), and the OmniQuant comparison address several of my concerns. However, the 70B evaluation (perplexity only on Llama-2-70B) and the baseline coverage (weight-only AWQ, LeanQuant, SchurQuant) remain only partially resolved, so I will maintain my score.

Thank you for acknowledging the additional experiments. We hope you will reconsider your score for the following reasons.

1. 70B evaluation

The original review questioned whether RCDQ’s gains extend beyond the models of up to 12B parameters evaluated in the submission. Our added Llama-2-70B experiment directly tests this question, comparing RCDQ against GPTQ, GPTAQ, QEP, and Qronos under matched calibration and evaluation settings. RCDQ achieves the lowest WikiText-2 perplexity, 3.7378 versus 3.8400 for the strongest baseline.

This representative large-model experiment demonstrates RCDQ’s effectiveness at 70B scale, where it outperforms all four baselines. We therefore do not think that testing another 70B model is necessary to support this conclusion.

2. AWQ comparisons

Our submission already included AWQ at W4A8, and our response added W4A4 comparisons. RCDQ achieves lower perplexity on both Llama-2-7B and Llama-3-8B under both settings. The relative reductions are approximately 2.9% and 6.9% at W4A8, and 15.1% and 17.5% at W4A4.

AWQ has therefore received direct experimental coverage across two models and two quantization settings. Given these consistent improvements, we do not think an additional weight-only AWQ comparison is necessary.

3. LeanQuant, SchurQuant, and baseline coverage

Although we cite LeanQuant and SchurQuant in related work, we consider QEP, LoaQ, and weight-MSE scale search more directly relevant to testing RCDQ’s specific contributions. The revised Appendix H therefore adds detailed comparisons and ablations examining group-wise compensation, the residual-output reference, and reference-guided scale selection.

To broaden the baseline comparisons, we also added OmniQuant and a KronQ-based weight-only baseline. Together with the baselines already evaluated in the main paper, these additions provide substantial evidence of RCDQ’s competitive performance. We do not think that establishing this requires experiments against every method cited in related work.

We would appreciate it if you could consider these new results alongside the original experiments when reassessing the paper.

Reviewer_jGQiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a 2-stage quantization framework for per-group quantization. Stage-1 considers error from previous layers before quantization, and Stage-2 search the quantization scale based on the error direction from the previous quantized group. Experiments across different model families and scale demonstrate that RCDQ achieves stable improvement.

Critical strengths
  1. The motivation is clear. new weight and new scales are indeed what we needed before quantizing each group

  2. The experimental results are impressive, achieves stable improvement across different settings

  3. The analysis of perturbation from previsous quantized groups is interesting.

Critical weaknesses
  1. The novelty is over claimed. Stage-1 is largely overlapped with QEP and interleaved execution is the standard workflow of modern compensation-based quantization methods. In GPTAQ and ResComp, they search the quantization scale using mean-square-error before quantizing each group, using the compensated weights. Therefore, the main contribution is intergrating the quantization error of pervious groups into the full-layer output error when search quantization scales.

  2. The framework combines multiple techniques, such as QEP and LoaQ. but the relavant ablations are missing.

  3. The hyperparameters of alpha and beta rely on empirical choices.No ablation or sensitivity analysis are provided.

Questions for authors

please see the weakness above

1 comment

Thank you for your review. RCDQ addresses how the improvement achieved by continuous correction can be preserved when the corrected weights are discretized. Stage 1 improves reconstruction; Stage 2 evaluates fake-quant candidates against the same FP reference to preserve these gains.

1. Novelty and QEP/LoaQ ablations. With calibration inputs XX, initial FP-reference discrepancy EE, A=XX⊤+λIA=XX^\top+\lambda I and B=EX⊤B=EX^\top, QEP-style whole-layer compensation under the same quadratic objective is Δ=αBA−1\Delta=\alpha BA^{-1}. RCDQ instead uses Eq. (17):

ΔGt=α(BGt−∑s<tΔGsAGs,Gt)AGt,Gt−1.\Delta_{G_t}=\alpha(B_{G_t}-\sum_{s<t}\Delta_{G_s}A_{G_s,G_t})A_{G_t,G_t}^{-1}.

This uses cross-group correlations to account for earlier continuous corrections; it generally differs from partitioning a whole-layer solution.

Appendix B proves that GPTQ rounding/feedback on column jj leaves later reference-gradient statistics unchanged, (δK):,k=0(\delta K)_{:,k}=0 for k>jk>j, in exact arithmetic with fixed statistics, group order and coefficients, and matched positive-definite curvature for compensation and feedback. Stage 1 corrections can thus be precomputed independently of earlier scales/rounding. Eq. (18) still includes actual preceding GPTQ feedback in the weights presented to Stage 2, so scale selection remains sequential.

We compare official QEP, QEP+RCDQ-stage2, and full RCDQ. We also replace one component of full RCDQ at a time:

  • Joint (QEP-related): replace group-wise compensation with one whole-layer update Δ=αBA−1\Delta=\alpha BA^{-1} before quantization; retain the same reference, α, damping, Stage 2 and GPTQ.
  • Linear (LoaQ-related): for o_proj/down_proj, match FP projection outputs before residual addition instead of outputs after adding the residual branch. Recompute reference statistics; keep other projections, compensation/scoring formulas and coefficients unchanged. This isolates the LoaQ-inspired residual-output reference.

Matched C4 calibration/evaluation, symmetric W2A16/G128; lower PPL is better. L=Llama; bold/underline mark best/second.

ModelQEPQEP+S2JointLinearRCDQ
L2-7B53.256620.626720.1720‾\underline{20.1720}22.774518.8507
L2-13B23.053523.921820.7554‾\underline{20.7554}20.901916.7183
L3-8B121.253875.947533.7888‾\underline{33.7888}91.171529.2348

RCDQ improves over QEP+S2 in W2 quantizatoin by 8.61%–61.51%; replacing either component increases PPL on the tested models.

Scale scoring differs from searching compensated weights. GPTAQ/ResComp incorporate reference discrepancies in weight updates and search scales on compensated weights. RCDQ directly scores fake-quant perturbations D(f)D(f) using the post-compensation reference gradient KG+K_G^{+}:

Sβ(f)=D(f)AGGD(f)⊤−2β⟨D(f),KG+⟩.S_\beta(f)=D(f)A_{GG}D(f)^\top-2\beta\langle D(f),K_G^{+}\rangle.

The linear term distinguishes candidates with equal curvature-weighted cost but different directions relative to the reference. This applies even to the first group, where upstream quantization may already leave a reference discrepancy; its role therefore extends beyond preceding groups' quantization errors.

Keeping Stage 1, candidates and all other settings fixed, replacing only the score with weight MSE gives the following W3A16/G128 C4 PPL:

ModelWeight MSERCDQ score
L2-7B8.10917.5991
L2-13B6.95236.9255
L3-8B10.936410.8318
L3.1-8B10.976310.8968
L3.2-3B13.069412.8612

Original Table 4 shows complementary gains: Llama-2-7B W3 WikiText-2 PPL is 6.4136/6.0690/6.0269/5.8689 for GPTQ/Stage 1/Stage 2/both. Fig. 1 shows that, after identical continuous compensation, Stage 2 reduces the reconstruction error reintroduced by discretization. Together, these results distinguish RCDQ from combining existing compensation and scale search.

2. α/β rationale and sensitivity. Eq. (8) ensures a non-increasing local objective for 0≤α≤2. On Llama-2-7B W3, disabling Stage 2 and sweeping α=0.1,0.2,…,0.6 yields the lowest C4 PPL, 7.8885, at α=0.4. β=1 is the exact local fake-quant objective increment with other groups fixed; actual GPTQ adds sequential rounding/feedback, motivating attenuated guidance.

A full 4×4 grid, α∈{0.2,0.4,0.6,1}, β∈{0,0.05,0.2,1}, places the fixed defaults (0.4,0.05) within 0.023%, 0.021%, 0.416% and 0.518% relative C4 PPL of the best tested pair on Llama-2-7B, Llama-3-8B, Mistral-7B-v0.3 and DeepSeek-V2-Lite at W3. A finer 6×6 Llama-2-7B W3 sweep gives PPL 7.5935–7.6486; default 7.5991 is only 0.074% above the minimum.

At fixed α=0.4, β=0.05 beats β=1 in all six tested W2/W3 settings. On Llama-2-7B W2, β=0/0.05/1 gives PPL 21.2739/18.8507/23.0591. β=0 retains curvature-weighted search, isolating the reference term's 11.39% gain. These results support shared defaults without per-model tuning.

Reviewer_AAz2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a two-stage PTQ framework to improve existing full-precision reference-based compensation methods. Stage 1 applies a group-wise continuous weight update to compensate for output deviations from the FP reference, while Stage 2 performs a heuristic search for quantization scales followed by standard GPTQ quantization.

Critical strengths

The overall presentation is sound, and the motivation is reasonably clear.

The 3-bit quantization results are promising on several evaluated models.

The proof in Appendix B showing that delta K=0 on subsequent unquantized columns establishes the validity of precomputing the Stage 1 compensation updates independently of previous GPTQ rounding decisions.

Critical weaknesses

The empirical selection of alpha and beta in Stages 1 and 2 makes the method somewhat heuristic.

The paper lacks sufficient justification and ablation studies for the selection of alpha and beta.

There is limited justification for the selection of 100 predefined scale multipliers. No ablation is provided on the candidate set's size or range.

As reported in Appendix D, RCDQ nearly doubles peak GPU memory compared with GPTQ on Llama-2-7B. A rough extrapolation of the reported 9.1 GiB peak memory to 70B-scale models suggests memory requirements exceeding the capacity of a single 80GB GPU. This raises concerns about the practicality and scalability of the proposed method, particularly given the absence of experiments on 70B-scale or larger models

It remains unclear whether the method generalizes to substantially different architectures, such as models using GDN. Additionally, prior FP-compensation based methods suffer from not working on the MoE models (different expert selections), this limits their extension to latest MoE models. RCDQ seems inherit this weakness.

It remains unclear whether the method generalizes to substantially different architectures, such as models using Gated DeltaNet (GDN) or Mixture-of-Experts (MoE). Prior FP-reference compensation methods may encounter difficulties with MoE models when the full-precision and quantized trajectories activate different experts. RCDQ may inherit similar challenges, but this is not investigated.

Questions for authors

Please see my comments above, particularly regarding the missing experiments on larger models and different architectures, as well as the missing ablations for the two empirical hyperparameters.

Additionally, I have several questions:

How does RCDQ compare with other recent PTQ methods, such as QuIP# and KronQ, which have usually demonstrated strong performance under W3 and W2 quantization compared with conventional GPTQ-based methods?

How does RCDQ perform under more aggressive activation quantization, such as W4A4 and W2A4?

What is the intuition behind selecting 100 scale multipliers within the range of 0.21–1.20? Does the optimal search range change with weight precision, especially for sub-3-bit quantization?

How does RCDQ compare with prior methods on recent reasoning-capable models? While QA tasks and MMLU are useful for evaluating the effects of PTQ on base LLMs, reasoning benchmarks on more capable open-source models are increasingly relevant. One possible evaluation would be Muse Glimmer 30B (a dense, non-MoE model) on AIME.

2 comments

Thank you for your review. Our main results retain α=0.4, β=0.05 and group size 128.

1. α/β sensitivity. α controls the continuous step; Eq. (8) ensures a non-increasing local objective for 0≤α≤2. β attenuates the reference-gradient term because fake-quant scoring precedes actual GPTQ rounding/feedback. A 4×4 grid, α∈{0.2,0.4,0.6,1}, β∈{0,0.05,0.2,1}, places the defaults within 0.023%, 0.021%, 0.416% and 0.518% relative C4 PPL of the best tested pair on Llama-2-7B, Llama-3-8B, Mistral-7B and DeepSeek-V2-Lite at W3. A finer 6×6 Llama-2-7B W3 sweep gives PPL 7.5935–7.6486; default 7.5991 is only 0.074% above the minimum. At fixed α=0.4, β=0.05 beats β=1 in all six tested W2/W3 settings.

2. Scale range. GPTQ's MSE scale search setting grid=100,maxshrink=0.8 yields 80 multipliers over [0.21,1.00]. We retain the 0.01 step and add 20 expansion candidates, giving 100 over [0.21,1.20]. On Llama-2-7B, varying only the upper bound gives WikiText-2 PPL:

Bits1.001.201.40
W214.966613.294012.9850
W35.90295.86895.8655
W45.56385.55515.5571

Removing expansion increases W2 PPL by 12.58%. The optimum depends on precision and model. For W2, extending the lower or upper endpoint gives:

Model[0.21,1.20][0.05,1.20][0.21,1.40]
Mistral-7B-v0.313.865314.341114.5220
Mistral-Nemo-12B14.252216.111718.6807

Wider ranges do not consistently help; thus we retain a 100-candidate budget.

3. 70B and memory. Llama-2-70B W3A16 WikiText-2 PPL after checkpoint reload is 3.8712/3.8515/3.8409/3.8400/3.7378 for GPTQ/GPTAQ/QEP/Qronos/RCDQ. RCDQ improves over the strongest baseline by 2.66%. Quantization on one A100-80GB with CPU offloading takes 208.36 min; peak GPU allocated memory is 25.85 GiB during quantization and 38.30 GiB during reload/evaluation. Peak quantization CPU RSS is 254.03 GiB. Blocks are processed individually, references are cached on CPU, and workspaces serve the current projection/group; GPU peaks therefore cannot be extrapolated linearly from total parameter count.

4. GDN and MoE. Added W3A16 results below report RCDQ; bold/underline denote best/second among five quantized methods. W=WikiText-2 PPL, C=C4 PPL; remaining metrics are accuracy (%).

ModelWC6-task meanMMLUGSM8K
OLMo-Hybrid-7B (GDN)7.452711.864669.7265.7366.49‾\underline{66.49}
DeepSeek-V2-Lite (MoE)6.79539.878665.9150.43‾\underline{50.43}22.14

Six-task evaluation is 0-shot, MMLU 5-shot, and GSM8K 8-shot greedy strict-match. Native GDN recurrence/convolution is retained. For MoE, FP/current paths route independently with FP routers. Token-aligned aggregate block outputs define a common reference despite different experts. Expert down compensation uses routing-weighted inputs and updates the common residual; gate/up use GPTQ. W3 expert-set mismatch rates are 32.00%/36.60% on WikiText-2/C4, so the gains include actual routing mismatches.

5. KronQ/QuIP#. For this comparison, we plug KronQ's input-side RHT into RCDQ for fairness. It is independent of our core pipeline; Stage 1/2 and GPTQ updates are unchanged. KronQ retains BiIP. Both use symmetric W2A16 and matched data. We select KronQ as it reports gains over QuIP# in most comparisons without E8P/finetuning (§5.2/Table 1).

Cells show RCDQ/KronQ; bold marks the better result. L=Llama; W/C=PPL; tasks=accuracy (%).

ModelWC6-task meanMMLU
L2-7B8.2398/8.436111.5044/12.175855.03/53.4528.96/25.67
L2-13B7.0500/11.21259.9211/16.814760.06/56.9032.02/28.14
L3-8B11.5974/14.924018.3850/24.701055.15/52.7129.86/26.59
L3.1-8B11.4632/12.592318.1171/22.856954.52/52.2729.85/26.43
L3.2-3B14.9738/15.868323.5425/26.346849.92/49.9326.16/26.39

RCDQ wins all 10 PPL comparisons and 4/5 models on each downstream metric.

6. W4A4. With shared full Hadamard rotation, per-token symmetric A4, and calibration/evaluation data, reloaded WikiText-2 PPL is:

ModelQuaRotAWQ (rotation-adapted)RCDQ
L2-7B6.1162‾\underline{6.1162}7.11456.0384
L3-8B8.2272‾\underline{8.2272}9.76068.0511

7. Reasoning. On DeepSeek-R1-Distill-Llama-8B at W3A16, RCDQ achieves MATH-500/6-task mean/MMLU accuracy of 77.40/62.59/48.91%, versus 74.60/61.21/48.55% for the strongest baseline, Qronos. MATH-500 improves by 2.8 points (14/500 additional correct answers). All methods share C4 calibration, prompts, per-question seeds, 0-shot, one sample/question, temperature=0.6, top_p=0.95, a 32,768-token output budget and Math-Verify scoring.

We will add these results and full protocols to the revised paper.

Thank you for the responses and for running additional experiments.

Some of my concerns have been addressed, particularly regarding the W4A4 results. However, I remain unconvinced of the method's generalizability, given its reliance on empirically selected hyperparameters such as α\alpha, β\beta, and the scale ranges. In particular, it remains unclear whether the same hyperparameter values and scale ranges would work reliably across other models without model-specific retuning. Although the responses to points 1 and 2 attempt to justify these heuristics, the supporting evidence is drawn from a limited set of models at W3 and does not sufficiently address these concerns.

I also remain unconvinced with the MoE results. In particular, updating weights when routing differs between full-precision and quantized inputs remains a concern.

For these reasons, I will maintain my original score. Thank you again for the additional effort and clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.