acceptodds
Under review as a conference paper at ICLR 2027

SCOPE-Q: ONE-CHECKPOINT QUANTIZATION-AWARE TRAINING FOR CROSS-FRAMEWORK LLM DEPLOYMENT

Abstract

Large language models (LLMs) are often deployed through several frameworks, such as OpenVINO, TensorRT, Core ML, and AIMET, each of which quantizes the model with its own quantization configuration. Most LLM quantization methods optimize a model for a single, fixed quantizer. We find that this assumption is costly in practice: at the same 4-bit precision, the MMLU accuracy of one checkpoint varies by 12–15 points across these four frameworks, mainly because of differences in their quantization configurations. In this paper, we propose SCOPEQ, a unified quantization-aware training method that produces a single checkpoint for multiple deployment frameworks. SCOPE-Q learns a low-rank update to the frozen pretrained weights while sampling one target quantization configuration at each training step, and distills knowledge from the full-precision model. After training, the update is merged into the weights, yielding a standard floating-point checkpoint that each framework quantizes without modification. Experiments on four LLMs ranging from 0.5B to 4.6B parameters show that a single SCOPE-Q checkpoint improves worst-case accuracy across frameworks by 7.5–8.4 points over round-to-nearest quantization and reduces the accuracy gap between frameworks by 44–69%, while training only 1.1–3.6% of the parameters.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, quantization, quantization-aware training, low-rank adaptation, weight-only quantization, cross-framework deployment, model compression, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2puPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SCOPE-Q trains one floating-point checkpoint for deployment under a predefined set of quantization configurations. It learns low-rank updates to frozen pretrained weights, samples a target configuration at each training step, and combines language-modeling and distillation losses. Quantization is applied to the merged weights during training, with scales recomputed rather than learned. The updates are then folded into a standard checkpoint that each deployment framework quantizes independently. Experiments across four toolchains show improvements in worst-case MMLU accuracy and reduced variation between frameworks, supported by single-configuration controls and additional analyses of scale, common configurations, and other tasks.

Critical strengths

The export design is practical. Training through quantization of the merged weights and recomputing scales makes the learned improvement portable through an ordinary floating-point checkpoint. Deployment requires no adapter or retained training-time quantizer state. This directly addresses the inconvenience of maintaining separate artifacts for several deployment targets.

The paper includes a useful controlled comparison. Table 3 and Appendix F hold the rank, data, budget, and objective fixed while varying the training configurations. These results support an additional benefit from multi-configuration training beyond the underlying low-rank QAT recipe. Reporting worst-case accuracy alongside the framework gap is also appropriate, since a smaller gap alone could hide degradation of the strongest target.

Critical weaknesses

The comparison with production QAT recipes leaves the accuracy–cost trade-off unresolved. Table 5 changes the learning rate, training budget, parameterization, and QAT implementation across methods. SCOPE-Q uses 2,000 steps at (10^{-4}), while the TensorRT and AIMET baselines use 500–1,000 steps at (10^{-5}). These differences make it difficult to attribute the relative performance to the proposed approach. The matched controls on Qwen2.5-0.5B and Qwen3-1.7B are meaningful and should receive credit. However, they do not establish how the single-checkpoint approach compares with separately tuned specialists across the larger models, where the results are more mixed. I would like to understand how much accuracy the one-checkpoint constraint costs under comparable training recipes and what training resources it saves.

The deployment motivation needs stronger evidence against a common configuration. In Table 4, common group-128 RTN already exceeds native SCOPE-Q in worst-case accuracy on Qwen2.5-1.5B (57.95 versus 54.25) and Gemma-4 E2B (40.49 versus 39.35). The independently configured user setting is reasonable, but the paper also motivates native configurations through hardware preferences without measuring the associated latency, memory, or energy trade-offs. Appendix M checks accuracy on two runtimes using an earlier checkpoint; it does not establish that trade-off for the final method. This matters because shared-weight training across quantization granularities and symmetry choices already appears in EQ-Net, as Section 3 acknowledges. The contribution therefore depends substantially on demonstrating the value of this particular deployment setting.

Questions for authors

Specialized baselines: On the larger models, how does SCOPE-Q compare with separately tuned single-configuration models using the same low-rank parameterization, data, and objective, with each specialist evaluated on its own target? Please report both accuracy and training cost.

Deployment trade-off: Using the final checkpoint, what accuracy, latency, and memory trade-offs arise between native SCOPE-Q and common group-128 RTN on the relevant devices? Which concrete deployment constraints make the common configuration unsuitable?

Evaluation breadth: Do the gains persist beyond MMLU when evaluated through the actual deployment toolchains on at least one larger model? Appendix H’s additional tasks use simulated quantizers on the 0.5B model, so they provide limited evidence for the broader deployment claim.

3 comments

Response (1/3)

We thank the reviewer for a precise review and for crediting the export design and the matched controls. The three questions are the ones we would have asked ourselves, and we have run new experiments for each. In brief:

  • Matched specialists at a larger scale, with cost. On Qwen2.5-1.5B and Gemma-4 E2B, three single-configuration specialists trained with the identical recipe together cost about two to three times one SCOPE-Q run. Each specialist wins its own column. On Qwen2.5-1.5B, SCOPE-Q's worst case exceeds every specialist's by 3–6 points; on Gemma-4 E2B, the per-channel specialist has the higher worst case (41.51 vs 40.04), while SCOPE-Q has the higher mean (43.39 vs 42.77). We report both outcomes.
  • Deployment trade-off. On an Apple M4, 4-bit block-128 weights are not executed by the Neural Engine; latency is 6.7× (prefill) and 3.8× (decode) that of per-channel weights, at the same size. TensorRT offers no per-channel INT4.
  • Beyond MMLU through the toolchains. On Qwen3-1.7B, ARC-Challenge, HellaSwag, and WinoGrande through all four toolchains show the MMLU pattern: the largest gains on the per-channel frameworks and a higher worst case on every task.

All MMLU numbers use the paper's 5,700-question protocol; each framework quantizes with its own toolchain at its native configuration.

Q1. Specialized baselines on the larger models, with accuracy and training cost

We agree that Table 5 changes budget, learning rate and parameterization across methods and answers a deployment question rather than the controlled one. The controlled comparison is Table 3, which we have extended to Qwen2.5-1.5B and Gemma-4 E2B: one checkpoint per distinct configuration, trained with the identical recipe (rank 32, WikiText-2, 2,000 steps, the same objective), each evaluated on all four frameworks, with wall-clock training time on the GPU of Section 5.1 (the Gemma-4 E2B runs used an A100 80 GB). Bold cells are each specialist on its own target. The SCOPE-Q rows are new training runs; the paper's runs (Table 4) have worst cases 54.25 and 39.35.

Qwen2.5-1.5B

checkpointOpenVINOTensorRTCore MLAIMETworstgaptraining
RTN59.5158.2346.6549.7746.6512.86none
OpenVINO specialist (affine g64)60.2858.3048.6050.7948.6011.68157 min
TensorRT specialist (sym g128)59.6358.8950.1451.3750.149.49152 min
per-channel specialist51.8452.7454.4755.6151.843.77149 min
SCOPE-Q, one checkpoint60.1158.5454.8855.5854.885.23221 min

Gemma-4 E2B

checkpointOpenVINOTensorRTCore MLAIMETmeanworstgaptraining
RTN47.0041.1931.8934.4238.6331.8915.11none
OpenVINO specialist (affine g64)49.35*44.3333.0434.6840.3533.0416.31237 min
TensorRT specialist (sym g128)45.9345.9636.1937.5141.4036.199.77228 min
per-channel specialist44.2542.8942.4441.5142.7741.512.74224 min
SCOPE-Q, one checkpoint47.3245.3740.8240.0443.3940.047.28228 min

* Above the pretrained full-precision 49.00 but below this specialist's own unquantized 50.18 (same protocol): training raises Gemma's unquantized accuracy, as Table 10 already shows (+0.82 for SCOPE-Q), and the specialist pays only 0.83 points of quantization cost on its own grid.

Response (2/3)

On both models each specialist wins its own column. The two recipes have the same per-step cost, since both evaluate one quantizer per step: Section 5.1 reports 149 minutes for a SCOPE-Q run on Qwen2.5-1.5B under this recipe, the same as a single specialist here, so three specialists cost about three SCOPE-Q runs (458 against 149–221 minutes on Qwen2.5-1.5B, 689 against 228 on Gemma-4 E2B). The two models then differ, and we report both. On Qwen2.5-1.5B, SCOPE-Q matches or beats the per-channel specialist on the per-channel frameworks, and its worst case exceeds every specialist's by 3.0–6.3 points, because each specialist loses accuracy on the configurations it was not trained for (the per-channel specialist loses 7.7 points on OpenVINO relative to RTN). On Gemma-4 E2B the per-channel specialist is stronger on the per-channel frameworks (42.44 and 41.51 against 40.82 and 40.04), so it has the higher worst case, by 1.5 points, and the smaller gap, while it loses 3.1 points to SCOPE-Q on OpenVINO and 2.5 on TensorRT, leaving SCOPE-Q the higher mean (43.39 against 42.77). The one-checkpoint constraint therefore costs 0.2–2.0 points on the strongest frameworks; on the weakest it gains 3–6 points on the three Qwen models and loses 1.5 on Gemma-4 E2B, where the paper already reports SCOPE-Q's weakest results (Section 5.3). We have added this comparison to Appendix F and summarized it in Section 5.4, stated the Gemma outcome explicitly, and labeled Table 5 as deployment context rather than a matched comparison. We did not re-run 7B; Section 5.6's statement that framework-specific QAT can be stronger where feasible stands, and the Gemma result points the same way.

Q2. Deployment trade-off between native SCOPE-Q and common group-128 RTN

Accuracy. Table 4 stands: common group-128 RTN exceeds native SCOPE-Q in worst case by 3.7 points on Qwen2.5-1.5B (57.95 vs 54.25) and 1.1 on Gemma-4 E2B (40.49 vs 39.35). The question is what that accuracy costs on the device.

Latency and size. We measured this on Apple silicon, the target whose native configuration differs most from the common one. We built Core ML models from the real projection weights of Qwen2.5-0.5B (attention q/o and MLP gate/down chains, all 24 layers), quantized them to 4 bits with coremltools 9.0 at per-channel, block-128 and block-64, and timed them (Apple M4, macOS 27.2.1; median of 50 calls after warm-up):

MLP chain, 64-token prefillfloat16int4 per-channel (native)int4 block-128 (common)int4 block-64
Neural Engine6.42 ms1.76 ms11.82 ms11.76 ms
GPU10.57 ms9.43 ms9.55 ms9.55 ms
compiled size399.0 MB100.1 MB102.9 MB106.0 MB

For single-token decode the Neural Engine takes 1.74 ms per-channel against 6.55 ms block-128. Core ML's compute plan explains it: with per-channel scales every linear layer is assigned to the Neural Engine; with block-128 or block-64 scales every linear layer is assigned to the CPU. On this device the 3.7 points of common RTN therefore buy a 6.7× prefill and 3.8× decode slowdown at the same model size. The measurement is configuration-level, so it does not depend on the checkpoint; the earlier checkpoint of Appendix M does not affect it.

Response (3/3)

Concrete constraints. A release for the Neural Engine cannot use blockwise INT4 without losing the accelerator, and a release for TensorRT cannot use per-channel INT4, which ModelOpt does not ship. Where the hardware is indifferent, as on the Mac GPU, the common configuration is free, and Table 16 shows the SCOPE-Q checkpoint is the better one to apply it to (best cell in every row). We did not measure OpenVINO CPU latency, a Hexagon device, energy, or peak runtime memory, which differs between the configurations only by scale storage (under 3% of the weight size). This is the evidence on which the contribution relative to shared-weight training such as EQ-Net rests: in the setting we study, the configurations are fixed by hardware and toolchains, and the deliverable is an ordinary floating-point checkpoint. We have added the table to Appendix C and the trade-off to Section 5.5.

Q3. Do the gains persist beyond MMLU through the deployment toolchains on a larger model?

Yes. We evaluated Qwen3-1.7B on ARC-Challenge (full test set), HellaSwag (first 2,000 validation examples), and WinoGrande (full validation set), zero-shot, length-normalized accuracy, with each framework quantizing the pretrained and the SCOPE-Q checkpoint through its own toolchain at its native configuration (pretrained → SCOPE-Q):

frameworkARC-ChallengeHellaSwagWinoGrande
OpenVINO42.49 → 41.8151.85 → 50.6058.17 → 61.40
TensorRT38.31 → 43.4351.05 → 51.9558.17 → 60.46
Core ML31.91 → 38.4048.70 → 51.5053.99 → 60.14
AIMET34.13 → 36.7747.70 → 50.6555.72 → 61.09

The MMLU pattern holds on every task: the largest gains fall on the per-channel frameworks (Core ML +6.5, +2.8, +6.2; AIMET +2.6, +3.0, +5.4), and the worst case over the four frameworks rises on all three tasks (ARC 31.91 → 36.77, HellaSwag 47.70 → 50.60, WinoGrande 53.99 → 60.14). OpenVINO, the strongest framework before training, changes by −0.7, −1.3 and +3.2. These results now support the deployment claim in Appendix H; the earlier simulated-quantizer table is retained for the perplexity measurements.

Reproducibility. Our general comment reports a re-run of the Qwen2.5-0.5B experiments, which reproduce to two decimals, and a fourth training run that widens the seed spread; Table 2 now also reports the four-run mean. The matched controls the reviewer credited are unaffected: the three specialists of Table 11 reproduce exactly.

Summary of changes. Matched specialist comparison extended to Qwen2.5-1.5B and Gemma-4 E2B with training cost (Appendix F, summarized in Section 5.4); Gemma outcome stated (Table 5 labeled as deployment context); Neural Engine measurements in Appendix C and the trade-off in Section 5.5; toolchain evaluations on three tasks in Appendix H; and Appendix D extended with the four-run analysis. We would be glad to run any further comparison the reviewer considers helpful.

Reviewer_AG7QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SCOPE-Q is a LoRA-based QAT method that produces a single floating-point checkpoint, which each framework then quantizes with its own native configuration. The method consistently outperforms RTN and, on most models, the framework-specific PTQ and QAT baselines.

Critical strengths
  1. Achieves strong accuracy while keeping each framework’s native quantization configuration unchanged.

  2. Trains efficiently on a single GPU with low resource requirements, despite being a QAT method.

Critical weaknesses

I currently lean toward rejection, mainly because (1) the reason for not using a common configuration is not clearly justified, and (2) stronger baselines applied consistently across frameworks are needed.

  1. Configuration vs Framework. Beyond vendor recommendations, the paper does not sufficiently justify why each framework must use a different configuration. Under a common configuration, RTN’s worst-case accuracy improves and the gap narrows substantially (Table 4). In this setting, the paper should clarify how SCOPE-Q differs from standard single-configuration QAT. It should also explain how the frameworks’ implementations differ when the same configuration is applied.

  2. Weak baselines. The paper should compare against calibration-free methods such as HQQ or SINQ (Muller et al., 2026), whose quantization cost is similar to RTN. It should also include a state-of-the-art PTQ method such as YAQA (Tseng et al., 2026) with an INT quantizer.

Questions for authors
  1. Under a common configuration, how does SCOPE-Q differ from standard single-configuration QAT?
  2. What system-level costs, such as latency or memory overhead, would a common configuration incur on each target hardware?
  3. Could you report results for HQQ, SINQ, and YAQA with an INT quantizer?
8 comments

Response (1/3)

We thank the reviewer for a careful review. Both main concerns, why each framework should keep its native configuration rather than adopt a common one and how SCOPE-Q compares with stronger calibration-free baselines, were under-supported in the submission; we ran new experiments for both. In brief:

  • Common configuration. At symmetric group-128 on all four frameworks, SCOPE-Q and a single-configuration group-128 QAT checkpoint are within 0.1 point in worst-case accuracy (43.00 vs 43.09); under native configurations the single-configuration checkpoint loses 6.6 points and SCOPE-Q (seed 0) 2.0, at identical training cost.
  • Hardware cost. On the Apple M4 Neural Engine, block-128 INT4 weights are not executed by the accelerator; latency is 6.7× (prefill) and 3.8× (decode) that of per-channel weights at the same size. TensorRT has no per-channel INT4.
  • Calibration-free baselines. HQQ and SINQ are strong with their own parameters (worst-case 38.8, 39.5; RTN 32.5) but fall below RTN once a framework requantizes the released weights (29.4, 29.8); SCOPE-Q reaches 39.1 ± 2.1 over four runs under that requantization.
  • Reproducibility. Paper numbers reproduce to two decimals; a fourth run widens the seed spread, and Table 2 will report the four-run mean (worst-case 39.1 instead of 40.13). A pooled bootstrap still places SCOPE-Q above the best specialist (+1.5, [+0.6, +2.4]) and RTN (+6.6, [+5.2, +8.0]).

New MMLU numbers use the paper's 5,700-question protocol, each framework's own toolchain at its native configuration (OpenVINO affine group-64, TensorRT symmetric group-128, Core ML and AIMET symmetric per-channel), and Qwen2.5-0.5B unless stated otherwise.

W1 / Q1. Under a common configuration, how does SCOPE-Q differ from single-configuration QAT?

They are equivalent in accuracy and cost; they differ when the common configuration is not applied. We constrained all four frameworks to symmetric group-128 (the setting of Table 4 and Tables 15–16) and compared SCOPE-Q with a checkpoint trained by the identical recipe (rank, data, budget, objective) against group-128 only:

checkpointconfigurationOpenVINOTensorRTCore MLAIMETworstgap
RTNcommon (Table 4)44.9839.4639.3341.9539.335.65
g128 QATcommon43.0944.8945.0244.3743.091.93
SCOPE-Q, seed 0common (new evaluation) (Table 16)44.7244.1644.1843.0043.001.72
g128 QATnative (Table 11)45.0044.8938.3536.5436.548.46
SCOPE-Q, seed 0native (Tables 4, 9)45.1643.3040.9641.3240.964.20

Both SCOPE-Q rows are the seed-0 checkpoint of Table 9. The g128 QAT and SCOPE-Q common-configuration rows are new evaluations on the rebuilt pipeline; Table 4 reports 43.30 worst case and 2.24 gap for this SCOPE-Q checkpoint, so the two checkpoints are within 0.1 point on the new evaluation and within 0.3 using Table 4's value. Table 2 reports the three-seed mean (40.13). Under native configurations the single-configuration checkpoint loses 6.6 points of worst-case accuracy and SCOPE-Q 2.0. Single-configuration QAT is the right choice only when the releaser controls every deployment; SCOPE-Q costs nothing then and remains usable when deployments are independent.

Training cost. Identical by construction: both evaluate one quantizer per step, and only the sampled configuration differs. On one RTX A6000 both take 60 minutes on Qwen2.5-0.5B (59 in Section 5.1) with the same peak memory (frozen model, frozen teacher, rank-32 adapters).

Why implementations differ under one configuration. The 0.28–5.65-point gap at group-128 (Tables 4, 15) comes from what the configuration does not fix: handling of dimensions not divisible by the group size, ModelOpt's [−7, 7] clipping against NNCF's [−8, 7], coremltools' numpy rounding tie rule, and AIMET computing min-max encodings on calibration activations even in weight-only mode. We will list these in Appendix I.

Response (2/3)

Q2. What system-level costs would a common configuration incur on each target hardware?

The submission answered this only by citation (Table 7), which we agree is insufficient. We measured it on Apple silicon, the target whose native configuration differs most from the common one. We built Core ML models from the real projection weights of Qwen2.5-0.5B (attention q/o and MLP gate/down chains, all 24 layers), quantized them to 4 bits with coremltools 9.0 at per-channel, block-128 and block-64, and timed them on the Neural Engine, GPU and CPU (Apple M4, macOS 27.2.1; median of 50 calls after warm-up):

MLP chain, 64-token prefillfloat16int4 per-channel (native)int4 block-128 (common)int4 block-64
Neural Engine6.42 ms1.76 ms11.82 ms11.76 ms
GPU10.57 ms9.43 ms9.55 ms9.55 ms
CPU11.90 ms12.12 ms11.86 ms11.87 ms
compiled size399.0 MB100.1 MB102.9 MB106.0 MB
MLP chain, 1-token decodefloat16int4 per-channelint4 block-128int4 block-64
Neural Engine6.38 ms1.74 ms6.55 ms6.67 ms
GPU4.69 ms1.62 ms1.72 ms1.78 ms

The attention chain shows the same pattern (Neural Engine prefill: 0.51 ms per-channel vs 1.97 ms block-128). Core ML's compute plan explains it: with per-channel scales every linear operation is assigned to the Neural Engine; with block-128 or block-64 scales every linear is assigned to the CPU, so a blockwise model requested on the Neural Engine actually runs on the CPU. The common configuration therefore costs 6.7× the prefill and 3.8× the decode latency of the native one, with no size benefit. On the Mac GPU both run at the same speed, consistent with Apple's guidance that per-block quantization targets the GPU and per-channel the Neural Engine: the constraint belongs to the accelerator, not the toolchain.

Other targets. TensorRT: ModelOpt has no per-channel INT4 weight-only mode (Table 7), so the common configuration is the native one and there is no cost to measure. OpenVINO: affine group-64 and symmetric group-128 are both supported on CPU; we did not measure their latency difference within the discussion period and expect it to be small, as both are grouped formats of the same kernel family. Qualcomm Hexagon: we had no device and cite Qualcomm's documentation of its blockwise format as block scales that are integer multiples of a per-channel scale, which is why per-channel is native there. Runtime memory: we report compiled size and did not measure peak runtime memory, which is dominated by the weights and KV cache; the configurations differ only in scale storage (under 3% of the weight size here).

A Neural Engine release therefore cannot adopt group-128 without a 4–7× latency penalty, and a TensorRT release cannot adopt per-channel at all. We will add these measurements to Appendix C and cite them in Sections 1 and 5.5.

Response (3/3)

W2 / Q3. Could you report HQQ, SINQ and YAQA with an INT quantizer?

We added HQQ and SINQ on Qwen2.5-0.5B at each framework's granularity, scored two ways. The paper's setting is a released floating-point checkpoint that each framework quantizes with its own data-free procedure: NNCF, ModelOpt, coremltools and AIMET all take a floating-point model and derive their own quantizer parameters, and none imports HQQ or SINQ parameters (pre-quantized GPTQ/AWQ checkpoints load into runtimes such as vLLM, not into these toolchains). We score (a) with each method's own parameters, as an in-framework implementation would achieve, and (b) after the framework requantizes the released weights, which is what reaches the device today.

methodscored withOpenVINOTensorRTCore MLAIMETworst
RTN (Table 2)framework45.9639.4632.4935.2832.49
HQQown parameters46.8145.8138.7738.7738.77
HQQframework requantizes46.3240.8129.4434.0429.44
SINQown parameters45.9345.5839.5339.5339.53
SINQframework requantizes44.5334.1629.8234.4429.82
SCOPE-Q, seed 0framework45.1643.3040.9641.3240.96
SCOPE-Q, mean of 4 runsframework46.0943.4339.1341.2139.13 ± 2.1

With their own parameters both are strong (at group-128 both exceed RTN by over six points; both use an affine grid, so those cells are not TensorRT's symmetric grid). After requantization the benefit does not survive: both fall below RTN on the per-channel frameworks, and SINQ falls 5.3 points below RTN on TensorRT (11.4 below its own-parameter result), because the optimized scales and zero points are discarded. SCOPE-Q carries its improvement in the floating-point weights: its four-run worst case after requantization (39.1) is comparable to what HQQ and SINQ reach only with parameters that do not reach the device. The methods are complementary: HQQ and SINQ on the SCOPE-Q checkpoint at per-channel reach 39.33 and 42.51 with their own parameters, against 38.77 and 39.53 on the pretrained one. We will add this to Section 5.3.

YAQA needs a full-model Kronecker-factored Hessian and a custom quantizer whose released code targets a different format; we could not obtain an INT4 per-channel or group variant these frameworks consume within the discussion period, so this request remains unmet. We will cite it as the strongest calibration-based alternative.

Reproducibility and statistical robustness. On a rebuilt pipeline with the public toolchains, all RTN, single-configuration and SCOPE-Q seed results reproduce Tables 6, 9 and 11 to two decimals. The target-list order affects the sampled sequence, so the same seed with a different order is a different run; one such run reached a worst case of 36.12 (Table 9 seeds: 40.96, 39.51, 39.91). Over four runs the worst case is 39.1 ± 2.1 and the mean accuracy 42.5 ± 0.4, against RTN's 32.49 and the best specialist's 37.60 (per-channel, Table 11). A paired item-level bootstrap pooled over the four runs (B = 10,000) gives +1.53 [+0.58, +2.43] over the specialist and +6.64 [+5.18, +8.00] over RTN. These intervals reflect question sampling; across training runs the margins over the specialist are +3.36, +1.91, +2.31 and −1.48, so the advantage holds in three of four runs and in the pooled estimate, and Table 8's seed-0 interval is the most favorable. Table 2 will report the four-run mean and Appendix D all runs and the pooled bootstrap; we also post this as a general comment.

Presentation. We will define the framework, configuration, and target with a worked example in Section 2 and place the common-configuration comparison beside Table 4 in Section 5.5.

We hope these measurements show that the native configurations are hardware constraints with a measured cost rather than vendor preferences, and we would be glad to run any further comparison the reviewer considers necessary.

Reviewer_v1LgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors find: same model, same quantization precision, same benchmark, performance varies across different frameworks. It is because different frameworks they have different quantization settings(per channel/grounp, groupsize, etc).

The author proposed SCOPE-Q: Fine-tune the model using LoRA method, based on the loss defined by next-token cross-entropy and perplexity between fine-tuned, quantized model and original one.

As a results, the gap of different frameworks' quantization configureation, narrows down a lot.

Critical strengths
  1. The evaluation supports original proposed problem. The LLM performance among different quantization configuraztion is much more stable after fine-tuning. SCOPE-Q improves worst-case accuracy and didn't impact best-case accuracy.
Critical weaknesses
  1. In training process, the quantization methods are sampled from a preset. And the main evaluations are also based on those configurations. Based on results shown in appendix, for the quantization method out of the training scope, benefits disappear. So I think it is the limitation of generalization.
  2. From my understanding, SCOPE-Q should be the final step before model release, but only trained on Wiki-Text2. However, some model capabilities such as function-calling may have been acquired from prior specialized fune-tuning. My concern is whether these capabilities are preserved after SCOPE-Q.
Questions for authors

Overall, the paper addresses a deployment problem and demonstrates that a single SCOPE-Q checkpoint can narrow down the gap amoung different quant-configs.

My main concerns are the scope of these benefits and the preservation of existing model capabilities. Section 5.6 acknowledges that the gains do not extend to unseen quantization configurations, raising questions about the additional training needed as deployment config updates. More importantly, it is unclear whether training SCOPE-Q on WikiText-2 preserves capabilities learned through earlier fine-tuning, such as function-calling. The decreases in unquantized MMLU accuracy reported in Table 10 motivate this concern.

3 comments

Response (1/3)

We thank the reviewer for a clear review. Both concerns, the scope of the benefit beyond the trained configurations and the preservation of capabilities acquired by earlier fine-tuning, are legitimate, and we have run new experiments for each. In brief:

  • Generalization. On an untrained combination inside the trained range, symmetric group-64 (the group size of the OpenVINO target with the symmetric grid of the other targets), the SCOPE-Q checkpoint gains +2.9 points over the pretrained checkpoint. The limitation of Appendix K is therefore confined to configurations coarser than any trained one. Adding a target costs one retraining run (59 minutes on Qwen2.5-0.5B and 3.5 hours on Gemma-4 E2B, Section 5.1).
  • Capability preservation. We trained SCOPE-Q on Qwen2.5-1.5B-Instruct and measured IFEval and MMLU before and after. Unquantized instruction following drops by 3.9 points (prompt-level strict), confirming the reviewer's concern in part. Deployed through the frameworks, the SCOPE-Q model retains more instruction following than the deployed pretrained model on Core ML, where quantization hurts most (+3.0), the same within 0.3 points on TensorRT and AIMET, and less on OpenVINO (−3.9); its worst case over the four frameworks improves from 16.8 to 19.8. On MMLU through the same toolchains the deployed SCOPE-Q model leads on Core ML (+6.4) and AIMET (+5.5), by 0.9 on TensorRT, and is equal on OpenVINO; the worst case rises from 49.25 to 55.60.

All MMLU numbers use the paper's 5,700-question protocol; each framework quantizes with its own toolchain at its native configuration.

Generalization to configurations outside the training set

We agree that SCOPE-Q optimizes a predefined set of target configurations and claims nothing for configurations outside it, as Section 5.6 and Appendix K state. Two points bound the practical cost, and a new experiment narrows the limitation.

First, the target set is known at release time: it is the list of frameworks the release supports. Adding a target means adding its configuration to the set and retraining, which takes 59 minutes on Qwen2.5-0.5B and 3.5 hours on Gemma-4 E2B on the single GPU of Section 5.1, with no change on the deployment side.

Second, both configurations of Appendix K lie outside the range of trained group sizes: group-32 is finer than every trained configuration and per-tensor coarser. We have now evaluated the Qwen2.5-0.5B checkpoints on an untrained combination inside that range, symmetric group-64. Group-64 was trained only with the affine grid of the OpenVINO target, and the symmetric grid only at group-128 and per-channel, so this grid was never seen in training (simulated quantizer, as in Appendix K):

configurationpretrainedSCOPE-Qchange
symmetric group-64 (trained group size, untrained grid)42.5845.46+2.88

Robustness therefore carries over to an untrained grid within the trained range of group sizes. Together with Appendix K, the picture is that the limitation is confined to configurations coarser than any trained one, where a 4-bit per-tensor grid also reduces the pretrained model to chance. We have added this result to Appendix K and restated Sections 5.6 and 6 accordingly.

Response (2/3)

Preservation of capabilities acquired by earlier fine-tuning

The submission did not test this, and the reviewer is right to ask, since Table 10 shows an unquantized cost of training. The distillation term is the mechanism intended to preserve such capabilities: the student matches the full-precision model's next-token distribution at every position, a property of the model rather than of WikiText-2, and the update is low-rank (2.4% of the parameters here, with B initialized at zero). To measure it directly, we trained SCOPE-Q on Qwen2.5-1.5B-Instruct with the paper's recipe (152 minutes) and evaluated instruction following with IFEval (541 prompts, lm-evaluation-harness, prompt-level and instruction-level strict accuracy) and MMLU, both unquantized and after each framework's own quantization:

checkpointweightsIFEval prompt-strictIFEval inst-strictMMLU
pretrained instructunquantized27.940.861.67
pretrained instructOpenVINO 4-bit27.041.558.93
pretrained instructTensorRT 4-bit23.536.657.42
pretrained instructCore ML 4-bit16.829.149.25
pretrained instructAIMET 4-bit21.635.550.74
SCOPE-Qunquantized24.036.160.33
SCOPE-QOpenVINO 4-bit23.136.158.91
SCOPE-QTensorRT 4-bit23.336.958.32
SCOPE-QCore ML 4-bit19.833.755.60
SCOPE-QAIMET 4-bit21.335.156.23

Three things are visible. First, SCOPE-Q training on WikiText-2 does cost instruction following in the unquantized model: 3.9 points of prompt-level and 4.7 points of instruction-level strict accuracy, and 1.3 MMLU points. The reviewer's concern is therefore partly borne out, and we state it as a limitation with these numbers. Second, the comparison that matters for deployment is between the two quantized models. Plain quantization already costs the pretrained instruct model 11.1 points of prompt-level IFEval on Core ML and 6.3 on AIMET; the deployed SCOPE-Q model retains more instruction following than the deployed pretrained model on Core ML (+3.0 prompt-level, +4.6 instruction-level), the same within 0.3 points on TensorRT and AIMET, and less on OpenVINO (−3.9), the framework whose fine affine grid loses almost nothing to quantization. The worst case over the four frameworks rises from 16.8 to 19.8 and the mean is unchanged (21.9 vs 22.2), so the training trades a small uniform loss for protection where quantization hurts most. MMLU through the same toolchains shows the same pattern more strongly: plain quantization costs the pretrained instruct model 12.4 points on Core ML and 10.9 on AIMET, and the deployed SCOPE-Q model leads the deployed pretrained model by 6.4 and 5.5 points there, by 0.9 on TensorRT, and is equal on OpenVINO (58.91 vs 58.93), with the worst case rising from 49.25 to 55.60. Third, the drop from unquantized to deployed is smaller for SCOPE-Q (IFEval 24.0 → 19.8 at worst, MMLU 60.33 → 55.60) than for the pretrained model (27.9 → 16.8 and 61.67 → 49.25), which is what the method is designed to do.

On function calling specifically. IFEval measures instruction following rather than function calling; we could not run a function-calling benchmark within the discussion period and use IFEval as the closest available measure of a capability acquired in post-training. We state this scope explicitly in the new appendix.

Response (3/3)

Mitigations. Two follow directly from the mechanism, and we have added them to Section 6: distilling on a corpus that includes instruction data rather than WikiText-2 alone, and distilling on the model's own generations, which covers behaviors the training text does not. We could not run them within the discussion period. All IFEval numbers are without a chat template, applied identically to every row, so absolute values are lower than the model card's but the before/after comparison is like for like.

On soundness. We would gently ask the reviewer to weigh the controls the submission already contains, each run to test a claim rather than assume it: a matched single-configuration ablation (Table 3), multiple training seeds (Table 9), paired item-level bootstraps (Table 8), and a negative result on the obvious mechanism (Table 13). The limitation the reviewer identified concerns the scope of the claim, which Section 6 states and which the new group-64 result narrows; the capability cost is real and is now measured rather than unknown. Our general comment also reports a re-run of the Qwen2.5-0.5B experiments, which reproduce to two decimals, and a fourth training run that widens the seed spread; Table 2 now also reports the four-run mean.

Summary of changes. Sections 5.6 and 6 and Appendix K were restated with the group-64 result; the instruct-model experiment was added as a new appendix with the table above, the unquantized cost stated as a limitation in Section 6 together with the two mitigations and the function-calling scope; and Appendix D was extended with the four-run seed analysis. We hope these measurements address the reviewer's concerns, and we would be glad to run any further test the reviewer considers helpful.

Comments on the paper

1 comment

General comment to all reviewers and the area chair: revised PDF and summary of the discussion

We thank the three reviewers for careful, constructive reviews and for the follow-up discussion. We have uploaded a revised PDF. Every addition is typeset in blue; no text of the submission was removed. Table, section, and appendix numbers in our responses refer to the submitted version; the revision adds tables within Appendices C, D, E, F, H, I, and K, so later appendix tables are renumbered there, while appendix letters A–M are unchanged and two appendices (N, O) are new.

What the discussion asked for and what we added.

  • Why native configurations rather than a common one (AG7Q, 2puP). Measured on an Apple M4: Core ML executes blockwise 4-bit weights on the CPU rather than the Neural Engine, so the common group-128 configuration costs 6.7× the prefill and 3.8× the decode latency of the native per-channel configuration at the same size, while on the Mac GPU the two are equal; TensorRT offers no per-channel INT4. Appendix C and Section 5.5.
  • SCOPE-Q versus single-configuration QAT under a common configuration (AG7Q). Equivalent (worst case 43.00 vs 43.09 on Qwen2.5-0.5B); under native configurations the single-configuration checkpoint falls to 36.54 and SCOPE-Q to 40.96. The implementation differences that remain under one configuration are listed. Appendix I.
  • Stronger training-free baselines (AG7Q). HQQ and SINQ are strong with their own quantizer parameters but fall below RTN once a framework requantizes the released weights. Pre-SINQ, which folds SINQ's column scales into a floating-point checkpoint, matches SCOPE-Q on the two grouped-grid frameworks (TensorRT 43.51 vs 43.43) and stays below RTN on the two per-channel frameworks (worst case 30.96 vs RTN 32.49 and SCOPE-Q 39.13 ± 2.09). Appendix N.
  • Matched specialists at larger scale, with cost (2puP, AG7Q). On Qwen2.5-1.5B and Gemma-4 E2B, three specialists with the identical recipe cost about three SCOPE-Q runs. Against one specialist per framework, the one-checkpoint constraint costs 0.2–2.0 points on fine-grained configurations and at most 1.5 points on coarse ones, depending on the model; the per-channel specialist's deficit on its own grid on Qwen2.5-0.5B is explained by co-adaptation to that grid (unquantized MMLU −4.03 vs −1.2 to −1.6 for SCOPE-Q's four runs). Appendices E and F.
  • Beyond MMLU through the toolchains (2puP). On Qwen3-1.7B, ARC-Challenge, HellaSwag and WinoGrande through all four toolchains show the MMLU pattern; the worst case rises on every task. Appendix H.
  • Configurations outside the training set (v1Lg, AG7Q). On untrained grids within the trained range of group sizes and finer (symmetric group-64 and group-32), SCOPE-Q gains 2.3–3.1 points over two runs, more than any single-configuration specialist; the limitation is confined to configurations coarser than any trained one. Appendix K, Sections 5.6 and 6.
  • Capabilities acquired by post-training (v1Lg). On Qwen2.5-1.5B-Instruct, SCOPE-Q costs 3.9 IFEval points unquantized, stated as a limitation; deployed through the frameworks it raises the IFEval worst case from 16.8 to 19.8 and the MMLU worst case from 49.25 to 55.60. Function calling is not measured. Appendix O and Section 6.

Corrections we report ourselves. All Qwen2.5-0.5B results of Tables 6, 9, 10 and 11 reproduce to two decimals on a rebuilt pipeline. A fourth training run, differing only in the order of the target list, widens the seed spread: the worst case over four runs is 39.13 ± 2.09 rather than 40.13 ± 0.75 over three. The claims over RTN are unaffected; the margin over the best single-configuration run is +1.53 (interval [+0.58, +2.43]) and holds in three of four runs. Table 2's caption and Appendix D report this. The group-32 entry of Table 18 (46.04 for both checkpoints) is superseded by a re-evaluation with the training simulator, in which the pretrained model scores 43.56; Section 5.6 and Appendix K are corrected.

The claim, as now stated in Sections 1 and 6. For a single released checkpoint whose deployments the releaser does not control, SCOPE-Q improves the worst case over RTN on every model, matches the best training-free one-checkpoint baseline on grouped grids and exceeds it on per-channel ones, and costs a small, measured accuracy loss against hardware-matched specialists in exchange for one artifact and one training run. Where the releaser controls every deployment, a common configuration or one specialist per hardware remains a sound alternative.

We are grateful for a discussion that has made the paper's claim more precise, and we would be glad to run any further checks the reviewers or the area chair consider necessary.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.