- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SCOPE-Q trains one floating-point checkpoint for deployment under a predefined set of quantization configurations. It learns low-rank updates to frozen pretrained weights, samples a target configuration at each training step, and combines language-modeling and distillation losses. Quantization is applied to the merged weights during training, with scales recomputed rather than learned. The updates are then folded into a standard checkpoint that each deployment framework quantizes independently. Experiments across four toolchains show improvements in worst-case MMLU accuracy and reduced variation between frameworks, supported by single-configuration controls and additional analyses of scale, common configurations, and other tasks.
Critical strengths
The export design is practical. Training through quantization of the merged weights and recomputing scales makes the learned improvement portable through an ordinary floating-point checkpoint. Deployment requires no adapter or retained training-time quantizer state. This directly addresses the inconvenience of maintaining separate artifacts for several deployment targets.
The paper includes a useful controlled comparison. Table 3 and Appendix F hold the rank, data, budget, and objective fixed while varying the training configurations. These results support an additional benefit from multi-configuration training beyond the underlying low-rank QAT recipe. Reporting worst-case accuracy alongside the framework gap is also appropriate, since a smaller gap alone could hide degradation of the strongest target.
Critical weaknesses
The comparison with production QAT recipes leaves the accuracy–cost trade-off unresolved. Table 5 changes the learning rate, training budget, parameterization, and QAT implementation across methods. SCOPE-Q uses 2,000 steps at (10^{-4}), while the TensorRT and AIMET baselines use 500–1,000 steps at (10^{-5}). These differences make it difficult to attribute the relative performance to the proposed approach. The matched controls on Qwen2.5-0.5B and Qwen3-1.7B are meaningful and should receive credit. However, they do not establish how the single-checkpoint approach compares with separately tuned specialists across the larger models, where the results are more mixed. I would like to understand how much accuracy the one-checkpoint constraint costs under comparable training recipes and what training resources it saves.
The deployment motivation needs stronger evidence against a common configuration. In Table 4, common group-128 RTN already exceeds native SCOPE-Q in worst-case accuracy on Qwen2.5-1.5B (57.95 versus 54.25) and Gemma-4 E2B (40.49 versus 39.35). The independently configured user setting is reasonable, but the paper also motivates native configurations through hardware preferences without measuring the associated latency, memory, or energy trade-offs. Appendix M checks accuracy on two runtimes using an earlier checkpoint; it does not establish that trade-off for the final method. This matters because shared-weight training across quantization granularities and symmetry choices already appears in EQ-Net, as Section 3 acknowledges. The contribution therefore depends substantially on demonstrating the value of this particular deployment setting.
Questions for authors
Specialized baselines: On the larger models, how does SCOPE-Q compare with separately tuned single-configuration models using the same low-rank parameterization, data, and objective, with each specialist evaluated on its own target? Please report both accuracy and training cost.
Deployment trade-off: Using the final checkpoint, what accuracy, latency, and memory trade-offs arise between native SCOPE-Q and common group-128 RTN on the relevant devices? Which concrete deployment constraints make the common configuration unsuitable?
Evaluation breadth: Do the gains persist beyond MMLU when evaluated through the actual deployment toolchains on at least one larger model? Appendix H’s additional tasks use simulated quantizers on the 0.5B model, so they provide limited evidence for the broader deployment claim.
3 comments
Response (2/3)
On both models each specialist wins its own column. The two recipes have the same per-step cost, since both evaluate one quantizer per step: Section 5.1 reports 149 minutes for a SCOPE-Q run on Qwen2.5-1.5B under this recipe, the same as a single specialist here, so three specialists cost about three SCOPE-Q runs (458 against 149–221 minutes on Qwen2.5-1.5B, 689 against 228 on Gemma-4 E2B). The two models then differ, and we report both. On Qwen2.5-1.5B, SCOPE-Q matches or beats the per-channel specialist on the per-channel frameworks, and its worst case exceeds every specialist's by 3.0–6.3 points, because each specialist loses accuracy on the configurations it was not trained for (the per-channel specialist loses 7.7 points on OpenVINO relative to RTN). On Gemma-4 E2B the per-channel specialist is stronger on the per-channel frameworks (42.44 and 41.51 against 40.82 and 40.04), so it has the higher worst case, by 1.5 points, and the smaller gap, while it loses 3.1 points to SCOPE-Q on OpenVINO and 2.5 on TensorRT, leaving SCOPE-Q the higher mean (43.39 against 42.77). The one-checkpoint constraint therefore costs 0.2–2.0 points on the strongest frameworks; on the weakest it gains 3–6 points on the three Qwen models and loses 1.5 on Gemma-4 E2B, where the paper already reports SCOPE-Q's weakest results (Section 5.3). We have added this comparison to Appendix F and summarized it in Section 5.4, stated the Gemma outcome explicitly, and labeled Table 5 as deployment context rather than a matched comparison. We did not re-run 7B; Section 5.6's statement that framework-specific QAT can be stronger where feasible stands, and the Gemma result points the same way.
Q2. Deployment trade-off between native SCOPE-Q and common group-128 RTN
Accuracy. Table 4 stands: common group-128 RTN exceeds native SCOPE-Q in worst case by 3.7 points on Qwen2.5-1.5B (57.95 vs 54.25) and 1.1 on Gemma-4 E2B (40.49 vs 39.35). The question is what that accuracy costs on the device.
Latency and size. We measured this on Apple silicon, the target whose native configuration differs most from the common one. We built Core ML models from the real projection weights of Qwen2.5-0.5B (attention q/o and MLP gate/down chains, all 24 layers), quantized them to 4 bits with coremltools 9.0 at per-channel, block-128 and block-64, and timed them (Apple M4, macOS 27.2.1; median of 50 calls after warm-up):
| MLP chain, 64-token prefill | float16 | int4 per-channel (native) | int4 block-128 (common) | int4 block-64 |
|---|---|---|---|---|
| Neural Engine | 6.42 ms | 1.76 ms | 11.82 ms | 11.76 ms |
| GPU | 10.57 ms | 9.43 ms | 9.55 ms | 9.55 ms |
| compiled size | 399.0 MB | 100.1 MB | 102.9 MB | 106.0 MB |
For single-token decode the Neural Engine takes 1.74 ms per-channel against 6.55 ms block-128. Core ML's compute plan explains it: with per-channel scales every linear layer is assigned to the Neural Engine; with block-128 or block-64 scales every linear layer is assigned to the CPU. On this device the 3.7 points of common RTN therefore buy a 6.7× prefill and 3.8× decode slowdown at the same model size. The measurement is configuration-level, so it does not depend on the checkpoint; the earlier checkpoint of Appendix M does not affect it.
Response (3/3)
Concrete constraints. A release for the Neural Engine cannot use blockwise INT4 without losing the accelerator, and a release for TensorRT cannot use per-channel INT4, which ModelOpt does not ship. Where the hardware is indifferent, as on the Mac GPU, the common configuration is free, and Table 16 shows the SCOPE-Q checkpoint is the better one to apply it to (best cell in every row). We did not measure OpenVINO CPU latency, a Hexagon device, energy, or peak runtime memory, which differs between the configurations only by scale storage (under 3% of the weight size). This is the evidence on which the contribution relative to shared-weight training such as EQ-Net rests: in the setting we study, the configurations are fixed by hardware and toolchains, and the deliverable is an ordinary floating-point checkpoint. We have added the table to Appendix C and the trade-off to Section 5.5.
Q3. Do the gains persist beyond MMLU through the deployment toolchains on a larger model?
Yes. We evaluated Qwen3-1.7B on ARC-Challenge (full test set), HellaSwag (first 2,000 validation examples), and WinoGrande (full validation set), zero-shot, length-normalized accuracy, with each framework quantizing the pretrained and the SCOPE-Q checkpoint through its own toolchain at its native configuration (pretrained → SCOPE-Q):
| framework | ARC-Challenge | HellaSwag | WinoGrande |
|---|---|---|---|
| OpenVINO | 42.49 → 41.81 | 51.85 → 50.60 | 58.17 → 61.40 |
| TensorRT | 38.31 → 43.43 | 51.05 → 51.95 | 58.17 → 60.46 |
| Core ML | 31.91 → 38.40 | 48.70 → 51.50 | 53.99 → 60.14 |
| AIMET | 34.13 → 36.77 | 47.70 → 50.65 | 55.72 → 61.09 |
The MMLU pattern holds on every task: the largest gains fall on the per-channel frameworks (Core ML +6.5, +2.8, +6.2; AIMET +2.6, +3.0, +5.4), and the worst case over the four frameworks rises on all three tasks (ARC 31.91 → 36.77, HellaSwag 47.70 → 50.60, WinoGrande 53.99 → 60.14). OpenVINO, the strongest framework before training, changes by −0.7, −1.3 and +3.2. These results now support the deployment claim in Appendix H; the earlier simulated-quantizer table is retained for the perplexity measurements.
Reproducibility. Our general comment reports a re-run of the Qwen2.5-0.5B experiments, which reproduce to two decimals, and a fourth training run that widens the seed spread; Table 2 now also reports the four-run mean. The matched controls the reviewer credited are unaffected: the three specialists of Table 11 reproduce exactly.
Summary of changes. Matched specialist comparison extended to Qwen2.5-1.5B and Gemma-4 E2B with training cost (Appendix F, summarized in Section 5.4); Gemma outcome stated (Table 5 labeled as deployment context); Neural Engine measurements in Appendix C and the trade-off in Section 5.5; toolchain evaluations on three tasks in Appendix H; and Appendix D extended with the four-run analysis. We would be glad to run any further comparison the reviewer considers helpful.
Response (1/3)
We thank the reviewer for a precise review and for crediting the export design and the matched controls. The three questions are the ones we would have asked ourselves, and we have run new experiments for each. In brief:
All MMLU numbers use the paper's 5,700-question protocol; each framework quantizes with its own toolchain at its native configuration.
Q1. Specialized baselines on the larger models, with accuracy and training cost
We agree that Table 5 changes budget, learning rate and parameterization across methods and answers a deployment question rather than the controlled one. The controlled comparison is Table 3, which we have extended to Qwen2.5-1.5B and Gemma-4 E2B: one checkpoint per distinct configuration, trained with the identical recipe (rank 32, WikiText-2, 2,000 steps, the same objective), each evaluated on all four frameworks, with wall-clock training time on the GPU of Section 5.1 (the Gemma-4 E2B runs used an A100 80 GB). Bold cells are each specialist on its own target. The SCOPE-Q rows are new training runs; the paper's runs (Table 4) have worst cases 54.25 and 39.35.
Qwen2.5-1.5B
Gemma-4 E2B
* Above the pretrained full-precision 49.00 but below this specialist's own unquantized 50.18 (same protocol): training raises Gemma's unquantized accuracy, as Table 10 already shows (+0.82 for SCOPE-Q), and the specialist pays only 0.83 points of quantization cost on its own grid.