- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
QGATE repairs disagreement between quantized LLMs and their full-precision references. It targets the worst tolerance-normalized behavioral slice, proposes native-format precision and scale patches, and accepts changes after a full discovery-ledger check. A final report combines exact counts on discovery/evaluation universes with simultaneous upper bounds on independent samples. Three deployments report 37.2-60.3% reductions on their initial worst slices with small storage and measured latency increases.
Critical strengths
- Slice-specific disagreement is a useful complement to perplexity and average accuracy. Explicitly reporting unresolved slices is a sensible deployment practice.
- The native-format patch design avoids adding inference modules. The scale-only result and the grammar/scoring ablations help explain the benefits of the proposed loop.
- The paper explicitly distinguishes full-census measurements from sampled bounds and discloses configurations that remain uncertified. This is more informative than reporting only the repaired worst slice.
Critical weaknesses
- The evidence for generalizable repair is insufficient. Eighteen of the 23 slices, including the headline bias and safety targets, are optimized and evaluated on the same complete universes (Section 4.4; Appendix A.2). Exact census counts are valid descriptions of these sets, but do not establish improvement on unseen examples. The five independent regime-L slices are unchanged on cfg1 (Table 8), so the independent evaluation does not demonstrate the claimed repair benefit. The practical question is whether targeting these flips repairs a behavior beyond the examples used during search.
- Behavioral agreement is not safety or fairness. The objective counts beneficial and harmful departures from the reference identically. Moreover, the safety probe asks the model to choose between comply/refuse/partial labels (Appendix A.6), rather than measuring its actual generated responses. The paper needs directional error analysis and held-out behavioral tests before the bias/safety motivation is convincingly supported.
- Key implementation details remain missing despite the long appendix. Equation 8 specifies a scale-fitting objective but not the solver, iterations, initialization, or constraints imposed by packed EXL2/GGUF scales. Channel-range enumeration, shortlist size, the actual surcharge entries, and measured search time are also absent. A fixed seed does not resolve these omissions. Counting accepted steps and sweep bounds is not an adequate account of candidate-scoring cost.
- The baseline evidence is difficult to interpret. Several comparisons are adapted '-style' implementations, including a diffusion-model calibration method ported to autoregressive models. They require explicit hyperparameters, fidelity checks, and actual compute accounting. Appendix A.5 also acknowledges that the four strongest competitors do not differ significantly at the allocated threshold. Point-estimate superiority is useful, but weaker than a convincing advantage over the nearest alternatives.
- The deployment and certification claims need tighter boundaries. The hard latency constraint applies to an additive surcharge model, not measured latency; three final measurements do not validate its accuracy across candidate combinations. The statistical theorem is a standard Clopper-Pearson/Bonferroni construction conditional on independent sealed samples, and offers no generalization guarantee for the 18 optimized universes or correctness/safety guarantee. The contribution is principally an empirical repair procedure, rather than a new verification result.
Minor Comments:
- The abstract contains too many numerical results and configuration-specific details. Retain only the central result and use the remaining space to explain the problem, method, and significance.
- Use leading zeros consistently in tables: write
0.xxxrather than.xxx. - Export diagrams and plots as vector graphics rather than embedding rasterized screenshots. Figure 1 is currently a high-resolution raster, but its small labels remain difficult to read at publication size. A vector export with larger labels and a simpler layout would improve readability; increasing image resolution alone would not address the problem.
- The repeated challenge-by-challenge narration and protocol qualifications make a relatively simple loop harder to follow. Reduce repetition and state the main methodological choices directly.
Questions for authors
- On held-out examples from the repaired BBQ and safety categories, how much improvement remains?
- What are the precise scale-solving algorithm, legal parameter space, shortlist construction, candidate count, and total GPU-hours? Can the patch sequences and per-example ledgers be released?
- How closely does the surcharge model predict measured latency across budget points and patch combinations?
- Beyond GPU memory usage and added model storage, what is the time overhead of the full repair procedure?