acceptodds
Under review as a conference paper at ICLR 2027

QGATE: Counterexample-Guided Precision Patching of Quantized LLMs with Fail-Closed Slice Reporting

Abstract

Quantized large language models can retain aggregate quality while changing their answers on specific slices of behavior, such as bias categories, low-resource languages, and safety prompts. Post-training quantization optimizes aggregate loss and does not localize these changes; existing repair methods add runtime modules, train weights, or rely on solvers that do not scale to 7B-parameter models, and none reports what it leaves unfixed. We propose QGATE, a training-free loop that treats every input on which a quantized model and its full-precision reference disagree (a flip) as a counterexample. QGATE targets the slice whose flip rate is largest relative to its tolerance, shortlists weight regions by the activation traffic of its flipped inputs, and applies patches native to the serving format: raising the bit-width of weight groups, re-solving stored quantization scales, and correcting biases where present. A patch is accepted only when a full discovery sweep, charging it for the flips it creates, confirms the improvement within a byte budget and a latency cap. Once the patched model is frozen, QGATE reports all 23 slices of its manifest: sampled slices carry simultaneous Clopper-Pearson upper bounds on sealed pools, exhaustively evaluated slices carry exact rates, and any slice not shown to be within tolerance is declared uncertified (fail-closed). On three shipped Qwen2.5-7B and Llama-3.1-8B configurations in EXL2 and GGUF, QGATE cuts the flip rate of the initially worst tolerance-normalized slice by 37.2–60.3% within a 96 MB budget, changes perplexity by at most 0.35%, and leads all eleven matched-cost baselines on the primary one (37.2% vs. at most 23.0%). On Llama, re-solving scales alone adds no bytes yet cuts more than raising bit-widths alone with 80 MB (46.6% vs. 43.8%). It certifies every primary-configuration slice and declares 2 and 7 slices uncertified on the stressed ones instead of passing them.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training quantization, quantized LLM repair, counterexample-guided repair, mixed-precision patching, slice-level behavioral flips, fail-closed residual certification

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_cc8KOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

QGATE repairs disagreement between quantized LLMs and their full-precision references. It targets the worst tolerance-normalized behavioral slice, proposes native-format precision and scale patches, and accepts changes after a full discovery-ledger check. A final report combines exact counts on discovery/evaluation universes with simultaneous upper bounds on independent samples. Three deployments report 37.2-60.3% reductions on their initial worst slices with small storage and measured latency increases.

Critical strengths
  1. Slice-specific disagreement is a useful complement to perplexity and average accuracy. Explicitly reporting unresolved slices is a sensible deployment practice.
  2. The native-format patch design avoids adding inference modules. The scale-only result and the grammar/scoring ablations help explain the benefits of the proposed loop.
  3. The paper explicitly distinguishes full-census measurements from sampled bounds and discloses configurations that remain uncertified. This is more informative than reporting only the repaired worst slice.
Critical weaknesses
  1. The evidence for generalizable repair is insufficient. Eighteen of the 23 slices, including the headline bias and safety targets, are optimized and evaluated on the same complete universes (Section 4.4; Appendix A.2). Exact census counts are valid descriptions of these sets, but do not establish improvement on unseen examples. The five independent regime-L slices are unchanged on cfg1 (Table 8), so the independent evaluation does not demonstrate the claimed repair benefit. The practical question is whether targeting these flips repairs a behavior beyond the examples used during search.
  2. Behavioral agreement is not safety or fairness. The objective counts beneficial and harmful departures from the reference identically. Moreover, the safety probe asks the model to choose between comply/refuse/partial labels (Appendix A.6), rather than measuring its actual generated responses. The paper needs directional error analysis and held-out behavioral tests before the bias/safety motivation is convincingly supported.
  3. Key implementation details remain missing despite the long appendix. Equation 8 specifies a scale-fitting objective but not the solver, iterations, initialization, or constraints imposed by packed EXL2/GGUF scales. Channel-range enumeration, shortlist size, the actual surcharge entries, and measured search time are also absent. A fixed seed does not resolve these omissions. Counting accepted steps and sweep bounds is not an adequate account of candidate-scoring cost.
  4. The baseline evidence is difficult to interpret. Several comparisons are adapted '-style' implementations, including a diffusion-model calibration method ported to autoregressive models. They require explicit hyperparameters, fidelity checks, and actual compute accounting. Appendix A.5 also acknowledges that the four strongest competitors do not differ significantly at the allocated threshold. Point-estimate superiority is useful, but weaker than a convincing advantage over the nearest alternatives.
  5. The deployment and certification claims need tighter boundaries. The hard latency constraint applies to an additive surcharge model, not measured latency; three final measurements do not validate its accuracy across candidate combinations. The statistical theorem is a standard Clopper-Pearson/Bonferroni construction conditional on independent sealed samples, and offers no generalization guarantee for the 18 optimized universes or correctness/safety guarantee. The contribution is principally an empirical repair procedure, rather than a new verification result.

Minor Comments:

  1. The abstract contains too many numerical results and configuration-specific details. Retain only the central result and use the remaining space to explain the problem, method, and significance.
  2. Use leading zeros consistently in tables: write 0.xxx rather than .xxx.
  3. Export diagrams and plots as vector graphics rather than embedding rasterized screenshots. Figure 1 is currently a high-resolution raster, but its small labels remain difficult to read at publication size. A vector export with larger labels and a simpler layout would improve readability; increasing image resolution alone would not address the problem.
  4. The repeated challenge-by-challenge narration and protocol qualifications make a relatively simple loop harder to follow. Reduce repetition and state the main methodological choices directly.
Questions for authors
  1. On held-out examples from the repaired BBQ and safety categories, how much improvement remains?
  2. What are the precise scale-solving algorithm, legal parameter space, shortlist construction, candidate count, and total GPU-hours? Can the patch sequences and per-example ledgers be released?
  3. How closely does the surcharge model predict measured latency across budget points and patch combinations?
  4. Beyond GPU memory usage and added model storage, what is the time overhead of the full repair procedure?
Reviewer_iiE5OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

using the wrong template.

Critical strengths

NA

Critical weaknesses

using the wrong template.

Questions for authors

NA

Reviewer_sQpaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

QGATE is a training-free framework for patching quantized LLMs to reduce disagreement with a full-precision reference on predefined behavioral slices. It prioritizes tolerance-normalized slice violations and applies backend-native precision, scale, and bias patches under storage and modeled latency constraints. The final report combines held-out statistical bounds for sampled slices with exact in-sample rates for exhaustively evaluated slices.

Critical strengths
  1. The framework addresses a practical limitation of aggregate quantization metrics by targeting slice-level behavioral disagreement and accounting for collateral flips.
Critical weaknesses
  1. The authors should further clarify how AI was used to distinguish AI assistance from automated research. Some visual elements in the figures do not match their intended meanings, suggesting that the figures may have been generated directly by AI. The scope of AI use therefore requires further clarification. These mismatches also make the figures difficult to interpret.
  2. Some terminology, such as “slice” and “flip/damage,” is confusing and further reduces readability.
  3. The range of models evaluated is limited.
  4. Reduced disagreement with the full-precision model does not necessarily imply improved correctness, fairness, or safety.
Questions for authors

In addition to the writing and figure-related issues mentioned above, there is a notable template issue: the page headers state “ICLR 2026.” Taken together, these concerns lead me to lean toward rejection at this stage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.