- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GAP is an evaluation protocol for comparing GGUF-quantized vision-language models by how close they stay to their BF16 reference. The BF16 model generates one response per image–question pair. Every quantized candidate is teacher-forced on that response, and per-item mean KL divergence is compared between two candidates with a paired t-test. Each comparison ends as a significant verdict or as explicitly unresolved. A pilot set is used to choose mode-specific token caps that reduce cost per item. The study covers 126 checkpoints (mostly near Q4), 8 models, 6 benchmarks and 15 model–mode configurations. For close pairs, pairing raises SNR by a median 3.7× in thinking mode and 2.0× in non-thinking mode, which cuts the items needed for 80% power by about 15.5× in thinking mode. Close non-thinking pairs still need about 11,000 items. Rankings can change between inference modes and when the reference responses come from a different model.
Critical strengths
The empirical scale is large and practically relevant: 126 community GGUF checkpoints, both inference modes, and released artifacts. Practitioners choosing between near-identical quants can use it.
Critical weaknesses
-
Methodological novelty is low. Teacher-forced KL divergence on on-policy reference responses is already standard (Nikolić et al. 2026; Helcig et al. 2026). Item-level paired testing is taken from Miller (2024). That pairing reduces variance when per-item divergences are positively correlated is a textbook identity (§3.3), so the 15.5× gain confirms an expected result rather than revealing a new one. The token-cap procedure, the remaining component, is a heuristic read off Fig. 1. The paper itself says its behavior is "unstable" for close pairs.
-
The protocol does not resolve the case it targets, and part of the procedure is circular. The target use case is close candidates during quantization development. Close non-thinking pairs need about 11k items, which is beyond the budget. Caps are assigned by pair category ("close" means at some cap), but that category is estimated from the same data, so a pair must be measured before its cap can be chosen. The paper also never connects KL gaps of 2.5–5% to downstream task differences. It acknowledges that KL divergence loses correlation with task scores in the near-lossless regime, so it is unclear whether resolving these gaps matters in practice.
Questions for authors
For resolved close pairs, does the KL-favored candidate also score higher (or flip fewer answers) on the benchmark tasks? Even a rank correlation on the main set would address significance.
7 comments
Downstream evaluation (W2 & Q1), Part 1/3: Summary and four-level comparison
Dear Reviewer oASM,
To directly address W2 and Q1 on downstream task relevance, Part 1 answers Q1 and compares GAP KLD, top-1 agreement, answer flips, and 3-seed task accuracy; Part 2 analyzes benchmark sample-size limits; Part 3 reports new-seed replications and per-benchmark checks. All results here are being added to the next revision.
Summary
- On close pairs that GAP resolves, the KL-favored candidate does not consistently score higher on downstream benchmarks: excluding one tied pair in each count, it achieves higher accuracy in 21/49 pairs (43%) and changes fewer answers relative to BF16 in 28/49 pairs (57%).
- At 3 seeds, the benchmark itself resolves only 3 of the 67 close pairs (1 matching GAP, 1 opposite, 1 unresolved by GAP), because close-pair accuracy gaps (median 0.47 pp) sit well below the benchmark's single-pair detection floor (1.20–1.90 pp; two seeds of BF16 alone disagree on 6.8–18.4% of questions).
- Across all candidates of a model, lower GAP KLD tracks greedy top-1 agreement ( = 0.94–1.00) and goes with fewer same-seed answer flips against BF16 ( to ; 27/27 pairs agree whenever the flip-rate difference itself is Holm-significant). Once accuracy gaps reach pp (41 pairs, all in Gemma-4-E4B-it), the benchmark resolves 40, and all 40 agree with GAP.
- Consistent with Nikolic et al. (2026), KLD measures behavioral divergence rather than whether flips turn right-to-wrong or wrong-to-right. GAP serves as a fast fidelity filter (resolving 50/67 close pairs in minutes) rather than a close-pair accuracy predictor, complementing final benchmark evaluation.
Setup & Pairwise Verdicts
We ran four thinking-mode VLMs (adding GLM-4.6V-Flash to the three announced; 56 quantized checkpoints plus BF16) on 1,800 visual questions (300 per benchmark) up to 32,768 tokens across 3 seeds, graded by majority vote of three LLM judges from three vendors. Because judge disagreements exposed defective items (e.g., reference keys contradicting the image), we curated the pool under image-only rules into 1,657 scored questions with 127 grading notes; every Holm-corrected headline count shifts by at most two pairs before vs. after curation (44 vs. 45 benchmark-resolved pairs; 40/43 vs. 41/44 agreement with GAP). Pairs are tested on the 1,329–1,610 loop-free clean questions per model via item-level paired -tests with Holm correction at .
| Model | Pairs (close) | GAP resolves (close) | Bench. resolves (close) | Both resolve (close) | Close: higher acc.† | Close: fewer flips† |
|---|---|---|---|---|---|---|
| Qwen3.6-35B-A3B | 45 (5) | 45 (5) | 0 (0) | – | 3/5 | 4/5 |
| Gemma-4-E4B-it | 105 (23) | 97 (15) | 40 (0) | 40 same, 0 opp. (close: 0) | 6/15 | 7/14 |
| Muse-Glimmer-30B | 136 (23) | 131 (18) | 5 (3)* | 1 same, 3 opp. (close: 1 same, 1 opp.) | 7/18 | 11/18 |
| GLM-4.6V-Flash | 91 (16) | 87 (12) | 0 (0) | – | 5/11 | 6/12 |
| All 4 models | 377 (67) | 360 (50) | 45 (3)* | 41 same, 3 opp. (close: 1 same, 1 opp.) | 21/49 | 28/49 |
* Parentheses: close pairs (<5% pilot KLD gap); Muse's 3rd benchmark-resolved close pair is unresolved by GAP. † Point-estimate direction on GAP-resolved close pairs (omitting 1 exact tie each).
Candidate Metric Ranges & Rank Correlations with GAP KLD
Candidate spans across checkpoints (GAP KLD at ; Top-1 on main set; flips and accuracy on 3-seed clean set):
| Model | Cands | GAP KLD | Top-1 agr. | Answer flips vs. BF16 | 3-seed Acc. |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B | 10 | 0.0033–0.0145 | 96.7–98.5% | 6.7–7.7% | 86.6–87.2% |
| Gemma-4-E4B-it | 15 | 0.0177–0.1921 | 87.3–95.9% | 17.4–25.2% | 46.0–56.9% |
| Muse-Glimmer-30B | 17 | 0.0215–0.1317 | 88.7–95.5% | 6.9–8.6% | 79.2–80.9% |
| GLM-4.6V-Flash | 14 | 0.0107–0.0338 | 94.3–96.7% | 9.8–11.6% | 69.4–71.1% |
Kendall's with GAP KLD (positive = lower KLD aligns with higher Top-1, fewer flips, or higher accuracy):
| Model | Top-1 | Flips | Flips * | Acc. | Acc. * |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B | +1.00 | +0.38 | 0.16 | +0.67 | 0.007† |
| Gemma-4-E4B-it | +0.94 | +0.66 | 0.0006† | +0.48 | 0.013† |
| Muse-Glimmer-30B | +1.00 | +0.63 | 0.0004† | +0.10 | 0.59 |
| GLM-4.6V-Flash | +0.96 | +0.44 | 0.028 | +0.22 | 0.27 |
* : uncorrected two-sided -value testing (Top-1 in all 4 models). † Significant after Holm's correction over the 4 models.
Notes
- All 40 wide-gap agreements come from Gemma-4-E4B-it (38 involving three weaker checkpoints at 46.0–52.0% vs. 54.7–56.9%, where WikiText-2 KLD also agrees 40/40), and Qwen's with accuracy spans just 0.6 pp (0 pairs resolved).
- Within Gemma's 8-checkpoint Q4 cluster (54.7–56.0% accuracy), neither GAP (, ) nor WikiText-2 KLD (, ) correlates detectably with accuracy.
Best regards,
The Authors
Downstream evaluation (W2 & Q1), Part 2/3: Why 3-seed benchmarks cannot separate close pairs
Dear Reviewer oASM,
During quantization development, engineers modify calibration or bit-allocation recipes and need an evaluation with enough statistical power to tell genuine behavioral changes apart from sampling noise. Below we quantify the sample sizes that standard free-form VLM benchmarks would require to separate closely matched pairs (close pairs: candidate pairs with <5% relative KLD gap on the pilot set; 67 pairs across the four models, or 66 omitting 1 exact accuracy tie) on each model's clean items (the 1,329–1,610 curated questions per model where every checkpoint and seed finished without a repetition loop).
Why 3-Seed Benchmarks Do Not Separate Close Pairs (MDE & Sample Size)
- Two runs of the exact same BF16 reference model on different seeds already receive opposite correctness grades (BF16 seed-to-seed disagreement below) on 6.8–18.4% of clean items, which is why seed-to-seed sampling noise accounts for 71–99% of pairwise accuracy variance at 3 seeds.
- Consequently, every model's median close-pair accuracy gap (0.25–0.63 pp; 0.47 pp overall) sits well below what the 3-seed benchmark can detect at 80% power (Detectable gap (MDE): 1.20–1.90 pp). At observed gaps, resolving half of the 66 non-tied close pairs at 3 seeds would require ~ 17,800 clean items (~53,400 full free-form generations per candidate), whereas GAP resolves distributional fidelity on the median close pair in 413 clean items with a single teacher-forced pass per item.
- Using a 0.5 pp target—near the overall close-pair median of 0.47 pp, and conservative for Qwen (0.25 pp) and GLM (0.33 pp), which need even larger samples—detecting 0.5 pp at 3 seeds requires 7,669–23,140 clean items (6–14$\times$ our current sets, or ~ 23,000–69,400 full generations per candidate). Simply running more seeds on the existing 1,329–1,610 items does not solve this either: between-question variance caps the infinite-seed () detectable gap at 0.66–0.82 pp in three of the four models (never reaching 0.5 pp), and in Qwen3.6-35B-A3B reaching 0.5 pp still takes ~18 seeds (~24,200 full generations per candidate).
| Model | Clean items | BF16 seed-to-seed disagreement | Detectable gap (MDE)* | Close-pair median gap | Clean items needed for 0.5 pp |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B | 1,329 | 6.8–7.5% | 1.20 pp | 0.25 pp | 7,669 |
| Gemma-4-E4B-it | 1,610 | 16.3–18.4% | 1.90 pp | 0.52 pp | 23,140 |
| Muse-Glimmer-30B | 1,477 | 7.1–7.4% | 1.29 pp | 0.63 pp | 9,881 |
| GLM-4.6V-Flash | 1,557 | 8.7–9.7% | 1.52 pp | 0.33 pp | 14,477 |
* Detectable gap (MDE) is the single-pair accuracy difference detected with 80% power at two-sided over 3 seeds. After Holm multiplicity correction across all close pairs of a model, the 3-seed MDE rises to 1.45–2.56 pp. Accounting for loop and defect filtering, detecting 0.5 pp at 3 seeds would require collecting 10,387–25,871 raw benchmark questions.
Free-form VLM benchmarks therefore cannot separate close quantization candidates at practical iteration budgets. As Part 1 shows, GAP does not predict which close candidate scores higher in task accuracy; what it provides is a statistically resolved distributional fidelity signal (tracking top-1 agreement and answer-flip volume in minutes) during quantization iteration, prior to final pre-deployment benchmark checks.
Best regards,
The Authors
Dear Reviewer oASM,
Thank you for the precise and critical review. We agree with several of your points, and Revision 1 changes the paper accordingly; pointers below refer to Revision 1. The downstream evaluation you ask for is in progress (see the last section).
W1. Methodological novelty
We agree: teacher-forced KLD on on-policy reference responses (Nikolić et al., 2026; Helcig et al., 2026), item-level paired testing (Miller, 2024), and the variance identity behind pairing are established, and we do not claim them. You are also right that the original token-cap procedure was a heuristic read off Fig. 1. Revision 1 replaces that component and states what the measurements add:
KLD ± SEcomparison; on the full main set that test resolves 24/128 of them under Holm's correction, versus 104/128 with pairing (Sec. 5.5).W2. Circularity and the close non-thinking case
W2 (downstream relevance) and Q1
Revision 1 does not yet contain task-level evidence. What it adds is decision-level evidence on the main set (App. L, Table 28): the rank correlation you suggest, computed against greedy top-1 token agreement with the reference instead of task scores. Kendall's τ between the KLD and top-1 rankings of each model's candidates is 0.82–0.98 in the seven non-thinking configurations and 0.94–1.00 in seven of the eight thinking configurations; the exception is Kimi-VL-A3B-Thinking-2506 (0.61). At the level of close pairs, both metrics resolve 78 of the 128 thinking pairs and agree in direction on 70; all 8 disagreements are in that Kimi configuration.
Your question asks for the next step, and we are running it: free-form generation with quantized candidates of three models (Qwen3.6-35B-A3B, Gemma-4-E4B-it, Muse-Glimmer-30B). We run thinking mode only, where GAP resolves most close pairs, and a subset of the candidates that form close pairs. As Sec. 2 notes, KLD tracks how many answers change rather than in which direction, so answer flips against the reference are the natural test alongside task scores. For resolved close pairs we will report (i) whether the KL-favored candidate flips fewer answers and scores higher, and (ii) the rank correlation between KLD and both answer-flip rate and benchmark score across these candidates. We will post these results, whichever way they come out, before November 16 (11:59pm Anywhere on Earth).
Thank you again for your incisive comments; they showed us exactly where the paper needed to improve.
Best regards,
The Authors