acceptodds
Under review as a conference paper at ICLR 2027

GAP: Generation-Aligned Paired Fidelity Evaluation for GGUF-Quantized VLMs

Abstract

Reliably distinguishing small fidelity differences between quantized candidates is challenging during quantization development. We introduce GAP, a Generation-Aligned Paired fidelity evaluation protocol for GGUF-quantized vision-language models. GAP generates image-conditioned responses with the full-precision reference, teacher-forces all candidates on these shared responses, and applies item-level paired tests to their fidelity divergence (e.g., KLD) from the reference to detect small fidelity differences with quantified uncertainty. A pilot-based procedure selects mode-specific token caps to reduce per-item evaluation cost, enabling larger samples and greater statistical power within a fixed budget. Across 126 quantized checkpoints near the Q4 regime, six benchmarks, and 15 model-mode configurations, the median paired-to-unpaired SNR ratio is 3.7 in thinking mode and 2.0 in non-thinking mode for candidate pairs with relative fidelity gaps below 5%; pairing reduces the estimated median sample size per candidate required for 80% power by approximately 15.5× relative to an independent-item unpaired design for such pairs in thinking mode, whereas close non-thinking pairs still need an estimated 11,000 items, beyond our budget. Fidelity rankings can also differ between thinking and non-thinking modes, while responses generated by a different reference model can change verdicts, including statistically significant reversals. These findings support evaluating candidates in their intended inference mode using responses from their corresponding reference model. We release the associated artifacts on Hugging Face Hub: https://huggingface.co/datasets/Anonymous778862/anony778862cr.

Primary area:
datasets and benchmarks
Keywords:
large language models, evaluation, benchmark, quantization, vision language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oASMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GAP is an evaluation protocol for comparing GGUF-quantized vision-language models by how close they stay to their BF16 reference. The BF16 model generates one response per image–question pair. Every quantized candidate is teacher-forced on that response, and per-item mean KL divergence is compared between two candidates with a paired t-test. Each comparison ends as a significant verdict or as explicitly unresolved. A pilot set is used to choose mode-specific token caps that reduce cost per item. The study covers 126 checkpoints (mostly near Q4), 8 models, 6 benchmarks and 15 model–mode configurations. For close pairs, pairing raises SNR by a median 3.7× in thinking mode and 2.0× in non-thinking mode, which cuts the items needed for 80% power by about 15.5× in thinking mode. Close non-thinking pairs still need about 11,000 items. Rankings can change between inference modes and when the reference responses come from a different model.

Critical strengths

The empirical scale is large and practically relevant: 126 community GGUF checkpoints, both inference modes, and released artifacts. Practitioners choosing between near-identical quants can use it.

Critical weaknesses
  1. Methodological novelty is low. Teacher-forced KL divergence on on-policy reference responses is already standard (Nikolić et al. 2026; Helcig et al. 2026). Item-level paired testing is taken from Miller (2024). That pairing reduces variance when per-item divergences are positively correlated is a textbook identity (§3.3), so the 15.5× gain confirms an expected result rather than revealing a new one. The token-cap procedure, the remaining component, is a heuristic read off Fig. 1. The paper itself says its behavior is "unstable" for close pairs.

  2. The protocol does not resolve the case it targets, and part of the procedure is circular. The target use case is close candidates during quantization development. Close non-thinking pairs need about 11k items, which is beyond the budget. Caps are assigned by pair category ("close" means ∣ρ^∣<5|\hat\rho|<5% at some cap), but that category is estimated from the same data, so a pair must be measured before its cap can be chosen. The paper also never connects KL gaps of 2.5–5% to downstream task differences. It acknowledges that KL divergence loses correlation with task scores in the near-lossless regime, so it is unclear whether resolving these gaps matters in practice.

Questions for authors

For resolved close pairs, does the KL-favored candidate also score higher (or flip fewer answers) on the benchmark tasks? Even a rank correlation on the main set would address significance.

7 comments

Dear Reviewer oASM,

Thank you for the precise and critical review. We agree with several of your points, and Revision 1 changes the paper accordingly; pointers below refer to Revision 1. The downstream evaluation you ask for is in progress (see the last section).

W1. Methodological novelty

We agree: teacher-forced KLD on on-policy reference responses (Nikolić et al., 2026; Helcig et al., 2026), item-level paired testing (Miller, 2024), and the variance identity behind pairing are established, and we do not claim them. You are also right that the original token-cap procedure was a heuristic read off Fig. 1. Revision 1 replaces that component and states what the measurements add:

  • The cap is now an explicit, validated rule (Sec. 3.4): the smallest power-of-two cap above 64 tokens with group-mean paired-SNR recovery over 90% and at most 5% significance flips against the full horizon (12,000 / 32,768 tokens). The instability we noted occurs at short caps, which the rule excludes. Caps fixed on the pilot set are evaluated unchanged on held-out items: they recover 92.4% of close-pair SNR, re-applying the rule selects the same caps, and none of 5,522 resolved comparisons reverses sign (Sec. 5.1, Table 9). The caps also transfer across models (Table 6): no model reverses the sign of any resolved comparison, and per-model close-pair recovery is 78–98%. Recovery below the 90% selection target costs power, not direction, and the longest-response models can recalibrate a higher cap with the same rule.
  • What the pairing measurements add. Beyond the textbook identity, our finding is that the pairing gain matters more for close pairs than for non-close pairs. The median SNR gain is 3.68 for close versus 2.08 for non-close thinking pairs (1.98 versus 1.44 in non-thinking mode; Table 3). Non-close pairs need under 350 items at the median with or without pairing. Close thinking pairs need 376 items with pairing versus 2,956 for the unpaired test on the same items, which is the usual KLD ± SE comparison; on the full main set that test resolves 24/128 of them under Holm's correction, versus 104/128 with pairing (Sec. 5.5).

W2. Circularity and the close non-thinking case

  • Circularity. You are right about the original design. Pair categories and caps are now fixed on the 600-item pilot set and never re-estimated on the disjoint 2,400-item main set. More importantly, no category is needed to choose a cap: using 512 / 2,048 tokens for every pair meets all criteria in every pooled cohort and changes none of the 1,323 main-set verdicts (Sec. 5.1, App. C).
  • The target case. In thinking mode, the median close pair needs 376 items, and 104/128 are resolved. In non-thinking mode we agree the case is not solved at our budget: the out-of-sample estimate is 5,887 items (about 11,000 originally), and pairing raises resolved close pairs from 7 to 23 of 87. The cause is short responses (half under 129 tokens; App. F), and GAP reports these pairs as unresolved rather than ranking them.

W2 (downstream relevance) and Q1

Revision 1 does not yet contain task-level evidence. What it adds is decision-level evidence on the main set (App. L, Table 28): the rank correlation you suggest, computed against greedy top-1 token agreement with the reference instead of task scores. Kendall's τ between the KLD and top-1 rankings of each model's candidates is 0.82–0.98 in the seven non-thinking configurations and 0.94–1.00 in seven of the eight thinking configurations; the exception is Kimi-VL-A3B-Thinking-2506 (0.61). At the level of close pairs, both metrics resolve 78 of the 128 thinking pairs and agree in direction on 70; all 8 disagreements are in that Kimi configuration.

Your question asks for the next step, and we are running it: free-form generation with quantized candidates of three models (Qwen3.6-35B-A3B, Gemma-4-E4B-it, Muse-Glimmer-30B). We run thinking mode only, where GAP resolves most close pairs, and a subset of the candidates that form close pairs. As Sec. 2 notes, KLD tracks how many answers change rather than in which direction, so answer flips against the reference are the natural test alongside task scores. For resolved close pairs we will report (i) whether the KL-favored candidate flips fewer answers and scores higher, and (ii) the rank correlation between KLD and both answer-flip rate and benchmark score across these candidates. We will post these results, whichever way they come out, before November 16 (11:59pm Anywhere on Earth).


Thank you again for your incisive comments; they showed us exactly where the paper needed to improve.

Best regards,

The Authors

AuthorsOpenReview ↗

Downstream evaluation (W2 & Q1), Part 1/3: Summary and four-level comparison

Dear Reviewer oASM,

To directly address W2 and Q1 on downstream task relevance, Part 1 answers Q1 and compares GAP KLD, top-1 agreement, answer flips, and 3-seed task accuracy; Part 2 analyzes benchmark sample-size limits; Part 3 reports new-seed replications and per-benchmark checks. All results here are being added to the next revision.

Summary

  • On close pairs that GAP resolves, the KL-favored candidate does not consistently score higher on downstream benchmarks: excluding one tied pair in each count, it achieves higher accuracy in 21/49 pairs (43%) and changes fewer answers relative to BF16 in 28/49 pairs (57%).
  • At 3 seeds, the benchmark itself resolves only 3 of the 67 close pairs (1 matching GAP, 1 opposite, 1 unresolved by GAP), because close-pair accuracy gaps (median 0.47 pp) sit well below the benchmark's single-pair detection floor (1.20–1.90 pp; two seeds of BF16 alone disagree on 6.8–18.4% of questions).
  • Across all candidates of a model, lower GAP KLD tracks greedy top-1 agreement (τb\tau_b = 0.94–1.00) and goes with fewer same-seed answer flips against BF16 (τb=+0.38\tau_b = +0.38 to +0.66+0.66; 27/27 pairs agree whenever the flip-rate difference itself is Holm-significant). Once accuracy gaps reach ≥2\ge 2 pp (41 pairs, all in Gemma-4-E4B-it), the benchmark resolves 40, and all 40 agree with GAP.
  • Consistent with Nikolic et al. (2026), KLD measures behavioral divergence rather than whether flips turn right-to-wrong or wrong-to-right. GAP serves as a fast fidelity filter (resolving 50/67 close pairs in minutes) rather than a close-pair accuracy predictor, complementing final benchmark evaluation.

Setup & Pairwise Verdicts

We ran four thinking-mode VLMs (adding GLM-4.6V-Flash to the three announced; 56 quantized checkpoints plus BF16) on 1,800 visual questions (300 per benchmark) up to 32,768 tokens across 3 seeds, graded by majority vote of three LLM judges from three vendors. Because judge disagreements exposed defective items (e.g., reference keys contradicting the image), we curated the pool under image-only rules into 1,657 scored questions with 127 grading notes; every Holm-corrected headline count shifts by at most two pairs before vs. after curation (44 vs. 45 benchmark-resolved pairs; 40/43 vs. 41/44 agreement with GAP). Pairs are tested on the 1,329–1,610 loop-free clean questions per model via item-level paired tt-tests with Holm correction at T∗T^*.

ModelPairs (close)GAP resolves (close)Bench. resolves (close)Both resolve (close)Close: higher acc.†Close: fewer flips†
Qwen3.6-35B-A3B45 (5)45 (5)0 (0)–3/54/5
Gemma-4-E4B-it105 (23)97 (15)40 (0)40 same, 0 opp. (close: 0)6/157/14
Muse-Glimmer-30B136 (23)131 (18)5 (3)*1 same, 3 opp. (close: 1 same, 1 opp.)7/1811/18
GLM-4.6V-Flash91 (16)87 (12)0 (0)–5/116/12
All 4 models377 (67)360 (50)45 (3)*41 same, 3 opp. (close: 1 same, 1 opp.)21/4928/49

* Parentheses: close pairs (<5% pilot KLD gap); Muse's 3rd benchmark-resolved close pair is unresolved by GAP. † Point-estimate direction on GAP-resolved close pairs (omitting 1 exact tie each).

Candidate Metric Ranges & Rank Correlations with GAP KLD

Candidate spans across checkpoints (GAP KLD at T∗=2,048T^* = 2{,}048; Top-1 on main set; flips and accuracy on 3-seed clean set):

ModelCandsGAP KLDTop-1 agr.Answer flips vs. BF163-seed Acc.
Qwen3.6-35B-A3B100.0033–0.014596.7–98.5%6.7–7.7%86.6–87.2%
Gemma-4-E4B-it150.0177–0.192187.3–95.9%17.4–25.2%46.0–56.9%
Muse-Glimmer-30B170.0215–0.131788.7–95.5%6.9–8.6%79.2–80.9%
GLM-4.6V-Flash140.0107–0.033894.3–96.7%9.8–11.6%69.4–71.1%

Kendall's τb\tau_b with GAP KLD (positive = lower KLD aligns with higher Top-1, fewer flips, or higher accuracy):

ModelTop-1 τb\tau_bFlips τb\tau_bFlips pp*Acc. τb\tau_bAcc. pp*
Qwen3.6-35B-A3B+1.00+0.380.16+0.670.007†
Gemma-4-E4B-it+0.94+0.660.0006†+0.480.013†
Muse-Glimmer-30B+1.00+0.630.0004†+0.100.59
GLM-4.6V-Flash+0.96+0.440.028+0.220.27

* pp: uncorrected two-sided pp-value testing τb=0\tau_b = 0 (Top-1 p<10−5p < 10^{-5} in all 4 models). † Significant after Holm's correction over the 4 models.

Notes

  • All 40 wide-gap agreements come from Gemma-4-E4B-it (38 involving three weaker checkpoints at 46.0–52.0% vs. 54.7–56.9%, where WikiText-2 KLD also agrees 40/40), and Qwen's τb=+0.67\tau_b = +0.67 with accuracy spans just 0.6 pp (0 pairs resolved).
  • Within Gemma's 8-checkpoint Q4 cluster (54.7–56.0% accuracy), neither GAP (τb=−0.29\tau_b = -0.29, p=0.40p = 0.40) nor WikiText-2 KLD (τb=+0.21\tau_b = +0.21, p=0.55p = 0.55) correlates detectably with accuracy.


Best regards,

The Authors

AuthorsOpenReview ↗

Downstream evaluation (W2 & Q1), Part 2/3: Why 3-seed benchmarks cannot separate close pairs

Dear Reviewer oASM,

During quantization development, engineers modify calibration or bit-allocation recipes and need an evaluation with enough statistical power to tell genuine behavioral changes apart from sampling noise. Below we quantify the sample sizes that standard free-form VLM benchmarks would require to separate closely matched pairs (close pairs: candidate pairs with <5% relative KLD gap on the pilot set; 67 pairs across the four models, or 66 omitting 1 exact accuracy tie) on each model's clean items (the 1,329–1,610 curated questions per model where every checkpoint and seed finished without a repetition loop).

Why 3-Seed Benchmarks Do Not Separate Close Pairs (MDE & Sample Size)

  • Two runs of the exact same BF16 reference model on different seeds already receive opposite correctness grades (BF16 seed-to-seed disagreement below) on 6.8–18.4% of clean items, which is why seed-to-seed sampling noise accounts for 71–99% of pairwise accuracy variance at 3 seeds.
  • Consequently, every model's median close-pair accuracy gap (0.25–0.63 pp; 0.47 pp overall) sits well below what the 3-seed benchmark can detect at 80% power (Detectable gap (MDE): 1.20–1.90 pp). At observed gaps, resolving half of the 66 non-tied close pairs at 3 seeds would require ~ 17,800 clean items (~53,400 full free-form generations per candidate), whereas GAP resolves distributional fidelity on the median close pair in 413 clean items with a single teacher-forced pass per item.
  • Using a 0.5 pp target—near the overall close-pair median of 0.47 pp, and conservative for Qwen (0.25 pp) and GLM (0.33 pp), which need even larger samples—detecting 0.5 pp at 3 seeds requires 7,669–23,140 clean items (6–14$\times$ our current sets, or ~ 23,000–69,400 full generations per candidate). Simply running more seeds on the existing 1,329–1,610 items does not solve this either: between-question variance caps the infinite-seed (S→∞S \to \infty) detectable gap at 0.66–0.82 pp in three of the four models (never reaching 0.5 pp), and in Qwen3.6-35B-A3B reaching 0.5 pp still takes ~18 seeds (~24,200 full generations per candidate).
ModelClean itemsBF16 seed-to-seed disagreementDetectable gap (MDE)*Close-pair median gapClean items needed for 0.5 pp
Qwen3.6-35B-A3B1,3296.8–7.5%1.20 pp0.25 pp7,669
Gemma-4-E4B-it1,61016.3–18.4%1.90 pp0.52 pp23,140
Muse-Glimmer-30B1,4777.1–7.4%1.29 pp0.63 pp9,881
GLM-4.6V-Flash1,5578.7–9.7%1.52 pp0.33 pp14,477

* Detectable gap (MDE) is the single-pair accuracy difference detected with 80% power at two-sided α=0.05\alpha = 0.05 over 3 seeds. After Holm multiplicity correction across all close pairs of a model, the 3-seed MDE rises to 1.45–2.56 pp. Accounting for loop and defect filtering, detecting 0.5 pp at 3 seeds would require collecting 10,387–25,871 raw benchmark questions.

Free-form VLM benchmarks therefore cannot separate close quantization candidates at practical iteration budgets. As Part 1 shows, GAP does not predict which close candidate scores higher in task accuracy; what it provides is a statistically resolved distributional fidelity signal (tracking top-1 agreement and answer-flip volume in minutes) during quantization iteration, prior to final pre-deployment benchmark checks.


Best regards,

The Authors

Reviewer_C1ssOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GAP, a fidelity evaluation protocol that teacher-forces GGUF-quantized VLMs on shared reference-generated responses and applies item-level paired tests to their divergence from the reference model. It also uses pilot experiments to select mode-specific scoring token caps. Experiments across eight models, 126 quantized checkpoints, and six benchmarks show improved statistical efficiency from pairing, an estimated 15.5-fold reduction in median sample requirements for close thinking-mode pairs, and sensitivity of candidate rankings to inference mode and response source.

Critical strengths
  1. A useful and statistically principled evaluation target. GAP addresses the practical difficulty of distinguishing closely matched quantized candidates through shared contexts, item-level pairing, and explicit uncertainty estimates. Its distinction between unresolved comparisons and equivalence makes the protocol valuable for avoiding unsupported rankings during quantization development.

  2. Broad empirical coverage with consequential findings. The study covers diverse models, quantized checkpoints, and visual benchmarks, and demonstrates that fidelity rankings can reverse across inference modes or reference-response sources. These findings directly inform evaluation design and support measuring candidates under conditions representative of their intended deployment.

Critical weaknesses
  1. The incremental methodological contribution is insufficiently isolated. Reference-generated teacher forcing, divergence metrics, and paired statistical testing have existing foundations, while the experiments do not clearly establish the additional benefit of generation alignment over alternative shared-context designs. This limits the evidence that GAP offers a substantial methodological advance beyond integrating established techniques.

  2. The headline efficiency claim relies on projected rather than empirically validated power. Required sample sizes are estimated from observed moments under working distributional assumptions, and bootstrap intervals do not establish that future evaluations achieve the predicted power. Without direct power validation and measured computational costs, the claimed fixed-budget advantage remains less firmly supported than the observed variance reduction.

Questions for authors
  1. Clarification: Were the projected sample requirements checked against empirical rejection rates on independent data or repeated subsamples, particularly for close candidate pairs? Any existing validation would materially strengthen confidence in the headline sample-efficiency result.

  2. Clarification: What explicit decision rule maps the pilot results to the selected token caps, and what existing evidence shows that these choices preserve candidate ordering relative to longer scoring horizons? This matters because truncation changes the evaluation target as well as its computational cost.

4 comments

Dear Reviewer C1ss,

Thank you for the constructive review and the positive assessment. We respond to each point below; all pointers refer to Revision 1.

W1. Isolating the methodological contribution and the benefit of generation alignment

We agree that reference-generated teacher forcing, divergence metrics, and paired tests have existing foundations; we do not claim them. Our methodological contribution is paired-SNR token-cap (T∗T^*) calibration (Sec. 3.4): a rule that lowers scoring cost without reversing any resolved verdict (see Q2), which makes KLD evaluation affordable under each VLM's own deployment configuration, including long reasoning traces.

The benefit of generation alignment over other shared contexts is that candidates are scored under that configuration: the model's chat template and inference mode, on responses from its own reference model. Revision 1 shows that each part matters:

  • Chat template and inference mode (Sec. 5.4). For dual-mode models, 5 of 430 pairs resolved in both thinking and non-thinking modes favor opposite candidates. A fixed text corpus is scored without the chat template, returns one verdict per pair, and cannot reflect this.
  • Reference model (Sec. 5.3). On the same images and prompts, scoring on another model's responses changes 36/132 close cross-family verdicts (8 reversed), versus 14/825 non-close; even within a model family, 10/44 close verdicts change.
  • Fixed text corpus (Sec. 5.2, App. D). Text-only KLD on WikiText-2 and PG essays agrees with GAP at text gaps >10% but opposes it in 17–38% of both-resolved pairs below 5%, also with an item-level paired test on text windows and a shared vision projector.

For close candidates, these shared contexts are not interchangeable with the candidate's own reference responses. For widely separated candidates, text-only KLD remains an adequate fast screen.

W2 / Q1. Empirical power validation and computational cost

We added the validation you describe (Sec. 5.5, App. K, Table 27). Pair categories and caps are fixed on the pilot set, the close-thinking budget is projected from pilot data alone (n=381n = 381 items), and we draw 1,000 subsamples of that size without replacement from the held-out main set, so no normality assumption enters. Across the 128 close thinking pairs, the paired test resolves a median of 82.9% of subsamples, matching the 80% target, versus 0.0% for Welch's unpaired test on the same items and 11.2% on independent items. Resampling each pair at its own pilot-projected budget gives 80.5% (96 eligible pairs). One limitation remains: close non-thinking pairs are underpowered at our 2,400-item budget (26.4% resolved under Holm vs. 8.0% without pairing), and GAP reports them as unresolved.

For cost, Revision 1 reports token counts rather than time (Table 8): the close-pair caps score 31.0% and 32.3% of response tokens on the pilot set, and 32.3% and 33.3% on held-out items. We are currently measuring wall-clock cost on a dedicated machine, both for scoring at T∗T^* and for GAP against downstream VLM evaluation, and will post the results before November 16 (11:59pm Anywhere on Earth).

Q2. Decision rule for the token caps and preservation of ordering

The rule (Sec. 3.4, App. C): for each group of pilot comparisons (inference mode × close / non-close), T∗T^* is the smallest power-of-two cap above 64 tokens at which (i) the group-mean paired-SNR recovery relative to the full horizon (12,000 / 32,768 tokens) exceeds 90% and (ii) at most 5% of comparisons flip between resolved and unresolved under Holm's correction. This selects 512 / 2,048 tokens for close pairs and 256 / 1,024 for non-close pairs in non-thinking / thinking mode (Table 1).

Ordering is preserved. Among comparisons resolved at the longer horizon, none reverses the sign of its mean difference at T∗T^*: 0 of 5,614 on the pilot set, 0 of 5,522 on held-out items scored to the same full horizons, and 0 of 6,567 on the 2,400-item main set against its 1,024 / 4,096-token horizon (Table 9). On the held-out items, the unchanged caps recover 92.4% of close-pair SNR with 2.9% / 4.2% flips, and re-applying the rule there selects the same caps (Sec. 5.1); leave-one-model-out selection matches in 6/7 and 8/8 close-pair folds (Table 7).

You are right that truncation changes the target: with a cap, the test concerns fidelity over the scored prefix, and it remains valid for that target because the cap is fixed before scoring. What truncation costs is power, not direction. On held-out items, 13/69 non-thinking and 13/242 thinking close comparisons resolved at the full horizon become unresolved at the cap, and none is resolved in the opposite direction. Behavior varies by model: the two with the longest responses (Qwen3.5-4B, Qwen3.6-35B-A3B) recover 78–88% of close-pair SNR and benefit from recalibrating a higher cap with the same rule (App. C, Table 6).


We would be glad to clarify any remaining points.

Best regards,

The Authors

AuthorsOpenReview ↗

Cost analysis: GAP at T* vs 16384 / 32768

Dear Reviewer C1ss,

To directly address W2 and Q2 regarding measured computational costs, Part 1 quantifies the wall-clock savings from truncating GAP at T∗T^{\ast} (vs. full-length scoring), and Part 2 evaluates GAP's fixed-budget advantage relative to free-form generation. All results here will be included in the next revision.

Summary

  • Truncating at T∗T^{\ast} reduces GAP's scoring time by 5–39% (one-slot; up to 47% in parallel) in non-thinking mode and 15–68% in thinking mode at micro-batch 2,048.
  • These numbers are conservative: under the paper's micro-batch 512 setting (one-slot), savings are larger in every case (5–48% in non-thinking and 16–72% in thinking mode). Comparing 512 vs. 2,048 on the same items shows no pairwise sign reversals and only minor borderline significance flips (3.4% at T∗T^{\ast} and 2.5% at full length, within the paper's 5% cap tolerance).

Setup & Results

Timed alone on a 96 GB NVIDIA RTX PRO 6000 Blackwell GPU across all 8 models (42 non-thinking and 49 thinking quantized candidates, 200 shared items each), covering both reference and candidate scoring (model loading excluded; the GPU metric kernel differs from the paper's CPU kernel by less than 3×10−113 \times 10^{-11}). Cells show non-thinking / thinking mean seconds over NN candidates (micro-batch 2,048; T∗=T^{\ast} = 512 / 2,048 vs. full 16,384 / 32,768 tokens).

ModelNOne slot: fullT∗T^{\ast}SavedParallel: fullT∗T^{\ast}Saved
Gemma-4-31B-it6 / 6258 / 475233 / 32410% / 32%does not fit––
Gemma-4-E4B-it6 / 671 / 10053 / 8426% / 16%56 / 7935 / 6637% / 16%
GLM-4.6V-Flash5 / 5128 / 185121 / 1486% / 20%109 / 16399 / 1289% / 21%
InternVL3.5-30B-A3B6 / 6271 / 415258 / 3275% / 21%204 / 300195 / 2425% / 19%
Kimi-VL-A3B6 / 689 / 13977 / 10813% / 22%69 / 10855 / 8121% / 25%
Muse-Glimmer-30B– / 7– / 490– / 418– / 15%– / 423– / 360– / 15%
Qwen3.5-4B6 / 6126 / 38377 / 12439% / 68%109 / 32458 / 10347% / 68%
Qwen3.6-35B-A3B7 / 7339 / 873232 / 35431% / 59%does not fit––

Notes

  • Longer responses yield larger savings: thinking mode and verbose models (e.g., Qwen3.5-4B and Qwen3.6-35B-A3B, where T∗T^{\ast} covers under a quarter of response tokens) save the most (31–68%).
  • Image and prompt processing is not shortened by T∗T^{\ast} (taking 74–92% of one-slot time at T∗T^{\ast} in non-thinking and 48–75% in thinking mode), which explains why wall-clock savings are smaller than the share of skipped output tokens. Full micro-batch 512 tables and details will be included in the next revision.
AuthorsOpenReview ↗

Cost analysis: GAP at T* vs free-form generation with llama-server

Dear Reviewer C1ss,

Continuing W2, Part 2 evaluates GAP's fixed-budget advantage under measured wall-clock cost, comparing parallel GAP scoring at T∗T^{\ast} against concurrent candidate generation in llama-server (standard practice for downstream benchmark evaluation).

Summary

  • On one GPU, parallel GAP at T∗T^{\ast} takes 74–98% less time (3.8–42.7× faster) than candidate generation across all thinking-mode cases, and 42–94% less time (1.7–16.0× faster) in 5 of 6 non-thinking cases (with comparable time on InternVL3.5-30B-A3B in non-thinking mode).
  • Under a fixed wall-clock compute budget, GAP can therefore evaluate roughly 4–40× more items or candidates in thinking mode than free-form generation, compounding the statistical sample-efficiency gains in W2/Q1.

Setup & Results

Same GPU and 200 items/candidate as Part 1. llama-server runs plain decoding up to 16,384 / 32,768 tokens across 6–16 concurrent slots (chosen prior to timing based on per-slot GPU memory). Model loading and one-time BF16 reference generation (needed once to evaluate the full-precision baseline and shared across all quantized candidates) are excluded. Cells show non-thinking / thinking mean seconds. († Only one of seven candidates fits in parallel on 96 GB.)

ModelGenerationGAP (T∗T^{\ast})SavedSpeedup
Gemma-4-31B-it558 / 2792does not fit––
Gemma-4-E4B-it182 / 37235 / 6681% / 82%5.1× / 5.6×
GLM-4.6V-Flash169 / 67299 / 12842% / 81%1.7× / 5.2×
InternVL3.5-30B-A3B219 / 1101195 / 24211% / 78%1.1× / 4.6×
Kimi-VL-A3B721 / 104255 / 8192% / 92%13.1× / 12.8×
Muse-Glimmer-30B– / 1378– / 360– / 74%– / 3.8×
Qwen3.5-4B923 / 439658 / 10394% / 98%16.0× / 42.7×
Qwen3.6-35B-A3B879† / 3537†166† / 243†81%† / 93%†5.3׆ / 14.6׆

Notes

  • Teacher forcing avoids step-by-step decoding, making full-length parallel GAP 1.1–10.4× (non-thinking) and 3.3–13.6× (thinking) faster than generation; truncating at T∗T^{\ast} widens this to 1.1–16.0× and 3.8–42.7×.
  • On InternVL3.5-30B-A3B (non-thinking), GAP is comparable in speed to generation (15% slower to 42% faster across the six candidates; slower on 3/6, with the 11% mean reflecting two candidates with a repetition-looped answer).
  • Testing 1.5–2× more llama-server slots across 15 model-mode cases did not speed up generation in 10/15 cases (1.01–1.50× runtime; 5–33% faster in 5/15, within run-to-run variance). Enabling multi-token prediction (MTP) on the four supported models shortens generation by 6.1–29.5% (non-thinking) and 4.1–50.0% (thinking), yet parallel GAP at T∗T^{\ast} still saves 78–93% and 79–98% of time on the three with parallel GAP settings.

All detailed tables and ablations will be included in the next revision. Thank you again for your concrete advice and patience.

Best,

The Authors

AuthorsOpenReview ↗

Follow-up to the cost analysis: both sides at their best setting

Dear Reviewer C1ss,

Summary

After our comment of 17 Nov, we checked whether each side of the cost comparison ran at its best setting. We asked an AI agent (Claude Fable 5.2, high effort) to search the settings of llama-server generation and of GAP, and we re-measured every case in which it found a better one.

  • Generation was not at its best setting in 5 of 15 model–mode cases; there it becomes 9–26% faster. GAP becomes 7% faster for one model.
  • With both sides at their best setting, the comparison is unchanged in thinking mode (T; GAP takes 74–98% less time) and in 5 of 6 non-thinking (NT) cases (42–94%).
  • On InternVL3.5-30B-A3B (NT) GAP has no cost advantage: the saving falls from 11% to 3%, i.e. parity.

How it was measured

On 17 Nov the slot counts were chosen from GPU memory before timing, and we had tested more slots but not fewer. We had not searched GAP's setting. The search covered all 15 cases on the same GPU type. For generation it varied the number of slots, the context per slot, a shared context pool, and the server's host-memory prompt cache. For GAP it varied sequences per process, processes, micro-batch and context.

With several slots, the same seed does not reproduce the answers: two identical runs agreed on 8 of 8 items with one slot and on 0 of 8 with eight slots. Answer lengths, and so job times, change between runs. The search therefore used a fixed-length replay, in which each request decodes exactly as many tokens as in our original job. By a rule fixed before the first measurement, a setting counted as better if it was faster than ours by more than the larger of 5% and twice the variation between identical repeats. One case was decided by judgment: for Kimi-VL-A3B (NT) our 16-slot setting varied by 44% between repeats, so the rule could not decide; we adopted 8 slots because each of its three runs was faster than every 16-slot run.

For each better setting we re-ran all candidates on the 200 items with the original method. On the same machine we also re-ran our original setting as a control.

Results

Table A. Generation at our original setting, run again, against the better setting on the same machine (mean s per candidate, 200 items). NT: non-thinking, T: thinking.

ModelSettingCand.Control → betterChange
Gemma-4-31B (NT)6 → 6 slots, cache off6/6582 → 528 s−9%
Gemma-4-31B (T)6 → 12 slots, cache off2/62,894 → 2,579 s−11%
InternVL3.5 (NT)12 → 6 slots, cache off6/6253 → 186 s−26%
Kimi-VL (NT)16 → 8 slots4/6646 → 475 s−26%
Qwen3.6-35B (NT)12 → 6 slots3/7906 → 827 s−9%
InternVL3.5 (T)8 → 16 slots6/61,070 → 1,030 s−4%, not clear
Qwen3.6-35B (T)12 → 8 slots2/73,885 → 3,820 s−2%, not clear

Table B. Rows of the 17 Nov table that change when both sides use their best setting (all candidates, 200 items; posted → best).

ModelGen. sGAP s at T*SavedSpeedup
InternVL3.5 (NT)219 → 186195 → 18111% → 3%1.1× → 1.0×
InternVL3.5 (T)1,101 → 1,097242 → 22378% → 80%4.6× → 4.9×
Kimi-VL (NT)721 → 5255592% → 90%13.1× → 9.6×
Gemma-4-31B (NT / T)‡558 → 528 / 2,792 → 2,483233 / 32458% → 56% / 88% → 87%2.4× → 2.3× / 8.6× → 7.7×
Qwen3.6-35B (NT)‡815 → 79023272% → 71%3.5× → 3.4×

‡ Parallel GAP does not fit on 96 GB, so GAP runs with one slot. These rows were not in the 17 Nov table; its Qwen3.6-35B row (the one candidate that fits) stays at 81%. InternVL3.5 (T): 1,097 s is a re-measurement at 8 slots that replaces 1,101 s; only GAP's setting changes that row.

Limits

  • We did not search server threads, batch size


Best,

The Authors

Reviewer_SdEDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper's main contribution is a method named GAP, which is an evaluation protocol for deciding which of two closely performing quantized vision-language models better preserves the behavior of its full-precision reference.

Critical strengths
  • The paper addresses an important practical challenge in quantization development: reliably distinguishing candidates whose fidelity differences are too small to detect using conventional benchmark evaluations.

  • The statistical framework is formulated well. The paired confidence intervals, unresolved comparisons, and sample-size calculations aid the statistical aspects of the framework.

Critical weaknesses
  • The novelty of the paper seems limited. The key elements of GAP seems to have been built on top of / borrowing ideas of establishes approaches like paired statistical inference and teacher force divergence measurement. Hence its unclear the boundary between its methodological contribution from its empirical and engineering contributions.

  • The reported 15.5× sample-size reduction is derived from plug-in statistical power calculations rather than direct repeated-sampling validation. These estimates depend on measured effect sizes, variance estimates, and assumptions about the distribution of paired differences. Hence the empirically validity of power improvement seems to be questionable.

  • In Section 5.4, pair categories and statistical power estimates are obtained using the same main evaluation data. This may influence the estimated efficiency of the method for pairs leading towards a bias (unless i am mistaken)

Questions for authors
  • What is the principal methodological novelty of GAP compared with existing reference-generated fidelity evaluation combined with standard paired statistical testing?

  • Could the authors empirically validate the predicted sample-size improvements through repeated subsampling or independent evaluation sets?

  • How sensitive are the results to defining close candidate pairs on the pilot set rather than using the same main-set data for categorization and power estimation?

5 comments
AuthorsOpenReview ↗

About W1/Q1 and W2/Q2

Dear Reviewer SdED,

Thank you for the careful review. Revision 1 addresses each of the points; all pointers below refer to Revision 1.

W1 / Q1. Methodological novelty and its boundary

We agree that teacher-forced divergence and item-level paired inference are established, and we do not claim them as novel. To make the boundary explicit:

  • Methodological: paired-SNR token-cap (T∗T^*) calibration (Sec. 3.4). Scoring full image-conditioned responses, especially long reasoning traces, is what makes paired evaluation of close VLM candidates expensive, and an arbitrary truncation can change verdicts. We define an explicit rule (the smallest power-of-two cap whose group-mean paired-SNR recovery exceeds 90% with at most 5% significance flips against a full-length horizon) and validate it out of sample: caps fixed on the pilot set recover 92.4% of full-horizon close-pair SNR on held-out items, and no resolved comparison reverses sign (Sec. 5.1, App. C).
  • Empirical: when the protocol is needed. Text-only KLD agrees with GAP at text gaps >10% but opposes it in 17–38% of both-resolved pairs below 5% (Sec. 5.2, App. D); scoring on another model's reference changes 36/132 close cross-family verdicts (Sec. 5.3); and resolved verdicts can reverse between thinking and non-thinking modes (Sec. 5.4).
  • Engineering: our customized llama.cpp scorer and the released per-token KLD archives for 124 checkpoints.

To our knowledge, none of the prior reference-generated fidelity studies (Sec. 2) selects the number of scored tokens by a validated rule or tests when a text-only proxy suffices.

W2 / Q2. Empirical validation by repeated subsampling

We added exactly this (Sec. 5.5, App. K, Table 27). With pair categories and T∗T^* fixed on the pilot set, we project the close-thinking budget from pilot data alone (n=381n = 381 items) and draw 1,000 subsamples of that size without replacement from the held-out main set, so no normality assumption enters. Across the 128 close thinking pairs, the paired test resolves a median of 82.9% of subsamples, matching the 80% target, versus 0.0% for Welch's unpaired test on the same items and 11.2% on independent items. A second check that resamples each pair at its own pilot-projected budget gives 80.5% (96 eligible pairs). On the full main set, pairing resolves 104/128 close thinking pairs under Holm's correction versus 24/128 for the same-item unpaired test.

The plug-in ratios are now reported against both baselines, with bootstrap intervals (Table 3): 7.9× fewer items than an unpaired test on the same items and 14.5× fewer than on independent items (15.5× in the original submission).

One limitation remains: close non-thinking pairs are underpowered at our 2,400-item budget (26.4% resolved under Holm vs. 8.0% without pairing), reflecting short responses rather than a failure of pairing (App. F). GAP reports such pairs as unresolved rather than ranking them.

AuthorsOpenReview ↗

About W3/Q3

W3 / Q3. Categorization and power estimated on the same data

You are right. In the original Sec. 5.4 (now Sec. 5.5), pairs were categorized and their sample sizes estimated on the same main set. In Revision 1, pair categories and T∗T^* are fixed on the 600-item pilot set and never re-estimated on the disjoint 2,400-item main set, where all power quantities are computed (Secs. 3.3, 4, 5.5).

To answer the sensitivity question directly, we recomputed Table 3 on the same Revision 1 data with main-set instead of pilot categories. In thinking mode nothing changes: 129 vs. 128 close pairs (124 shared), a median paired N^80\hat{N}_{80} of 376 and an SNR gain of 3.68 under both, and identical 7.9× / 14.5× reductions. (The original Table 1 value of 353 predates the additional repetition screening of the main set described in App. H.) In non-thinking mode the estimate is sensitive: 81 vs. 87 close pairs (62 shared), with a median N^80\hat{N}_{80} of 10,949 under main-set categories versus 5,887 under pilot categories; the 25 pairs that are close only on the pilot set are moderate or wide on the main set (median N^80\hat{N}_{80} 1,754). Under either definition, the median close non-thinking pair needs more items than our 2,400-item budget, so that conclusion does not depend on the choice.

Table 3 recomputed on the same Revision 1 data and main-set items, with pair categories taken from the main set versus the pilot set. Brackets are 95% bootstrap intervals; >10N marks medians beyond the pair-specific search limit, as in Table 3.

ModeCategoryCategories fromPairsgabg_{ab}N^80\hat{N}_{80} pairedN^80\hat{N}_{80} unp. (same)N^80\hat{N}_{80} unp. (indep.)
Non-thinkingclosemain set812.0010,949 [5,096, 13,059]>10N [17,214, >10N]>10N [>10N, >10N]
Non-thinkingclosepilot (Table 3)871.985,887 [3,310, 7,828]19,009 [10,400, >10N]>10N [14,266, >10N]
Non-thinkingnon-closemain set5251.43159 [132, 177]285 [243, 318]340 [298, 386]
Non-thinkingnon-closepilot (Table 3)5191.44159 [130, 174]280 [239, 313]335 [293, 380]
Thinkingclosemain set1293.68376 [301, 455]2,956 [2,779, 3,524]5,435 [4,772, 6,039]
Thinkingclosepilot (Table 3)1283.68376 [301, 455]2,956 [2,779, 3,524]5,435 [4,772, 6,039]
Thinkingnon-closemain set5882.077 [6, 7]15 [14, 17]20 [18, 22]
Thinkingnon-closepilot (Table 3)5892.087 [6, 7]16 [14, 17]20 [18, 22]


We would be glad to clarify any remaining points. Thank you.


Best regards,

The Authors

AuthorsOpenReview ↗

Update on W2/Q2: sample-size reduction validated by direct resampling

Dear Reviewer SdED,


A brief update on W2/Q2 with two results completed since our first reply.


The reduction factors, without the power formula. Our earlier check validated the paired sample size (n=381n = 381) by subsampling, but the reduction factors themselves (7.9× / 14.5×) were still plug-in estimates: their bootstrap intervals describe sampling uncertainty and do not remove the power formula. We have now estimated N80N_{80} of all three tests by resampling main-set questions (1,000 replicates per pair; 128 close thinking pairs). The median paired N80N_{80} is 342 (plug-in: 376), and the unpaired tests need 8.7× and 15.8× as many items (plug-in: 7.9× and 14.5×).


Why the values differ, and which one the paper reports. The formula assumes normally distributed item-level differences. In a minority of pairs a few items carry most of the variance, and for these the formula overstates the paired sample size; the median therefore moves from 376 to 342. We keep the plug-in values (7.9× / 14.5×) as the reported factors because they are the more conservative ones, and add the resampled values beside them. Two limits: for a single pair the two estimates can differ (ratio 0.41–1.16 under the paired test), and the factor varies across models (5.6× to 21.7×).


Independent reference responses. This check comes from our downstream evaluation (reply to Reviewer oASM), where the reference model answered every benchmark question with three sampling seeds. We scored all candidates of its four thinking-mode models (Qwen3.6-35B-A3B, Gemma-4-E4B-it, Muse-Glimmer-30B, GLM-4.6V-Flash; 377 pairs) on each of the three sets of reference responses: 374 pairs receive the same verdict on all three sets, and no pair is resolved in opposite directions.


Both results will be in the next revision. The downstream-benchmark and wall-clock results are in our replies to Reviewers oASM and C1ss.


Best regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response and Summary of Revisions (Revision 1)

Dear Area Chair and Reviewers,

Thank you for your constructive feedback. We uploaded Revision 1 of GAP and are updating the artifact archive. To improve readability, we rewrote Sec. 1, formalized the T∗T^* rule (Sec. 3.4, App. C), and streamlined Sec. 5.5 (moving length details to App. F). We list section pointers below to answer some common questions.

Methodological novelty and positioning (Secs. 1–2) [oASM W1, C1ss W1, SdED W1/Q1] We agree that teacher-forced KLD and item-level paired tests are established components; our contribution is making them reliable at scale for VLMs. To our knowledge, prior quantization studies used them diagnostically on small setups. Resolving close VLM quantizations requires hundreds to thousands of multimodal items, often with long reasoning traces. GAP makes such evaluation feasible with an out-of-sample-validated protocol that introduces paired-SNR token-cap (T∗T^*) calibration to bound per-item cost without sign reversals, and defines when GAP is needed over text-only KLD.

1. When GAP is needed and why own-reference alignment matters (Secs. 5.2–5.3, App. D) [C1ss W1, oASM W1, SdED W1/Q1] Text-only KLD (llama-perplexity and a paired-window variant, on WikiText-2 and PG essays) agrees with GAP at wide gaps (>10%), but opposes GAP in 17–38% of both-resolved pairs below a 5% text gap—even with a shared vision projector (Sec. 5.2, App. D). On the same images and prompts (Sec. 5.3), scoring on a foreign rather than the model's own VLM reference also changes 36/132 close cross-family verdicts (8 reversals). Text-only KLD thus serves as a fast screen (>10% gap), while close VLM candidates require own-reference GAP.

2. Explicit token-cap rule validated on full-length responses (Secs. 3.4, 5.1, App. C) [oASM W1, C1ss Q2] We defined an explicit rule (>90% paired-SNR recovery, ≤5% Holm verdict flips) against pilot horizons of 12,000 / 32,768 tokens (≥99.0% complete responses pooled; ≥95.8% per model, Table 21). Extending 100 disjoint main-set items per benchmark to the same horizons selects the identical caps with zero sign reversals on resolved pairs, and the full main set is scored to 1,024 / 4,096 tokens for 2× headroom checks.

3. Out-of-sample calibration and updated N^80\hat{N}_{80} estimates (Secs. 3.3–3.5, 4, 5.1, 5.5) [oASM W2, SdED W3/Q3] Pair categories and $T^* $ are fixed on the 600-item pilot set (also used for Sec. 5.3/App. D controls) and never re-estimated on the disjoint 2,400-item main set used in Secs. 5.1, 5.2, 5.5, and App. L (combined only in a Sec. 5.4 check). Out-of-sample categorization removes in-sample small-gap selection bias on the main set, updating close non-thinking median paired N^80\hat{N}_{80} from 10,949 (original submission, Table 1) to 5,887 (Revision 1, Table 3), while close thinking N^80\hat{N}_{80} stays stable (353 → 376; same tables). In practice, default caps (T∗T^* = 512 / 2,048) pass all criteria in every pooled cohort without a pilot split or pre-categorization.

4. Pairing gains vs. shared-item baselines and empirical power (Sec. 5.5, App. K) [oASM W1/W2, C1ss W2/Q1, SdED W2/Q2] Alongside independent-item unpaired tests (14.5× median budget reduction on close thinking pairs; 15.5× originally), we added Welch's test on the same items—matching the KLD ± SE comparison on shared prompts—showing a 7.9× reduction from pairing alone. Held-out subsampling (1,000 draws at pilot-projected n=381n = 381) validates power without normality assumptions: on close thinking pairs, the paired test resolves a median 82.9% of draws vs. 0.0% for same-item Welch. Underpowered pairs (64/87 close non-thinking) remain unresolved rather than given noisy rankings.

5. Linking KLD to reference token decisions (New App. L) [oASM W2/Q1] To verify behavioral consistency with the reference, we added greedy top-1 token agreement in App. L: candidates with lower GAP KLD achieve higher top-1 agreement (same rank-1 candidate in 15/15 setups; zero opposite Holm verdicts in 14/15). While not replacing free-form evaluation, this provides complementary evidence that lower GAP KLD reflects closer token decisions.

6. Multiplicity control and schedule-consistency audit (Secs. 3.3, 4, 5.3, Supplementary) Uniform Holm correction (incl. Sec. 5.3) leaves all qualitative conclusions unchanged. Auditing cross-schedule scoring found 0 / 11,529 same-layout prefix discrepancies across all 124 retained checkpoints, while two Gemma-4 UD-Q3_K_XL checkpoints showed schedule-dependent KLD shifts from token 0; we excluded both (126 → 124 checkpoints, 1,381 → 1,323 pairs; 0 close pairs affected; see why-do-we-remove-UD-Q3_K_XL-from-Gemma-4.md).


Update (Nov 16 AoE): the planned results are posted.

  • Downstream evaluation [oASM W2/Q1]: Parts 1–3 in Reviewer oASM's thread.
  • Measured compute cost [C1ss W2]: two comments in Reviewer C1ss's thread.
  • Resampled sample-size validation [SdED W2/Q2]: update in Reviewer SdED's thread.
AuthorsOpenReview ↗

General Response and Summary of Revisions (Revision 2)

Dear Area Chair and Reviewers,

Thank you sincerely for your detailed reviews, patience, and constructive discussions. We have uploaded Revision 2, incorporating all promised experiments and a full readability pass. Due to file volume, the updated code and artifacts at https://huggingface.co/datasets/Anonymous778862/GAP-revision-2 are uploading and will finish by November 24 (AoE).

Protocol simplification: close / non-close and one T∗T^* per mode (Secs. 3.3–3.4, 5.1; Apps. A, J).

Replacing Revision 1's four-tier split (tight, narrow, moderate, wide), Revision 2 groups pilot pairs into close (<5%<5\% relative KLD gap) and non-close (≥5%\ge 5\%). We especially thank Reviewer C1ss for prompting GPU wall-clock measurements, which directly guided us to simplify GAP: because image-and-prompt prefill takes 74–92% (non-thinking) and 48–75% (thinking) of one-slot T∗T^* time (App. F, Table 24), scoring at 512/2,048512 / 2{,}048 takes nearly the same time as 256/1,024256 / 1{,}024 (+1.6–6.5% and +8.2–29.1%; Sec. 5.1; App. A). With score reuse and 87.8% of candidate–mode evaluations in a close pair, one cap T∗=512/2,048T^* = 512 / 2{,}048 per mode suffices without prior categorization (1,323/1,323 main-set verdicts unchanged).

1. Downstream benchmarks and multi-seed highlight [oASM W2/Q1, SdED W2/Q2]

All three parts from Reviewer oASM's thread (56 thinking candidates of four VLMs, 377 pairs / 67 close, three seeds) are incorporated:

  • Parts 1 & 3 (flips, accuracy correlations, curation, replications): Comparisons against greedy top-1 agreement (App. I, Table 40), same-seed correctness flips, three-seed task accuracy (including loss of accuracy correlation on close Q4 pairs; Nikolic et al., 2026), and text KLD appear in Sec. 5.5 (Table 5) and App. G.1–G.2, G.6–G.7 (Tables 27–29, 34), with three-layer curation in App. H (Tables 37–39). New-seed replications (Gemma-4-E4B-it, Muse-Glimmer-30B, plus 6–9 seed checks) and per-benchmark / QAT-vs.-PTQ checks appear in App. G.3, G.5 (Tables 30, 32–33).

  • Part 2 & new multi-seed highlight (seed sensitivity vs. GAP stability): Downstream benchmarks are sensitive to random decoding seeds: two BF16 seeds disagree on 6.8–18.4% of clean questions, swinging single-seed rankings (τb=−0.58\tau_b = -0.58 to +0.61+0.61) and raising three-seed MDEs (1.20–1.90 pp) above median close-pair gaps (0.25–0.63 pp; 3/67 close pairs resolved vs. GAP's 50/67; Sec. 5.5; App. G.4, Table 31). By contrast, GAP is largely unaffected by seed-to-seed BF16 reference variation: across three independent BF16 seeds, single-seed rankings reach τb=+0.94\tau_b = +0.94 to +1.00+1.00 and verdicts match across all three seeds on 374/377 pairs (64/67 close) with zero reversals (Secs. 1, 5.5, 6; App. G.6, Tables 35–36). Variance decomposition shows that one BF16 reference response per question suffices, as spending 3×3\times compute on 3×3\times more questions lowers close-pair SE by 29% over averaging three seeds (Table 36).

2. Measured GPU wall-clock cost and setting search [C1ss W2/Q2]

All three posts in Reviewer C1ss's thread appear in Secs. 5.1, 5.4 (Table 4) and Apps. A, F (Tables 19–26): (i) T∗T^* vs. 16,384 / 32,768 savings, prefill share, GPU-kernel check, and micro-batch 512 vs. 2,048 tables (Tables 22–24, 26); (ii) GAP at T∗T^* vs. multi-slot llama-server decoding (74–98% saved in thinking mode, or 2.1–11.3 min/candidate at N^80=376\widehat{N}_{80}=376, plus MTP and slot-sweep ablations; Tables 4, 25); and (iii) the both-sides setting search with fixed-length replay and same-machine controls (App. F, Table 21).

3. Bootstrap-resampled N^80\widehat{N}_{80} and categorization sensitivity [SdED W2/Q2, W3/Q3; C1ss W2/Q1]

We thank Reviewer SdED for pushing us to validate these assumptions directly. Bootstrap-resampled N^80\widehat{N}_{80} (342 vs. 376 plug-in on close thinking pairs; 8.7×8.7\times vs. 7.9×7.9\times over same-item Welch's test), per-model breakdowns, three-seed checks, and held-out subsampling appear in Sec. 5.4 (Table 3) and App. D (Tables 15–17); split stability and main-set vs. pilot categorization sensitivity appear in App. D (Tables 13–14) and App. J.

4. Framing and readability pass [SdED W1/Q1; all reviewers]

Following Reviewer SdED (W1/Q1) and all reviewers' presentation feedback, we refined Secs. 1 and 3 to frame GAP as a practically calibrated VLM fidelity protocol, rewrote prose and captions in plain English, and added an Appendix Contents roadmap (p. 16).

We are deeply grateful to the Area Chair and Reviewers oASM, C1ss, and SdED for helping make this work much stronger.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.