- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper unifies contrastive decoding for vision-language models (VLMs) under the classifier-free guidance (CFG) framework, showing that existing methods can be analyzed in terms of their reference distributions, guidance strength, and guidance duration. Through systematic experiments, the authors identify a key limitation of teacher-forced references: conditioning the image-free branch on generated answer prefixes leaks visual information and weakens the guidance signal. They propose SWITCH, which initially uses a fixed image-free reference and switches to a teacher-forced reference after the first tokens. Combined with image deletion as the negative reference and a fixed guidance scale of , the resulting VLM-CFG method requires no additional training. Evaluation on ten benchmarks across five models shows improvements of 5.69 and 1.72 percentage points over unguided decoding on Qwen2.5-VL-7B and Qwen3-VL-8B, respectively, with a reported inference cost of 1.04×. However, gains are limited on InternVL3.5-8B, indicating that the method's effectiveness may depend on the underlying model.
Critical strengths
-
Unified theoretical framework. The paper formulates contrastive decoding as classifier-free guidance, providing a principled framework for understanding and comparing existing methods.
-
Insightful analysis and simple design. The paper identifies answer-prefix leakage as a key limitation of teacher-forced references and proposes SWITCH to balance guidance strength and reference freshness.
-
Empirical effectiveness and efficiency. Experiments across multiple models and ten benchmarks demonstrate consistent gains on several backbones, with only 1.04× inference cost for the proposed method.
Critical weaknesses
-
The mechanistic interpretation of the guidance signal remains incomplete. The paper acknowledges that the image-free reference is not necessarily the true marginal distribution, so the guidance signal contains both image-related information and reference-distribution bias. Although the prefix-swap experiments support the proposed explanation of guidance decay, the relative contributions of these effects to the observed performance gains remain unclear.
-
The generalizability of the proposed configuration is not fully established. SWITCH relies on a fixed switching threshold () and guidance scale (). Although the paper provides partial sensitivity analyses, their coverage across model families and task distributions remains limited. The negligible overall gain on InternVL3.5-8B further suggests that the effectiveness of the proposed configuration is model-dependent.
-
The efficiency advantage is strongly dependent on answer length. The reported 1.04× inference cost primarily reflects the short-answer evaluation setting, whereas the appendix estimates 1.62–1.68× overhead on longer-answer benchmarks. This substantially qualifies the efficiency claim and leaves the practical trade-off less clear for long-form generation.
Questions for authors
-
Baseline tuning and comparison fairness. Could the authors clarify how the hyperparameters of the compared methods were selected for each backbone? Were the baselines evaluated under their original settings or tuned on validation data? Would the main conclusions hold under a comparable tuning protocol?
-
Practical inference efficiency. Could the authors provide measured end-to-end latency and throughput, in addition to forward-pass-based cost estimates, for representative short- and long-answer tasks? This would help clarify the practical overhead under realistic inference conditions.
-
Robustness to decoding settings. All evaluations use greedy decoding. Do the authors have preliminary results or a technical rationale for whether the proposed method would remain effective under temperature and top- sampling?
-
Sensitivity to prompt and stopping conventions. Since the method can affect both token selection and when generation terminates, how sensitive are the results to benchmark prompt templates and stopping criteria? Could the authors clarify whether these settings are kept identical across all compared methods?
8 comments
Response 1, Part 2
W2) & Q1) Fixed settings and baseline tuning
One setting for every model and benchmark. VLM-CFG has two settings, and neither is tuned per benchmark or per model. The switch point K = 8 was set from answer-length statistics before reading any score, and α = 1.5 is the standard guidance scale for text generation [2]. Both stay fixed across all 13 models and every benchmark, so VLM-CFG is a simple, generic change to the decoding loop with nothing to tune. The baselines are treated the same way: each runs at its published setting, the same on every backbone, and no method, ours included, is tuned on validation data.
Sweeping is not worth it. To test whether fixed settings favour us, Table R3 gives every method the same scale sweep as VLM-CFG, on two models from different families.
Table R3. Mean accuracy (%) over OK-VQA, VizWiz, PhD, TreeBench and V* as the guidance scale s grows. Each method applies s to its own contrast (VCD and SID: α_cd = s − 1; PAI: γ = s; VLM-CFG: α = s, without the gate). † Default setting. Bold: best of each row.
| Scale s | 1.25 | 1.5 | 2 | 3 | 4 |
|---|---|---|---|---|---|
| Qwen2.5-VL-7B (unguided 59.89) | |||||
| VCD | 61.16 | 61.42 | 61.36† | 60.56 | 60.64 |
| SID | 61.54 | 62.44 | 62.70† | 62.41 | 62.09 |
| PAI (first rung γ = 1.1) | 63.07† | 65.05 | 64.41 | 63.56 | 63.27 |
| VLM-CFG | 64.12 | 64.73† | 64.30 | 63.80 | 63.64 |
| Molmo2-4B (unguided 60.81) | |||||
| VCD | 61.16 | 60.95 | 60.00† | 59.59 | 58.86 |
| SID | 61.23 | 60.90 | 60.62† | 60.35 | 59.88 |
| PAI (first rung γ = 1.1) | 61.65† | 60.36 | 59.53 | 59.03 | 58.77 |
| VLM-CFG | 62.38 | 62.10† | 61.70 | 61.28 | 61.25 |
Three results stand out.
- Tuning the baselines does not change the conclusion. Even with each baseline at its best scale for each model, picked on these test results, VLM-CFG at its fixed α = 1.5 beats VCD, SID and M3ID on both models, and PAI on Molmo2-4B.
- VLM-CFG is the most robust. It is the only method above unguided decoding at every scale on both models, and it moves by about one point across the whole ladder, against up to three for PAI.
- α = 1.5 is the best single setting. Averaged over the two models, it beats every setting of every method, its own included (63.42, against 62.71 for the best baseline setting, PAI at γ = 1.5). Retuning α per model would gain at most 0.28 points.
The only setting above ours is PAI at γ = 1.5 on Qwen2.5-VL-7B, by 0.32. PAI combines contrastive decoding with attention manipulation. At γ = 1.5 its contrastive term runs at our scale, and the attention boost on top helps slightly on Qwen2.5-VL-7B but costs 1.74 points on Molmo2-4B, where PAI at γ = 1.5 falls below unguided decoding.
M3ID has no constant scale: its weight grows with the distance from the image, and it acts only when the model is uncertain about the next token. We therefore sweep its rate λ from 0.005 to 0.08 (default 0.02). It stays within 0.5 points of unguided decoding at every λ on both models, because recent VLMs are rarely that uncertain on short answers: on Qwen2.5-VL-7B, M3ID leaves 99.0% of answers unchanged. The full sweep, per benchmark, is now in Appendix A.5 of the revision.
Response 1, Part 3
W2 (continued). Model dependence: small gains on InternVL3.5, and 13 diverse VLMs
We agree with the reviewer: the gain of plain guidance depends on the model, and it is small on InternVL3.5. Understanding why led us to a simple, label-free fix. We also extended the evaluation from 6 to 13 VLMs, chosen to span very different architectures: nine series from 3B to 8B parameters, models with few vision tokens (FastVLM; LLaVA-Mini, with one), a unified understanding and generation model (BAGEL) and a Mamba language model (Cobra).
Why the gain varies. Guidance corrects the language prior (Eq. 2), the answer the model would give without looking. Asked what color a fire truck is, a model answers "red" without seeing any image, and it often gives the same answer with the image, whatever the image shows. If the truck in the image is yellow, guidance pushes the answer towards "yellow". An answer the image has already decided is different: guidance has nothing left to correct, and its push can only make plausible edits, which cost accuracy when they land on a correct answer. Where a model makes many such edits, as InternVL3.5 does, they cost almost as much as guidance gains on the other answers.
Prior-Match Gating (PMG) therefore guides only the answers the prior decided. It finds them by asking the model blind: besides its unguided answer with the image, , the model answers the same prompt with the image removed, , the input its negative stream already reads. When the two match, the image did not change the answer, so the answer is the prior's and PMG keeps the guided answer ; otherwise it keeps :
where tests whether the first words of the two answers agree. PMG needs no labels and no extra model. It is the per-answer analogue of selective guidance in diffusion, which applies guidance only where it helps: within a limited interval of the trajectory [3], or only when the sample resembles the negative concept [4].
Table R4. Ablation of PMG: VLM-CFG with and without the gate (mean over the ten benchmarks of Table 3; gain over unguided in parentheses).
| Model | Unguided | VLM-CFG | VLM-CFG + PMG |
|---|---|---|---|
| Qwen2.5-VL-3B | 53.73 | 57.15 (+3.42) | 57.40 (+3.67) |
| Qwen2.5-VL-7B | 54.64 | 59.14 (+4.50) | 59.16 (+4.52) |
| Qwen3-VL-4B | 55.91 | 56.86 (+0.95) | 57.00 (+1.09) |
| Qwen3-VL-8B | 59.95 | 61.40 (+1.45) | 61.35 (+1.40) |
| InternVL3.5-4B | 54.00 | 54.34 (+0.34) | 55.11 (+1.11) |
| InternVL3.5-8B | 55.95 | 56.16 (+0.21) | 57.16 (+1.21) |
| Molmo2-4B | 56.40 | 58.05 (+1.65) | 58.52 (+2.12) |
| Molmo2-8B | 54.69 | 57.74 (+3.05) | 58.04 (+3.35) |
| BAGEL-7B-MoT | 54.42 | 54.07 (−0.35) | 57.10 (+2.68) |
| FastVLM-7B | 52.35 | 51.13 (−1.22) | 54.18 (+1.83) |
| LLaVA-Mini | 43.77 | 44.57 (+0.80) | 44.77 (+1.00) |
| LLaVA-OV-1.5-8B | 56.90 | 59.02 (+2.12) | 60.44 (+3.54) |
| Cobra-3B | 34.72 | 33.93 (−0.79) | 37.98 (+3.26) |
| Mean gain | +1.24 | +2.37 | |
| Models above unguided | 10 / 13 | 13 / 13 |
PMG turns the three losses of plain guidance (BAGEL, FastVLM-7B, Cobra) into gains and raises InternVL3.5-8B, the model the reviewer names, from +0.21 to +1.21 points. Where plain guidance already works, as on the Qwen models, it changes little.
Response 1, Part 1
We thank the reviewer for the positive and thoughtful review. We are glad that the reviewer found the guidance view a principled way to compare contrastive decoders, the analysis of prefix leakage insightful, and the resulting design simple. We answer each point below, and every change is included in the revised paper.
W1) What the guidance signal carries
We agree that this is the central question about the signal, and it can be measured. By our bias decomposition (Section 3.1, Eq. 3), the guidance term is image evidence plus a reference bias: the gap between the image-free prediction and the prediction averaged over all images. The bias depends on the question but not on the image. Two tests bound its share of the gain.
Removing the bias. Replacing the image-free reference with the model's prediction averaged over other images moves the reference toward that average, which removes most of the bias and leaves the image evidence. Table R1 does this on Qwen2.5-VL-7B, averaging the reference over M images drawn at random from the same benchmark.
Table R1. Accuracy (%) on Qwen2.5-VL-7B of unguided decoding and of VLM-CFG (α = 1.5) with different references.
A single random image is a noisy estimate that carries the content of a wrong image, and it trails deletion by 0.86 points. Averages of 4, 8 and 16 images match deletion: they lie within 0.54 points of it, and no per-benchmark difference is significant (95% paired bootstrap). With most of the bias removed, guidance keeps nearly all of its gain over unguided decoding (1.9 to 2.6 points, against 2.5 for deletion), so the gain is carried by the image evidence. Deletion obtains it with one image-free pass instead of M image passes.
An image-blind shift explains little. On PhD, our largest yes-or-no gain, we also test the simplest bias, one shift toward "yes" shared by every question, added to the unguided model (the test of [1]).
Table R2. PhD on Qwen2.5-VL-7B. The PhD index is the harmonic mean of the two recalls.
A shared shift can only trade no recall for yes recall: it gains 1.5 points at VLM-CFG's yes rate, and 2.9 even when tuned on the test set. VLM-CFG gains 7.1 while keeping no recall, and of the 136 answers it changes, in both directions, 85% become correct.
An exact split would need the prediction averaged over all images consistent with the question, which is not feasible; both tests indicate that the bias contributes little and the image evidence carries the gain. The revision adds this analysis as Section 4.2.1, which Section 3.1 now points to right after the bias decomposition; the full Table R1 is in Appendix A.2 and Table R2 in Appendix A.6.