acceptodds
Under review as a conference paper at ICLR 2027

Contrastive Decoding Is Classifier-Free Guidance: Unifying Visual Guidance for Vision-Language Models

Abstract

Classifier-free guidance is standard in image and video generation, yet is not commonly used in vision-language models. We show that the field has effectively used it without recognizing it: existing contrastive decoding methods are classifier-free guidance with different references, distinguished by the bias each reference carries. This unified formulation exposes a common design space for autoregressive generation. Exploring this space, we find that corrupted-image references increasingly align with simply deleting the image, while weaker models and degraded attention provide no better guidance. Guidance operators from diffusion likewise reduce to vanilla guidance on different scales. The key distinction from diffusion lies in the prefix: an autoregressive reference must condition on some answer prefix, but it is unclear which. Existing methods use the model’s own prefix, leaking image information into the image-free reference, and rapidly collapsing the guidance signal. A prefix-free reference avoids this leakage but becomes stale and overguides long answers. Switching between the two balances these complementary failure modes. Empirically, our approach improves Qwen2.5-VL-7B by 5.7 points and Qwen3-VL-8B by 1.7 points across ten general benchmarks, outperforming four published contrastive decoders at 1.04× the cost of unguided decoding, compared with up to 2× for prior methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
CFG, VLM, LLM

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_M3LTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper unifies contrastive decoding for vision-language models (VLMs) under the classifier-free guidance (CFG) framework, showing that existing methods can be analyzed in terms of their reference distributions, guidance strength, and guidance duration. Through systematic experiments, the authors identify a key limitation of teacher-forced references: conditioning the image-free branch on generated answer prefixes leaks visual information and weakens the guidance signal. They propose SWITCH, which initially uses a fixed image-free reference and switches to a teacher-forced reference after the first K=8K=8 tokens. Combined with image deletion as the negative reference and a fixed guidance scale of α=1.5\alpha=1.5, the resulting VLM-CFG method requires no additional training. Evaluation on ten benchmarks across five models shows improvements of 5.69 and 1.72 percentage points over unguided decoding on Qwen2.5-VL-7B and Qwen3-VL-8B, respectively, with a reported inference cost of 1.04×. However, gains are limited on InternVL3.5-8B, indicating that the method's effectiveness may depend on the underlying model.

Critical strengths
  1. Unified theoretical framework. The paper formulates contrastive decoding as classifier-free guidance, providing a principled framework for understanding and comparing existing methods.

  2. Insightful analysis and simple design. The paper identifies answer-prefix leakage as a key limitation of teacher-forced references and proposes SWITCH to balance guidance strength and reference freshness.

  3. Empirical effectiveness and efficiency. Experiments across multiple models and ten benchmarks demonstrate consistent gains on several backbones, with only 1.04× inference cost for the proposed method.

Critical weaknesses
  1. The mechanistic interpretation of the guidance signal remains incomplete. The paper acknowledges that the image-free reference is not necessarily the true marginal distribution, so the guidance signal contains both image-related information and reference-distribution bias. Although the prefix-swap experiments support the proposed explanation of guidance decay, the relative contributions of these effects to the observed performance gains remain unclear.

  2. The generalizability of the proposed configuration is not fully established. SWITCH relies on a fixed switching threshold (K=8K=8) and guidance scale (α=1.5\alpha=1.5). Although the paper provides partial sensitivity analyses, their coverage across model families and task distributions remains limited. The negligible overall gain on InternVL3.5-8B further suggests that the effectiveness of the proposed configuration is model-dependent.

  3. The efficiency advantage is strongly dependent on answer length. The reported 1.04× inference cost primarily reflects the short-answer evaluation setting, whereas the appendix estimates 1.62–1.68× overhead on longer-answer benchmarks. This substantially qualifies the efficiency claim and leaves the practical trade-off less clear for long-form generation.

Questions for authors
  1. Baseline tuning and comparison fairness. Could the authors clarify how the hyperparameters of the compared methods were selected for each backbone? Were the baselines evaluated under their original settings or tuned on validation data? Would the main conclusions hold under a comparable tuning protocol?

  2. Practical inference efficiency. Could the authors provide measured end-to-end latency and throughput, in addition to forward-pass-based cost estimates, for representative short- and long-answer tasks? This would help clarify the practical overhead under realistic inference conditions.

  3. Robustness to decoding settings. All evaluations use greedy decoding. Do the authors have preliminary results or a technical rationale for whether the proposed method would remain effective under temperature and top-pp sampling?

  4. Sensitivity to prompt and stopping conventions. Since the method can affect both token selection and when generation terminates, how sensitive are the results to benchmark prompt templates and stopping criteria? Could the authors clarify whether these settings are kept identical across all compared methods?

8 comments
AuthorsOpenReview ↗

Response 1, Part 1

We thank the reviewer for the positive and thoughtful review. We are glad that the reviewer found the guidance view a principled way to compare contrastive decoders, the analysis of prefix leakage insightful, and the resulting design simple. We answer each point below, and every change is included in the revised paper.


W1) What the guidance signal carries

We agree that this is the central question about the signal, and it can be measured. By our bias decomposition (Section 3.1, Eq. 3), the guidance term is image evidence plus a reference bias: the gap between the image-free prediction and the prediction averaged over all images. The bias depends on the question but not on the image. Two tests bound its share of the gain.

Removing the bias. Replacing the image-free reference with the model's prediction averaged over other images moves the reference toward that average, which removes most of the bias and leaves the image evidence. Table R1 does this on Qwen2.5-VL-7B, averaging the reference over M images drawn at random from the same benchmark.

Table R1. Accuracy (%) on Qwen2.5-VL-7B of unguided decoding and of VLM-CFG (α = 1.5) with different references.

ReferenceVizWizTreeBenchMMStarBLINKMean
Unguided74.0840.7462.4925.3350.66
Image deleted (ours)78.0142.9662.1729.3353.12
1 random image77.3542.2261.4628.0052.26
Average of 4 random images77.8042.7262.1729.3353.00
Average of 8 random images77.7541.9862.5130.6753.22
Average of 16 random images77.7142.2263.0427.3352.58

A single random image is a noisy estimate that carries the content of a wrong image, and it trails deletion by 0.86 points. Averages of 4, 8 and 16 images match deletion: they lie within 0.54 points of it, and no per-benchmark difference is significant (95% paired bootstrap). With most of the bias removed, guidance keeps nearly all of its gain over unguided decoding (1.9 to 2.6 points, against 2.5 for deletion), so the gain is carried by the image evidence. Deletion obtains it with one image-free pass instead of M image passes.

An image-blind shift explains little. On PhD, our largest yes-or-no gain, we also test the simplest bias, one shift toward "yes" shared by every question, added to the unguided model (the test of [1]).

Table R2. PhD on Qwen2.5-VL-7B. The PhD index is the harmonic mean of the two recalls.

Yes %Yes recallNo recallPhD index
Unguided41.5064.4181.4171.92
+ "yes" shift, at VLM-CFG's yes rate47.4170.9276.1073.41
+ "yes" shift, tuned on test52.3977.2972.5174.82
VLM-CFG47.3476.4981.8179.06

A shared shift can only trade no recall for yes recall: it gains 1.5 points at VLM-CFG's yes rate, and 2.9 even when tuned on the test set. VLM-CFG gains 7.1 while keeping no recall, and of the 136 answers it changes, in both directions, 85% become correct.

An exact split would need the prediction averaged over all images consistent with the question, which is not feasible; both tests indicate that the bias contributes little and the image evidence carries the gain. The revision adds this analysis as Section 4.2.1, which Section 3.1 now points to right after the bias decomposition; the full Table R1 is in Appendix A.2 and Table R2 in Appendix A.6.

AuthorsOpenReview ↗

Response 1, Part 2


W2) & Q1) Fixed settings and baseline tuning

One setting for every model and benchmark. VLM-CFG has two settings, and neither is tuned per benchmark or per model. The switch point K = 8 was set from answer-length statistics before reading any score, and α = 1.5 is the standard guidance scale for text generation [2]. Both stay fixed across all 13 models and every benchmark, so VLM-CFG is a simple, generic change to the decoding loop with nothing to tune. The baselines are treated the same way: each runs at its published setting, the same on every backbone, and no method, ours included, is tuned on validation data.

Sweeping is not worth it. To test whether fixed settings favour us, Table R3 gives every method the same scale sweep as VLM-CFG, on two models from different families.

Table R3. Mean accuracy (%) over OK-VQA, VizWiz, PhD, TreeBench and V* as the guidance scale s grows. Each method applies s to its own contrast (VCD and SID: α_cd = s − 1; PAI: γ = s; VLM-CFG: α = s, without the gate). † Default setting. Bold: best of each row.

Scale s1.251.5234
Qwen2.5-VL-7B (unguided 59.89)
VCD61.1661.4261.36†60.5660.64
SID61.5462.4462.70†62.4162.09
PAI (first rung γ = 1.1)63.07†65.0564.4163.5663.27
VLM-CFG64.1264.73†64.3063.8063.64
Molmo2-4B (unguided 60.81)
VCD61.1660.9560.00†59.5958.86
SID61.2360.9060.62†60.3559.88
PAI (first rung γ = 1.1)61.65†60.3659.5359.0358.77
VLM-CFG62.3862.10†61.7061.2861.25

Three results stand out.

  • Tuning the baselines does not change the conclusion. Even with each baseline at its best scale for each model, picked on these test results, VLM-CFG at its fixed α = 1.5 beats VCD, SID and M3ID on both models, and PAI on Molmo2-4B.
  • VLM-CFG is the most robust. It is the only method above unguided decoding at every scale on both models, and it moves by about one point across the whole ladder, against up to three for PAI.
  • α = 1.5 is the best single setting. Averaged over the two models, it beats every setting of every method, its own included (63.42, against 62.71 for the best baseline setting, PAI at γ = 1.5). Retuning α per model would gain at most 0.28 points.

The only setting above ours is PAI at γ = 1.5 on Qwen2.5-VL-7B, by 0.32. PAI combines contrastive decoding with attention manipulation. At γ = 1.5 its contrastive term runs at our scale, and the attention boost on top helps slightly on Qwen2.5-VL-7B but costs 1.74 points on Molmo2-4B, where PAI at γ = 1.5 falls below unguided decoding.

M3ID has no constant scale: its weight grows with the distance from the image, and it acts only when the model is uncertain about the next token. We therefore sweep its rate λ from 0.005 to 0.08 (default 0.02). It stays within 0.5 points of unguided decoding at every λ on both models, because recent VLMs are rarely that uncertain on short answers: on Qwen2.5-VL-7B, M3ID leaves 99.0% of answers unchanged. The full sweep, per benchmark, is now in Appendix A.5 of the revision.

AuthorsOpenReview ↗

Response 1, Part 3


W2 (continued). Model dependence: small gains on InternVL3.5, and 13 diverse VLMs

We agree with the reviewer: the gain of plain guidance depends on the model, and it is small on InternVL3.5. Understanding why led us to a simple, label-free fix. We also extended the evaluation from 6 to 13 VLMs, chosen to span very different architectures: nine series from 3B to 8B parameters, models with few vision tokens (FastVLM; LLaVA-Mini, with one), a unified understanding and generation model (BAGEL) and a Mamba language model (Cobra).

Why the gain varies. Guidance corrects the language prior (Eq. 2), the answer the model would give without looking. Asked what color a fire truck is, a model answers "red" without seeing any image, and it often gives the same answer with the image, whatever the image shows. If the truck in the image is yellow, guidance pushes the answer towards "yellow". An answer the image has already decided is different: guidance has nothing left to correct, and its push can only make plausible edits, which cost accuracy when they land on a correct answer. Where a model makes many such edits, as InternVL3.5 does, they cost almost as much as guidance gains on the other answers.

Prior-Match Gating (PMG) therefore guides only the answers the prior decided. It finds them by asking the model blind: besides its unguided answer with the image, y^v\hat y^{v}, the model answers the same prompt with the image removed, y^∅\hat y^{\varnothing}, the input its negative stream already reads. When the two match, the image did not change the answer, so the answer is the prior's and PMG keeps the guided answer y^α\hat y^{\alpha}; otherwise it keeps y^v\hat y^{v}:

y^=y^α if m(y^v,y^∅)=1, and y^=y^v otherwise,\hat{y} = \hat{y}^{\alpha} \text{ if } m(\hat{y}^{v}, \hat{y}^{\varnothing}) = 1, \text{ and } \hat{y} = \hat{y}^{v} \text{ otherwise,}

where mm tests whether the first words of the two answers agree. PMG needs no labels and no extra model. It is the per-answer analogue of selective guidance in diffusion, which applies guidance only where it helps: within a limited interval of the trajectory [3], or only when the sample resembles the negative concept [4].

Table R4. Ablation of PMG: VLM-CFG with and without the gate (mean over the ten benchmarks of Table 3; gain over unguided in parentheses).

ModelUnguidedVLM-CFGVLM-CFG + PMG
Qwen2.5-VL-3B53.7357.15 (+3.42)57.40 (+3.67)
Qwen2.5-VL-7B54.6459.14 (+4.50)59.16 (+4.52)
Qwen3-VL-4B55.9156.86 (+0.95)57.00 (+1.09)
Qwen3-VL-8B59.9561.40 (+1.45)61.35 (+1.40)
InternVL3.5-4B54.0054.34 (+0.34)55.11 (+1.11)
InternVL3.5-8B55.9556.16 (+0.21)57.16 (+1.21)
Molmo2-4B56.4058.05 (+1.65)58.52 (+2.12)
Molmo2-8B54.6957.74 (+3.05)58.04 (+3.35)
BAGEL-7B-MoT54.4254.07 (−0.35)57.10 (+2.68)
FastVLM-7B52.3551.13 (−1.22)54.18 (+1.83)
LLaVA-Mini43.7744.57 (+0.80)44.77 (+1.00)
LLaVA-OV-1.5-8B56.9059.02 (+2.12)60.44 (+3.54)
Cobra-3B34.7233.93 (−0.79)37.98 (+3.26)
Mean gain+1.24+2.37
Models above unguided10 / 1313 / 13

PMG turns the three losses of plain guidance (BAGEL, FastVLM-7B, Cobra) into gains and raises InternVL3.5-8B, the model the reviewer names, from +0.21 to +1.21 points. Where plain guidance already works, as on the Qwen models, it changes little.

Reviewer_fbZfOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

The paper argues that visual contrastive decoding is classifier-free guidance with a different negative stream. With that view it goes through the choice of negative, the guidance operators from diffusion (APG, CFG-Zero*, TCFG, EP-CFG), and the prefix the image-free stream sees. The take-aways are that dropping the image with plain guidance is enough, that the operators reduce to different guidance scales under greedy decoding, and that the teacher-forced reference loses most of its signal after one token while a prefix-free reference overguides long answers. SWITCH uses the prefix-free reference for the first K = 8 tokens and teacher forcing after that, and with one fixed setting gives +5.7 on Qwen2.5-VL-7B and +1.7 on Qwen3-VL-8B over ten benchmarks at 1.04x cost.

Critical strengths

The prefix analysis is the part I found most interesting, and it is done carefully: the teacher-forced reference loses 47-78% of its push after the first generated token (Table 8a), checked with swapped and shuffled prefixes (Tables 8c, 9). Earlier contrastive decoders never state what their reference has seen, and here it turns into a method that costs almost nothing. I also appreciate the general benchmarks, current models, and that the appendix reports what did not work.

Critical weaknesses

My main concern is how the main comparison is set up. Nine of the ten Table 6 benchmarks also appear in the Sec. 4.1 analysis or the appendix. K = 8 was set before looking at scores, which is fine, but alpha = 1.5 is only "fixed in advance," and the paper itself notes that the best scale differs by benchmark (Table 18). Meanwhile the four baselines in Table 6 run at their published settings, tuned on older models, and none of them gets the scale sweep that B.7 promises. Also, "PhD" in Table 6 is a different protocol from "PhD-rat." in Table 3, and the unguided numbers differ between the two tables, so it would help to say which runs these are.

Second, the paper itself says the O3 and PhD-rat gains come from the stop decision (strength oracle in Table 3; App. A.2). I appreciate the honesty, but it means part of the gain is about when the answer ends, not about looking at the image. The PhD column in Table 6 (67.50 -> 79.15, the largest gain) is the yes/no protocol, and App. C.2 itself cites work showing that earlier contrastive decoding gains came from a one-directional shift of the output distribution. That is exactly the thing to rule out here. Finally, in Table 19 the fixed configuration gives only +0.95 on Qwen3-4B and +0.07 on InternVL3.5-8B, below SID and PAI, with no discussion; the abstract reports only the two larger Qwen models.

Questions for authors
  1. How was alpha = 1.5 chosen? I would appreciate seeing Table 6 with each baseline given a scale sweep like the one Table 18 gives SWITCH.
  2. For the Table 6 PhD gain, can you show it is not just a shift in the yes/no answer distribution? For O3, does an unguided run truncated after the first label recover most of the F1 gain?
  3. Everything is greedy. Sec. 3.4 notes that under sampling the factor a survives as an inverse temperature, so the "all operators are a scale" story may not hold there. Any results with sampling?
  4. HallusionBench and AMBER are listed in B.8 but I could not find their results. Where are they?
7 comments
AuthorsOpenReview ↗

Response 1, Part 1

We thank the reviewer for a careful and constructive review. We are glad that the prefix analysis was the most interesting part: what the reference stream reads is a choice every contrastive decoder makes without stating it, and it is the core of our method. We also appreciate the recognition of the general benchmarks, the current models and the reported negative results. We answer each point below, and every change is included in the revised paper.


W1) & Q1) How α = 1.5 was chosen, and a scale sweep for every method

Why a small scale. α = 1.5 is the standard guidance scale for autoregressive text generation. Classifier-free guidance for language models [1] uses it as its main setting, finds the best scale below 2 for most prompts, and reports that reasoning chains degrade above 1.5. Our paper explains why decoding needs a small scale. Diffusion revises the whole image at every step, so a step that overshoots is corrected by the next one. A decoder commits to each token once and conditions every later token on it, so a scale strong enough to force one wrong token derails the rest of the answer.

Why one value. The best scale does vary with the task (Table 22): on Molmo2-4B, for instance, PhD keeps gaining up to α = 3, while VizWiz peaks at 1.25 on both models. But retuning α buys little. On the two models in Table R1, the best scale is 1.5 on Qwen2.5-VL-7B and 1.25 on Molmo2-4B, only 0.28 points higher. A sweep for every new model or benchmark would cost far more than it returns, so we fixed α = 1.5 in advance and never tuned it per benchmark. We see this as a practical strength: VLM-CFG is a one-line change to the decoding loop, with nothing to tune.

A scale sweep for every method. We agree that the baselines should not be held to their published settings. Table R1 gives each of them the same sweep as VLM-CFG, on two models from different families.

Table R1. Mean accuracy (%) over OK-VQA, VizWiz, PhD, TreeBench and V* as the guidance scale s grows. Each method applies s to its own contrast (VCD and SID: α_cd = s − 1; PAI: γ = s; VLM-CFG: α = s, without the gate). † Default setting. Bold: best of each row.

Scale s1.251.5234
Qwen2.5-VL-7B (unguided 59.89)
VCD61.1661.4261.36†60.5660.64
SID61.5462.4462.70†62.4162.09
PAI (first rung γ = 1.1)63.07†65.0564.4163.5663.27
VLM-CFG64.1264.73†64.3063.8063.64
Molmo2-4B (unguided 60.81)
VCD61.1660.9560.00†59.5958.86
SID61.2360.9060.62†60.3559.88
PAI (first rung γ = 1.1)61.65†60.3659.5359.0358.77
VLM-CFG62.3862.10†61.7061.2861.25

Three results stand out.

  • Tuning the baselines does not change the conclusion. Even with each baseline at its best scale for each model, picked on these test results, VLM-CFG at its fixed α = 1.5 beats VCD, SID and M3ID on both models, and PAI on Molmo2-4B.
  • VLM-CFG is the most robust. It is the only method above unguided decoding at every scale on both models, and it moves by about one point across the whole ladder, against up to three for PAI.
  • α = 1.5 is the best single setting. Averaged over the two models, it beats every setting of every method, its own included (63.42, against 62.71 for the best baseline setting, PAI at γ = 1.5).

The only setting above ours is PAI at γ = 1.5 on Qwen2.5-VL-7B, by 0.32. PAI combines contrastive decoding with attention manipulation. At γ = 1.5 its contrastive term runs at our scale, and the attention boost on top helps slightly on Qwen2.5-VL-7B but costs 1.74 points on Molmo2-4B, where PAI at γ = 1.5 falls below unguided decoding.

M3ID has no constant scale: its weight grows with the distance from the image, and it acts only when the model is uncertain about the next token. We therefore sweep its rate λ from 0.005 to 0.08 (default 0.02). It stays within 0.5 points of unguided decoding at every λ on both models, because recent VLMs are rarely that uncertain on short answers: on Qwen2.5-VL-7B, M3ID leaves 99.0% of answers unchanged. The full sweep, per benchmark, is now in Appendix A.5 of the revision.

AuthorsOpenReview ↗

Response 1, Part 2


W1) (continued) What PhD-rat is

PhD-rat is PhD with a reason before the answer. It asks the same questions as PhD [2], but prompts the model to give a one-sentence reason before its yes or no, and scores that answer; PhD (Table 3) asks for the yes or no directly. We built PhD-rat for the reference analysis of Table 6: the reference rules differ only once the answer has a prefix, and PhD-rat places about 30 tokens before the decision. Because the prompts differ, so do the unguided scores, and each table compares every method with the unguided run of its own protocol. The revision describes both protocols in the benchmark appendix.


W2) Is the PhD gain a mirage, a shift toward "yes"?

No, it is not a mirage. We ran the test that the mirage paper [3] uses for exactly this claim (its Section 5.1). It pushes the unguided model toward "yes", either through the prompt (PBA: "Answer Yes if possible.") or at the output layer (OLM: answer "yes" whenever p(yes) and p(no) are within τ), and asks whether that shift alone reproduces the gain. Table R2 applies it to PhD.

Table R2. PhD results using Qwen2.5-VL-7B. The PhD index, the benchmark's official score, is the harmonic mean of the two recalls. The yes-logit bias adds b to the "yes" logit, the general form of any shift toward "yes". "Tuned on test" picks τ or b on these same questions, an upper bound no real method could reach.

Yes %Yes recallNo recallAccuracyPhD index
Unguided41.5064.4181.4172.9171.92
VLM-CFG47.3476.4981.8179.1579.06
Shifts of the unguided model toward "yes"
PBA (prompt)41.7062.4279.0270.7269.74
OLM at VLM-CFG's yes rate (τ = 0.328)47.3470.7876.1073.4473.34
OLM, tuned on test (τ = 0.625)52.9977.9571.9874.9774.85
Yes-logit bias, tuned on test (b = 1.4)52.3977.2972.5174.9074.82

Guidance does say "yes" more often (47.3% against 41.5%), so the question is fair. But a shift toward "yes" can only buy yes recall with no recall, and the index penalises that trade. Even tuned on the test set, the best shift gains 2.9 points; at exactly VLM-CFG's yes rate, OLM gains 1.4. VLM-CFG gains 7.1, raising yes recall by 12.1 points while no recall holds (81.4 → 81.8).

The flipped answers show why. VLM-CFG turns 112 unguided "no" answers into "yes" and is right on 92 of them; OLM at the same yes rate turns 88 and is right on only 48. VLM-CFG also turns 24 "yes" answers into "no", which no shift toward "yes" can do, and is right on 23. Guidance does not lower the bar for "yes"; it changes the answers that the image contradicts. We add this test to the revision as Appendix A.6.

AuthorsOpenReview ↗

Response 1, Part 3


W2) & Q2). Does part of the gain come from where the answer ends, rather than from the image?

On O3, yes, as the paper states, and the suggested control now confirms it. Section 4.2 attributes the O3 gain to the stop decision. O3 asks which of seven named properties make an object stand out; the unguided model lists 2.4 of them on average, against 1.8 in the ground truth, and guidance shortens the list to 1.3. We thank the reviewer for this simple and reliable test: truncating the unguided answer after its first label raises F1 from 57.39 to 64.97, nearly all of VLM-CFG's 65.17 (Table R3). We add it to Appendix A.5 and cite it next to that statement.

Table R3. O3 on Qwen2.5-VL-7B.

Properties listedF1
Ground truth1.78–
Unguided2.4057.39
Unguided, truncated after the first label≤ 164.97
VLM-CFG1.2965.17

The conclusions do not rest on O3. O3 and O3-box are two of the eleven benchmarks. Without them, VLM-CFG still improves all 13 models, is the strongest of the five decoders on all 13, and its mean gain rises from 2.33 to 2.57 points. On Qwen2.5-VL-7B its lead over PAI grows from 0.94 to 1.77, because PAI gains more on these two benchmarks than we do.

The gain mainly comes from the image. Outside O3, four results show that guidance changes what the answer says, not only where it ends.

1. The changed answers follow the image. PhD asks for a single yes or no, so there is no list to shorten. There, VLM-CFG changes 136 answers, in both directions, and 85% of them become correct (Table R2).

2. Removing the reference bias keeps the gain. The image-free reference carries a bias that does not depend on the image (Section 3.1, Eq. 3). Averaging the reference over 8 random images removes most of this bias, and guidance keeps its gain: +2.56 points, against +2.46 with the image deleted (Section 4.2.1).

3. Qualitative examples. Figures 9 to 12 (Appendix A.8) show 28 questions where VLM-CFG corrects the unguided answer, across language priors, counterfactual images, illusions, counting, small or blurred details, text and diagrams, and four failures. In Figure 10, asked what a cat is sitting on, the model with the image gives "chair" and "mirror" 35.2% each and answers "chair", the answer it prefers without the image (15.6%, against under 0.1% for "mirror"). Guidance subtracts that prior, and VLM-CFG answers "mirror" with 96.9%.

4. Gating on the image adds gain. If the gain came from where answers end, it would not depend on whether the image changed the answer. It does. Guidance corrects the language prior (Eq. 2), the answer the model would give without looking. Asked what color a fire truck is, a model answers "red" without seeing any image, and it often gives the same answer with the image, whatever the image shows. If the truck in the image is yellow, guidance pushes the answer towards "yellow". An answer the image has already decided leaves guidance nothing to correct, and its push can only make plausible edits, which cost accuracy when they land on a correct answer. Our new Prior-Match Gating (PMG, Section 3.5) therefore guides only the answers the prior decided. It finds them by asking the model blind: besides its unguided answer with the image, y^v\hat y^{v}, the model answers the same prompt with the image removed, y^∅\hat y^{\varnothing}, the input its negative stream already reads. When the two match, the image did not change the answer, so the answer is the prior's and PMG keeps the guided answer y^α\hat y^{\alpha}; otherwise it keeps y^v\hat y^{v}:

y^=y^α  if  m(y^v,y^∅)=1,y^=y^v  otherwise,\hat y = \hat y^{\alpha} \ \text{ if } \ m(\hat y^{v}, \hat y^{\varnothing}) = 1, \qquad \hat y = \hat y^{v} \ \text{ otherwise},

where mm tests whether the first words of the two answers agree. PMG needs no labels and no extra model. It is the per-answer analogue of selective guidance in diffusion, which applies guidance only where it helps: within a limited interval of the trajectory [4], or only when the sample resembles the negative concept [5]. Guidance raises the answers that match the blind answer by 3.7 to 12.0 points and leaves the others nearly unchanged, and with the gate VLM-CFG gains on all 13 models (Tables R4 and R5, next comment).

Reviewer_SRHoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes visual contrastive decodings as CFG and explains why we can view them as CFG through a systematic analysis, including negative references and guidance operators. Furthermore, this work shows that existing visual contrastive decoding methods can leak visual information through the decoding stage during token generation, which can weaken the guidance signal. To solve this problem, they propose VLM-CFG, which switches the early step of contrastive decoding to use the cached prefix-free reference. Experiments on various models and benchmarks show consistent performance and cost gains.

Critical strengths

S1. The analysis of the work provides existing visual contrastive decoding methods under a CFG formulation and shows which design parts actually lead to performance improvements. This gives useful insights for the analysis of contrastive decoding.

S2. This work investigates the reduction of the guidance signal under different answer prefixes and prefix windows, which can provide motivation for the prefix-free reference-based method.

Critical weaknesses

W1. The main evaluation performance gains are concentrated in short-answer benchmarks. The proposed reference switching operates only when the inference output length is long, so it rarely activates on those kinds of benchmarks. As shown in Table 3, most of these benchmarks show the same performance results for I-1 and SWITCH. Furthermore, on longer response benchmarks, SWITCH does not consistently outperform TF and reaches similar computational cost.

W2. The VLM-CFG results vary across the model families. Most gains are shown in Qwen-series models, while InternVL shows much less gain on average and high variation across benchmarks. This result limits whether the proposed VLM-CFG method really can improve the inference performance in general.

Questions for authors

Q1. How much the reference switching actually occurs in the given main benchmarks?

Q2. It may be better to check end-to-end latency for short and long answer generation separately.

6 comments
AuthorsOpenReview ↗

Response 1, Part 1

We thank the reviewer for the positive assessment and the constructive comments. We are glad that the reviewer found the CFG view of contrastive decoding useful for seeing which design choices matter (S1), and that the analysis of the reference prefix motivates our method (S2). We address each point below, and all changes are included in the revised paper.


W1) What does the switch add?

Existing decoders take TF for granted. Contrastive decoders come from text decoding, where the second stream naturally reads the answer generated so far; Teacher-Forcing (TF). Our CFG view raises a question these methods never ask: which prefix should the image-free stream read? Diffusion does not face it, since both streams see the same noisy sample. In visual autoregressive generation, however, SoftCFG [1] observed that the gap between the two streams vanishes as decoding proceeds. This led us to measure the same effect in VLMs: under TF, a single generated token removes about two thirds of the guidance signal, because the answer describes the image and leaks it into the stream that should not see it (Finding 2). Reading only the question (I-1) avoids this leak.

SWITCH builds on our analysis of both rules. Neither rule works on every answer. Always using I-1 beats always using TF on every short benchmark of Table 6, by 1.2 to 4.9 points, but on long answers its push never fades, and it falls up to 12.4 points below TF. Since answer length is unknown before decoding, SWITCH uses I-1 for the first K = 8 tokens, while the answer is decided, and TF afterwards.

An idle switch is not an idle method. A fire rate near 0% may suggest that the switch is useless there. It is the opposite: while the switch is idle, VLM-CFG runs I-1, the rule our analysis calls for on short answers. On single-token answers, I-1 and TF coincide, and on short multi-token answers, always using TF loses 2.74 points (Table R1). Idle, the switch keeps the leak out of short answers; firing, it keeps the stale push of I-1 out of long ones. This is the design, and Table R1 confirms both sides.

Four new long-answer benchmarks. To test the long-answer side, we added NaturalBench, MathVista, We-Math, and DynaMath (69 to 299 tokens) to Table 6. Table R1 summarizes the revised table, and Table 6 provides all benchmarks.

On long answers SWITCH matches TF, by design. We agree that SWITCH does not outperform TF on long answers. After the switch it is TF, so it inherits the accuracy of TF and most of its cost; its role there is to avoid the losses of I-1. Section 3.4 states that SWITCH is identical to TF after the switch, and Section 4.2.3 reports that it matches TF on long answers (61.40 against 61.36). Our answer to W3 reports latency separately for short and long answers.

Table R1. Reference rules on Qwen2.5-VL-7B over 11 benchmarks (revised Table 6). Short: answers shorter than K tokens on average (3 benchmarks); Long: the other 8.

RuleShortLongAll
No guidance57.9159.5359.09
Always TF62.4861.3661.66
Always I-165.2257.7659.79
SWITCH65.2261.4062.44

SWITCH keeps the gain of I-1 on short answers and the accuracy of TF on long ones (+3.64 over always using I-1), and it is the best rule overall. It is also the only rule that never falls below unguided decoding.

AuthorsOpenReview ↗

Response 1, Part 2


W2) Gains vary across model families and are small on InternVL

We agree with the reviewer: the gain of plain guidance depends on the model, and it is small on InternVL3.5. Understanding why led us to a simple, label-free fix. We also extended the evaluation from 6 to 13 VLMs, chosen to span very different architectures: nine series from 3B to 8B parameters, models with few vision tokens (FastVLM; LLaVA-Mini, with one), a unified understanding and generation model (BAGEL) and a Mamba language model (Cobra).

Why the gain varies. Guidance corrects the language prior (Eq. 2), the answer the model would give without looking. Asked what color a fire truck is, a model answers "red" without seeing any image, and it often gives the same answer with the image, whatever the image shows. If the truck in the image is yellow, guidance pushes the answer towards "yellow". An answer the image has already decided is different: guidance has nothing left to correct, and its push can only make plausible edits, which cost accuracy when they land on a correct answer. Where a model makes many such edits, as InternVL3.5 does, they cost almost as much as guidance gains on the other answers.

Prior-Match Gating (PMG) therefore guides only the answers the prior decided. It finds them by asking the model blind: besides its unguided answer with the image, y^v\hat y^{v}, the model answers the same prompt with the image removed, y^∅\hat y^{\varnothing}, the input its negative stream already reads. When the two match, the image did not change the answer, so the answer is the prior's and PMG keeps the guided answer y^α\hat y^{\alpha}; otherwise it keeps y^v\hat y^{v}:

y^=y^α  if  m(y^v,y^∅)=1,y^=y^v  otherwise,\hat y = \hat y^{\alpha} \ \text{ if } \ m(\hat y^{v}, \hat y^{\varnothing}) = 1, \qquad \hat y = \hat y^{v} \ \text{ otherwise},

where mm tests whether the first words of the two answers agree. PMG needs no labels and no extra model. It is the per-answer analogue of selective guidance in diffusion, which applies guidance only where it helps: within a limited interval of the trajectory [2], or only when the sample resembles the negative concept [3].

Table R2. Ablation of PMG: VLM-CFG with and without the gate (mean over the ten benchmarks of Table 3; gain over unguided in parentheses).

ModelUnguidedVLM-CFGVLM-CFG + PMG
Qwen2.5-VL-3B53.7357.15 (+3.42)57.40 (+3.67)
Qwen2.5-VL-7B54.6459.14 (+4.50)59.16 (+4.52)
Qwen3-VL-4B55.9156.86 (+0.95)57.00 (+1.09)
Qwen3-VL-8B59.9561.40 (+1.45)61.35 (+1.40)
InternVL3.5-4B54.0054.34 (+0.34)55.11 (+1.11)
InternVL3.5-8B55.9556.16 (+0.21)57.16 (+1.21)
Molmo2-4B56.4058.05 (+1.65)58.52 (+2.12)
Molmo2-8B54.6957.74 (+3.05)58.04 (+3.35)
BAGEL-7B-MoT54.4254.07 (−0.35)57.10 (+2.68)
FastVLM-7B52.3551.13 (−1.22)54.18 (+1.83)
LLaVA-Mini43.7744.57 (+0.80)44.77 (+1.00)
LLaVA-OV-1.5-8B56.9059.02 (+2.12)60.44 (+3.54)
Cobra-3B34.7233.93 (−0.79)37.98 (+3.26)
Mean gain+1.24+2.37
Models above unguided10 / 1313 / 13

PMG turns the three losses of plain guidance (BAGEL, FastVLM-7B, Cobra) into gains and lifts both InternVL3.5 models above +1.1 points. Where plain guidance already works, as on the Qwen models, it changes little.

Table R3. Comparison with contrastive decoding methods on all 13 VLMs (mean over the ten benchmarks of Table 3; VLM-CFG uses PMG). ‡ Includes NaturalBench. —: undefined for this model (see part 3/3).

ModelUnguidedVCDM3IDPAISIDVLM-CFG
Qwen2.5-VL-3B53.7352.7551.8754.8255.6457.40
Qwen2.5-VL-7B‡56.6957.5656.3660.0458.5660.98
Qwen3-VL-4B55.9156.1655.4356.0457.0457.00
Qwen3-VL-8B59.9559.9159.5860.4260.5261.35
InternVL3.5-4B‡56.2054.8355.9256.1655.6157.29
InternVL3.5-8B55.9555.2255.7956.5256.4957.16
Molmo2-4B56.4056.2855.8556.4556.2658.52
Molmo2-8B54.6955.9254.1655.9956.2458.04
BAGEL-7B-MoT‡56.4156.2155.7257.2855.8058.83
FastVLM-7B52.3549.7849.9152.4748.4654.18
LLaVA-Mini43.7743.8942.3543.98—44.77
LLaVA-OV-1.5-8B56.9057.2555.7557.6558.8760.44
Cobra-3B34.7233.3033.69——37.98
Mean gain (11 models with all methods)−0.30−0.80+0.79+0.39+2.36
Models above unguided5 / 130 / 1311 / 127 / 1113 / 13

VLM-CFG improves on all 13 models, across every architecture, and is the strongest decoder on 12; on Qwen3-VL-4B, SID leads by 0.04.

AuthorsOpenReview ↗

Response 1, Part 3


W2) (continued) Why PAI and SID cannot run on Cobra

Both steer the attention from text tokens to image tokens: PAI amplifies it, and SID ranks image tokens by it. Cobra's Mamba language model has no attention, so both methods are undefined there, and LLaVA-Mini passes a single vision token, which leaves SID nothing to rank. This is not a corner case: RWKV-based VLMs have no attention either, and recent hybrids such as Qwen3.5 and Nemotron Nano V2 VL keep attention in only a minority of layers. VLM-CFG, like VCD and M3ID, needs only a forward pass without the image, which every VLM supports.

Revision: new Section 3.5 (PMG and its link to diffusion guidance); Table 3 now shows Qwen2.5-VL-7B, InternVL3.5-4B and BAGEL, and Table 21 (Appendix A.5) the other ten models, per benchmark; Appendix A.11 details PMG.


W3) End-to-end latency for short and long answers

We agree, and we have measured it. Table R4 reports wall-clock seconds per sample on Qwen2.5-VL-7B (one V100 GPU, batch size 1, greedy decoding, each method at its default setting). The time covers image preprocessing, the vision encoder, prefill and every decoding step of every stream a method runs.

Table R4. Latency (s/sample), with cost relative to unguided decoding in parentheses. Benchmarks are ordered by answer length.

PhDV*TreeBenchOK-VQAVizWizNaturalBenchOCRBenchWe-Math
# tokens1.01.01.32.42.569152.5296
Unguided1.115.645.701.204.405.558.0411.65
VCD2.30 (2.07×)11.41 (2.02×)11.42 (2.00×)2.41 (2.01×)8.69 (1.98×)11.09 (2.00×)15.80 (1.97×)24.53 (2.11×)
M3ID1.36 (1.23×)5.84 (1.04×)6.06 (1.06×)1.51 (1.26×)4.84 (1.10×)8.06 (1.45×)16.52 (2.05×)25.15 (2.16×)
PAI1.49 (1.34×)6.18 (1.10×)6.25 (1.10×)1.62 (1.35×)4.95 (1.13×)10.54 (1.90×)18.37 (2.28×)30.18 (2.59×)
SID2.28 (2.05×)11.12 (1.97×)11.49 (2.02×)2.45 (2.04×)8.89 (2.02×)12.27 (2.21×)18.26 (2.27×)28.79 (2.47×)
VLM-CFG1.38 (1.24×)6.17 (1.09×)6.11 (1.07×)1.49 (1.24×)4.87 (1.11×)8.30 (1.50×)12.35 (1.54×)22.85 (1.96×)

Short answers. VLM-CFG costs 1.07 to 1.24 times unguided decoding, against about 2 times for VCD and SID. The difference lies in the negative stream. VCD and SID encode a second, degraded image and prefill all its visual tokens again, whereas the negative stream of VLM-CFG has no image, so its extra prefill covers only the text prompt. M3ID and PAI also remove the image and cost about the same as VLM-CFG.

Long answers. Here every method runs its second stream at each token, so costs rise to 1.45 to 2.6 times. VLM-CFG is the fastest on OCRBench and We-Math, on OCRBench partly because its answers are shorter (141 tokens, against 164 for M3ID and 168 for PAI).

Overall, VLM-CFG is faster than PAI, VCD and SID on all eight benchmarks and at most 6% slower than M3ID.

From estimate to measurement. The submission's 1.04× was computed from forward-pass counts on short answers, as its Appendix A.4 stated. The revision replaces it with the measurements above throughout (abstract, Figure 1, Table 3, Section 4.1 and Table 19). The conclusion is unchanged: guided decoding costs no more than PAI, VCD or SID on any benchmark we timed.


We thank the reviewer again for the helpful comments. We hope our answers resolve the concerns, and we would be glad to clarify anything further.


References

[1] SoftCFG: Uncertainty-guided Stable Guidance for Visual Autoregressive Model

[2] Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models

[3] Dynamic Negative Guidance of Diffusion Models

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision: 13 VLMs, fair tuning and measured cost

Summary of the revision

We thank all reviewers for their careful and constructive feedback. We were glad that all three found our work principled and the analysis of the reference prefix insightful.


The revised paper incorporates every suggestion; the main additions are:

  1. 13 VLMs and consistent gains. We extended the evaluation from 6 to 13 VLMs from nine series (3B to 8B), chosen to differ in architecture: a unified generation and understanding model (BAGEL), a Mamba language model (Cobra) and models with very few vision tokens. A new label-free gate, Prior-Match Gating (Section 3.5), applies guidance only to answers the model would also give without the image. VLM-CFG now improves all 13 models, by 1.0 to 4.3 points, and is the strongest of five decoders on 12 (Tables 3 and 21).

  2. The gain comes from the image. Replacing the image-free reference with an average over random images removes most of the reference bias and keeps the gain (Section 4.2.1). On PhD, a shift toward "yes" explains at most 2.9 of the 7.1 point gain, and 85% of the answers guidance changes become correct (Appendix A.6).

  3. Fair tuning. Every baseline receives the same scale sweep as ours, with the best setting picked on the test results. VLM-CFG at its fixed α = 1.5 still beats all of them except PAI on one model, by 0.32, and it is the only method above unguided decoding at every scale (Table 22).

  4. Measured cost. End-to-end latency replaces the forward-pass estimate. On short answers VLM-CFG costs 1.07 to 1.24 times unguided decoding, against about 2 times for VCD and SID, and it is faster than PAI, VCD and SID on all eight benchmarks we timed (Table 19).

  5. Robustness. Four long-answer benchmarks extend the reference analysis: SWITCH is the best rule overall and the only one that never falls below unguided decoding (Table 6). Under nucleus and pure sampling, VLM-CFG remains the strongest decoder, and its gain grows as sampling becomes noisier (Appendix A.7).


We are grateful for the reviewers' time and suggestions, which have made the paper clearer and more complete.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.