acceptodds
Under review as a conference paper at ICLR 2027

Erasure or Silence? A Placebo-Controlled Evaluation of Training-Free Hallucination Mitigation in Vision-Language Models

Abstract

Hallucinating objects has been a problem for LVLMs since the beginning. Recent advances in LVLMs have introduced notable hallucination-reduction methods: some introduce training-free hallucination mitigation by intervening in the weight space, and others introduce decoding-time interventions. However, most of these hallucination mitigation frameworks evaluate their success using evaluation metrics such as CHAIR and POPE scores, which measure the level of hallucination based on the number of claims made by the model, not on the ground truth. But the model can manipulate this hallucination score just by generating less. In this paper, we are introducing (Suppression-Corrected Assessment of Reduction), which proposes a placebo-controlled frontier for the unedited model to use as a baseline for their hallucination reduction methods, to check that their mitigation gain is not just coming from the verbosity cheat of the model output. Our intuition is that if the hallucination mitigation method does not improve hallucination risk at the same coverage compared to the results that can be achieved by mere dummy changes, then the method was not worth it in the first place. Our frontier shows how the risk (hallucination score, in this case ) can be achieved in different coverage () levels, just by changing the placebo controls (generation length, beam width, temperature, a logit bias on object words), interventions that do not add any visual info, just control how much the model asserts.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, object hallucination, hallucination mitigation, evaluation methodology, CHAIR, POPE, training-free intervention, placebo control, risk-coverage trade-off, benchmark auditing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4qf5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether performance improvements in training-free hallucination mitigation methods truly stem from a reduction in hallucinations or can be achieved by altering the model’s generative behavior. The authors propose SCAR, which constructs a placebo frontier by applying interventions to the original model that do not introduce additional visual information, and evaluates hallucination mitigation methods under the same object coverage. The authors extend this framework to discriminative tasks, evaluating it based on AUROC and threshold-corrected accuracy gains. Experiments show that the object hallucination reduction reported by different methods can generally be matched by placebo interventions under the same coverage.

Critical strengths
  1. This manuscript approaches the research from an evaluative perspective, arguing that if a model reduces the number of correct object mentions, a decrease in the CHAIR score does not necessarily indicate improved visual comprehension. This is a reasonable research perspective.
  2. The authors conducted sufficient experiments to demonstrate the validity and soundness of the proposed insight.
Critical weaknesses
  1. There are still some methodological details that require further discussion; please see the “Questions” section for more details.
  2. The manuscript’s paragraph structure and writing style could be further optimized.
Questions for authors
  1. In Section 3.1, before Equations (1)–(2), the authors write: “Dropping ground-truth objects thus does not affect the hallucination metric.” However, based on the authors’ definition in Equation (2), this statement may not hold true or may be ambiguous. Please clarify the intended meaning here.

  2. In Section 3.1, the authors define operations that do not introduce additional visual information as “placebo,” including beam search, temperature, object-token bias, and random projection. Subsequently, in Section 3.3, Equations (3)–(5), the authors further use the placebo frontier to estimate how much of the gain stems from suppression. I would like to know why “not introducing new visual information” implies “only changing suppression without improving visual grounding”? For example, changing the decoding strategy or model weights—while not introducing additional visual information—could very well affect the model’s efficiency in utilizing existing visual evidence.

  3. In Section 3.3, the authors define the null band using the maximum leave-one-out SCAR score for each placebo configuration. Why can the maximum leave-one-out score serve as a statistical threshold for determining “whether a true reduction in hallucinations exists”? This metric may depend heavily on the sampling density of the placebo grid. Suppose a particular placebo configuration is especially important on the frontier; if it is removed, the frontier formed by the remaining configurations may deteriorate significantly, resulting in a higher leave-one-out score for that configuration. However, this does not necessarily mean that other methods must exceed the same margin to be considered an effective improvement. Therefore, I ask the authors to clarify: Is $ B_{\mathrm{LOO}}$ actually a statistical threshold with an explicit false-positive control, or is it an empirical tolerance for the sparsity of the placebo grid? This may directly affect the most important conclusion in the paper: “None of the evaluated methods clears the null band.”

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4qf5 (1/2)

Thank you for three precise questions. Each pointed at a real flaw, and each is corrected in the revised PDF (major changes in blue). The conclusion you flagged in Q3 holds without the band: none of the 25 re-scored configurations has an S interval above 0, and 12 have intervals below 0. The revised passages are quoted below so that they can be judged here.

Your pointStatusWhat changedWhere
Q1: "Dropping ground-truth objects thus does not affect the hallucination metric" conflicts with Eq. (2)FixedReplaced by an exact statement of how omission moves CHAIRᵢ§3.1
Q2: "no new visual information" does not imply "only suppression"Agreed, rewrittenMechanism claims removed. S is a comparison with a specified control family, not a test of suppression§1, §3.1, §3.3
Q3: is the max-LOO band a calibrated threshold or a grid-sparsity tolerance?Clarified, and measuredIt is an empirical tolerance. Verdicts now come from intervals, and what those intervals can detect is measured§3.3, §5, App. G
W2: structure and writingRevisedQuantities are defined before claims; informal wording is replacedAbstract, §1–§3, §6

Q1. Fixed. You are right. Under Eq. (2), dropping a correct mention shrinks only the denominator, so it raises CHAIRᵢ whenever a hallucinated mention remains (H/(M−1) > H/M for H > 0). What we meant is that CHAIRᵢ has no term for ground-truth objects a caption never mentions. §3.1 now reads:

CHAIRᵢ is the fraction of object mentions that are hallucinated over the total number of object mentions. It is normalised by the objects a caption mentions, so it has no term for ground-truth objects the caption leaves out, and omission moves it only through its denominator. Dropping a correct mention alone raises CHAIRᵢ when H > 0. [...] Coverage (Eq. 1) is the quantity that records this loss.

Eqs. (1)–(2) are unchanged.

Q2. Agreed, rewritten. It does not imply it. A decoding change or a random weight fold can change how well the model uses the visual evidence already in its input, as you say. "Content-free" now describes only what an intervention is given (§3.1):

A content-free control is an intervention on the unedited model that uses only the model's own weights and its ordinary input (image and prompt). It uses no annotations, detectors, retrieved evidence, trained scorers, or other information about which objects are present. A control may change how effectively the model uses the visual evidence already in its input; SCAR does not assume otherwise.

and §3.3 states what the score can and cannot support:

S(M) > 0 means that M reaches lower CHAIRᵢ than every operating point of the content-free control family, and every mixture of them, at M's coverage; S(M) ≤ 0 means that some such operating point matches M's risk there. S therefore measures the incremental benefit of M over re-operating the unedited model within this family, with no extra information. The family is specified, not exhaustive. S does not identify a mechanism: S ≤ 0 does not show that M "only suppresses", and S > 0 does not show that M improves grounding.

So if a control improves grounding, that improvement is part of the reference, and a method is credited for what it adds beyond it. Eqs. (3)–(5) are unchanged. σ is renamed from "suppression share" to "placebo-matched share", and "verbosity cheat" and "not worth it" are removed from the abstract.

A direct check of what the score responds to (new; 500 images, 2,000 paired resamples). We biased object words down using the image's annotations, then repeated the run with degraded annotations:

annotations used (bias strength)hallucinated mentions removed [95% CI]S [95% CI]verdict
correct (δ = −4)71.4% [59.6, 81.3]+0.674 [+0.532, +0.790]above
75% correct (δ = −4)62.4% [48.2, 73.8]+0.406 [+0.111, +0.575]above
50% correct (δ = −4)65.4% [52.9, 75.7]+0.242 [−0.243, +0.473]inconclusive
permuted (δ = −2)33.1% [12.8, 48.8]−0.407 [−1.030, −0.131]below

At fixed δ = −4 the point estimate of S falls as label accuracy falls, while the removed shares remain similar; the permuted arm, run at δ = −2, is below the frontier. The intervals of the three δ = −4 arms overlap and each arm is a single run, so this is an observed pattern, not a trend test. What it does establish is that removing hallucinated mentions is not enough to obtain a positive S.

(Part 2: Q3 and W2.)

AuthorsOpenReview ↗

Response to Reviewer 4qf5 (2/2)

Q3. It is an empirical tolerance for grid sparsity, not a threshold with false-positive control, and the main result no longer uses it.

The band is the largest leave-one-out S among the content-free cells. The case you describe is the one that occurs: on LLaVA/COCO a single isolated 32-token cell sets the hull band at +0.136, while the 95th percentile of the leave-one-out scores is +0.058. It also depends on grid density: thinning the grid at random from 25 to 8 cells widens its spread (App. G). These are properties of the designed grid, not a calibrated null distribution. §3.3 now says this, and "clears the band" remains only as a descriptive label.

The conclusion you flag is re-established without the band. Verdicts come from the 95% bootstrap interval of S: below the frontier if the interval lies below 0, above if it lies above 0, inconclusive otherwise. On the submitted audit (500 images), re-scored on the exact frontier of realizable mixtures:

Plane (VCD, PAI, ProjectAway, Nullu)belowinconclusiveabove
LLaVA-1.5-7B / COCO8100
MiniGPT-4 / COCO110
LLaVA-1.5-7B / AMBER220
MiniGPT-4 / AMBER100
All 25 configurations12130

(OPERA, DeCo and the MiniGPT-4 PAI and VCD rows could not be re-scored on this frontier and keep their submitted intervals; none lies above 0.) An inconclusive row is not read as evidence of absence, and each is reported with its interval: for PAI on LLaVA/COCO, S = +0.092 [−0.202, +0.281], and the upper end of its ε interval is +1.36 points against a 1.95-point reduction. These intervals describe image-sampling uncertainty for a fixed control family; they do not adjust for the choice of family, which is why the family is specified and released.

What these intervals can detect: measured. We injected annotation-derived gains of graded size into the unedited model and scored each exactly like a method (500 images; ε and S on the same frontier):

injected armε, CHAIRᵢ points [95% CI]S [95% CI]verdict
oracle, δ = −0.5+0.50 [−0.40, +1.62]+0.094 [−0.082, +0.268]inconclusive
oracle, δ = −1+1.93 [+0.78, +3.23]+0.359 [+0.169, +0.524]above
oracle, δ = −2+3.22 [+1.89, +4.73]+0.570 [+0.406, +0.691]above

On these runs the 1.93-point rung is resolved and the 0.50-point rung is not; PAI's reduction is 1.95 points. On these runs the test returned "above" at 500 images for the 1.93-point rung; an inconclusive row is reported with its interval and is not read as a zero. Each rung is a single run, so this brackets what 500 images resolve and is not a power curve; §5 states the limit.

The same test returns positive verdicts for published methods (new). MARINE, which adds detector evidence, is above the frontier at its published setting on our caption prompt (500 images): S = +0.364 [+0.118, +0.459]. (On its own short-caption prompt its coverage exceeds every control, so we leave it unscored rather than extrapolate.) PAI's published setting is above the frontier at 5,000 images: ε = +0.45 points [+0.04, +0.85], S = +0.085 [+0.009, +0.156], which is 35% of its 1.28-point reduction on those images. PAI and controls use different attention code, which alone moved CHAIRᵢ by −0.18 points at 500 images. The frontier spans a narrow range with six cells and has no matched-code reference at this size; the lower bound is close to 0. The whole margin cannot be attributed to PAI's intervention. Neither result clears its plane's band, so the band criterion you questioned would have registered neither. Submitted result values are retained; corrected mixture scores are reported separately. The abstract and conclusion keep the audit's result and report these two positive verdicts next to it.

W2. Revised. §3 defines coverage, risk, content-free control, the exact mixture frontier, S and the three verdicts before the first result. The pre-registration paragraph moves to the appendix. The abstract's informal phrases are replaced and it states the results.

In terms of your assessment. Soundness: you rated the concerns as minor or fixable; Q1–Q3 are each fixed in the paper, and the conclusion Q3 questioned now rests on intervals whose sensitivity is measured. Presentation: definitions precede claims, and the revised passages are quoted above. Informativeness: the paper reports what the test resolves at 500 images, what it does not, and that it credits published methods when there is a gain to credit.

If these changes address your concerns, we would be grateful if you would consider updating your rating.

Reviewer_gtedOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SCAR, a placebo-controlled evaluation protocol for training-free hallucination mitigation. It compares methods against a coverage–risk frontier constructed from content-free interventions on the unedited model. For POPE, it separates discrimination improvements from changes in yes/no decision bias. Experiments audit six methods, primarily on LLaVA-1.5-7B, with extensions to MiniGPT-4 and AMBER.

Critical strengths
  1. An important evaluation question. Distinguishing improved visual grounding from conservative generation is consequential. Comparing hallucination risk at matched coverage provides a stronger reference than reporting CHAIR improvements alone.
  2. Useful controls and complementary analyses. The POPE re-thresholding analysis, oracle/permuted controls, and robustness checks support the protocol’s usefulness. The finding that simple baseline adjustments reproduce substantial reported gains is informative.
Critical weaknesses
  1. Limited empirical scope for the broader conclusions. The evaluation primarily uses two older 7B models and object-level metrics. Some transferred configurations lack model-specific published settings. Testing stronger contemporary models and practical evidence-enhancing methods would better establish whether the findings generalize.
  2. The frontier comparison does not establish a suppression mechanism. Failure to outperform the selected placebo frontier shows limited incremental benefit under this protocol, but does not establish that a method merely suppresses claims or fails to improve visual representations. Matching aggregate object coverage also leaves differences in object identity, difficulty, attributes, and relations uncontrolled. Moreover, placebo configurations are selected from a sweep, while most methods are evaluated at fixed operating points, leaving the attainable frontiers of the methods incompletely characterized.
Questions for authors
  1. Do the conclusions hold on stronger contemporary VLMs and methods that explicitly enhance visual evidence?
  2. How do mitigation methods compare when their decoding parameters are swept using the same search budget as the placebo family?
  3. Does the conclusion remain unchanged when matching coverage by object category or difficulty rather than only aggregate recall?
2 comments
AuthorsOpenReview ↗

Response to Reviewer gted (1/2)

Thank you. Your three questions each asked for an experiment; all three are done (§5, App. G). In short:

  • Q1. MARINE, a published method that adds detector evidence, is above the frontier. On Qwen3-VL-8B and Gemma-4-E4B the protocol detects the annotation oracle, and VCD is inconclusive on both.
  • Q2. Swept over the controls' own nine settings, PAI improves on the content-free frontier at lower coverage, and its best setting keeps a gain on held-out images. Nullu and VCD, swept identically, are inconclusive.
  • Q3. With coverage matched inside object-frequency and difficulty strata, no tested LLaVA/COCO method row has an S interval wholly above 0 within a stratum.

We also agree with W2: the comparison does not establish a suppression mechanism, and the paper no longer says it does. Major changes are in blue in the revised PDF.

Your pointStatusEvidenceWhere
W1/Q1: contemporary models; evidence-enhancing methodsMeasuredMARINE above the frontier; Qwen3-VL-8B and Gemma-4-E4B with their own frontiers§5, App. G
W1: transferred settingsLabelledPublished settings, transfers and adaptations are distinguishedTab. 2 and Tab. 28 (App. G)
W2: no suppression mechanismAgreed, rewrittenS is a comparison with a specified control family§3.1, §3.3
W2/Q2: controls swept, methods fixedMeasuredSame nine settings for PAI, Nullu and VCD; held-out test§5, App. G
W2/Q3: object identity and difficultyMeasuredPer-stratum frontiersApp. G

W2 (mechanism). Agreed, rewritten. You are right: failing to outperform the frontier shows limited incremental benefit under this protocol, not that a method merely suppresses claims. Controls may improve use of existing evidence. §3.3 reads:

S(M) > 0 means that M reaches lower CHAIRᵢ than every operating point of the content-free control family, and every mixture of them, at M's coverage; S(M) ≤ 0 means that some such operating point matches M's risk there. S therefore measures the incremental benefit of M over re-operating the unedited model within this family, with no extra information. The family is specified, not exhaustive. S does not identify a mechanism: S ≤ 0 does not show that M "only suppresses", and S > 0 does not show that M improves grounding.

W2/Q2 (sweep symmetry). Measured. PAI (α = 0.5), Nullu (k = 4) and VCD (α = 1) were each run over the controls' nine settings on LLaVA-1.5-7B (base: beam 3, 64 tokens; then one knob at a time: object-word bias δ = −0.5, −1, −2; beam 1, 2, 5; 32 or 128 tokens; 500 images, 2,000 paired resamples). Choosing the best of nine noisy settings favours whatever is swept: for scale, one content-free grid swept against another reaches a best-setting S of +0.043 [+0.005, +0.146]. Under a rule fixed before any method sweep was scored, each method is therefore also compared with a second content-free grid, swept the same way against the same reference and paired in every resample (the "excess" column):

swept methodbest settingbest S, selection-aware 95% CIexcess over a swept content-free gridverdict
PAI (controls run through PAI's own code)with δ = −0.5+0.328 [+0.187, +0.489]+0.292 [+0.132, +0.450]above
Nullubeam 5+0.093 [−0.070, +0.353]+0.051 [−0.134, +0.300]inconclusive
VCDbeam 5−0.034 [−0.229, +0.227]−0.077 [−0.298, +0.166]inconclusive

PAI's gain comes from PAI combined with a mild object-word bias, at lower coverage: ε = +1.02 CHAIRᵢ points [+0.06, +1.85] at coverage 0.62 and +1.28 [+0.04, +2.05] at 0.66 (three pointwise readouts, not a simultaneous band; lower bounds near 0). At 0.70, near its published operating point, the gap is inconclusive (−0.01 [−1.10, +1.81]), as is the published setting itself against the same controls (S = +0.087 [−0.194, +0.287]). Fixed operating points understated what PAI can reach on this GPU. Nullu and VCD remain inconclusive over this grid.

Held-out test. Before the sweeps were scored we fixed a second rule: take each method's best setting against default-code controls on these 500 images and re-test it on 4,500 other images. PAI's is above the frontier there (S = +0.169 [+0.089, +0.241], ε = +0.84 points [+0.42, +1.26]). Nullu's is below it (S = −0.142 [−0.231, −0.055]), so its advantage on the selection images does not replicate. Limit: the held-out controls are six default-code settings bracketing these coverages, while PAI runs through its own attention code, a difference that alone moved CHAIRᵢ by −0.18 points at 500 images. This is therefore not a matched-code replication; the 500-image sweep in the table is matched.

(Part 2: Q1, Q3 and the paper's conclusion.)

AuthorsOpenReview ↗

Response to Reviewer gted (2/2)

W1/Q1 (methods that add visual evidence). Measured. MARINE at its published setting (γ = 0.7, greedy, DETR + RAM++ object lists) is above the content-free frontier on our caption prompt: S = +0.364 [+0.118, +0.459]. The same detector lists, used only to rank the candidates of a fixed search, are also above the frontier (S = +0.589 [+0.400, +0.714]); that search ranked by the model's own likelihood is inconclusive (+0.040 [−0.257, +0.253]). On MARINE's own short-caption prompt its coverage exceeds every control, so it is unscored; PAI, VCD and Nullu there are inconclusive with wide intervals. For Q1, MARINE is above the frontier on our prompt; the original audit's verdicts do not extend to this evidence-adding method.

W1/Q1 (contemporary models). Measured. Qwen3-VL-8B-Instruct and Gemma-4-E4B, each with its own nine-setting content-free frontier on the same 500 images and prompt, with a base 64-token budget:

modelrowS [95% CI]verdict
Qwen3-VL-8Bannotation oracle (δ = −4)+0.630 [+0.375, +0.780]above
Qwen3-VL-8Bpermuted annotations (δ = −2)−0.267 [−0.698, −0.022]below
Qwen3-VL-8BVCD α = 1, swept; selection-aware CI−0.035 [−0.381, +0.433]inconclusive
Gemma-4-E4Bannotation oracle (δ = −4)+0.744 [+0.588, +0.883]above
Gemma-4-E4Bpermuted annotations (δ = −2)−0.142 [−0.711, +0.073]inconclusive
Gemma-4-E4BVCD α = 1, fixed−0.031 [−0.590, +0.254]inconclusive

On both models the protocol detects the oracle, and VCD is inconclusive on both. The permuted control is below on Qwen3-VL but not on Gemma, where its interval reaches 0, so that specificity check replicates on one of the two. Limits: only VCD could be ported to these models in the discussion period; both frontiers sit at lower coverage (0.3305–0.5804 and 0.2944–0.5058); and the Qwen sweep has no second content-free grid, so no excess is reported for it.

W1 (transferred settings). Labelled. Table 2 flags the MiniGPT-4 VCD and DeCo rows as LLaVA-1.5 settings, since none is published for MiniGPT-4. The Qwen grid evaluates adaptations, not a published model-specific default. New rows are scored against a frontier from their own GPU; VCD's published setting, whose noise draw depends on the GPU, is below the frontier on the first GPU and inconclusive on the second.

W2/Q3 (object category and difficulty). Measured. We built a separate frontier inside each object-frequency stratum (head, mid, tail) and each difficulty stratum (easy, medium, hard, by the unedited model's recall), so coverage is matched within the stratum. No tested LLaVA/COCO method row has an S interval wholly above 0 within a stratum, while the annotation oracle does in the head (S = +0.703 [+0.507, +0.814]), easy (+0.748 [+0.535, +0.885]) and hard (+0.544 [+0.184, +0.734]) strata. Limits: each stratum holds only 17–92 hallucinated mentions, so intervals are wide; these frontiers are convex, not exact-mixture; strata are coarse groups, not exact object identity; and attributes and relations are outside CHAIR's scope, which the paper lists as a limitation.

What this does to the paper's conclusion. The submitted audit stands at 500 images: of the 25 configurations re-scored on the exact frontier, 12 are below it, 13 are inconclusive and none is above (OPERA, DeCo and the MiniGPT-4 PAI and VCD rows keep their submitted intervals, none above 0). At 5,000 images PAI's published setting is above the frontier: ε = +0.45 points [+0.04, +0.85], S = +0.085 [+0.009, +0.156], which is 35% of its 1.28-point reduction there. The held-out code mismatch applies here too, with a narrow six-cell reference, the lower bound is close to 0, and no matched-code frontier exists at this scale, so the whole margin cannot be attributed to PAI's intervention. MARINE and both PAI comparisons have positive S intervals but do not clear the descriptive bands of their own pre-registered frontiers. Submitted result values are retained; mixture re-scores and new checks are reported separately (§5, App. G), and the abstract and conclusion report PAI and MARINE as positive verdicts next to the audit.

In terms of your assessment. Soundness: the concerns you rated minor or fixable are each addressed: the mechanism claim is replaced by a defined comparison, and sweep symmetry, scope and strata are measured with their limits. Presentation: §3 defines every quantity before use; new results are in one table in §5, with details in App. G. Informativeness: under identical sweeps the protocol credits one method family and leaves two inconclusive, and it credits a published evidence-adding method.

If these changes address your concerns, we would be grateful if you would consider updating your rating.

Reviewer_PSF9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper re-examines reported gains from training-free methods for reducing hallucinations in vision-language models. The main comparison asks whether a method improves over content-free controls built from the same base model at a similar level of object recall. The proposed SCAR score compares each method with a reference curve from a control family that includes decoding changes and random weight projections. The POPE experiments also check whether a method changes the ordering of yes/no scores or mainly shifts the decision threshold. The paper studies six methods on LLaVA-1.5-7B and repeats part of the study on MiniGPT-4, using COCO and AMBER. Under this evaluation, none of the tested published settings goes beyond the reference band, while a control that uses image annotations does.

Critical strengths

1- The paper focuses on an important evaluation issue: a method may reduce object hallucinations simply by mentioning fewer correct objects. SCAR considers both hallucination and coverage and compares each method with content-free controls built from the same base model. This makes trade-offs visible that CHAIR alone can hide. SCAR could be a complementary metric alongside recall and F1.

2- The POPE analysis complements existing findings by Mirage and gives more information than accuracy alone. The authors report AUROC and yes-answer rates. For the reported rows, they also compare each method with the unedited model after adjusting its threshold to match the method's yes rate. This helps show whether an improvement comes from better score ordering or mostly from a change in the tendency to answer yes.

3- The experiments cover six different mitigation methods. The control that uses image annotations and the permuted-annotation control are useful points of comparison. The bootstrap confidence intervals and extra ablations also make some of the uncertainty and sensitivity of the results easier to see.

Critical weaknesses

1- My main concern is whether the model's natural generation under different decoding settings can reasonably be treated as a placebo. A recent work, HaloProbe[1], uses a trained probe as an external score in beam search to favor captions with fewer predicted hallucinations. The model's standard decoding is left unchanged, although the probe changes which beam is kept. Its reported results provide an example in which better search and selection improve caption quality without modifying the model's internals. More generally, with a larger search budget and an accurate score that balances hallucination and coverage, one could select substantially better captions from the model's own greedy output. Therefore, core claims like "how much of the reported improvement is real and how much is the placebo effect" (line 104) require extra caution.

[1] Zohrabi et al., "HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models", ICML 2026.

2- The second concern is whether the analysis can detect small remaining improvements. For PAI on LLaVA/COCO, Table 5 reports an improvement of 0.44 CHAIR percentage points over the reference curve, with a 95% interval of [-0.86, 1.29]. This result is still inconclusive. Table 19 also suggests that only larger gains can be reliably detected in this setting. Therefore, detecting the larger effect of the annotation-based control does not show that the test is sensitive to smaller but realistic gains. The discussion should more clearly separate settings that are clearly below the reference curve from those with uncertain results, and it should better explain the choice of the null-band cutoff.

3- My third concern is how directly this evaluation explains the gains reported in the original papers. Most POPE results are based on the first-step logits for Yes and No, while OPERA is evaluated using generated text because it changes decoding (Table 3). Appendix C shows a fairly large difference between these first-token accuracies and the published generated-answer accuracies for MiniGPT-4. It is therefore unclear whether the conclusions would remain the same if all methods were tested using generated answers. For captioning, the controls are tested across several decoding settings, while the methods are mainly tested by changing the method setting under one shared decoding setup, apart from a few alternatives. The use of only two older 7B models and a limited set of prompts also makes it hard to know how well the results generalize.

4- The convex-hull reference curve may not represent mixtures that can actually be achieved when CHAIR is pooled over the dataset. Randomly choosing controls across images makes expected coverage change linearly, but pooled CHAIR weights the results by the number of object mentions. For the 32-token and unedited MiniGPT-4 controls in Table 12, matching Nullu's AMBER coverage gives about 9.31% CHAIR, compared with 8.30% from the straight line between the two controls. Appendix A reports its check of this approximation only for the original LLaVA/COCO scoring region. This example does not change Nullu's negative result, and Table 15 includes a sensitivity check using the piecewise-linear reference curve. Still, the paper should measure the effect on scores, null bands, and uncertainty in every setting using a curve that is consistent with the pooled counts.

Questions for authors

1- Table 8, footnote b, says that the saved Nullu k=8 weight difference has rank 4 and matches the k=4 fold, but Table 4 reports different results. Which checkpoints were used for generation and for the operator analysis? Can the authors provide their measured edit ranks?

2- PAI with alpha=0 changes 18 of 100 captions (Table 5). How much do CHAIR and coverage change on the full evaluation set? If the controls are run through the same PAI code with alpha=0, does PAI at alpha=0.5 still improve over the reference curve?

3- How do these methods compare with SCAR's reference curve on newer vision-language models, such as Qwen3-VL and Gemma 4?

4 comments
AuthorsOpenReview ↗

Response to Reviewer PSF9 (1/3)

Thank you for a review that engages with the method's equations. All seven points are answered below, each with a correction or a new measurement. Submitted result values are retained; mixture re-scores and new runs are reported separately. In short:

  • W1. Search with an informative scorer is not a placebo, and the paper no longer implies it: such a scorer is evaluated as a method. One fixed search is inconclusive when ranked by the model's likelihood and above the frontier when ranked by a detector.
  • W2. Below-frontier and inconclusive rows are separated, no conclusion uses the band, and we measured what the intervals resolve at 500 and at 5,000 images.
  • W3. POPE is re-run with generated answers; PAI, Nullu and VCD are swept over the controls' grid; a second prompt and two newer models are added.
  • W4. You are right. The frontier is now exact for pooled counts wherever captions are available, and no verdict turns positive.
Your pointStatusKey evidence (500 images, 2,000 paired resamples unless stated)
W1: is decoding a placebo? (HaloProbe)Clarified, and measuredSame search: likelihood S = +0.040 [−0.257, +0.253]; detector +0.589 [+0.400, +0.714]
W2: small gains; below vs uncertain; bandMeasured, restructured25 re-scored rows: 12 below, 13 inconclusive, 0 above. PAI at 5,000 images: ε = +0.45 [+0.04, +0.85]
W3: generated POPE; fixed points; models; promptsMeasuredGenerated POPE: no verdict more favourable to a method. Sweeps: PAI above, Nullu and VCD inconclusive. Second prompt, two newer models
W4: hull not realizableFixedExact mixture frontier; your example reproduces (8.30% vs 9.17%)
Q1: Nullu ranksFixed, and measuredOriginal k = 4 checkpoint: rank 4 in all 16 edited layers; rebuilt k = 1, 2, 8: ranks 1, 2, 8
Q2: PAI α = 0 on the full setMeasured−0.18 points; PAI re-scored against controls run through its own code
Q3: Qwen3-VL, Gemma 4MeasuredOracle above on both; VCD inconclusive on both

New runs use a second GPU, and each is scored against a frontier rebuilt on it. Major changes are marked in blue in the revised PDF. Tab. 4 is new, so later tables are numbered one higher than in the submission (your Table 8 is now Tab. 9).

W1. Clarified, and measured. You are right that better search and selection can improve captions without touching the model's internals. The revised definition (§3.1) draws the line at what an intervention is given, not at where it acts:

A content-free control is an intervention on the unedited model that uses only the model's own weights and its ordinary input (image and prompt). It uses no annotations, detectors, retrieved evidence, trained scorers, or other information about which objects are present. [...] Beam search ranked by the model's own likelihood is therefore a control, while the same search ranked by a trained probe, a detector or the annotations is evidence-informed selection, which SCAR scores as a method.

A HaloProbe-style scorer is therefore evaluated as a method, and §2 cites it beside MARINE. To test the distinction we held one search fixed (5 sampled candidates per 20-token chunk) and changed only the ranker:

rankercoverageCHAIRᵢS [95% CI]verdict
model likelihood0.69985.55%+0.040 [−0.257, +0.253]inconclusive
DETR + RAM++ object lists0.67672.04%+0.589 [+0.400, +0.714]above
annotations0.78451.52%+0.879 [+0.833, +0.918]above

For the distinction your HaloProbe example raises: the informed ranker is credited as a method (above the frontier), and likelihood-ranked search, a control, is inconclusive. Model-only procedures may also help: the control family is specified, not exhaustive (§3.3), and PAI, which uses only the model's own input, is above the frontier when swept (W3). The sentence you quote at l.104 (first contribution, §1) now reads: "how much of a reported CHAIRᵢ reduction exceeds what content-free operation of the unedited model reaches at the same coverage."

W4. Fixed. For a per-image mixture of controls i and j, expected coverage is linear in λ, but pooled risk is (λHᵢ + (1−λ)Hⱼ) / (λMᵢ + (1−λ)Mⱼ). §3.2 now defines the exact mixture frontier as the minimum of this ratio at matched coverage. A Charnes–Cooper transform gives a linear program with two equality constraints, so pairs suffice. Your example reproduces: hull 8.30%, best realizable pair 9.17%, your pair 9.31%. Effect on every plane: bands move from +0.136, +0.191, +0.262, +0.321 (hull) to +0.157, +0.214, +0.279, +0.438; Nullu k = 8 on MiniGPT-4/AMBER stays below (S = −0.278 [−0.518, −0.114]; hull −0.412); Nullu k = 2 and VCD α = 0.5 on LLaVA/COCO move from below to inconclusive; no row becomes positive. OPERA, DeCo and the MiniGPT-4 PAI and VCD rows were not re-scored and are identified as such.

AuthorsOpenReview ↗

Response to Reviewer PSF9 (2/3)

W2. Measured; below-frontier and inconclusive rows are separated.

(a) Verdicts. You are right that PAI's result is inconclusive, and the paper now says so. Verdicts come from the 95% bootstrap interval of S: below, inconclusive, above. Of the 25 configurations of the submitted audit that could be re-scored on the exact frontier, 12 are below, 13 are inconclusive and none is above. For PAI on LLaVA/COCO, S = +0.092 [−0.202, +0.281], and the upper end of its ε interval is +1.36 points against a 1.95-point reduction; this is an interval endpoint, not a bound on the true gain. The hull upper endpoint was +1.29 points, the figure you quote. The band is the largest leave-one-out S among the content-free cells, an empirical tolerance for grid sparsity that a single isolated cell sets on LLaVA/COCO (32-token budget, coverage 0.589). Its rank-test level of 1/24 would hold only under an exchangeability we do not assume. No conclusion uses it.

(b) Sensitivity. You are right that detecting the strong oracle shows nothing about smaller gains, so we measured it. Annotation-derived gains were injected into the unedited model and scored like a method (ε and S on the same frontier):

injected armε, CHAIRᵢ points [95% CI]S [95% CI]verdict
oracle δ = −0.5+0.50 [−0.40, +1.62]+0.094 [−0.082, +0.268]inconclusive
oracle δ = −1+1.93 [+0.78, +3.23]+0.359 [+0.169, +0.524]above
oracle δ = −4, 75% correct labels+1.81 [+0.41, +2.75]+0.406 [+0.111, +0.575]above
oracle δ = −4, 50% correct labels+0.89 [−0.64, +2.00]+0.242 [−0.243, +0.473]inconclusive
permuted labels δ = −2−1.62 [−3.31, −0.54]−0.407 [−1.030, −0.131]below

On these runs a 1.93-point margin is resolved and a 0.50-point margin is inconclusive; PAI's reduction is 1.95 points. Each arm is a single run; this is not a power curve. The permuted arm removes 33.1% [12.8, 48.8] of hallucinated mentions and is still below the frontier.

(c) More images. PAI α = 0.5 and Nullu k = 8 at 5,000 images, against a pre-registered exact-mixture frontier with a narrow span and six control settings:

row at 5,000 imagesS [95% CI]ε [95% CI]verdict
PAI α = 0.5+0.085 [+0.009, +0.156]+0.45 [+0.04, +0.85]above
Nullu k = 8−0.073 [−0.163, +0.003]−0.39 [−0.84, +0.01]inconclusive

The 500-image row you called inconclusive stays inconclusive. On 5,000 images PAI is above the frontier, by 35% of its 1.28-point reduction. Controls use default attention code and PAI its own, which alone moved CHAIRᵢ by −0.18 points at 500 images. No matched-code frontier exists at 5,000, so this margin does not isolate PAI's intervention. Its lower bound is close to 0, and it is inside its plane's band. For Nullu the upper end of the ε interval is +0.01 points, against +1.58 at 500 images. Its verdict sits at the cut-off (with three more control settings the interval is [−0.164, −0.000], below), and it uses the rebuilt edit of Q1.

W3(a). Methods at fixed points. Measured. PAI (α = 0.5), Nullu (k = 4) and VCD (α = 1) were each run over the controls' nine settings (beam 3 and 64 tokens, then one knob at a time: bias δ = −0.5, −1, −2; beam 1, 2, 5; 32 or 128 tokens). Choosing the best of nine noisy settings favours whatever is swept, so under a rule fixed before any method sweep was scored, each method is also compared with a second content-free grid swept the same way against the same reference ("excess"):

swept methodbest settingbest S, selection-aware CIexcess over a swept content-free gridverdict
PAI (controls through PAI's own code)with δ = −0.5+0.328 [+0.187, +0.489]+0.292 [+0.132, +0.450]above
Nullubeam 5+0.093 [−0.070, +0.353]+0.051 [−0.134, +0.300]inconclusive
VCDbeam 5−0.034 [−0.229, +0.227]−0.077 [−0.298, +0.166]inconclusive

PAI's gain comes with a mild object-word bias, at lower coverage: ε = +1.02 [+0.06, +1.85] at coverage 0.62 and +1.28 [+0.04, +2.05] at 0.66 (three pointwise readouts, not a simultaneous band; lower bounds near 0); at 0.70, near its published point, the gap is inconclusive (−0.01 [−1.10, +1.81]). Fixed operating points understated what PAI can reach; Nullu and VCD remain inconclusive. Under a second pre-fixed rule each method's best setting against default-code controls was re-tested on 4,500 disjoint images: PAI's is above the frontier (S = +0.169 [+0.089, +0.241], ε = +0.84 [+0.42, +1.26]; six bracketing default-code controls; same code mismatch and band status as in (c)), and Nullu's is below (S = −0.142 [−0.231, −0.055]), so its advantage does not replicate.

AuthorsOpenReview ↗

Response to Reviewer PSF9 (3/3)

W3(b). Generated POPE: no listed method has a more favourable verdict. For the listed configurations, we re-ran POPE with generated answers (LLaVA-1.5-7B, 9,000 questions, beam 3). Each method is compared with the unedited model re-thresholded to the method's own yes-rate; the matched-rate reference still uses the unedited model's first-token score, so this is not an entirely generated-answer calibration baseline.

methodgenerated accuracy (unedited: 84.73%)ε_acc, generated, points [95% CI]ε_acc, first-token, as submitted
PAI α = 0.584.18%+0.51 [+0.09, +0.92]+0.76 [+0.48, +1.09]
VCD α = 182.06%−1.11 [−1.61, −0.58]−1.18 [−1.67, −0.72]
Nullu k = 482.83%−0.76 [−1.26, −0.27]−0.04 [−0.11, +0.07]
Nullu k = 8 (rebuilt edit)84.40%−0.56 [−1.00, −0.13]+0.07 [−0.07, +0.18]

The last column is from the first GPU and the original checkpoints, so the two columns are shown side by side and not differenced. PAI keeps a positive residual although its raw generated accuracy is below the unedited model's; under a stricter answer parser that residual is inconclusive (+0.36 [−0.09, +0.76]) and the other three stay below. On MiniGPT-4 (your App. C point), Nullu (k = 8) raises generated accuracy from 51.19% to 54.16%, but its residual is −0.40 [−0.67, −0.11] (stricter parser: −0.64 [−1.02, −0.31]). That result is limited to our checkpoint and prompt and does not reproduce the published absolute accuracies.

W3(c). Prompts. MARINE at its published setting (γ = 0.7, greedy, DETR + RAM++ lists) is above the frontier on our caption prompt: S = +0.364 [+0.118, +0.459], inside its plane's band. On its own short-caption prompt its coverage exceeds every control, so we leave it unscored rather than extrapolate; PAI, VCD and Nullu on that prompt are inconclusive with wide intervals.

Q1. Fixed, and measured. You are right: footnote b (now Tab. 9) conflated the k = 4 fold of the operator analysis with the k = 8 generation label. Generation used one checkpoint per rank; the operator analysis used k = 4, and its lower-rank ρ values use directions within that fold. For W_edited − W_raw, the original k = 4 checkpoint has rank 4 in all 16 edited layers. Edits rebuilt for k = 1, 2 and 8 have their nominal rank in every layer, with smallest singular-value gaps of 1082×, 1557× and 1612×, and their largest per-layer statistic differs from the original edit logs by 4.58 × 10⁻⁴ (relative). The original k = 1, 2 and 8 checkpoints were not available for direct measurement, so those three are rebuild checks.

Q2. Measured on all 500 images. At α = 0, PAI's attention code leaves 422 captions unchanged; CHAIRᵢ moves from 6.35% to 6.18% (−0.18 points from unrounded counts) and coverage from 0.6901 to 0.6907. Our 100-image estimate (−0.04) understated this, and we correct it. PAI is therefore now scored against nine controls run through its own code. Your second question then has two answers: at its published setting the comparison is inconclusive (S = +0.087 [−0.194, +0.287]); run over the controls' grid, PAI is above that reference at the lower coverage points in W3a.

Q3. Measured. Each model has its own nine-setting frontier on the same 500 images and prompt, with a 64-token base budget; entries are S [95% CI].

modelannotation oracle (δ = −4)permuted annotations (δ = −2)VCD α = 1
Qwen3-VL-8B+0.630 [+0.375, +0.780], above−0.267 [−0.698, −0.022], belowswept: −0.035 [−0.381, +0.433], inconclusive
Gemma-4-E4B+0.744 [+0.588, +0.883], above−0.142 [−0.711, +0.073], inconclusivefixed: −0.031 [−0.590, +0.254], inconclusive

On both models the test detects the oracle, and VCD is inconclusive. Limits: VCD is the only published method we could port in the discussion period; the permuted control is below on Qwen3-VL only; the Qwen sweep has no second content-free grid; and both frontiers sit at lower coverage (0.3305–0.5804 and 0.2944–0.5058). VCD's published setting on LLaVA-1.5 is below the frontier on the first GPU and inconclusive on the second, since its noise draw depends on the GPU.

Assessment. Soundness: you rated the concerns as minor or fixable; W4 and Q1 are fixed, Q2 is corrected on the full set, and W1–W3 are each answered by a measurement with its limits stated. Presentation: quantities are defined before use, and uncertain rows are no longer described as negative. Informativeness: the protocol credits an informed ranker, MARINE and a swept PAI, next to a submitted audit that stands at 500 images; the abstract and conclusion report both.

If these changes address your concerns, we would be grateful if you would consider updating your rating.

Thank you for the response and additional experiments. I now find the paper more convincing as a complementary evaluation framework and lean toward acceptance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions and new results (all reviewers)

We thank the reviewers. Without the null band, no configuration in the submitted audit has an S interval above 0. The revision corrects S's interpretation and pooled-count mixtures; new runs test sensitivity and scope (§5, App. G; major changes in blue).

ConcernRaised byResponse
InterpretationPSF9 W1, gted W2, 4qf5 Q2Agreed, rewritten. S is a comparison with a specified, non-exhaustive control family; mechanism language is removed. A scorer that carries image information is evaluated as a method: one fixed search is inconclusive when ranked by the model's likelihood (S = +0.040 [−0.257, +0.253]) and above the frontier when ranked by a detector (+0.589 [+0.400, +0.714])
Band and sensitivityPSF9 W2, 4qf5 Q3Clarified, and measured. Verdicts (below, inconclusive, above) come from 95% bootstrap intervals of S; the band is a descriptive label. Of two injected gains at 500 images, a 1.93-point margin [+0.78, +3.23] is resolved and a 0.50-point margin [−0.40, +1.62] is not (single runs, not a power curve)
Pooled-count mixturesPSF9 W4Fixed. Exact-mixture re-score of available rows on all four planes; the reviewer's example reproduces; two rows move from below to inconclusive and none becomes positive
Equal configuration gridsgted Q2, PSF9 W3Measured. PAI, Nullu and VCD run over the controls' nine settings, each also compared with a second swept content-free grid. PAI is above; Nullu and VCD are inconclusive
Scope and calibrationgted W1/Q1/Q3, PSF9 W3/Q3Measured. MARINE; Qwen3-VL-8B and Gemma-4-E4B with their own frontiers (oracle detected on both, VCD inconclusive on both); a second prompt; POPE with generated answers; per-stratum frontiers
ProvenancePSF9 Q1, Q2Fixed, and measured. Edit ranks measured (original k = 4 checkpoint; rebuilt k = 1, 2, 8); PAI's α = 0 drift measured on all 500 images (−0.18 points); PAI's n = 500 sweep uses matched-code controls
Presentationall threeRevised. Quantities are defined before claims; inconclusive rows are no longer described as negative; the §3.1 sentence flagged by 4qf5 is corrected

What this does to the paper's conclusion. The submitted audit stands at 500 images: of the 25 configurations re-scored on the exact frontier, 12 are below it, 13 are inconclusive and none is above (OPERA, DeCo and the MiniGPT-4 PAI and VCD rows keep their submitted intervals, none above 0). An inconclusive row is not read as evidence of absence; the single-run check above illustrates sensitivity on these images. The abstract and conclusion report these positives next to the audit:

New positive resultValue [95% CI]Limits
MARINE, published setting, our caption prompt, 500 imagesS = +0.364 [+0.118, +0.459]Unscored on its own short-caption prompt, where its coverage exceeds every control
PAI sweep with matched-code controls, 500 imagesselection-aware best S = +0.328 [+0.187, +0.489]; excess over a swept content-free grid +0.292 [+0.132, +0.450]One GPU; PAI combined with mild object-word bias at lower coverage; three pointwise gaps, not a simultaneous band; lower bounds near 0; its published setting stays inconclusive
PAI's best swept setting, 4,500 held-out imagesε = +0.84 points [+0.42, +1.26]Six bracketing default-code controls against PAI's own code; not a matched-code replication
PAI's published setting, 5,000 imagesε = +0.45 points [+0.04, +0.85], 35% of its 1.28-point reductionSame code mismatch (the code path alone moved CHAIRᵢ by −0.18 points at 500 images); narrow six-cell frontier; lower bound near 0

MARINE and both large-sample PAI comparisons do not clear their own pre-registered descriptive bands, and neither large-sample PAI margin isolates the intervention. Nullu's selected setting falls below the frontier on held-out images.

Scope limits in the paper. On the two newer models only VCD could be ported, both frontiers sit at lower coverage under the 64-token budget, Qwen's sweep has no second content-free grid, and the permuted control is below on Qwen but inconclusive on Gemma. The LLaVA/COCO stratum checks use coarse groups with 17–92 hallucinated mentions each, on convex frontiers. For generated-answer POPE the reference uses first-token scores; PAI's residual is parser-sensitive and its raw accuracy is below the unedited model's. Each new comparison uses a frontier from its own GPU, and VCD's published setting is below the frontier on the first GPU and inconclusive on the second.

Submitted result values are retained; corrected mixture scores and new runs are reported separately. All three reviews rated the soundness concerns minor or fixable; each now has its correction or measurement in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.