- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether performance improvements in training-free hallucination mitigation methods truly stem from a reduction in hallucinations or can be achieved by altering the model’s generative behavior. The authors propose SCAR, which constructs a placebo frontier by applying interventions to the original model that do not introduce additional visual information, and evaluates hallucination mitigation methods under the same object coverage. The authors extend this framework to discriminative tasks, evaluating it based on AUROC and threshold-corrected accuracy gains. Experiments show that the object hallucination reduction reported by different methods can generally be matched by placebo interventions under the same coverage.
Critical strengths
- This manuscript approaches the research from an evaluative perspective, arguing that if a model reduces the number of correct object mentions, a decrease in the CHAIR score does not necessarily indicate improved visual comprehension. This is a reasonable research perspective.
- The authors conducted sufficient experiments to demonstrate the validity and soundness of the proposed insight.
Critical weaknesses
- There are still some methodological details that require further discussion; please see the “Questions” section for more details.
- The manuscript’s paragraph structure and writing style could be further optimized.
Questions for authors
-
In Section 3.1, before Equations (1)–(2), the authors write: “Dropping ground-truth objects thus does not affect the hallucination metric.” However, based on the authors’ definition in Equation (2), this statement may not hold true or may be ambiguous. Please clarify the intended meaning here.
-
In Section 3.1, the authors define operations that do not introduce additional visual information as “placebo,” including beam search, temperature, object-token bias, and random projection. Subsequently, in Section 3.3, Equations (3)–(5), the authors further use the placebo frontier to estimate how much of the gain stems from suppression. I would like to know why “not introducing new visual information” implies “only changing suppression without improving visual grounding”? For example, changing the decoding strategy or model weights—while not introducing additional visual information—could very well affect the model’s efficiency in utilizing existing visual evidence.
-
In Section 3.3, the authors define the null band using the maximum leave-one-out SCAR score for each placebo configuration. Why can the maximum leave-one-out score serve as a statistical threshold for determining “whether a true reduction in hallucinations exists”? This metric may depend heavily on the sampling density of the placebo grid. Suppose a particular placebo configuration is especially important on the frontier; if it is removed, the frontier formed by the remaining configurations may deteriorate significantly, resulting in a higher leave-one-out score for that configuration. However, this does not necessarily mean that other methods must exceed the same margin to be considered an effective improvement. Therefore, I ask the authors to clarify: Is $ B_{\mathrm{LOO}}$ actually a statistical threshold with an explicit false-positive control, or is it an empirical tolerance for the sparsity of the placebo grid? This may directly affect the most important conclusion in the paper: “None of the evaluated methods clears the null band.”
2 comments
Response to Reviewer 4qf5 (2/2)
Q3. It is an empirical tolerance for grid sparsity, not a threshold with false-positive control, and the main result no longer uses it.
The band is the largest leave-one-out S among the content-free cells. The case you describe is the one that occurs: on LLaVA/COCO a single isolated 32-token cell sets the hull band at +0.136, while the 95th percentile of the leave-one-out scores is +0.058. It also depends on grid density: thinning the grid at random from 25 to 8 cells widens its spread (App. G). These are properties of the designed grid, not a calibrated null distribution. §3.3 now says this, and "clears the band" remains only as a descriptive label.
The conclusion you flag is re-established without the band. Verdicts come from the 95% bootstrap interval of S: below the frontier if the interval lies below 0, above if it lies above 0, inconclusive otherwise. On the submitted audit (500 images), re-scored on the exact frontier of realizable mixtures:
| Plane (VCD, PAI, ProjectAway, Nullu) | below | inconclusive | above |
|---|---|---|---|
| LLaVA-1.5-7B / COCO | 8 | 10 | 0 |
| MiniGPT-4 / COCO | 1 | 1 | 0 |
| LLaVA-1.5-7B / AMBER | 2 | 2 | 0 |
| MiniGPT-4 / AMBER | 1 | 0 | 0 |
| All 25 configurations | 12 | 13 | 0 |
(OPERA, DeCo and the MiniGPT-4 PAI and VCD rows could not be re-scored on this frontier and keep their submitted intervals; none lies above 0.) An inconclusive row is not read as evidence of absence, and each is reported with its interval: for PAI on LLaVA/COCO, S = +0.092 [−0.202, +0.281], and the upper end of its ε interval is +1.36 points against a 1.95-point reduction. These intervals describe image-sampling uncertainty for a fixed control family; they do not adjust for the choice of family, which is why the family is specified and released.
What these intervals can detect: measured. We injected annotation-derived gains of graded size into the unedited model and scored each exactly like a method (500 images; ε and S on the same frontier):
| injected arm | ε, CHAIRᵢ points [95% CI] | S [95% CI] | verdict |
|---|---|---|---|
| oracle, δ = −0.5 | +0.50 [−0.40, +1.62] | +0.094 [−0.082, +0.268] | inconclusive |
| oracle, δ = −1 | +1.93 [+0.78, +3.23] | +0.359 [+0.169, +0.524] | above |
| oracle, δ = −2 | +3.22 [+1.89, +4.73] | +0.570 [+0.406, +0.691] | above |
On these runs the 1.93-point rung is resolved and the 0.50-point rung is not; PAI's reduction is 1.95 points. On these runs the test returned "above" at 500 images for the 1.93-point rung; an inconclusive row is reported with its interval and is not read as a zero. Each rung is a single run, so this brackets what 500 images resolve and is not a power curve; §5 states the limit.
The same test returns positive verdicts for published methods (new). MARINE, which adds detector evidence, is above the frontier at its published setting on our caption prompt (500 images): S = +0.364 [+0.118, +0.459]. (On its own short-caption prompt its coverage exceeds every control, so we leave it unscored rather than extrapolate.) PAI's published setting is above the frontier at 5,000 images: ε = +0.45 points [+0.04, +0.85], S = +0.085 [+0.009, +0.156], which is 35% of its 1.28-point reduction on those images. PAI and controls use different attention code, which alone moved CHAIRᵢ by −0.18 points at 500 images. The frontier spans a narrow range with six cells and has no matched-code reference at this size; the lower bound is close to 0. The whole margin cannot be attributed to PAI's intervention. Neither result clears its plane's band, so the band criterion you questioned would have registered neither. Submitted result values are retained; corrected mixture scores are reported separately. The abstract and conclusion keep the audit's result and report these two positive verdicts next to it.
W2. Revised. §3 defines coverage, risk, content-free control, the exact mixture frontier, S and the three verdicts before the first result. The pre-registration paragraph moves to the appendix. The abstract's informal phrases are replaced and it states the results.
In terms of your assessment. Soundness: you rated the concerns as minor or fixable; Q1–Q3 are each fixed in the paper, and the conclusion Q3 questioned now rests on intervals whose sensitivity is measured. Presentation: definitions precede claims, and the revised passages are quoted above. Informativeness: the paper reports what the test resolves at 500 images, what it does not, and that it credits published methods when there is a gain to credit.
If these changes address your concerns, we would be grateful if you would consider updating your rating.
Response to Reviewer 4qf5 (1/2)
Thank you for three precise questions. Each pointed at a real flaw, and each is corrected in the revised PDF (major changes in blue). The conclusion you flagged in Q3 holds without the band: none of the 25 re-scored configurations has an S interval above 0, and 12 have intervals below 0. The revised passages are quoted below so that they can be judged here.
Q1. Fixed. You are right. Under Eq. (2), dropping a correct mention shrinks only the denominator, so it raises CHAIRᵢ whenever a hallucinated mention remains (H/(M−1) > H/M for H > 0). What we meant is that CHAIRᵢ has no term for ground-truth objects a caption never mentions. §3.1 now reads:
Eqs. (1)–(2) are unchanged.
Q2. Agreed, rewritten. It does not imply it. A decoding change or a random weight fold can change how well the model uses the visual evidence already in its input, as you say. "Content-free" now describes only what an intervention is given (§3.1):
and §3.3 states what the score can and cannot support:
So if a control improves grounding, that improvement is part of the reference, and a method is credited for what it adds beyond it. Eqs. (3)–(5) are unchanged. σ is renamed from "suppression share" to "placebo-matched share", and "verbosity cheat" and "not worth it" are removed from the abstract.
A direct check of what the score responds to (new; 500 images, 2,000 paired resamples). We biased object words down using the image's annotations, then repeated the run with degraded annotations:
At fixed δ = −4 the point estimate of S falls as label accuracy falls, while the removed shares remain similar; the permuted arm, run at δ = −2, is below the frontier. The intervals of the three δ = −4 arms overlap and each arm is a single run, so this is an observed pattern, not a trend test. What it does establish is that removing hallucinated mentions is not enough to obtain a positive S.
(Part 2: Q3 and W2.)