acceptodds
Under review as a conference paper at ICLR 2027

MOON: Reducing Object Hallucination with Repair-Aware Visual Guidance

Abstract

Vision-language models often describe objects that are absent from an image. External visual guidance can correct these errors, but it can also introduce false mentions or remove correct content. We introduce MOON, a controller that learns when and how to guide a frozen vision-language model. Its central idea is to predict what guidance will change. Adding a mention repairs an omission when the object is present and causes damage when it is absent; deleting a mention reverses these roles. MOON combines these effects with estimates of object presence and the unguided model's behavior to choose an action before generation. Across three backbone updates, it reduces average weighted object loss over 8–128-image paired-audit budgets by 1.9–2.3% relative to a direct risk predictor trained on the same observations. Both methods also use a pool of unguided target responses. The advantage extends to prompt and dataset changes and narrows as paired supervision increases. Controlled comparisons support repair/damage structure as a useful inductive bias for adapting visual guidance with limited paired data.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, object hallucination, hallucination mitigation, visual grounding, adaptive visual guidance, low-data adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qFEMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes MOON, a repair-aware controller for reducing object hallucination in frozen vision-language models. Instead of always applying external visual guidance, MOON predicts whether a candidate guidance action is likely to improve or harm the final response. The method separates guidance effects into four cases: false addition, false deletion, true addition, and true deletion, and combines these with estimated object presence and the unguided model’s behavior to compute expected action risk before generation.
But the practical scope narrower than the framing suggests, and the method introduces substantial modeling and system complexity for relatively modest gains that are concentrated in a specific low-data regime.

Critical strengths

1. The paper studies an important practical failure mode of visual guidance.
External visual evidence can reduce hallucination, but incorrect guidance can also introduce false mentions or remove correct content. Modeling when guidance should be applied is therefore a meaningful problem.

2. The decomposition is well motivated.
Separating true additions, false additions, true deletions, and false deletions gives a way to reason about when guidance helps or hurts compared with directly predicting a single risk value.

3. The experimental controls are careful.
The paper compares against parameter-matched direct predictors, link-matched variants, signed recomposition, random partitioning, and learned combiners.

Critical weaknesses

1. The motivation is overclaimed.
The paper is framed as reducing object hallucination in vision-language models, but the actual setting is more specific: selecting among a small predefined set of external-guidance actions for a frozen model under limited paired supervision. This is narrower than the general hallucination-mitigation framing suggests.

2. The main conceptual contribution is relatively incremental.
The central observation that adding or deleting an object mention can either repair or damage the answer depending on object presence is natural once the problem is written as mention transitions relative to the Base model. Much of the contribution therefore lies in the parameterization and recomposition of these effects rather than in a fundamentally new formulation of hallucination mitigation.

3. The method is fairly complex relative to the size of the reported gain.
MOON requires multiple visual experts, calibrated object-presence estimates, Base-state estimators, several repair/damage factors, source priors, target Base responses, and paired guided responses. The main improvement over a matched direct-risk predictor is around 1.9–2.3% in low-budget AULC and becomes much smaller as more paired data are available.

4. The interpretation of the learned factors is weaker than the paper initially suggests.
Under risk-only training, the four factors are latent and are not uniquely identifiable as event probabilities. MOON-S improves factor calibration, but this does not yield a statistically resolved improvement in policy quality, suggesting that the decomposition may function mainly as a useful parameterization rather than as a strongly validated mechanistic model.

5. The strongest gains are concentrated in a narrow low-data regime.
The advantage is clearest with 8–128 paired target images and becomes practically equivalent to several more flexible predictors at 512 paired images. This significantly narrows the regime in which the method is clearly preferable.

6. External comparisons are limited compared with the extensive internal ablations.
The paper contains many carefully controlled comparisons among its own variants, but the main shared-evidence comparison with recent hallucination-mitigation methods is limited to LLaVA-1.5/COCO. A broader comparison across the evaluated backbones would better establish practical competitiveness.

7. The practical deployment story is not fully convincing.
The controller itself is cheap, but the full system still requires external experts and may invoke two decoder branches. Warm latency is notably above Base, and cold execution is dominated by expert loading. It is unclear when this additional system complexity is preferable to simply improving the visual evidence model or directly adapting the VLM.

8. The presentation is hard to follow and makes the central contribution harder to identify than necessary.
The paper introduces many overlapping variants and names, including IG, IG RT, IG+, MOON-R, MOON-S, Flat, Flat-σ, Signed-Recompose, FreeMix, anchor adaptation, and Joint CA. The appendix also contains a large number of comparison families and tables. The central message that structured factorization helps direct risk prediction in low-data adaptation could be presented much more simply.

Questions for authors

Q1. What is the intended practical deployment scenario where MOON’s added?
The main advantage appears in the low-paired-data regime, while deployment still requires multiple visual experts, fitted estimators, and sometimes two decoder branches. It would be useful to clarify when this setup is preferable to collecting more target data, improving the detector, or directly adapting the VLM.

Q2. How strongly should the factors (such as repair/damage) be interpreted as actual mechanisms rather than as a structured parameterization?
Since the risk-only factors are not identifiable as event probabilities, and explicit event supervision improves calibration without clearly improving the final policy, the main contribution may be better understood as an inductive bias for low-data risk prediction.

Q3. Can the authors simplify the method hierarchy and clearly state the recommended final configuration?
It is currently difficult to distinguish the main deployed method from analysis-only variants and transfer variants. A concise table separating essential components, optional components, and diagnostic variants would improve readability.

Q4. How practically meaningful is the reported low-budget gain?
The paper reports improvements in weighted loss and AULC, but it would be easier to interpret the benefit if the authors translated the gain into a concrete number of avoided false object mentions or corrected omissions per 100 images, together with the added runtime and supervision cost.

Reviewer_jCkrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MOON, a visual-guidance controller for frozen vision-language models that aims to reduce object hallucination without damaging correct content. It separately models object presence, unguided model mention behavior, and guidance-induced additions and deletions. These components are recomposed into expected excess risk relative to the unguided model, enabling intervention selection before generation.

The main contribution is to use a repair/damage decomposition as an inductive bias for risk prediction under limited paired supervision, together with an analysis of Base-only and single-anchor adaptation. Validation covers three backbone release updates, prompt changes, and cross-dataset transfer. Against a direct risk predictor matched in features, supervision, parameter count, and source-risk shrinkage, MOON-R reduces average weighted loss by 1.9–2.3% over paired budgets of 8–128 images. Structural ablations, human mention audits, native metrics, and downstream capability checks provide additional support, although the advantage decreases with more paired data.

Critical strengths
  1. Strong controlled evaluation of the central empirical claim. The paper goes beyond endpoint comparisons through prior-matched, parameter-matched, random-partition, link-matched, conditional-recomposition, and learned-combiner controls. These experiments make the claim that low-budget improvements arise from the structured readout, rather than additional supervision or source information, substantially more convincing. This is the primary reason supporting acceptance.

  2. Useful analysis of adaptation conditions and limitations. The framework separates evidence validity, Base behavior, and guidance effects, and explicitly establishes that presence and Base observations alone do not identify target guidance risks. The conditional-effect and anchor-contrast transport assumptions, together with measured natural-shift residuals, provide a useful account of when limited-supervision adaptation can work. This adds conceptual value beyond the empirical controller.

Critical weaknesses
  1. The practical benefit is modest and concentrated in the low-budget regime. The principal improvement over the matched direct predictor is a 1.9–2.3% relative reduction in weighted loss, and the methods are equivalent within the specified tolerance at 512 paired images. Small paired budgets also do not imply minimal overall supervision: the standard setting still uses 512 target Base images and shared source acquisition. In addition, the primary audit is restricted to 80 canonical object classes. These factors limit the direct implications for low-cost, open-vocabulary deployment.

  2. Attribution specifically to repair/damage structure remains incomplete. MOON-R fits four latent factors using risk supervision alone, so these factors need not represent the actual addition/deletion probabilities. The ablations support the structured readout, but do not fully exclude an explanation based primarily on context-dependent feasible risk ranges or other functional constraints, rather than repair/damage semantics. MOON-S improves event calibration, but its policy-loss improvement over MOON-R is not statistically established after correction. Consequently, the evidence is stronger for the structure as a predictive inductive bias than as a model of the underlying intervention mechanism.

Questions for authors
  1. Could the authors report the false-positive, false-negative, and recall changes associated with the low-budget improvements under the principal matched comparisons? How much of the AULC benefit comes from fewer false mentions, and how much is associated with changes in true-object coverage? This would help establish the practical significance of the relatively small weighted-loss improvement.

  2. Does the advantage over Flat remain when sample efficiency is measured by total acquisition budget rather than paired-image count alone? In particular, could the smaller-Base-pool results explicitly quantify the magnitude and stability of the advantage, beyond its significance direction

Reviewer_i978OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

MOON is a controller that decides, before generation, whether and how to use external visual-expert evidence (DETR, RAM++, Grounding DINO) to reduce object hallucination in frozen VLMs. Its key idea is to explicitly separate "repair" (fixing an omission) from "damage" (introducing a false mention) as distinct, estimable effects of each guidance action, combined with presence estimates into a risk-based action-selection rule. Across three backbone update pairs, MOON-R beats a matched direct-risk-predictor baseline (Flat) by 1.9–2.3% relative weighted loss in the 8–128 paired-image low-budget regime.

Critical strengths
  1. The repair/damage split isn't just a heuristic — it's backed by an actual error bound in the appendix
  2. The robustness checks go well beyond the core result which includes prompt changes, dataset transfer, different false-positive costs, smaller data budgets making the main claim much more convincing.
Critical weaknesses
  1. Only ILVAD and AFIP are empirically tested; the well-known inference-time decoding baselines named in related work (VCD, OPERA) are never run, nor are other common ones (DoLa, ICD, VISTA) — so MOON's advantage over this broader, directly comparable family of methods remains unverified.
  2. The paper needs an accuracy–inference-cost tradeoff analysis against other baselines including inference time intervention techniques.
  3. The paper's own ablation raises doubts about the external comparison: a much simpler internal baseline (Flat residual) nearly matches full MOON and already beats ILVAD/AFIP, suggesting the gain over prior work may come from the shared evidence setup rather than MOON's core contribution.
  4. All tested backbones sit in a narrow 7B–8B band; the paper gives no evidence on either end of the scale — smaller models (under ~7B), where guidance may matter more, or larger ones (13B+), where hallucination behavior and responsiveness to guidance may differ substantially.
Questions for authors

Please refer to the Weaknesses Section

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.