acceptodds
Withdrawn from ICLR 2027

LIFE: LOCAL INTRINSIC DIMENSION-GUIDED FEATURE EDITING FOR ONE-TOKEN, ONE-LAYER HALLUCINATION REPAIR IN VISION-LANGUAGE MODELS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large vision-language models (LVLMs) can hallucinate, producing fluent responses with details that the image does not support. Hallucination leaves traces in the model’s internal representations, but the decoder layer where these traces are clearest varies across models, benchmarks, and individual inputs, and existing methods inspect or edit a layer that is fixed in advance. We introduce **LIFE** , ocal ntrinsic Dimension-Guided eature diting, an inference-time framework that decides where to inspect, whether to intervene, and how little to change. **LIFE-Detect** uses intrinsic dimension to locate hallucination-sensitive layers without labels, at two scales: the population-level intrinsic dimension of calibration representations identifies a shared layer for each model and benchmark, and the local intrinsic dimension (LID) of an individual input selects a layer for that input. A small MLP probe at the selected layer decides whether the response will likely be a hallucinated one. **LIFE-Mitigate** then edits one hidden state, the representation of a single token at a single decoder layer, by moving it along a direction given by the probe with a step size chosen for that input, and regenerates the response. The LVLM is never fine-tuned and never back-propagated through; the only trained component is the probe, and the correction optimizes one scalar. Across three LVLM backbones and five benchmarks, LIFE reduces the AMBER hallucination rate from 21.3% to 18.8% on LLaVA-1.6 and from 14.2% to 11.0% on Qwen3-VL, and more than halves sentence-level CHAIR on all three backbones, at a per-sample cost dominated by regenerating the response.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-language models, hallucination detection, hallucination mitigation, local intrinsic dimensionality, inference-time intervention

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qUzEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper uses a model judge to label generated responses, trains layer-specific probes to detect hallucinations, and uses intrinsic dimension to select an intervention layer. For flagged responses, it edits one token’s hidden state along a probe-guided direction. The authors report results across three vision-language models and five benchmarks.

Critical strengths

None

Critical weaknesses
  1. The supervision and intervention are too coarse-grained. A response may contain both correct and hallucinated content, yet the probe receives only a response-level binary label and edits a single hidden state. The paper does not show that this intervention targets unsupported content while preserving correct content.

  2. The reliability of the automatically generated labels is insufficiently validated. GPT-5-mini can misjudge image content; systematic labeling errors could affect both the probe and its editing direction. A human-audited label assessment and a noise-sensitivity analysis are needed.

  3. The specificity of the learned signal is unclear. The probe may detect general errors, task difficulty, or distribution shift rather than hallucination-specific patterns. Comparisons with difficulty-matched non-hallucination errors and cross-task tests would help distinguish these explanations.

Questions for authors
  1. See weakness
  2. I think the paper’s formatting and writing need work. I hope the authors will revise it manually to reduce the signs of AI-generated writing.
Reviewer_Er38OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes LIFE, an inference-time method for reducing hallucination in LVLMs. It has two parts. LIFE-Detect picks a decoder layer in two ways: a dataset-level layer from the FEPoID intrinsic-dimension criterion of Wang et al. (2026), and a per-input layer given by the argmax of local intrinsic dimension (LID) across layers. An MLP probe trained on LLM-judge labels at that layer then decides whether the response is hallucinated. LIFE-Mitigate edits a single hidden state (the last token of the first generated sentence) at the selected layer. It moves the state along the negative normalized gradient of the probe, chooses the step size by optimizing one scalar, and then regenerates the rest of the response. Experiments use LLaVA-1.6-7B, Qwen3-VL-8B, and InternVL3.5-8B on AMBER, POPE, MME, HaloQuest, and CHAIR, plus LLaVA-1.5-7B on POPE-Random. The paper reports that the FEPoID layer is within two layers of the best-AUROC layer in all 12 model-benchmark pairs. It also reports lower AMBER hallucination rates (e.g., 21.3 to 18.8 on LLaVA-1.6), CHAIR_S more than halved on all three backbones, POPE gains over four inference-time baselines, and lower latency than VCD, VTI, and OPERA.

Critical strengths
  1. The paper checks that FEPoID, a label-free layer criterion proposed for text-only LLMs, also finds a near-best probe layer in three LVLMs. The FEPoID layer is within two layers of the best-AUROC layer in all 12 model-benchmark pairs (Fig. 3). This is a useful sanity check for people who want to probe LVLMs without tuning the layer on labeled data.

  2. The intervention itself is small and cheap: one hidden state, one scalar, and only on inputs the probe flags. The paper is also fairly honest about where the per-input rule does not help (Table 2, Appendix D, Limitations).

Critical weaknesses
  1. Most of the mitigation results seem to be measured on data the probe was trained on. Section 4.1 says the mitigation examples for AMBER, MME, HaloQuest, and CHAIR "were selected from the corresponding probe-training splits", and only POPE uses a separate set. For MME, for example, the probe pool has 2,375 labeled samples and 874 of them are used for mitigation. Since the probe both decides which inputs to edit and gives the edit direction, this could inflate the results a lot (e.g., CHAIR_S going from 58.8 to 10.4 on Qwen3-VL). Some numbers also don't line up. Table 4 shows only 9-16 of 1,004 AMBER outputs corrected for InternVL3.5, but Table 2 shows its hallucination rate falling from 22.43 to 14.2, which is about 80 responses. For Qwen3-VL on POPE, Table 4 implies 1,211 initially incorrect answers out of 4,500 (about 73% accuracy), while Table 3 reports 93.24% for the vanilla model. Until these points are clarified, I can't trust the main mitigation numbers.

  2. The per-input LID layer selection, which is the paper's main new idea and is in the title, is not shown to help. On AMBER every layer, including L/2 and the last layer, gives about the same result (Table 2). On MME and HaloQuest, the per-input layer is best in only 4 of 12 rows, its detection AUROC beats the fixed FEPoID layer in 6 of 12 settings (Table 10), and with QT representations it is worse than FEPoID for all three models (Table 9). The other components come from earlier work. FEPoID and first-sentence truncation are from Wang et al. (2026). Editing hidden states with a hallucination probe has been done before (e.g., TruthPrInt, ICCV 2025; Activation Steering Decoding, ACL 2025). Input-dependent layer selection for steering also exists (Where to Steer, 2026). None of these are cited. In addition, LIFE uses in-domain labels for every benchmark, while all the baselines are training-free, and there are no simple controls that would show the edit itself matters.

Questions for authors

Questions that could change my assessment:

  1. Are the mitigation examples for AMBER, MME, HaloQuest, and CHAIR part of the probe's training split? If so, could you re-run mitigation on examples the probe never saw during training?

  2. Could you explain the mismatch between Table 4 and Tables 2/3? This includes the InternVL3.5 AMBER counts, the POPE counts for Qwen3-VL and InternVL3.5, and why POPE has N = 4,500 in Table 4 but 27,000 questions in Section 4.1.

  3. Could you add three controls: regenerating the continuation without any edit, editing along a random direction with the same norm, and, for yes/no benchmarks, simply flipping the answer whenever the probe flags it? These would show how much of the gain comes from the edit rather than from the in-domain probe.

Clarifications:

  1. On POPE and MME, the answer is usually a single word. With FST, the first sentence is kept and only the continuation is regenerated, so how does the yes/no answer change? Is a different token position used for these benchmarks?

  2. Do the gains hold when the probe is trained on one benchmark and applied to another (e.g., trained on AMBER, tested on HaloQuest)? Without this, the method needs new labels for every benchmark.

Suggestion (doesn't affect my score):

Please cite and discuss TruthPrInt, Activation Steering Decoding, Where to Steer, and ITI, and compare with at least one probe-based steering baseline that uses the same labels.

Reviewer_Q7CKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses visual hallucination in large vision-language models (LVLMs) with an inference-time method called LIFE. The authors argue that the decoder layer where hallucination is most visible differs across models, benchmarks and inputs, while existing representation-editing methods use a fixed layer and edit every input. LIFE has two parts. LIFE-Detect uses intrinsic dimension to choose a layer without labels, at two scales: the population-level intrinsic dimension of a calibration set gives one layer per model and benchmark (using the FEPoID criterion), and the local intrinsic dimension (LID) of each input selects a layer for that input. A small MLP probe at the selected layer then decides whether the response is likely hallucinated. LIFE-Mitigate runs only on flagged inputs. It edits a single hidden state, the last token of the first generated sentence at the selected layer, along the probe's gradient direction, optimizes only the step size, and regenerates the rest of the response. The LVLM is never fine-tuned or back-propagated through. Experiments on LLaVA-1.6, Qwen3-VL and InternVL3.5 across AMBER, POPE, MME, HaloQuest and CHAIR show lower hallucination than the base models and several inference-time baselines, e.g., AMBER hallucination rate falling from 21.3% to 18.8% on LLaVA-1.6 and sentence-level CHAIR more than halving on all three models. The paper also reports a comparison with VCD, VTI and OPERA on LLaVA-1.5, latency measurements, query-token experiments and sensitivity to the neighborhood size.

Critical strengths
  1. Label-free layer localization works well empirically. In all twelve model–benchmark pairs, the FEPoID layer is within two layers of the best-AUROC layer, and within one layer in eight (Sec. 4.2, Figure 3). Its detection AUROC is close to the best fixed layer, which needs labels to choose (Table 10). This is a useful, cheap way to choose where to probe a new LVLM.

  2. A small, well-defined intervention. LIFE edits one hidden state, optimizes one scalar and leaves the LVLM untouched (Sec. 3.3). The cost analysis is clear: localization and correction cost about 4% of one decoding step, and the main overhead is regeneration (Figure 5). The paper is also open about its limitations, e.g., that the per-input rule is not uniformly better than the fixed layer (Table 2, Table 9, Limitations).

Critical weaknesses
  1. Sec. 4.1 and Appendix A.1 state that the mitigation examples for AMBER, MME, HaloQuest and CHAIR "were selected from the corresponding probe-training splits". The probe decides which inputs to edit and also gives the edit direction, so on these four benchmarks it has already seen the labels of the examples it corrects. This likely inflates the mitigation results a lot. For example, MME gains of up to 9 points (Table 2) imply almost no correct-to-incorrect flips, even though the probe's held-out AUROC on MME is only 84–87% (Table 10). POPE is the only benchmark with a separate mitigation set, and there the gains are much smaller (e.g., 91.57 → 92.59 F1 on LLaVA-1.6, Table 3). Since this affects most of the headline results, including the CHAIR and AMBER numbers in the abstract, I don't think the main claims are supported as reported.

  2. On AMBER, every layer gives nearly the same hallucination rate, which the authors themselves attribute to "the gated edit itself rather than ... where it is applied" (Sec. 4.3). On MME and HaloQuest, Li∗L^*_i is best or tied in only 4 of 12 rows, and it trails fixed layers in several of them (e.g., Qwen3-VL MME: 94.2 vs. 97.6 for FoID; Table 2). In the QT setting it is worse than FoID and the last layer for all three models (Table 9). Its detection AUROC is above FEPoID in only 6 of 12 settings (Table 10). The geometric argument for taking the argmax of LID over layers (Appendix B) is also called a "working hypothesis". So the evidence mostly supports FEPoID, which comes from Wang et al. (2026), not the new per-input rule.

  3. No baseline edits with a random direction of the same norm, or one that uses the probe alone (e.g., flipping the answer on binary tasks such as POPE and MME when the probe flags it). Without these, it's unclear whether the gains come from the probe-gradient edit or simply from regenerating flagged outputs. The number of correct outputs made incorrect is also never reported, although Sec. 4 lists "preserving reliable outputs" as a research question.

Questions for authors
  1. Can you report AMBER, MME, HaloQuest and CHAIR mitigation results on examples that were not used to train the probes?
  2. How many correct outputs became incorrect after mitigation on each benchmark? The MME numbers in Tables 2 and 4 suggest almost none. Is that right?
  3. How does LIFE compare with (a) the same edit using a random direction of the same norm, and (b) a probe-only baseline that flips the answer on binary tasks when the probe flags it?
  4. What is the latency per sample on AMBER or CHAIR, where the regenerated continuation is long? (Clarification)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.