acceptodds
Under review as a conference paper at ICLR 2027

HCA-Edit: Hybrid Category-Aware Editing for Visual Hallucination Mitigation

Abstract

Activation editing offers an efficient, training-free strategy to mitigate visual hallucinations in large vision-language models (LVLMs). However, visual hallucinations span distinct semantic categories-specifically object, attribute, and relation errors. Estimating a single correction direction from mixed hallucination examples obscures category-specific information and favors categories with larger activation magnitudes. Furthermore, applying a uniform editing strength across layers overlooks the variation of these category signals across network depth. To address these limitations, we propose HCA-Edit (Hybrid Category-Aware Editing), a training-free framework that combines category-balanced direction construction with category-aware layer gating. Using paired calibration data constructed through targeted semantic perturbations, HCA-Edit estimates correction directions separately for object, attribute, and relation hallucinations, normalizes each direction within each layer, and combines them with equal weights. This balances category contributions independently of their raw direction magnitudes. To allocate editing strength across depth, we normalize category-specific activation-difference signals across layers and take their maximum across categories, retaining strong responses from any category. The resulting shared gate assigns continuous, layer-specific editing strengths while preserving the balanced category mixture. Extensive experiments across multiple LVLMs and benchmarks demonstrate that HCA-Edit effectively reduces visual hallucinations while maintaining general visual understanding capabilities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large vision-language models, Knowledge editing, Hallucination

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gkXkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses visual hallucination mitigation in LVLMs by showing that separately estimating correction directions for object, attribute, and relation hallucinations and then fusing them is more effective than learning a single direction from mixed categories. The proposed HCA-Edit normalizes and combines category-specific directions and uses category-aware layer gating to adapt the editing strength across layers. Experiments across multiple LVLMs and benchmarks show consistent hallucination reduction while largely preserving general visual capabilities.

Critical strengths

The paper addresses the important and practically relevant problem of hallucination mitigation in LVLMs, while considering multiple hallucination categories beyond object-level hallucinations.

Critical weaknesses
  • LURE-3C is presented as a major contribution and serves as the basis of the proposed method. However, its construction largely extends the paired-data workflow used by Nullu from object hallucinations to attribute and relation hallucinations. The paper also reports no human annotation or systematic validation of the generated category labels, limiting both its novelty and reliability.

  • The effectiveness of the category-balanced direction construction is not well supported by the ablation results. In Table 4, “Category-balanced + Uniform” performs worse than “Pooled + Uniform,” leaving the independent benefit of category-specific direction estimation unclear and casting doubt on the stated motivation for estimating category-specific directions before combining them. Moreover, the main evaluations are object-centric (CHAIR and POPE), while the limited three-category evaluation reports increased attribute hallucination severity (Color sub-task in MME and Attribute dimension of LLaVA-Bench), providing insufficient evidence that the method benefits all three hallucination categories.

Questions for authors
  1. What is the motivation for introducing the floor ff in Eq. (7) and the additional editing strength αtext\alpha_{\text{text}} in Eq. (8)? Please clarify why both components are necessary and provide ablations removing them.

  2. Although three LVLMs are considered, POPE is evaluated only on the two LLaVA models, while MME and LLaVA-Bench are evaluated only on LLaVA-1.5-7B. Why are the corresponding results on Qwen2.5-VL-7B omitted, and does the method remain effective on these benchmarks across model families?

  3. HCA-Edit introduces several hyperparameters, including αtext\alpha_{\text{text}}, ff, pp, and amax⁡a_{\max}, potentially creating a substantial tuning burden in practical deployment. It would be helpful to provide a practical method for selecting hyperparameters.

  4. Additional references on Attention Intervention to consider:

[1] Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. Jiang Z, Chen J, Zhu B, et al. CVPR 2025.

Reviewer_pLpbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HCA-Edit estimates separate activation directions from object, attribute and relation perturbations, normalizes and averages them, and uses category-derived signals to allocate editing strength across layers. The directions and gates are fixed at inference. Evaluation covers object hallucination benchmarks and a small model-judged study of the three error categories.

Critical strengths

HCA-Edit gives substantial object-hallucination reductions while maintaining recall. The intervention is simple at inference, and the pooled, single-category and gating controls make several design choices directly inspectable.

Critical weaknesses

• Equal category weighting is only partly justified. Normalization prevents categories with large activation norms from dominating, but that does not establish that object, attribute and relation directions are equally useful, or that every input benefits from the same mixture. The paper needs evidence for choosing equal weights beyond the normalization argument. Moreover, the calibration contrast may not represent the model's own hallucination tendency, since directions are extracted from reference descriptions and synthetically perturbed descriptions supplied to the model.

• It is unclear how well the correction directions transfer beyond the calibration distribution. Results on several benchmarks are encouraging, but they do not isolate sensitivity to the words, phrasing and scenes used in the 500 calibration contexts. A direction could exploit regularities in those contexts without capturing a broadly transferable correction.

Questions for authors

• How does equal weighting compare with unequal mixtures chosen on held-out data, and does the preferred mixture differ across question or error types?

• With calibration directions and coefficients fixed, do the gains persist for unseen vocabulary, paraphrased descriptions and shifted scene distributions?

• Do the synthetic-pair directions align with corrections derived from the backbone's actual hallucinations in free generation? What changes under meaning-preserving paraphrase controls?

Reviewer_Yp2POpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HCA-Edit:a training-free activation-editing method for reducing visual hallucinations. The authors construct LURE-3C by generating object-level, attribute-level, and relation-level perturbations of reference captions. HCA-Edit estimates a separate correction direction for each category and combines the normalized directions with equal weights. It uses category-conditioned activation differences to determine the editing strength at each layer. Experiments on LLaVA-1.5 and Qwen2.5-VL show strong improvements on CHAIR and POPE. The paper also reports results on MME and LLaVA-Bench, together with several ablation studies.

Critical strengths
  1. The method follows clearly from the paper’s motivating analysis. The paper identifies two possible problems with existing activation-editing methods: mixing different hallucination categories into one direction and using the same editing strength at every layer. HCA-Edit addresses these issues through category-balanced directions and layer-dependent gating. The method does not require training or additional inference passes.
  2. The improvements on object-hallucination benchmarks are strong and consistent. On CHAIR, HCA-Edit substantially reduces sentence-level hallucination across all three tested model configurations. Recall is also maintained or improved. The POPE results are similarly competitive. The paper provides useful ablations and reports the inference overhead, which makes the practical trade-offs relatively clear.
Critical weaknesses
  1. The evidence for three-category mitigation remains limited.CHAIR and POPE mainly test object hallucinations, while MME is not category-specific. The only category-level evaluation uses GPT-5.6 on 60 LLaVA-Bench examples, where attribute severity actually increases from 0.53 to 0.57. Overall, the evidence is convincing for object hallucination, but remains limited for attribute and relation hallucinations.
  2. The novelty over related activation-editing methods needs clearer support. AFTER already uses category-level, attribute-level, and relation-level factual signals, while LTS-FS already varies steering strength across layers. HCA-Edit has meaningful differences, but the paper does not compare with AFTER. Table 4 also shows that neither component improves over the pooled-and-uniform baseline when used alone. This suggests that the gain depends on their interaction, while the individual contribution of each component remains unclear.
Questions for authors
  1. Which generative model was used to construct LURE-3C? Was the resulting dataset checked by human annotators for category purity?
  2. Can the authors provide a validated evaluation for object, attribute, and relation hallucinations separately? Please also explain why attribute hallucination severity becomes worse on LLaVA-Bench.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.