HCA-Edit: Hybrid Category-Aware Editing for Visual Hallucination Mitigation

Activation editing offers an efficient, training-free strategy to mitigate visual hallucinations in large vision-language models (LVLMs). However, visual hallucinations span distinct semantic categories-specifically object, attribute, and relation errors. Estimating a single correction direction from mixed hallucination examples obscures category-specific information and favors categories with larger activation magnitudes. Furthermore, applying a uniform editing strength across layers overlooks the variation of these category signals across network depth. To address these limitations, we propose HCA-Edit (Hybrid Category-Aware Editing), a training-free framework that combines category-balanced direction construction with category-aware layer gating. Using paired calibration data constructed through targeted semantic perturbations, HCA-Edit estimates correction directions separately for object, attribute, and relation hallucinations, normalizes each direction within each layer, and combines them with equal weights. This balances category contributions independently of their raw direction magnitudes. To allocate editing strength across depth, we normalize category-specific activation-difference signals across layers and take their maximum across categories, retaining strong responses from any category. The resulting shared gate assigns continuous, layer-specific editing strengths while preserving the balanced category mixture. Extensive experiments across multiple LVLMs and benchmarks demonstrate that HCA-Edit effectively reduces visual hallucinations while maintaining general visual understanding capabilities.