TRACE: Tracing Representation Ancestry for Hallucination Mitigation in Large Vision-Language Models

Large Vision-Language Models (LVLMs) often generate fluent but visually unsupported content. Prior studies commonly explain such hallucinations through reduced visual attention or increased reliance on textual context, implicitly associating information with the modality of the token positions that carry it. However, Transformer layers repeatedly mix information across positions, allowing visual evidence to propagate into question and previously generated tokens. As a result, token modality does not always reveal where information originally comes from. We introduce TRACE, a provenance-based framework for analyzing and mitigating hallucination in LVLMs. TRACE traces information back to its source by propagating contributions from visual inputs, user queries, and generation history through intermediate representations. This analysis reveals a recurring pattern before hallucination, which we call provenance takeover: generation-history provenance becomes increasingly dominant over visual provenance as the model approaches a hallucinated token. Building on this observation, TRACE detects abnormal takeover during decoding and reinforces the corresponding visual-origin component of the current representation. This enables our method to intervene selectively when the model begins to drift away from visual evidence, without requiring additional training. Across many benchmarks, TRACE consistently reduces hallucination-related errors across LVLMs while largely preserving overall multimodal performance with modest inference-time overhead. These results establish information provenance as a useful perspective for understanding hallucination and a practical signal for improving visual grounding in LVLMs.