Can Video-Language Models Be Induced to Hallucinate Objects? A Study of Semantic Implantation via Feature Alignment

Video-Language Models (Vid-LMs) have achieved remarkable success in multimodal understanding, yet the robustness of their internal visual–semantic alignment remains poorly understood. In this work, we ask a fundamental question: Can Vid-LMs be induced to hallucinate non-existent objects by directly manipulating their intermediate multimodal representations? We introduce Adversarial Object Hallucination (AOH), a controlled probing framework for semantic implantation that injects target object concepts into the model's latent space via subtle perturbations. AOH operates via intermediate feature alignment, forcing the representation of an original, object-free video to mimic the latent semantics of a counterpart video where the target object is naturally present. To support systematic analysis, we construct AOH-Bench, a first-of-its-kind benchmark comprising 555 high-fidelity video pairs that enable direct comparison between implanted hallucinations and genuine object perception. Our evaluation of state-of-the-art Vid-LMs reveals three key phenomena: (1) Universality: all evaluated models consistently exhibit object hallucination under AOH; (2) Cross-scale Transferability: hallucination-inducing videos optimized on small models (e.g., 0.5B) transfer effectively to much larger models (e.g., 7B); (3) Stealthiness: hallucinations emerge without noticeable shifts in visual attention, suggesting a covert disruption of the visual–language semantic interface rather than perceptual mislocalization. Overall, our findings expose a systemic vulnerability in Vid-LMs, highlighting a fundamental gap between visual perception and linguistic description.