- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents VidHaze, an adversarial perturbation approach to inducing hallucinations in Video-LLMs. Motivated by the observation that correct sentences are more sensitive to spatiotemporal corruptions than hallucinated ones, VidHaze selects a corrupted hard negative via Thompson Sampling and contrasts it with the clean video to build a hallucination-oriented soft target for PGD optimization. Experiments on VATEX and MSRVTT across four Video-LLMs show higher hallucination rates than adapted baselines in most settings.
Critical strengths
- The evaluation is extensive, covering four Video-LLMs, two datasets, and both white-box and black-box settings.
- Finding 3 is an interesting analysis. It shows with normalized metrics (HSR, HUR) that VidHaze increases hallucination density rather than merely expanding the generated response.
Critical weaknesses
- The motivation for deliberately increasing hallucinations is unclear, since even the latest Video-LLMs already hallucinate on clean videos (Table 1). The threat model, i.e., who would launch this attack and why, is not discussed.
- The VATEX results of Gemma4 and LLaVA-NV in Table 8 are identical to Table 1, including judge-dependent metrics (Qual., HSR, HUR), although the judge is changed. Please clarify.
- Eq. (17) is essentially VCD [1] in other words. Since s⁺ − s⁻ = ℓʰ − ℓᶜ, it reduces to softmax(2ℓʰ − ℓᶜ), i.e., VCD with the roles swapped. This component is also not ablated in Table 3.
- TVA already induces fluent hallucinations in Table 1, which contradicts Figure 1 and shows that hallucination induction in Video-LLMs is not a new problem.
- Simple baselines are missing, e.g., feeding the corrupted video directly, random noise under the same ε, or an untargeted attack.
- No human evaluation verifies the LLM judges.
- The gains in Table 2 are marginal (about 2.3 HSR points over Mirage on average), and no variance is reported.
Questions for authors
Why build the target from spatiotemporal hard negatives instead of the model’s second-best tokens on the original video?
Looking Forward to Your Engagement in the Rebuttal Discussion
Thank you for your initial feedback. We believe we have thoroughly addressed your concerns through additional experiments and clarifications. We would appreciate your continued engagement in the discussion.