- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies multi-agent hallucination snowballing (MHS) in MLLM-based multi-agent systems and recasts it as two empirically separable failure modes: intra-agent generation, attributed to an under-reliance on visual evidence, and inter-agent propagation, attributed to an over-reliance on unreliable memory evidence. The two modes are identified through latent-space interventions that add or remove an evidence source and track its relative support across decoder layers. Building on this diagnosis, the authors propose a training-free framework, DELVE, comprising two complementary modules. PVER reinforces latent representations using dimension-level visual-evidence weights derived from the initial agent's pristine (memory-uncontaminated) visual evidence, applied through a norm-preserving update. CUMS treats cross-layer directional inconsistency as a proxy for memory conflict and adaptively attenuates conflicting upstream contributions while retaining consistent ones. The method is evaluated across six-plus benchmarks, four MLLMs, and three MAS topologies, with ablation, parameter-sensitivity, interaction-scale, and overhead analyses.
Critical strengths
-
A well-posed and empirically grounded problem formulation. The paper operationalizes an otherwise loosely defined phenomenon into two measurable and separately addressable mechanisms. The decomposition is not purely conceptual: it is supported by latent-space observations obtained through controlled addition and removal of evidence sources (Eqs. 3–4), which gives the subsequent module design a clear diagnostic rationale rather than an ad hoc combination of components.
-
A training-free design with bounded marginal cost and a principled formulation. DELVE requires no parameter updates and no preference or SFT data. Its reported inference cost is approximately 1.61× the baseline (Table 5): evidence extraction is performed by a fixed number of prompt-level forward passes per agent and reused throughout decoding, leaving only O(Ld) element-wise operations per generated token.
Critical weaknesses
- Gains diminish on stronger base models, and the ablation is confined to the weakest base. Improvements vanish or reverse on the strongest base (Qwen3-VL-8B: POPE avg-F1 88.2→88.2, mesh POPE-R 90.1→89.8, MME-Total 2354.6→2321.5), without explanation. Both the ablation (Table 4) and the sensitivity study (Fig. 3) use only the weakest base, LLaVA-1.5-7B, so the modules' effectiveness and complementarity on a strong base remain unverified. An ablation on a strong base, together with a brief account of why the gains contract, would help clarify whether the mechanism still holds in that regime.
- Limited evidence relative to the most relevant competitors. The honest omission of ViF (Appendix C.1) still leaves relative standing undetermined, and other methods (Table 3) are compared only on hallucination benchmarks under random—never on general benchmarks—so a "better capability preservation" claim is unsupported (such data exist, e.g. Saliency reports MME/MM-Vet).
- Inference efficiency relative to the training-free alternative. The (, Table 5) latency penalty is roughly an order of magnitude higher than ViF ( latency overhead, as self-reported in ViF's original paper). Moreover, overhead is measured only on short-form POPE without memory footprint or scaling curves (resolution/turn count), leaving deployment costs on visual-heavy or long-context workloads unknown.
Questions for authors
- Is there an explanation for the flat/negative results on Qwen3-VL-8B, and do PVER and CUMS still contribute (and remain complementary) on strong bases?
- Is cross-layer directional inconsistency empirically a valid proxy for genuine memory error, as opposed to correct-but-differently-phrased upstream content?
7 comments
Rebuttal to Reviewer 2fTE (2/4)
Q2: 1) The omission of ViF in Appendix C.1 leaves DELVE's relative standing against a directly relevant competitor undetermined. Can the authors provide a comparison? The honest omission of ViF (Appendix C.1) still leaves relative standing undetermined, and 2) other methods (Table 3) are compared only on hallucination benchmarks under random—never on general benchmarks—so a “better capability preservation” claim is unsupported (such data exist, e.g. Saliency reports MME/MM-Vet).
A2: 1) We thank the reviewer for this constructive suggestion. We are committed to fair and comprehensive comparisons and have made extensive efforts to reproduce ViF. As discussed in Appendix C.1, the difficulty extends beyond the incomplete inference implementation in the public release. ViF requires additional training, yet its paper and codes fail to provide sufficient details regarding the training process, training data, and implementation configurations for the evaluated backbones, making the reported results difficult to reproduce and independently verify. In particular, the uncertainty regarding the training data prevents us from verifying that our implementation matches the original training conditions.
During our research, we implemented the inference process according to the paper and fine-tuned the resulting model using COCO and additional llava tuning image-text datasets. We denote this data-adapted reimplementation as ViF*. The results are reported in Tab. 2 below.
Tab. 2: Performance comparison of the baseline, ViF*, and our DELVE on POPE and MMHal-Bench with LLaVA-1.5-7B and Qwen3-VL-8B.
| Backbone | Method | POPE-R F1 ↑ | POPE-P F1 ↑ | POPE-A F1 ↑ | POPE Avg. F1 ↑ | MMHal-Bench Score ↑ |
|---|---|---|---|---|---|---|
| LLaVA-1.5-7B | Baseline | 86.1 | 78.2 | 77.1 | 80.5 | 2.38 |
| LLaVA-1.5-7B | ViF* | 85.3 | 75.6 | 75.1 | 78.7 | 2.14 |
| LLaVA-1.5-7B | DELVE | 88.0 | 83.7 | 79.2 | 83.6 | 2.40 |
| Qwen3-VL-8B | Baseline | 90.1 | 87.9 | 86.6 | 88.2 | 2.69 |
| Qwen3-VL-8B | ViF* | 88.1 | 86.4 | 85.3 | 86.6 | 2.32 |
| Qwen3-VL-8B | DELVE | 90.2 | 88.1 | 86.4 | 88.2 | 2.75 |
Our DELVE consistently outperforms ViF*, further demonstrating the superiority and effectiveness of our method. Specifically, ViF* underperforms the corresponding baselines on both architectures. Together with the insufficiently specified training conditions in the original work, these results suggest that the available implementation details do not provide a sufficient basis for a reliable and verifiable reproduction of the reported results. We therefore did not present these results as the performance of the ViF method, which could unfairly disadvantage it.
To address the reviewer's concerns, we additionally implemented ViF-TF, a training-free variant retaining ViF’s Attention Reallocation component without the trained visual-relay component. Within the limited rebuttal period, we evaluated this simplified baseline on LLaVA-1.5-7B using all 9,000 POPE questions and the open-ended MMHal-Bench task, as shown in Tab. 3.
Tab. 3: Performance comparison of the baseline, ViF-TF, and our DELVE on POPE and MMHal-Bench with LLaVA-1.5-7B.
| Backbone | Method | POPE-R F1 ↑ | POPE-P F1 ↑ | POPE-A F1 ↑ | POPE Avg. F1 ↑ | MMHal-Bench Score ↑ |
|---|---|---|---|---|---|---|
| LLaVA-1.5-7B | Baseline | 86.1 | 78.2 | 77.1 | 80.5 | 2.38 |
| LLaVA-1.5-7B | ViF-TF | 87.3 | 79.4 | 78.2 | 81.6 | 2.29 |
| LLaVA-1.5-7B | DELVE | 88.0 | 83.7 | 79.2 | 83.6 | 2.40 |
Although ViF-TF achieves a modest improvement on POPE, increasing the average F1 score from 80.5 to 81.6, our DELVE consistently outperforms ViF-TF across all reported metrics, achieving an average F1 score of 83.6, further demonstrating the effectiveness of our approach. More importantly, we observe that while ViF-TF provides certain benefits on binary yes/no benchmarks such as POPE, its performance on open-ended generation tasks such as MMHal-Bench deteriorates compared with the baseline model. This degradation is reflected not only in the score drop from 2.38 to 2.29, but also in severe generation failures observed in its outputs, including a significant loss of coherence, frequent repetitive generation loops, and garbled text. These observations suggest that the additional training required by the original ViF framework may be essential for mitigating such generation failures, but we are unable to reproduce this training process due to the lack of sufficient training details and reproducibility resources.
Consequently, ViF-TF cannot be considered a representative substitute for the original ViF, as removing its trainable components fundamentally alters its generation behavior. Therefore, a direct comparison against ViF-TF alone would not constitute a reliable assessment of the complete ViF framework.
Rebuttal to Reviewer 2fTE (3/4)
A2: 2) We thank the reviewer for highlighting this potential ambiguity. We would like to clarify that we do NOT claim better capability preservation. In Section 4.1, observation 3, we state: “Our method effectively mitigates hallucinations while preserving base models’ original capabilities on general VQA benchmarks, with slight performance improvements observed on most benchmarks.” This statement refers to the comparisons between DELVE and its corresponding base-model baselines in Table 2, which evaluates MME, MMBench, and MM-Vet. It does not assert a comparative advantage over other mitigation methods on general benchmarks. Our motivation for including these evaluations is to examine potential capability costs alongside hallucination reduction. Such trade-offs have been discussed in prior work: for example, LLaVA-RLHF reports capability degradation at the 7B scale and notes that brief or evasive responses can receive high scores on MMHAL-BENCH, highlighting the importance of evaluating helpfulness and general capabilities separately [A]. We have clarified the scope of this statement in the revised manuscript to avoid implying a broader comparative claim, with the corresponding changes highlighted in red in Section 4.1 and our Abstract.
Q3: 1) The (1.61$\times$, Table 5) latency penalty is roughly an order of magnitude higher than ViF (1.08-1.13$\times$ latency overhead, as self-reported in ViF's original paper). 2) Table 5 measures overhead only on short-form POPE, without memory footprint or scaling with image resolution and interaction turns. Can deployment costs be clarified?
A3: We thank the reviewer for this valuable suggestion.
1) We would like to clarify that DELVE is fully training-free, whereas ViF employs a two-stage training procedure, as described in Appendix C.1 of our paper. To be specific, training-free methods usually involve repeated full-model decoding can introduce substantial inference costs. For example, the recent Saliency baseline incurs a 6.93× slowdown under the evaluation protocol. As explained in Appendix B.3, DELVE instead extracts evidence through prompt-only forward passes and reuses the resulting modulation vectors throughout generation. Its interventions operate directly on existing hidden states without additional full-model evaluations at each decoding step, keeping the measured slowdown below 2×, at 1.61×. This demonstrates the competitive inference efficiency of our training-free approach. Although ViF achieves lower inference overhead, it requires additional training and data, which limits its flexibility and ease of deployment across different models. Therefore, inference latency alone does not fully reflect the overall computational and deployment costs, making a direct comparison based solely on inference latency potentially misleading and unfair.
2) DELVE reinforces visual evidence and suppresses unreliable contextual contributions directly in latent space, adding lightweight operations during generation. To address the reviewer’s concern and evaluate its efficiency beyond short-form POPE, we have additionally measured inference time on MMHal-Bench and Hall-Bench within the rebuttal period. The results are summarized in Tab. 4 and Tab. 5 below.
Tab. 4: Inference latency comparison between Baseline and DELVE.
| Benchmark | Baseline time (s/sample) ↓ | DELVE time (s/sample) ↓ | DELVE / Baseline ↓ |
|---|---|---|---|
| MMHal-Bench | 11.266 | 19.029 | |
| Hall-Bench | 18.595 | 28.110 |
Tab. 5: Inference latency comparison between Saliency and DELVE.
| Benchmark | Saliency time (s/sample) ↓ | DELVE time (s/sample) ↓ | DELVE / Saliency ↓ |
|---|---|---|---|
| MMHal-Bench | 537.493 | 19.029 | |
| Hall-Bench | 115.395 | 28.110 |
Across both benchmarks, DELVE incurs only a 1.51–1.69× inference slowdown relative to the baseline, below the at least 2× overhead commonly associated with decoding-based methods. Moreover, its computational overhead remains manageable even with complex contexts and high-resolution images, without exhibiting exponential growth in these settings. Notably, DELVE is 4.1–28.2× faster than Saliency, a representative decoding-based method, further demonstrating its computational efficiency and practical applicability. We have incorporated these results and the corresponding analysis into Appendix B.3 of the revised manuscript, with the revisions highlighted in red.
Rebuttal to Reviewer 2fTE (1/4)
Q1: 1) Why do gains vanish or reverse on Qwen3-VL-8B? 2) Since ablation study Table 4 only evaluate LLaVA-1.5-7B, do PVER and CUMS still contribute and remain complementary on a strong backbone?
A1: We thank the reviewer for highlighting these results.
1) On Qwen3-VL-8B, DELVE improves average POPE F1 across all three topologies, despite a slight decline on POPE-R under Mesh. Under Random, it also reduces HS, showing that improved final-answer performance is accompanied by reduced hallucination snowballing.
The stronger backbone already achieves substantially higher baseline performance, leaving a smaller observed margin for improvement. DELVE adjusts the contributions of visual and upstream-memory evidence through inference-time modulation; this adjustment produces task-dependent gains and trade-offs. For MME, the 33.1-point total decrease is concentrated in perception (1669.7→1636.2), while cognition remains essentially unchanged (685.0→685.4). Meanwhile, all three MMBench accuracy measures improve. The results therefore identify a specific capability trade-off on MME rather than a uniform decline in general performance.
2) Yes. PVER and CUMS remain effective and complementary on stronger backbones. We have extended our ablation study to Qwen3-VL-8B. As shown in Tab. 1 below, combining PVER and CUMS achieves the best results across all three benchmarks on both backbones, supporting their complementary contributions. These additional results and the corresponding revisions have been incorporated into Section 4.2 of the revised manuscript and highlighted in red.
Tab. 1: Ablation study on LLaVA-1.5-7B and Qwen3-VL-8B across three benchmarks. ✓ stands for use.