acceptodds
Under review as a conference paper at ICLR 2027

Seeing Clearly, Remembering Selectively: Mitigating Multi-Agent Hallucination Snowballing via Evidential Modulation

Abstract

As Multimodal Large Language Models (MLLMs) continue to advance, they are increasingly integrated into multi-agent systems for collaborative visual reasoning and task completion. However, MLLMs may still generate responses inconsistent with visual or contextual evidence, and such hallucinations can propagate across agents, causing hallucination snowballing and undermining system-level robustness. Existing mitigation methods primarily strengthen individual agents' visual grounding, leaving the reliability of information exchanged between agents insufficiently addressed. In this work, we investigate this problem from an evidential perspective by manipulating agentic visual and memory evidence to trace the influence on latent representations. Our analysis indicates that the generation of hallucination within an agent is linked to the disregard of visual evidence, whereas the propagation of hallucination between agents is associated with reliance on memory evidence supported by cross-layer inconsistencies. Based on these observations, we propose Dual Evidential Adaptive Modulation (DELVE), a training-free framework with two complementary modules. Specifically, Pristine Visual Evidence Reinforcement (PVER) derives dimension-level evidential weights to selectively reinforce latent representations with stable and pristine visual evidence of the initial agent, providing a stronger visual reference throughout agent interactions to mitigate intra-agent hallucination. Conflict-Aware Unreliable Memory Suppression (CUMS) exploits cross-layer directional inconsistency as a proxy for potential memory conflict, selectively attenuating inconsistent memory contributions while retaining coherent upstream information, thereby mitigating inter-agent hallucination propagation. Experiments on six hallucination and visual question answering benchmarks, spanning four MLLMs and three communication topologies, demonstrate improved hallucination mitigation while largely preserving general multimodal capabilities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-agent systems, hallucination mitigation, MLLMs, evidence modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2fTEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-agent hallucination snowballing (MHS) in MLLM-based multi-agent systems and recasts it as two empirically separable failure modes: intra-agent generation, attributed to an under-reliance on visual evidence, and inter-agent propagation, attributed to an over-reliance on unreliable memory evidence. The two modes are identified through latent-space interventions that add or remove an evidence source and track its relative support across decoder layers. Building on this diagnosis, the authors propose a training-free framework, DELVE, comprising two complementary modules. PVER reinforces latent representations using dimension-level visual-evidence weights derived from the initial agent's pristine (memory-uncontaminated) visual evidence, applied through a norm-preserving update. CUMS treats cross-layer directional inconsistency as a proxy for memory conflict and adaptively attenuates conflicting upstream contributions while retaining consistent ones. The method is evaluated across six-plus benchmarks, four MLLMs, and three MAS topologies, with ablation, parameter-sensitivity, interaction-scale, and overhead analyses.

Critical strengths
  • A well-posed and empirically grounded problem formulation. The paper operationalizes an otherwise loosely defined phenomenon into two measurable and separately addressable mechanisms. The decomposition is not purely conceptual: it is supported by latent-space observations obtained through controlled addition and removal of evidence sources (Eqs. 3–4), which gives the subsequent module design a clear diagnostic rationale rather than an ad hoc combination of components.

  • A training-free design with bounded marginal cost and a principled formulation. DELVE requires no parameter updates and no preference or SFT data. Its reported inference cost is approximately 1.61× the baseline (Table 5): evidence extraction is performed by a fixed number of prompt-level forward passes per agent and reused throughout decoding, leaving only O(Ld) element-wise operations per generated token.

Critical weaknesses
  • Gains diminish on stronger base models, and the ablation is confined to the weakest base. Improvements vanish or reverse on the strongest base (Qwen3-VL-8B: POPE avg-F1 88.2→88.2, mesh POPE-R 90.1→89.8, MME-Total 2354.6→2321.5), without explanation. Both the ablation (Table 4) and the sensitivity study (Fig. 3) use only the weakest base, LLaVA-1.5-7B, so the modules' effectiveness and complementarity on a strong base remain unverified. An ablation on a strong base, together with a brief account of why the gains contract, would help clarify whether the mechanism still holds in that regime.
  • Limited evidence relative to the most relevant competitors. The honest omission of ViF (Appendix C.1) still leaves relative standing undetermined, and other methods (Table 3) are compared only on hallucination benchmarks under random—never on general benchmarks—so a "better capability preservation" claim is unsupported (such data exist, e.g. Saliency reports MME/MM-Vet).
  • Inference efficiency relative to the training-free alternative. The +61%+61\% (1.61×1.61\times, Table 5) latency penalty is roughly an order of magnitude higher than ViF (+8–13%+8\text{--}13\% latency overhead, as self-reported in ViF's original paper). Moreover, overhead is measured only on short-form POPE without memory footprint or scaling curves (resolution/turn count), leaving deployment costs on visual-heavy or long-context workloads unknown.
Questions for authors
  • Is there an explanation for the flat/negative results on Qwen3-VL-8B, and do PVER and CUMS still contribute (and remain complementary) on strong bases?
  • Is cross-layer directional inconsistency empirically a valid proxy for genuine memory error, as opposed to correct-but-differently-phrased upstream content?
7 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 2fTE (1/4)

Q1: 1) Why do gains vanish or reverse on Qwen3-VL-8B? 2) Since ablation study Table 4 only evaluate LLaVA-1.5-7B, do PVER and CUMS still contribute and remain complementary on a strong backbone?

A1: We thank the reviewer for highlighting these results.

1) On Qwen3-VL-8B, DELVE improves average POPE F1 across all three topologies, despite a slight decline on POPE-R under Mesh. Under Random, it also reduces HS, showing that improved final-answer performance is accompanied by reduced hallucination snowballing.

The stronger backbone already achieves substantially higher baseline performance, leaving a smaller observed margin for improvement. DELVE adjusts the contributions of visual and upstream-memory evidence through inference-time modulation; this adjustment produces task-dependent gains and trade-offs. For MME, the 33.1-point total decrease is concentrated in perception (1669.7→1636.2), while cognition remains essentially unchanged (685.0→685.4). Meanwhile, all three MMBench accuracy measures improve. The results therefore identify a specific capability trade-off on MME rather than a uniform decline in general performance.

2) Yes. PVER and CUMS remain effective and complementary on stronger backbones. We have extended our ablation study to Qwen3-VL-8B. As shown in Tab. 1 below, combining PVER and CUMS achieves the best results across all three benchmarks on both backbones, supporting their complementary contributions. These additional results and the corresponding revisions have been incorporated into Section 4.2 of the revised manuscript and highlighted in red.

Tab. 1: Ablation study on LLaVA-1.5-7B and Qwen3-VL-8B across three benchmarks. ✓ stands for use.

PVERCUMSPOPE-R · LLaVAPOPE-R · QwenPOPE-A · LLaVAPOPE-A · QwenMMHal-Bench · LLaVAMMHal-Bench · Qwen
86.190.177.186.62.352.82
✓87.791.077.887.02.502.95
✓87.091.076.186.82.442.76
✓✓88.091.579.287.22.563.07
AuthorsOpenReview ↗

Rebuttal to Reviewer 2fTE (2/4)

Q2: 1) The omission of ViF in Appendix C.1 leaves DELVE's relative standing against a directly relevant competitor undetermined. Can the authors provide a comparison? The honest omission of ViF (Appendix C.1) still leaves relative standing undetermined, and 2) other methods (Table 3) are compared only on hallucination benchmarks under random—never on general benchmarks—so a “better capability preservation” claim is unsupported (such data exist, e.g. Saliency reports MME/MM-Vet).

A2: 1) We thank the reviewer for this constructive suggestion. We are committed to fair and comprehensive comparisons and have made extensive efforts to reproduce ViF. As discussed in Appendix C.1, the difficulty extends beyond the incomplete inference implementation in the public release. ViF requires additional training, yet its paper and codes fail to provide sufficient details regarding the training process, training data, and implementation configurations for the evaluated backbones, making the reported results difficult to reproduce and independently verify. In particular, the uncertainty regarding the training data prevents us from verifying that our implementation matches the original training conditions.

During our research, we implemented the inference process according to the paper and fine-tuned the resulting model using COCO and additional llava tuning image-text datasets. We denote this data-adapted reimplementation as ViF*. The results are reported in Tab. 2 below.

Tab. 2: Performance comparison of the baseline, ViF*, and our DELVE on POPE and MMHal-Bench with LLaVA-1.5-7B and Qwen3-VL-8B.

BackboneMethodPOPE-R F1 ↑POPE-P F1 ↑POPE-A F1 ↑POPE Avg. F1 ↑MMHal-Bench Score ↑
LLaVA-1.5-7BBaseline86.178.277.180.52.38
LLaVA-1.5-7BViF*85.375.675.178.72.14
LLaVA-1.5-7BDELVE88.083.779.283.62.40
Qwen3-VL-8BBaseline90.187.986.688.22.69
Qwen3-VL-8BViF*88.186.485.386.62.32
Qwen3-VL-8BDELVE90.288.186.488.22.75

Our DELVE consistently outperforms ViF*, further demonstrating the superiority and effectiveness of our method. Specifically, ViF* underperforms the corresponding baselines on both architectures. Together with the insufficiently specified training conditions in the original work, these results suggest that the available implementation details do not provide a sufficient basis for a reliable and verifiable reproduction of the reported results. We therefore did not present these results as the performance of the ViF method, which could unfairly disadvantage it.

To address the reviewer's concerns, we additionally implemented ViF-TF, a training-free variant retaining ViF’s Attention Reallocation component without the trained visual-relay component. Within the limited rebuttal period, we evaluated this simplified baseline on LLaVA-1.5-7B using all 9,000 POPE questions and the open-ended MMHal-Bench task, as shown in Tab. 3.

Tab. 3: Performance comparison of the baseline, ViF-TF, and our DELVE on POPE and MMHal-Bench with LLaVA-1.5-7B.

BackboneMethodPOPE-R F1 ↑POPE-P F1 ↑POPE-A F1 ↑POPE Avg. F1 ↑MMHal-Bench Score ↑
LLaVA-1.5-7BBaseline86.178.277.180.52.38
LLaVA-1.5-7BViF-TF87.379.478.281.62.29
LLaVA-1.5-7BDELVE88.083.779.283.62.40

Although ViF-TF achieves a modest improvement on POPE, increasing the average F1 score from 80.5 to 81.6, our DELVE consistently outperforms ViF-TF across all reported metrics, achieving an average F1 score of 83.6, further demonstrating the effectiveness of our approach. More importantly, we observe that while ViF-TF provides certain benefits on binary yes/no benchmarks such as POPE, its performance on open-ended generation tasks such as MMHal-Bench deteriorates compared with the baseline model. This degradation is reflected not only in the score drop from 2.38 to 2.29, but also in severe generation failures observed in its outputs, including a significant loss of coherence, frequent repetitive generation loops, and garbled text. These observations suggest that the additional training required by the original ViF framework may be essential for mitigating such generation failures, but we are unable to reproduce this training process due to the lack of sufficient training details and reproducibility resources.

Consequently, ViF-TF cannot be considered a representative substitute for the original ViF, as removing its trainable components fundamentally alters its generation behavior. Therefore, a direct comparison against ViF-TF alone would not constitute a reliable assessment of the complete ViF framework.

AuthorsOpenReview ↗

Rebuttal to Reviewer 2fTE (3/4)

A2: 2) We thank the reviewer for highlighting this potential ambiguity. We would like to clarify that we do NOT claim better capability preservation. In Section 4.1, observation 3, we state: “Our method effectively mitigates hallucinations while preserving base models’ original capabilities on general VQA benchmarks, with slight performance improvements observed on most benchmarks.” This statement refers to the comparisons between DELVE and its corresponding base-model baselines in Table 2, which evaluates MME, MMBench, and MM-Vet. It does not assert a comparative advantage over other mitigation methods on general benchmarks. Our motivation for including these evaluations is to examine potential capability costs alongside hallucination reduction. Such trade-offs have been discussed in prior work: for example, LLaVA-RLHF reports capability degradation at the 7B scale and notes that brief or evasive responses can receive high scores on MMHAL-BENCH, highlighting the importance of evaluating helpfulness and general capabilities separately [A]. We have clarified the scope of this statement in the revised manuscript to avoid implying a broader comparative claim, with the corresponding changes highlighted in red in Section 4.1 and our Abstract.

Q3: 1) The (1.61$\times$, Table 5) latency penalty is roughly an order of magnitude higher than ViF (1.08-1.13$\times$ latency overhead, as self-reported in ViF's original paper). 2) Table 5 measures overhead only on short-form POPE, without memory footprint or scaling with image resolution and interaction turns. Can deployment costs be clarified?

A3: We thank the reviewer for this valuable suggestion.

1) We would like to clarify that DELVE is fully training-free, whereas ViF employs a two-stage training procedure, as described in Appendix C.1 of our paper. To be specific, training-free methods usually involve repeated full-model decoding can introduce substantial inference costs. For example, the recent Saliency baseline incurs a 6.93× slowdown under the evaluation protocol. As explained in Appendix B.3, DELVE instead extracts evidence through prompt-only forward passes and reuses the resulting modulation vectors throughout generation. Its interventions operate directly on existing hidden states without additional full-model evaluations at each decoding step, keeping the measured slowdown below 2×, at 1.61×. This demonstrates the competitive inference efficiency of our training-free approach. Although ViF achieves lower inference overhead, it requires additional training and data, which limits its flexibility and ease of deployment across different models. Therefore, inference latency alone does not fully reflect the overall computational and deployment costs, making a direct comparison based solely on inference latency potentially misleading and unfair.

2) DELVE reinforces visual evidence and suppresses unreliable contextual contributions directly in latent space, adding lightweight operations during generation. To address the reviewer’s concern and evaluate its efficiency beyond short-form POPE, we have additionally measured inference time on MMHal-Bench and Hall-Bench within the rebuttal period. The results are summarized in Tab. 4 and Tab. 5 below.

Tab. 4: Inference latency comparison between Baseline and DELVE.

BenchmarkBaseline time (s/sample) ↓DELVE time (s/sample) ↓DELVE / Baseline ↓
MMHal-Bench11.26619.0291.689×1.689\times
Hall-Bench18.59528.1101.512×1.512\times

Tab. 5: Inference latency comparison between Saliency and DELVE.

BenchmarkSaliency time (s/sample) ↓DELVE time (s/sample) ↓DELVE / Saliency ↓
MMHal-Bench537.49319.0290.035×0.035\times
Hall-Bench115.39528.1100.244×0.244\times

Across both benchmarks, DELVE incurs only a 1.51–1.69× inference slowdown relative to the baseline, below the at least 2× overhead commonly associated with decoding-based methods. Moreover, its computational overhead remains manageable even with complex contexts and high-resolution images, without exhibiting exponential growth in these settings. Notably, DELVE is 4.1–28.2× faster than Saliency, a representative decoding-based method, further demonstrating its computational efficiency and practical applicability. We have incorporated these results and the corresponding analysis into Appendix B.3 of the revised manuscript, with the revisions highlighted in red.

Reviewer_sv5FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the challenge of hallucination snowballing in multimodal multi-agent systems, where agents inherit and amplify each other's errors. It introduces DELVE, a training-free method that enhances visual evidence extracted before agent interactions and suppresses upstream memory contributions showing cross-layer directional disagreements. Experiments across six benchmarks, four vision-language models, and three communication topologies demonstrate a general reduction in hallucination propagation and improved answer quality. Ablation studies further validate the complementary roles of the two components. Although performance gains vary depending on the setting, the inference process takes approximately 1.61 times longer than the baseline.

Critical strengths
  1. Complementary Mechanisms. DELVE addresses both hallucination generation and propagation, with ablations supporting the benefit of combining visual reinforcement and memory suppression.
  2. Broad evaluation. Tests across four models, six benchmarks, and three communication topologies show benefits beyond a single setting, including reduced error propagation.
Critical weaknesses
  1. The evidence does not validate CUMS as a memory-reliability mechanism. CUMS uses cross-layer directional disagreement to identify unreliable upstream contributions. However, i) Figure 1 provides only qualitative examples, without measuring whether disagreement distinguishes correct from incorrect upstream information; and ii) Appendix A assumes that corresponding dimensions are comparable across terminal layers without validating this assumption. Its mathematical derivation explains directional discounting, but does not establish a connection to factual reliability. This gap concerns the paper’s central mechanism: the reported improvements do not establish that CUMS selectively suppresses erroneous memory while preserving useful information.
  2. The experiments do not isolate the benefit of the defining design choices. Table 4 establishes that combining PVER and CUMS helps, but omits the controls needed to explain why: i) conflict-based versus uniform memory suppression at matched strength; and ii) cached versus current-context visual evidence, with adaptive versus uniform weighting. Without these comparisons, the results cannot distinguish the proposed mechanisms from the simpler effects of increasing visual influence and reducing memory influence. This leaves the empirical justification for DELVE’s specific design incomplete and is a major reason for my negative assessment.
Questions for authors
  1. How well does the CUMS disagreement score distinguish correct from incorrect upstream information across models? Does it also suppress useful memory that produces cross-layer disagreement?
  2. Do the proposed mechanisms outperform strength-matched uniform memory suppression and visual reinforcement using current-context evidence? How does the cached reference behave when the initial visual interpretation is incorrect?
  3. What final values of λ\lambda and KtK_t were used, and how were they selected? Do the smaller gains persist across independently sampled communication graphs?
7 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer sv5F (1/4)

Q1: 1) Figure 1 provides only qualitative examples, without measuring whether disagreement distinguishes correct from incorrect upstream information. How well does the CUMS disagreement score distinguish correct from incorrect upstream information across models? 2) The reported improvements do not establish that CUMS selectively suppresses erroneous memory while preserving useful information. 3) Does it also suppress useful memory that produces cross-layer disagreement?

A1: Thank you for these valuable suggestions, which help us clarify the role of disagreement and strengthen the quantitative evaluation.

1) Figure 1 illustrates how upstream information affects the consistency of the receiving model's internal representations. Our motivation is consistent with prior work that uses agreement among intermediate-layer predictions as a confidence signal [A]. In CUMS, disagreement is an internal proxy for instability in memory-induced representations. It is not assigned the meaning of a factual truth label. CUMS uses this signal to attenuate inconsistent memory-induced components, without requiring a binary correct/incorrect judgment for each upstream statement. External fact-checking would be a separate and complex verification pipeline, potentially involving retrieval infrastructure or an additional verifier. CUMS operates entirely through the existing model's representations and introduces neither an external knowledge base nor a stronger inference model. The controlled experiment below evaluates preservation of downstream benefits under correct memories, rather than the disagreement score’s ability to discriminate correct from incorrect memories.

2) CUMS does not explicitly classify upstream memory as factually correct or incorrect. Such verification would introduce an additional fact-checking mechanism, potentially involving external retrieval or a stronger verifier. Instead, CUMS attenuates memory-induced components according to their cross-layer inconsistency, which can also affect useful memory. We therefore directly evaluate whether this modulation preserves useful contextual benefits and improves the receiving model’s answers, as shown in Tab. 1. Specifically, we use the same 59 audited MMHal questions for both base models. For each question, we construct a factually correct memory and a fact-preserving paraphrase, and evaluate each with CUMS off/on. Every pair shares the same image, question, memory, prompt, PVER tensor, and decoding settings. A separate no-memory condition identifies whether the supplied memory actually helps. All answer transitions below use fixed, blinded automated task-correctness judgments.

Tab. 1: CUMS evaluation on 59 MMHal questions with correct and paraphrased memories.

BackboneCorrect-memory variantOff: correct answersOff: incorrect answersInitially correct answers retained with CUMS onInitially incorrect answers corrected
LLaVA-1.5-7BOriginal273225/270/32
LLaVA-1.5-7BParaphrase293029/290/30
Qwen3-VL-8B-InstructOriginal481148/483/11
Qwen3-VL-8B-InstructParaphrase491049/491/10

Each row contains 59 paired evaluations. For Qwen3, the two variants therefore yield 48+49=97 initially correct responses, all of which remain correct, and 11+10=21 initially incorrect responses, of which 3+1=4 are corrected. Thus, CUMS retains 97/97 initially correct responses and corrects 4/21 (19.05%) initially incorrect responses, with no observed correctness regressions in these conditions.

3) Yes. CUMS may attenuate useful memories, while preserving useful-memory benefits in our controlled experiments. To be specific, the memory is considered useful if it leads to a correct answer with CUMS disabled, while the answer without memory is incorrect. We then check whether that answer remains correct with CUMS on, as depicted in Tab. 2.

Tab. 2: Evaluation of CUMS in preserving useful memory across 59 MMHal questions.

BackboneOriginal useful-memory benefits retainedParaphrased useful-memory benefits retained
LLaVA-1.5-7B9/912/12
Qwen3-VL-8B-Instruct8/85/5

Every useful-memory case above exhibits nonzero disagreement and receives nonzero modulation, yet its observed downstream benefit is preserved. These results support consistency-based contextual modulation: useful components may receive attenuation, but their downstream benefits remain available. For Qwen3-VL-8B-Instruct, this preservation accompanies additional answer corrections. This complements PVER’s visual evidence reinforcement, consistent with our main-paper MMHal ablation improving from 2.50 with PVER alone to 2.56 with PVER+CUMS. These additional experiments described above have been incorporated into Section 4.3 of our revised paper.

AuthorsOpenReview ↗

Rebuttal to Reviewer sv5F (2/4)

Q2: 1) Appendix A assumes that corresponding dimensions are comparable across terminal layers without validating this assumption. 2) The derivation explains directional discounting, but does not establish factual reliability. How is this assumption justified?

A2: Thank you for your careful review.

1) As described in [B], Transformer blocks share a common residual coordinate system, supporting our assumption that corresponding dimensions are comparable across terminal layers. More specifically, CUMS measures memory-induced changes at decoder-block outputs within the same frozen model. Transformer blocks add their updates to a common residual stream [B], so corresponding coordinates share the same computational axes. After decision-anchor pooling, the memory perturbation satisfies: δℓ+1(u)=δℓ(u)+Δfℓ+1(u),\delta_{\ell+1}^{(u)}=\delta_{\ell}^{(u)}+\Delta f_{\ell+1}^{(u)}, where Δfℓ+1(u)\Delta f_{\ell+1}^{(u)} is the difference between residual updates with and without memory. Consequently, a sign reversal means that subsequent computation counteracts and reverses the preceding memory effect along that same axis. This comparison focuses on the propagation of source-induced perturbations without assuming that individual coordinates correspond to fixed semantic concepts.

2) As discussed in A1, our reliability refers to the stability and coherence of the activation changes induced by upstream information across terminal layers. This differs from factual reliability. Establishing whether an upstream statement is factually correct requires verification beyond this consistency statistic, for example through a stronger verifier or independent evidence.

The assumption underlying CUMS is that memory contributions lacking stable internal support should receive less evidential trust. This is motivated by existing layer-wise research. Specifically, Ju et al. [C] show that contextual knowledge is encoded differently across layers, with greater encoding in upper layers, and DoLa [D] demonstrates that cross-layer dynamics provide useful signals for reducing factual hallucinations. Within the shared residual coordinates discussed above, a directional reversal means that later computation counteracts the preceding memory-induced influence. CUMS therefore interprets such disagreement as insufficiently coherent support for retaining that influence and applies reliability discounting accordingly. Moreover, the ablation results support the utility of this interpretation: adding CUMS to PVER improves POPE-A F1 from 77.8 to 79.2 and MMHal score from 2.50 to 2.56. These gains support regulating memory influence through internal consistency, while preserving the distinction between evidential reliability and factual verification.

To address the reviewer’s concern and provide a more comprehensive theoretical account, we have revised and expanded Appendix A, with the revisions highlighted in red.

AuthorsOpenReview ↗

Rebuttal to Reviewer sv5F (3/4)

Q3: 1) Table 4 establishes that combining PVER and CUMS helps, but omits the controls needed to explain why: i) conflict-based versus uniform memory suppression at matched strength; and ii) cached versus current-context visual evidence, with adaptive versus uniform weighting. 2) How does the cached reference behave when the initial visual interpretation is incorrect?

A3: We thank the reviewer for the valuable comments.

1) On the one hand, CUMS uses cross-layer conflicts to identify memory-induced components for attenuation, allowing directionally consistent contributions to undergo less suppression and thereby preserving upstream information that uniform memory suppression might otherwise discard. On the other hand, PVER provides a fixed visual reference extracted before upstream memory is introduced. Together, these designs aim to provide the model with consistent visual latent evidence that remains independent of contextual contamination. Compared with relying on visual evidence extracted from the current context, this approach offers both a stronger conceptual rationale and greater computational efficiency.

To address the concern, we have added controlled ablations on Qwen3-VL-8B, comparing conflict-based suppression against uniform suppression at matched strength, and examining cached versus current-context visual evidence with adaptive versus uniform weighting. The experimental results are shown in Tab. 3.

Tab. 3: Comparison of the full DELVE method with matched-strength uniform suppression and current-context visual evidence baselines.

ModelVariantPOPE-A F1MMHal-Bench Score
Qwen3-VL-8BUni. Memory86.62.69
Curr. Visual86.92.62
Ours87.22.75

DELVE achieves the best overall performance among the evaluated variants on Qwen3-VL-8B. These results support the benefits of conflict-aware suppression and adaptive reinforcement of the cached visual reference. The additional ablation results and implementation details have been incorporated into Appendix C.5 as shown in red.

2) We would like to clarify that our cached reference is extracted by treating the entire image as an evidence source and measuring its effect on latent representations. It is not constructed from an explicit textual interpretation or preselected image regions. Thus, even when the initial interpretation is incorrect, the cached reference may still retain useful visual cues that subsequent agents can exploit. Through context-dependent weighting and reinforcement, PVER may help later agents make better use of these cues and reduce hallucinations.

Reviewer_hu6xOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles two issues in MLLMs in the context of multi-modal reasoning tasks for multi-agent systems. The first one is each agent's hallucination (authors call this intra-agent hallucination) and the second one is propagation of error among agents in the multi-agent system (authors call this one multi-agent hallucination snowballing). The main claim in the paper is that the first effect is mainly due to ignoring the visual modality in a multi-modal reasoning task, and the main reason for the second problem is accumulation of error and reliance on inaccurate memory in multi-agent systems while agent are exchanging the information. Authors claim that the main focus in the literature has been on fixing the first issue, while the second one hasn't been sufficiently addressed. They are proposing two methods (Pristine Visual Evidence Reinforcement and Conflict-Aware Unreliable Memory Suppression) within a framework called Dual Evidential Adaptive Modulation to address these two problems.

Critical strengths

I think the problem tackled in this paper is an interesting and important problem and also the way that authors have broken it down into two components makes sense. I think it is a strength point that they have tried to show via ablations that each of their two modules has its own impact and those impacts are not overlapping or randomly impacting the results, but adding up. I also think the evaluation set up has a good coverage in terms of number of backbone models, communication topologies, and standard hallucination benchmarks. However, I have one important comment about the evaluation part which I've mentioned in the weaknesses section.

Critical weaknesses

I think the most important issue in the evaluation part is that although the main claim in the paper is mitigating the snowballing effect for the multi-agent hallucination, one of the main options for benchmarking this claim (ViF work) has not been included properly in the evaluation. Authors mentioned that they were not able to reproduce it, and although this could be a fair argument, but I think they could implement a simple version of it themselves, because this is one of the main options for benchmarking their central claim in the paper.

Questions for authors
  • Would it be possible that sign inconsistency between layers also be a result of complicated but correct memory? and if this is possible, can CUMS module potentially suppress some useful stuff? I think authors need to discuss this point in more details to help better clarify the CUMS functionality in their framework.

  • The proposed CUMS module, uses the inconsistency of cross-layer sign as a measure for conflicting memory aspect, however this is mostly discussed in a qualitative way, like heatmaps. It would help if authors can add some quantitative ways of demonstrating this point.

  • Can authors add more details on how they have used the methods they have listed in Table 3 (like: llava-rlhf, vista, etc) in the MAS setting? what did they choose for hyperparameters? is the model applied to every single agent? etc. This would help to better understand and interpret the reported results

  • Is applying PVER for all layers versus limiting CUMS to certain layers is a design choice or it is finding from authors' observations and analysis? Can authors clarify this?

5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer hu6x (1/3)

Q1: ViF is a directly relevant baseline for multi-agent hallucination snowballing. The inability to reproduce it does not fully justify its omission, and could the authors implement a simplified version for comparison?

A1: We thank the reviewer for this constructive suggestion. Following the reviewer’s suggestion, we additionally implemented ViF-TF, a training-free simplified version retaining ViF’s Attention Reallocation module without the trained visual-relay component. Within the rebuttal period, we evaluated this simplified baseline on LLaVA-1.5-7B using all 9,000 POPE questions and the open-ended MMHal-Bench task, as shown in Tab.1.

Tab. 1 Performance comparison of the baseline, ViF-TF, and our DELVE on POPE and MMHal-Bench with LLaVA-1.5-7B.

BackboneMethodPOPE-R F1 ↑POPE-P F1 ↑POPE-A F1 ↑POPE Avg. F1 ↑MMHal-Bench Score ↑
LLaVA-1.5-7BBaseline86.178.277.180.52.38
LLaVA-1.5-7BViF-TF87.379.478.281.62.29
LLaVA-1.5-7BDELVE88.083.779.283.62.40

Although ViF-TF achieves a modest improvement on POPE, our DELVE consistently outperforms ViF-TF across all reported metrics, further demonstrating the effectiveness of our method. In addition, we observe that while ViF-TF provides certain benefits on binary yes/no benchmarks such as POPE, its performance on open-ended generation tasks such as MMHal-Bench deteriorates compared with the baseline model. This degradation is reflected not only in the score drop from 2.38 to 2.29, but also in severe generation failures observed in its outputs, including a significant loss of coherence, frequent repetitive generation loops, and garbled text. The observations suggest that the additional training required by the ViF framework may be essential for mitigating such generation failures, but we are unable to reproduce this training process due to the lack of sufficient training details and reproducibility resources.

We are committed to fair and comprehensive comparisons and have made extensive efforts to reproduce ViF. As discussed in Appendix C.1, the difficulty extends beyond the incomplete inference implementation in the public release. ViF requires additional training, yet its paper and code fail to provide sufficient details regarding the training process, training data, and implementation configurations for the evaluated backbones, making the reported results difficult to reproduce and independently verify. In particular, the uncertainty regarding the training data prevents us from verifying that our implementation matches the original training conditions.

During our research, we have implemented the inference process according to the paper and fine-tuned the resulting model using COCO and additional LLaVA tuning image-text datasets. We denote this data-adapted reimplementation as ViF*. The results are reported below Tab. 2.

Tab. 2 Performance comparison of the baseline, ViF*, and our DELVE on POPE and MMHal-Bench with LLaVA-1.5-7B and Qwen3-VL-8B.

BackboneMethodPOPE-R F1 ↑POPE-P F1 ↑POPE-A F1 ↑POPE Avg. F1 ↑MMHal-Bench Score ↑
LLaVA-1.5-7BBaseline86.178.277.180.52.38
LLaVA-1.5-7BViF*85.375.675.178.72.14
LLaVA-1.5-7BDELVE88.083.779.283.62.40
Qwen3-VL-8BBaseline90.187.986.688.22.69
Qwen3-VL-8BViF*88.186.485.386.62.32
Qwen3-VL-8BDELVE91.588.187.288.92.75

ViF* underperforms the corresponding baselines on both architectures. Together with the insufficiently specified training conditions in the original work, these results suggest that the available implementation details do not provide a sufficient basis for a reliable and verifiable reproduction of the reported results. We therefore did not present these results as the performance of the ViF method, which could unfairly disadvantage it.

AuthorsOpenReview ↗

Rebuttal to Reviewer hu6x (2/3)

Q2: 1) Can cross-layer sign inconsistency also arise from complex but correct memory, causing CUMS to suppress useful information? 2) Beyond qualitative heatmaps, can the authors quantitatively validate the connection between inconsistency and unreliable upstream memory?

A2: We thank the reviewer for this constructive suggestion.

1) Yes. CUMS does not rely on additional large language models or RAG-based knowledge bases for factual verification. Instead, it applies lightweight modulation to memory-induced components based on cross-layer sign consistency, allowing even factually correct memories to undergo attenuation. As shown in the experiments in A2 2) and the Ablation Study (Section 4.2), such attenuation does not necessarily compromise downstream benefits. Although occasional correctness regressions may occur, the gains from hallucination reduction outweigh the potential loss of useful information.

2) To directly evaluate preservation of useful-memory benefits, we conducted a controlled receiver-level experiment on 59 audited MMHal questions using LLaVA-1.5-7B and Qwen3-VL-8B-Instruct, as shown in Tab.3. Each question was evaluated with a verified factually correct memory, a fact-preserving paraphrase, and a no-memory control. Both base models received identical frozen memories. Paired CUMS off/on runs shared the same image, question, memory, prompt, PVER tensor, and decoding settings. Correctness was assessed by a fixed, image-grounded automated judge blinded to the intervention. We define memory as useful when it corrects an otherwise incorrect no-memory answer with CUMS off, and evaluate whether this benefit survives with CUMS on.

Tab. 3: CUMS evaluation on 59 MMHal-Bench questions with correct and paraphrased memories. Counts (n/Nn/N) report retained correct answers, corrected errors, and preserved useful-memory benefits.

BackboneMemoryCorrect answers: off → onUseful benefits retained
LLaVA-1.5-7BOriginal27/59 → 25/599/9
LLaVA-1.5-7BParaphrase29/59 → 29/5912/12
Qwen3-VL-8B-InstructOriginal48/59 → 51/598/8
Qwen3-VL-8B-InstructParaphrase49/59 → 50/595/5

All observed useful-memory benefits are preserved under both original and paraphrased memories, despite nonzero disagreement and modulation in every useful-memory case. This preservation concerns the subset of memories that correct an otherwise incorrect no-memory answer. Across all responses, LLaVA exhibits two correctness regressions with original memories (27/59 to 25/59), whereas Qwen3 retains all 97/97 initially correct responses and corrects 4/21 initially incorrect responses across the two variants. These results directly support preservation of the observed downstream benefits of correct memories, rather than discrimination between factually correct and incorrect memories. We have incorporated these experiments and clarified their scope in Section 4.3.

Q3: How were the Table 3 baselines, such as LLaVA-RLHF and VISTA, adapted to the MAS setting? Please specify their hyperparameters and whether each method was applied to every agent.

A3: To ensure a fair comparison, all baselines are evaluated within the same MAS framework as DELVE, using the same fixed Random communication graph, upstream-memory construction, interaction turns, task prompts, and shared base generation configuration, including greedy decoding and consistent response-length limits. For each baseline experiment, we use the corresponding baseline model or method for every agent in the MAS, following the same communication and evaluation protocol as DELVE. Specifically, we integrate LLaVA-RLHF and RLSF-V through the authors’ official implementations, using their trained models as the agent backends. For the training-free baselines, including VISTA, SHIELD, and Saliency, the method-specific hyperparameters follow the authors’ official code without additional specific tuning. Their original inference-time mechanisms are preserved under the shared evaluation protocol.

We thank the reviewer for highlighting the need to make these implementation details more explicit. To address this concern and improve reproducibility, we have added a dedicated section (i.e., Appendix B.2) to our supplementary Appendix. This subsection describes the integration of each baseline into our MAS framework, its agent-level application, the source of its method-specific hyperparameter settings, and the shared prompting and generation configurations used for comparison. The newly added details are highlighted in red for ease of reference.

AuthorsOpenReview ↗

Rebuttal to Reviewer hu6x (3/3)

Q4: Is applying PVER to all layers while restricting CUMS to selected layers an empirical finding or a design choice? What observations support this choice?

A4: We thank the reviewer for this insightful question. The different layer scopes are design choices motivated by the distinct objectives of PVER and CUMS, with empirical support from our additional layer-ablation experiments and the parameter analysis.

1) As described in Section 3.2, PVER uses layer-specific pristine visual evidence with context-dependent dimension weighting to strengthen visual grounding. Applying this reinforcement across all layers allows visual evidence to support successive stages of representation refinement. Restricting PVER to a small subset of layers limits the stages at which this explicit reinforcement is applied. To empirically examine this design choice, we compared all-layer intervention against interventions applied to the first three, last three, and three randomly selected decoder layers, using LLaVA-1.5-7B on the POPE-A benchmark with a newly sampled random MAS graph topology. As shown below Tab. 4, all-layer reinforcement achieves better performance on POPE-A, supporting this choice in the evaluated settings.

Tab. 4 Effect of intervention layer selection on POPE-A with LLaVA-1.5-7B.

PVER intervention scopePOPE-A F1 (%) ↑
First 3 layers80.68
Last 3 layers81.10
Random 3 layers80.70
All layers82.12

2) Unlike visual reinforcement, CUMS selectively discounts upstream-memory contributions according to their cross-layer directional inconsistency. We compute this inconsistency and apply suppression within the last KlK_l layers to focus the intervention on representations closer to the output prediction. This design is motivated by the depth-dependent encoding of contextual knowledge studied by Ju et al. [A], which is already cited in Section 3.3. It is also consistent with the observations of Chuang et al. [B], who show that predictions of fact-critical tokens can continue to change substantially in later layers. These observations motivate assessing memory conflict near the final prediction stage, where the intervention is intended to be less sensitive to intermediate representational variations that may not persist to the output. To further address the reviewer’s concern, we also evaluated matched-strength uniform suppression across all decoder layers on POPE-A and MMHal-Bench, as depicted in Tab.5. The results further validate the effectiveness of our design rationale.

Tab. 5: Comparison of the full DELVE method with matched-strength uniform suppression baseline.

ModelVariantPOPE-A F1MMHal-Bench Score
Qwen3-VL-8BUni. Memory86.62.69
Ours87.22.75

The terminal-layer restriction also limits the additional layer-wise memory-evidence processing and token-level suppression operations. Our sensitivity analysis in Section 4.2 and Figure 3(c) shows relatively stable performance over a moderate range of KlK_l. We therefore adopt Kl=3K_l=3 as a practical balance between performance and the additional intervention overhead.

References:

[A] Ju, Tianjie, et al. "How large language models encode context knowledge? A layer-wise probing study." Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). 2024.

[B] Chuang, Yung-Sung, et al. "Dola: Decoding by contrasting layers improves factuality in large language models." International Conference on Learning Representations. Vol. 2024. 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.