acceptodds
Under review as a conference paper at ICLR 2027

CARE-Flow: Mitigating Inter-Image Hallucination in Multi-Image MLLMs

Abstract

Hallucination remains a major reliability challenge for multimodal large language models (MLLMs), and becomes more complex when reasoning across multiple images. Even when visual content remains grounded, models may select irrelevant evidence, omit required evidence, or incorrectly combine supported evidence. We call such failures in using grounded evidence across images inter-image hallucinations. Standard multimodal inference entangles determining what evidence a question requires, acquiring that evidence from the images, and forming the response, making it difficult to establish which visual information actually drives the inference. Because observed association cannot establish such dependence, the use of evidence must be tested through intervention. We propose CARE-Flow, a causal framework that decouples evidence specification and acquisition from response formation through an explicit evidence mediator. This mediator makes image-specific contributions independently testable before response generation and constrains downstream inference to the established evidence, yielding a constructed conditional front-door structure for tracing visual influence. Existing multi-image benchmarks mainly score final answers and provide limited diagnosis of how evidence use fails. We therefore introduce CARE-Bench, a 12,000-question benchmark that diagnoses failures in evidence formation and combination across multiple visual domains. Across three MLLMs and seven multi-image benchmarks, CARE-Flow consistently improves accuracy, outperforming the strongest baselines by 4.28–7.53 percentage points on CARE-Bench.

Primary area:
generative models
Keywords:
Multimodal models, multi-image, large language models, reasoning, hallucination

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cTGsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CARE-Flow is a training-free causal framework for reducing inter-image hallucination in multi-image MLLMs. The paper distinguishes inter-image hallucination — where visual evidence is grounded but used incorrectly across images — from conventional single-image hallucination. It identifies three failure modes: incorrect evidence selection (grounded substitution), missing required evidence (omission), and incorrect evidence combination (operator error).

The framework operates in three stages. First, it specifies the evidence structure required by the question before any response is generated, fixing the required operand roles and composition operator. Second, it causally certifies each proposed operand by intervening on image-specific attention value-output paths and comparing the necessity scores of declared versus undeclared sources. Operands that fail certification are marked as unknown rather than silently dropped. Third, it performs a hard reset discarding all visual states and runs a fresh text-only computation over only the certified evidence, preventing downstream inference from bypassing the established mediator. Responses are then verified against both a certified bridge and a structure-matched null bridge, with the front-door gap used to filter responses that are supported mainly by linguistic priors.

The paper also introduces CARE-Bench, a 12,000-question diagnostic benchmark covering operand substitution, operand omission, and operator errors across COCO-VQA, museum images, and controlled generated scenes. Experiments across three MLLMs and seven benchmarks show consistent accuracy improvements, with the largest gains on CARE-Bench. Ablations confirm that causal certification and the hard reset are the two most critical components.

Critical strengths

S1: Technically distinctive approach to evidence certification. Decomposing attention output by image source and intervening on value-output paths is more direct than methods that rely solely on attention magnitude to infer evidence provenance. Combining this with explicit evidence specification and a hard reset provides a principled mechanism for controlling which visual information enters downstream inference, rather than redistributing attention weights as in DAB.

S2: Ablation results support the core components. Table 3 shows that removing causal certification drops macro-average accuracy from 40.47 to 31.75, and removing the hard reset drops it to 35.20. Replacing causal certification with attention-based certification recovers only to 37.00, remaining 3.47 points below the full method. These gaps suggest the components are not interchangeable and each contributes meaningfully.

S3: Evaluation covers multiple models and public benchmarks. Results are reported across three frozen backbones on six public benchmarks plus CARE-Bench, with mean and standard deviation over five runs. Appendix analyses further break down results by model scale, domain, and evidence complexity, which strengthens the generalization claim.

Critical weaknesses

W1: The certification criterion may be too strict for redundant or jointly required evidence.

Certification requires every declared source to contribute more strongly than any undeclared source. This fails in two common cases. First, when two images contain the same object, the redundant image also receives a high necessity score, causing the declared source to fail certification even when it is correct. Second, for facts that require two images jointly — such as a change across views — intervening on each image alone produces low scores for both, so neither can be certified even though both are genuinely required.

The paper does not report how often certification rejects correct operands, what fraction of unknown-marked operands were semantically valid proposals, or how these rejections affect final accuracy. Experiments with redundant sources, jointly required evidence, and varying intervention strengths would clarify when the criterion is unnecessarily strict and at what cost.

W2: The distinction from DAB and related methods needs sharper articulation.

The paper compares against DAB (Dynamic Attention Balancing), which redistributes attention across images to reduce multi-image hallucination. The introduction states that CARE-Flow differs by explicitly testing source dependence through intervention rather than adjusting attention allocation. However, the paper does not include a direct analysis of when the two methods succeed and fail on the same examples, nor does it explain why attention redistribution is insufficient for the failure modes CARE-Flow targets. Since both methods operate without training and both address cross-image evidence use, a clearer empirical or mechanistic comparison — for example, showing cases where DAB fails due to insufficient source separation but CARE-Flow succeeds due to certification — would strengthen the contribution claim.

W3: The hard reset discards all visual computation, which may hurt performance when certified evidence is incomplete.

After certification, CARE-Flow removes all visual tokens and key-value caches and proceeds with text-only inference over the certified bridge. When certification fails on one or more operands — marking them as unknown — the downstream model must reason over an incomplete evidence set with no access to any residual visual signal. Table C.7 shows that atom-veracity errors account for a substantial and growing fraction of residual errors after CARE-Flow, suggesting that some failures arise from the model being unable to recover visual information that was not successfully certified. The paper does not ablate a softer fallback — for example, retaining low-confidence operands rather than marking them unknown, or preserving a limited visual context alongside the bridge — and it is unclear whether a less aggressive reset would improve performance on examples where certification partially fails.

W4: CARE-Bench is heavily dominated by one domain.

Table 5 shows that COCO-VQA accounts for 11,132 of the 12,000 questions (92.8%), with Te Papa contributing only 368 and Generated 500. The benchmark statistics and aggregate accuracy are therefore almost entirely determined by the COCO-VQA domain. Domain-macro reporting is used in the appendix but not consistently in the main results, which makes it difficult to assess how well CARE-Flow generalizes to the less common domains. The imbalance also raises the question of whether the benchmark construction pipeline scales to more visually diverse or linguistically varied settings, since the vast majority of instances come from a single well-studied dataset.

Questions for authors

For facts that require two images jointly — such as identifying that the same object appears across views — intervening on one image at a time will produce low necessity scores for both, since neither alone is sufficient. Does this mean the framework systematically fails to certify valid multi-source operands of this type? And has the authors considered a joint intervention approach, for example masking pairs of images together, to handle such cases?

4 comments
AuthorsOpenReview ↗

Response to cTGs (1/4)

We thank the reviewer for the constructive feedback on source certification and multi-image evidence use.

W1: Redundant and jointly required evidence.

CARE-Flow fixes question-required evidence roles before binding sources. The original criterion Dj>0D_j>0 compares individual declared-source necessity with competing-source necessity and therefore tests conditional source dependence.

For redundant evidence, multiple images may support the same fact, but acceptable source attribution must satisfy the question-required role; visually matching yet question-irrelevant sources are distinguished from valid alternatives. For jointly required evidence, non-decomposable facts (e.g., cross-view identity) may retain a multi-image source set SjS_j. Even when a source is insufficient in isolation, weakening it while retaining the other images can reduce support for the joint fact, yielding positive conditional necessity NjkN_{jk}.

To evaluate these cases directly, we conducted additional controlled experiments comparing original individual-source and joint + source-constrained certification across both evidence types and three intervention strengths (Table R2). At ϵ=0.9\epsilon=0.9, redundant valid rejection decreases from 66.33% to 7.14%, correct acceptance rises from 33.67% to 92.86%, and final-answer accuracy increases from 31.63% to 87.76%; false acceptance changes from 6.12% to 6.63%. For jointly required evidence, valid rejection decreases from 31.63% to 11.22%, correct acceptance rises from 68.37% to 88.78%, and final-answer accuracy improves from 64.29% to 83.67%, with false acceptance changing from 2.04% to 3.06%.

Across ϵ∈{0.3,0.6,0.9}\epsilon\in\{0.3,0.6,0.9\}, joint + source-constrained certification lowers valid rejection for both evidence types at every tested strength (Table R2). Group-level support combined with question-specific source constraints consistently recovers valid operands under redundant and joint source dependencies. The reported false-acceptance and retention rates further characterize the tradeoff between valid-evidence recovery and source selectivity.

We have summarized the certification findings in Section 6.3 and added the complete intervention-strength analysis to the Appendix, with the new material highlighted in blue.

Table R2. Controlled diagnostics on redundant and jointly required evidence (%). Valid rejection and correct acceptance are fractions of valid proposals; false acceptance is the fraction of invalid proposals accepted; retention is the fraction of all proposals retained. Accuracy is final-answer accuracy on the corresponding controlled questions.

Evidence typeεCertificationValid rejection ↓Correct acceptance ↑False acceptance ↓RetentionAccuracy ↑
Redundant0.3Original69.39%30.61%6.63%14.63%28.57%
Redundant0.3Joint + source constraint12.24%87.76%8.16%34.69%80.61%
Redundant0.6Original67.35%32.65%6.12%14.97%30.61%
Redundant0.6Joint + source constraint8.16%91.84%7.14%35.37%85.71%
Redundant0.9Original66.33%33.67%6.12%15.31%31.63%
Redundant0.9Joint + source constraint7.14%92.86%6.63%35.37%87.76%
Jointly required0.3Original38.78%61.22%3.06%22.45%55.10%
Jointly required0.3Joint + source constraint16.33%83.67%4.59%30.95%75.51%
Jointly required0.6Original34.69%65.31%2.55%23.47%60.20%
Jointly required0.6Joint + source constraint13.27%86.73%3.57%31.29%80.61%
Jointly required0.9Original31.63%68.37%2.04%24.15%64.29%
Jointly required0.9Joint + source constraint11.22%88.78%3.06%31.63%83.67%
AuthorsOpenReview ↗

Response to cTGs (2/4)

W2: Comparison with DAB.

DAB enhances multi-image processing through token-level attention balancing, while CARE-Flow controls inference-level evidence formation and use. DAB redistributes attention over image tokens; CARE-Flow explicitly specifies question-required evidence roles, tests operand-level source dependence through interventions, and constrains how certified operands are composed. These operations directly address source-role assignment and cross-image evidence-combination errors. In source identification, intervention-based attribution achieves 0.72 AUROC / 0.58 AUPRC versus 0.61/0.42 for attention-based attribution.

To evaluate the two methods on the same individual questions, we conducted additional paired evaluations on 500 Generated-domain questions per backbone (Table R3). CARE-Flow achieves net accuracy gains of +5.60%, +5.60%, and +5.00% over DAB on LLaVA-NeXT-7B, Qwen3.5-9B, and InternVL3-8B. CARE-Flow-only correct cases outnumber DAB-only correct cases for all three backbones, demonstrating consistent question-level advantages in the paired comparison. A matched certification-rule comparison also yields +4.20%, +4.00%, and +4.40% over attention-based certification, supporting the contribution of operand-level source testing. The reconstructed CARE-Flow accuracies agree with the Appendix Generated-domain results.

We have incorporated the paired gains into Section 6.3 and added the full comparison to the Appendix, with both revisions highlighted in blue.

Table R3. Paired outcome analysis on Generated-500 (n=500n=500 per backbone). Full method compares CARE-Flow with DAB; Cert. rule only compares causal with attention-based certification. Each row partitions matched questions into four disjoint outcomes totaling 100%. Net gain = CARE only − Baseline only.

ComparisonBackboneBaselinenBoth correct (%)Baseline only (%)CARE only (%)Both wrong (%)
Full methodLLaVA-NeXT-7BDAB50028.005.6011.2055.20
Full methodQwen3.5-9BDAB50042.603.208.8045.40
Full methodInternVL3-8BDAB50027.004.609.6058.80
Cert. rule onlyLLaVA-NeXT-7BAttn. Cert.50029.605.409.6055.40
Cert. rule onlyQwen3.5-9BAttn. Cert.50044.602.806.8045.80
Cert. rule onlyInternVL3-8BAttn. Cert.50027.404.809.2058.60
AuthorsOpenReview ↗

Response to cTGs (3/4)

W3: Hard reset and incomplete evidence.

Hard reset restricts downstream inference to the established evidence bridge, ensuring that subsequent generation uses certified operands. The existing ablation shows that removing hard reset reduces macro-average accuracy from 40.47% to 35.20%.

Appendix C.7 also shows that the absolute rates of Atom Conflict and A-V (incorrect answers associated with atom conflicts) both decrease across all three backbones. Their higher share among residual errors reflects larger reductions in other error categories.

To evaluate this concern directly, we conducted additional certification-stratified experiments on hard reset and evidence recovery using 1,500 model–question records (500 Generated-domain questions × three backbones; Table R4): 780 fully certified and 720 partially certified, with groups fixed by the original CARE-Flow decisions. For fully certified cases, hard reset improves accuracy from 62.18% to 66.67% and lowers structural hallucination from 31.67% to 26.92%. The same pattern holds for partially certified cases (13.47% → 16.11% accuracy; 60.56% → 57.64% structural hallucination).

Targeted evidence recovery further improves partially certified cases to 25.14% accuracy and 48.47% structural hallucination, while the same recovery without a final reset yields 22.08% and 51.81%. This comparison distinguishes two complementary functions: recovery supplies previously unresolved evidence, and reset enforces the use of the resulting certified bridge. Among the tested strategies for partially certified records, targeted recovery followed by hard reset achieves the highest accuracy and lowest structural hallucination.

We have added the certification-stratified findings to Section 6.3 and the full hard-reset and recovery evaluation to the Appendix, with the additions highlighted in blue.

Table R4. Certification-stratified evaluation on 1,500 model–question records. Accuracy, structural hallucination, and other wrong are mutually exclusive outcomes and sum to 100% within each subgroup (up to rounding). Subgroups are fixed across strategies.

Certification statusStrategynAccuracy ↑Structural halluc. ↓Other wrong ↓
Fully certifiedCARE-Flow78066.67%26.92%6.41%
w/o Hard Reset78062.18%31.67%6.15%
Partially certifiedCARE-Flow72016.11%57.64%26.25%
w/o Hard Reset72013.47%60.56%25.97%
Low-confidence fallback72017.08%56.11%26.81%
Targeted evidence recovery72025.14%48.47%26.39%
Recovery w/o Hard Reset72022.08%51.81%26.11%
AuthorsOpenReview ↗

Response to cTGs (4/4)

W4 — CARE-Bench domain imbalance and diagnostic coverage.

We address two concerns: whether COCO-VQA's 92.8% share drives the gains and whether additional questions contribute distinct diagnostic structures. CARE-Bench spans complementary sources: COCO-VQA (everyday scenes), Te Papa (less familiar real images), and Generated (controlled visual facts). The construction pipeline validates evidence and answer uniqueness in eligible real-image pools; the final domain sizes reflect valid-question yield.

To evaluate the effect of domain weighting, we conducted additional balanced evaluations (Table R5). Gains are positive for all nine backbone–domain combinations (+6.40% to +14.19%), including Te Papa and Generated. With equal domain weights, macro gains are +12.84%, +9.66%, and +7.13% on LLaVA-NeXT-7B, Qwen3.5-9B, and InternVL3-8B. Equal-size evaluation (368 questions/domain, n=1,104n=1{,}104) yields gains of 12.84 ± 0.82, 9.66 ± 0.76, and 7.13 ± 0.73 (mean ± SD, %). COCO-VQA accounts for one-third of the balanced score. Therefore, CARE-Bench's domain imbalance does not alter our main empirical conclusion: CARE-Flow consistently improves performance across all three visual domains.

We also measured new diagnostic structures discovered per additional 100 questions under coarse and fine taxonomies. Marginal discovery decreases from the first to the last window in all three domains (Table R5). These results quantify diminishing marginal structural novelty as the benchmark grows and show that successive questions increasingly consolidate coverage of the diagnostic configurations captured by both taxonomies.

We have added the domain-balanced findings to Sections 5 and 6.2 and the full balanced and diagnostic-discovery analyses to the Appendix, with the additions highlighted in blue.

Table R5. A: Domain-macro accuracy and domain-wise rates on all 12,000 questions (COCO-VQA / Te Papa / Generated). B: Balanced evaluation with 368 questions per domain, reporting mean ± SD (%). C: New coarse/fine evidence configurations per 100 questions, comparing first and last windows. Columns 5–6 mean Vanilla/CARE-Flow for A–B and First 100/Last 100 for C.

AnalysisBackbone / DomainnMetricVanilla / First 100CARE-Flow / Last 100Difference
A: Domain macroLLaVA-NeXT-7B12000Accuracy (%)33.1746.0112.84
A: Domain ratesLLaVA-NeXT-7B12000COCO / Te Papa / Generated (%)27.18 / 45.92 / 26.4038.72 / 60.11 / 39.2011.54 / 14.19 / 12.80
A: Domain macroQwen3.5-9B12000Accuracy (%)44.0553.719.66
A: Domain ratesQwen3.5-9B12000COCO / Te Papa / Generated (%)41.53 / 47.61 / 43.0050.21 / 59.51 / 51.408.68 / 11.90 / 8.40
A: Domain macroInternVL3-8B12000Accuracy (%)30.8537.987.13
A: Domain ratesInternVL3-8B12000COCO / Te Papa / Generated (%)29.02 / 33.32 / 30.2036.63 / 40.71 / 36.607.61 / 7.39 / 6.40
B: BalancedLLaVA-NeXT-7B1104Accuracy: mean ± SD (%)33.17 ± 0.8646.01 ± 0.9412.84 ± 0.82
B: BalancedQwen3.5-9B1104Accuracy: mean ± SD (%)44.05 ± 0.9553.71 ± 0.969.66 ± 0.76
B: BalancedInternVL3-8B1104Accuracy: mean ± SD (%)30.85 ± 0.8837.98 ± 0.937.13 ± 0.73
C: DiscoveryCOCO-VQA11132New coarse configs /100: first → last23.290.00-23.29
C: DiscoveryCOCO-VQA11132New fine configs /100: first → last26.280.00-26.28
C: DiscoveryTe Papa368New coarse configs /100: first → last7.460.07-7.39
C: DiscoveryTe Papa368New fine configs /100: first → last8.680.49-8.19
C: DiscoveryGenerated500New coarse configs /100: first → last77.432.07-75.36
C: DiscoveryGenerated500New fine configs /100: first → last159.2220.58-138.64

Question for Authors — Joint intervention. CARE-Flow supports multi-image source sets SjS_j. Individual-source necessity is measured with the other images retained, allowing each source to contribute conditionally to a joint fact; joint intervention measures collective support. At ϵ=0.9\epsilon=0.9, joint + source-constrained certification reduces valid rejection from 31.63% to 11.22%, improves correct acceptance from 68.37% to 88.78%, and changes false acceptance from 2.04% to 3.06% (Table R2); valid acceptance also improves at ϵ=0.3\epsilon=0.3 and 0.6. These results show that individual interventions identify conditional contributions, while group-level certification recovers additional valid jointly supported evidence.

Reviewer_hobyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies a specific type of hallucination in multi-image MLLMs, referred to as inter-image hallucination, where models may select irrelevant visual evidence, omit required evidence, or incorrectly combine evidence across images. To address this problem, the authors propose CARE-Flow, a causal framework that divides the inference process into three stages. The first stage specifies the evidence required to answer the question and how it should be combined. The second stage uses causal interventions to verify the source dependence of individual evidence items and filters out uncertified evidence. The third stage removes the original visual inputs and performs response generation or verification using only the certified evidence and the predefined composition operator.

In addition, the authors introduce CARE-Bench, a benchmark designed to evaluate inter-image hallucinations through three types of errors: operand substitution, operand omission, and operator error.

Experiments across three MLLM backbones and seven benchmarks show that CARE-Flow consistently outperforms the compared baselines in terms of accuracy.

Critical strengths
  • Problem formulation and benchmark. The paper identifies and formalizes inter-image hallucination, a relatively new research problem that includes failure modes specific to multi-image reasoning. The authors also introduce CARE-Bench, a benchmark designed to evaluate these errors and facilitate future research in this direction.

  • Comprehensive experiments. The paper conducts extensive experiments across three MLLM backbones and seven benchmarks, where CARE-Flow consistently outperforms all compared baselines. The authors also provide ablation studies to demonstrate the contributions of individual components.

Critical weaknesses
  • Clarity and presentation. While this is not a major methodological weakness, the notation and explanations in Section 4 are often unclear. Several important symbols and operations are introduced without sufficiently explicit definitions. For example, in Section 4.1, the first formula introduces symbols such as GG and FjF_j without clearly explaining their meanings. In Section 4.2, the mapping performed by Certify⁡(E~,N)\operatorname{Certify}(\widetilde{E}, N) is not explicitly defined and needs to be inferred from the surrounding text and Algorithm 1 in the appendix. Additionally, in Section 4.3, the role of ΔFD\Delta_{FD} in the final inference procedure is primarily conveyed through equations, with limited textual explanation. Although these details can eventually be understood, they require substantial effort from the reader and make the methodology unnecessarily difficult to follow.
Questions for authors
  • Figure 3(a). Could the authors clarify how the heatmap in Figure 3(a) is constructed, particularly how the head-causal alignment scores are computed? The heatmap shows noticeable clusters of high values that are contiguous along the attention-head index dimension. Since neighboring attention-head indices do not necessarily imply functional similarity, it is unclear why such patterns emerge. Were any sorting, smoothing, interpolation, or other post-processing techniques applied when generating the heatmap?
2 comments
AuthorsOpenReview ↗

Methodological Clarity and Figure Interpretation

We thank the reviewer for the helpful suggestions on methodological clarity and figure interpretation.

Point 1 — Notation and methodology. We have clarified the methodological notation in the revised main text, with the additions highlighted in blue (Sections 4.1–4.2). In Section 4.1, G=(g1,…,gK)\mathbf G=(g_1,\ldots,g_K) gates outgoing visual attention value-output paths (G=1\mathbf G=\mathbf1 is the original computation), and Fj(g)F_j(\mathbf g) is the mean teacher-forced log-likelihood of atom cjc_j under these gates. In Section 4.2, Certify⁡(E~,N)\operatorname{Certify}(\widetilde{\mathcal E},N) uses source-necessity scores NN to compute DjD_j, retains cjc_j when Dj>0D_j>0, and otherwise sets its value to unknown while preserving the role and source. Section 4.3 defines ΔFD(a)=R1(a)−R0(a)\Delta_{\mathrm{FD}}(a)=R_1(a)-R_0(a) as the increase in verifier support from certified evidence relative to the matched null mediator. A candidate is valid if R1(a)>0R_1(a)>0 and ΔFD(a)>0\Delta_{\mathrm{FD}}(a)>0; the highest-scoring valid candidate is selected, with abstention or regeneration if none qualifies.

Point 2 — Construction and interpretation of Figure 3(a). Our visualization is inspired by the head-oriented causal attribution analysis of Wang et al., From Heads to Neurons: Causal Attribution and Steering in Multi-Task Vision–Language Models (Findings of ACL 2026). The purpose of the atlas is to identify where image-level attention allocation aligns with the intervention-measured causal support of image sources for atomic evidence. It reveals how this alignment varies across decoder layers and heads, with higher alignment concentrated in a subset of heads, predominantly in middle and later layers. This head-level view complements our source-identification experiment by showing where source-sensitive attention patterns emerge within the model. We compute HCA on LLaVA-NeXT-Interleave-7B using 59 COCO-VQA and 100 Te Papa questions, yielding 434 fixed atomic evidence items. The procedure is:

(1) Source attention. For atom jj, layer ℓ\ell, head hh, and image kk, normalize image-level attention mass AjkℓhA_{jk}^{\ell h}:

pjkℓh=Ajkℓh∑rAjrℓh.p_{jk}^{\ell h}=\frac{A_{jk}^{\ell h}}{\sum_r A_{jr}^{\ell h}}.

(2) Causal source support. Intervene on each image's outgoing visual value-output paths and measure the change in the atom log-likelihood, yielding necessity NjkN_{jk}. Normalize positive contributions:

qjk=[Njk]+∑r[Njr]+.q_{jk}=\frac{[N_{jk}]_+}{\sum_r[N_{jr}]_+}.

(3) Head-Causal Alignment. Compare the two image distributions using base-2 Jensen–Shannon divergence:

HCAℓh=1−Ej[JSD2(pjℓh∥qj)].\mathrm{HCA}_{\ell h}=1-\mathbb E_j[\mathrm{JSD}_2(p_j^{\ell h}\Vert q_j)].

Of 434 fixed atoms, 50 have no positive source contribution and cannot yield a normalized qjq_j. We therefore average over 384 valid atoms. HCA lies in [0,1][0,1], with higher values indicating stronger attention–causal-source agreement.

Head ordering and rendering. Building on the head-oriented visualization of Wang et al. (Findings of ACL 2026), we further introduce head-column reordering to improve visual clarity. Specifically, we reorder the 32 head columns using their 17-dimensional HCA profiles over decoder layers 16–32. Euclidean-distance Ward clustering forms five clusters, arranged with higher-alignment clusters centrally and lower-alignment clusters toward the sides. The same permutation is applied to all 32 layers, explaining the contiguous bright regions. The heatmap uses contrast stretching in its color mapping and interpolation="nearest" for cellwise rendering; no smoothing or numerical interpolation is applied. The code for Figure 3(a) has already been released in the figures/ directory of our anonymous repository, with the original HCA data included as comments in the source code.

Thank you for the detailed response and additional experiments. Based on the authors' clarifications, while also acknowledging the concern raised by Reviewer cTGs regarding the limitations of the certification criterion in handling redundant evidence, I believe my initial rating remains appropriate and will keep it unchanged.

Reviewer_um5HOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the hallucination in MLLMs during cross-image reasoning, where visual evidence exists but is erroneously selected, omitted, or combined. The paper proposes CARE-Flow, a causal framework that decouples evidence retrieval from answer generation, thereby constraining reasoning within the bounds of available evidence. Furthermore, the paper introduces CARE-Bench to evaluate evidence formation and combination during cross-image reasoning.

Critical strengths
  1. Problem Formulation. The paper separates evidence formation from evidence combination, offering a finer-grained analysis of how hallucinations arise in cross-image reasoning.
  2. Comprehensive Experiments. The experiments not only demonstrate the effectiveness of CARE-Flow but also analyze its efficiency trade-offs. The experimental coverage is thorough.
Critical weaknesses

The causal conclusion of CARE-Flow is conditional. It guarantees only that the answer can be influenced by visual information solely through the evidence bridge, but it does not guarantee that the evidence bridge itself is faithful, complete, or role-correct.

Questions for authors

How does CARE-Flow ensure that the evidence bridge is faithful, complete, and role-correct?

2 comments
AuthorsOpenReview ↗

Reliability of CARE-Bridge

We thank the reviewer for the thoughtful comments on the reliability of CARE-Bridge.

Evidence-bridge correctness.

Theorem 1 establishes the mediated visual-information pathway. We directly evaluate the reliability of the evidence admitted to CARE-Bridge along three complementary dimensions:

Faithfulness. Source-specific interventions test whether a proposed operand causally depends on its declared visual sources. The criterion Dj>0D_j>0 requires declared sources to contribute positively and more strongly than competing sources. Source identification achieves 0.72 AUROC / 0.58 AUPRC, outperforming attention (0.61/0.42) and masking (0.52/0.38).

Completeness. The Specify stage fixes the question-required evidence roles before acquisition. Every role remains explicitly represented in CARE-Bridge, with uncertified values marked unknown to preserve the specified evidence structure and identify unresolved roles. Table 2 reports lower operand-omission errors across all three backbones.

Role correctness. Proposed operands are assigned to question-specified roles. Source contrast evaluates support from declared versus competing images, while the fixed operator preserves the required composition. Table 2 reports reduced operand-substitution and operator errors across all three backbones.

To assess bridge correctness, we conducted new bridge-level experiments on 500 Generated-domain questions per backbone (Table R1). Compared with the proposed bridge, CARE-Flow improves factual faithfulness by 7.41%–10.65% and role accuracy by 6.64%–10.43%, with corresponding question-level completeness decreases of 2.00%–2.80%. At the same observed retention as attention-based certification, CARE-Flow achieves higher faithfulness, completeness, role accuracy, and role recall for all three backbones. These matched-retention results demonstrate that CARE-Flow fills the same proportion of evidence-role slots with more source-supported and role-correct operands, directly strengthening the reliability of CARE-Bridge.

We have added a concise summary of these bridge-level results to Section 6.3 and the full evaluation to the Appendix, with the additions highlighted in blue.

Table R1. Bridge-level reliability on CARE-Bench Generated-500 (n=500n=500 per backbone; all metrics in %). Faithful: retained operands supported by declared sources. Complete: questions with every required role correctly filled. Role Acc.: role correctness among retained operands. Retention: non-unknown role slots. Role Recall: gold roles filled by supported, role-correct operands. All methods share the initial proposals.

BackboneMethodnFaithful ↑Complete ↑Role Acc. ↑Retention ↑Role Recall ↑
LLaVA-NeXT-7BProposed Bridge50078.40%41.20%77.16%95.29%74.91%
Attn. Cert.50082.48%31.20%81.02%80.59%66.17%
CARE-Flow50089.05%38.40%87.59%80.59%73.03%
Qwen3.5-9BProposed Bridge50087.80%63.80%87.20%96.47%86.14%
Attn. Cert.50090.41%53.80%89.04%85.88%79.28%
CARE-Flow50095.21%61.80%93.84%85.88%83.65%
InternVL3-8BProposed Bridge50082.10%52.80%80.86%95.29%77.40%
Attn. Cert.50086.33%42.80%84.89%81.76%69.29%
CARE-Flow50091.37%50.80%89.21%81.76%75.22%
Reviewer_um5HOpenReview ↗

Reviewer Response

Thank you for the additional experimental results and the detailed clarification. The authors' response has satisfactorily addressed my concerns, and I will raise my rating to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.