acceptodds
Under review as a conference paper at ICLR 2027

EVIDENCE-GUIDED MULTIMODAL LARGE LANGUAGE MODELS VIA VISUAL CAUSAL REASONING

Abstract

Multimodal large language models (MLLMs) can answer open-ended questions about images, yet their outputs can be driven by linguistic priors that are only weakly supported by the visual input. We introduce EG-MLLM, an evidence guided architecture that inserts an explicit, testable evidence path between the vision encoder and language decoder. A prompt-conditioned selector identifies task-relevant visual tokens; a sparse structured reasoner models their dependen cies; counterfactual token interventions distinguish answer-critical evidence from matched nuisance context; and a bottleneck cross-attention adapter injects the resulting representation into the decoder. The training objective combines autore gressive likelihood, evidence consistency, and an intervention margin that requires removing selected evidence to affect the target likelihood more than removing nuisance tokens. We evaluate the implemented evidence pathway with a controlled 6,000-instance evidence-binding experiment and four 2,000-instance test conditions defined by fixed shortcut and noise settings. EG-MLLM reaches 98.65% accuracy under a deliberately reversed shortcut correlation and 95.40% under heavy nuisance corruption while maintaining 98.33% and 94.80% evidence precision, respectively. A matched run without the intervention objective obtains nearly identical answer accuracy but satisfies the intervention margin on only 49.65% of shortcut-shift cases, compared with 98.30% for the full model. These results show that sparse evidence selection, structured relation binding, and intervention-sensitive training can jointly improve robustness and make the model’s supporting evidence explicitly testable

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language models, evidence‑guided reasoning, visual grounding, counterfactual intervention, multimodal faithfulness, hallucination mitigation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_kqGoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes EG-MLLM, an evidence-guided architecture combining prompt-conditioned token selection, a sparse graph reasoner, a bottleneck adapter, and an intervention-margin objective. The objective encourages removing selected evidence to reduce the target likelihood more than removing matched nuisance tokens. Evaluation is restricted to a synthetic binary relation task with 6,000 training examples and four 2,000-example test conditions. The full model achieves 98.65% accuracy under shortcut reversal and 95.40% under nuisance corruption. Removing the intervention objective leaves accuracy essentially unchanged but reduces intervention-margin satisfaction from 98.30% to 49.65%. No end-to-end natural-image MLLM results are reported.

Critical strengths
  • The controlled diagnostic makes evidence quality directly measurable. Section 4 and Appendix B specify known answer-critical tokens and deliberately vary shortcut correlation and nuisance corruption. This enables evaluation beyond answer accuracy and provides a useful setting for examining whether a model selects relevant evidence under distribution shifts.

  • The intervention-loss ablation isolates a meaningful behavioral difference. The matched comparison in Section 4.4 reports essentially unchanged shortcut-shift accuracy—98.80% without the loss versus 98.65% with it—but substantially different intervention-margin satisfaction. This supports the narrow conclusion that the objective changes the model’s response to the specified interventions. The authors also appropriately distinguish this operational criterion from identifiable causal effects in Section 3.4.

Critical weaknesses
  • W1 — The evidence does not establish the proposed contribution in the intended MLLM setting. All numerical results come from comparing two scalar attributes encoded in 32-dimensional synthetic token vectors, with exactly two ground-truth evidence tokens (Section 4.1; Appendix B). This bypasses visual feature extraction from images, natural-language understanding and generation, ambiguous evidence, and imperfect evidence supervision. The 7B-class configuration and parameter counts are architectural specifications rather than evidence of successful integration or training. Section 5 explicitly acknowledges this limitation, which is welcome, but does not resolve it. As submitted, the paper establishes a small diagnostic demonstration rather than the effectiveness of an evidence-guided MLLM. Evaluation on an actual vision-language backbone with a documented, practically obtainable supervision scheme is necessary to support the central contribution.

  • W2 — The experiments do not adequately identify the benefits of the proposed mechanisms. In Section 4.2, Soft Attention receives no evidence supervision, whereas the evidence-selection variants do. Top-K + Evidence simultaneously removes the graph and adapter and uses a pooled readout that the authors state loses role–attribute binding. Its near-chance performance therefore does not establish that the proposed graph is necessary or preferable to a simpler binding-capable model. Comparisons with matched supervision and capacity, alongside separate graph and adapter ablations, are missing. Moreover, the intervention loss does not improve accuracy in Table 1; its demonstrated benefit is satisfaction of the same 0.15-margin criterion used for training. This is valid evidence of objective compliance, but provides limited independent support for broader faithfulness. Evaluation under alternative interventions or controlled changes to the true causal attributes would help distinguish semantic evidence dependence from sensitivity to the trained masking procedure.

Questions for authors
  • Clarification — diagnostic training and supervision: What is the exact objective used for each diagnostic model, including the evidence-supervision loss and its weight? How does this objective correspond to Equation 11, and which components replace the vision encoder and autoregressive decoder?

  • Evidence request — attribution of architectural gains: Under identical evidence supervision and comparable capacity, how does EG-MLLM compare with a simple role-aware readout or standard attention model that preserves role–attribute binding? Can the graph and adapter contributions be isolated separately? These comparisons would materially affect W2.

  • Clarification and evidence request — intervention interpretation: Are selection and graph construction recomputed after masking, and can unselected tokens reach the predictor through another pathway? What are the separate distributions of ΔE\Delta_E and ΔN\Delta_N, and does the reported gain persist under an intervention other than the masking procedure used in training? This would clarify whether improved margin satisfaction reflects transferable evidence dependence.

Reviewer_YcycOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EG-MLLM, an evidence-guided framework designed to make multimodal language models rely more on relevant visual evidence rather than correlations or language priors. The method combines prompt-conditioned visual token selection, sparse graph-based reasoning, counterfactual evidence interventions, and a bottleneck adapter to incorporate selected evidence into the language decoder. The authors introduce an intervention-based training objective that encourages removing relevant evidence to have a greater impact on the prediction than removing irrelevant tokens. To validate the method, they conduct experiments on synthetic feature vectors, using 6000 training instances and four test conditions with 2000 instances each. Results show strong robustness to shortcut correlations and noise, while the intervention objective substantially improves evidence-dependence scores without improving prediction accuracy. However, the proposed framework is not evaluated on real images or pretrained MLLMs.

Critical strengths

I think the motivation is meaningful, as correct predictions do not necessarily mean that a model relies on the appropriate visual evidence. Encouraging models to ground their answers in relevant visual information could help improve their reliability and reduce hallucinations.

Critical weaknesses

The experimental results do not sufficiently support the paper's central claims. The paper argues that correct predictions do not necessarily imply reliance on appropriate visual evidence, and proposes an intervention-based method to improve evidence-grounded reasoning. However, the experiments are conducted entirely on synthetic feature vectors rather than real images or pretrained MLLMs. More importantly, the main improvement from the intervention objective is measured using essentially the same criterion that the objective explicitly optimizes, while answer accuracy remains nearly identical or slightly worse. Therefore, the results do not convincingly demonstrate that the proposed method improves visual grounding, reduces reliance on spurious visual or linguistic cues, or mitigates hallucinations in real-world MLLMs.

Also, some figures are difficult to read and contain visible text errors. In particular, Figure 2 includes noticeable typos and distorted text that should have been corrected before submission.

Questions for authors

Please see weakness.

Reviewer_MPyaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes EG-MLLM, an evidence-guided architecture combining prompt-conditioned token selection, a sparse graph reasoner, a bottleneck adapter, and an intervention-margin objective. The objective encourages masking selected evidence to reduce the target log-likelihood more than masking matched nuisance tokens. The empirical evaluation consists entirely of a synthetic binary classification task with 16 feature-vector tokens, two ground-truth evidence tokens, and an answer-correlated shortcut. The structured variants achieve high accuracy under shortcut reversal and nuisance corruption. Adding the intervention objective substantially increases satisfaction of the corresponding intervention margin, while leaving answer accuracy essentially unchanged.

Critical strengths
  1. The distinction between prediction accuracy and evidence dependence is useful. Testing whether selected evidence functionally influences a prediction is a reasonable complement to reporting selection precision or attention weights.

  2. The controlled setup permits targeted diagnostics. Known evidence tokens and manipulated shortcut correlations allow the authors to examine evidence selection, relational binding, and intervention sensitivity separately.

Critical weaknesses

W1. The experiments do not substantiate the proposed MLLM contribution.

Section 3 describes a vision encoder, an autoregressive language decoder, teacher-forced language modeling, and adapters inserted into decoder layers. However, all reported numerical results come from a small synthetic classification experiment using 32-dimensional token proxies. No experiment demonstrates integration with an actual pretrained MLLM, natural-image understanding, language generation, or hallucination reduction.

Synthetic studies can provide valuable evidence, but the connection between this diagnostic and the claimed application remains untested. The reference 7B configuration, computed parameter counts, and dataset adapters do not establish that the proposed architecture works in the setting motivating the paper.

W2. The comparisons confound evidence supervision with architectural changes.

Section 4.2 states that Soft Attention receives no evidence supervision, whereas Top-K + Evidence does. The evidence-supervision objective and its application across the EG-MLLM variants are not clearly specified in Section 3.6.

Furthermore, Top-K + Evidence removes both the graph and the adapter and uses a pooled readout. Its poor performance therefore does not isolate the contribution of graph-based reasoning. Pooling raw role and attribute features can discard the information needed for the ordered comparison by construction.

An evidence-supervised attention baseline and a parameter-matched, role-aware non-graph baseline would help determine whether the improvements arise from supervision, representation capacity, or the proposed graph and adapter. The current comparisons do not separate these factors.

W3. The intervention result provides limited independent evidence of faithfulness.

The main intervention result measures satisfaction of the same margin used for training, with the same threshold of 0.15. Improved performance on held-out instances demonstrates generalization of the optimized criterion, but does not independently establish faithful reasoning.

Specifically, the likelihood difference simplifies to:

Δ_E − Δ_N = ℓ(Z \ N) − ℓ(Z \ E).

The objective therefore ranks two perturbed-input likelihoods. It does not by itself ensure that masking evidence changes the predicted answer, nor does it exclude learning a response specific to the masking mechanism. The paper also acknowledges that representation masking can create out-of-distribution states.

Reporting the individual likelihood drops, prediction changes, and sensitivity to alternative interventions would strengthen the interpretation. This matters because removing the intervention loss leaves shortcut-shift accuracy essentially unchanged: 98.80% without the loss versus 98.65% with it. The demonstrated benefit is therefore narrower than a general improvement in robustness or reasoning quality.

W4. The executed experiment is insufficiently specified.

The synthetic experiment is the sole empirical basis of the paper, yet its training configuration is less clearly documented than the full-scale reference configuration. Important omissions include the evidence-supervision loss, the classification readout and its relationship to the autoregressive objective, and the concrete instantiation of evidence–text consistency in the synthetic setting. The nuisance-corruption magnitude and several diagnostic training details are also unclear.

All results use a single seed, without reporting variability across training runs. Although the manuscript refers to repository files and unit tests, these references do not resolve the missing methodological details in the supplied PDF.

Questions for authors
  1. What exactly replaces the vision encoder and language decoder in the executed synthetic experiment? Which components and losses from Section 3 are actually used?

  2. What is the evidence-supervision objective, and which methods receive it? Can the authors separate the effects of supervision, model capacity, graph reasoning, and the adapter?

  3. Does the intervention-trained model remain more faithful under replacement or perturbation mechanisms different from the training mask? What are the separate distributions of Δ_E and Δ_N, and how often do the interventions change the predicted label?

  4. Has an actual pretrained MLLM been trained and evaluated with the proposed pathway? If not, how should the contribution be scoped to the synthetic evidence-binding setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.