- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes EG-MLLM, an evidence-guided architecture combining prompt-conditioned token selection, a sparse graph reasoner, a bottleneck adapter, and an intervention-margin objective. The objective encourages removing selected evidence to reduce the target likelihood more than removing matched nuisance tokens. Evaluation is restricted to a synthetic binary relation task with 6,000 training examples and four 2,000-example test conditions. The full model achieves 98.65% accuracy under shortcut reversal and 95.40% under nuisance corruption. Removing the intervention objective leaves accuracy essentially unchanged but reduces intervention-margin satisfaction from 98.30% to 49.65%. No end-to-end natural-image MLLM results are reported.
Critical strengths
-
The controlled diagnostic makes evidence quality directly measurable. Section 4 and Appendix B specify known answer-critical tokens and deliberately vary shortcut correlation and nuisance corruption. This enables evaluation beyond answer accuracy and provides a useful setting for examining whether a model selects relevant evidence under distribution shifts.
-
The intervention-loss ablation isolates a meaningful behavioral difference. The matched comparison in Section 4.4 reports essentially unchanged shortcut-shift accuracy—98.80% without the loss versus 98.65% with it—but substantially different intervention-margin satisfaction. This supports the narrow conclusion that the objective changes the model’s response to the specified interventions. The authors also appropriately distinguish this operational criterion from identifiable causal effects in Section 3.4.
Critical weaknesses
-
W1 — The evidence does not establish the proposed contribution in the intended MLLM setting. All numerical results come from comparing two scalar attributes encoded in 32-dimensional synthetic token vectors, with exactly two ground-truth evidence tokens (Section 4.1; Appendix B). This bypasses visual feature extraction from images, natural-language understanding and generation, ambiguous evidence, and imperfect evidence supervision. The 7B-class configuration and parameter counts are architectural specifications rather than evidence of successful integration or training. Section 5 explicitly acknowledges this limitation, which is welcome, but does not resolve it. As submitted, the paper establishes a small diagnostic demonstration rather than the effectiveness of an evidence-guided MLLM. Evaluation on an actual vision-language backbone with a documented, practically obtainable supervision scheme is necessary to support the central contribution.
-
W2 — The experiments do not adequately identify the benefits of the proposed mechanisms. In Section 4.2, Soft Attention receives no evidence supervision, whereas the evidence-selection variants do. Top-K + Evidence simultaneously removes the graph and adapter and uses a pooled readout that the authors state loses role–attribute binding. Its near-chance performance therefore does not establish that the proposed graph is necessary or preferable to a simpler binding-capable model. Comparisons with matched supervision and capacity, alongside separate graph and adapter ablations, are missing. Moreover, the intervention loss does not improve accuracy in Table 1; its demonstrated benefit is satisfaction of the same 0.15-margin criterion used for training. This is valid evidence of objective compliance, but provides limited independent support for broader faithfulness. Evaluation under alternative interventions or controlled changes to the true causal attributes would help distinguish semantic evidence dependence from sensitivity to the trained masking procedure.
Questions for authors
-
Clarification — diagnostic training and supervision: What is the exact objective used for each diagnostic model, including the evidence-supervision loss and its weight? How does this objective correspond to Equation 11, and which components replace the vision encoder and autoregressive decoder?
-
Evidence request — attribution of architectural gains: Under identical evidence supervision and comparable capacity, how does EG-MLLM compare with a simple role-aware readout or standard attention model that preserves role–attribute binding? Can the graph and adapter contributions be isolated separately? These comparisons would materially affect W2.
-
Clarification and evidence request — intervention interpretation: Are selection and graph construction recomputed after masking, and can unselected tokens reach the predictor through another pathway? What are the separate distributions of and , and does the reported gain persist under an intervention other than the masking procedure used in training? This would clarify whether improved margin satisfaction reflects transferable evidence dependence.