- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a causal-decomposition evaluation framework for MLLMs on compositional tasks, using a structural causal model (SCM) and ternary interventions (injecting correct, incorrect, or empty answers) at the unit-task level. It introduces "capability metrics" to separate intrinsic from cascading errors, and "contribution metrics" (N/S-Scores) to quantify the necessity and sufficiency of prerequisites. The framework is instantiated in CADET, a large-scale, human-annotated benchmark evaluated on 12 frontier MLLMs. The results demonstrate that this approach uncovers critical insights obscured by standard end-to-end accuracy, such as isolating cascading failures and identifying specific bottleneck prerequisites.
Critical strengths
-
Reframing compositional evaluation as a controlled intervention over a prerequisite graph is effective. It moves beyond uninformative scalar scores to provide actionable diagnostics about where errors arise and which prerequisites act as bottlenecks, revealing how providing prerequisites drastically shifts model capability rankings.
-
The CADET benchmark is a carefully constructed artifact with extensive, newly authored unit-task questions. Its verified correct/incorrect annotations and planned open-source release make it a reusable resource for future research.
Critical weaknesses
-
Incremental Value Over Conditional Accuracy: The prerequisite structures are designer-specified rather than discovered, making the "causal" claims somewhat stipulated. More importantly, the paper fails to establish that the complex N/S-Score machinery provides diagnostic insights beyond what simple conditional accuracies ( vs. ) already offer. Without demonstrating a case where the N/S ranking changes the diagnostic conclusion compared to basic conditional accuracy, the causal framework risks being a formal re-packaging rather than a novel capability.
-
Intervention Asymmetry and Confounding Variables: The (correct) and (incorrect) interventions are not symmetric: often injects additional visual annotations (e.g., bounding boxes) and risks direct answer leakage, whereas is text-only. This discrepancy confounds central metric contrasts (like IC vs. RC), and the current appendix ablation on just two models is insufficient to rule out the impact of this visual scaffolding.
Questions for authors
-
Can you provide a concrete example where the N/S-Score ordering diverges from simple conditional accuracy ordering in a way that actually changes the diagnostic conclusion?
-
How much of the metric contrasts is attributable to the added visual annotations in the condition, and can the text-only ablation be extended to all models and incorrect-contrast conditions?
-
What mechanisms prevent the injections from directly leaking the final answer across different tasks?
-
How sensitive are the N- and S-Score bottleneck rankings to changes in the designer-specified prerequisite graph (e.g., adding or removing a parent node)?
5 comments
Reponse to Reviewer haKM [part 2]
Q2 & W2: How much of the metric contrasts is attributable to the added visual annotations, and can the text-only ablation be extended to all models and incorrect-contrast conditions?
A: Following the reviewer's suggestion, we extended the text-only ablation (, injecting the same correct textual answer while keeping the original image unmodified) in Appendix D.4.1 from 2 to 9 models across 5 model families (the remaining 3 could not complete within the rebuttal window), and added the symmetric incorrect-contrast condition (, 0) where both states intervene purely via text. The 9-model averaged results are reported below:
| Category | NC | IC (pa = 1) | (pa = ) | Δ (IC − ) |
|---|---|---|---|---|
| Perception | 0.594 | 0.801 | 0.760 | +0.041 |
| Spatial | 0.551 | 0.669 | 0.649 | +0.020 |
| Temporal | 0.461 | 0.626 | 0.614 | +0.012 |
| Cognitive | 0.419 | 0.738 | 0.669 | +0.069 |
| Overall | 0.493 | 0.696 | 0.660 | +0.035 |
| Metric | Main (a = 1) | Text-Only (a = ) |
|---|---|---|
| S(a, ∅) | 0.248 | 0.173 |
| N(a, ∅) | 0.106 | 0.108 |
| S(a, 0) | 0.137 | 0.122 |
| N(a, 0) | 0.203 | 0.163 |
Across all 9 models, we observe the same consistent pattern as on the initial 2 models: IC is only slightly higher than (+0.035 overall, and +0.012–0.020 in Temporal and Spatial), showing that the textual answer alone accounts for the vast majority of the capability gain over NC (0.493 → 0.660) while preserving all capability findings in Sec. 5.2 (e.g., Cognitive rising from lowest to above Spatial and Temporal). Similarly, text-only N- and S-Scores are slightly lower than in the main setting; consistent with Appendix D.4.1, these drops concentrate on a few Perception prerequisites that benefit from visual localization cues, whereas Spatial, Temporal, and Cognitive prerequisites remain closely aligned along the diagonal. Overall, S(, ∅), N(, ∅), S(, 0), and N(, 0) strongly correlate with their main-setting counterparts (Pearson r = 0.80, 0.86, 0.92, and 0.87, respectively), and all causal contribution findings (including bottleneck sparsity and N(, 0) > N(, ∅)) hold unchanged. We have updated Appendix D.4.1 with the full 9-model results.
Q3: What mechanisms prevent the injections from directly leaking the final answer across different tasks?
A: In CADET tasks, upstream prerequisites target individual capabilities, with their questions, answers, and visual annotations (where applicable) typically focusing on local information that differs from the final task query and answer (Table 4; Appendix B.2). For example, in Multi-View Sequencing (Fig. 1), the final task asks for the chronological order of five images (e.g., E, A, D, C, B), and its answer is distinct from those of its prerequisites. For cases where an upstream step could potentially overlap with the final answer, the authors carefully reviewed each prerequisite and applied dedicated filtering; for instance, in the unfolded Maze chain (Appendix D.3), where steps share the same exit-identification question and ground-truth answer, omits the preceding steps' textual answers and marks the path only up to step t−1 on the image.
Empirically, our results across all 12 models are consistent with this: substantial residual errors and unrecovered failures remain across all capability categories, both under full-prerequisite injection (1 − IC in Fig. 3, Table 8) and across various prerequisite injection conditions (Figs. 5, 9, 20). Moreover, as shown in Q2 above, text-only injection (, keeping the original image unmodified) closely tracks IC across the 9 evaluated models, indicating that the visual annotations in likewise do not give away the final answer.
Reponse to Reviewer haKM [part 3]
Q4: How sensitive are the N- and S-Score bottleneck rankings to changes in the designer-specified prerequisite graph (e.g., adding or removing a parent node)?
A: Theoretically, our SCM specifies the prerequisite reference set placed under controlled intervention rather than assuming a complete graph, so all N- and S-Scores remain valid lower bounds on PN and PS under any parent addition or removal (Theorems A.3–A.4; Appendix E). Because any unmodeled factor remains un-intervened in the natural state , removing a parent is operationally identical to fixing , whereas adding one extends from ∅ to .
- Consequently, both the scores and relative rankings of shared parents are strictly invariant to adding or removing other parents for all three natural-environment metrics (): (our primary repair metric), , and (Def 3.7–3.8).
- For the remaining five metrics under active backgrounds (), shared-parent rankings (1) remain stable whenever the added or removed node is non-critical or affects other parents uniformly, and (2) reorder only when it selectively interacts with a specific parent (e.g., as a redundant backup in A ∨ B), which rightly reflects that parent's updated causal role under the new reference set.
Take as an example: for any two shared parents with , substituting into Definition 3.7 (Eq. 5), this ranking is equivalent to with before removing and after when considering parent removal. Switching from 1 to ∅ at both side preserves this inequality whenever is non-critical or shifts both sides comparably (case 1), and flips it only when selectively interacts with one parent, e.g., acting as a redundant backup that sustains Y under at but not at (case 2). Conversely, adding a parent reverses this transition from (before addition) to (after), under which the same analysis and conclusions hold symmetrically.
Reponse to Reviewer haKM
We sincerely thank Reviewer haKM for the insightful review and constructive feedback, particularly for recognizing our controlled intervention formulation over prerequisite graphs, actionable bottleneck diagnostics, and newly authored CADET benchmark.
Q1&W1: Causal framework value over conditional accuracy, and concrete examples of N/S-score ordering diverges from conditional accuracy.
A: Compositional tasks in MLLMs often depend on multiple interdependent prerequisites, so we adopt the SCM and probabilities-of-causation framework to model muti-prerequisite dependencies and attribute task performance through N-, S- score for necessity and sufficiency. Although each single N- or S-Score (like PN and PS) is expressed as a normalized contrast of conditional probabilities, below points make it go beyond conditional accuracy:
For example, our N- and S-Scores reveal markedly different problem-solving mechanisms across model tiers and individual models (Sec. 5.3, Fig. 6; App. D.2, Figs. 18–19, 29–36): on Pattern-2 tasks, SOTA models operate with mutually compensable, redundant pathways (low N, high S, analogous to disjunctive A ∨ B below), whereas other models lack such redundancy and shift toward higher N and lower S (closer to conjunctive A ∧ B). Furthermore, across the ternary contrasts, our N-Scores guide failure attribution by separating omission loss (N(1, ∅)) from misinformation damage (N(1, 0), N(∅, 0), e.g., on cognitive prerequisites; Figs. 7, 9), while our S-Scores guide targeted repair (e.g., the single top-S(1, ∅) prerequisite recovering 84% of the full-prerequisite gain; Fig. 8b) by separating non-monotonic scaffolding (S(0, ∅)) from genuine causal sufficiency (S(1, ∅), S(1, 0)).
Theoretical grounding in probabilities of causation: More broadly, these design choices directly reflect why the causal attribution literature distinguishes counterfactual attribution from direct conditional probability contrasts (Tian & Pearl, 2000; Pearl, 2022):
Following Reviewer haKM's suggestion, we select Traffic Causation on GPT-5.4 as a representative example and report the prerequisite rankings (1=highest) across the eight N/S-Scores alongside Δ=P(Y=1∣do(Ai=1))−P(Y=1∣do(Ai=0)) below. Different metrics lead to distinct diagnostic conclusions on this task: S(1, ∅) selects X₅ (Temporal Sequencing) for natural single-point repair, N(1, 0) / S(1, 0) highlight X₆ (False Correlation Rejection) under prerequisite corruption (sensitivity to correctness), while the conditional accuracy contrast Δ selects X₂ (State Change Recognition).