acceptodds
Under review as a conference paper at ICLR 2027

Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis

11 views

Abstract

End-to-end accuracy on compositional tasks records how often MLLMs fail, but cannot distinguish whether a failure reflects an intrinsic deficit in the targeted capability or a cascading error from an upstream prerequisite. We propose a causal decomposition framework that isolates these two failure modes through controlled interventions on the prerequisite dependencies of each task. Our capability metrics (NC, IC, RC) score each task under unassisted, correct, or incorrect prerequisites to diagnose where failures arise; contribution metrics (N-Score, S-Score), adapted from probabilities of causation, quantify each prerequisite's necessity and sufficiency to determine why. We instantiate the framework in CADET, a diagnostic benchmark of 10 composite tasks decomposed into 46 unit tasks with over 33,000 human-annotated questions spanning perception, spatial, temporal, and cognitive categories. Diagnosing frontier MLLMs with our framework uncovers systematic patterns that end-to-end accuracy obscures. Capability-wise, supplying correct prerequisites eliminates 54% of errors on cognitive tasks, lifting them from weakest to above spatial and temporal. Prerequisite-wise, causal contributions are concentrated in a few critical prerequisites, and supplying the single most important one alone captures 84% of the gain from supplying all prerequisites.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Evaluation and Diagnosis, Benchmark, Task Decomposition, Structural Causal Models

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_haKMOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a causal-decomposition evaluation framework for MLLMs on compositional tasks, using a structural causal model (SCM) and ternary interventions (injecting correct, incorrect, or empty answers) at the unit-task level. It introduces "capability metrics" to separate intrinsic from cascading errors, and "contribution metrics" (N/S-Scores) to quantify the necessity and sufficiency of prerequisites. The framework is instantiated in CADET, a large-scale, human-annotated benchmark evaluated on 12 frontier MLLMs. The results demonstrate that this approach uncovers critical insights obscured by standard end-to-end accuracy, such as isolating cascading failures and identifying specific bottleneck prerequisites.

Critical strengths
  1. Reframing compositional evaluation as a controlled intervention over a prerequisite graph is effective. It moves beyond uninformative scalar scores to provide actionable diagnostics about where errors arise and which prerequisites act as bottlenecks, revealing how providing prerequisites drastically shifts model capability rankings.

  2. The CADET benchmark is a carefully constructed artifact with extensive, newly authored unit-task questions. Its verified correct/incorrect annotations and planned open-source release make it a reusable resource for future research.

Critical weaknesses
  1. Incremental Value Over Conditional Accuracy: The prerequisite structures are designer-specified rather than discovered, making the "causal" claims somewhat stipulated. More importantly, the paper fails to establish that the complex N/S-Score machinery provides diagnostic insights beyond what simple conditional accuracies (P(Y=1∣X=1)P(Y=1\mid X=1) vs. P(Y=1∣X=0)P(Y=1\mid X=0)) already offer. Without demonstrating a case where the N/S ranking changes the diagnostic conclusion compared to basic conditional accuracy, the causal framework risks being a formal re-packaging rather than a novel capability.

  2. Intervention Asymmetry and Confounding Variables: The A=1A=1 (correct) and A=0A=0 (incorrect) interventions are not symmetric: A=1A=1 often injects additional visual annotations (e.g., bounding boxes) and risks direct answer leakage, whereas A=0A=0 is text-only. This discrepancy confounds central metric contrasts (like IC vs. RC), and the current appendix ablation on just two models is insufficient to rule out the impact of this visual scaffolding.

Questions for authors
  1. Can you provide a concrete example where the N/S-Score ordering diverges from simple conditional accuracy ordering in a way that actually changes the diagnostic conclusion?

  2. How much of the metric contrasts is attributable to the added visual annotations in the A=1A=1 condition, and can the text-only ablation be extended to all models and incorrect-contrast conditions?

  3. What mechanisms prevent the A=1A=1 injections from directly leaking the final answer across different tasks?

  4. How sensitive are the N- and S-Score bottleneck rankings to changes in the designer-specified prerequisite graph (e.g., adding or removing a parent node)?

5 comments
AuthorsOpenReview ↗

Reponse to Reviewer haKM

We sincerely thank Reviewer haKM for the insightful review and constructive feedback, particularly for recognizing our controlled intervention formulation over prerequisite graphs, actionable bottleneck diagnostics, and newly authored CADET benchmark.

Q1&W1: Causal framework value over conditional accuracy, and concrete examples of N/S-score ordering diverges from conditional accuracy.

A: Compositional tasks in MLLMs often depend on multiple interdependent prerequisites, so we adopt the SCM and probabilities-of-causation framework to model muti-prerequisite dependencies and attribute task performance through N-, S- score for necessity and sufficiency. Although each single N- or S-Score (like PN and PS) is expressed as a normalized contrast of conditional probabilities, below points make it go beyond conditional accuracy:

  • (1) dividing by baseline success or failure in the denominators normalizes out accuracy differences, making N/S scores comparable across models and tasks and enabling further diagnosis analysis (e.g., SOTA models’ redundant problem-solving pattern discovery in paper & examples below).
  • (2) moreover, under our asymmetric background states to the remaining prerequisites A−iA_{-i} (A−i=aA_{-i} = a for N-Score and A−i=a′A_{-i} = a' for S-Score) (Sec 3.4), N and S-scores across the four state pairs are distinct, with each capturing an independent causal mechanism in multi-prerequisite reasoning (Sec. 3.4, Tab. 1; App. A.3.2).

For example, our N- and S-Scores reveal markedly different problem-solving mechanisms across model tiers and individual models (Sec. 5.3, Fig. 6; App. D.2, Figs. 18–19, 29–36): on Pattern-2 tasks, SOTA models operate with mutually compensable, redundant pathways (low N, high S, analogous to disjunctive A ∨ B below), whereas other models lack such redundancy and shift toward higher N and lower S (closer to conjunctive A ∧ B). Furthermore, across the ternary contrasts, our N-Scores guide failure attribution by separating omission loss (N(1, ∅)) from misinformation damage (N(1, 0), N(∅, 0), e.g., on cognitive prerequisites; Figs. 7, 9), while our S-Scores guide targeted repair (e.g., the single top-S(1, ∅) prerequisite recovering 84% of the full-prerequisite gain; Fig. 8b) by separating non-monotonic scaffolding (S(0, ∅)) from genuine causal sufficiency (S(1, ∅), S(1, 0)).

 

Theoretical grounding in probabilities of causation: More broadly, these design choices directly reflect why the causal attribution literature distinguishes counterfactual attribution from direct conditional probability contrasts (Tian & Pearl, 2000; Pearl, 2022):

  • Even under exogeneity, interventional probabilities and their unscaled differences measure population-average effects, ignoring the conditioning events (X = x, Y = 1) for PN and (X = x′, Y = 0) for PS. Conditioning on these actual outcomes divides the population-level difference by baseline success P(Yx=1)P(Y_x = 1) for PN and by baseline failure 1−P(Yx′=1)1 - P(Y_{x'} = 1) for PS; without these denominators, a factor accounting for a large fraction of actual successes or failures yields a near-zero raw difference whenever the corresponding baseline rate is small, distorting comparisons across tasks, intervention regimes, or models with different baseline accuracies.
  • Finally, necessity and sufficiency are two independent aspects of causal explanation that cannot be captured by a single scalar contrast. In disjunctive structures (A ∨ B), an alternative process sustains the outcome so PN = 0, whereas PS = 1 captures each factor's sufficiency when the other is inactive; conversely, in conjunctive structures (A ∧ B), both factors have PN = 1, whereas PS depends on the state of the other factor and drops to 0 when the other is inactive.

 

Following Reviewer haKM's suggestion, we select Traffic Causation on GPT-5.4 as a representative example and report the prerequisite rankings (1=highest) across the eight N/S-Scores alongside Δ=P(Y=1∣do(Ai=1))−P(Y=1∣do(Ai=0))\Delta = P(Y=1 \mid do(A_i=1)) - P(Y=1 \mid do(A_i=0)) below. Different metrics lead to distinct diagnostic conclusions on this task: S(1, ∅) selects X₅ (Temporal Sequencing) for natural single-point repair, N(1, 0) / S(1, 0) highlight X₆ (False Correlation Rejection) under prerequisite corruption (sensitivity to correctness), while the conditional accuracy contrast Δ selects X₂ (State Change Recognition).

MetricX₁X₂X₃X₄X₅X₆
Δ412356
S(1, ∅)523416
N(1, ∅)124424
S(1, 0)432551
N(1, 0)423651
S(∅, 0)532351
N(∅, 0)412443
S(0, ∅)344214
N(0, ∅)233313
AuthorsOpenReview ↗

Reponse to Reviewer haKM [part 2]

Q2 & W2: How much of the metric contrasts is attributable to the added visual annotations, and can the text-only ablation be extended to all models and incorrect-contrast conditions?

A: Following the reviewer's suggestion, we extended the text-only ablation (Ai=1tA_i = 1_t, injecting the same correct textual answer while keeping the original image unmodified) in Appendix D.4.1 from 2 to 9 models across 5 model families (the remaining 3 could not complete within the rebuttal window), and added the symmetric incorrect-contrast condition (1t1_t, 0) where both states intervene purely via text. The 9-model averaged results are reported below:

CategoryNCIC (pa = 1)IC1t\rm IC_{1_t} (pa = 1t1_t)Δ (IC − IC1t\rm IC_{1_t})
Perception0.5940.8010.760+0.041
Spatial0.5510.6690.649+0.020
Temporal0.4610.6260.614+0.012
Cognitive0.4190.7380.669+0.069
Overall0.4930.6960.660+0.035
MetricMain (a = 1)Text-Only (a = 1t1_t)
S(a, ∅)0.2480.173
N(a, ∅)0.1060.108
S(a, 0)0.1370.122
N(a, 0)0.2030.163

Across all 9 models, we observe the same consistent pattern as on the initial 2 models: IC is only slightly higher than IC1t\rm IC_{1_t} (+0.035 overall, and +0.012–0.020 in Temporal and Spatial), showing that the textual answer alone accounts for the vast majority of the capability gain over NC (0.493 → 0.660) while preserving all capability findings in Sec. 5.2 (e.g., Cognitive rising from lowest to above Spatial and Temporal). Similarly, text-only N- and S-Scores are slightly lower than in the main setting; consistent with Appendix D.4.1, these drops concentrate on a few Perception prerequisites that benefit from visual localization cues, whereas Spatial, Temporal, and Cognitive prerequisites remain closely aligned along the diagonal. Overall, S(1t1_t, ∅), N(1t1_t, ∅), S(1t1_t, 0), and N(1t1_t, 0) strongly correlate with their main-setting counterparts (Pearson r = 0.80, 0.86, 0.92, and 0.87, respectively), and all causal contribution findings (including bottleneck sparsity and N(1t1_t, 0) > N(1t1_t, ∅)) hold unchanged. We have updated Appendix D.4.1 with the full 9-model results.

 


Q3: What mechanisms prevent the Ai=1A_i = 1 injections from directly leaking the final answer across different tasks?

A: In CADET tasks, upstream prerequisites target individual capabilities, with their questions, answers, and visual annotations (where applicable) typically focusing on local information that differs from the final task query and answer (Table 4; Appendix B.2). For example, in Multi-View Sequencing (Fig. 1), the final task asks for the chronological order of five images (e.g., E, A, D, C, B), and its answer is distinct from those of its prerequisites. For cases where an upstream step could potentially overlap with the final answer, the authors carefully reviewed each prerequisite and applied dedicated filtering; for instance, in the unfolded Maze chain (Appendix D.3), where steps share the same exit-identification question and ground-truth answer, Ai=1A_i = 1 omits the preceding steps' textual answers and marks the path only up to step t−1 on the image.

Empirically, our results across all 12 models are consistent with this: substantial residual errors and unrecovered failures remain across all capability categories, both under full-prerequisite injection (1 − IC in Fig. 3, Table 8) and across various prerequisite injection conditions (Figs. 5, 9, 20). Moreover, as shown in Q2 above, text-only injection (IC1t\rm IC_{1_t}, keeping the original image unmodified) closely tracks IC across the 9 evaluated models, indicating that the visual annotations in Ai=1A_i = 1 likewise do not give away the final answer.

AuthorsOpenReview ↗

Reponse to Reviewer haKM [part 3]

Q4: How sensitive are the N- and S-Score bottleneck rankings to changes in the designer-specified prerequisite graph (e.g., adding or removing a parent node)?

A: Theoretically, our SCM specifies the prerequisite reference set placed under controlled intervention rather than assuming a complete graph, so all N- and S-Scores remain valid lower bounds on PN and PS under any parent addition or removal (Theorems A.3–A.4; Appendix E). Because any unmodeled factor remains un-intervened in the natural state ∅∈{1,0,∅}\varnothing \in \lbrace 1, 0, \varnothing \rbrace, removing a parent XkX_{k} is operationally identical to fixing Ak=∅A_{k} = \varnothing, whereas adding one extends AkA_{k} from ∅ to {1,0,∅}\lbrace 1, 0, \varnothing \rbrace.

  • Consequently, both the scores and relative rankings of shared parents are strictly invariant to adding or removing other parents for all three natural-environment metrics (A−i=∅A_{-i} = \varnothing): S(1,∅)S_{(1,\varnothing)} (our primary repair metric), S(0,∅)S_{(0,\varnothing)}, and N(∅,0)N_{(\varnothing,0)} (Def 3.7–3.8).
  • For the remaining five metrics under active backgrounds (A−i∈{1,0}A_{-i} \in \lbrace 1, 0 \rbrace), shared-parent rankings (1) remain stable whenever the added or removed node is non-critical or affects other parents uniformly, and (2) reorder only when it selectively interacts with a specific parent (e.g., as a redundant backup in A ∨ B), which rightly reflects that parent's updated causal role under the new reference set.

 

Take N(1,∅)N_{(1,\varnothing)} as an example: for any two shared parents with N(1,∅)(Xi)>N(1,∅)(Xj)>0N_{(1,\varnothing)}(X_i) \gt N_{(1,\varnothing)}(X_j) \gt 0, substituting into Definition 3.7 (Eq. 5), this ranking is equivalent to P(Y=1∣do(Ai=∅,A−i,k=1,Ak=c))<P(Y=1∣do(Aj=∅,A−j,k=1,Ak=c)),P(Y = 1 \mid do(A_i = \varnothing, A_{-i,k} = 1, A_k = c)) \lt P(Y = 1 \mid do(A_j = \varnothing, A_{-j,k} = 1, A_k = c)), with c=1c = 1 before removing XkX_k and c=∅c = \varnothing after when considering parent removal. Switching cc from 1 to ∅ at both side preserves this inequality whenever XkX_k is non-critical or shifts both sides comparably (case 1), and flips it only when XkX_k selectively interacts with one parent, e.g., acting as a redundant backup that sustains Y under Aj=∅A_j = \varnothing at c=1c = 1 but not at c=∅c = \varnothing (case 2). Conversely, adding a parent XkX_k reverses this transition from c=∅c = \varnothing (before addition) to c=1c = 1 (after), under which the same analysis and conclusions hold symmetrically.

Reviewer_24XYOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how to diagnose failures in MLLMs on compositional tasks. The authors argue that end-to-end accuracy cannot distinguish failures caused by intrinsic capability limitations from those resulting from errors propagated through prerequisite steps.

To address this issue, the paper proposes a causal task decomposition framework that models task dependencies using Structural Causal Models (SCMs). It introduces three capability metrics (NC, IC, RC) based on interventions with correct, incorrect, or absent prerequisite answers, along with N-Score and S-Score to quantify the necessity and sufficiency of individual prerequisites.

The authors construct CADET, a benchmark comprising 10 composite tasks and over 33,000 human-annotated questions. Experiments on 12 MLLMs reveal that providing correct prerequisites can eliminate a substantial proportion of errors in cognitive tasks, while significant capability gaps remain in spatial and temporal tasks. Furthermore, supplying only the single most influential prerequisite recovers most of the improvement achieved by providing all prerequisites.

Critical strengths

Strength

  1. The paper addresses an important limitation of existing MLLM evaluation by distinguishing intrinsic capability limitations from cascading errors.
  2. The proposed framework is well-structured, with clear mathematical definitions and interpretable metrics for diagnosing capability limitations and prerequisite contributions.
  3. The experiments are comprehensive and reveal interesting findings, particularly that a few critical prerequisites dominate final-task performance.
Critical weaknesses

Weakness

  1. The incorrect prerequisite answers are primarily sampled from the valid answer space rather than reflecting errors that models naturally make. Although this ensures standardized comparisons across models, such errors may differ from realistic upstream mistakes.

  2. The evaluation is limited to 10 predefined task types, some of which have relatively simple dependency structures, such as Maze and Map Navigation.I would encourage the authors to include additional open-ended or interactive tasks with dynamic dependencies, such as Traffic Causation.

Questions for authors

See weaknesses above. If the authors can adequately address the above two weaknesses, I will increase my rating score.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 24XY

We sincerely thank Reviewer 24XY for the constructive review and positive assessment, particularly for recognizing our SCM-based diagnostic framework, interpretable capability and contribution metrics, and comprehensive empirical findings.

W1: The incorrect prerequisite answer sampled may differ from realistic upstream mistakes.

A: Regarding Ai=0A_i = 0 wrong answer sampling: As Reviewer 24XY rightly noted, our main Ai=0A_i = 0 setting samples from the valid answer space of unit task XiX_i so that perturbations remain standardized and model-independent, ensuring that RC and state-0 contribution scores are directly comparable across models (Sec. 4; Appendix E). To avoid implausible errors that models could trivially rule out, numerical tasks are sampled within a bounded local neighborhood around the ground truth (e.g., within ±2 of a ground-truth value of 5 rather than distant values like 20), and multiple-choice tasks are sampled from the valid option candidates (Appendix B.3.2).

Experiment on Ai=0mA_i = 0_m using realistic model errors: Following Reviewer 24XY's suggestion, we further construct a realistic upstream-mistake regime (Ai=0mA_i = 0_m) and evaluate it across 9 models (4 SOTA and 5 Others across 5 model families) on the 8 non-Markov tasks (Appendix D.4.3). Under Ai=0mA_i = 0_m, we select each prerequisite's incorrect answer in three tiers:

  • (1) we first prioritize the model's own incorrect answer from NC (29.3%);
  • (2) if unavailable (the model answered correctly), we use the most frequent cross-model incorrect answer (39.8%); and
  • (3) if neither exists (all models answered correctly), we retain the original sampled answer (30.9%).

The main results are reported in the tables below.

0m0_m Error Source DistributionCapability Metrics
CategoryTier1: Model's OwnTier2: Cross-ModelTier3: Original SampledNCRCRC0m\rm RC_{0_m}RC0m\rm RC_{0_m} − RCRC − NCRC0m\rm RC_{0_m} − NC
Perception22.5%40.4%37.1%0.6820.5380.472−0.066−0.144−0.210
Spatial34.0%38.3%27.7%0.5920.4820.438−0.044−0.110−0.154
Temporal41.2%39.6%19.2%0.4610.4180.400−0.018−0.043−0.061
Cognitive39.8%52.8%7.4%0.4240.4370.443+0.007+0.013+0.019
Overall29.3%39.8%30.9%0.5180.4560.431−0.026−0.062−0.087

Capability: Overall, RC0m\rm RC_{0_m} closely tracks RC across categories and models (with cross-model rankings highly consistent at Kendall τ = 0.94 and the IC − RC gap preserved), confirming that all capability findings in Section 5.2 hold under realistic model errors:

  • In particular, RC0m\rm RC_{0_m} − NC follows the same cross-category progression as RC − NC: whereas incorrect prerequisites degrade Perception (−0.210), Spatial (−0.154), and Temporal (−0.061), Cognitive shows no degradation and retains a slight positive shift (+0.019, vs. +0.013 under RC), confirming that the non-monotonicity in Section 5.2 (Property 3.6) holds under realistic upstream errors.
  • Meanwhile, RC0m\rm RC_{0_m} is slightly lower than RC overall (avg. Δ = −0.026), with modest drops in Perception (−0.066) and Spatial (−0.044), indicating that realistic upstream model errors are slightly more misleading than the sampled wrong answers; restricting strictly to the subset of instances where all injected prerequisites come from actual model errors (Tiers 1–2, excluding Tier 3) shows the same pattern (avg. Δ = −0.018; Appendix D.4.3, Table 16).
AuthorsOpenReview ↗

Response to Reviewer 24XY [part 2]

(W1 continue)

N(1, ∅)N(1, 0)N(1, 0m0_m)N(1, 0m0_m) − N(1, 0)N(1, 0) − N(1, ∅)N(1, 0m0_m) − N(1, ∅)S(1, ∅)S(1, 0)S(1, 0m0_m)S(1, 0m0_m) − S(1, 0)S(1, 0) − S(1, ∅)S(1, 0m0_m) − S(1, ∅)
Perception0.0780.1990.202+0.003+0.120+0.1240.1950.0370.028−0.010−0.157−0.167
Spatial0.1010.2110.266+0.055+0.110+0.1640.2860.1640.163−0.001−0.122−0.123
Temporal0.0920.1400.131−0.009+0.047+0.0380.2150.0970.099+0.002−0.118−0.116
Cognitive0.3860.5080.529+0.021+0.123+0.1440.4690.5040.532+0.028+0.035+0.064
Overall0.1080.2070.223+0.016+0.099+0.1150.2440.1200.118−0.002−0.125−0.126
SOTA / Others0.086 / 0.1250.171 / 0.2360.187 / 0.251+0.016 / +0.015+0.084 / +0.111+0.100 / +0.1270.298 / 0.2010.157 / 0.0900.150 / 0.092−0.007 / +0.002−0.142 / −0.111−0.148 / −0.109

Contribution: For the contribution metrics, following Appendix D.4, N(1, 0m0_m) and S(1, 0m0_m) closely match N(1, 0) and S(1, 0) across all prerequisites (Pearson r = 0.98 and 0.99, MAE 0.029 and 0.017 on the 9-model means), with no prerequisite crossing the 0.15 threshold. Consistent with models being more misled by their own mistakes above (RC0m\rm RC_{0_m} < RC), in N(1, 0m0_m) (Eq. 5), where the baseline IC(Y) is fixed and 0m0_m enters only the single corrupted prerequisite, we observe a slightly larger average drop under single-node corruption that shifts mean N modestly upward (0.207 → 0.223); whereas in S(1, 0m0_m) (Eq. 6), where 0m0_m enters both the remaining background (A−i=0mA_{-i} = 0_m) and the baseline RC0m\rm RC_{0_m}(Y), both accuracies empirically decrease by similar amounts on average and largely offset each other (mean S: 0.120 → 0.118). Furthermore, all Section 5.3 findings on (1, 0) hold consistently under (1, 0m0_m) (table above):

  • Our finding in Sec. 5.3 (Fig. 9) that injecting incorrect prerequisites is consistently more destructive than omitting them across all four categories (N(1, 0) − N(1, ∅) = +0.047 to +0.123; +0.099 overall) holds under (1, 0m0_m): N(1, 0m0_m) − N(1, ∅) is positive across every category, ranging from +0.038 to +0.164 (+0.115 overall).
  • Our finding in Sec. 5.3 (Fig. 9) that, unlike cognitive target tasks benefiting from incorrect prerequisites, cognitive prerequisites are the most sensitive to correctness, causing the steepest drop when corrupted and the strongest recovery when fixed (N(1, 0) = 0.508 vs. 0.140–0.211 for other categories; S(1, 0) = 0.504 vs. 0.037–0.164), likewise holds under (1, 0m0_m): cognitive prerequisites score highest on both metrics (N(1, 0m0_m) = 0.529 vs. 0.131–0.266; S(1, 0m0_m) = 0.532 vs. 0.028–0.163).
  • Our finding in Sec. 5.3 (Fig. 7) that SOTA models exhibit lower vulnerability to single-prerequisite corruption alongside stronger single-point recovery than other models (N(1, 0): 0.171 vs. 0.236; S(1, 0): 0.157 vs. 0.090) also holds under (1, 0m0_m): SOTA models pair lower N(1, 0m0_m) (0.187 vs. 0.251) with higher S(1, 0m0_m) (0.150 vs. 0.092).

We have added these results to Appendix D.4.

AuthorsOpenReview ↗

Response to Reviewer 24XY [part 3]

W2: The evaluation is limited to 10 predefined task types, and extension to additional open-ended or interactive tasks with dynamic dependencies.

A: We sincerely thank Reviewer 24XY for the positive evaluation of our framework and findings, and especially for recognizing the design of Traffic Causation and suggesting extensions to additional open-ended or interactive tasks with dynamic dependencies.

We would like to first share more rationale and considerations about our 10 tasks in CADET:

Multi-step Markov chain structures in Maze and Map Navigation: At the overall task-level SCM, Maze and Map Navigation indeed have relatively simple dependency structures, yet each encodes a multi-step, sequentially dependent Markov chain unit task that iteratively solves recurring sub-problems (Sec. 4; Appendix A.4). Similarly, Multistep Reasoning incorporates a two-task interleaved Markov chain alternating between position tracking and open-ended action prediction, and unfolding these structures yields sequential dependency chains spanning multiple steps up to 16+ steps (Appendix A.4, D.3). We systematically study these unfolded multi-step structures and their empirical capability patterns across reasoning depth in Sec. 5.4 (Fig. 10; Appendix D.3). We have also made the notation and description of Markov chain structures clearer in Fig. 2 and Sec. 4 of the revised paper.

Task selection rationale and structural breadth: In selecting the 10 tasks for CADET, we focused on unsaturated real-world compositional problems across single-image, multi-image, and video modalities and all four capability categories (perception, spatial, temporal, and cognitive), with SCM structures ranging from shallow aggregations (e.g., Dice Sum) and iterative Markov chains (e.g., Maze, Multistep Reasoning) to deep multi-path cascades (e.g., open-ended Traffic Causation) (Sec. 4; Appendix B.2). Covering both sparser structures (Pattern 1) and denser multi-parent cascades (Pattern 2) also enables our analysis to uncover how different models behave across varying dependency structures (Sec. 5.3, Fig. 6); for instance, top SOTA models maintain low omission sensitivity (N) and strong single-point recoverability (S) even on dense multi-parent tasks, whereas other models shift sharply toward higher N and lower S as prerequisite dependencies grow more complex. Indeed, Traffic Causation, which the reviewer highlights, already exemplifies this direction as CADET's most complex task, with a deep six-node multi-path cascade incorporating open-ended causal reasoning over dynamic video events (Appendix B.2).

Extending to broader tasks and automated decomposition: We fully agree that diagnosing broader open-ended and interactive tasks, as well as extending causal evaluation to wider and more complex SCM dependencies and task scenarios, is a closely related and a very promising direction for future work. However, within the short rebuttal window, we are unable to construct an additional task to the same standard as existing CADET tasks, since each task is newly created from scratch to mitigate data contamination and annotated through nearly six months of close collaboration between the authors and over 100 raters to ensure benchmark quality and reliability (Sec. 4; Appendix B.3.2). Rather, we hope that building and open-sourcing CADET can provide a high-quality, reliable benchmark of human-verified unit tasks and SCMs to support and validate future work in this direction, particularly automated task decomposition, such as leveraging capable models and agents to automatically decompose, evaluate, and diagnose more complex task scenarios (Sec. 6; Appendix E).

Reviewer_PXa6OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a task-decomposition framework to diagnose failures in multimodal models. It measures performance with missing, correct, and incorrect prerequisite information using the NC, IC, and RC metrics, and proposes two normalized intervention metrics (N-Score and S-Score) to quantify prerequisite contributions. The framework is evaluated on CADET, a dataset with 10 composite tasks, 46 unit tasks, and over 33,000 unit-task questions built from 1,144 instances.

The main empirical finding is valuable: getting a prerequisite question right on its own does not mean that correcting it will give the biggest improvement in the final output. I appreciate the controlled prompt intervention design and the inclusion of question-only and text-only controls. However, my main concerns are about interpreting prompt interventions as true model capability state interventions, and whether the resulting failure-attribution claims are strong enough.

Critical strengths
  1. Controlled interventions offer a better diagnostic approach than observational error analysis. Comparing model performance when prerequisite information is provided, omitted, or modified directly tests how the model responds to that information. This is a clear improvement over trying to infer failure causes solely from subtask accuracy or generated reasoning traces, and it works well for closed-weight models.

  2. The benchmark provides actionable comparisons across prerequisites. The explicit task dependencies and correct/incorrect annotations allow for systematic measurement of omission sensitivity and single-prerequisite recovery. The finding that the most helpful prerequisite is often not the one with the lowest standalone accuracy is a useful practical insight for prioritizing system improvements. Additionally, the question-only and text-only controls help separate answer content from visual-annotation and prompt scaffolding effects.

Critical weaknesses
  1. Prompt interventions do not strictly measure internal model capabilities. Injecting prerequisite text only changes the input prompt; it does not directly set or modify the model's internal capability or latent state. The model might ignore, override, or misinterpret the provided answer, or the added text might simply alter its overall reasoning path. While Appendix E acknowledges this limitation, the main text still treats IC as intrinsic capability and uses IC − NC to estimate cascading error.

Furthermore, Assumption 3.3 needs a clearer justification. The lack of an internal state readout does not automatically mean that injecting text is equivalent to conditioning on a latent subtask outcome. The lower-bound claims make sense for prompt-based interventions, but this is distinct from proving the necessity or sufficiency of internal model capabilities.

  1. Aggregate gains and retrospective prerequisite selection require clearer qualification. The reported 54% reduction in errors comes from a net accuracy difference. Without reporting paired instance transitions (e.g., how many wrong predictions became correct versus correct ones becoming wrong), it is unclear how many original errors were actually fixed. Similarly, the 84% single-prerequisite result relies on retrospectively selecting the best prerequisite using measured S-Scores. While this shows that intervention benefits can be concentrated, it does not prove that one can reliably select the correct prerequisite for new, unseen instances. Reporting paired rescue/harm counts and clarifying whether prerequisite selection was evaluated on held-out instances would make the claims much stronger.
Questions for authors
  1. Interpretation of causal variables: Are the PN/PS guarantees intended to apply to prompt-information regimes or to the model’s internal prerequisite outcomes? Beyond defining the intervention operationally, how does Assumption 3.3 justify connecting textual injection to internal capability states?

  2. Paired error transitions: For the cognitive-category results, what percentage of instances change from incorrect to correct (rescued) versus correct to incorrect (harmed) when correct prerequisites are injected? Does the overall conclusion hold when looking at raw rescue rates alongside net accuracy gains?

  3. Single-prerequisite selection: At what level is the highest-S prerequisite selected (per task, per model, or per instance)? Was the 84% recovery rate evaluated on the same instances used to select the best prerequisite, and is this ranking stable across resampling or held-out splits?

3 comments
AuthorsOpenReview ↗

Response to Reviewer PXa6

We sincerely thank Reviewer PXa6 for the thoughtful review and encouraging feedback, particularly for recognizing our controlled prompt intervention design, ablation controls, and actionable diagnostic comparisons across prerequisites.

Q1 & W1: Interpretation of causal variables regarding prompt regimes vs. model internal capability states.

A: Our formulation aligns with Reviewer PXa6's reading: both the capability metrics and the PN/PS lower-bound guarantees apply to prompt-information regimes (Ai∈{1,0,∅}A_i \in \lbrace 1, 0, \varnothing \rbrace), not to latent internal states of the model. Accordingly, how a model handles an injected prerequisite (whether adopting it, checking it against the visual input, or overriding it) is part of the behavior measured by NC, IC, and RC rather than an external assumption. We agree that making this scope explicit upfront in the main text improves clarity: in the revised paper, we have added a clarifying note directly in Sec. 3.3 (p. 4, alongside the full discussion in Appendix E, p. 39) stating that IC(XiX_i) measures the model's ability to solve XiX_i when all declared prerequisites are correctly supplied in the input (including integrating that information), and IC − NC measures the error removed by supplying them in context.

Defining interventions at the input level rather than assuming that prompt injection sets a matching internal state is our intentional modeling choice, consistent with how controlled interventions are treated in Pearl's framework. As Pearl (2022) emphasizes, PN and PS are "global (i.e., input–output) features of a causal model, depending only on the function Yx(u)Y_x(u) ... oblivious to the internal workings of the circuit." In black-box MLLM evaluation, assuming perfect internal uptake of input interventions would be unrealistic and would hide behaviors that the metrics are meant to capture: failing to utilize a correct prerequisite in context rightly registers as target-node residual error (1 − IC), whereas overriding a false prerequisite via visual evidence registers in resilience (RC). Keeping the boundary at the input preserves clean input–output semantics for black-box evaluation while leaving internal state tracing to future mechanistic work.

Regarding Assumption 3.3: it is not intended to equate textual injection with conditioning on a latent internal state, and we agree that lacking an internal readout would not justify that leap. Instead, Assumption 3.3 adapts Pearl's exogeneity condition (Pearl, 2009; 2022; Tian & Pearl, 2000) for bounding PN/PS. Whereas classical observational studies passively observe X and must assume exogeneity to identify P(yx)P(y_x) from P(y∣x)P(y \mid x), MLLM evaluation inverts this setting (Remark 3.4): unassisted runs admit no faithful passive readout of intermediate steps (Turpin et al., 2023; Chen et al., 2025b), whereas we directly run controlled experiments by assigning regimes Ai∈{1,0,∅}A_i \in \lbrace 1, 0, \varnothing \rbrace externally across identical instances. Because AiA_i is assigned by the evaluation protocol independently of the unmodeled noise variables (N), exogeneity on the regime variable AiA_i itself, (Ya,Ya′)⊥Ai(Y_a, Y_{a'}) \perp A_i and hence Pc(⋅∣Ai=a)=Pc(⋅∣do(Ai=a))P_c(\cdot \mid A_i = a) = P_c(\cdot \mid do(A_i = a)), holds by construction, which is the only property used in our proofs (Appendix A.2, Theorems A.3 and A.4). Writing Xi=xX_i = x on the left side of the original Eq. (2) was meant only as shorthand that upstream parents are instantiated via AiA_i, and we see how it could be read as referring to a latent state; we have updated Eq. (2) and Remark 3.4 (Sec. 3.2, p. 4) in the revision to state exogeneity directly in terms of AiA_i, matching Appendix A.2.

AuthorsOpenReview ↗

Response to Reviewer PXa6 [part 2]

Q2&W2: Paired rescued (incorrect to correct) and harmed (correct to incorrect) rates from NC to IC, and whether the conclusion holds under raw rescue rates.

A: Following the reviewer's suggestion, we report the paired instance transition rates: rescued (p01p_{01}, incorrect to correct) and harmed (p10p_{10}, correct to incorrect), when injecting correct prerequisites (NC → IC) below. In the Cognitive category, 38.5% of instances flip from incorrect to correct (p01p_{01}) while only 5.8% flip from correct to incorrect (p10p_{10}). Under the raw rescue rate among original errors (p01p_{01}/(1 − NC)), 64.3% of original Cognitive errors are fixed (72%–77% for SOTA models), exceeding the 54% net error reduction ((IC − NC)/(1 − NC)) reported in the paper.

CategoryNCICRescued(p01p_{01})Harmed(p10p_{10})IC-NCp01p_{01}/(1−NC)p10p_{10}/NC(IC−NC)/(1−NC) (paper reported)
Perception0.5910.79724.9%4.4%+0.20560.9%7.4%50.2%
Spatial0.5590.67217.6%6.3%+0.11339.9%11.2%25.6%
Temporal0.4570.61823.9%7.8%+0.16144.0%17.1%29.6%
Cognitive0.4010.72838.5%5.8%+0.32764.3%14.5%54.6%
Overall0.4890.69126.5%6.3%+0.20251.9%12.9%39.5%

Our findings in the paper hold under raw rescue rates as well: Cognitive shows the largest improvement (p01p_{01}) across all categories, aligning with (IC − NC)/(1 − NC), while Spatial and Temporal still leave most errors (1 − IC) after prerequisite injection. We have added these detailed results and per-model breakdown to Appendix D.1 (Tables 10–11).

 


Q3 & W2: At what level is the highest-S prerequisite selected, and does the single-prerequisite recovery hold stably on held-out splits?

A: In our paper (Sec. 5.3, Fig. 8b), the highest-S(1, ∅) prerequisite is selected per overall task and per model, as our goal is to characterize each model's distinct bottleneck behavior. The 84% recovery rate reported in the paper was evaluated across all instances of each task.

Following the reviewer's suggestion, we added held-out split evaluations using 2-fold (50/50) and 5-fold (80/20) cross-validation across task instances, repeated 10 times and averaged: in each split, we identify the highest-S(1, ∅) prerequisite (and each baseline criterion in Fig. 8b) on the selection split, and evaluate the percentage of full-prerequisite gain (IC − NC) recovered on the held-out split. The averaged results are shown below:

Pattern 1 (All / 2-fold / 5-fold)Pattern 2 (All / 2-fold / 5-fold)Overall (All / 2-fold / 5-fold)
S(1, ∅)93.2% / 90.6% / 91.9%74.8% / 69.2% / 69.6%84.4% / 80.4% / 81.2%
N(1, ∅)89.9% / 88.3% / 89.4%42.7% / 41.6% / 41.8%67.3% / 66.0% / 66.7%
min(NC)85.6% / 84.3% / 85.5%36.3% / 34.6% / 34.8%62.0% / 60.6% / 61.3%
min(IC)90.1% / 89.1% / 89.7%33.9% / 35.7% / 34.1%63.2% / 63.6% / 63.1%
max(IC − NC)40.9% / 39.8% / 39.9%39.8% / 38.9% / 39.7%40.4% / 39.4% / 39.8%
Random51.8% / 51.7% / 51.7%24.8% / 24.8% / 24.7%38.8% / 38.8% / 38.8%

On held-out instances, the S(1, ∅)-selected prerequisite recovers 80.4% (2-fold) and 81.2% (5-fold) of the full-prerequisite gain overall, closely matching the 84.4% in-sample result, and continues to substantially outperform all other selection criteria across both splits (especially on multi-parent Pattern 2 tasks: ~70% vs. 35% for min(NC)). Across resampled splits (2-fold / 5-fold), the selected top-1 S(1, ∅) prerequisite matches the full-sample choice in 87.4% / 93.7% of folds, and within-task prerequisite rankings remain highly consistent for both S(1, ∅) (Kendall τ = 0.85 / 0.92, Pearson r = 0.94 / 0.98) and N(1, ∅) (τ = 0.82 / 0.91, r = 0.90 / 0.97), confirming that our findings hold stably on unseen instances. We have added these results to Appendix D.2.

Thank you for the thoughtful response. The held-out results give me more confidence that the selected prerequisite remains effective on new instances of the same task. Your clarification that the causal guarantees apply to the information supplied in the prompt also eases my concern about how to interpret the causal variables.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.