acceptodds
Under review as a conference paper at ICLR 2027

SciReC: Diagnostic Evaluation of Multimodal Multi-Turn Relational Reasoning with Adaptive Interaction

Abstract

Relational reasoning requires the process of perceptual understanding, comparing, and integrating the underlying relationships between concepts. This ability consists of multiple categories, such as analogical, structural, and cause-effect, each capturing a different aspect of higher-order understanding. To examine the performance of multimodal large language models (MLLM) on these relational inference tasks, we developed SciReC, a model-adaptive multimodal academic dialog benchmark. As the relational reasoning process involves multiple representations and various factors (visual understanding, exhibiting knowledge, and memory recall), we propose DMRA, a deficit-based diagnostic framework that quantifies the contribution of these components to identify the primary cause of unsuccessful cases. Claude 4.6 achieved the best performance on the overall relational score with 73%, followed by GPT 5.4 with 68%. Performance trends indicate that open-source models achieve their lowest scores on spatial relations, while proprietary models struggle more with hierarchical and sequential relations. Across domains, model performance is lowest on Astronomy and highest on Psychology. The results of DMRA reveal that relational reasoning is the primary source of error across all models, followed by memory limitations.

Primary area:
datasets and benchmarks
Keywords:
relational reasoning, MLLM, multi-turn interaction, conversational AI, scientific dialogue, deficit-based analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ecsZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SciReC, a model-adaptive multimodal benchmark for multi-turn academic dialogue, designed to evaluate and diagnose the relational reasoning capabilities of multimodal large language models (MLLMs) across complex concepts. Unlike previous benchmarks based on static dialogue flows, SciReC dynamically adjusts subsequent question types according to the model's responses at each turn, covering relational reasoning, knowledge gaps, visual perception, and memory retrieval, thereby enabling adaptive, interactive evaluation. The paper also proposes DMRA, a diagnostic framework based on deficit analysis. Through a two-stage error decomposition, DMRA attributes errors to upstream capabilities, including vision, knowledge, and memory, or to higher-order relational reasoning itself, with the aim of accurately identifying the underlying sources of reasoning failures.

Critical strengths
  1. The study introduces a model-adaptive, dynamic multi-turn evaluation paradigm for academic concepts, addressing the limitations of previous multimodal dialogue benchmarks that rely on fixed, static contexts. SciReC incorporates a conditional interaction flow governed by scoring thresholds: model responses are evaluated immediately, and subsequent question types are adjusted adaptively to probe memory retrieval, memory validation, visual perception, or domain knowledge as needed. This design offers greater flexibility than conventional benchmarks and enables a more comprehensive assessment of the relevant model capabilities.

  2. The experimental analysis indicates that the primary sources of failure differ across models at different capability levels. For frontier models, the central bottleneck lies in relating concepts across images rather than understanding individual images, while retrieval from conversational memory is a common bottleneck across models. These findings provide valuable empirical guidance for targeted improvements to multimodal large language models.

Critical weaknesses
  1. During dynamic multi-turn interactions, the system relies on Claude 4.6 to assign scores on a 1–10 scale in real time, determining subsequent question branches and the error attribution calculations. Preferences for particular linguistic patterns or even minor scoring errors could introduce cascading biases into the entire dialogue trajectory and the subsequent DMRA error decomposition. Furthermore, updates to proprietary API versions pose substantial challenges to long-term evaluation consistency and the cost of reproducing the evaluation using open-source alternatives.

  2. The authors acknowledge that residual errors may remain in the dataset, and the construction pipeline is not fully automated for large-scale expansion to additional evaluation domains. Moreover, the reasoning questions involve only a small number of images and remain relatively simple compared with complex real-world problems.

Questions for authors
  1. How sensitive are the evaluation results to the capabilities of the judge model? In particular, could differences in judgments for responses near the threshold alter the subsequent evaluation flow? Given the potential unavailability of proprietary model APIs, please clarify whether using commonly available open-source models as judges would substantially affect the results.

  2. The paper decomposes errors into relational, memory, knowledge, and visual factors using task-specific score deficits and their weighted combinations. How is the agreement between this decomposition and the actual sources of failure validated? Could expert attribution or targeted information provision experiments be used for validation, for example, by testing whether supplying relevant historical facts or necessary knowledge improves performance on the original relational question as predicted by the attribution? Additionally, how sensitive are the rankings of primary and secondary causes to the weights, smoothing parameter, and scoring errors?

  3. Are there additional calibration criteria or empirical analyses supporting the choice of T = 7 in the scoring scheme? What are the advantages and disadvantages of asking a judge model to make a direct binary pass/fail judgment, compared with the current approach of applying a threshold to a numerical score?

Questions for future investigation:

  1. The paper states that all relational questions underwent manual review, whereas other question types were not subject to the same filtering procedure, and the AI use statement acknowledges the possibility of residual generation errors. If the benchmark is expanded to additional disciplines, which construction and quality assurance stages would still require human involvement, what would the associated costs be, and are feasible automated quality assurance approaches available?

  2. The relational questions in the paper primarily involve two images, whereas complex real-world tasks may require integrating more images, knowledge across chapters, and longer dialogue histories. How to generate questions that require longer-range and more complex understanding is a worthwhile direction for further investigation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ecsZ (part 1)

We thank the reviewer for the insightful comments. We are pleased that you recognize SciReC’s model-adaptive diagnostic evaluation for identifying why models fail and its value in guiding MLLM improvement. Below, we provide detailed responses to your specific inquiries:

"How sensitive are the results to the judge model? Please clarify whether using open-source models as judges would affect the results."

We thank the reviewer for raising the question of sensitivity to the choice of judge model. Claude 4.6 was retained as the primary judge based on its stronger agreement with human evaluation reported in Appendix B. We additionally evaluated the same 199 responses in Appendix B using the open-source Qwen3.5-9B model, and results are provided in the table below.

CategoryNThree-Judge ConsensusThree-Judge ConflictClaude–GPT AgreementClaude–Qwen Agreement
Knowledge8691.9%8.1%94.2%97.7%
Relation2454.2%45.8%58.3%83.3%
Memory2286.4%13.6%95.5%90.9%
Visual6170.5%29.5%90.2%75.4%
Memory validation757.1%42.9%57.1%100.0%
Overall20079.0%21.0%87.5%88.5%

Overall, all three judges produced the same pass/fail decision on 79.0% of cases. When Claude 4.6 is used as the reference, GPT-5.4 agrees with its decision in 87.5% of cases, while Qwen3.5-9B agrees in 177/200 cases (88.5%). The open-source Qwen judge exhibits overall decision consistency with Claude comparable to that of another proprietary judge, especially in the relational reasoning category.

In SciReC, the evaluation flow is affected specifically at two decision points: a relational reasoning score below 7 triggers the diagnostic evaluation, and, within this evaluation, a memory score below 7 triggers the additional memory-validation step. Threshold disagreements for other diagnostic components, such as knowledge or visual understanding, may affect the resulting diagnostic attribution but do not alter which evaluation step is executed.

For relational reasoning, Claude 4.6 and Qwen3.5-9B agree on the threshold decision for 20/24 cases (83.3%), with four cases receiving different decisions around T=7. For memory evaluation, they agree on 20/22 cases (90.9%), with two threshold disagreements; these disagreements can affect whether validation is performed only for cases that have entered the diagnostic branch.

The results indicate that replacing Claude 4.6 with the open-source Qwen3.5-9B does not substantially alter the aggregate evaluation. Differences near the threshold can affect individual branching decisions, particularly for relational reasoning; however, a small score difference around the threshold does not by itself produce a large relational attribution. DMRA considers the magnitude of the relational deficit together with the measured task deficits when determining the primary failure source. Therefore, small judge variations may alter whether further diagnosis is triggered, but substantial relational attribution requires a correspondingly larger unexplained relational deficit.

"Are there additional calibration criteria supporting the choice of T = 7? What are the advantages and disadvantages of asking a judge model to make a direct binary pass/fail judgment, compared with the current approach?"

We thank the reviewer for raising this question. The choice of T=7 is grounded directly in the semantic definition of our scoring rubric rather than selected as a numerical value; see Figures 28 and 29. Scores of 4–6 represent responses in which the main relationship is missing, incorrect, contradicted, or poorly explained, whereas scores of 7–8 require the main relationship and most key facts to be correctly captured. Thus, T=7 represents the predefined boundary between an insufficient response and one that correctly captures the core relationship.

A direct binary pass/fail judgment could simplify this decision and eliminate the need to subsequently apply a numerical threshold. However, it would discard information about the magnitude of the error. For example, responses receiving scores of 1 and 6 would both be classified as failures despite representing substantially different levels of correctness. This distinction is particularly important for DMRA because the magnitude of component deficits contributes to the diagnostic attribution. The numerical scoring provides a semantic pass/fail threshold at T=7T=7 while capturing deficit severity.

We acknowledge that numerical scoring can be sensitive to judge calibration, particularly near the threshold. To reduce this variation, we added explicit scoring constraints defining when responses should fall above or below the threshold. We further assessed the reliability of these decisions through human–judge agreement and cross-judge consistency analyses.

AuthorsOpenReview ↗

Response to Reviewer ecsZ (part 2)

"How is the agreement between the decomposition and the actual sources of failure validated? Could expert attribution or targeted information provision experiments be used for validation?"

We thank the reviewer for this important suggestion. In the revised evaluation, we conducted three complementary validation analyses: human validation of the identified primary failure source, controlled perturbation experiments, and an alternative bottleneck-based attribution analysis.

First, we conducted human validation of the primary failure attribution. Human validators independently examined same 24 model failures and 368 diagnostic questions in total to identify the most plausible and secondary failure source, which was then compared with the DMRA diagnosis. The two evaluators agreed on 23 of 24 cases (95.8%; Cohen’s κ = 0.86), demonstrating strong inter-rater reliability. DMRA’s primary attribution agreed with Human Evaluator 1 in 20 of 24 cases (83.3%) and with Human Evaluator 2 in 19 of 24 cases (79.2%). This provides independent evidence that the primary failure identified by DMRA is consistent with human assessment rather than being determined solely by the attribution formula.

Second, we conducted controlled memory and visual perturbation experiments to test whether DMRA responds to a deliberately introduced and known source of failure. For memory, we removed the preceding multimodal context; for visual understanding, we degraded the corresponding visual information. For memory perturbation, memory became the primary attribution in 87.5% of cases. Across visual cases, visual performance decreased by 7.75 points per figure, and Visual became primary in 80% of cases. These results show that DMRA shifts attribution toward the deliberately impaired component while leaving unaffected components largely unchanged.

Third, we conducted an alternative bottleneck analysis to determine whether relational reasoning remains the primary failure when the additive residual formulation is removed. Under the original DMRA, relational reasoning accounted for 65.9%, 71.7%, and 69.4% of the primary failures for Claude 4.6, GPT-5.4, and Qwen3.5-9B, respectively. Under the bottleneck formulation, the corresponding values were 49.7%, 60.5%, and 52.8%. Relational reasoning remained the most frequent primary failure mode for all three models, indicating that the main conclusion is preserved under an alternative bottleneck attribution mechanism.

Together, these analyses evaluate DMRA from complementary perspectives: human validation examines agreement with independent attribution, controlled perturbations test whether known introduced failures produce the predicted diagnostic shifts, and the bottleneck analysis tests whether the main failure pattern persists without the original additive residual mechanism. Detailed experimental settings and results for each analysis are provided in our responses to the corresponding reviewer comments.

The authors acknowledge that errors may remain in the dataset, and the construction pipeline is not fully automated for large-scale expansion to additional domains. The reasoning questions involve a small number of images and remain simple compared with complex real-world problems.

We thank the reviewer for highlighting these limitations. We acknowledge that some undetected generation errors may remain in the synthetic data. This risk is not unique to our dataset but is a general limitation of studies that use LLM-assisted synthetic data generation, where generated samples may contain occasional inconsistencies even after quality-control procedures. To mitigate this risk, we defined explicit generation rules, manually reviewed a subset of the generated data against both these rules and the source materials, and validated the relational reasoning data used in the experiment. Nevertheless, we conservatively acknowledge the possibility of remaining generation errors rather than assuming that LLM-generated data are error-free.

The current construction pipeline includes two manual quality-control steps: filtering non-scientific images and human validation of the generated relational reasoning questions. Because the model identifies relationships by connecting information across different parts of the textbook, human verification is important to ensure that the generated relationships and questions are valid. Extending the dataset to a new textbook or domain would therefore require these two manual validation steps to maintain data quality.

Our relational tasks use a controlled number of figures. However, a small number of images does not necessarily imply simple reasoning. This controlled design was intentional to isolate prerequisite and relational reasoning failures, while automated construction, broader domains, and more complex multi-image reasoning remain important directions for future work. We will include these limitations in the final draft.

AuthorsOpenReview ↗

Response to Reviewer ecsZ (part 3)

How sensitive are the rankings of primary and secondary causes to the weights, smoothing parameter, and scoring errors?

We thank the reviewer for this question. Although the DMRA weighting parameters are fixed design constants defining the attribution policy rather than tuned hyperparameters, we conducted a robustness analysis of the constants λ\lambda, τ\tau, and λfig\lambda_{\mathrm{fig}} around their predefined values while keeping equal importance across upstream tasks. We did not vary the passing threshold T=7T=7, as doing so would change the semantic definition of a deficit. We evaluated Claude 4.6 answers to determine whether variations alter the primary diagnostic attribution and ranking.

The results are presented in the Table. Primary Stable represents the percentage of individual cases whose primary failure reason remains unchanged relative to the original DMRA configuration, while Secondary Stable represents the percentage whose secondary failure reason remains unchanged. For a more detailed analysis, we additionally evaluated two joint parameter configurations:

Joint-Weak:

λ=0.1,τ=0.05,λfig=0.25\lambda=0.1,\quad \tau=0.05,\quad \lambda_{\mathrm{fig}}=0.25

Joint-Strong:

λ=0.4,τ=0.2,λfig=1.0\lambda=0.4,\quad \tau=0.2,\quad \lambda_{\mathrm{fig}}=1.0

Under the original configuration, the primary failure reasons were 65.87% Relational, 22.75% Memory, 8.98% Visual, and 2.40% Knowledge. Across all parameter variations, the ordering remained unchanged: Relational > Memory > Visual > Knowledge. Relational attribution ranged from 61.08% to 70.06%, Memory from 17.96% to 25.75%, Visual from 8.38% to 10.18%, and Knowledge from 1.80% to 2.99%.

Primary rankings were highly stable, with 93.41–99.40% stability across individual parameter variations, while secondary rankings ranged from 91.62–99.40%. Under the deliberately stronger joint perturbation, primary stability remained 94.61%, secondary stability was 89.82%, and the same two leading causes were retained in 95.21% of cases. These results show that DMRA’s main diagnostic conclusions and primary-cause assignments are robust and not strongly dependent on the exact parameter values.

Removing validation weighting (λ=0\lambda=0) produced a noticeable shift in attribution, particularly from Memory toward Relational; however, the overall conclusion remained unchanged. Removing figure smoothing (λfig=0\lambda_{\mathrm{fig}}=0) produced the largest shift in the primary-cause distribution but did not alter the ordering of the dominant diagnostic causes. Finally, τ=0.1\tau=0.1 implements the intended tie-handling policy but does not drive the overall conclusions.

SettingMemoryKnowledgeVisualRelationalPrimary StableSecondary Stable
Original22.752.408.9865.87100.00100.00
λ=0.00\lambda = 0.0017.961.8010.1870.0693.4192.81
λ=0.10\lambda = 0.1022.162.408.9866.4799.4098.20
λ=0.30\lambda = 0.3023.352.408.9865.2797.0195.21
λ=0.40\lambda = 0.4022.752.408.9865.8796.4192.81
τ=0.00\tau = 0.0025.752.408.3863.4797.0194.61
τ=0.05\tau = 0.0523.352.408.3865.8799.4098.20
τ=0.15\tau = 0.1522.162.998.9865.8799.4099.40
τ=0.20\tau = 0.2020.962.998.9867.0798.2098.20
λfig=0.00\lambda_{\mathrm{fig}} = 0.0025.752.9910.1861.0895.2191.62
λfig=0.25\lambda_{\mathrm{fig}} = 0.2523.952.408.9864.6798.8097.60
λfig=0.75\lambda_{\mathrm{fig}} = 0.7522.162.408.9866.4799.4097.60
λfig=1.00\lambda_{\mathrm{fig}} = 1.0020.962.408.9867.6698.2096.41
Joint-Weak25.152.408.3864.0797.6095.21
Joint-Strong20.962.998.9867.0794.6189.82

To assess whether robustness was model-specific, we repeated the sensitivity analysis across all three models. Primary-cause assignments remained highly stable, and the overall diagnostic ranking remained unchanged across the tested configurations. Thus, although the attribution percentages and a small number of case-level primary or secondary assignments changed under parameter variation, these changes did not alter the main ordering of diagnostic causes or the overall conclusions drawn from DMRA. These results indicate that the reported diagnostic findings are robust to reasonable variations in the weighting and smoothing constants across all three evaluated models.

ModelsPrimary Range (%)Secondary Range (%)
Claude 4.693.41–99.4089.82–99.40
GPT-5.495.12–100.0092.68–100.00
Qwen3.5-9B95.49–99.6592.71–99.65
AuthorsOpenReview ↗

Response to Reviewer ecsZ (part 4)

Questions for future investigation: If the benchmark is expanded to additional disciplines, which construction and quality assurance stages would still require human involvement, what would the associated costs be, and are feasible automated quality assurance approaches available?

We thank the reviewer for raising this important question. When extending SciReC to additional textbooks or disciplines, most of the dataset construction pipeline can remain automated, while two stages currently require human quality control. First, extracted figures are manually filtered to remove non-scientific or unsuitable images. Second, the generated relational questions are manually validated. This step is particularly important because the model constructs relationships by connecting information across different figures and text of a textbook; therefore, human verification ensures that the identified relationship, question, and expected answer are supported by the source material.

The associated cost includes both model inference and human review. GPT is used to generate the questions and ground truths, so adding new textbooks requires additional API/inference costs proportional to the amount of processed content and generated questions. Human effort is additionally required for figure filtering and relational-question validation. Therefore, the total cost depends on the textbook size, number of extracted figures, and number of generated questions rather than being a fixed cost.

These manual stages could also be partially automated in future extensions. For example, multimodal models could pre-filter unsuitable figures and an independent model could verify whether a generated relational question and ground truth are supported by the corresponding figures and source text. Agreement between multiple independent validators could further identify uncertain cases for human review. However, we would retain human verification for ambiguous or low-confidence relational questions rather than relying entirely on automated quality assurance. Thus, a scalable extension could use automated validation as an initial filter and reserve human review for uncertain cases, substantially reducing the manual effort while maintaining quality control.

Questions for future investigation: How to generate questions that require longer-range and more complex understanding is a worthwhile direction for further investigation.

We agree that extending relational reasoning to multiple figures and cross-chapter relationships would provide an important direction for future work. However, the current setting already includes substantial long-range conversational context. The dialogue history accumulates questions throughout the chapter, including relational, memory, knowledge, and visual-understanding questions, rather than containing only the two figures used in a relational question. Thus, although individual relational questions primarily compare two figures, the model must operate within an already extended multimodal dialogue history. Nevertheless, extending the relational task itself to require integration across multiple figures, chapters, and even longer histories would provide an additional level of complexity and is an important direction for future investigation.

We believe these clarifications and experiments address all your concerns and questions. We look forward to further discussion if needed.

Reviewer_NW51OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SciReC, an adaptive multimodal dialogue benchmark containing 189 dialogues and 656 relational questions derived from 12 textbooks across eight academic domains. A judge scores model responses and determines subsequent diagnostic probes. The proposed DMRA framework uses memory, knowledge, and visual scores to allocate failures between upstream deficits and residual relational errors. Experiments evaluate seven MLLMs, with Claude 4.6 achieving 73.8% overall accuracy, and report diagnostic profiles alongside a human-adjudicated judge-selection study.

Critical strengths

(1) The adaptive protocol integrates diagnosis into evaluation. Conditioning follow-up questions on model performance enables targeted assessment of memory, knowledge, and visual understanding within the dialogue.

(2) Textbook-grounded generation, iterative human quality control, and human adjudication of judge disagreements provide useful construction safeguards. Coverage across eight academic domains broadens multimodal dialogue evaluation beyond everyday scenarios.

Critical weaknesses

(1) The conclusion that relational reasoning is the main source of failure is not sufficiently supported. DMRA assigns errors left unexplained by memory, knowledge, and visual checks to relational reasoning, without ruling out other explanations such as unclear questions, incomplete checks, or scoring mistakes.

(2) The reported error diagnoses lack independent validation. The human study evaluates agreement on answer scores, rather than whether DMRA correctly identifies why an answer failed. Expert diagnoses or controlled tests with known error sources are needed to establish the reliability of these conclusions.

Questions for authors

(1) What evidence shows that errors left unexplained by the memory, knowledge, and visual checks are failures to combine information from the figures? Could unclear questions, incomplete checks, or scoring mistakes explain some of these cases?

(2) Do DMRA’s diagnoses agree with independent expert assessments? Can it identify the expected error type when a known problem is deliberately introduced, such as removing earlier context or degrading an image? Please also assess whether changing its weighting parameters changes the main conclusions.

5 comments
AuthorsOpenReview ↗

Response to Reviewer NW51 (part 1)

Thank you for your insightful review and suggestions on our paper. We are pleased that you recognize SciReC’s diagnostic and adaptive evaluation design, broad academic coverage, and rigorous dataset construction and validation. We are also pleased that you recognize SciReC’s contribution in extending multimodal dialogue evaluation beyond conventional static benchmarks and everyday scenarios. Below, we provide detailed responses to your specific inquiries:

What evidence shows that errors left unexplained by the memory, knowledge, and visual checks are failures to combine information from the figures? Could unclear questions, incomplete checks, or scoring mistakes explain some of these cases?

We thank the reviewer for raising this important concern. The relational attribution is not intended to represent arbitrary unexplained error. Memory, knowledge, and visual questions separately evaluate whether the information required for the relational task is retained, known, and correctly understood. After accounting for these prerequisites, the relational task additionally requires cross-image correspondence, comparison, information integration, and inference, which are explicitly considered components of relational reasoning in SciReC. Therefore, when the prerequisites are adequately satisfied, but the relational question fails, the remaining failure emerges at the stage where information from the figures must be combined.

We nevertheless agree that unclear questions, incomplete diagnostic checks, or scoring errors could contribute to individual cases, but they can influence all diagnostic question types, not just relational. General requirements such as language and task understanding are also shared across memory, knowledge, visual, and relational evaluations rather than being unique to the relational stage. Therefore, these sources of uncertainty should not be assumed to accumulate exclusively in the relational residual, which directly affects the related task score.

To mitigate these concerns, all relational questions undergo human validation to reduce question ambiguity and ground-truth errors, while Claude 4.6 was selected as the primary judge based on its agreement with human evaluation to reduce scoring-related uncertainty. We further evaluate these concerns through controlled perturbation, human attribution validation, and an alternative bottleneck formulation, as detailed in the corresponding responses.

"Do DMRA’s diagnoses agree with independent expert assessments?"

To address this concern, we added a controlled diagnostic validation experiment. We selected 24 failed relational reasoning cases, with eight from each model: Claude 4.6, GPT-5.4, and Qwen3.5-9B. These cases contained 368 diagnostic questions in total, including 142 knowledge, 138 visual, 48 memory, 24 relational, and 16 memory questions. Two human evaluators were given the flow of the conversation, including all types of questions, model answers, and ground truth. They were asked, “Based on the available evidence, what are the primary and secondary sources of the model’s error?” The evaluation was blinded to the DMRA diagnosis and task scores. We measured inter-annotator agreement and subsequently compared the adjudicated human diagnoses with those produced by DMRA.

• Primary agreement: DMRA's primary diagnosis matches the human validator's first diagnosis.

• Secondary diagnosis: For cases where the primary diagnosis does not match but DMRA's primary diagnosis matches the human validator's second diagnosis.

EvaluatorPrimary AgreementSecondary Diagnoses
Human 120/24 (83.3%)22/24 (91.7%)
Human 219/24 (79.2%)23/24 (95.8%)

Human evaluators independently identified the primary source of failure and, when applicable, provided a secondary diagnosis. Two independent human evaluators assessed the source of failure for each case by identifying a primary failure component and, when applicable, a secondary diagnosis. The two evaluators agreed on 23 of 24 cases (95.8%), with a Cohen's κ of 0.86, indicating strong inter-rater reliability.

Primary diagnostic agreement was defined as agreement between the human evaluator's primary diagnosis and DMRA's primary attribution. Human Evaluator 1 showed primary agreement in 83.3%, while Human Evaluator 2 showed agreement in 79.2%. When these secondary human diagnoses were considered, the diagnostic agreement increased to 22 of 24 cases (91.7%) and 23 of 24 cases (95.8%), respectively.

Together, the high inter-rater reliability and consistent alignment between independent human diagnoses and DMRA provide evidence that DMRA's diagnostic attributions meaningfully reflect the failure components identified through human assessment, rather than arising solely from its score-based formulation.

AuthorsOpenReview ↗

Response to Reviewer NW51 (part 2)

"Can it identify the expected error type when a known problem is deliberately introduced, such as removing earlier context or degrading an image?"

We thank the reviewer for this valuable suggestion. We would like to emphasize that conducting these controlled perturbation experiments was nontrivial, as they could not be performed by simply modifying the scores obtained from the original evaluation. Each controlled intervention requires a new evaluation pipeline: identifying and matching suitable cases, constructing memory or visual perturbations, re-evaluating the best-performing three models: GPT-5.4, Claude 4.6, and Qwen3.5, scoring the new responses, integrating them with the original diagnostic scores, and recomputing DMRA attribution.

A key challenge was identifying suitable cases for controlled intervention. We required cases near the relational decision threshold with reliable measurements for all other diagnostic components and correctly matched questions, figures, ground truths, and model outputs. Cases with substantial pre-existing deficits were excluded because they could confound the effect of the introduced perturbation. These strict criteria resulted in a limited controlled-perturbation test set.

Removing earlier context

We conducted a controlled memory perturbation on eight relational reasoning cases from three models, selecting cases with relational, knowledge, and visual scores ≥6\geq 6 to minimize other deficits. We removed the preceding multimodal context for the relational and memory questions and restored the corresponding figure for memory validation. Responses were scored using the same protocol as the main study.

The perturbation substantially affected relational performance. Across the eight cases, the relational score decreased by an average of 3.5 points. DMRA identified Memory as the primary diagnostic attribution in 7 of the 8 cases (87.5%). Across all cases, Memory accounted for an average of 77.2% of the diagnostic attribution, compared with 22.3% for the relational residual, 0.2% for Knowledge, and 0.3% for Visual understanding. The negligible Knowledge and Visual contributions are consistent with the controlled manipulation, since these components were not deliberately perturbed. In the only case where Memory was not the primary attribution, the induced change in Memory was smaller than the existing Relational deficit. Consequently, DMRA produced nearly equal attributions to Memory (48.5%) and Relational (51.3%), with Relational remaining slightly higher.

CaseMemoryKnowledgeVisualRelationalPrimary Diagnosis
163.5%0.3%0.3%35.8%Memory
2100.0%0.0%0.0%0.0%Memory
366.1%0.0%0.0%33.9%Memory
4100.0%0.0%0.0%0.0%Memory
599.3%0.0%0.7%0.0%Memory
690.0%0.0%0.0%10.0%Memory
750.1%1.6%0.9%47.4%Memory
848.5%0.0%0.3%51.3%Relational
Overall77.2% avg.0.2%0.3%22.3%Memory: 7/8 (87.5%)

Degraded Image Input

For visual perturbation, we identified five relational reasoning cases from three models, selecting cases with relational, knowledge, and visual scores ≥5\geq 5 to minimize other deficits. As higher scores are hard to find, we had to reduce the score to 5. We added a mask in the center of the images for relational and visual questions. Responses were scored using the same protocol as the main study.

Across five controlled visual-perturbation cases, visual performance decreased by an average of 7.75 points per figure, accompanied by an average 1.60-point decrease in relational reasoning performance. DMRA attributed an average of 69.4% of the observed deficit to visual understanding, compared with 28.4% to the relational residual, 2.2% to knowledge, and 0.0% to memory. Visual understanding was identified as the primary diagnostic attribution in 4 of 5 cases (80%). In the remaining case, Visual still accounted for 44.9% of the attribution but was slightly below the Relational residual (51.4%) due to a large pre-existing relational deficit.

CaseMemoryKnowledgeVisualRelationalPrimary Diagnosis
10.0%0.0%64.0%36.0%Visual
20.0%0.0%62.0%38.0%Visual
30.0%0.0%83.3%16.7%Visual
40.0%7.1%92.8%0.1%Visual
50.0%3.7%44.9%51.4%Relational
Average0.0%2.2%69.4%28.4%Visual: 4/5 (80%)

Together, these controlled perturbation experiments demonstrate that DMRA can identify the expected failure component when a known memory or visual deficit is deliberately introduced, providing direct evidence that its diagnostic attribution responds to the underlying source of failure.

AuthorsOpenReview ↗

Response to Reviewer NW51 (part 3)

“Assess whether changing its weighting parameters changes the main conclusions.”

We thank the reviewer for this suggestion. Although the DMRA weighting parameters are fixed design constants defining the attribution policy rather than learned or tuned hyperparameters, we conducted a robustness analysis of the constants λ\lambda, τ\tau, and λfig\lambda_{\mathrm{fig}} around their predefined values while keeping equal importance across upstream tasks. We did not vary the passing threshold T=7T=7, as doing so would change the semantic definition of a deficit. We evaluated Claude 4.6 model answers to determine whether these variations alter the primary diagnostic attribution and ranking.

The results are provided in the table below. Primary Stable represents the percentage of individual cases whose primary failure reason remains unchanged relative to the original DMRA configuration, while Secondary Stable represents the percentage whose secondary failure reason remains unchanged. For a more detailed analysis, we additionally evaluated two joint parameter configurations:

Joint-Weak:

λ=0.1,τ=0.05,λfig=0.25\lambda=0.1,\quad \tau=0.05,\quad \lambda_{\mathrm{fig}}=0.25

Joint-Strong:

λ=0.4,τ=0.2,λfig=1.0\lambda=0.4,\quad \tau=0.2,\quad \lambda_{\mathrm{fig}}=1.0

Under the original configuration, the primary failure reasons were 65.87% Relational, 22.75% Memory, 8.98% Visual, and 2.40% Knowledge. Across all parameter variations, the ordering remained unchanged: Relational > Memory > Visual > Knowledge. Relational attribution ranged from 61.08% to 70.06%, Memory from 17.96% to 25.75%, Visual from 8.38% to 10.18%, and Knowledge from 1.80% to 2.99%.

Primary rankings were highly stable, with 93.41–99.40% stability across individual parameter variations, while secondary rankings ranged from 91.62–99.40%. Under the deliberately stronger joint perturbation, primary stability remained 94.61%, secondary stability was 89.82%, and the same two leading causes were retained in 95.21% of cases. These results show that DMRA’s main diagnostic conclusions and primary-cause assignments are robust and not strongly dependent on the exact parameter values.

Removing validation weighting (λ=0\lambda=0) produced a noticeable shift in attribution, particularly from Memory toward Relational; however, the overall conclusion remained unchanged. Removing figure smoothing (λfig=0\lambda_{\mathrm{fig}}=0) produced the largest shift in the primary-cause distribution but did not alter the ordering of the dominant diagnostic causes. Finally, τ=0.1\tau=0.1 implements the intended tie-handling policy but does not drive the overall conclusions.

SettingMemoryKnowledgeVisualRelationalPrimary StableSecondary Stable
Original22.752.408.9865.87100.00100.00
λ=0.00\lambda = 0.0017.961.8010.1870.0693.4192.81
λ=0.10\lambda = 0.1022.162.408.9866.4799.4098.20
λ=0.30\lambda = 0.3023.352.408.9865.2797.0195.21
λ=0.40\lambda = 0.4022.752.408.9865.8796.4192.81
τ=0.00\tau = 0.0025.752.408.3863.4797.0194.61
τ=0.05\tau = 0.0523.352.408.3865.8799.4098.20
τ=0.15\tau = 0.1522.162.998.9865.8799.4099.40
τ=0.20\tau = 0.2020.962.998.9867.0798.2098.20
λfig=0.00\lambda_{\mathrm{fig}} = 0.0025.752.9910.1861.0895.2191.62
λfig=0.25\lambda_{\mathrm{fig}} = 0.2523.952.408.9864.6798.8097.60
λfig=0.75\lambda_{\mathrm{fig}} = 0.7522.162.408.9866.4799.4097.60
λfig=1.00\lambda_{\mathrm{fig}} = 1.0020.962.408.9867.6698.2096.41
Joint-Weak25.152.408.3864.0797.6095.21
Joint-Strong20.962.998.9867.0794.6189.82

To assess whether robustness was model-specific, we repeated the sensitivity analysis across all three models. Primary-cause assignments remained highly stable, and the overall diagnostic ranking remained unchanged across the tested configurations. Thus, although the attribution percentages and a small number of case-level primary or secondary assignments changed under parameter variation, these changes did not alter the main ordering of diagnostic causes or the overall conclusions drawn from DMRA. These results indicate that the reported diagnostic findings are robust to reasonable variations in the weighting and smoothing constants across all three evaluated models.

ModelsPrimary Range (%)Secondary Range (%)
Claude 4.693.41–99.4089.82–99.40
GPT-5.495.12–100.0092.68–100.00
Qwen3.5-9B95.49–99.6592.71–99.65

We believe our response and new experiments address your concerns comprehensively and encourage you to reevaluate our submission. We look forward to further discussion.

Reviewer_mDcnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents SciReC, a benchmark built from 12 OpenStax college textbooks covering 8 academic subjects to test how well MLLMs handle relational reasoning across multiple images. Instead of using a fixed question script, the benchmark uses Claude 4.6 as an online judge to branch dynamically into diagnostic sub-questions (checking memory, visual perception, and knowledge) whenever a model fails a high-level relational query. The authors also propose DMRA, an error attribution formula that splits failures into upstream perceptual/knowledge deficits versus true relational reasoning errors. Results are shown across 7 models, with Claude 4.6 and GPT-5.4 performing best.

Critical strengths

(1) The dynamic branching setup is definitely the most interesting part of this work. Most multi-image QA datasets follow a rigid script regardless of what the model says. Here, probing intermediate perceptual or memory failures right after a high-level relational mistake allows the benchmark to isolate why the model failed rather than just recording a wrong answer.

(2) Grounding the relations in college-level science and economics diagrams provides a much higher cognitive ceiling than existing multi-image benchmarks, which often rely on casual photo albums or simple visual puzzles.

Critical weaknesses

(1) The DMRA formulation in Eq. (5) looks mathematically circular. The relational error is defined strictly as a residual: Crel=F−min⁡(F,U)C_{rel} = F - \min(F, U), where UU is the upstream deficit (perception, knowledge, memory). For stronger models like Claude 4.6 or GPT-5.4, their single-image perception and basic knowledge are already quite good, meaning UU is small by definition. As a result, the residual CrelC_{rel} is mathematically forced to absorb almost all the remaining error. Fig. 5 claims that top models primarily suffer from relational reasoning deficits (~70%), but to me this looks like an artifact of how Eq. 5 was formulated rather than a genuine cognitive finding. Also, summing these components linearly with arbitrary weights (λ=0.2,τ=0.1\lambda=0.2, \tau=0.1) lacks theoretical grounding.

(2) There is no blind / text-only baseline, which is a major red flag for this benchmark. In Section 3.2, the authors state that relations and reference answers were extracted purely from textbook text using GPT-5.4 without looking at the images, and the questions were populated using these text extractions. Since OpenStax textbooks are widely represented in the pretraining corpora of frontier models, we need to know how many questions can be answered from parametric language memory alone. Without an image-free ablation (evaluating models with the visual inputs masked), it is impossible to tell whether SciReC tests multimodal reasoning or just textbook memorization.

(3) Using Claude 4.6 as the online routing judge introduces a clear evaluation confounder, especially since Claude 4.6 is also reported as the top-performing model in Table 1. If the judge exhibits even slight self-preference at the margin, it will trigger fewer penalty branches for itself, confounding the conversational scores across different model families.

Questions for authors

(1) Can you report results for Claude 4.6, GPT-5.4, and Qwen-3.5 when image inputs are completely masked (text-only evaluation)? What proportion of the relational questions can be solved without looking at the figures?

(2) If DMRA is formulated as a bottleneck (e.g. failure occurs if perception OR knowledge drops below a threshold) instead of an additive residual, does relational reasoning still show up as the primary failure mode in Figure 5?

(3) Did you check whether Claude 4.6 has an intrinsic self-preference bias when serving as the routing judge compared to using GPT-5.4 as an external evaluator?

(4) In your qualitative inspection, did you see models quoting textbook text verbatim even when the diagram contradicted the answer?

3 comments
AuthorsOpenReview ↗

Response to Reviewer mDcn (part 1)

Thank you for your insightful review and suggestions on our paper. We are pleased that you recognized SciReC’s contribution in advancing static multimodal benchmarks through adaptive diagnostic evaluation, conditional interaction, and grounded relational reasoning, while providing broader cognitive evaluation than existing multi-image benchmarks. Below, we provide detailed responses to your specific inquiries:

“The residual is mathematically forced to absorb almost all the remaining error. Fig. 5 claims that top models primarily suffer from relational reasoning deficits (~70%), but to me this looks like an artifact of how Eq. 5 was formulated rather than a genuine cognitive finding.”

We thank the reviewer for raising this concern. We would like to clarify that the relational component in DMRA is not intended to represent an arbitrary residual or an unspecified source of error. In SciReC, relational reasoning is the downstream process of integrating information and relations across images to derive an inference, while perception, knowledge, and memory serve as prerequisites. Thus, failures remaining after these prerequisites are accounted for are attributed to the relational reasoning process. Importantly, strong prerequisite performance does not mathematically force a large relational deficit. The relational deficit still depends on how poorly the model performs on the relational task itself. For example, consider two cases with the same small combined upstream deficit of 0.4 from memory, knowledge, and visual understanding:

• If the relational-task deficit is 1, the remaining relational deficit is 1−0.4=0.61-0.4=0.6.

• If the relational-task deficit is 5, the remaining relational deficit is 5−0.4=4.65-0.4=4.6.

Thus, even with similarly strong prerequisite performance, relational attribution can be small or large depending on the observed relational-task failure. Relational attribution therefore becomes prominent only when the upstream deficits are insufficient to explain the relational deficit. This is also reflected at the model level: weaker prerequisite performance increases upstream attribution, whereas stronger prerequisite performance shifts attribution toward relational reasoning. This pattern is consistent with DMRA’s intended diagnostic behavior rather than assigning unexplained failures to relational reasoning by construction.

Finally, we conducted human attribution validation, controlled memory and visual perturbation, bottleneck analysis, and parameter-sensitivity analysis to validate the reliability and robustness of DMRA, which are provided in our responses to the related reviewer comments.

“Summing these components linearly with arbitrary weights lacks theoretical grounding.”

We thank the reviewer for raising the concern regarding the weighting scheme. The weights in DMRA are not intended to represent theoretically unique estimates of the relative importance of each component; rather, they operationalize how evidence from the diagnostic evaluations is combined. The linear aggregation is likewise intentional: DMRA is designed as an interpretable error-attribution model that partitions the contribution of measured prerequisite deficits to an observed relational failure.

To examine the effect of parameters, we conducted a sensitivity analysis by varying the weighting parameters individually and jointly. Across the tested settings, relational reasoning consistently remains the largest attributed failure component, ranging from 61.08% to 70.06%, compared with 65.87% under the original configuration. More importantly, the primary diagnosis remains unchanged for 93.41–99.40% of cases under individual parameter variations, while secondary diagnosis stability ranges from 91.62–99.40%. Even under joint perturbations of multiple parameters, primary diagnosis stability remains 94.61–97.60%. To determine whether the observed robustness was model-specific, we repeated the parameter sensitivity analysis for all three evaluated models. Across the tested weighting and smoothing configurations, primary-cause assignments remained highly stable for Claude 4.6, GPT-5.4, and Qwen3.5-9B. Importantly, the overall ranking of the diagnostic causes did not change across the tested parameter configurations. These results indicate that the resulting diagnostic ranking is highly stable and does not depend on a particular weight configuration.

Sensitivity analysisTested rangeRelational attribution rangePrimary diagnosis stabilitySecondary diagnosis stability
λ\lambda0.00–0.4065.27–70.06%93.41–99.40%92.81–98.20%
τ\tau0.00–0.2063.47–67.07%97.01–99.40%94.61–99.40%
λfig\lambda_{\mathrm{fig}}0.00–1.0061.08–67.66%95.21–99.40%91.62–97.60%
Joint perturbationWeak–Strong64.07–67.07%94.61–97.60%89.82–95.21%
Original—65.87%100%100%
AuthorsOpenReview ↗

Response to Reviewer mDcn (part 2)

"Can you report results for Claude 4.6, GPT-5.4, and Qwen-3.5 when image inputs are completely masked (text-only evaluation)? What proportion of the relational questions can be solved without looking at the figures?"

We thank the reviewer for raising this meaningful question regarding potential data leakage and whether the questions can be answered without visual information. To reduce potential data leakage, we replace textbook figure names with anonymous identifiers, preventing models from exploiting textbook-specific figure references or numbering to retrieve memorized OpenStax content. In addition, the relational questions are generated using templates rather than being directly extracted from the textbooks; the questions are explicitly stated in the source text. We also conducted human-expert validation on a subset of the questions to verify that they could not be answered from the textual content alone and required information from the corresponding figures. Together, these measures reduce the likelihood that successful performance results from memorized textbook content or source-specific cues rather than reasoning over the provided visual information.

To further address this concern, we conducted an additional experiment in which the figures were removed from all 656 relational questions and the models were asked to answer using only the textual questions. The results are provided in the table below. Across all three evaluated models, only a small proportion of relational questions could be solved without access to the figures. GPT-5.4 achieved the highest proportion at 13.72% (90/656), followed by Claude 4.6 at 6.86% (45/656) and Qwen3.5-9B at 6.40% (42/656). These results indicate that the large majority of the relational questions require information provided by the associated figures rather than being answerable from textual information alone.

Judge ModelQuestions Scored ≥ 7Proportion Solved Without FiguresAverage Score
GPT-5.49013.72%3.52/10
Claude 4.6456.86%2.79/10
Qwen3.5-9B426.40%2.69/10

“Did you check whether Claude 4.6 has an intrinsic self-preference bias when serving as the routing judge compared to using GPT-5.4 as an external evaluator?”

We thank the reviewer for raising the concern regarding potential self-preference bias when Claude 4.6 evaluates responses generated by Claude 4.6. Claude 4.6 was selected as the primary judge based on its stronger agreement with human evaluations. To examine this effect, we independently re-evaluated the same 662 Claude 4.6 responses using GPT-5.4 as the judge and compared the resulting scores. The results do not indicate score inflation due to Claude self-preference bias. GPT-5.4 assigned higher scores in 75.23% of cases, whereas Claude assigned higher scores in only 7.25%, with GPT-5.4 producing a 1.08-point higher mean score overall (8.21 vs. 7.13). The two judges gave the exact same score for 116 cases (17.52%). These results provide no evidence of self-preference score inflation; instead, Claude 4.6 appears to apply a more conservative scoring criterion than GPT 5.4.

Evaluation of Claude 4.6 ResponsesClaude 4.6 JudgeGPT-5.4 Judge
Mean score7.138.21
Higher score than the other judge48 (7.25%)498 (75.23%)
Exact score agreement116 (17.52%)116 (17.52%)

"In your qualitative inspection, did you see models quoting textbook text verbatim even when the diagram contradicted the answer?”

We thank the reviewer for raising this question. We did not have a specific intervention in which the figures were modified to contradict the textbook content, and the models were subsequently tested on these altered images. However, we inspected Claude 4.6 and GPT 5.4 responses from our main experimental setting to determine whether the models referenced any textbook in their answers. We did not observe cases in which the models quoted or explicitly referenced the specific source textbook in their answers.

AuthorsOpenReview ↗

Response to Reviewer mDcn (part 3)

“If DMRA is formulated as a bottleneck (e.g. failure occurs if perception OR knowledge drops below a threshold) instead of an additive residual, does relational reasoning still show up as the primary failure mode in Figure 5?”

We thank the reviewer for this suggestion. Since the reviewer did not specify a particular mathematical formulation for the bottleneck alternative, we implemented a simple bottleneck formulation using the same threshold as DMRA (T = 7). For each figure, the deficit of each upstream component is calculated directly from its evaluation score: DM,K,V=max⁡(0,T−SM,K,V)D_{M,K,V} = \max(0, T - S_{M,K,V}) For Memory, validation provides additional evidence: improvement after validation increases the memory deficit by one, no change leaves it unchanged, and a lower validation score decreases it by one; no memory deficit is assigned when the original memory score already satisfies the threshold. In DMRA, the validation-memory difference is calculated in its weighting mechanism, but the bottleneck method uses discrete scores for finding the largest deficits; therefore, the effect of validation on memory questions changes discretely to show any score changes.

For cases containing multiple figures, the largest deficit across figures is retained for each upstream component. The relational deficit is similarly calculated as:

DR=max⁡(0,T−SR)D_R = \max(0, T - S_R)

Unlike DMRA's additive attribution, the bottleneck formulation directly selects the component with the largest deficit:

Cprimary=arg⁡max⁡(DM,DK,DV,DR)C_{\mathrm{primary}} = \arg\max(D_M, D_K, D_V, D_R)

Equal maximum deficits are recorded as ties.

The conclusion remains consistent across all three models. Under the original additive analysis, relational reasoning is the most common first cause for Claude 4.6 (65.9%), GPT-5.4 (71.7%), and Qwen3.5-9B (69.4%). Under the bottleneck formulation, relational reasoning remains the largest primary failure mode for Claude 4.6 (49.7%), GPT-5.4 (60.5%), and Qwen3.5-9B (52.8%). Therefore, the main conclusion of Figure 5 is preserved under an alternative bottleneck-based formulation and is not an artifact of the additive residual formulation.

ModelMethodMemoryKnowledgeVisualRelationalTied
Claude 4.6DMRA22.8%2.4%9.0%65.9%—
Bottleneck29.34%4.79%13.17%49.70%2.99%
GPT-5.4DMRA18.0%2.9%7.3%71.7%—
Bottleneck21.46%4.39%9.76%60.49%3.90%
Qwen3.5-9BDMRA19.1%4.5%6.9%69.4%—
Bottleneck24.31%6.94%11.46%52.78%4.51%

The differences in memory and relational diagnoses are partly attributable to the different granularity and decision mechanisms of the two formulations. DMRA performs continuous, distributed attribution across components whereas the bottleneck formulation makes a discrete assignment based solely on the largest deficit. Because the evaluation scores are relatively coarse, even small differences between memory and relational deficits can switch the primary bottleneck, making this formulation more sensitive to evaluator-assigned scores. This difference in granularity and score sensitivity explains the increased memory attribution and corresponding reduction in relational attribution under the bottleneck formulation. Despite these attribution shifts, relational reasoning remains the most frequently identified primary failure mode across all three models, preserving the main conclusion.

We would be happy to consider an alternative bottleneck formulation if the reviewer has a specific formulation or additional analysis in mind.

We believe our response and new experiments address your concerns comprehensively and encourage you to reevaluate our submission. We look forward to further discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response to Reviewers

We greatly appreciate the reviewers’ thoughtful and constructive feedback. We are pleased to note their unanimous positive assessments, which highlight the strengths of our work across multiple dimensions.

  • Reviewers mDcn, NW51, and ecsZ highlight the contribution of SciReC in integrating diagnosis into evaluation that identifies why the model fails rather than just recording incorrect answers, and reveals how primary sources of failure differ across models with varying capability levels.

  • Reviewers mDcn and ecsZ acknowledge the significance of SciReC in addressing fixed and static contexts regardless of model output commonly used in multi-image QA datasets by introducing a model-adaptive, dynamic multi-turn evaluation paradigm.

  • Reviewers mDcn and NW51 highlight that the implemented conditional interaction flow design offers more flexibility than conventional benchmarks and enables a more comprehensive, targeted assessment of the relevant model capabilities within dialogue.

  • The reviewers acknowledge that coverage of SciReC across 8 college-level academic domains broadens multimodal dialogue evaluation beyond everyday scenarios (NW51), and grounded relations provide a much higher cognitive ceiling than existing multi-image benchmarks (mDcn).

  • Reviewer ecsZ recognizes the significance of findings of SciReC in providing valuable empirical guidance for MLLM improvement.

  • Reviewer NW51 emphasizes the critical stages of dataset construction safeguards of SciReC: textbook-grounded generation, iterative human quality control, and human adjudication of judge disagreements.

  • Additionally, we are pleased that all reviewers, mDcn, NW51, and ecsZ, acknowledged that they learned something interesting and useful from the study.

We have addressed each reviewer’s comments in detail. Below, we summarize our responses to the key points and offer additional clarifications.

Summary of Ablations

  • Human validation of DMRA: Tested whether DMRA diagnoses agree with independently identified failure sources. Human evaluators showed 95.8% inter-rater agreement (κ = 0.86), while DMRA's primary diagnosis agreed with Human 1 in 83.3% and Human 2 in 79.2% of cases. This provides independent evidence that DMRA's primary diagnoses correspond to failure sources identified through human assessment, rather than arising solely from the attribution formula.

  • DMRA parameter sensitivity: Tested whether the conclusions depend on the weighting and figure-balancing constants. Primary diagnoses, relational reasoning remained highly stable (93.41–100% across the three models), and the overall ordering of diagnostic causes remained unchanged, with Relational > Memory > Visual > Knowledge. The results show that DMRA’s primary failure rankings remain robust across different weighting configurations.

  • Alternative bottleneck formulation: Tested whether the dominance of relational failures is caused by DMRA's additive residual formulation. We implemented an alternative bottleneck-based diagnosis that identifies the component with the largest score deficit relative to the threshold. Relational remained the most frequent primary failure: 49.7% for Claude 4.6, 60.5% for GPT-5.4, and 52.8% for Qwen3.5-9B under the bottleneck formulation. This resulst shows that the dominance of relational reasoning is not specific to the additive residual formulation and persists under an alternative method.

  • Controlled memory perturbation: Tested whether deliberately removing historical context causes DMRA to identify Memory as the source of failure. Memory became the primary diagnosis in 87.5%.

  • Controlled visual perturbation: Tested whether deliberately degrading figure information causes DMRA to identify visual understanding as the source of failure. Visual scores decreased by 7.75 points per figure on average, and visual became the primary diagnosis (80%).

  • Text-only evaluation: Tested whether relational questions could be solved without figures. Only 13.72% (GPT-5.4), 6.86% (Claude 4.6), and 6.40% (Qwen3.5-9B) were successfully answered, showing that most questions require visual information.

  • Claude self-preference bias: Tested whether Claude 4.6 favors its own responses when acting as judge. GPT-5.4 actually assigned higher scores in 75.23% of cases, providing no evidence of Claude self-preference score inflation.

  • Qwen3.5-9B as an open-source judge: Tested whether replacing the proprietary judge substantially changes evaluation decisions. Qwen3.5-9B achieved 88.5% overall agreement with Claude 4.6, comparable to GPT-5.4's 87.5%, indicating that the aggregate evaluation is not strongly dependent on a proprietary judge.

We thank the reviewers for their time and detailed feedback. Together, these findings show the robustness and reliability of DMRA in identifying meaningful sources of model failure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.