- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces SciReC, a model-adaptive multimodal benchmark for multi-turn academic dialogue, designed to evaluate and diagnose the relational reasoning capabilities of multimodal large language models (MLLMs) across complex concepts. Unlike previous benchmarks based on static dialogue flows, SciReC dynamically adjusts subsequent question types according to the model's responses at each turn, covering relational reasoning, knowledge gaps, visual perception, and memory retrieval, thereby enabling adaptive, interactive evaluation. The paper also proposes DMRA, a diagnostic framework based on deficit analysis. Through a two-stage error decomposition, DMRA attributes errors to upstream capabilities, including vision, knowledge, and memory, or to higher-order relational reasoning itself, with the aim of accurately identifying the underlying sources of reasoning failures.
Critical strengths
-
The study introduces a model-adaptive, dynamic multi-turn evaluation paradigm for academic concepts, addressing the limitations of previous multimodal dialogue benchmarks that rely on fixed, static contexts. SciReC incorporates a conditional interaction flow governed by scoring thresholds: model responses are evaluated immediately, and subsequent question types are adjusted adaptively to probe memory retrieval, memory validation, visual perception, or domain knowledge as needed. This design offers greater flexibility than conventional benchmarks and enables a more comprehensive assessment of the relevant model capabilities.
-
The experimental analysis indicates that the primary sources of failure differ across models at different capability levels. For frontier models, the central bottleneck lies in relating concepts across images rather than understanding individual images, while retrieval from conversational memory is a common bottleneck across models. These findings provide valuable empirical guidance for targeted improvements to multimodal large language models.
Critical weaknesses
-
During dynamic multi-turn interactions, the system relies on Claude 4.6 to assign scores on a 1–10 scale in real time, determining subsequent question branches and the error attribution calculations. Preferences for particular linguistic patterns or even minor scoring errors could introduce cascading biases into the entire dialogue trajectory and the subsequent DMRA error decomposition. Furthermore, updates to proprietary API versions pose substantial challenges to long-term evaluation consistency and the cost of reproducing the evaluation using open-source alternatives.
-
The authors acknowledge that residual errors may remain in the dataset, and the construction pipeline is not fully automated for large-scale expansion to additional evaluation domains. Moreover, the reasoning questions involve only a small number of images and remain relatively simple compared with complex real-world problems.
Questions for authors
-
How sensitive are the evaluation results to the capabilities of the judge model? In particular, could differences in judgments for responses near the threshold alter the subsequent evaluation flow? Given the potential unavailability of proprietary model APIs, please clarify whether using commonly available open-source models as judges would substantially affect the results.
-
The paper decomposes errors into relational, memory, knowledge, and visual factors using task-specific score deficits and their weighted combinations. How is the agreement between this decomposition and the actual sources of failure validated? Could expert attribution or targeted information provision experiments be used for validation, for example, by testing whether supplying relevant historical facts or necessary knowledge improves performance on the original relational question as predicted by the attribution? Additionally, how sensitive are the rankings of primary and secondary causes to the weights, smoothing parameter, and scoring errors?
-
Are there additional calibration criteria or empirical analyses supporting the choice of T = 7 in the scoring scheme? What are the advantages and disadvantages of asking a judge model to make a direct binary pass/fail judgment, compared with the current approach of applying a threshold to a numerical score?
Questions for future investigation:
-
The paper states that all relational questions underwent manual review, whereas other question types were not subject to the same filtering procedure, and the AI use statement acknowledges the possibility of residual generation errors. If the benchmark is expanded to additional disciplines, which construction and quality assurance stages would still require human involvement, what would the associated costs be, and are feasible automated quality assurance approaches available?
-
The relational questions in the paper primarily involve two images, whereas complex real-world tasks may require integrating more images, knowledge across chapters, and longer dialogue histories. How to generate questions that require longer-range and more complex understanding is a worthwhile direction for further investigation.
4 comments
Response to Reviewer ecsZ (part 2)
"How is the agreement between the decomposition and the actual sources of failure validated? Could expert attribution or targeted information provision experiments be used for validation?"
We thank the reviewer for this important suggestion. In the revised evaluation, we conducted three complementary validation analyses: human validation of the identified primary failure source, controlled perturbation experiments, and an alternative bottleneck-based attribution analysis.
First, we conducted human validation of the primary failure attribution. Human validators independently examined same 24 model failures and 368 diagnostic questions in total to identify the most plausible and secondary failure source, which was then compared with the DMRA diagnosis. The two evaluators agreed on 23 of 24 cases (95.8%; Cohen’s κ = 0.86), demonstrating strong inter-rater reliability. DMRA’s primary attribution agreed with Human Evaluator 1 in 20 of 24 cases (83.3%) and with Human Evaluator 2 in 19 of 24 cases (79.2%). This provides independent evidence that the primary failure identified by DMRA is consistent with human assessment rather than being determined solely by the attribution formula.
Second, we conducted controlled memory and visual perturbation experiments to test whether DMRA responds to a deliberately introduced and known source of failure. For memory, we removed the preceding multimodal context; for visual understanding, we degraded the corresponding visual information. For memory perturbation, memory became the primary attribution in 87.5% of cases. Across visual cases, visual performance decreased by 7.75 points per figure, and Visual became primary in 80% of cases. These results show that DMRA shifts attribution toward the deliberately impaired component while leaving unaffected components largely unchanged.
Third, we conducted an alternative bottleneck analysis to determine whether relational reasoning remains the primary failure when the additive residual formulation is removed. Under the original DMRA, relational reasoning accounted for 65.9%, 71.7%, and 69.4% of the primary failures for Claude 4.6, GPT-5.4, and Qwen3.5-9B, respectively. Under the bottleneck formulation, the corresponding values were 49.7%, 60.5%, and 52.8%. Relational reasoning remained the most frequent primary failure mode for all three models, indicating that the main conclusion is preserved under an alternative bottleneck attribution mechanism.
Together, these analyses evaluate DMRA from complementary perspectives: human validation examines agreement with independent attribution, controlled perturbations test whether known introduced failures produce the predicted diagnostic shifts, and the bottleneck analysis tests whether the main failure pattern persists without the original additive residual mechanism. Detailed experimental settings and results for each analysis are provided in our responses to the corresponding reviewer comments.
The authors acknowledge that errors may remain in the dataset, and the construction pipeline is not fully automated for large-scale expansion to additional domains. The reasoning questions involve a small number of images and remain simple compared with complex real-world problems.
We thank the reviewer for highlighting these limitations. We acknowledge that some undetected generation errors may remain in the synthetic data. This risk is not unique to our dataset but is a general limitation of studies that use LLM-assisted synthetic data generation, where generated samples may contain occasional inconsistencies even after quality-control procedures. To mitigate this risk, we defined explicit generation rules, manually reviewed a subset of the generated data against both these rules and the source materials, and validated the relational reasoning data used in the experiment. Nevertheless, we conservatively acknowledge the possibility of remaining generation errors rather than assuming that LLM-generated data are error-free.
The current construction pipeline includes two manual quality-control steps: filtering non-scientific images and human validation of the generated relational reasoning questions. Because the model identifies relationships by connecting information across different parts of the textbook, human verification is important to ensure that the generated relationships and questions are valid. Extending the dataset to a new textbook or domain would therefore require these two manual validation steps to maintain data quality.
Our relational tasks use a controlled number of figures. However, a small number of images does not necessarily imply simple reasoning. This controlled design was intentional to isolate prerequisite and relational reasoning failures, while automated construction, broader domains, and more complex multi-image reasoning remain important directions for future work. We will include these limitations in the final draft.
Response to Reviewer ecsZ (part 3)
How sensitive are the rankings of primary and secondary causes to the weights, smoothing parameter, and scoring errors?
We thank the reviewer for this question. Although the DMRA weighting parameters are fixed design constants defining the attribution policy rather than tuned hyperparameters, we conducted a robustness analysis of the constants , , and around their predefined values while keeping equal importance across upstream tasks. We did not vary the passing threshold , as doing so would change the semantic definition of a deficit. We evaluated Claude 4.6 answers to determine whether variations alter the primary diagnostic attribution and ranking.
The results are presented in the Table. Primary Stable represents the percentage of individual cases whose primary failure reason remains unchanged relative to the original DMRA configuration, while Secondary Stable represents the percentage whose secondary failure reason remains unchanged. For a more detailed analysis, we additionally evaluated two joint parameter configurations:
Joint-Weak:
Joint-Strong:
Under the original configuration, the primary failure reasons were 65.87% Relational, 22.75% Memory, 8.98% Visual, and 2.40% Knowledge. Across all parameter variations, the ordering remained unchanged: Relational > Memory > Visual > Knowledge. Relational attribution ranged from 61.08% to 70.06%, Memory from 17.96% to 25.75%, Visual from 8.38% to 10.18%, and Knowledge from 1.80% to 2.99%.
Primary rankings were highly stable, with 93.41–99.40% stability across individual parameter variations, while secondary rankings ranged from 91.62–99.40%. Under the deliberately stronger joint perturbation, primary stability remained 94.61%, secondary stability was 89.82%, and the same two leading causes were retained in 95.21% of cases. These results show that DMRA’s main diagnostic conclusions and primary-cause assignments are robust and not strongly dependent on the exact parameter values.
Removing validation weighting () produced a noticeable shift in attribution, particularly from Memory toward Relational; however, the overall conclusion remained unchanged. Removing figure smoothing () produced the largest shift in the primary-cause distribution but did not alter the ordering of the dominant diagnostic causes. Finally, implements the intended tie-handling policy but does not drive the overall conclusions.
| Setting | Memory | Knowledge | Visual | Relational | Primary Stable | Secondary Stable |
|---|---|---|---|---|---|---|
| Original | 22.75 | 2.40 | 8.98 | 65.87 | 100.00 | 100.00 |
| 17.96 | 1.80 | 10.18 | 70.06 | 93.41 | 92.81 | |
| 22.16 | 2.40 | 8.98 | 66.47 | 99.40 | 98.20 | |
| 23.35 | 2.40 | 8.98 | 65.27 | 97.01 | 95.21 | |
| 22.75 | 2.40 | 8.98 | 65.87 | 96.41 | 92.81 | |
| 25.75 | 2.40 | 8.38 | 63.47 | 97.01 | 94.61 | |
| 23.35 | 2.40 | 8.38 | 65.87 | 99.40 | 98.20 | |
| 22.16 | 2.99 | 8.98 | 65.87 | 99.40 | 99.40 | |
| 20.96 | 2.99 | 8.98 | 67.07 | 98.20 | 98.20 | |
| 25.75 | 2.99 | 10.18 | 61.08 | 95.21 | 91.62 | |
| 23.95 | 2.40 | 8.98 | 64.67 | 98.80 | 97.60 | |
| 22.16 | 2.40 | 8.98 | 66.47 | 99.40 | 97.60 | |
| 20.96 | 2.40 | 8.98 | 67.66 | 98.20 | 96.41 | |
| Joint-Weak | 25.15 | 2.40 | 8.38 | 64.07 | 97.60 | 95.21 |
| Joint-Strong | 20.96 | 2.99 | 8.98 | 67.07 | 94.61 | 89.82 |
To assess whether robustness was model-specific, we repeated the sensitivity analysis across all three models. Primary-cause assignments remained highly stable, and the overall diagnostic ranking remained unchanged across the tested configurations. Thus, although the attribution percentages and a small number of case-level primary or secondary assignments changed under parameter variation, these changes did not alter the main ordering of diagnostic causes or the overall conclusions drawn from DMRA. These results indicate that the reported diagnostic findings are robust to reasonable variations in the weighting and smoothing constants across all three evaluated models.
| Models | Primary Range (%) | Secondary Range (%) |
|---|---|---|
| Claude 4.6 | 93.41–99.40 | 89.82–99.40 |
| GPT-5.4 | 95.12–100.00 | 92.68–100.00 |
| Qwen3.5-9B | 95.49–99.65 | 92.71–99.65 |
Response to Reviewer ecsZ (part 4)
Questions for future investigation: If the benchmark is expanded to additional disciplines, which construction and quality assurance stages would still require human involvement, what would the associated costs be, and are feasible automated quality assurance approaches available?
We thank the reviewer for raising this important question. When extending SciReC to additional textbooks or disciplines, most of the dataset construction pipeline can remain automated, while two stages currently require human quality control. First, extracted figures are manually filtered to remove non-scientific or unsuitable images. Second, the generated relational questions are manually validated. This step is particularly important because the model constructs relationships by connecting information across different figures and text of a textbook; therefore, human verification ensures that the identified relationship, question, and expected answer are supported by the source material.
The associated cost includes both model inference and human review. GPT is used to generate the questions and ground truths, so adding new textbooks requires additional API/inference costs proportional to the amount of processed content and generated questions. Human effort is additionally required for figure filtering and relational-question validation. Therefore, the total cost depends on the textbook size, number of extracted figures, and number of generated questions rather than being a fixed cost.
These manual stages could also be partially automated in future extensions. For example, multimodal models could pre-filter unsuitable figures and an independent model could verify whether a generated relational question and ground truth are supported by the corresponding figures and source text. Agreement between multiple independent validators could further identify uncertain cases for human review. However, we would retain human verification for ambiguous or low-confidence relational questions rather than relying entirely on automated quality assurance. Thus, a scalable extension could use automated validation as an initial filter and reserve human review for uncertain cases, substantially reducing the manual effort while maintaining quality control.
Questions for future investigation: How to generate questions that require longer-range and more complex understanding is a worthwhile direction for further investigation.
We agree that extending relational reasoning to multiple figures and cross-chapter relationships would provide an important direction for future work. However, the current setting already includes substantial long-range conversational context. The dialogue history accumulates questions throughout the chapter, including relational, memory, knowledge, and visual-understanding questions, rather than containing only the two figures used in a relational question. Thus, although individual relational questions primarily compare two figures, the model must operate within an already extended multimodal dialogue history. Nevertheless, extending the relational task itself to require integration across multiple figures, chapters, and even longer histories would provide an additional level of complexity and is an important direction for future investigation.
We believe these clarifications and experiments address all your concerns and questions. We look forward to further discussion if needed.
Response to Reviewer ecsZ (part 1)
We thank the reviewer for the insightful comments. We are pleased that you recognize SciReC’s model-adaptive diagnostic evaluation for identifying why models fail and its value in guiding MLLM improvement. Below, we provide detailed responses to your specific inquiries:
"How sensitive are the results to the judge model? Please clarify whether using open-source models as judges would affect the results."
We thank the reviewer for raising the question of sensitivity to the choice of judge model. Claude 4.6 was retained as the primary judge based on its stronger agreement with human evaluation reported in Appendix B. We additionally evaluated the same 199 responses in Appendix B using the open-source Qwen3.5-9B model, and results are provided in the table below.
Overall, all three judges produced the same pass/fail decision on 79.0% of cases. When Claude 4.6 is used as the reference, GPT-5.4 agrees with its decision in 87.5% of cases, while Qwen3.5-9B agrees in 177/200 cases (88.5%). The open-source Qwen judge exhibits overall decision consistency with Claude comparable to that of another proprietary judge, especially in the relational reasoning category.
In SciReC, the evaluation flow is affected specifically at two decision points: a relational reasoning score below 7 triggers the diagnostic evaluation, and, within this evaluation, a memory score below 7 triggers the additional memory-validation step. Threshold disagreements for other diagnostic components, such as knowledge or visual understanding, may affect the resulting diagnostic attribution but do not alter which evaluation step is executed.
For relational reasoning, Claude 4.6 and Qwen3.5-9B agree on the threshold decision for 20/24 cases (83.3%), with four cases receiving different decisions around T=7. For memory evaluation, they agree on 20/22 cases (90.9%), with two threshold disagreements; these disagreements can affect whether validation is performed only for cases that have entered the diagnostic branch.
The results indicate that replacing Claude 4.6 with the open-source Qwen3.5-9B does not substantially alter the aggregate evaluation. Differences near the threshold can affect individual branching decisions, particularly for relational reasoning; however, a small score difference around the threshold does not by itself produce a large relational attribution. DMRA considers the magnitude of the relational deficit together with the measured task deficits when determining the primary failure source. Therefore, small judge variations may alter whether further diagnosis is triggered, but substantial relational attribution requires a correspondingly larger unexplained relational deficit.
"Are there additional calibration criteria supporting the choice of T = 7? What are the advantages and disadvantages of asking a judge model to make a direct binary pass/fail judgment, compared with the current approach?"
We thank the reviewer for raising this question. The choice of T=7 is grounded directly in the semantic definition of our scoring rubric rather than selected as a numerical value; see Figures 28 and 29. Scores of 4–6 represent responses in which the main relationship is missing, incorrect, contradicted, or poorly explained, whereas scores of 7–8 require the main relationship and most key facts to be correctly captured. Thus, T=7 represents the predefined boundary between an insufficient response and one that correctly captures the core relationship.
A direct binary pass/fail judgment could simplify this decision and eliminate the need to subsequently apply a numerical threshold. However, it would discard information about the magnitude of the error. For example, responses receiving scores of 1 and 6 would both be classified as failures despite representing substantially different levels of correctness. This distinction is particularly important for DMRA because the magnitude of component deficits contributes to the diagnostic attribution. The numerical scoring provides a semantic pass/fail threshold at T=7 while capturing deficit severity.
We acknowledge that numerical scoring can be sensitive to judge calibration, particularly near the threshold. To reduce this variation, we added explicit scoring constraints defining when responses should fall above or below the threshold. We further assessed the reliability of these decisions through human–judge agreement and cross-judge consistency analyses.