acceptodds
Withdrawn from ICLR 2027

MultiKnee: A Multi-Sequence Multi-View MRI Benchmark for Evidence-Chain Knee Diagnosis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical imaging benchmarks for multimodal large language models (MLLMs) predominantly rely on single-image inputs or coarse-grained labels, as fine-grained annotations aligned across sequences and views are prohibitively expensive. We introduce **MultiKnee**, a hierarchically structured multi-sequence, multi-view knee MRI benchmark with fine-grained annotations at scale: 11,575 studies, 104,175 sequence-view units, and 821,823 VQAs across six anatomical systems, 71 clinical attributes, and 34 diagnostic categories. VQAs are organized as **explicitly linked Observation-Evidence-Diagnosis chains** that trace a clinical reasoning pathway: from perceiving images across sequences and views, to interpreting them within anatomical context, and reaching the final diagnosis. Across 20 model configurations, controlled-observation analysis reveals substantial differences in how models utilize complementary sequence and view information. Stage-wise analysis further reveals that models' reasoning bottlenecks lie primarily in evidence interpretation and diagnosis rather than observation. MultiKnee thus provides a large-scale infrastructure for multi-sequence, multi-view MRI research and a framework for understanding how MLLMs reason from observations to diagnoses.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Evidence, Multi-Sequence, Multi-View, Knee MRI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yQCGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MultiKnee, a knee MRI benchmark, and OEDChain, a hierarchical evaluation framework. The dataset contains 11,575 studies, 104,175 sequence–view units, and 821,823 VQA items. Each chain includes Observation, Evidence, and Diagnosis questions about the same anatomical target within one study. The benchmark covers six anatomical systems. The authors evaluate 20 model configurations under full observation. They compare full, sequence-restricted, and view-restricted inputs for nine configurations. Validation experiments also include MRI removal and a comparison of question-generation workflows. The results show differences in model performance across clinical levels and input conditions.

Critical strengths

1.MultiKnee contains 273,941 complete O–E–D chains. The three questions in each chain concern the same anatomical target within the same study. This structure supports stage-wise evaluation of a shared target rather than diagnosis accuracy alone. The patient-disjoint split also reduces the risk of patient overlap between training and testing.

2.The linked Observation–Evidence–Diagnosis structure produces concrete and actionable findings. The paper shows that Claude Opus 4.7 reaches 86.51% at Observation but only 42.56% at Diagnosis. The paper also shows that Qwen3.8-27B and Gemini 3.1-Pro have similar O–D gaps but fail at different stages. These findings are directly useful for model developers. The validation experiments increase confidence in the benchmark: removing MRI images reduces accuracy by 18–59 points, and the multi-stage construction pipeline achieves a 79.68% expert pass rate versus 41.52% for single-pass generation.

Critical weaknesses

1.The construction process includes 2,236 expert-reviewed questions. In Table G, rounds 21 and 22 each include 100 reviewed questions. Their expert pass rates are 85% and 78%, respectively. The authors correct and re-audit identified errors. These pass rates therefore cannot directly establish the error rate of the final test set. The expert experiment in Section 5.5 compares two generation workflows. It does not directly validate the finalized test annotations. The effect of residual annotation errors on model rankings and stage gaps remains unclear. The authors should provide an independent, stratified expert audit of the final test set. The results should include answer correctness, MRI answerability, and confidence intervals for each O/E/D level.

2.Section 5.3 reports that full input does not always outperform restricted input. For Claude Opus 4.7, the diagnosis-level full-observation advantages are −16.18 and −16.63 percentage points relative to the average single-view and single-sequence results. However, Section 5.1 does not specify the actual slice counts, image resolutions, or visual-token budgets under these conditions. The lower accuracy with full input could reflect poor integration across sequences. It could also reflect multi-image processing load or input truncation. The current experiments do not distinguish these explanations.

3.Equation (2) in Section 3.3 measures differences between aggregate stage accuracies. Section 5.2 uses these gaps to discuss failures at different reasoning stages. However, a shared anatomical target does not ensure equal question difficulty. Differences in questions and distractors could also produce these gaps. Higher Evidence accuracy and lower Diagnosis accuracy do not directly establish correct evidence interpretation followed by an incorrect diagnosis on the same chain.

Questions for authors

1.The authors should report audit sample sizes, answer correctness, and MRI answerability by O/E/D level and anatomical system.The authors should also report the main model rankings and stage gaps on the expert-verified subset.

2.The authors should specify the actual slice counts and slice-selection rules under full and restricted inputs. They should also report image resolutions, visual-token budgets, and truncation policies for each model. A control with matched image counts and resolutions is needed for representative models, particularly those with lower diagnosis accuracy under full input in Section 5.3. The authors should report whether these performance declines persist under the matched settings.

3.Table E still contains placeholders for the checkpoints and evaluation periods of all four fine-tuned models.

Reviewer_AhHbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This study builds an image benchmark dataset, called MultiKnee, a hierarchically structured multi-sequence, multi-view knee MRI benchmark with fine-grained annotations at scale: 11,575 studies, 104,175 sequence-view units, and 821,823 VQAs across six anatomical systems, 71 clinical attributes, and 34 diagnostic categories. Its main contribution is the scale of knee images, although the accessibility of those images remains unclear. The innovations and contributions are insufficiently described.

Critical strengths

A dataset of 11,575 studies, 104,175 sequence-view units, and 821,823 VQAs across six anatomical systems, 71 clinical attributes, and 34 diagnostic categories.

Critical weaknesses

The innovations and contributions of this work have been greatly exaggerated. It did not cite a prior work (MedThinkVQA, ICLR 2026), which studies and interprets multiple images and integrates evidence and reasoning. While MedThinkVQA covers a much broader semantic types of medical images, MULTIKNEE only involves images of knees. Together with 3DReasonKnee and MRNet, the authors may need to readdress the contributions of their work.

Questions for authors

Although this work has created a large cohort of knee images, it remains unclear the quality of the annotations, clinical coverage, accessibility, and the research questions the dataset can address.

Reviewer_zHuvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MultiKnee, a large-scale VQA dataset for multi-sequence, multi-view knee MRI, comprising 11,575 examinations, 273,941 Observation–Evidence–Diagnosis (OED) chains, and 821,823 questions. The authors analyze model performance and information dependence using level-wise accuracy under full-input, sequence-restricted, and view-restricted conditions. The study addresses a practically important problem: diagnostic accuracy alone cannot adequately characterize a model’s medical imaging capabilities. The dataset and evaluation suite offer potential value. However, the current experiments do not sufficiently support the interpretations regarding “localizing reasoning bottlenecks” and “using complementary imaging information.” Stronger evidence is also needed regarding the quality of the final dataset and its reproducibility.

Critical strengths

1.The task has a clear clinical focus. Compared with general medical VQA, this work focuses on knee MRI and covers multiple anatomical systems, sequences, and views, facilitating the study of model performance with complex imaging inputs.

2.The level-wise evaluation provides useful descriptive insights. Organizing O/E/D questions around the same anatomical target can reveal task-specific differences obscured by overall diagnostic accuracy. Models exhibit different performance patterns across levels and input restrictions, warranting further investigation.

3.The dataset construction pipeline incorporates quality control. The authors combine report parsing, image localization, cross-model review, error feedback, and physician spot checks, and provide image-removal experiments and comparisons of construction pipelines. These measures provide a useful foundation for validation.

Critical weaknesses

1.Appendix B.1 explicitly states that O, E, and D questions are evaluated independently, without providing the model with answers to preceding questions. Consequently, the scores at the three levels reflect marginal performance on related tasks and cannot directly establish reasoning failures at the O→E or E→D transitions. For example, higher E accuracy than D accuracy does not imply that the corresponding proportion of samples involves correct evidence assessment followed by an incorrect diagnosis, because cases with incorrect E answers but correct D answers may also occur.

2.Table G shows that, even in later rounds, samples that passed automated review achieved only 78%–87% acceptance in physician spot checks. This indicates that non-negligible issues remain after automated review. Although the authors state that identified errors are fed back for correction, the manuscript does not provide an independent human audit conducted after corrections were completed and the test set was frozen. Since the dataset is the paper’s core contribution, the authors should report the quality of the final test set in terms of question-level correctness, answerability from the MRI inputs, and chain-level consistency, together with inter-expert agreement and confidence intervals. Spot checks conducted during individual construction rounds are insufficient to establish whether the final benchmark supports fine-grained model comparisons.

3.The paper emphasizes fine-grained alignment across input units, but the appendix primarily describes selecting a relevant slice and generating bounding boxes and candidate masks. This procedure does not sufficiently explain how correspondence for the same lesion is established and validated across the nine input units. The authors should clarify whether “aligned” refers to semantic associations involving the same case and anatomical target, or to validated spatial or lesion-level correspondence. If it primarily refers to the former, the contribution claims should be qualified accordingly.

4.Appendix C.2 states that questions are trained independently during fine-tuning, with only the option letter as the target and without passing preceding answers or imposing an explicit chain-consistency objective. Thus, the performance gains support the effectiveness of within-task supervision, but do not establish that the gains arise from the OED chain structure itself.

Questions for authors

1.Given that the three levels are answered independently, how do the authors distinguish differences in question difficulty from actual reasoning failures between levels? Could the authors add experiments that intervene on the evidence provided at preceding levels?

2.What are the correctness rate and chain-level consistency rate of the final frozen test set, as assessed through an independent, blinded expert review?

3.How many slices does each input unit actually contain, and what image resolution and visual input budget are used? What proportion of questions remain answerable under the restricted-input conditions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.