- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MultiKnee, a knee MRI benchmark, and OEDChain, a hierarchical evaluation framework. The dataset contains 11,575 studies, 104,175 sequence–view units, and 821,823 VQA items. Each chain includes Observation, Evidence, and Diagnosis questions about the same anatomical target within one study. The benchmark covers six anatomical systems. The authors evaluate 20 model configurations under full observation. They compare full, sequence-restricted, and view-restricted inputs for nine configurations. Validation experiments also include MRI removal and a comparison of question-generation workflows. The results show differences in model performance across clinical levels and input conditions.
Critical strengths
1.MultiKnee contains 273,941 complete O–E–D chains. The three questions in each chain concern the same anatomical target within the same study. This structure supports stage-wise evaluation of a shared target rather than diagnosis accuracy alone. The patient-disjoint split also reduces the risk of patient overlap between training and testing.
2.The linked Observation–Evidence–Diagnosis structure produces concrete and actionable findings. The paper shows that Claude Opus 4.7 reaches 86.51% at Observation but only 42.56% at Diagnosis. The paper also shows that Qwen3.8-27B and Gemini 3.1-Pro have similar O–D gaps but fail at different stages. These findings are directly useful for model developers. The validation experiments increase confidence in the benchmark: removing MRI images reduces accuracy by 18–59 points, and the multi-stage construction pipeline achieves a 79.68% expert pass rate versus 41.52% for single-pass generation.
Critical weaknesses
1.The construction process includes 2,236 expert-reviewed questions. In Table G, rounds 21 and 22 each include 100 reviewed questions. Their expert pass rates are 85% and 78%, respectively. The authors correct and re-audit identified errors. These pass rates therefore cannot directly establish the error rate of the final test set. The expert experiment in Section 5.5 compares two generation workflows. It does not directly validate the finalized test annotations. The effect of residual annotation errors on model rankings and stage gaps remains unclear. The authors should provide an independent, stratified expert audit of the final test set. The results should include answer correctness, MRI answerability, and confidence intervals for each O/E/D level.
2.Section 5.3 reports that full input does not always outperform restricted input. For Claude Opus 4.7, the diagnosis-level full-observation advantages are −16.18 and −16.63 percentage points relative to the average single-view and single-sequence results. However, Section 5.1 does not specify the actual slice counts, image resolutions, or visual-token budgets under these conditions. The lower accuracy with full input could reflect poor integration across sequences. It could also reflect multi-image processing load or input truncation. The current experiments do not distinguish these explanations.
3.Equation (2) in Section 3.3 measures differences between aggregate stage accuracies. Section 5.2 uses these gaps to discuss failures at different reasoning stages. However, a shared anatomical target does not ensure equal question difficulty. Differences in questions and distractors could also produce these gaps. Higher Evidence accuracy and lower Diagnosis accuracy do not directly establish correct evidence interpretation followed by an incorrect diagnosis on the same chain.
Questions for authors
1.The authors should report audit sample sizes, answer correctness, and MRI answerability by O/E/D level and anatomical system.The authors should also report the main model rankings and stage gaps on the expert-verified subset.
2.The authors should specify the actual slice counts and slice-selection rules under full and restricted inputs. They should also report image resolutions, visual-token budgets, and truncation policies for each model. A control with matched image counts and resolutions is needed for representative models, particularly those with lower diagnosis accuracy under full input in Section 5.3. The authors should report whether these performance declines persist under the matched settings.
3.Table E still contains placeholders for the checkpoints and evaluation periods of all four fine-tuned models.