- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces OCT-Bench to evaluate OCT understanding in multimodal language models. It contains 10,076 multiple-choice questions built from 4,137 images across seven public datasets, covering 20 tasks grouped into Perception, Cognition, and Reasoning. Questions are generated with GPT-4o and screened through model-based checks and independent review by three ophthalmologists. The authors evaluate 20 models and report substantial differences across tasks, with clinical reasoning generally receiving lower scores than basic perception. GPT-5.4-mini obtains the highest reported overall score of 62.01%.
Critical strengths
-
The paper looks beyond disease classification and includes tasks on anatomy, lesion features, and clinical decisions. This broader scope is a useful direction for OCT evaluation.
-
Having three ophthalmologists independently review the candidate questions represents a substantial annotation effort. Their checks of medical correctness, visual answerability, and ambiguity are a valuable part of the benchmark construction.
-
The evaluation includes 20 general-purpose and medical models, with scores reported separately for all 20 tasks. This level of detail makes the results easier to examine.
Critical weaknesses
1. The task counts and reported accuracies do not line up.
Sec. 4.1 describes ordinary MCQ accuracy, with invalid responses counted as incorrect. However, some results in Table 3 cannot be obtained from the task sizes in Figure 8A. For example, T17 has 468 questions, but GPT-5.4-mini scores 38.42%. The neighboring possible scores are 179/468 = 38.25% and 180/468 = 38.46%. Similar discrepancies occur for T12 (703 questions, 35.01%) and T15 (865 questions, 34.66%). These differences cannot be explained by rounding alone.
2. The benchmark does not establish that images are needed to answer some tasks.
T01’s generation prompt requires the correct answer to always be “OCT.” Selecting the OCT option therefore solves the task without looking at the image, regardless of option order. Its high scores cannot by themselves demonstrate visual modality recognition.
The concern is less direct, but more consequential, for T19 and T20. T19 uses a fixed disease–treatment mapping, while T20 generates disease-specific follow-up scenarios with prescribed management answers. The clinical history and treatment response in the question may already reveal enough to choose an answer. No text-only or mismatched-image control is reported, so the visual contribution remains unclear. T19 also mixes diagnostic accuracy with treatment selection, while T20 tests synthetic follow-up decisions rather than prognosis against real longitudinal outcomes.
3. Separate task scores do not show where the clinical reasoning process breaks down.
The paper claims that its hierarchy can distinguish weak visual grounding from failures in higher-level clinical inference, and later discusses error propagation. But the experiments mainly compare accuracy across separate task types. A model scoring well on Perception and poorly on Reasoning has not necessarily extracted the relevant evidence correctly before failing at the next step. The tasks may differ in image content, difficulty, and source dataset.
The error analysis has a related gap. The evaluation requires only an option letter, yet Sec. 4.5 assigns errors to knowledge deficits and reasoning breakdowns, and Figure 7 compares extended explanations. Please explain how those rationales were collected and how the error categories were assigned. A wrong option alone does not identify its cause, and a separately prompted explanation need not reflect the process behind the original answer.
4. The training-data analysis does not establish whether the evaluated models have seen the benchmark images.
Appendix A.4 identifies OCT exposure, but OCT-Bench draws from seven public datasets, so dataset- or image-level overlap remains relevant. Generating new questions does not remove possible familiarity with the underlying scans. Please distinguish known overlap from unknown exposure and, where possible, report results on sources confirmed to be held out. The source–task imbalance in Figure 8E also needs attention, since differences across capability levels may partly reflect different image sources rather than the intended capability alone.
- The task taxonomy needs stronger justification.
Sec. 3.2 motivates the taxonomy through the clinical OCT workflow, but does not explain how the 20 tasks were selected or whether experts reviewed their coverage and placement. Why, for example, is disease classification assigned to Reasoning rather than Cognition? The three levels are described conceptually, but there is no validation that the tasks distinguish these abilities rather than simply differ in content or difficulty. This matters because the paper uses the grouping to draw conclusions about perception versus reasoning. [Secs. 3.1–3.2]
- Sec. 3.2 lists medical correctness, visual answerability, task relevance, and ambiguity as review criteria, but how were these applied in practice? Did reviewers use a checklist or scoring rubric, and what counted as a pass? Were they shown the proposed answer, source label, or generation context, or asked to answer independently first? Please also clarify how disagreements were resolved and whether revised questions went through another independent review. For staging, treatment, and follow-up tasks, how did reviewers decide that the image and question provided enough information for one uniquely correct answer?
Questions for authors
Please see the weaknesses above.
Summary of Our Rebuttal to Reviewer 2oef
We sincerely thank you for your careful review and constructive feedback. Your comments have helped us improve the rigor and clarity of our work. In response to your concerns, we have conducted additional experiments and provided detailed clarifications, summarized as follows:
Visual dependency: We evaluated seven models across all tasks under original-image, no-image, and mismatched-image conditions. Average accuracy dropped from 57.47% to 35.51% and 41.98%, respectively, demonstrating the importance of correctly matched OCT images.
Error analysis: We clarified the supplementary error analysis protocol, including model-generated explanations and targeted follow-up questions, and explicitly limited its interpretation to qualitative evidence rather than causal claims.
Benchmark validity: We further justified the clinical workflow-based task taxonomy and detailed the independent review process involving three ophthalmologists. Only questions unanimously approved by all three experts across all four quality criteria were retained.
Limitations: We explicitly acknowledged the uncertainty surrounding potential training-image overlap and source–task confounding, which cannot be fully ruled out.
Additionally, we corrected the numerical inconsistencies in Table 3, with all changes below 0.1% and no impact on the main conclusions.
We hope these additional experiments, corrections, and methodological clarifications adequately address your concerns and reinforce the reliability of our benchmark. We would sincerely appreciate your reconsideration of our paper and a potentially more favorable rating in light of these improvements. Thank you again for your rigorous review and valuable feedback.