acceptodds
Withdrawn from ICLR 2027

Can Multimodal Large Language Models Understand OCT?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Optical coherence tomography (OCT) is essential for the diagnosis and treatment of retinal diseases. Although multimodal large language models (MLLMs) have shown strong potential in medical image analysis, existing benchmarks mainly focus on coarse-grained disease classification or visual question answering, failing to comprehensively evaluate the cognitive process from visual perception to clinical reasoning. To address this gap, we present OCT-Bench, a comprehensive benchmark for OCT image understanding. OCT-Bench contains 10,076 high-quality multiple-choice questions derived from 4,137 OCT images across seven public datasets. Following the clinical interpretation workflow, we establish a hierarchical taxonomy of 20 fine-grained tasks spanning three capability dimensions: Perception, Cognition, and Reasoning. These tasks cover imaging attributes, retinal anatomy, lesion characteristics, spatial relationships, disease assessment, treatment decision-making, and prognosis. We evaluate 20 representative MLLMs, including close-source, open-source, and medical-domain models. Results show that current models remain far from reliable OCT understanding, while neither medical-domain adaptation nor larger model scale consistently improves performance across capability levels. OCT-Bench provides a comprehensive benchmark for evaluating OCT understanding and facilitates the development of clinically grounded MLLMs.

Primary area:
datasets and benchmarks
Keywords:
Optical Coherence Tomography, Multimodal Large Language Models, Medical Image Understanding, Medical Reasoning, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2oefOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces OCT-Bench to evaluate OCT understanding in multimodal language models. It contains 10,076 multiple-choice questions built from 4,137 images across seven public datasets, covering 20 tasks grouped into Perception, Cognition, and Reasoning. Questions are generated with GPT-4o and screened through model-based checks and independent review by three ophthalmologists. The authors evaluate 20 models and report substantial differences across tasks, with clinical reasoning generally receiving lower scores than basic perception. GPT-5.4-mini obtains the highest reported overall score of 62.01%.

Critical strengths
  1. The paper looks beyond disease classification and includes tasks on anatomy, lesion features, and clinical decisions. This broader scope is a useful direction for OCT evaluation.

  2. Having three ophthalmologists independently review the candidate questions represents a substantial annotation effort. Their checks of medical correctness, visual answerability, and ambiguity are a valuable part of the benchmark construction.

  3. The evaluation includes 20 general-purpose and medical models, with scores reported separately for all 20 tasks. This level of detail makes the results easier to examine.

Critical weaknesses

1. The task counts and reported accuracies do not line up.

Sec. 4.1 describes ordinary MCQ accuracy, with invalid responses counted as incorrect. However, some results in Table 3 cannot be obtained from the task sizes in Figure 8A. For example, T17 has 468 questions, but GPT-5.4-mini scores 38.42%. The neighboring possible scores are 179/468 = 38.25% and 180/468 = 38.46%. Similar discrepancies occur for T12 (703 questions, 35.01%) and T15 (865 questions, 34.66%). These differences cannot be explained by rounding alone.

2. The benchmark does not establish that images are needed to answer some tasks.

T01’s generation prompt requires the correct answer to always be “OCT.” Selecting the OCT option therefore solves the task without looking at the image, regardless of option order. Its high scores cannot by themselves demonstrate visual modality recognition.

The concern is less direct, but more consequential, for T19 and T20. T19 uses a fixed disease–treatment mapping, while T20 generates disease-specific follow-up scenarios with prescribed management answers. The clinical history and treatment response in the question may already reveal enough to choose an answer. No text-only or mismatched-image control is reported, so the visual contribution remains unclear. T19 also mixes diagnostic accuracy with treatment selection, while T20 tests synthetic follow-up decisions rather than prognosis against real longitudinal outcomes.

3. Separate task scores do not show where the clinical reasoning process breaks down.

The paper claims that its hierarchy can distinguish weak visual grounding from failures in higher-level clinical inference, and later discusses error propagation. But the experiments mainly compare accuracy across separate task types. A model scoring well on Perception and poorly on Reasoning has not necessarily extracted the relevant evidence correctly before failing at the next step. The tasks may differ in image content, difficulty, and source dataset.

The error analysis has a related gap. The evaluation requires only an option letter, yet Sec. 4.5 assigns errors to knowledge deficits and reasoning breakdowns, and Figure 7 compares extended explanations. Please explain how those rationales were collected and how the error categories were assigned. A wrong option alone does not identify its cause, and a separately prompted explanation need not reflect the process behind the original answer.

4. The training-data analysis does not establish whether the evaluated models have seen the benchmark images.

Appendix A.4 identifies OCT exposure, but OCT-Bench draws from seven public datasets, so dataset- or image-level overlap remains relevant. Generating new questions does not remove possible familiarity with the underlying scans. Please distinguish known overlap from unknown exposure and, where possible, report results on sources confirmed to be held out. The source–task imbalance in Figure 8E also needs attention, since differences across capability levels may partly reflect different image sources rather than the intended capability alone.

  1. The task taxonomy needs stronger justification.

Sec. 3.2 motivates the taxonomy through the clinical OCT workflow, but does not explain how the 20 tasks were selected or whether experts reviewed their coverage and placement. Why, for example, is disease classification assigned to Reasoning rather than Cognition? The three levels are described conceptually, but there is no validation that the tasks distinguish these abilities rather than simply differ in content or difficulty. This matters because the paper uses the grouping to draw conclusions about perception versus reasoning. [Secs. 3.1–3.2]

  1. Sec. 3.2 lists medical correctness, visual answerability, task relevance, and ambiguity as review criteria, but how were these applied in practice? Did reviewers use a checklist or scoring rubric, and what counted as a pass? Were they shown the proposed answer, source label, or generation context, or asked to answer independently first? Please also clarify how disagreements were resolved and whether revised questions went through another independent review. For staging, treatment, and follow-up tasks, how did reviewers decide that the image and question provided enough information for one uniquely correct answer?
Questions for authors

Please see the weaknesses above.

1 comment
AuthorsOpenReview ↗

Summary of Our Rebuttal to Reviewer 2oef

We sincerely thank you for your careful review and constructive feedback. Your comments have helped us improve the rigor and clarity of our work. In response to your concerns, we have conducted additional experiments and provided detailed clarifications, summarized as follows:

  1. Visual dependency: We evaluated seven models across all tasks under original-image, no-image, and mismatched-image conditions. Average accuracy dropped from 57.47% to 35.51% and 41.98%, respectively, demonstrating the importance of correctly matched OCT images.

  2. Error analysis: We clarified the supplementary error analysis protocol, including model-generated explanations and targeted follow-up questions, and explicitly limited its interpretation to qualitative evidence rather than causal claims.

  3. Benchmark validity: We further justified the clinical workflow-based task taxonomy and detailed the independent review process involving three ophthalmologists. Only questions unanimously approved by all three experts across all four quality criteria were retained.

  4. Limitations: We explicitly acknowledged the uncertainty surrounding potential training-image overlap and source–task confounding, which cannot be fully ruled out.

Additionally, we corrected the numerical inconsistencies in Table 3, with all changes below 0.1% and no impact on the main conclusions.

We hope these additional experiments, corrections, and methodological clarifications adequately address your concerns and reinforce the reliability of our benchmark. We would sincerely appreciate your reconsideration of our paper and a potentially more favorable rating in light of these improvements. Thank you again for your rigorous review and valuable feedback.

Reviewer_3wLFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OCT-Bench, a domain-specific benchmark designed to evaluate the capability of multimodal large language models (MLLMs) to understand OCT images. It proposes a hierarchical evaluation framework that simulates the clinical interpretation workflow and enables fine-grained assessment across three dimensions: visual perception, medical cognition, and clinical reasoning. The benchmark defines 20 distinct tasks, with questions generated by GPT-4o under task guidance and subjected to dual quality control by GPT-4o and ophthalmologists. General-purpose MLLMs and medical domain-specific MLLMs are evaluated under a unified zero-shot setting.

Critical strengths
  • Clinically meaningful question construction. Current MLLMs are insufficient in fine-grained OCT understanding, and relevant data resources are less abundant than those for fundus imaging. Although hierarchical evaluation is not entirely new, the OCT-specific emphasis on retinal anatomy, lesion morphology, spatial relationships, and clinical management provides meaningful domain specialization.

  • The benchmark construction pipeline combines task-guided generation prompt, medical knowledge references, automated filtering, and ophthalmologist review.

  • Task-level reporting is provided. Several potentially valuable anomalous findings challenge simplistic assumptions about medical domain adaptation, and these observations provide useful evidence for future OCT-oriented systems development.

Critical weaknesses
  • The novelty positioning relative to closely related recent ophthalmology benchmarks, such as X-PCR and FunBench, is insufficient. A specialized benchmark with deeper OCT-specific anatomical and pathological coverage could still constitute a meaningful contribution; however, the paper currently emphasizes the novelty of its hierarchical organization without adequately distinguishing it from these closely related works. Table 1 further weakens this argument: general benchmarks such as MMBench and MathVista were not designed to assess clinical ophthalmology capabilities. They lack OCT-specific tasks and therefore provide limited information about the unique contribution of the proposed benchmark.

  • The current hierarchical organization of tasks according to the clinical interpretation process is methodologically reasonable, but there is insufficient evidence that these tasks reliably measure separable capabilities. It is necessary to demonstrate that each question type primarily depends on its target capability and to distinguish task-level performance differences from genuine error propagation. The current experiments mainly report accuracy for each task group and lack a systematic analysis of error dependencies across capability levels for the same OCT cases.

  • The anomalous results lack mechanistic analysis. T03 mainly involves visual morphological discrimination, and the extremely low accuracy of Hulu-Med and Lingshu on T03 warrants further analysis of the specific reasons. The paper currently mainly reports the phenomenon and does not adequately explain the significant differences among medical models, nor their relatively strong performance on other tasks.

  • The benchmark is primarily designed for single-turn, image-based multiple-choice evaluation of standalone MLLMs. Ophthalmic AI research is increasingly considering tool-augmented systems and agents capable of using specialized image analysis tools, retrieving clinical knowledge, and integrating evidence from multiple sources. Its intended scope, long-term scalability, and relationship with tool-augmented clinical AI systems have not been adequately discussed.

Questions for authors
  • Compared with the six-stage clinical reasoning evaluation in X-PCR and the hierarchical, modular evaluation in FunBench, both works have already explored hierarchical or progressive evaluation for ophthalmic multimodal understanding, and OCT is included in their modality coverage. Beyond focusing on OCT and providing more fine-grained tasks, what model capability deficiencies can OCT-Bench reveal that existing benchmarks cannot identify?

  • The current experimental results are derived from different task groups and cannot directly demonstrate that errors at early stages lead to failures at later stages. For the same OCT image, is it possible to design paired questions that span different capability levels? Can the analysis determine whether errors at the perceptual or cognitive level predict subsequent errors in diagnosis or treatment?

  • Please analyze why the strong medical domain models Hulu-Med and Lingshu perform anomalously poorly on T03, especially given that Fleming-VL-8B performs much better on T03. Can the authors provide class-wise confusion matrices or prediction distributions to distinguish systematic answer bias from genuine visual recognition failure? Is there a medical semantic bias whereby domain-fine-tuned models are more inclined to use familiar medical representations rather than select morphological vocabulary?

  • This paper covers a broad range of models, but several evaluated general open-source models are relatively old compared with the submission period. Please explain the rationale for model selection to improve the representativeness of the evaluation. For example, the base model of Fleming-VL is InternVL3; including InternVL3 in the comparison would better support analysis of the differences.

1 comment
AuthorsOpenReview ↗

Summary of Our Rebuttal to Reviewer 3wLF

We sincerely thank you for your constructive feedback and recognition of our work. In response to your concerns, we have provided detailed clarifications and conducted additional analyses, summarized as follows:

  1. Benchmark novelty: We clarified that OCT-Bench follows the actual clinical OCT interpretation workflow, enabling precise localization of capability deficiencies at different stages of clinical interpretation, rather than merely reporting overall performance.

  2. Cross-level error dependencies: We conducted same-image paired analyses, revealing a strong association between diagnostic correctness and treatment-planning performance, with accuracy differences of 63.73 and 46.43 percentage points across two models.

  3. T03 anomaly analysis: We analyzed prediction distributions across all 344 questions, identifying systematic class biases in Hulu-Med and Lingshu rather than output-parsing failures.

  4. Model coverage and domain adaptation: We evaluated newer models, including InternVL3, and compared Fleming-VL with its base model. Results revealed uneven benefits from medical domain adaptation, with reasoning improving by 7.33 points but cognition decreasing by 4.03 points.

We also clarified the benchmark's scope and its potential extension to tool-augmented clinical AI systems.

We believe these additional analyses provide stronger support for OCT-Bench's contributions and hope they address your concerns. We would sincerely appreciate your reconsideration of our paper and a potentially more favorable rating. Thank you again for your valuable feedback, and we welcome any further discussion.

Reviewer_Y3hAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OCT-Bench to evaluate MLLMs on OCT image understanding. It contains 10,076 multiple-choice questions based on 4,137 images from seven public datasets. The taxonomy follows the clinical interpretation workflow. It covers Perception, Cognition, and Reasoning through nine capability groups and twenty tasks. Evaluations of twenty open-source, closed-source, and medical-domain models reveal substantial limitations, particularly in fine-grained structural recognition and clinical reasoning.

Critical strengths
  1. The paper follows the clinical workflow for OCT interpretation. It groups tasks into Perception, Cognition, and Reasoning, with nine capability groups and twenty tasks. These tasks cover visual features, medical knowledge, and clinical decisions. This makes it easier to compare model performance across tasks.

  2. The benchmark combines seven public datasets and evaluates twenty models from different categories. The authors report that three ophthalmologists reviewed the questions. The generation templates and task-level results help readers check how the benchmark was built and provide a useful reference for further research.

Critical weaknesses
  1. MIRAGE (Asadi et al., 2026) found that models could achieve substantial accuracy on several general and medical VQA benchmarks without image inputs. This highlights the need to test the role of images in visual evaluation. Some questions in OCT-Bench may provide non-visual clues, such as fixed links between annotation colors and lesion types. The paper does not report performance when images are removed or replaced, so the effect of these clues remains unclear. Since the benchmark aims to assess OCT image understanding, further evidence is needed to show how much its scores reflect the use of visual information rather than question wording or annotation patterns.

  2. Figure 6 attributes many errors to knowledge deficits and reasoning failures, and the paper uses these results to identify major bottlenecks. However, the main evaluation requests only an answer option. The manuscript does not adequately explain the evidence and criteria used to distinguish these error types. An incorrect choice alone cannot reliably establish whether the cause was a missed visual finding, missing medical knowledge, or faulty reasoning. This limits the support for the conclusions about failure mechanisms. The absence of Claude and newer Qwen vision models also limits how broadly the findings can be generalized.

Asadi M, O'Sullivan J W, Cao F, et al. Mirage: The illusion of visual understanding. arXiv preprint arXiv:2603.21687, 2026.

Questions for authors
  1. Could you select a few representative tasks and compare accuracy with the original images, no images, and mismatched images, while keeping the questions and answer options unchanged? This would help assess how much performance depends on visual information.

  2. Could you include a representative Claude model and a newer Qwen model with visual capabilities, such as Qwen3-VL or Qwen3.5? Do their results support the paper’s main conclusions?

  3. Under the option-only evaluation protocol, how were errors classified as visual failures, knowledge deficits, or reasoning errors? Please describe the evidence, labeling criteria, and review process used for Figure 6.

  4. OCT5k is based on OCT volumes. OIMHS and MMC-AMD also include eyes with multiple B-scans. Why was evaluation limited to single images, and how was the image for each question selected? During expert review, how was the sufficiency of the image and question text established for diagnosis, staging, and management questions?

  5. Could you provide a higher-resolution version of Figure 8 to improve the readability of its text and visual details?

2 comments
AuthorsOpenReview ↗

Summary of Our Rebuttal to Reviewer Y3hA

We sincerely thank you for your constructive feedback and recognition of our work. In response to your concerns, we have conducted additional experiments and provided detailed clarifications:

  1. Visual dependency: Seven models showed substantial accuracy drops without images (57.47% → 35.51%) or with mismatched images (57.47% → 41.98%), confirming the importance of visual evidence. Additional color-perturbation experiments further examined annotation bias.
  2. Newer models: We evaluated Qwen3-VL, Qwen3.5, InternVL3, and Molmo2. The results consistently confirm the substantial perception–reasoning gap, reaching 31.43 percentage points even for Qwen3.5-27B.
  3. Methodological clarifications: We clarified the qualitative error analysis protocol, the rationale for single B-scan evaluation, and the three-ophthalmologist validation process. We also committed to improving Figure 8.

We hope these additional results and clarifications adequately address your concerns and strengthen the paper. We would sincerely appreciate it if you could reconsider your assessment and update your rating in light of the new evidence.

Thank you again for your valuable time and feedback.

Thank you for the additional experiments, which address many of my original concerns. I appreciate the value of a fine-grained benchmark for OCT interpretation. However, I still have reservations about the novelty and overall contribution of the work. In particular, the distinction from existing ophthalmology benchmarks and the insights gained from the evaluation could be clearer. I maintain my original score, though I consider this a borderline paper.

Comments on the paper

1 comment

Dear Reviewers,

We’re now past the halfway point of the ICLR discussion period. Please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.