- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents the OphthoFlow, an intelligent agent workflow for ophthalmology clinical decision-making, and the OphthoBench, a multi-level evaluation benchmark, aiming to bridge the gap between the lack of clinical reasoning ability in visual base models and the tendency of visual hallucinations in large language models (LLMs).
Critical strengths
OphthoBench breaks the limitation of previous medical agents relying solely on subjective scoring of small samples. By unifying prompt words, parsers and scoring logic, it rigorously compares blind testing LLM and end-to-end VLM.
Critical weaknesses
-
The gate compliance performance is fragile and reveals serious flaws in the multi-step dynamic clinical reasoning. Among the 11 condition tasks at Level 3 (i.e., determining whether to invoke subsequent grading or prognosis tools based on the positive or negative results of the preceding screening tools), the agent performed the worst. Even when excluding errors in the classification of the underlying tools and only evaluating the logical consistency of the agent itself (the "Gating-output-based compliance" metric), the best overall performer, GPT-5.3 backbone, still forcibly invoked the excluded downstream tools in 32.1% of the queries that should have been terminated (the violation rate of Qwen3.5-122B was even as high as 54.8%).
-
Innovation in the core algorithm is limited, and the system's safety boundaries and modal routing are highly dependent on pre-hardcoded rules rather than autonomous reasoning by the agent.OphthoFlow's workflow is essentially a wrapper around a standard two-node LangGraph state machine.The "99%–100% rejection rate for out-of-range images" and "same-modal tool safe binding" that the paper highlights as core highlights are actually completely statically completed by the deterministic pre-script in the prepare node without calling any LLM.
-
Level 3 workflow evaluation only evaluates the calling trajectory and ignores the final clinical conclusion, and lacks quantitative verification of multi-tool conflict resolution.In the introduction, the paper criticizes existing models for their inability to "integrate cross-modal evidence or reconcile conflicting findings." However, in Level 2, each question in OphthoBench is only bound to a single designated tool, while in Level 3, which involves a combination of multiple tools, the scorer does not read the agent's final text answer at all, and only mechanically compares the coincidence between the tool call log and the preset trajectory.
-
The dependency of the backbone model is extremely strong, and open source and lightweight models have experienced a cliff like decline in formatted output and tool coverage. Although the paper claims that "any LLM can serve as the backbone", experimental data shows that the system performance is highly tied to the top commercial closed source model GPT-5.3.
-
The quality of the toolset (OphthoTools) varies greatly, with some tools having rough designs or lacking substantial clinical validation value. Among the 61 tools, 46 only fine tune simple two-layer perceptrons on frozen RETFound encoders, which results in poor performance of some tools. For example, the AUROC of 12 systemic disease prognostic tools is only between 0.587 and 0.705 (AUPRC as low as 0.004 to 0.073), and even the authors admit that their predictive ability is very weak.
-
The blind arrangement of pure text cuts off the visual context, causing the system to lose its ability to perceive and correct visual features outside the tool vocabulary. The core design principle of OphthoFlow is to ensure that the arranger never touches the image pixels and relies entirely on the preset JSON fields returned by the tool for inference.
-
There is a potential cyclic bias in the normalization and retrieval evaluation of answers based on the same source big model, and the benchmark dataset has limitations in templating and demographic confounding.
Questions for authors
-
In Level 3 conditional tasks, GPT-5.3 still violated controlled downstream tools in 32.1% of cases, even though the pre tools had clearly returned negative results. Can the author further analyze whether this phenomenon is due to the execution inertia caused by the mandatory generation of a global planning message at the beginning of the round?
-
What is the final diagnostic accuracy given by the intelligent agent when it calls multiple tools as required (such as simultaneously calling detectors, classifiers, and multi disease panels) and the outputs of each tool conflict?
-
The method limits the intelligent agent to only see textual tool output and cannot touch image pixels. When faced with uncontrolled datasets, if visual language models (such as GPT-5.3 or Qwen3.5-122B) are allowed to directly observe the original image while calling OphthoTools, will their performance be further improved compared to the current "pure blind text orchestration", or will it decrease due to visual prior interference?
4 comments
Response to Reviewer wF5T (2/2)
Q7. Regarding text-only orchestration
We clarify that this is our core design choice, for three main reasons: (i) Privacy. No medical image is sent to a commercial LLM, then our system can work with commercial LLM with high reasoning capability; (ii) Hallucination mitigation. Visual findings come from validated tools, not from a VLM's impression; (iii) Flexibility. A new capability needs a new tool. A VLM would need retraining.
Our results support it:
| GPT-5.3 | Accuracy | Macro-F1 |
|---|---|---|
| Sees the image, no tools | 52.8 | 49.1 |
| Never sees the image, uses tools | 61.7 | 59.5 |
VLMs are also less safe. They abstain on only 61–74% of out-of-scope images (agents: 99–100%).
For features outside the tool vocabulary, the remedy is a new tool. The clinician also are able to sees the image and overlays beside the trace.
Q8. Regarding evaluation bias in normalization and retrieval evaluation
We clarify that most free-text answers are evaluated using deterministic rules or agreement among three open-weight judges (GLM-4.6V, Llama-4-Scout, and Qwen3.5-122B). GPT-5.3 is involved only in normalizing 13.2% of free-text answers. The agent consistently outperforms the best VLM by 8.4 macro-F1 points across all questions and 6.8 points when excluding free-text questions, indicating that its advantage is not driven by evaluation bias.
Similarly, the retrieval ablation uses an independent grader, Claude Opus 4.8 (Anthropic), rather than the agent’s underlying model, GPT-5.3 (OpenAI). The grader follows a checklist with which an ophthalmologist agreed on 98.0% of reviewed statements, further mitigating concerns about evaluation bias.
Q9. Regarding conflicting in tool outputs
We clarify that each tool returns a confidence score, so the agent can reason about which result to trust in case of conflicting in tool outputs. To support it, we give two cases studies below:
Case A. The DR screen and the DR grader disagree; the agent runs lesion segmentation, finds no lesions and records no DR.
| Question (level 3) | "Assess this fundus for diabetic retinopathy and report the severity grade you would record. In addition, screen it for other retinal disease." |
| Conflict | DR screen: no DR (0.76). DR grader: mild NPDR (0.57). |
| What the agent did | Called DR lesion segmentation: no hemorrhages, exudates, cotton-wool spots or drusen. Multi-disease panel: normal. |
| Answer (excerpt) | "Given the absence of segmented DR lesions and the relatively modest confidence of the severity model, I would record this image as: No apparent diabetic retinopathy (ICDR: no_DR)." |
| Reference | No DR (ICDR grade 0). |
Case B. The same question and the same kind of disagreement; this time the lesions support the grader, and the agent records moderate NPDR.
| Question (level 3) | Same template as case A. |
| Conflict | DR screen: no DR (0.82). DR grader: moderate NPDR (0.51). |
| What the agent did | Called DR lesion segmentation: a few hemorrhages, hard exudates and cotton-wool spots. Multi-disease panel: no confident alternative diagnosis. |
| Answer (excerpt) | "Recorded severity grade: Moderate nonproliferative diabetic retinopathy. The detected hemorrhages, exudates, and cotton-wool spots are compatible with moderate NPDR under ICDR criteria." It also names the disagreement and suggests OCT for macular edema. |
| Reference | Moderate NPDR (ICDR grade 2). |
Thank you for the rebuttal and additional experiments. However, I believe several fundamental concerns remain insufficiently addressed.
-
The authors argue that gate violations only introduce redundant tool calls without affecting accuracy. However, the reported correctness decreases from 99.0% to 95.5% when extra calls occur.
-
The claim of being the first comprehensive ophthalmic agent system requires stronger comparisons with existing medical agent frameworks. Furthermore, using learned modality and quality classifiers does not resolve the concern about the largely predefined orchestration and safety mechanisms.
-
The reported improvements support the usefulness of specialized visual tools, but they do not demonstrate that restricting the LLM to textual tool outputs is generally preferable to alternative multimodal agent designs. Information omitted or incorrectly represented by upstream tools cannot necessarily be recovered through subsequent reasoning.
-
The shared-encoder design is computationally attractive, but the ability to incorporate new tools through lightweight heads does not demonstrate robust transferability to new diseases, modalities, or clinical settings. The relatively weak systemic disease prognosis results also raise concerns about the consistency of tool quality across clinical functions.
Therefore, while I appreciate the authors' responses and additional efforts, I will maintain my original rating.
Follow-up Response to Reviewer wF5T
Thank you for the follow-up. We respond to each point below.
1. Gate violations: 99.0% vs. 95.5%
These two numbers come from two different sets of queries, so they cannot show that the extra call lowers accuracy. The key number is the recall of required tools: 100.0% with the extra call. No required step is skipped.
A fair test compares on the same queries. We did this in two ways.
Comparison 1: a backbone that follows the gate.
| Agent | Gate violations | VQA accuracy | Tool recall | Full coverage |
|---|---|---|---|---|
| GPT-5.3 | 32.1% | 60.0 | 94.9 | 87.3 |
| Gemma-4-31B | 2.0% | 58.2 | 86.6 | 64.8 |
Gemma-4-31B follows the gate almost always, yet it is less accurate and misses far more required tools. Its Level-3 final answers are only on par (78.0 vs. 77.2).
Comparison 2: GPT-5.3 forced to follow the gate. A deterministic guard cuts its violations from 32.1% to 0.4%. Full coverage then falls from 90.2% to 83.5%, because the guard blocks the grader when the screen is wrong.
So stricter gate compliance does not make the agent better. The violation is one redundant call by an over-cautious model (Appendix C.11).
2. Comparison with existing medical agents
A comparison with existing agents is neither reasonable nor feasible.
- General medical agents (MedAgents, MDAgents, MMedAgent, MedRAX). They are text-only or built for other specialties, and have no ophthalmic tools.
- Ophthalmic agents (AgentEYE, EyeAgent). They release no runnable tools or benchmark, so they cannot be run.
- What we compare: nine VLMs (including the medical MedGemma-27B and LLaVA-Med), seven agent backbones, a tool-oracle, and rule-based routers.
To our knowledge, no ophthalmic agent is public. If you know of one, we will gladly add it.
3. "Largely predefined" orchestration and safety
Only the admission check is predefined. It decides whether an image is retinal and gradable. This is intended: a safety boundary must hold every time, so it should not depend on the LLM.
Everything after that is decided by the LLM. If the orchestration were predefined, fixed rules could reproduce it. They cannot:
- Rule-based routers fall far behind. Even with the task structure built into their rules, they trail the agent by 12.4 to 28.4 F1 points at Level 3 (Appendix C.14).
- The agent adapts. With the diagnosis tools removed, it still answers 60.3% of glaucoma questions through the cup-disc ratio.
- The agent resolves conflicts. In Cases 15 and 16, it settles the same tool disagreement in opposite directions, and both answers are correct (Appendix D).
4. Text-only orchestration vs. multimodal agents
Our framework is not limited to text. The LLM can be replaced by a VLM that receives the image together with the tool outputs. We chose the text-only design for three reasons:
- Privacy. Hospitals often cannot send medical images to commercial models. With text-only orchestration, the strongest commercial LLMs can be used while the image stays local.
- Reliability. Every tool has a measured accuracy and returns a confidence score, so the LLM and the clinician know how much to trust each finding. A VLM reading the image directly gives no such guarantee.
- Cost. No image tokens are sent.
Our results support this choice, on the same backbone:
| GPT-5.3 | Accuracy | Macro-F1 |
|---|---|---|
| Sees the image, no tools | 52.8 | 49.1 |
| Never sees the image, uses tools | 61.7 | 59.5 |
On information lost by the tools: this risk exists for any tool-using agent, multimodal or not. We reduce it with quality checks, confidence scores, and cross-checking between tools.
5. Transferability
Our system transfers to new diseases, modalities, and clinical settings by design.
- New diseases. The tools build on RETFound, a retinal foundation model made for transfer. A new disease needs only a new lightweight head.
- New modalities. Our tools already cover four. Foundation models such as VisionFM cover more and can be plugged in.
- New clinical settings. A new function needs only a new tool; the agent is unchanged. On 21 datasets never used for training, binary diagnosis stays at 75.0 (76.3 on training datasets, Appendix C.4).
6. Systemic disease prognosis
We agree this is the weakest group, and report it openly.
- It is the hardest task. These tools predict the future onset of a disease from one image. The others detect visible disease.
- The core clinical tools are strong: AUROC 0.943 (glaucoma) and 0.925 (AMD, DR); kappa 0.82 (grading); C-index 0.889 (AMD progression).
- The clinician decides. Our system supports clinicians; it does not replace them. Every tool comes with a measured accuracy and a confidence score, so clinicians can decide whether to use a systemic prediction.
We hope this resolves your concerns and kindly ask you to reconsider your rating.
Response to Reviewer wF5T (1/2)
Thank you for your review. We appreciate your recognition that our work breaks the limitation of previous medical agents relying solely on subjective scoring of small samples. Below, we address your remaining concerns:
Q1. Regarding gate compliance
We analyzed these violations in detail. In short, GPT-5.3 is over-cautious compared to Gemma-4-31B. A violation is one redundant tool call. It only costs efficiency, not the accuracy.
What a violation is. The screening tool says "no disease", and the agent still calls the grader. We compared GPT-5.3 queries with and without this extra call:
It is a backbone behavior, not a workflow flaw. With the same workflow, Gemma-4-31B violates the gate on only 2.0% of queries.
Q2. Regarding innovation of our work
We clarify that our major contributions are (i) the first ophthalmic agentic AI system covering multiple clinical functions and imaging modalities with tools, retrieval grounding, and an auditable trace; and (ii) the first reproducible benchmark that separates tool error from orchestration error. We believe that a contribution does not necessarily be an algorithm. In a high-stakes domain, a deployable system and a rigorous way to evaluate it are the missing pieces.
Q3. Regarding hard-coded safety
We clarify that deterministic safety is intentional in our design. A safety boundary must hold every time, so it should not depend on LLM discretion. Moreover, the admission step is also not a hand-written rule: it uses learned modality and quality classifiers.
Q4. Regarding end-to-end accuracy in level 3 evaluation
Based on your comment, we conduct experiment to score the final answers at Level 3. We score the agent's final free-text answer, not its tool calls. This experiment covers 2 agents including GPT-5.3 and Gemma-4-31B, and 18,209 questions.
We can see that agents achieve end-to-end accuracy around 70% and open-weight Gemma agent matches GPT-5.3 on every metric.
We also conduct experiment on a stratified sample of 2,000 questions to compare agent with VLM baselines.
We can see that both agents beat the VLM on every target and metric. The backbone is the same, so the gap comes from the workflow and its tools. In sum, the agent's final clinical conclusion at Level 3 is measured, and it is more accurate than a VLM reading the image directly. We will add these results to the revision.
Q5. Regarding LLM backbone dependence
We clarify that "Any LLM can serve as the backbone" is an architectural statement. Specifically, the LLM reads only tool outputs, never images. So, any LLM can be plugged in, open-source or commercial, with no privacy risk. We agree with you that system performance does depend on the backbone's reasoning and tool-use ability, and our paper says so (Section 5). But the system is not tied to GPT-5.3. The best open-weight model (Gemma-4-31B) is close:
Q6. Regarding tool design and quality
We clarify that the lightweight design is intentional. Tools are small heads on a shared, frozen RETFound encoder. This design allows the system stays compact as tools are added. A new tool needs only a new head, nothing else is retrained. Then, deployment cost is minimal, which matters for on-site use.
We also emphasize that most tools perform strongly: AUROC 0.943 (glaucoma), 0.925 (AMD), 0.925 (DR); kappa 0.82 (grading); C-index 0.889 (AMD progression). The only weak group is systemic disease prognosis. It predicts disease onset within the next 10 years from one image, which is a very hard task. However, we note that in practice, patients are assessed repeatedly, and predictions are more reliable over shorter windows. Following your comment, we have revised our manuscript to reflect this discussion.