acceptodds
Under review as a conference paper at ICLR 2027

OphthoFlow: An Agentic Workflow for Clinical Decision Support in Ophthalmology

Abstract

Ophthalmic imaging guides the management of the leading causes of irreversible blindness and, because the retina offers a direct, non-invasive view of the microvasculature and central nervous system, also serves as an inexpensive marker of systemic health. Yet neither of the two dominant AI paradigms fits clinical workflows well. Vision foundation models produce predictions they cannot reason about, and large language models (LLMs) reason fluently but hallucinate and cannot analyze images reliably. We present OphthoFlow, an agentic workflow for clinical decision support in ophthalmology. An LLM orchestrator, OphthoAgent, plans and calls OphthoTools, a suite of 61 validated task-specific models covering six clinical functions across four ophthalmic imaging modalities, and grounds its recommendations with OphthoRAG, a hybrid retrieval module over a curated ophthalmology knowledge base, behind a clinician-facing interface, OphthoChat, that exposes every step as an auditable trace. The orchestrator is text-based, so any LLM can serve as its backbone and raw images never leave the local deployment. To evaluate the workflow, we build OphthoBench, a benchmark spanning 57 datasets that separately measures clinical tool accuracy, end-to-end clinical visual question answering (VQA) over 25,000 items, and clinical workflow orchestration over 14,000 queries from 56 clinical tasks with 24 evaluation metrics, complemented by a robustness evaluation on tool-removal scenarios and on an adversarial set of out-of-scope images and ungradable retinal images where the correct response is to abstain or to warn, and a fairness analysis across demographic groups. With a GPT-5.3 backbone, OphthoAgent answers within 1.1 percentage points of a tool-oracle model that always calls the correct tool, exceeds the best of nine end-to-end vision-language models (VLMs) by 8.4 macro-F1 points, identifies the required tool set with an F1 of 92.3, and abstains with a correct reason on 99% of out-of-scope images.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
agentic AI, LLM, VLM, medical imaging, ophthalmology

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wF5TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents the OphthoFlow, an intelligent agent workflow for ophthalmology clinical decision-making, and the OphthoBench, a multi-level evaluation benchmark, aiming to bridge the gap between the lack of clinical reasoning ability in visual base models and the tendency of visual hallucinations in large language models (LLMs).

Critical strengths

OphthoBench breaks the limitation of previous medical agents relying solely on subjective scoring of small samples. By unifying prompt words, parsers and scoring logic, it rigorously compares blind testing LLM and end-to-end VLM.

Critical weaknesses
  1. The gate compliance performance is fragile and reveals serious flaws in the multi-step dynamic clinical reasoning. Among the 11 condition tasks at Level 3 (i.e., determining whether to invoke subsequent grading or prognosis tools based on the positive or negative results of the preceding screening tools), the agent performed the worst. Even when excluding errors in the classification of the underlying tools and only evaluating the logical consistency of the agent itself (the "Gating-output-based compliance" metric), the best overall performer, GPT-5.3 backbone, still forcibly invoked the excluded downstream tools in 32.1% of the queries that should have been terminated (the violation rate of Qwen3.5-122B was even as high as 54.8%).

  2. Innovation in the core algorithm is limited, and the system's safety boundaries and modal routing are highly dependent on pre-hardcoded rules rather than autonomous reasoning by the agent.OphthoFlow's workflow is essentially a wrapper around a standard two-node LangGraph state machine.The "99%–100% rejection rate for out-of-range images" and "same-modal tool safe binding" that the paper highlights as core highlights are actually completely statically completed by the deterministic pre-script in the prepare node without calling any LLM.

  3. Level 3 workflow evaluation only evaluates the calling trajectory and ignores the final clinical conclusion, and lacks quantitative verification of multi-tool conflict resolution.In the introduction, the paper criticizes existing models for their inability to "integrate cross-modal evidence or reconcile conflicting findings." However, in Level 2, each question in OphthoBench is only bound to a single designated tool, while in Level 3, which involves a combination of multiple tools, the scorer does not read the agent's final text answer at all, and only mechanically compares the coincidence between the tool call log and the preset trajectory.

  4. The dependency of the backbone model is extremely strong, and open source and lightweight models have experienced a cliff like decline in formatted output and tool coverage. Although the paper claims that "any LLM can serve as the backbone", experimental data shows that the system performance is highly tied to the top commercial closed source model GPT-5.3.

  5. The quality of the toolset (OphthoTools) varies greatly, with some tools having rough designs or lacking substantial clinical validation value. Among the 61 tools, 46 only fine tune simple two-layer perceptrons on frozen RETFound encoders, which results in poor performance of some tools. For example, the AUROC of 12 systemic disease prognostic tools is only between 0.587 and 0.705 (AUPRC as low as 0.004 to 0.073), and even the authors admit that their predictive ability is very weak.

  6. The blind arrangement of pure text cuts off the visual context, causing the system to lose its ability to perceive and correct visual features outside the tool vocabulary. The core design principle of OphthoFlow is to ensure that the arranger never touches the image pixels and relies entirely on the preset JSON fields returned by the tool for inference.

  7. There is a potential cyclic bias in the normalization and retrieval evaluation of answers based on the same source big model, and the benchmark dataset has limitations in templating and demographic confounding.

Questions for authors
  1. In Level 3 conditional tasks, GPT-5.3 still violated controlled downstream tools in 32.1% of cases, even though the pre tools had clearly returned negative results. Can the author further analyze whether this phenomenon is due to the execution inertia caused by the mandatory generation of a global planning message at the beginning of the round?

  2. What is the final diagnostic accuracy given by the intelligent agent when it calls multiple tools as required (such as simultaneously calling detectors, classifiers, and multi disease panels) and the outputs of each tool conflict?

  3. The method limits the intelligent agent to only see textual tool output and cannot touch image pixels. When faced with uncontrolled datasets, if visual language models (such as GPT-5.3 or Qwen3.5-122B) are allowed to directly observe the original image while calling OphthoTools, will their performance be further improved compared to the current "pure blind text orchestration", or will it decrease due to visual prior interference?

4 comments
AuthorsOpenReview ↗

Response to Reviewer wF5T (1/2)

Thank you for your review. We appreciate your recognition that our work breaks the limitation of previous medical agents relying solely on subjective scoring of small samples. Below, we address your remaining concerns:

Q1. Regarding gate compliance

We analyzed these violations in detail. In short, GPT-5.3 is over-cautious compared to Gemma-4-31B. A violation is one redundant tool call. It only costs efficiency, not the accuracy.

What a violation is. The screening tool says "no disease", and the agent still calls the grader. We compared GPT-5.3 queries with and without this extra call:

With extra call (684)Without (1,569)
Recall of required tools100.099.5
Clear answers that are correct95.599.0
  • No required tool is skipped. The extra call is added on top of the required tools, never in place of one.
  • The conclusion stays right. After an extra call, 95.5% of clear answers are still correct.
  • The only cost is efficiency: one redundant tool call.

It is a backbone behavior, not a workflow flaw. With the same workflow, Gemma-4-31B violates the gate on only 2.0% of queries.

Q2. Regarding innovation of our work

We clarify that our major contributions are (i) the first ophthalmic agentic AI system covering multiple clinical functions and imaging modalities with tools, retrieval grounding, and an auditable trace; and (ii) the first reproducible benchmark that separates tool error from orchestration error. We believe that a contribution does not necessarily be an algorithm. In a high-stakes domain, a deployable system and a rigorous way to evaluate it are the missing pieces.

Q3. Regarding hard-coded safety

We clarify that deterministic safety is intentional in our design. A safety boundary must hold every time, so it should not depend on LLM discretion. Moreover, the admission step is also not a hand-written rule: it uses learned modality and quality classifiers.

Q4. Regarding end-to-end accuracy in level 3 evaluation

Based on your comment, we conduct experiment to score the final answers at Level 3. We score the agent's final free-text answer, not its tool calls. This experiment covers 2 agents including GPT-5.3 and Gemma-4-31B, and 18,209 questions.

AgentOverall acc.Overall macro-F1
GPT-5.368.764.3
Gemma-4-31B70.164.9

We can see that agents achieve end-to-end accuracy around 70% and open-weight Gemma agent matches GPT-5.3 on every metric.

We also conduct experiment on a stratified sample of 2,000 questions to compare agent with VLM baselines.

SystemOverall acc.Overall macro-F1
Agent GPT-5.370.064.7
Agent Gemma-4-31B71.165.4
VLM GPT-5.366.450.6
Agent GPT-5.3 minus VLM [95% CI]+3.6 [+1.7, +5.4]+14.2 [+9.9, +18.3]

We can see that both agents beat the VLM on every target and metric. The backbone is the same, so the gap comes from the workflow and its tools. In sum, the agent's final clinical conclusion at Level 3 is measured, and it is more accurate than a VLM reading the image directly. We will add these results to the revision.

Q5. Regarding LLM backbone dependence

We clarify that "Any LLM can serve as the backbone" is an architectural statement. Specifically, the LLM reads only tool outputs, never images. So, any LLM can be plugged in, open-source or commercial, with no privacy risk. We agree with you that system performance does depend on the backbone's reasoning and tool-use ability, and our paper says so (Section 5). But the system is not tied to GPT-5.3. The best open-weight model (Gemma-4-31B) is close:

MetricGPT-5.3Gemma-4-31B
VQA accuracy60.058.2
VQA macro-F158.256.8
Tool selection98.798.0
Level-3 tool-set F192.389.6

Q6. Regarding tool design and quality

We clarify that the lightweight design is intentional. Tools are small heads on a shared, frozen RETFound encoder. This design allows the system stays compact as tools are added. A new tool needs only a new head, nothing else is retrained. Then, deployment cost is minimal, which matters for on-site use.

We also emphasize that most tools perform strongly: AUROC 0.943 (glaucoma), 0.925 (AMD), 0.925 (DR); kappa 0.82 (grading); C-index 0.889 (AMD progression). The only weak group is systemic disease prognosis. It predicts disease onset within the next 10 years from one image, which is a very hard task. However, we note that in practice, patients are assessed repeatedly, and predictions are more reliable over shorter windows. Following your comment, we have revised our manuscript to reflect this discussion.

AuthorsOpenReview ↗

Response to Reviewer wF5T (2/2)

Q7. Regarding text-only orchestration

We clarify that this is our core design choice, for three main reasons: (i) Privacy. No medical image is sent to a commercial LLM, then our system can work with commercial LLM with high reasoning capability; (ii) Hallucination mitigation. Visual findings come from validated tools, not from a VLM's impression; (iii) Flexibility. A new capability needs a new tool. A VLM would need retraining.

Our results support it:

GPT-5.3AccuracyMacro-F1
Sees the image, no tools52.849.1
Never sees the image, uses tools61.759.5

VLMs are also less safe. They abstain on only 61–74% of out-of-scope images (agents: 99–100%).

For features outside the tool vocabulary, the remedy is a new tool. The clinician also are able to sees the image and overlays beside the trace.

Q8. Regarding evaluation bias in normalization and retrieval evaluation

We clarify that most free-text answers are evaluated using deterministic rules or agreement among three open-weight judges (GLM-4.6V, Llama-4-Scout, and Qwen3.5-122B). GPT-5.3 is involved only in normalizing 13.2% of free-text answers. The agent consistently outperforms the best VLM by 8.4 macro-F1 points across all questions and 6.8 points when excluding free-text questions, indicating that its advantage is not driven by evaluation bias.

Similarly, the retrieval ablation uses an independent grader, Claude Opus 4.8 (Anthropic), rather than the agent’s underlying model, GPT-5.3 (OpenAI). The grader follows a checklist with which an ophthalmologist agreed on 98.0% of reviewed statements, further mitigating concerns about evaluation bias.

Q9. Regarding conflicting in tool outputs

We clarify that each tool returns a confidence score, so the agent can reason about which result to trust in case of conflicting in tool outputs. To support it, we give two cases studies below:

Case A. The DR screen and the DR grader disagree; the agent runs lesion segmentation, finds no lesions and records no DR.

Question (level 3)"Assess this fundus for diabetic retinopathy and report the severity grade you would record. In addition, screen it for other retinal disease."
ConflictDR screen: no DR (0.76). DR grader: mild NPDR (0.57).
What the agent didCalled DR lesion segmentation: no hemorrhages, exudates, cotton-wool spots or drusen. Multi-disease panel: normal.
Answer (excerpt)"Given the absence of segmented DR lesions and the relatively modest confidence of the severity model, I would record this image as: No apparent diabetic retinopathy (ICDR: no_DR)."
ReferenceNo DR (ICDR grade 0).

Case B. The same question and the same kind of disagreement; this time the lesions support the grader, and the agent records moderate NPDR.

Question (level 3)Same template as case A.
ConflictDR screen: no DR (0.82). DR grader: moderate NPDR (0.51).
What the agent didCalled DR lesion segmentation: a few hemorrhages, hard exudates and cotton-wool spots. Multi-disease panel: no confident alternative diagnosis.
Answer (excerpt)"Recorded severity grade: Moderate nonproliferative diabetic retinopathy. The detected hemorrhages, exudates, and cotton-wool spots are compatible with moderate NPDR under ICDR criteria." It also names the disagreement and suggests OCT for macular edema.
ReferenceModerate NPDR (ICDR grade 2).

Thank you for the rebuttal and additional experiments. However, I believe several fundamental concerns remain insufficiently addressed.

  1. The authors argue that gate violations only introduce redundant tool calls without affecting accuracy. However, the reported correctness decreases from 99.0% to 95.5% when extra calls occur.

  2. The claim of being the first comprehensive ophthalmic agent system requires stronger comparisons with existing medical agent frameworks. Furthermore, using learned modality and quality classifiers does not resolve the concern about the largely predefined orchestration and safety mechanisms.

  3. The reported improvements support the usefulness of specialized visual tools, but they do not demonstrate that restricting the LLM to textual tool outputs is generally preferable to alternative multimodal agent designs. Information omitted or incorrectly represented by upstream tools cannot necessarily be recovered through subsequent reasoning.

  4. The shared-encoder design is computationally attractive, but the ability to incorporate new tools through lightweight heads does not demonstrate robust transferability to new diseases, modalities, or clinical settings. The relatively weak systemic disease prognosis results also raise concerns about the consistency of tool quality across clinical functions.

Therefore, while I appreciate the authors' responses and additional efforts, I will maintain my original rating.

AuthorsOpenReview ↗

Follow-up Response to Reviewer wF5T

Thank you for the follow-up. We respond to each point below.

1. Gate violations: 99.0% vs. 95.5%

These two numbers come from two different sets of queries, so they cannot show that the extra call lowers accuracy. The key number is the recall of required tools: 100.0% with the extra call. No required step is skipped.

A fair test compares on the same queries. We did this in two ways.

Comparison 1: a backbone that follows the gate.

AgentGate violationsVQA accuracyTool recallFull coverage
GPT-5.332.1%60.094.987.3
Gemma-4-31B2.0%58.286.664.8

Gemma-4-31B follows the gate almost always, yet it is less accurate and misses far more required tools. Its Level-3 final answers are only on par (78.0 vs. 77.2).

Comparison 2: GPT-5.3 forced to follow the gate. A deterministic guard cuts its violations from 32.1% to 0.4%. Full coverage then falls from 90.2% to 83.5%, because the guard blocks the grader when the screen is wrong.

So stricter gate compliance does not make the agent better. The violation is one redundant call by an over-cautious model (Appendix C.11).

2. Comparison with existing medical agents

A comparison with existing agents is neither reasonable nor feasible.

  • General medical agents (MedAgents, MDAgents, MMedAgent, MedRAX). They are text-only or built for other specialties, and have no ophthalmic tools.
  • Ophthalmic agents (AgentEYE, EyeAgent). They release no runnable tools or benchmark, so they cannot be run.
  • What we compare: nine VLMs (including the medical MedGemma-27B and LLaVA-Med), seven agent backbones, a tool-oracle, and rule-based routers.

To our knowledge, no ophthalmic agent is public. If you know of one, we will gladly add it.

3. "Largely predefined" orchestration and safety

Only the admission check is predefined. It decides whether an image is retinal and gradable. This is intended: a safety boundary must hold every time, so it should not depend on the LLM.

Everything after that is decided by the LLM. If the orchestration were predefined, fixed rules could reproduce it. They cannot:

  • Rule-based routers fall far behind. Even with the task structure built into their rules, they trail the agent by 12.4 to 28.4 F1 points at Level 3 (Appendix C.14).
  • The agent adapts. With the diagnosis tools removed, it still answers 60.3% of glaucoma questions through the cup-disc ratio.
  • The agent resolves conflicts. In Cases 15 and 16, it settles the same tool disagreement in opposite directions, and both answers are correct (Appendix D).

4. Text-only orchestration vs. multimodal agents

Our framework is not limited to text. The LLM can be replaced by a VLM that receives the image together with the tool outputs. We chose the text-only design for three reasons:

  • Privacy. Hospitals often cannot send medical images to commercial models. With text-only orchestration, the strongest commercial LLMs can be used while the image stays local.
  • Reliability. Every tool has a measured accuracy and returns a confidence score, so the LLM and the clinician know how much to trust each finding. A VLM reading the image directly gives no such guarantee.
  • Cost. No image tokens are sent.

Our results support this choice, on the same backbone:

GPT-5.3AccuracyMacro-F1
Sees the image, no tools52.849.1
Never sees the image, uses tools61.759.5

On information lost by the tools: this risk exists for any tool-using agent, multimodal or not. We reduce it with quality checks, confidence scores, and cross-checking between tools.

5. Transferability

Our system transfers to new diseases, modalities, and clinical settings by design.

  • New diseases. The tools build on RETFound, a retinal foundation model made for transfer. A new disease needs only a new lightweight head.
  • New modalities. Our tools already cover four. Foundation models such as VisionFM cover more and can be plugged in.
  • New clinical settings. A new function needs only a new tool; the agent is unchanged. On 21 datasets never used for training, binary diagnosis stays at 75.0 (76.3 on training datasets, Appendix C.4).

6. Systemic disease prognosis

We agree this is the weakest group, and report it openly.

  • It is the hardest task. These tools predict the future onset of a disease from one image. The others detect visible disease.
  • The core clinical tools are strong: AUROC 0.943 (glaucoma) and 0.925 (AMD, DR); kappa 0.82 (grading); C-index 0.889 (AMD progression).
  • The clinician decides. Our system supports clinicians; it does not replace them. Every tool comes with a measured accuracy and a confidence score, so clinicians can decide whether to use a systemic prediction.

We hope this resolves your concerns and kindly ask you to reconsider your rating.

Reviewer_Qo1yOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

OphthoFlow combines a text-based LLM orchestrator with 61 ophthalmic image tools, hybrid retrieval from NEI and EyeWiki, and a clinician-facing interface that exposes plans, tool calls, retrieved passages and answers. The tools span admission and image quality, demographics, diagnosis and grading, prognosis, segmentation and generation across four retinal imaging modalities. A key design choice is that the LLM sees image artifact identifiers and structured tool outputs rather than raw image pixels. The accompanying OphthoBench separates individual tool performance, end-to-end VQA and tool-call trajectories. It draws on 57 datasets, 25,000 VQA items and 14,000 workflow queries from 56 tasks, plus 2,000 out-of-scope or ungradable images.

With GPT-5.3 as orchestrator, the system reports 58.2 macro-F1 and 60.0 accuracy on VQA, 8.4 macro-F1 points above its strongest VLM baseline and 1.1 accuracy points below an oracle that always calls the designated tool. It selects the required tool set at F1 92.3 and covers every required tool on 87.3% of workflow queries. The experiments also expose weak spots: systemic-disease tools have AUROC 0.587–0.705, the agent uses about 16,969 tokens per VQA item, and GPT-5.3 calls a gated tool on 32.1% of queries even when the gating tool's own output excludes it. Out-of-scope abstention is strong, while warning rates on ungradable images range more widely across backbones.

Critical strengths

The three-level evaluation is the most valuable contribution. A designated-tool oracle gives a reference for how much VQA error comes from the tools themselves, while trajectory metrics reveal behavior that a correct final label would miss. The contrast between 92.3 tool-set F1 and 32.1% gating-output-based violation for GPT-5.3 is a good example: the agent is usually able to choose the right tools, yet still makes a consequential decision error after a gate. The admission checks and auditable trace make the system more inspectable than a direct VLM answer.

The breadth and the negative results add credibility. The paper evaluates several orchestrator backbones, compares with nine VLMs, removes tool groups, checks unsupported and ungradable images, and reports group differences rather than only average accuracy. The tool-removal drops of up to 26.1 points make it plausible that specialist models provide real signal beyond the language model. On 1,500 out-of-scope images, agents abstain 99–100% of the time; on 500 ungradable retinal images, warning rates range from 63% to 92%, indicating both a useful safeguard and remaining inconsistency. The methods appendix also gives fixed split and checkpoint-selection rules, using patient-level splits where identifiers exist, then eye-level or image-level splits where they do not.

Critical weaknesses
  • The strongest benchmark numbers do not test much of the clinical reasoning promised by the title. Each VQA item asks a templated question whose gold answer is an existing dataset label, and each has one designated tool. The 56 workflow tasks are defined with two ophthalmologists, but many are fixed chains or combinations of independent questions. This design gives reproducible scoring and tests orchestration, yet it says less about ambiguous cases, conflicting images, uncertain patient history or management choices. The paper includes a retrieval ablation for recommendation quality, but that part relies on an LLM-created reference and LLM grading rather than a prospective clinician assessment. I would interpret the results as evidence for a research workflow, not clinical decision support that has been validated in practice.

  • Error propagation and evaluation sensitivity are harder to assess. Some source datasets lack patient identifiers, so their stated split falls back to eye or image level; the paper does not clearly establish cross-dataset patient deduplication. This matters when 57 collections feed a common benchmark. The answer-normalization pipeline uses multiple LLM calls to map free text to labels, with the ground truth withheld, which reduces a simple leakage risk but does not establish that ambiguous outputs are judged consistently across agents and VLMs. Similarly, the abstention score depends partly on model-based classification of the reason given. The 8.4-point macro-F1 gain is meaningful, but its stability to independent human adjudication is not clear. Finally, the 16,969 versus 4,618 token gap matters for the practicality of the system, and the weak systemic prognosis tools require careful wording when their outputs feed management suggestions.

Questions for authors

For datasets without patient identifiers, what checks were performed to detect repeated eyes or patients across train and test, and to detect overlap between source datasets? The appendix describes patient/eye/image splitting within datasets, but cross-dataset overlap could still affect the level-1 and level-2 results. An estimate of the number of datasets and items covered by each split granularity would help assess this risk.

Was a stratified sample of answer normalizations, abstention reasons and retrieval recommendations adjudicated independently by ophthalmologists or trained human raters? Agreement by model and question type, especially on inferred rather than directly normalized labels, would show whether the headline VQA advantage is robust to the scoring method. For safety, can the authors explain why GPT-5.3 still calls a gated tool on 32.1% of queries where the gating tool says not to, and whether a deterministic guard can prevent that call without reducing answer accuracy?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Qo1y (1/2)

Thank you for your review and for recognizing the three-level evaluation as the most valuable contribution. Below, we address your remaining concerns:

Q1. Regarding clinical reasoning and validation in practice

A1. We agree that OphthoFlow is a research prototype, not a clinically validated product, and our paper already said so (Ethics Statement). Our goal is to provide a benchmark that is large, reproducible, and auditable. Prospective clinician assessment cannot be run at this scale (25,000 VQA items, 14,000 workflow queries).

Following your comments, to support the rigor of our evaluation, we added three things:

  • Human review of the automatic scoring. Ophthalmologists checked the answer normalizer and the retrieval evaluation. Agreement is 92.7–99.5% (details below).
  • Final-answer scoring at Level 3. We now score the agent's final conclusion, not only its tool calls. The GPT-5.3 agent reaches 68.7% accuracy on 11,287 queries. With the same backbone, it beats the VLM by +3.6 accuracy and +14.2 macro-F1 points (both 95% CIs exclude zero). Two judges from different model families give the same score on 99.4% of judgements. A human reviewer agreed with the judges on 100 of 100 sampled cases.
  • Conflicting evidence. We added two traced cases where the DR screen and the DR grader disagree. The agent calls lesion segmentation as a third opinion. In these cases, it resolves the conflict in opposite directions, and both match the reference.

Q2. Regarding retrieval ablation: ophthalmologist review

A2. To validate the correctness of our retrieval ablation, an ophthalmologist reviewed both LLM-based parts:

  • The reference checklist. 149 statements were reviewed, including every statement on which the LLM validators disagreed. The ophthalmologist agreed on 98.0% (146 of 149).
  • The grading. 20 cases (40 sessions) were reviewed without knowing which version used retrieval. The ophthalmologist agreed with the LLM grader on 92.7% of labels.

Q3. Regarding dataset splits and overlap

A3. We clarify that there is no leakage between training and test. Specifically, the table below gives the split unit for every dataset behind the tools.

Split unitMade byDatasetsL1 test imagesL2 itemsL3 items
PatientDataset creators411,6664,7222,331
PatientUs391,7997,8394,508
As publishedDataset creators930,1956,3604,329
EyeUs221,160286540
Test only, no tool trained on itn/a214,0965,7932,292
  • With patient identifiers: we split by patient and verified that training and test patients are disjoint.
  • Without patient identifiers: we use the split published by the dataset creators to avoid data leakage.
  • One exception, AIROGS. It has no identifiers and no published split, so we split it ourselves. Every AIROGS image comes from a different eye (de Vente et al., arXiv:2302.01738), so no eye appears on both sides.
  • 21 datasets are test-only. No tool was trained on them.

Q4. Regarding answer normalization and the 8.4-point gain

Most of the benchmark needs no LLM. 72.1% of items are closed-form and are scored directly against the label. Only free-text answers (27.9%) pass through the normalizer, and 44.5% of those are resolved by deterministic rules.

(Human evaluation) An ophthalmologist reviewed every LLM stage. Samples are stratified by task type and cover two agents and two VLMs.

Normalizer stageShare of free-text answersOphthalmologist agreement
Strict match25.9%deterministic
Negation rule18.6%deterministic
Three open-weight judges agree42.2%99.5% (200 sampled)
GPT-5.3, agrees with at least two judges8.6%98.0% (200 sampled)
GPT-5.3 alone4.6%93.3% (464 of 497, all reviewed)

The last row is the "inferred" case you ask about. We reviewed all of it. About 73% of the disagreements are cases where the normalizer was too cautious and returned no label.

The headline gain is stable under human adjudication. Applying the ophthalmologist's corrections changes agent accuracy by 0.04 points, and the agent-minus-VLM gap by at most 0.02 points.

Agents and VLMs are judged consistently. The normalizer never sees the gold label or which system wrote the answer. The agent's lead also does not come from the LLM-judged stage:

StageAgent GPT-5.3 minus best VLM (accuracy)
Strict match+15.1
Negation rule+7.4
Three judges agree-2.1

The gain holds without free-text answers. Excluding all free-text items, the agent still leads the best VLM by 6.8 macro-F1 points (58.3 vs. 51.5).

AuthorsOpenReview ↗

Response to Reviewer Qo1y (2/2)

Q5. Regarding abstention scoring

  • The abstention itself is not an LLM judgement. It is triggered by the admission step, which uses learned modality and quality classifiers. This is why agents abstain on 99–100% of out-of-scope images.
  • The reason classifier is constrained. It picks from a closed label set and must quote the reply as evidence. It does not see the image, the image family, or which system answered. Correctness then follows a fixed rule per image family (Appendix B, Tables 18–19).
  • The gap is too large to be a scoring artifact. Agents abstain on 99–100% of out-of-scope images, and VLMs on 61–74%.

Q6. Regarding token cost

We agree that the agent uses more tokens compared to VLM baselines because it calls tools and reads their outputs. However, this is what buys the accuracy, and we emphasize that the cost in dollars is small (see table below).

SystemAccuracyTokens per item$ per item
Agent GPT-5.361.117,5020.0203
VLM GPT-5.352.21,0830.0030
Agent Gemma-4-31B59.420,2860.0041
  • About two cents per question. The GPT-5.3 agent gains 8.9 accuracy points over its own VLM.
  • Caching helps the agent. 56% of its prompt tokens are cached, since the system prompt and tool catalog repeat at every step. This cuts its cost by 43%.
  • An open-weight agent is cheap. The Gemma agent is 7.2 points more accurate than the GPT-5.3 VLM at a similar price.

Q7. Regarding gate compliance and a deterministic guard

We analyzed these violations in detail. In short, GPT-5.3 is over-cautious compared to Gemma-4-31B. A violation is one redundant tool call. It only costs efficiency, not the accuracy.

What a violation is. The screening tool says "no disease", and the agent still calls the grader. We compared GPT-5.3 queries with and without this extra call:

With extra call (684)Without (1,569)
Recall of required tools100.099.5
Clear answers that are correct95.599.0
  • No required tool is skipped. The extra call is added on top of the required tools, never in place of one.
  • The conclusion stays right. After an extra call, 95.5% of clear answers are still correct.
  • The only cost is efficiency: one redundant tool call.

It is a backbone behavior, not a workflow flaw. With the same workflow, Gemma-4-31B violates the gate on only 2.0% of queries.

On a deterministic guard. We tested one. It refuses a gated call whenever the screening output rules it out, and it cuts violations from 32.1% to 0.4%. But it does more harm than the violation it removes:

  • It trusts the screen even when the screen is wrong. It then blocks the grader on eyes that do have the disease. Full coverage of required tools falls from 90.2% to 83.5%.
  • It cannot read the question. "Does this image show DR, and what grade applies?" asks for the grade in every case. On such queries, the guard would refuse a required tool on 36.7%.

So the guard replaces a harmless extra call with a harmful missed one. We therefore do not enable it, and we report this analysis in the revision.

Q8. Regarding ethics concerns

We take these concerns seriously. We believe they are addressed by the paper and by the new results above, so no further ethics review is needed.

  • No new human data. All 57 datasets are existing and de-identified. Controlled-access images never leave the approved environment, and the LLM never receives image pixels.
  • Presentation of recommendations. The paper states that OphthoFlow is a research prototype, not a medical device, and that it supports, not replaces, a qualified clinician (Ethics Statement). Every answer comes with an auditable trace of the tools and sources behind it.
  • Sensitive predictors. Age, sex, and ethnicity are clinically relevant risk factors, and their accuracy is reported in full at Level 1. Our fairness analysis covers 15,819 items and marks confounded comparisons instead of interpreting them (Appendix C.13). We do not report race gaps, because the data cannot support an unconfounded comparison, and the paper says so.
  • Weak tool outputs. The systemic tools are labeled as the hardest target throughout the paper. Their outputs are risk signals for future events, not diagnoses, and the agent reports them as such.
  • Safeguards. Admission checks stop out-of-scope images (agents abstain on 99–100%). A gate violation is a redundant call, not a skipped step (see Q7).
Reviewer_1ugqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This manuscript presents OphthoFlow, an agentic workflow for ophthalmic clinical decision support. It integrates a text-based LLM orchestrator, 61 specialized visual tools, evidence retrieval, and an auditable interactive interface. Together these cover tool invocation, knowledge retrieval, user interaction, and system evaluation, reflecting a fairly thorough engineering design. The main contribution of this work is OphthoBench, a three-tier evaluation framework that assesses tool accuracy, end-to-end answer correctness, and clinical workflow orchestration.

Critical strengths
  • The research question is well targeted, meaningful, and timely.

  • OphthoFlow is fairly complete. It covers tool invocation, RAG, an interactive interface, and system evaluation, and the engineering work is solid.

  • The main contribution is the benchmark. OphthoBench evaluates specialty tool performance, final answer correctness, and clinical workflow execution quality at three levels. In particular, its evaluation of tool invocation order and conditional dependencies can reveal system failure modes that conventional end-to-end accuracy metrics cannot capture.

  • The experimental results support the proposed system design. They show that specialized tool orchestration outperforms direct VLM answering. The experiments also show how different LLMs perform as backbone models.

Critical weaknesses
  • The routing ablations show that selective tool use helps. But on their own, they do not tell us how much of this gain comes from LLM orchestration itself. A simple rule-based or lightweight classifier router would be a useful baseline. Comparing it with LLM orchestration would show whether LLM scheduling is necessary and where it actually matters, especially for structured tasks with predefined tool dependencies.

  • OphthoBench evaluates workflow execution against predefined tool-call trajectories. The design is reasonable and reproducible. The issue is that predefined paths may not cover other clinically valid workflows, and they do not capture tool calls made for confirmation. This can affect how gate compliance is interpreted. The authors should explain how the reference workflows were established clinically and how such cases are handled.

Questions for authors
  • It would help to add a comparison between OphthoFlow and lightweight rule-based or classifier-based routing on Level-2 and Level-3 structured tasks. The current results show a gain from selective tool use, but they do not yet isolate the value of LLM orchestration. Such a comparison would clarify what LLM orchestration adds beyond choosing among predefined tools.
  • The difference in conditional gate compliance across LLM backbones needs explanation. GPT-5.3 has better tool coverage but clearly lower gate compliance than Gemma-4-31B. When a workflow trajectory deviates from the predefined rules, how should we interpret it? Is it ineffective tool use, a misunderstanding of the conditional rules, or actually a clinically reasonable extra verification step?
  • The clinical validation of the predefined workflow paths should be described. In particular, when multiple diagnostic sequences are possible or additional confirmatory tool calls are needed, the authors should explain how these paths were validated as clinically reasonable.
1 comment
AuthorsOpenReview ↗

Response to Reviewer 1ugq

Thank you for your review. We appreciate your recognition that the research question is well targeted, meaningful, and timely, that the system is complete and solid, and that the experiments support the design. Below, we address your remaining concerns:

Q1. Regarding comparison with rule-based orchestration

We ran this comparison at both levels 2 and 3. Rule-based routers fall behind the LLM, and the gap is largest on multi-step tasks.

Baselines. All routers see exactly what the LLM sees: the question, its options, the image modality, and the tool catalog.

  • TF-IDF classifier. A logistic regression trained on 2,500 questions written from the tool catalog alone.
  • Catalog matcher. No training. Rules from the system prompt filter the candidate tools. The rest are ranked by similarity to the question, with BM25 or PubMedBERT.

Level 2 evaluation results

RouterAccuracyMacro-F1Accuracy vs. agent [95% CI]
Tool-oracle60.658.4+1.1 [+0.8, +1.4]
Agent GPT-5.359.558.1ref
Catalog matcher, PubMedBERT55.953.1-3.6 [-4.0, -3.2]
TF-IDF classifier51.049.9-8.5 [-9.0, -8.1]
Catalog matcher, BM2549.946.7-9.6 [-10.1, -9.1]
  • The agent is 1.1 points from the tool-oracle. LLM routing costs almost nothing.
  • The routers lose 3.6 to 9.6 points. They pick the designated tool on 79.4–87.8% of items. The agent does so on 98.7%.

Level 3 evaluation results. We gave the rule-based routers an advantage here. Their rules split each query into clauses, place the screening tool before its grader, and skip a conditional grader after a negative screen. So the benchmark's structure is built into them.

RouterRecallF1Full coverageF1 vs. agent [95% CI]
Agent GPT-5.394.992.387.3ref
TF-IDF classifier84.779.959.6-12.4 [-12.7, -12.1]
Catalog matcher, BM2582.677.756.7-14.6 [-14.9, -14.3]
Catalog matcher, PubMedBERT66.563.933.2-28.4 [-28.8, -28.1]
  • The agent is best on every metric. It runs the complete reference workflow on 87.3% of queries. The best rule-based router does so on 59.6%.
  • Even with the query structure given for free, the routers trail by 12.4 to 28.4 F1 points.

Reworded questions. We rewrote 10% of the Level-2 questions into harder wording with the same meaning. The agent hardly moves (-0.4 accuracy points) and still calls the designated tool on 98.5% of items. The BM25 matcher drops from 78.8% to 72.1%. So the agent routes by reading the question, not by matching the benchmark's wording.

Where LLM orchestration matters. For level 2 evaluation with one question and one tool, a good matcher comes within a few points. Once a query has several parts as in level 3 evaluation, choosing and ordering tools from free text is where the LLM is needed. We will add these results to the revision.

Q2. Regarding the design of clinical workflows in level 3 evaluation

Clinical validation. Two board-certified ophthalmologists helped design and review all 56 tasks. Every task was finalized only with 100% agreement between them, before any query was generated (Appendix B.7). The clinical chains follow published diagnosis-to-grading pathways for diabetic retinopathy, AMD, and glaucoma.

One reference workflow per task. Each task has one reference: the minimal set of tools that answers the query. Other workflows can also reach a correct answer, but they do so by adding calls that the query does not need. We treat those calls as redundant, not as wrong.

How alternative workflows are scored. The metrics separate the two cases, so an extra call is never confused with a missing one:

  • Recall and full coverage check only the required tools. Extra calls do not lower them.
  • Precision and unnecessary tool use measure redundancy separately.
  • Order is enforced only where the clinical pathway requires it.

Q3. Regarding gate compliance

We analyzed these gate compliance violations in detail. In short, GPT-5.3 is over-cautious compared to Gemma-4-31B. A violation is one redundant tool call. It only costs efficiency, not the accuracy.

What a violation is. The screening tool says "no disease", and the agent still calls the grader. We compared GPT-5.3 queries with and without this extra call:

With extra call (684)Without (1,569)
Recall of required tools100.099.5
Clear answers that are correct95.599.0
  • No required tool is skipped. The extra call is added on top of the required tools, never in place of one.
  • The conclusion stays right. After an extra call, 95.5% of clear answers are still correct.

It is a backbone behavior, not a workflow flaw. With the same workflow, Gemma-4-31B violates the gate on only 2.0% of queries.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their careful and constructive feedback. We have uploaded a revised manuscript. All changes are marked in red. Below, we summarize what we added during the rebuttal and where to find it.

1. Final-answer accuracy at Level 3 (Reviewers wF5T, Qo1y)

Level 3 previously scored only the tool calls. We now also score the agent's final answer against ground-truth labels.

  • The agent answers 77–78% of diagnosis targets correctly, and grades severity with a QWK of 0.75.
  • With the same GPT-5.3 backbone, the agent beats the VLM by +3.6 accuracy and +14.2 macro-F1 points.
  • Two LLM judges from different families give the same score on 99.4% of judgements, and a human check agreed on 100 of 100 samples.

Where: Section 5.3; Appendix B.9 (protocol); Appendix C.10, Tables 52–53 (results).

2. Rule-based routing baselines (Reviewer 1ugq)

We compare LLM orchestration with a TF-IDF classifier and a catalog matcher (BM25, PubMedBERT).

  • Level 2: the routers trail the agent by 3.6 to 9.6 accuracy points.
  • Level 3: they trail by 12.4 to 28.4 F1 points, even with the task structure built into their rules.
  • On reworded questions, the agent's accuracy changes by only 0.4 points.

Where: Sections 5.2 and 5.3; Appendix B.12 (setup); Appendix C.14, Tables 60–62 (results).

3. Gate compliance analysis (all reviewers)

We analyzed why GPT-5.3 calls a gated tool on 32.1% of queries, and what it costs.

  • Cost. A violation is one redundant call. No required tool is skipped, and 95.5% of clear answers are still correct.
  • Cause. The rate depends on question wording: 0.5% with an explicit condition, 59.7% with a loose one. Gemma-4-31B stays at 2.0% overall.
  • Planning. The plan step reduces violations; without it, they rise from 32.1% to 41.9%.
  • Guard. A deterministic guard cuts violations to 0.4%, but it also blocks required tools, so we report it as an analysis only.

Where: Section 5.3; Appendix C.11, Tables 54–56.

4. Ophthalmologist review of the automatic scoring (Reviewers Qo1y, wF5T)

  • Answer normalizer. Agreement is 93.3–99.5% at every LLM stage. The corrections change agent accuracy by 0.04 points. Without free-text items, the agent still leads the best VLM by 6.8 macro-F1 points.
  • Retrieval ablation. Agreement is 98.0% on the checklist and 92.7% on the grading.

Where: Appendix C.6, Table 46 (normalizer); Appendix C.18 (retrieval ablation).

5. Dataset splits (Reviewer Qo1y)

We added a table with the split unit of every dataset used to train or test the tools, and the number of items from each group. Patient-level splits were verified to be disjoint.

Where: Appendix B, "Splits" paragraph and Table 4.

6. Dollar cost (Reviewer Qo1y)

The GPT-5.3 agent costs about two cents per question (\0.0203),andgains8.9accuracypointsoveritsownVLM.Promptcachinglowersitscostby430.0203), and gains 8.9 accuracy points over its own VLM. Prompt caching lowers its cost by 43%. The open-weight Gemma-4-31B agent costs \\0.0041.

Where: Appendix B.14 (pricing); Appendix C.17, Table 66 (results).

7. Conflicting tool outputs (Reviewers wF5T, Qo1y)

We added two traced cases where the DR detection tool and the severity grader disagree. The agent calls lesion segmentation to settle the conflict. It resolves the two cases in opposite directions, and both match the ground truth.

Where: Appendix D.16 and D.17 (Cases 15 and 16).

8. Clarifications in the text (Reviewers wF5T, Qo1y)

  • Backbone claim. We reworded "any LLM can serve as the backbone". The text now says the backbone can be swapped without changing any tool, and that orchestration quality depends on the backbone. Where: Abstract; Sections 3, 5, and 6.
  • Systemic prognosis tools. We state that these tools predict future disease onset from a single image, and that their outputs are risk signals, not diagnoses. Where: Section 5.1.

We believe these additions address the main concerns raised in the reviews. We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.