acceptodds
Under review as a conference paper at ICLR 2027

CodeLancet: Internalizing Physician Skills for Diagnosis via Agentic Coding

Abstract

Vision–language models (VLMs) now approach specialists at recognizing what a medical image shows. Yet many diagnoses must be examined rather than recognized. On these, their correctness is often hollow. On ejection fraction, which must be measured by tracing the left ventricle through the cardiac cycle, the most accurate frontier VLMs miss 74–89% of failing hearts and score well only because most hearts are normal. What they lack is less medical knowledge than a means of acting on the image, so they fall back on the population prior. Can a VLM learn to examine the way a physician does? Our key insight is that executable code turns examination into a sequence of executed and inspectable acts on the image. Unlike a fixed toolset, code lets the model check and correct its own evidence, compose operations no designer anticipated, adapt its effort to the case and make perception itself an act, choosing where and how finely to look. We present CodeLancet, a 4B agent that examines medical images through executable code. Skill-guided, rubric-audited demonstrations and outcome-based GRPO first teach it to conduct the examination. Yet a trajectory-level outcome reward credits every step of a correct examination alike, and over a third of these contain critical wrong steps. Focal-PPO instead credits each decision by the change in expected outcome it causes, estimated by branching rollouts around the decisions where failed examinations first went wrong, and the second stage cuts this share to a tenth. Across six benchmarks in fundus photography, echocardiography and gigapixel pathology, CodeLancet outperforms six frontier VLMs on every headline metric: at comparable overall accuracy it detects 72.5% of failing hearts, it nearly halves referral errors in retinopathy, and on whole-slide melanoma images it issues about a quarter fewer false relapse alerts, with a margin that widens as the answer depends more on examination than on recognition. Blinded specialists endorse 80–86% of its steps. Our results suggest that diagnosis is learned not only by seeing better but also by acting on what is seen, and that code can be to a model what the lancet is to a physician.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Medical Agent, Diagnosis, AI for Healthcare

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XQ1XOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains a 4B vision-language model to analyze medical images by writing and executing code. Training starts with teacher demonstrations that pass a process audit. It then adds error localization, trajectory repair, and Focal-PPO. The evaluation covers diabetic retinopathy grading, ejection fraction assessment, and melanoma relapse prediction. Tool comparisons, training ablations, and blinded physician reviews assess task performance and the model’s ability to handle faulty tool outputs.

Critical strengths

(1) The evaluation spans three medical imaging tasks. Frontier model baselines also receive the same tools and skill instructions. The gains on retinopathy grading and ejection fraction assessment are convincing. Access to more tools alone does not explain these gains. This gives the work practical value. (2) The ablations are fairly thorough. Comparisons of training starting points, removal of the segmentation tool, and physician reviews provide complementary evidence for the system. Figure 9 is particularly useful. It shows that focusing training around successfully repaired error locations helps.

Critical weaknesses

(1) The methodological novelty is not yet clear. Code-based actions and rollout-based credit estimation both have clear precedents. The main contribution appears to be the combination of error localization, repair, and targeted training. Figure 9 supports the choice of starting points. But there is no GRPO comparison using the same repair roots and execution budget. The added value of the local credit estimator therefore remains unclear. This weakens the case for a distinct methodological contribution. (2) The improvement in process quality lacks an independent comparison across training stages. The same model judge is used to filter demonstrations, locate errors, and assess final process defects. These steps may share the same judgment biases. The physician review supports the quality of the final trajectories, but it does not compare checkpoints. It therefore does not independently establish the reported drop in process defects. I remain cautious about the claim that the model has internalized physician skills.

Questions for authors

(1) How much of the gap between GRPO and Focal-PPO remains when both use the same repair roots and execution budget? This would help isolate the contribution of local credit estimation. (2) Could physicians blindly review trajectories from different checkpoints on the same cases? This would test whether process defects actually decrease across training stages. (3) Please specify the data splits and the input fields available to each model. In particular, which VisioMel clinical variables were provided? How was absolute ventricular volume computed for the ventricular dilation task?

1 comment

Thank you for the additional experiments, which address several of my earlier concerns. However, I still find the novelty limited: the method mainly combines existing code-agent and credit-estimation approaches with error localization and repair. I also share Reviewer v3gf’s concern about whether the benefits justify the system’s complexity, despite the promising OOD results.

I will maintain my score, while remaining open to further adjustments based on the authors' subsequent responses and discussions with other reviewers.

Reviewer_v3gfOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose an agentic framework for automatically diagnosing medical imaging. They utilize an LLM-based vision-language model and provide tool access via a programmable action space. Imaging inputs can be preprocessed via this programmable action space and then be fed into the model iteratively. They finetune their vision-language model via supervised fine-tuning and reinforcement learning, introducing capabilities to examine (preprocess) the image and predict diagnostic decision based on it. They compare their resulting agent with frontier LLMs on three types of medical modalities: retinal imaging, echocardiography, and whole-slide pathology imaging.

Critical strengths

1. Thorough Analysis and Ablation Studies

The authors thoroughly analyze their agent, identifying which parts of their framework enables the strongest gains. They further study the impact of specific tools and how to use them (e.g. the role of their programmable action space), as well of the impact of specific training stages.

Overall, these analyses provide interesting insights that may generalize beyond the studied applications.

2. Multiple Imaging Modalities

The authors study their agentic framework on three different imaging modalities. These three studies modalities can also be considered underexplored within vision-language models, e.g. due to data availability (echocardiography) or due to image size (whole-slide pathology imaging), especially when comparing these to thoroughly explored modalities such as chest X-rays or chest CTs. Also, these modalities have diverse properties: e.g., echocardiographies are videos, whole-slide pathology imaging is extremely high resolution).

Overall, the selected modalities provide a diverse evaluation benchmark beyond the commonly studied modalities.

Critical weaknesses

1. Missing Baselines and Low Performance

The authors only compared their approach against “frontier LLM” baselines, while not considering foundation models or specialist models for the considered benchmark tasks / modalities. To set their results in context and for a fair comparison, the authors should also evaluate specialist and/or modality-specific foundation models for each of their tasks. For example baseline models could include the following foundation models:

  • For diabetic retinopathy grading on retinal images: RETFound [1]
  • For EF prediction on echocardiographies: EchoPrime [2]
  • For five-year melanoma relapse on whole-slide pathology imaging: Prov-GigaPath [3]

Furthermore, the authors should discuss why their performance is often notably worse than smaller specialist models. For example:

  • Diabetic retinopathy (multiple datasets): deep-learning-based automatic grading models often perform >0.9 AUC and similar accuracies (see for example [1]). While a direct comparison is not possible (different dataset / grading schemes), these models seem to perform much better than the proposed model (which achieves only 0.81 accuracy)
  • EF prediction (EchoNet-Dynamic): the reference model of the original dataset publication achieves an R2 of 0.81 (MAE 4.1%). Compare this to the proposed model, which achives only an R2 of 0.38 (MAE 7.3%).
  • Five-year melanoma relapse (VisioMel whole-slide image): the winners of the challenge report an AUC of around 0.81 and an F1 score can be estimated from their confusion matrix as about 0.54 (third place: 0.56). Compare this to the proposed model, which achieves an F1 of 0.53 while not reporting the AUC

Overall, the authors need to set their work in context with common foundation and/or specialist models of their benchmark tasks. Without such comparisons and discussions, their results are not presented fairly and cannot be set into context.

[1] Zhou et al. “A foundation model for generalizable disease detection from retinal images”. Nature 622, 156–163 (2023)

[2] Vukadinovic *et al. “*Comprehensive echocardiogram evaluation with view primed vision language AI”. Nature 650, 970–977 (2026)

[3] Xu *et al. “*A whole-slide foundation model for digital pathology from real-world data”. Nature 630, 181–188 (2024)

[4] Asiri et al. “Deep learning based computer-aided diagnosis systems for diabetic retinopathy: A survey”, Artificial Intelligence in Medicine (2019).

2. Motivation and Complexity: Why agentic tool use?

What is the benefit of introducing agentic tool use if simpler and computationally cheaper models perform already quite good and much better than the proposed method (see Weakness 1). The authors hypothesize that “diagnoses must be examined rather than recognized”, but the success of simpler specialist models invalidates this hypothesis.

Even if the goal is the integration of different medical modalities into an LLM-based vision-language model, why use this agentic approach? An alternative would be to focus more on replacing the general-purpose vision encoder of the VLM by more specialized vision models based on medical foundation models for the relevant modalities. This would be much simpler (compared to the complex proposed training approach) and expected to perform better (see Weakness 1).

Overall, the authors propose an extremely complex approach, with expert-designed examination skills, multi-stage fine-tuning, and reinforcement learning with several complex rewards. This approach is computationally much more expensive than specialist models while often performing much worse. However, the authors do not clearly highlight the benefit of their approach compared to specialist models or even simple improvements to vision-language LLMs, such as replacement or improved pretraining of the vision encoder.

Questions for authors
  • why learn specialized procedure (crop, zoom, …) if this could be automatically done via specialized segmentation/location models and the resulting cropped/preprocessed images could be directly feed
  • Why define the skills manually instead of relying on the knowledge already present in the models
1 comment

Thank you for your answers.

(1) Thanks for adding the baselines, please also include them in the main paper. I agree that your framework can provide other benefits besides accuracy, so I don't think that it necessarily has to outperform the specialist baselines. However, if the performance is far worse than specialist methods, then I don't think there is a value in your framework. While for DR and Relapse your framework's performance is close to specialist methods, it completely fails on EF. Therefore, I cannot really see that your framework provides any value to EF prediction, especially in a clinical setting, it simply does not work on that task.

(2) You still did not answer what your core motivation is. Considering auditability, a simple hardcoded pipeline for EF estimation (segmentation followed by simple rules) can be easily audited and likely performs better than you method. Most importantly, for clinical practice the reliability of tools is important. If you audit a tool during a clinical trial, then you want it to perform similarly during later application. However, this may not be given for agent-based systems as their behavior is hard to predict. While you can later look at their decision paths, this would require humans to check every single decision path during application, which invalidates the value of those tools. Considering that, hardcoded workflows seems to be more easily to validate while still being auditable.

Furthermore, you are stating that VLMs fail because they try to recognize instead of examining evidence. However, specialist models seems to perform well (better than your method), while only using "recognition" instead of "examination". This seems to invalidate your hypothesis.

Keeping these aspects in mind, I still do not see the motivation of your work. Agentic systems are useful due to their flexibility. However, I do not think that this kind of flexibility is really needed for the tasks you study.

Reviewer_24zGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CodeLancet, which translates physicians' image-examination workflows into executable code. Through staged training, the model learns to select observation regions, perform measurements, verify evidence, and repair errors. The central contribution is to make consequential decisions within the examination process explicit targets for learning. Results across fundus imaging, echocardiography, and pathology, together with training and tool-interface comparisons, provide convincing support for this approach.

Critical strengths
  1. The paper studies whether medical VLMs can execute examinations and verify evidence rather than merely produce plausible explanations. This is a meaningful direction for medical tasks that require localized inspection and quantitative measurement.
  2. Fixed-tool comparisons use the same backbone and matched training conditions. Ablations of training stages, additional GRPO, and root placement help distinguish the effects of the interface, local repair, and decision-level training. Independent physician assessment further supports the quality of the final trajectories.
Critical weaknesses
  1. Independent validation of the magnitude of process improvement remains limited. Process-defect reductions are primarily assessed by the same judge used for training selection and repair verification. The physician evaluation supports final trajectory quality but does not directly assess changes across training stages (Tables 3 and 7). Please distinguish judge-assessed improvements from independently validated findings and report any available agreement with physician assessments.
  2. Patient-level data separation for VisioMel needs clarification. The manuscript does not clearly establish whether training, validation, and test sets are separated by patient (Appendices B and D). This matters for interpreting generalization to unseen patients. Please specify the splitting procedure; the reporting gap does not itself demonstrate data leakage.
Questions for authors
  1. Does the existing physician evaluation provide evidence of process improvements across training stages? Are agreement results available between physicians and the automated judge on process-defect identification?
  2. Are the VisioMel training, validation, and test sets patient-disjoint? Can different images from the same patient appear in different partitions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.