- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper trains a 4B vision-language model to analyze medical images by writing and executing code. Training starts with teacher demonstrations that pass a process audit. It then adds error localization, trajectory repair, and Focal-PPO. The evaluation covers diabetic retinopathy grading, ejection fraction assessment, and melanoma relapse prediction. Tool comparisons, training ablations, and blinded physician reviews assess task performance and the model’s ability to handle faulty tool outputs.
Critical strengths
(1) The evaluation spans three medical imaging tasks. Frontier model baselines also receive the same tools and skill instructions. The gains on retinopathy grading and ejection fraction assessment are convincing. Access to more tools alone does not explain these gains. This gives the work practical value. (2) The ablations are fairly thorough. Comparisons of training starting points, removal of the segmentation tool, and physician reviews provide complementary evidence for the system. Figure 9 is particularly useful. It shows that focusing training around successfully repaired error locations helps.
Critical weaknesses
(1) The methodological novelty is not yet clear. Code-based actions and rollout-based credit estimation both have clear precedents. The main contribution appears to be the combination of error localization, repair, and targeted training. Figure 9 supports the choice of starting points. But there is no GRPO comparison using the same repair roots and execution budget. The added value of the local credit estimator therefore remains unclear. This weakens the case for a distinct methodological contribution. (2) The improvement in process quality lacks an independent comparison across training stages. The same model judge is used to filter demonstrations, locate errors, and assess final process defects. These steps may share the same judgment biases. The physician review supports the quality of the final trajectories, but it does not compare checkpoints. It therefore does not independently establish the reported drop in process defects. I remain cautious about the claim that the model has internalized physician skills.
Questions for authors
(1) How much of the gap between GRPO and Focal-PPO remains when both use the same repair roots and execution budget? This would help isolate the contribution of local credit estimation. (2) Could physicians blindly review trajectories from different checkpoints on the same cases? This would test whether process defects actually decrease across training stages. (3) Please specify the data splits and the input fields available to each model. In particular, which VisioMel clinical variables were provided? How was absolute ventricular volume computed for the ventricular dilation task?
Thank you for the additional experiments, which address several of my earlier concerns. However, I still find the novelty limited: the method mainly combines existing code-agent and credit-estimation approaches with error localization and repair. I also share Reviewer v3gf’s concern about whether the benefits justify the system’s complexity, despite the promising OOD results.
I will maintain my score, while remaining open to further adjustments based on the authors' subsequent responses and discussions with other reviewers.