acceptodds
Withdrawn from ICLR 2027

MedCode: Medical Visual Reasoning beyond Predefined Toolkits

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical vision language models (VLM) have benefited from explicit reasoning, but most existing approaches reason over a visual representation that is encoded only once. Recent tool augmented methods allow models to revisit the image, yet their analyses remain limited to procedures defined in advance. In this paper, we present MedCode, a framework that enables medical VLMs to construct question specific image analyses through executable code. Instead of selecting from a predefined toolkit, MedCode writes programs using general image processing and numerical operations. We further develop MedCodeLab, a shared general purpose environment that executes these programs and returns textual results and derived images for subsequent reasoning. To teach this capability, we construct MedCode-50k, which contains 49,817 verified multi turn trajectories across eight imaging modalities and four question categories. We train an 8B model through trajectory based supervised fine tuning followed by agentic reinforcement learning on questions selected according to rollout difficulty. On the in distribution test set, MedCode raises the accuracy of its backbone from 75.2% to 82.8%. It also generalizes to four unseen data sources, where it improves average accuracy from 76.2% to 85.5%. MedCode consistently outperforms crop based reasoning and predefined image operations, demonstrating the value of constructing flexible analyses for medical visual reasoning.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Medical visual reasoning, Vision–language models, Code generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3APqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

MedCode trains Qwen3-VL-8B to answer medical multiple-choice questions by writing Python programs. A sandbox (MedCodeLab) executes them and returns text and images for further reasoning. MedCode-50k contains 49,817 Gemini 3.1 Pro trajectories on an OmniMedVQA subset (8 modalities, 4 question categories), filtered for answer correctness and at least one successful, nontrivial execution. Training is SFT on these trajectories, then GRPO on about 16k difficulty-selected questions with reward = accuracy + 0.5·format + 0.5·accuracy·execution. Accuracy rises from 75.2% to 82.8% in-distribution and from 76.2% to 85.5% on four held-out sources. Ablations cover program scope, training stage, reward gating and feedback modality.

Critical strengths
  1. The design choices are sound: randomized filenames prevent label leakage, execution is stateless, the prompt is given verbatim, and the correctness-gated reward is ablated (80.7 vs. 82.8). Gains are positive on every category and held-out source, which suggests more than source memorization. How much this matters depends on attributing the gains to code, which W1 shows the paper has not done.
  2. A useful resource, if released. About 50k executed multimodal trajectories could support work on medical tool use. The paper does not state any release plan.
Critical weaknesses

W1. Limited novelty, and the experiments do not isolate the effect of code. Training VLMs to generate and execute image-processing code via SFT followed by RL is established. Thyme, PyVision and DeepEyesV2 all do this, and none is cited or compared. MedCodeLab is a generic Python sandbox, and the training pipeline and reward are standard. The contribution therefore rests on the evidence, which has four gaps:

  1. No same-data, no-tool control. The baselines were never trained on OmniMedVQA, and the “10.1% over text-based reasoning” is the relative gain over the untrained backbone.
  2. Table 3a is not a fair toolkit comparison. It restricts the code-trained checkpoint at inference, so it never tests models trained on predefined toolkits.
  3. No framework+model baselines. Teacher accuracy with MedCodeLab is not reported. The acceptance rate (about 83%) is close to Gemini 3 Pro’s no-tool accuracy (82.9%).
  4. One backbone, one run, no variance.

W2. The evaluation and verification do not show that code yields valid evidence.

  1. Easy, near-distribution benchmark. OmniMedVQA is mostly templated classification, the held-out sources share modalities and templates with training, and no reasoning benchmark is used.
  2. Outcome-only verification. In the paper’s own CTR example, heart width is capped at 0.4× image width while thoracic width is maximized, which biases the ratio toward “normal.” The dermoscopy example assumes 1 mm ruler ticks. Both trajectories were accepted because the final answer matched.
  3. No causal or cost evidence. There is no test that execution outputs change answers. Turns, tokens and latency are not reported, even though the prompt mandates code and allows up to 10 turns.
Questions for authors
  1. How does a same-data, no-tool SFT+GRPO control perform?
  2. Were the Table 3a/3c conditions inference-time restrictions? How do toolset-specific models trained end to end compare, and which operations do the learned programs actually use?
  3. How many teacher samples were drawn per question? What is Gemini 3.1 Pro’s accuracy with and without MedCodeLab? How do frontier models with MedCodeLab and zero-shot Qwen3-VL-8B with MedCodeLab perform?
  4. What is the accuracy when observations are replaced with ones from another image, or emptied?
  5. What are the turns, tool calls, tokens and latency compared with the backbone and fixed-tool variants?
  6. Was the split done at question, image or patient level, and was overlap checked between held-out and training sources?
  7. What is the variance across seeds?
  8. How does MedCode perform on a reasoning benchmark outside OmniMedVQA?
Reviewer_PRrwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MedCode, a framework that enables medical vision-language models to write and execute question-specific image-analysis code rather than relying only on predefined tools. It introduces MedCodeLab, an execution environment that returns textual results and derived images, and MedCode-50k, a dataset of 49,817 filtered multi-turn trajectories. The authors train Qwen3-VL-8B through trajectory-based supervised fine-tuning followed by reinforcement learning. The resulting model achieves 82.8% accuracy on the in-distribution evaluation and 85.5% average accuracy across four held-out data sources, compared with 75.2% and 76.2%, respectively, for the original backbone.

Critical strengths

I find the execution environment and trajectory dataset valuable. MedCodeLab provides a consistent interface for running code and returning textual and visual observations during data collection, training, and inference. MedCode-50k provides approximately 50k interaction trajectories across eight medical imaging modalities, including code execution and subsequent reasoning. Together, they provide a useful basis for training and studying medical agents that interact with images rather than relying only on text-based reasoning.

Critical weaknesses
  1. The authors check code execution and final-answer correctness, but do not independently verify the medical accuracy of intermediate measurements. For example, the ruler-based size estimates and anatomical boundary estimates in Appendix B.3 are not checked against reference measurements or expert assessments. I am curious whether the gains mainly come from the additional medical training data or from the model actually extracting medically valid evidence from the images through code execution.

  2. Table 3(a) and Appendix C.3 compare different tool interfaces using the same checkpoint trained with the full code interface, without reporting comparable training for the restricted interfaces. The lower performance may therefore partly reflect a training–evaluation mismatch rather than limitations of predefined tools. This is a useful ablation, but insufficient to establish the claimed superiority of flexible code.

Questions for authors

1.What evidence shows that medically valid intermediate analysis contributes to the gains beyond additional training on medical data?

2.Does MedCode still outperform fixed-tool agents when both receive interface-specific training on the same data with comparable budgets?

Reviewer_1Ku4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MedCode enables medical VLMs to generate and execute question-specific image-analysis programs. The authors construct MedCode-50k and train an 8B model through trajectory-based SFT followed by GRPO. Reported macro accuracy improves from 75.2% to 82.8% in distribution and from 76.2% to 85.5% on four held-out sources.

Critical strengths

The shared execution environment offers a practical approach to supporting diverse imaging modalities without specialist perception models.

Improvements are consistent across question categories and held-out sources.

The paper provides useful implementation details, reward ablations, and illustrative execution trajectories.

Critical weaknesses
  1. Unfair baseline comparison. The fixed-tool and crop baselines use the same code-trained checkpoint but different interfaces, so the reported gains may partly reflect interface mismatch rather than the value of general code.

  2. Weak validation of generated analyses. Trajectories are filtered mainly by final-answer correctness and successful execution; the intermediate measurements are not independently verified for medical validity or relevance.

  3. Unclear data independence. The paper does not clearly state whether images, patients, or duplicate studies are separated across splits, leaving possible leakage concerns.

  4. Limited statistical and efficiency analysis. No confidence intervals, multiple training runs, random-selection control, latency, token usage, or execution-cost analysis is provided.

Questions for authors

Does the advantage persist when fixed-tool and no-tool baselines receive matched training and inference budgets?

Were splits grouped by image and patient, and were duplicate or related images checked across sources?

How frequently are generated measurements medically valid, and how much does accuracy fall when execution feedback is removed or shuffled?

Are the smaller ablation gains robust, and what is the inference-cost increase?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.