- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
MedCode trains Qwen3-VL-8B to answer medical multiple-choice questions by writing Python programs. A sandbox (MedCodeLab) executes them and returns text and images for further reasoning. MedCode-50k contains 49,817 Gemini 3.1 Pro trajectories on an OmniMedVQA subset (8 modalities, 4 question categories), filtered for answer correctness and at least one successful, nontrivial execution. Training is SFT on these trajectories, then GRPO on about 16k difficulty-selected questions with reward = accuracy + 0.5·format + 0.5·accuracy·execution. Accuracy rises from 75.2% to 82.8% in-distribution and from 76.2% to 85.5% on four held-out sources. Ablations cover program scope, training stage, reward gating and feedback modality.
Critical strengths
- The design choices are sound: randomized filenames prevent label leakage, execution is stateless, the prompt is given verbatim, and the correctness-gated reward is ablated (80.7 vs. 82.8). Gains are positive on every category and held-out source, which suggests more than source memorization. How much this matters depends on attributing the gains to code, which W1 shows the paper has not done.
- A useful resource, if released. About 50k executed multimodal trajectories could support work on medical tool use. The paper does not state any release plan.
Critical weaknesses
W1. Limited novelty, and the experiments do not isolate the effect of code. Training VLMs to generate and execute image-processing code via SFT followed by RL is established. Thyme, PyVision and DeepEyesV2 all do this, and none is cited or compared. MedCodeLab is a generic Python sandbox, and the training pipeline and reward are standard. The contribution therefore rests on the evidence, which has four gaps:
- No same-data, no-tool control. The baselines were never trained on OmniMedVQA, and the “10.1% over text-based reasoning” is the relative gain over the untrained backbone.
- Table 3a is not a fair toolkit comparison. It restricts the code-trained checkpoint at inference, so it never tests models trained on predefined toolkits.
- No framework+model baselines. Teacher accuracy with MedCodeLab is not reported. The acceptance rate (about 83%) is close to Gemini 3 Pro’s no-tool accuracy (82.9%).
- One backbone, one run, no variance.
W2. The evaluation and verification do not show that code yields valid evidence.
- Easy, near-distribution benchmark. OmniMedVQA is mostly templated classification, the held-out sources share modalities and templates with training, and no reasoning benchmark is used.
- Outcome-only verification. In the paper’s own CTR example, heart width is capped at 0.4× image width while thoracic width is maximized, which biases the ratio toward “normal.” The dermoscopy example assumes 1 mm ruler ticks. Both trajectories were accepted because the final answer matched.
- No causal or cost evidence. There is no test that execution outputs change answers. Turns, tokens and latency are not reported, even though the prompt mandates code and allows up to 10 turns.
Questions for authors
- How does a same-data, no-tool SFT+GRPO control perform?
- Were the Table 3a/3c conditions inference-time restrictions? How do toolset-specific models trained end to end compare, and which operations do the learned programs actually use?
- How many teacher samples were drawn per question? What is Gemini 3.1 Pro’s accuracy with and without MedCodeLab? How do frontier models with MedCodeLab and zero-shot Qwen3-VL-8B with MedCodeLab perform?
- What is the accuracy when observations are replaced with ones from another image, or emptied?
- What are the turns, tool calls, tokens and latency compared with the backbone and fixed-tool variants?
- Was the split done at question, image or patient level, and was overlap checked between held-out and training sources?
- What is the variance across seeds?
- How does MedCode perform on a reasoning benchmark outside OmniMedVQA?