- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents MedPilot, a multi-agent LLM system (built on Claude Opus 4.7) that generates complete medical imaging training pipelines from a short task description and a dataset path. The work is split into a planning stage (dataset analysis, pipeline design) and a coding stage (parallel module implementation, assembly). Each intermediate artefact must pass a check before it is passed on: consistency of the dataset descriptors, an LLM-based semantic review of the design, model checking of a CSP# translation of the workflow, static analysis, generated unit tests, and an assembly test. Failed artefacts are regenerated with the validator's feedback, up to three times. Validated pipelines are stored in a knowledge base and retrieved for later tasks. On six benchmarks (five segmentation, one 3D classification), MedPilot is compared with eight agentic or IDE-based systems over three trials each. It obtains the best mean score on every benchmark and needs no human intervention, while the baselines need roughly 2 to 20 interventions per run.
Critical strengths
- The problem is timely and the staged design is sensible. Checking the dataset descriptors, the design and each module before they are passed downstream addresses a real failure mode of code-generating agents.
- The paper is candid about several limits of its evidence. It states that the compared systems run on different LLMs (Appendix C.3), reports published nnU-Net and reference numbers that are higher than its own (Tables 3 and 7), and includes a limitations section and a detailed run trace.
- The evaluation covers six benchmarks, eight baselines from three families and three trials per pair, and it measures autonomy in addition to accuracy. The ablation removes each component in turn.
Critical weaknesses
- The experiments do not isolate the contribution that the paper claims. The compared systems differ in the underlying LLM (Appendix C.3; the models behind MGX and the three IDE assistants are unknown or not stated), and only MedPilot receives curated medical-imaging knowledge, skills and a cross-task memory. The paper's own ablation (Figure 4) shows that these additions, not the validation gates, account for most of the accuracy: removing the role-specific knowledge costs 13.7 points and removing the adaptive knowledge base 8.4, against 3.3 to 4.7 for the validators. The reduction in interventions comes mainly from the execution and assembly tests (3.5 and 2.5), which are standard practice, while the formal check is never isolated (it is merged with the semantic check, 0.57). The curated knowledge covers exactly the evaluated task families (Table 5), and no task outside them is tested.
- The autonomy result rests on a metric that is counted by the authors on runs they operate themselves. There is no rule for when a human steps in, who does it, or when a baseline run is declared finished, and interactive assistants are penalised for being interactive. For MedPilot, a failed check is bypassed after three attempts instead of being escalated (L171-174), so it is unclear what would count as an intervention. The accuracy comparison uses three trials; from the reported means and standard deviations, the gap to the strongest baseline is within noise on FLARE22, FLARE21 and OrganMNIST3D. Basic protocol details are missing: split sizes and whether official splits are used, the training budget of the synthesised pipelines, whether the knowledge base (which stores downstream performance) is reset between trials, and the datasets and runs behind Figure 4.
- The practical value is not established. nnU-Net is fully automatic, needs no API calls, and according to the paper's own Table 7 is 2 to 7 Dice points better on all four segmentation benchmarks for which published numbers are given. The supervised-only nnU-Net even beats MedPilot's semi-supervised pipeline on FLARE22 by 7 points. nnU-Net is not run as a baseline under the same splits, nor is it offered to the agents as a tool. The argument that MedPilot covers a broader problem then rests on a single classification dataset. The cost comparison is made against the authors' own estimate of expert effort (16 to 40 hours) and not against this existing automatic alternative.
Questions for authors
- Could the authors rerun the main comparison with all open frameworks on the same LLM, and report MedPilot without its curated knowledge and skills, or the baselines with them? A result on a task family not covered by the knowledge sources, such as detection or registration, would also help.
- How was the Intervention Burden collected: who intervened, under what rule, and were the baseline scores measured after the interventions? Do the scaffolds that AutoML-Agent and M3Builder require (Appendix C.4) count as E2 interventions? What would have counted as an intervention for MedPilot?
- What are the split sizes, are the official splits used (patient-level for Promise12MSBench), and how long is each synthesised pipeline trained? Is the knowledge base reset between the three trials, and is the stored performance measured on validation or test data?
- How often did each validator reject an artefact across the 18 runs, and how often did the PAT check fail on its own? On which datasets and how many runs is Figure 4 based (a ΔIB of 0.57 is not a multiple of 1/18)?
Answers to reviewer’s questions: