acceptodds
Under review as a conference paper at ICLR 2027

Autonomous Medical Imaging Pipeline Synthesis with Validation-Gated Refinement

Abstract

Large language models enable increasingly autonomous machine learning workflows, but reliable end-to-end pipeline construction remains difficult in specialised domains such as medical imaging. Pipeline synthesis requires coordinated decisions over dataset interpretation, preprocessing, model design, training, and integration, while an incorrect intermediate decision can invalidate subsequent stages. We introduce MedPilot, a validation-gated agentic framework for autonomous medical imaging pipeline synthesis. Rather than generating a pipeline in a single pass, MedPilot decomposes synthesis into specialised stages and validates intermediate artefacts before they are propagated downstream. Failed candidates receive structured feedback and are iteratively repaired, while validated solutions are accumulated in an adaptive knowledge base and reused as references for related tasks. We evaluate MedPilot on six medical imaging benchmarks spanning supervised and semi-supervised segmentation and 3D classification. MedPilot achieves zero human intervention across all evaluated runs, with Dice scores of up to 0.945 on segmentation and 98.6% accuracy on classification, while consistently outperforming the evaluated baselines. These results demonstrate the effectiveness of validation-gated refinement for reliable and autonomous medical imaging pipeline synthesis.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Automatic Programming, Medical Imaging, Agentic Workflow

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ooQbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents MedPilot, a multi-agent LLM system (built on Claude Opus 4.7) that generates complete medical imaging training pipelines from a short task description and a dataset path. The work is split into a planning stage (dataset analysis, pipeline design) and a coding stage (parallel module implementation, assembly). Each intermediate artefact must pass a check before it is passed on: consistency of the dataset descriptors, an LLM-based semantic review of the design, model checking of a CSP# translation of the workflow, static analysis, generated unit tests, and an assembly test. Failed artefacts are regenerated with the validator's feedback, up to three times. Validated pipelines are stored in a knowledge base and retrieved for later tasks. On six benchmarks (five segmentation, one 3D classification), MedPilot is compared with eight agentic or IDE-based systems over three trials each. It obtains the best mean score on every benchmark and needs no human intervention, while the baselines need roughly 2 to 20 interventions per run.

Critical strengths
  1. The problem is timely and the staged design is sensible. Checking the dataset descriptors, the design and each module before they are passed downstream addresses a real failure mode of code-generating agents.
  2. The paper is candid about several limits of its evidence. It states that the compared systems run on different LLMs (Appendix C.3), reports published nnU-Net and reference numbers that are higher than its own (Tables 3 and 7), and includes a limitations section and a detailed run trace.
  3. The evaluation covers six benchmarks, eight baselines from three families and three trials per pair, and it measures autonomy in addition to accuracy. The ablation removes each component in turn.
Critical weaknesses
  1. The experiments do not isolate the contribution that the paper claims. The compared systems differ in the underlying LLM (Appendix C.3; the models behind MGX and the three IDE assistants are unknown or not stated), and only MedPilot receives curated medical-imaging knowledge, skills and a cross-task memory. The paper's own ablation (Figure 4) shows that these additions, not the validation gates, account for most of the accuracy: removing the role-specific knowledge costs 13.7 points and removing the adaptive knowledge base 8.4, against 3.3 to 4.7 for the validators. The reduction in interventions comes mainly from the execution and assembly tests (3.5 and 2.5), which are standard practice, while the formal check is never isolated (it is merged with the semantic check, 0.57). The curated knowledge covers exactly the evaluated task families (Table 5), and no task outside them is tested.
  2. The autonomy result rests on a metric that is counted by the authors on runs they operate themselves. There is no rule for when a human steps in, who does it, or when a baseline run is declared finished, and interactive assistants are penalised for being interactive. For MedPilot, a failed check is bypassed after three attempts instead of being escalated (L171-174), so it is unclear what would count as an intervention. The accuracy comparison uses three trials; from the reported means and standard deviations, the gap to the strongest baseline is within noise on FLARE22, FLARE21 and OrganMNIST3D. Basic protocol details are missing: split sizes and whether official splits are used, the training budget of the synthesised pipelines, whether the knowledge base (which stores downstream performance) is reset between trials, and the datasets and runs behind Figure 4.
  3. The practical value is not established. nnU-Net is fully automatic, needs no API calls, and according to the paper's own Table 7 is 2 to 7 Dice points better on all four segmentation benchmarks for which published numbers are given. The supervised-only nnU-Net even beats MedPilot's semi-supervised pipeline on FLARE22 by 7 points. nnU-Net is not run as a baseline under the same splits, nor is it offered to the agents as a tool. The argument that MedPilot covers a broader problem then rests on a single classification dataset. The cost comparison is made against the authors' own estimate of expert effort (16 to 40 hours) and not against this existing automatic alternative.
Questions for authors
  1. Could the authors rerun the main comparison with all open frameworks on the same LLM, and report MedPilot without its curated knowledge and skills, or the baselines with them? A result on a task family not covered by the knowledge sources, such as detection or registration, would also help.
  2. How was the Intervention Burden collected: who intervened, under what rule, and were the baseline scores measured after the interventions? Do the scaffolds that AutoML-Agent and M3Builder require (Appendix C.4) count as E2 interventions? What would have counted as an intervention for MedPilot?
  3. What are the split sizes, are the official splits used (patient-level for Promise12MSBench), and how long is each synthesised pipeline trained? Is the knowledge base reset between the three trials, and is the stored performance measured on validation or test data?
  4. How often did each validator reject an artefact across the 18 runs, and how often did the PAT check fail on its own? On which datasets and how many runs is Figure 4 based (a ΔIB of 0.57 is not a multiple of 1/18)?
1 comment

Answers to reviewer’s questions:

  1. We thank the reviewer for these suggestions. Our evaluation focuses on complete systems under their native configurations, as replacing their LLM backbones or introducing external knowledge and skills could alter their intended operation. Importantly, MedPilot, M³Builder, and MLZero share the same Claude Opus 4.7 backbone. We also evaluated our framework with GPT-5.2, achieving zero human intervention across all six datasets, with competitive downstream performance (0.8090, 0.7901, 0.7669, 0.8667, 0.9396, and 0.9752). Regarding component contributions, Fig. 4 shows that planning-level validation improves downstream performance, while execution and assembly validation substantially reduce human intervention. Domain knowledge and adaptive knowledge further improve pipeline quality. We are also extending MedPilot to additional medical imaging tasks, including image registration and denoising, to further explore its applicability across different task families.
  2. Intervention Burden (IB) measures human corrective actions required to complete pipeline synthesis, following three categories: (E1) additional instructions, error reporting, or corrections; (E2) manually supplied templates or skeleton code; and (E3) manual modifications to generated code or configurations. Routine approvals for code execution, file changes, or dependency installation are excluded, as are autonomous corrections. Framework-required scaffolds for AutoML-Agent and M³Builder are treated as standard initial inputs rather than additional E2 interventions, since they are necessary for native framework execution (Appendix C.4). The same intervention criteria apply to MedPilot; any human corrective action during its synthesis process would be counted. MedPilot required none across the evaluated runs.
  3. All methods use identical, predefined data splits. We follow the official dataset splits where available; otherwise, we use an 80/20 training–validation split, consistently applied across all methods. The held-out test set is reserved exclusively for final evaluation (Appendix C.2). Training durations depend on the generated pipelines. Taking BTCV as example, CAMEL, MetaGPT X, MaCTG, AutoML-Agent, and M³Builder require 146, 257, 203, 106, and 235 minutes, respectively. Our framework requires 477 minutes. The adaptive knowledge base is reset between independent trials. Stored performance is measured on test data when available, or validation data otherwise.
  4. Based on execution logs from 18 runs, the numbers of runs with at least one rejection were: data-analysis file validation (1/18), data-analysis semantic validation (2/18), architecture file validation (1/18), architecture semantic validation (5/18), PAT formal verification (1/18), static-code validation (4/18), code-execution validation (7/18), and assembly validation (3/18). All observed failures were resolved within the 3 times retry budget (particularly within one repair attempt), without validation bypasses. Figure 4 is based on six datasets, with three independent trials per dataset (18 runs in total). We identified the transcription error in the reported ΔIB of 0.57, which should be 0.56 (10/18).
Reviewer_c9FhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes an agentic code generation method called “medPilot”. While existing tools such as Antigravity and Cursor (as well as agentic AutoML frameworks) generate code based on a retrospective approach—namely, generating a first draft and then fixing it if and where needed—the proposed method applies semantic checks (“gates”) as well as formal verifications at very early stages to validate the correctness of the “plan” before any Python code is generated. The authors argue that this early semantic gating is important in the medical domain because it verifies that the generated code follows standard medical procedures. For experimental evaluation, the medPilot method was evaluated against other agentic AutoML frameworks and code-assisting tools. The method showed advantages in the amount of human intervention needed, as well as the quality of downstream tasks such as segmentation.

Critical strengths

-- In general, incorporating early-stage verifications of standard medical procedures into AutoML tools is a desired and important contribution to AutoML in medical imaging.

-- The paper shows that the proposed agentic pipeline outperforms powerful existing tools such as Cursor and Antigravity on classical medical imaging tasks in an end-to-end evaluation mode.

Critical weaknesses

-- Technical novelty: I don’t find the paper's contribution to be a core ML component or method, but rather an applied method that is useful for certain safety and regulatory domains, such as medical imaging.

-- Evaluation procedure is mixed: I’m not sure if all compared harness methods used the same LLM backbone (e.g., medPilot is based on Claude Opus 4.7, but AutoML agent uses GPT 5.5). Also, Antigravity and Cursor are frameworks that rely on human intervention, so the metric of human intervention might need some adjustment for these types of methods.

Questions for authors

none

1 comment

Answers to reviewer's concerns (weaknesses):

  1. MedPilot's primary contribution is the integration of phase-specific validation mechanisms throughout autonomous pipeline synthesis, rather than a new underlying ML architecture. By combining domain-specific semantic checks, formal verification, and code-level validation, MedPilot automatically detects and repairs errors before they propagate downstream, thereby reducing the need for human intervention/correction. This validation-gated approach is applicable beyond medical imaging wherever pipeline correctness depends on domain-specific constraints.
  2. We retain each baseline's native LLM configuration to preserve its intended operating conditions. Claude Opus 4.7 was selected for MedPilot's main evaluation to enable backbone-aligned comparisons with M³Builder and MLZero. To further address this concern, we provide additional results from our GPT-5.2 configuration, which achieves zero human intervention across all six datasets, with downstream performance of 0.8090, 0.7901, 0.7669, 0.8667, 0.9396, and 0.9752, respectively. These results support the framework's effectiveness across different LLM backbones.
    For interactive assistants such as Cursor and Antigravity, Intervention Burden (IB) counts only human corrective actions, including additional instructions or error corrections, supplying templates or skeleton code, and manually modifying generated scripts or configurations. Routine approvals for code execution, file modifications, or dependency installation are not counted as interventions, nor are errors resolved autonomously. Thus, IB measures substantive human assistance rather than routine interactions required by the tools' interfaces.
Reviewer_Jze1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce MedPilot, an LLM-based multi-agent framework for autonomous medical imaging pipeline generation. The framework consists of a planning stage that analyses datasets and designs architecture, followed by a coding stage that implements and assembles executable components.

Critical strengths
  1. The paper addresses an important problem in medical imaging by automating dataset analysis, model design, training, and pipeline generation

  2. The evaluation across six medical imaging datasets is a strength, providing a broad assessment of the proposed framework across diverse tasks. The inclusion of comparisons with existing agentic baselines offers useful empirical insights into its performance.

Critical weaknesses
  1. The proposed framework appears to be an engineering integration of existing agentic techniques rather than a novel methodological contribution. Despite the promising results, the experiments do not clearly identify which components drive the improvements.
  2. Section 2.2 allows failed validation checks to be bypassed after three repair attempts, weakening the claimed reliability guarantees.
  3. The cost analysis also focuses mainly on pipeline synthesis and API expenses without clearly reporting total model training and evaluation costs.
  4. MedPilot performs below the reported nnU-Net results on all four segmentation benchmarks in Appendix D. However, the paper does not adequately investigate or explain the reasons for this performance gap, raising concerns about the effectiveness of the generated pipelines.
Questions for authors
  1. What is the key methodological novelty, and which components contribute most to the performance improvements?
  2. How frequently are failed validation checks bypassed, and how is pipeline reliability ensured?
  3. What is the total computational cost, including pipeline generation, training, and evaluation?
  4. Why does MedPilot consistently underperform nnU-Net across all four segmentation benchmarks, and what justifies its practical effectiveness?
1 comment

Answers to reviewer's questions:

  1. MedPilot's key novelty lies in integrating phase-specific validation mechanisms throughout pipeline synthesis, covering dataset analysis, architecture design, module implementation, and assembly. Unlike approaches that rely primarily on fixed templates or naive execution feedback, MedPilot detects and repairs intermediate errors before they propagate downstream during both the planning and execution phases.
  2. No validation checks were bypassed in our experiments; all failures were resolved within one repair attempt (Sec. 2.2). Reliability is supported by stage-specific validation and feedback-driven repair. We acknowledge this potential concern and will introduce a default fallback that executes a simplified pipeline using validated solutions from the prior/adaptive knowledge base when repeated refinement fails. Rather than pursuing high performance through complex optimisation, this fallback prioritises basic functionality and successful pipeline completion, while remaining subject to validation.
  3. For training costs, we provide representative measurements on BTCV. Existing multi-agent frameworks require 146–257 minutes of training with 6.0–25.5 GB GPU memory, while AutoML-Agent and M³Builder require 106 and 235 minutes with 14.4 and 15.8 GB, respectively. Our framework requires 477 minutes and 14.9 GB. Higher computational demands are associated with more comprehensive preprocessing, patch-based training, and augmentation strategies (Table 2), while architecture and training configurations also influence memory usage. MedPilot primarily targets autonomous pipeline synthesis rather than minimising training costs.
  4. nnU-Net is specifically optimised for supervised segmentation but requires manual dataset reformatting and configuration, whereas MedPilot autonomously synthesises complete pipelines from task specifications and raw datasets. For FLARE22, nnU-Net does not natively support semi-supervised learning, and its reported result uses only 50 labelled cases. It is also not directly applicable to classification tasks such as OrganMNIST3D. Furthermore, comprehensive nnU-Net training can take days to weeks, depending on the configuration. While MedPilot does not surpass nnU-Net in segmentation accuracy, it prioritises end-to-end autonomy, broader task adaptability, and reduced development effort. Thus, the comparison highlights a trade-off between specialised performance optimisation and flexible, fully autonomous pipeline development, rather than a direct competition in segmentation accuracy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.