acceptodds
Under review as a conference paper at ICLR 2027

MAGMA: Multimodal Agentic Medical AI with Context-Based Iterative Refinement

Abstract

Predictive machine learning models hold substantial promise in healthcare, yet their development typically requires significant domain expertise and careful methodological reporting. Recent agentic AI systems have demonstrated strong coding capabilities, including for end-to-end machine learning pipeline construction. However, their use in healthcare raises additional requirements since workflows must be clinically-grounded, transparent, and reported in sufficient detail to support appraisal and reproducibility. We introduce MAGMA (Multimodal AGentic Medical AI), an agentic system for end-to-end clinical model development and evaluation. MAGMA's central contribution is a judge-driven iterative refinement loop in which a judge agent audits each attempt against a rubric derived from established clinical prediction modelling guidelines and returns a composite refinement score combining predictive performance with reporting compliance. This score and the item-level feedback guide the execution of the next attempt, so that clinical reporting compliance drives refinement. We also constructed MAGMA-Suite, a new benchmark of 10 clinical case studies spanning structured electronic health records, CXR images, ECG signals, and clinical text. We also propose MAGMAScore, a guideline-derived 22-item metric for assessing the clinical grounding of autonomous modeling workflows. All systems are evaluated on a fixed, patient-disjoint held-out partition across four prompt templates and three independent runs per configuration, and results are reported on a completion-weighted basis that accounts for failed runs. Under this protocol, single-agent baselines complete only 22.5% of attempted runs, while MAGMA completes every attempted run, and attains the highest completion-weighted AUROC in 9 of the 10 case studies. In a blinded expert adjudication conducted by two clinical machine-learning assessors against a rubric held out from the refinement loop, MAGMA received the highest score in 8 of 10 case studies (mean 0.74 versus 0.62 across baseline packages). We release the system, benchmark, and evaluation pipelines to support reproducible research on autonomous machine learning in healthcare.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
AutoML, Clinical AI, Iterative Refinement, Agentic Systems

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_AoBjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes MAGMA, a multi-agent framework for automated clinical machine learning, featuring judge-driven iterative refinement and a guideline-based evaluation metric. However, I have serious concerns about the quality and reliability of this submission. The manuscript contains numerous inconsistencies, unfinished text, and basic errors. These problems raise concerns that this is a fully AI-generated paper. In its current form, I do not believe this paper meets the requirements of ICLR and should be rejected.

Critical strengths

N.A.

Critical weaknesses
  1. The manuscript contains serious inconsistencies in experimental results. For example, Table S13 reports 552 total runs, although the individual system counts sum to 576. Its completed-run total is also incorrect (139 instead of 162). Furthermore, the HealthFlow results in Tables S11 and S16 are inconsistent. These errors directly undermine confidence in the reported results.

  2. The manuscript contains obvious unfinished text. For example, Section 4.3 ends a paragraph with "It is worth noting that..." without completing the sentence.

  3. The main methodological figure is inconsistent with the text. Figure 1 uses "TRIPODScore" in the refinement formula, whereas the corresponding equation and method description use "MAGMAScore". This inconsistency appears in the central contribution of the paper and suggests insufficient verification of the manuscript.

  4. The manuscript contains additional basic reporting inconsistencies. For example, Figure 2 reports a 100% completion rate for HealthFlow, whereas Table S10 reports 47 completed runs out of 48 (97.9%). The presence of such easily verifiable discrepancies further raises concerns about the overall quality control.

Questions for authors

See weakness.

2 comments

[1/n] We thank the reviewer for their thorough comments. We also address the ethics flag directly and fully reject the accusation that our paper was AI generated.

I have significant concerns about the paper’s soundness that do not appear easy to address.

We have addressed each of the soundness concerns raised:

  1. We fixed inconsistencies across main results and supplementary tables.
  2. We completed an unfinished sentence.
  3. We updated the figure due to a typo.
  4. We fully reject the accusation that the paper was AI generated, and this undermines the amount of effort and resources that was put into the work by the authors.

We hope these revisions resolve the reviewer’s concerns about soundness. Each revision is described in more detail below

The manuscript contains serious inconsistencies in experimental results. For example, Table S13 reports 552 total runs, although the individual system counts sum to 576. Its completed-run total is also incorrect (139 instead of 162). Furthermore, the HealthFlow results in Tables S11 and S16 are inconsistent. These errors directly undermine confidence in the reported results.

The manuscript contains additional basic reporting inconsistencies. For example, Figure 2 reports a 100% completion rate for HealthFlow, whereas Table S10 reports 47 completed runs out of 48 (97.9%). The presence of such easily verifiable discrepancies further raises concerns about the overall quality control.

Thank you for identifying the discrepancies. All of them derive from one per-run record of 576 logged baseline runs (480 single-agent, 48 HealthFlow, 48 MLZero), excluding the 96 CliMB-DC and HACHI attempts that produced no classifiable terminal outcome.

All the discrepancies identified trace back to a correction to the HealthFlow results that was applied to Tables S10 and S11 and was not propagated to the remaining HealthFlow relevant cells in Figure 2, Table S16, and Table S13. The values requiring correction are as follows:

  • HealthFlow completion (Figure 2):
    • Previous: 100% (48/48)
    • Correct Value: 97.9% (47/48)
  • HealthFlow completion-weighted AUROC (Figure 2):
    • Previous: 0.810
    • Correct Value: 0.802
  • HealthFlow conditional AUROC (Figure 2):
    • Previous: 0.810
    • Correct Value: 0.819
  • HealthFlow CS1–CS4, completion-weighted AUROC/AUPRC (Table S16):
    • Previous: 0.792/0.089, 0.958/0.792, 0.681/0.029, 0.810/0.611
    • Correct Value: 0.817/0.143, 0.924/0.740, 0.694/0.029, 0.774/0.617
  • Attempted-run total (Table S13):
    • Previous: 552
    • Correct Value: 576
  • Completed-run total (Table S13):
    • Previous: 139
    • Correct Value: 162
  • Attempted-run total (Section 5):
    • Previous: 552
    • Correct Value: 576

Tables S9, S10, and S11 are correct as printed in the paper. Table S9 already carries the corrected HealthFlow row (97.9%, 0.819 ± 0.105 conditional, 0.802 ± 0.157 completion-weighted).

As it relates to CS2 and Healthflow’s results, the reviewer makes a correct comment, whereas completion-weighting assigns zero to failed runs, so for a cell with no failures it is the identity, and the conditional and weighted values must coincide. HealthFlow completed 12/12 on CS2, so Table S11’s 0.924 is correct. The same holds for CS1 and CS3. CS4 is the one HealthFlow cell where the two differ, 11/12 completed, giving 0.845 conditional against 0.774 weighted, and in the corrected tables that is the only such inconsistency.

With the updated values, MAGMA holds the highest completion rate and the higher completion-weighted AUROC (0.808 vs 0.802), with the caveat already stated in the paper that HealthFlow is evaluated on case studies 1 - 4 only and hence the two are not directly comparable. Those inconsistencies do not change the claim that MAGMA leads completion-weighted AUROC in 9 of 10 case studies, and is outperformed only on case study 2, by HealthFlow.

[2/2]

The manuscript contains obvious unfinished text. For example, Section 4.3 ends a paragraph with "It is worth noting that..." without completing the sentence.

The reviewer is correct, the paragraph in section 4.3 ends mid-sentence, and that is an editing error from our side that will be removed.

The main methodological figure is inconsistent with the text. Figure 1 uses "TRIPODScore" in the refinement formula, whereas the corresponding equation and method description use "MAGMAScore". This inconsistency appears in the central contribution of the paper and suggests insufficient verification of the manuscript.

The reviewer is correct. The metric was renamed from TRIPODScore to MAGMAScore throughout the process of building MAGMA and figure 1 was not updated. The formula shown is otherwise identical to the formula used in our system. The figure will be corrected.

Given the weakness, I would like to flag this paper as a fully AI-generated paper and should be desk rejected.

We take this flag seriously, and respond by noting that our AI Use Statement (p. 9) discloses that generative AI tools were used to assist with editing the text of this paper, and that all such text was reviewed and revised by the authors, who take responsibility for the final content. That disclosure was made at submission and remains accurate. ICLR’s policy for authors requires disclosure of such use and does not prohibit it; the Code of Ethics concern it identifies is a substantial falsehood, instance of plagiarism, or misrepresentation.

The manuscript was authored by the authors, and generative AI was used for text editing as disclosed, not to produce results, and the inconsistencies identified by the reviewer are stale aggregates arising from a correction applied inconsistently across tables, not fabricated or unverifiable results.

We do not dispute the reviewer’s broader point about quality control. The manuscript was submitted with an unfinished sentence, a figure carrying a superseded metric name, and a data correction propagated to two tables out of six. We are grateful they were caught, and they will be fully corrected.

Reviewer_JAmgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MAGMA, a multi-agent system that autonomously develops clinical prediction pipelines from natural-language instructions using health records, chest radiographs, ECGs, and clinical text. Its central contribution is iterative refinement guided by a judge combining predictive performance with clinical reporting compliance. MAGMA completed all runs and led completion-weighted AUROC on nine tasks, demonstrating greater execution reliability and reporting quality.

Critical strengths
  1. The paper addresses an important practical problem in clinical machine learning by developing a multi-agent system for end-to-end clinical model development and evaluation.

  2. The authors introduce a multimodal benchmark covering 10 clinical prediction tasks. The experimental results demonstrate promising execution reliability and workflow completeness, highlighting the practical potential of the proposed framework.

Critical weaknesses
  1. Limited methodological novelty. MAGMA primarily integrates existing techniques, including multi-agent collaboration, code validation, and iterative feedback. While this integration provides a systematic framework for clinical AutoML, the substantive methodological contributions beyond existing approaches are not sufficiently clear.

  2. Insufficient ablation studies. The experiments mainly compare the complete MAGMA framework against other systems, without adequately isolating the contributions of its core components. Consequently, the current results do not sufficiently demonstrate which components are responsible for the observed improvements or whether the proposed key mechanisms are essential to the overall performance gains.

  3. Concerns regarding the fairness of experimental comparisons. MAGMA allows up to 10 refinement attempts per run and incorporates dedicated validation and error-repair mechanisms, whereas the baselines differ in their underlying language models and execution strategies. The paper does not sufficiently report or control the computational resources and refinement budgets across methods. Therefore, it remains unclear to what extent the observed improvements can be attributed to the proposed framework rather than additional execution and error-repair opportunities.

Questions for authors

See my review comments on Weaknesses.

2 comments

[1/n] We thank the reviewer for their thorough comments and feedback, and address all weaknesses and questions below.

I have significant concerns about the paper’s soundness that do not appear easy to address.

We have addressed each of the soundness concerns raised:

  1. We clarified the novelty concerns.
  2. We conducted new ablation experiments and added their results.
  3. We conducted experiments with matched budgets.

We hope these revisions resolve the reviewer’s concerns about soundness. Each revision is described in more detail below.

Limited methodological novelty. MAGMA primarily integrates existing techniques, including multi-agent collaboration, code validation, and iterative feedback. While this integration provides a systematic framework for clinical AutoML, the substantive methodological contributions beyond existing approaches are not sufficiently clear.

We thank the reviewer, and accept that the individual components of MAGMA are established techniques. We argue that our novelty and contribution is as follows:

  1. Existing agentic AutoML systems, including those we run as baselines, treat clinical reporting guidelines as a post-hoc checklist applied after building the model, if at all. We built MAGMA to make guideline compliance a term in the signal that drives refinement, combining predictive performance and reporting compliance in a singular composite score that is used to refine performance with each attempt.
  2. The benchmark and evaluation protocol that we offer through MAGMA-Suite provides a contribution to evaluating clinical AutoML systems using a fixed patient-disjoint held-out partition. This is independent of MAGMA itself as a system.

Insufficient ablation studies. The experiments mainly compare the complete MAGMA framework against other systems, without adequately isolating the contributions of its core components. Consequently, the current results do not sufficiently demonstrate which components are responsible for the observed improvements or whether the proposed key mechanisms are essential to the overall performance gains. We thank the reviewer for highlighting the need to isolate the contributions of MAGMA's individual components. We conducted three targeted ablations to isolate the contributions of MAGMA's core components:

  1. No Validators: We removed both the data and model validator agents while retaining the judge and iterative refinement loop. This configuration examines the contribution of dedicated validation agents to pipeline reliability and predictive performance.
  2. No Guideline Feedback: We retained the full multi-agent architecture and iterative refinement loop but removed MAGMAScore and its associated 22-item guideline feedback from the refinement process. The judge instead optimized validation AUROC alone. This configuration isolates the contribution of guideline-driven refinement from performance-based refinement.

All configurations were evaluated using the same clinical prompt template and held-out evaluation protocol, with three independent runs per configuration. We report mean held-out AUROC and held-out AUPRC.

Case StudyConfigurationAUROCAUPRC
CS1No Guideline Feedback0.75810.3540
CS1Full MAGMA0.78000.3677

[2/2]

Concerns regarding the fairness of experimental comparisons. MAGMA allows up to 10 refinement attempts per run and incorporates dedicated validation and error-repair mechanisms, whereas the baselines differ in their underlying language models and execution strategies. The paper does not sufficiently report or control the computational resources and refinement budgets across methods. Therefore, it remains unclear to what extent the observed improvements can be attributed to the proposed framework rather than additional execution and error-repair opportunities.

We thank the reviewer for their comment. We report the protocol for submitted baselines first, then the matched-budget reruns.

  1. Single-agent baseline configuration
SettingValue
Model versionsgpt-5.5-2026-04-23, claude-opus-4-8, gemini-3.1-pro-preview, deepseek-v4-pro
Max output tokens32,000
Samplingprovider default
Promptoriginal template (4 variants) plus a single-agent system prompt
Tool accessnone; text in, text out, with no code execution, browsing or file access
Attempts1
Repair allowancenone
Feedback after an attemptnone
Conversation memorynot applicable
Compute per attempt1x A100, 200 GB system memory, 64 CPUs, 48 h
  1. Matched-budget reruns We matched the resource budget conditions for the base model of MAGMA. Similar to MAGMA:
  • The base model gets up to 10 attempts per run.
  • A run stops after 3 consecutive attempts without improvement (attempt AUROC exceeding the previous best by more than 1e-4), matching MAGMA’s plateau rule.
  • After each attempt, the baselines are provided with the exit code, the last 60 lines of stderr, the last 40 lines of stdout, the validation AUROC it printed, its best score so far, and how many attempts remain.
  • No judge, rubric or MAGMAScore feedback, and no held-out data.
  • Additionally, the conversation is resent each call, with the best attempt and the last 2 in full and older attempts as a one-line summary.
  • The best attempt is selected by the model’s own validation AUROC, then scored on held-out data independently, as is the case for MAGMA.

We ran the base model with a matched budget on CS1 only during the rebuttal period, which achieved AUROC of 0.844. We note:

  • This result is on a single run of the base model on one case study and one prompt, while MAGMA’s value is the mean of three runs that range from 0.713 to 0.854 and all prompts. A conclusive comparison requires three runs per system on all ten case studies and all four prompt templates, which we are not able to complete within the review period. As such, this result is not conclusive.
  • MAGMA’s main strength lies in its execution reliability. In the submitted results, MAGMA completed all 120 runs while the single-agent baselines completed 22.5% of single-attempt runs. Additionally, MAGMA leads completion-weighted AUROC in 9 of the 10 case studies. On CS1 itself, MAGMA’s AUROC with the other three prompt templates (0.824 to 0.858) is comparable to the matched-budget base model’s 0.844. As a result, whether a matched budget also closes the gap on the remaining case studies, in particular the bimodal ones, is not answered by this experiment.

We hope these additions address the reviewer’s concerns. We would be grateful if the reviewer would consider revising their score and we are happy to answer any further questions.

Reviewer_CcByOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper provides MAGMA, a multimodal clinical agentic system, MAGMA-Suite (10 unimodal/bimodal case studies across EHR, text, CXR, ECG) and MAGMAScore, a 22-item TRIPOD+AI-derived compliance signal, with a separate blinded 8-item human expert rubric. MAGMA completed all 120 runs, achieved strong completion-weighted predictive performance, and scored highest on expert workflow evaluation in 8/10 cases.

Critical strengths

The paper makes a good attempt to evaluate multimodal clinical agents in realistic end-to-end settings.

Critical weaknesses
  1. Validity of MAGMA-Suite: MAGMA-Suite comprises only ten heterogeneous case studies, varying substantially in input modality and prediction task. It remains unclear whether performance reflects task-specific characteristics or modality fusion. Despite this heterogeneity, the same 22-item MAGMAScore and eight-item rubric are applied across all tasks. Aggregate scores may therefore obscure modality effects or workflow quality, and no per-task validity or sensitivity analysis is reported.
  2. Effectiveness of the MAGMA framework: MAGMA is evaluated only on its own benchmark, with GPT-5.4 as the sole backbone for all agents. No external benchmark or backbone ablation isolates the architecture from the base model or demonstrates generalization. The claimed architectural advantage therefore remains underdetermined.
Questions for authors
  1. Could you provide concrete details for one or two of the ten case studies—specifically the exact input materials and the precise prediction task? The main text and Appendix C (Tables S2–S3) remain largely descriptive, which makes it difficult to understand what the benchmark actually operationalizes.
  2. Would the MAGMA framework’s results generalize to benchmarks or datasets beyond the two rubric instruments developed here?
  3. Are there validation experiments demonstrating that the LLM-as-a-Judge scoring the 22-item rubric produces correct, reliable, and consistent estimates of model performance?
3 comments

[1/n] We thank the reviewer for their thorough comments and feedback, and address all weaknesses and questions below.

I have significant concerns about the paper’s soundness that do not appear easy to address.

We have addressed each of the soundness concerns raised:

  1. We reconciled the results across the main text and appendix.
  2. We conducted ablations without guideline-based refinement.
  3. We clarified the purpose of the blinded expert assessment.
  4. We ran an experiment with a matched budget.

We hope these revisions resolve the reviewer’s concerns about soundness. Each revision is described in more detail below.

Validity of MAGMA-Suite: MAGMA-Suite comprises only ten heterogeneous case studies, varying substantially in input modality and prediction task. It remains unclear whether performance reflects task-specific characteristics or modality fusion. Despite this heterogeneity, the same 22-item MAGMAScore and eight-item rubric are applied across all tasks. Aggregate scores may therefore obscure modality effects or workflow quality, and no per-task validity or sensitivity analysis is reported.

The reviewer mentions evaluating MAGMA on its own benchmark. However, we note that the MAGMA-Suite benchmark is a task suite derived from pre-existing, publicly available datasets, Namely, we have built all 10 case studies in the suite from MIMIC-IV, MIMIC-CXR, MIMIC-ECG, MIMIC-Note, PhysioNet 2019 Challenge, NIH ChestX-Ray14, and PTB-XL, each obtained under its respective data use agreement. As such, we did not evaluate MAGMA on its own benchmark, we defined task definitions relevant for clinical AutoML, and fixed patient-disjoint partitions for fair evaluations. Additionally, our predictive results do not depend on any instrument built by our team, we merely compare systems based on their AUROC and AUPRC performance.

Would the MAGMA framework’s results generalize to benchmarks or datasets beyond the two rubric instruments developed here?

As it relates to external benchmarks, to the best of our knowledge, we are not aware of any benchmark that evaluates end-to-end clinical predictive model development on patient data with modality fusion, which is why we built MAGMA-Suite. The closest existing work tests different tasks:

  • MedAgent Bench (Jiang et al., 2025) places agents in a virtual EHR to execute clinical actions rather than to build models
  • CodeClinic (Ossowski et al., 2026) evaluates agents writing code over MIMIC-IV for surveillance and information seeking with no model training
  • BioML-bench (Miller et al., 2025) covers end-to-end biomedical ML on drug discovery, protein engineering, single-cell omics, and imaging

As it pertains to the rubric instruments used, we would like to clarify that only MAGMAScore was developed by our team, and we agree it should not be read as independent evidence, since it is also the refinement objective, which we state in section 5. The expert rubric, however, is adapted from the clinical AutoML requirements framework of Scott et al. (2024), and it was applied by two assessors who were not involved in this study and were blinded to system identity. We reinforce the conclusion that our workflow quality claim rests on that instrument, and not on MAGMAScore.

Effectiveness of the MAGMA framework: MAGMA is evaluated only on its own benchmark, with GPT-5.4 as the sole backbone for all agents. No external benchmark or backbone ablation isolates the architecture from the base model or demonstrates generalization. The claimed architectural advantage therefore remains underdetermined.

We thank the reviewer for highlighting the need to evaluate MAGMA's generalizability beyond its original backbone. To address this concern, we conducted an additional experiment using Gemini-3.1-Pro as an alternative backbone, while retaining MAGMA's multi-agent architecture and refinement mechanism.

On CS1, MAGMA with Gemini-3.1-Pro successfully completed all three independent runs, achieving a mean held-out AUROC of 0.800 and AUPRC of 0.350. This is better than Gemini-3.1-Pro single agent which achieved AUROC of 0.790 and AUPRC of 0.084 (Table S11).

These findings provide evidence that MAGMA's functionality is not restricted to GPT-5.4 and that its agentic workflow can be transferred to another LLM backbone.

[2/n]

Could you provide concrete details for one or two of the ten case studies—specifically the exact input materials and the precise prediction task? The main text and Appendix C (Tables S2–S3) remain largely descriptive, which makes it difficult to understand what the benchmark actually operationalizes.

We agree that Tables S2-S3 describe the tasks without thoroughly explaining what each system receives, so we will add a section in the appendix giving the full specification per case study. We provide a summary of the cohorts below, and give concrete details for two case study examples (CS1 unimodal and CS7 multimodal), as requested.

In the table below, we provide the task, cohort size, and label prevalence for all ten case studies:

CSTaskTrain NTrain prevalenceTest NTest prevalence
CS1Sepsis onset34,2857.29%6,0517.14%
CS2COPD diagnosis123,98610.16%21,92810.07%
CS3Pneumonia26,1853.30%4,6203.10%
CS4Arrhythmia16,03855.52%2,83156.45%
CS5Pleural effusion46,00326.29%8,02927.39%
CS6In-hospital mortality107,6976.02%18,8516.34%
CS730-day readmission119,46227.13%21,09626.77%
CS8Cardiomegaly46,25829.22%8,10429.62%
CS930-day readmission38,25438.46%6,66739.10%
CS1030-day readmission91,63630.73%16,11930.59%

Additionally, we provide the details for CS1 and CS7 below, as two randomly chosen case studies of unimodal and bimodal nature, respectively:

CS1: Sepsis onset (unimodal, EHR)CS7: 30-day readmission (bimodal, EHR + text)
SourcePhysioNet/CinC Challenge 2019, training sets A and B pooledMIMIC-IV v1.0 + MIMIC-Note (discharge)
CohortAny adult ICU patient with a record filePatients with >=1 admission, >=1 discharge note, and >=1 admission not ending in death
Train / test N34,285 / 6,051119,462 / 21,096
Predictors40 fields per hour: 8 vital signs, 26 laboratory values, 6 demographic/administrative, all listed in the table below. Record to date visible at each hourStructured MIMIC-IV tables plus the free-text discharge note
MissingnessHeavy, encoded as NaN, propagated to the system without imputationPropagated to the system without imputation
Index timeEach ICU hour t, predicting onset in hours t+1 to t+6Discharge time of each eligible admission
LabelNative Challenge SepsisLabel, already shifted by the Challenge to become positive 6 h before suspicion of sepsisDerived from admissions.csv: positive if a later admission begins within 30 days of the index discharge. Admissions ending in death excluded as index admissions
Patient-level rollupAny positive hourAny positive eligible admission
Prevalence (train / test)7.29% / 7.14%27.13% / 26.77%
FusionNot applicableNotes join to admissions on hadm_id (exact); multiple notes or admissions resolved by the patient-level rollup
Provided to the systemRaw .psv files, a concatenated aggregated.csv with a patient identifier, a per-patient summary table, and a data dictionary in the promptMIMIC-IV tree, discharge note table, and a data dictionary in the prompt. The readmission label is not pre-computed, so label construction is part of the task

[3/3]

Additionally, the predictors in CS1 are the same exact feature set used in the PhysioNet 2019 Challenge, in addition to the Sepsis label, which was also provided in the original PhysioNet 2019 dataset. The predictors are shown below:

CategoryFeatureDescriptionUnit
Vital signHRHeart ratebeats/min
Vital signO2SatPulse oximetry%
Vital signTempTemperature°C
Vital signSBPSystolic blood pressuremmHg
Vital signMAPMean arterial pressuremmHg
Vital signDBPDiastolic blood pressuremmHg
Vital signRespRespiratory ratebreaths/min
Vital signEtCO2End-tidal CO₂mmHg
LaboratoryBaseExcessBase excessmmol/L
LaboratoryHCO3Bicarbonatemmol/L
LaboratoryFiO2Fraction of inspired oxygenfraction
LaboratorypHArterial pH—
LaboratoryPaCO2Arterial partial pressure of CO₂mmHg
LaboratorySaO2Arterial oxygen saturation%
LaboratoryASTAspartate aminotransferaseIU/L
LaboratoryBUNBlood urea nitrogenmg/dL
LaboratoryAlkalinephosAlkaline phosphataseIU/L
LaboratoryCalciumCalciummg/dL
LaboratoryChlorideChloridemmol/L
LaboratoryCreatinineCreatininemg/dL
LaboratoryBilirubin_directDirect bilirubinmg/dL
LaboratoryGlucoseSerum glucosemg/dL
LaboratoryLactateLactic acidmg/dL
LaboratoryMagnesiumMagnesiummmol/dL
LaboratoryPhosphatePhosphatemg/dL
LaboratoryPotassiumPotassiummmol/L
LaboratoryBilirubin_totalTotal bilirubinmg/dL
LaboratoryTroponinITroponin Ing/mL
LaboratoryHctHematocrit%
LaboratoryHgbHemoglobing/dL
LaboratoryPTTPartial thromboplastin timeseconds
LaboratoryWBCWhite blood cell count×10³/µL
LaboratoryFibrinogenFibrinogenmg/dL
LaboratoryPlateletsPlatelet count×10³/µL
Demographic / adminAgeAge (capped at 100 for ≥90)years
Demographic / adminGender0 = female, 1 = male—
Demographic / adminUnit1ICU unit indicator: medical ICU (MICU)—
Demographic / adminUnit2ICU unit indicator: surgical ICU (SICU)—
Demographic / adminHospAdmTimeTime between hospital and ICU admissionhours
Demographic / adminICULOSICU length of stay (hourly time index)hours
LabelSepsisLabel1 from 6 h before sepsis onset (Sepsis-3), else 0—

We note that we deliberately designed the benchmark to not provide any details regarding missing data handling, feature window choice within the record provided, and any sub-cohort restrictions since those are capabilities that we are testing rather than protocol decisions.

Are there validation experiments demonstrating that the LLM-as-a-Judge scoring the 22-item rubric produces correct, reliable, and consistent estimates of model performance?

To address this concern, we conducted an ablation on CS1 in which MAGMAScore and its associated 22-item guideline feedback were removed from refinement, while retaining the judge, orchestrator, data and model workers, validators, and iterative refinement mechanism. The refinement objective was changed to validation AUROC alone, and we used the same clinical prompt template and held-out evaluation protocol, with three independent runs per configuration. The results are summarized below.

Case StudyConfigurationAUROCAUPRC
CS1No Guideline Feedback0.75810.3540
CS1Full MAGMA0.78000.3677

Full MAGMA achieved higher held-out AUROC (0.7800 vs. 0.7581) and slightly higher AUPRC (0.3677 vs. 0.3540) compared with the No Guideline Feedback configuration.

Although this does not directly assess the judge with guideline refinement since this would require extensive qualitative review, it does demonstrate its utility in this context. We will add this result to the appendix.

We hope these additions address the reviewer’s concerns. We would be grateful if the reviewer would consider revising their score and we are happy to answer any further questions.

Reviewer_8nEMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MAGMA coordinates agents that build, validate, and revise clinical prediction pipelines. Refinement uses validation performance and a reporting-guideline score. The evaluation covers ten tasks across four modalities, measuring execution success, held-out prediction performance, and expert-rated pipeline quality.

Critical strengths
  • Fixed patient-disjoint test partitions and explicit accounting for unsuccessful runs address weaknesses of evaluations based only on self-reported validation scores.
  • Multiple prompt formulations, repeated runs, and blinded expert assessment broaden the evaluation beyond a few successful demonstrations.
Critical weaknesses
  • The reported results cannot be reconciled across the main text and appendix: e.g. Figure 2 gives HealthFlow 100% completion, while Table S10 gives 47/48. For CS2, Table S11 reports AUROC 0.924 with 12/12 completed runs, whereas Table S16 reports 0.958 under failure-zero weighting. With no failures, these values should coincide. Table S13's completed counts sum to 162, but its total is 139. These discrepancies affect the central reliability and performance comparisons.
  • The evaluation does not isolate the value of guideline-based refinement from additional orchestration and retry opportunities. A comparison with the same agents and budgets but without guideline feedback is needed. The expert assessment also measures several properties already rewarded during refinement. Tables S1 and S7 both cover data processing, feature engineering, model comparison, tuning, and modality handling. Blinded scoring remains useful, but the claim that no MAGMAScore item appears in the external rubric overstates its independence.
Questions for authors
  • Can you reconcile Figure 2 and Tables S10, S11, S13, and S16 using one per-run record? Please identify which reported aggregate values require correction.
  • How much of the improvement remains when guideline feedback is removed while keeping the same agents, validation access, retry limits, and computational budget?
  • What model versions, tool access, repair allowances, and total resource budgets were available to each baseline? Which differences remain after matching these conditions?
  • How do the external rubric's criteria differ from the properties rewarded by MAGMAScore? Can you report results separately for overlapping and independently assessed criteria?
3 comments

[1/n] We thank the reviewer for their constructive feedback. We are glad that the reviewer appreciates the fixed patient-disjoint partitions, explicit accounting for unsuccessful runs, multiple prompt formulations, repeated runs, and blinded expert assessment.

I have significant concerns about the paper’s soundness that do not appear easy to address.

We have addressed each of the soundness concerns raised:

  1. We reconciled the results across the main text and appendix.
  2. We conducted ablations without guideline-based refinement.
  3. We clarified the purpose of the blinded expert assessment.
  4. We ran an experiment with a matched budget.

We hope these revisions resolve the reviewer’s concerns about soundness. Each revision is described in more detail below.

The reported results cannot be reconciled across the main text and appendix: e.g. Figure 2 gives HealthFlow 100% completion, while Table S10 gives 47/48. For CS2, Table S11 reports AUROC 0.924 with 12/12 completed runs, whereas Table S16 reports 0.958 under failure-zero weighting. With no failures, these values should coincide. Table S13's completed counts sum to 162, but its total is 139. These discrepancies affect the central reliability and performance comparisons.

Can you reconcile Figure 2 and Tables S10, S11, S13, and S16 using one per-run record? Please identify which reported aggregate values require correction.

Thank you for identifying the discrepancies. All of them derive from one per-run record of 576 logged baseline runs (480 single-agent, 48 HealthFlow, 48 MLZero), excluding the 96 CliMB-DC and HACHI attempts that produced no classifiable terminal outcome.

  • HealthFlow completion (Figure 2):
    • Previous: 100% (48/48)
    • Correct Value: 97.9% (47/48)
  • HealthFlow completion-weighted AUROC (Figure 2):
    • Previous: 0.810
    • Correct Value: 0.802
  • HealthFlow conditional AUROC (Figure 2):
    • Previous: 0.810
    • Correct Value: 0.819
  • HealthFlow CS1–CS4, completion-weighted AUROC/AUPRC (Table S16):
    • Previous: 0.792/0.089, 0.958/0.792, 0.681/0.029, 0.810/0.611
    • Correct Value: 0.817/0.143, 0.924/0.740, 0.694/0.029, 0.774/0.617
  • Attempted-run total (Table S13):
    • Previous: 552
    • Correct Value: 576
  • Completed-run total (Table S13):
    • Previous: 139
    • Correct Value: 162
  • Attempted-run total (Section 5):
    • Previous: 552
    • Correct Value: 576

Tables S9, S10, and S11 are correct as printed in the paper. Table S9 already carries the corrected HealthFlow row (97.9%, 0.819 ± 0.105 conditional, 0.802 ± 0.157 completion-weighted).

As it relates to CS2 and Healthflow’s results, the reviewer makes a correct comment, whereas completion-weighting assigns zero to failed runs, so for a cell with no failures it is the identity, and the conditional and weighted values must coincide. HealthFlow completed 12/12 on CS2, so Table S11’s 0.924 is correct. The same holds for CS1 and CS3. CS4 is the one HealthFlow cell where the two differ, 11/12 completed, giving 0.845 conditional against 0.774 weighted, and in the corrected tables that is the only such inconsistency.

With the updated values, MAGMA holds the highest completion rate and the higher completion-weighted AUROC (0.808 vs 0.802), with the caveat already stated in the paper that HealthFlow is evaluated on case studies 1 - 4 only and hence the two are not directly comparable. Those inconsistencies do not change the claim that MAGMA leads completion-weighted AUROC in 9 of 10 case studies, and is outperformed only on case study 2, by HealthFlow.

The evaluation does not isolate the value of guideline-based refinement from additional orchestration and retry opportunities. A comparison with the same agents and budgets but without guideline feedback is needed.

How much of the improvement remains when guideline feedback is removed while keeping the same agents, validation access, retry limits, and computational budget?

We thank the reviewer for this comment. To address this concern, we conducted an ablation on CS1 in which MAGMAScore and its associated 22-item guideline feedback were removed from refinement, while retaining the judge, orchestrator, data and model workers, validators, and iterative refinement mechanism. The refinement objective was changed to validation AUROC alone, and we used the same clinical prompt template and held-out evaluation protocol, with three independent runs per configuration. The results are summarized below.

Case StudyConfigurationAUROCAUPRC
CS1No Guideline Feedback0.75810.3540
CS1Full MAGMA0.78000.3677

Full MAGMA achieved higher held-out AUROC (0.7800 vs. 0.7581) and slightly higher AUPRC (0.3677 vs. 0.3540) compared with the No Guideline Feedback configuration.

We will add this result to the appendix.

[2/n]

The expert assessment also measures several properties already rewarded during refinement. Tables S1 and S7 both cover data processing, feature engineering, model comparison, tuning, and modality handling. Blinded scoring remains useful, but the claim that no MAGMAScore item appears in the external rubric overstates its independence.

How do the external rubric's criteria differ from the properties rewarded by MAGMAScore? Can you report results separately for overlapping and independently assessed criteria?

We thank the reviewer for their comment. Our statement that no MAGMAScore item appears in the external rubric is inaccurate and we will remove it. Our intended argument was that computing the MAGMAScore using the LLM judge is independent from the blinded expert adjudication as an evaluation approach.

While both MAGMAScore and the expert rubric have overlap in 6 of the eight items (Table 1 below), both items assess the same thing: whether a system follows accepted practice for building a clinical prediction model.

Table 1: Overlap between MAGMAScore and Expert Evaluation Rubric.

Expert itemScott FRMAGMAScore item
1. Modality HandlingFR015.1
2. Data Cleansing and TransformationFR032.3
3. Feature EngineeringFR023a.3
4. Model ComparisonFR063b.2
5. Metric DiversityFR135.3
6. OptimizationFR083b.5
7. Scoring Pipelines-none
8. Documentation-none directly

The independence that the expert assessment provides lies in how the scoring was done, rather than in what was scored: (i) the instrument is different, with 8 items on a 3-point scale taken from Scott et al. rather than 22 items derived from four sources, including TRIPOD+AI; and (ii) the scoring was done by two human assessors who were not involved in the study, who were blinded to system identity, and who had no knowledge of MAGMAScore or of any system’s score on it.

Table 2 below shows the item split, as requested by the reviewer, normalized to [0,1]:

SubsetMAGMABaselines
Overlapping (items 1 - 6)0.7130.573
Independent (items 7 - 8)0.8380.734
All 8 items0.7440.613

We will report those results in the appendix for transparency, but would like to re-iterate that the purpose of the human expert assessment was to conduct an unbiased external review of the systems.

[3/3]

What model versions, tool access, repair allowances, and total resource budgets were available to each baseline? Which differences remain after matching these conditions? We thank the reviewer for their comment. We report the protocol for submitted baselines first, then the matched-budget reruns.

  1. Single-agent baseline configuration
SettingValue
Model versionsgpt-5.5-2026-04-23, claude-opus-4-8, gemini-3.1-pro-preview, deepseek-v4-pro
Max output tokens32,000
Samplingprovider default
Promptoriginal template (4 variants) plus a single-agent system prompt
Tool accessnone; text in, text out, with no code execution, browsing or file access
Attempts1
Repair allowancenone
Feedback after an attemptnone
Conversation memorynot applicable
Compute per attempt1x A100, 200 GB system memory, 64 CPUs, 48 h
  1. Matched-budget reruns

We matched the resource budget conditions for the base model of MAGMA. Similar to MAGMA:

  • The base model gets up to 10 attempts per run.
  • A run stops after 3 consecutive attempts without improvement (attempt AUROC exceeding the previous best by more than 1e-4), matching MAGMA’s plateau rule.
  • After each attempt, the baselines are provided with the exit code, the last 60 lines of stderr, the last 40 lines of stdout, the validation AUROC it printed, its best score so far, and how many attempts remain.
  • No judge, rubric or MAGMAScore feedback, and no held-out data.
  • Additionally, the conversation is resent each call, with the best attempt and the last 2 in full and older attempts as a one-line summary.
  • The best attempt is selected by the model’s own validation AUROC, then scored on held-out data independently, as is the case for MAGMA.

We ran the base model with a matched budget on CS1 only during the rebuttal period, which achieved AUROC of 0.844. We note:

  • This result is on a single run of the base model on one case study and one prompt, while MAGMA’s value is the mean of three runs that range from 0.713 to 0.854 and all prompts. A conclusive comparison requires three runs per system on all ten case studies and all four prompt templates, which we are not able to complete within the review period. As such, this result is not conclusive.
  • MAGMA’s main strength lies in its execution reliability. In the submitted results, MAGMA completed all 120 runs while the single-agent baselines completed 22.5% of single-attempt runs. Additionally, MAGMA leads completion-weighted AUROC in 9 of the 10 case studies. On CS1 itself, MAGMA’s AUROC with the other three prompt templates (0.824 to 0.858) is comparable to the matched-budget base model’s 0.844. As a result, whether a matched budget also closes the gap on the remaining case studies, in particular the bimodal ones, is not answered by this experiment.

We hope these additions address the reviewer’s concerns. We would be grateful if the reviewer would consider revising their score and we are happy to answer any further questions.

Comments on the paper

2 comments

[1/n] Dear PC, SAC, AC, and Reviewers,

We sincerely thank you for your detailed and constructive feedback, which allowed us to substantially strengthen the manuscript. We are encouraged that the reviewers recognized the fixed patient-disjoint test partitions, the explicit accounting for failed runs, the blinded expert assessment, multiple prompt formulations, the practical importance of end-to-end clinical model development, and our multimodel benchmark.

Based on your feedback, we have made the following updates to the paper;

  • Corrections: we confirmed the inconsistencies identified by reviewers 8nEM and AoBj. All of the inconsistencies trace back to one correction to the HealthFlow results that was applied to Tables S9, S10, and S11 and was not propagated to Figure 2, Table S16, and Table S13. We list each printed and corrected value in our responses, and note that the corrections do not change our central claims.

  • Ablations: we added ablations of MAGMA without the guideline-based feedback, under the same agents, attempt limit, and compute. The results of our ablations show that the contribution of guideline-based feedback is task-dependent. Across CS1 to CS3, with three runs per configuration, removing guideline feedback lowered MAGMAScore on two of the three case studies (0.923 to 0.833 on CS1 and 0.924 to 0.864 on CS3), while its effect on held-out AUROC was mixed: lower on CS1 (0.780 to 0.758), unchanged on CS2 (0.952 to 0.954), and higher on CS3 (0.696 to 0.733). We therefore read guideline-driven refinement as improving reporting compliance, with a possible trade-off against predictive performance on some tasks, and we note that these results are preliminary given that they cover three of the ten case studies.

  • Matched-budget baselines: we reran MAGMA’s base model (GPT-5.4) as a single-agent with the same 10-attempt allowance and stopping rule as MAGMA, with execution feedback and no judge or rubric feedback. This was run on CS1 only during the rebuttal period, where it achieved a held-out AUROC of 0.844. We note that this result is not conclusive, since it is a single run of the base model on one case study and one prompt, while MAGMA’s value is the mean of three runs ranging from 0.713 to 0.854, and a full comparison on all ten case studies and all four prompt templates could not be completed within the review period. MAGMA’s main strength lies in its execution reliability, whereas in the submitted results, it completed all 120 runs against 22.5% of single-attempt runs for the single-agent baselines, and it leads completion-weighted AUROC in 9 of the 10 case studies. On CS1 itself, MAGMA’s AUROC with the other three prompt templates (0.824 to 0.858) is comparable to the matched-budget value. Whether a matched budget also closes the gap on the remaining case studies, in particular the bimodal ones, is not answered by this experiment.

  • Alternative backbone: We evaluated MAGMA using Gemini-3.1-Pro as an alternative backbone to assess whether the framework's capabilities extend beyond its original GPT-5.4 configuration. On CS1, Gemini-3.1-Pro successfully completed all three independent runs, achieving a mean held-out AUROC of 0.8004, AUPRC of 0.350, and MAGMAScore of 0.9697. For comparison, the original GPT-5.4 configuration achieved a held-out AUROC of 0.7800, AUPRC of 0.3677, and MAGMAScore of 0.9228 on the same case study. Gemini-3.1-Pro achieved slightly higher AUROC and guideline compliance, while GPT-5.4 achieved slightly higher AUPRC. These findings demonstrate that MAGMA can successfully operate with an alternative LLM backbone while maintaining comparable predictive performance and high guideline compliance. We emphasize that this experiment covers only one case study and therefore provides preliminary evidence of backbone flexibility rather than conclusive evidence of generalization across models and clinical tasks.

  • Expert rubric overlap: we removed the statement that no MAGMAScore item appears in the expert rubric, added an item-level mapping between the two instruments, and report expert scores separately for overlapping and non-overlapping items.

  • Protocol details: we added the full baseline configuration (model versions, tool access, attempts, feedback, compute) and a per-case-study specification of cohorts, predictors, labels and prevalence.

  • Clarifications: we revised the contribution statement, positioned MAGMA-Suite against existing agentic benchmarks, and state the absence of judge validation experiments as a limitation.

[2/2]

As it pertains to the ethics flag raised by AoBj, the reviewer flagged the submission as fully AI-generated on the basis of the reporting inconsistencies, an unfinished sentence in Section 4.3, and a superseded metric name in Figure 1. We fully reject this claim. All three are human errors that are resultant failures of verification, and while we do not defend them, they are not evidence of fabricated results. Every corrected value derives from a per-run record of logged experiments, and each inconsistency is explained by the single uncorrected propagation described above. Our AI Use Statement (p. 9) disclosed at submission that generative AI was used to assist with editing the text, consistent with the ICLR policy for authors. It was not used to produce results. We respectfully ask that the flag be assessed against this record. We provide further details in our point-by-point responses and have incorporated all additions into the revised manuscript. Thank you again for your time in reviewing our submission.

Sincerely,

The Authors of Submission #54176

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.