- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes MAGMA, a multi-agent framework for automated clinical machine learning, featuring judge-driven iterative refinement and a guideline-based evaluation metric. However, I have serious concerns about the quality and reliability of this submission. The manuscript contains numerous inconsistencies, unfinished text, and basic errors. These problems raise concerns that this is a fully AI-generated paper. In its current form, I do not believe this paper meets the requirements of ICLR and should be rejected.
Critical strengths
N.A.
Critical weaknesses
-
The manuscript contains serious inconsistencies in experimental results. For example, Table S13 reports 552 total runs, although the individual system counts sum to 576. Its completed-run total is also incorrect (139 instead of 162). Furthermore, the HealthFlow results in Tables S11 and S16 are inconsistent. These errors directly undermine confidence in the reported results.
-
The manuscript contains obvious unfinished text. For example, Section 4.3 ends a paragraph with "It is worth noting that..." without completing the sentence.
-
The main methodological figure is inconsistent with the text. Figure 1 uses "TRIPODScore" in the refinement formula, whereas the corresponding equation and method description use "MAGMAScore". This inconsistency appears in the central contribution of the paper and suggests insufficient verification of the manuscript.
-
The manuscript contains additional basic reporting inconsistencies. For example, Figure 2 reports a 100% completion rate for HealthFlow, whereas Table S10 reports 47 completed runs out of 48 (97.9%). The presence of such easily verifiable discrepancies further raises concerns about the overall quality control.
Questions for authors
See weakness.
2 comments
[2/2]
The manuscript contains obvious unfinished text. For example, Section 4.3 ends a paragraph with "It is worth noting that..." without completing the sentence.
The reviewer is correct, the paragraph in section 4.3 ends mid-sentence, and that is an editing error from our side that will be removed.
The main methodological figure is inconsistent with the text. Figure 1 uses "TRIPODScore" in the refinement formula, whereas the corresponding equation and method description use "MAGMAScore". This inconsistency appears in the central contribution of the paper and suggests insufficient verification of the manuscript.
The reviewer is correct. The metric was renamed from TRIPODScore to MAGMAScore throughout the process of building MAGMA and figure 1 was not updated. The formula shown is otherwise identical to the formula used in our system. The figure will be corrected.
Given the weakness, I would like to flag this paper as a fully AI-generated paper and should be desk rejected.
We take this flag seriously, and respond by noting that our AI Use Statement (p. 9) discloses that generative AI tools were used to assist with editing the text of this paper, and that all such text was reviewed and revised by the authors, who take responsibility for the final content. That disclosure was made at submission and remains accurate. ICLR’s policy for authors requires disclosure of such use and does not prohibit it; the Code of Ethics concern it identifies is a substantial falsehood, instance of plagiarism, or misrepresentation.
The manuscript was authored by the authors, and generative AI was used for text editing as disclosed, not to produce results, and the inconsistencies identified by the reviewer are stale aggregates arising from a correction applied inconsistently across tables, not fabricated or unverifiable results.
We do not dispute the reviewer’s broader point about quality control. The manuscript was submitted with an unfinished sentence, a figure carrying a superseded metric name, and a data correction propagated to two tables out of six. We are grateful they were caught, and they will be fully corrected.
[1/n] We thank the reviewer for their thorough comments. We also address the ethics flag directly and fully reject the accusation that our paper was AI generated.
We have addressed each of the soundness concerns raised:
We hope these revisions resolve the reviewer’s concerns about soundness. Each revision is described in more detail below
Thank you for identifying the discrepancies. All of them derive from one per-run record of 576 logged baseline runs (480 single-agent, 48 HealthFlow, 48 MLZero), excluding the 96 CliMB-DC and HACHI attempts that produced no classifiable terminal outcome.
All the discrepancies identified trace back to a correction to the HealthFlow results that was applied to Tables S10 and S11 and was not propagated to the remaining HealthFlow relevant cells in Figure 2, Table S16, and Table S13. The values requiring correction are as follows:
Tables S9, S10, and S11 are correct as printed in the paper. Table S9 already carries the corrected HealthFlow row (97.9%, 0.819 ± 0.105 conditional, 0.802 ± 0.157 completion-weighted).
As it relates to CS2 and Healthflow’s results, the reviewer makes a correct comment, whereas completion-weighting assigns zero to failed runs, so for a cell with no failures it is the identity, and the conditional and weighted values must coincide. HealthFlow completed 12/12 on CS2, so Table S11’s 0.924 is correct. The same holds for CS1 and CS3. CS4 is the one HealthFlow cell where the two differ, 11/12 completed, giving 0.845 conditional against 0.774 weighted, and in the corrected tables that is the only such inconsistency.
With the updated values, MAGMA holds the highest completion rate and the higher completion-weighted AUROC (0.808 vs 0.802), with the caveat already stated in the paper that HealthFlow is evaluated on case studies 1 - 4 only and hence the two are not directly comparable. Those inconsistencies do not change the claim that MAGMA leads completion-weighted AUROC in 9 of 10 case studies, and is outperformed only on case study 2, by HealthFlow.