- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper frames multimodal MDD pipeline development as recursive self-improvement (RSI). GSC builds a checked data adapter that keeps subject–label alignment fixed across feature recipes; CPE enumerates and evaluates combinations of feature recipes (sources, aggregation, fusion) and scikit-learn predictors; EGE uses multiple LLM roles (Planner/Engineer/Reviewer/Chair/Proposer) to review results, propose new components, and replace the incumbent pipeline only when the gain exceeds a bootstrap-based margin. Experiments on E-DAIC, MPDD-2025/2026 and two supplementary datasets (Mental Health Multimodal, Kaggle TESS/RAVDESS) report gains over DL (MulT, Flex-MoE, MCMoE) and LLM (Gemini 2.5 Flash, MDAgents) baselines.
Critical strengths
- Strong results on several MDD benchmarks. MERID clearly outperforms the reported DL and LLM baselines on E-DAIC (binary Macro-F1 0.68/0.71 vs. 0.60 for Flex-MoE; PHQ-8 CCC 0.33 vs. 0.05) and MPDD-2026 Young (PHQ-9 CCC 0.61 vs. 0.08), with consistent results across two LLM backbones (Opus 4.8, GPT-6 Astra). Gains are largest on severity estimation, where the baselines largely fail.
- Transparency. The appendix discloses prompts, run records, per-seed results, and negative findings (e.g., the preregistered RSI test, baseline prediction collapse), which makes the claims checkable.
Critical weaknesses
- LLM-assisted AutoML rather than RSI. What is improved is the configuration of traditional ML pipelines (feature recipes + scikit-learn predictors), while LLM weights, role definitions and protocol stay fixed (§3); this is close to well-known AutoML / agent-based pipeline search (Auto-WEKA, MFAS, DS-Agent, AutoML-Agent, all cited by the authors). The evidence for "recursive" improvement is also weak: main runs use a single revision round (T=1, App. B.5), the review arms do not beat random revision in the eight-round study (−0.0122 [−0.0346, 0.0003], App. C.2.1), and larger budgets/more rounds give no consistent gain.
- The core module (EGE) is under-specified. EGE is the main technical contribution, yet there is no algorithm box or complete workflow; its description is scattered across §3.4 and App. A/H. The only worked example (Fig. 7) raises the dev score while the test score is unchanged, so the novelty of the agent design is hard to assess.
- Missing ML/AutoML baselines. Only DL and LLM baselines are reported. Classical LR/SVM/Ridge/SVR baselines are described as implemented (App. F.2/F.3) but not reported in Table 1 or Table 21, and there is no same-budget AutoML comparison (grid/random search, Optuna) on the same features. Given the small cohorts, conventional ML may outperform DL regardless of the agent; indeed, the DL baselines collapse to majority-class predictions on E-DAIC and MPDD-2026 Elder (App. E).
- Ablation does not support the module contributions. Removing EGE (or CPE) improves results on Mental Health and MPDD-2025 Elder binary/quinary (Table 2). Moreover, the full model uses 5 seeds with an inherited pipeline while the removals use 3 seeds without one (§4.3), so the differences cannot be attributed to the modules.
- Limited evaluation reliability. MPDD-2026 test tracks contain only 22–23 subjects; several MPDD-2026 tasks have SD = 0.0000 across seeds (App. B.1), suggesting carried-over heads rather than search; two of the six Table 1 datasets are not MDD tasks (stress classification with a row-level split; acted-emotion proxies) yet enter the average rank; test predictions ensemble up to five candidates (§4.1), unlike most baselines.
Questions for authors
- Please report the classical baselines from App. F.2/F.3, and a same-budget grid/random search (or existing AutoML tool) over the same features and scikit-learn search space. How much of the gain remains attributable to the agent?
- Can you provide an ablation with matched seeds and initialization (inherited pipeline or not), and explain why removing EGE improves several tasks?
- In the main runs, how often did a REVISE round actually replace the incumbent, and did those replacements improve test performance?
3 comments
Thank you for the detailed response and additional experiments. However, the new results reinforce my main concern. Same-space random search matches MERID on average across the 13 tasks (0.552 vs. 0.549). The one large gap, on E-DAIC binary, is attributed by your own analysis to the inherited reference and the 1-SE margin, not to the LLM-driven search. The search baselines run without that reference. DS-Agent also reaches a comparable or higher score at similar cost. Combined with the low replacement rate (11 of 65 runs) and no consistent gains from additional rounds, the evidence for agent-driven recursive improvement remains limited. I will keep my score.
Thank you for the follow-up and engaging with our response. We answer each sentence below, and the answers share one point. Our introduction states that "a higher development score does not necessarily justify retaining a revision". MERID addresses this reliability problem. The new results (Appendices B.5 and I of the revision) show that random search and DS-Agent are unstable where MERID is stable and robust.
"Same-space random search matches MERID on average across the 13 tasks (0.552 vs. 0.549)."
We clarify that this average hides how strongly random search depends on its budget and its draw, and that it searches the space MERID's agents wrote (Q1 A1). We extended it to five budgets on E-DAIC binary (mean ± SD) on 100 runs:
| Method | Development score | Test Macro-F1 | Range |
|---|---|---|---|
| Random, 8 pairs | 0.616 | 0.592 ± 0.072 | 0.434–0.721 |
| Random, 16 pairs | 0.649 | 0.601 ± 0.084 | 0.417–0.730 |
| Random, 32 pairs | 0.681 | 0.580 ± 0.091 | 0.417–0.738 |
| Random, 64 pairs | 0.703 | 0.542 ± 0.090 | 0.417–0.738 |
| Random, 128 pairs | 0.732 | 0.483 ± 0.057 | 0.449–0.678 |
| Grid, all pairs | 0.736 | 0.459 | – |
| MERID (Opus 4.8) | 0.717 | 0.677 ± 0.015 | 0.665–0.694 |
On this task the largest budgets have the highest development scores and the lowest test scores, so the budget cannot be set from development data. Over the 13 tasks the mean of one run has an SD of 0.013, against 0.005 for MERID.
"The one large gap, on E-DAIC binary, is attributed by your own analysis to the inherited reference and the 1-SE margin, not to the LLM-driven search."
We see this attribution as support for MERID. The inherited reference and the margin are the recursion it proposes. RSI is "an accepted and inherited state change" [1], where the reference is the inherited state and the margin is the acceptance step. Our response (Q2 A2) shows that inheritance without acceptance is not enough, since a margin of 0 lowers E-DAIC binary from 0.684 to 0.495.
"The search baselines run without that reference."
We would like to point out that our response already reports the search with that reference. "MERID w/o EGE" in its Q2 table starts from the same inherited reference and selects by development score alone. It reaches 0.459 on E-DAIC binary, so giving the search the reference does not close the gap.
"DS-Agent also reaches a comparable or higher score at similar cost."
With 15 runs per system, DS-Agent is not higher than MERID, and MERID is far more stable and robust. The number of added runs was fixed in advance, and all 15 are reported (mean ± SD):
| Method | E-DAIC binary (Macro-F1) | MPDD-2025 Elder 1s ternary (official score) | LLM calls / cost (E-DAIC) |
|---|---|---|---|
| DS-Agent | 0.614 ± 0.099 | 0.629 ± 0.064 | 20.5 / $0.77 |
| MERID (Opus 4.8) | 0.681 ± 0.014 | 0.661 ± 0.046 | 10.6 / $0.58 |
On E-DAIC binary MERID's 15 seeds stay within 0.665–0.694, and its SD is seven times smaller than DS-Agent's. DS-Agent's runs range from 0.411 to 0.767, and its first five averaged 0.695. It also needs 1.9 times the LLM calls and ran on features that MERID's adapter aligned.
"Combined with the low replacement rate (11 of 65 runs) and no consistent gains from additional rounds, the evidence for agent-driven recursive improvement remains limited."
We respectfully read the low replacement rate differently. It is the acceptance rule at work. The agents proposed all 11 replacements, and 8 of them raised the test score (Q3 A3), so the rounds that change the pipeline improve it in most cases. Rounds without a verified gain keep the pipeline. DS-Agent overwrites its script in every iteration without such a check, and its E-DAIC binary test score moves by 0.21 within one run on average. In MERID the agents propose every change and a change is inherited only after it is verified, which is why its results stay stable.
Thank you again for your time and comments. We hope these responses address your concerns, and we would be grateful if you could reconsider your score.
References
[1] Y. Duan et al. The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement. arXiv:2609.11873, 2026.
We are truly grateful for the time you have taken to review our paper and your insightful review. Here we address your comments in the following.
A1. We clarify that MERID is RSI as the literature defines it.
A2. Each EGE round: the Planner, Engineer and Reviewer analyze the results, the Chair keeps the pipeline or asks for a revision, the Proposer adds candidates, and a candidate replaces the incumbent only if its development gain exceeds the incumbent's SE. In main-table runs, 8 of 11 replacements raised test scores. We will add it as an algorithm box.
A3. We added them. Classical baselines (App. F, development-selected) are below MERID on 12 of 15 tasks. Same-space search, no LLM or reference (mean test scores):
On E-DAIC binary they are 0.085–0.217 lower; on average they match MERID.
A4. We re-ran the ablation with matched seeds and initial state: removing EGE costs 0.217 on E-DAIC binary and 0.067 on MPDD-2025 Elder 5s ternary. Mental Health now favors the full system (+0.013); Table 2's reversal came from the mismatch. Removing CPE barely changes scores but raises LLM calls per run from 11.8 to 48.9.
A5.
A1. The agent builds the space that same-space search reuses, and MERID's margin avoids E-DAIC binary's collapse (0.677 vs 0.459). The released AutoML-Agent and DS-Agent code (MERID's LLM, features, splits) reach 0.597 and 0.695 on E-DAIC binary (MERID 0.677) at $8.21 and $0.78 per run (MERID $0.60); all paired intervals include 0.
A2. Re-run with matched seeds (0–4) and initial state (mean test scores; Elder: MPDD-2025):
A3. 11 times in 65 main-table runs; 8 raised the test score (mean +0.015).
In light of these responses, we hope we have addressed your concerns, and hope you will consider raising your score. If there are any additional notable points of concern that we have not yet addressed, please do not hesitate to share them, and we will promptly attend to those points.
References
[1] Y. Duan et al. The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement. arXiv:2609.11873, 2026.
[2] J. Zhang et al. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents. arXiv:2505.22954, 2025.