acceptodds
Under review as a conference paper at ICLR 2027

MELD: Recursive Self-Improvement via Hierarchical Coordination for Multi-Agent Scientific Discovery

99 views

Abstract

Multi-agent scientific discovery with large language models raises a coupled learning problem: research assignments determine both which experiments are pursued and the experiences from which agents adapt. We introduce MELD, a unified hierarchical test-time training framework that jointly adapts a team-level Coordinator policy and agent-level individual Scientist policies within a single discovery process. The Coordinator generates a joint portfolio of search branches and research directives over a shared derivation tree, while Scientists develop hypotheses and execution plans implemented by a shared frozen Executor. Both levels receive alternating GRPO-style parameter updates under a shared smooth best-of-budget objective, allowing team organization and individual research behavior to co-evolve in a recursive self-improvement loop. Our analysis establishes objective alignment for the hierarchy, characterizes conditions for coordination gains over independent parallel discovery and the first-order benefit of private adaptation, and derives a finite-budget regret bound for adapted models. On 22 MLGym and MLE-bench tasks, MELD is best on 20 of them and achieves the highest aggregate normalized scores on MLGym, using a 4B backbone for coordination and research. Most baselines use proprietary Scientists, and all systems share one frozen Executor.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Recursive Self-Improving, Multi-Agent System, Test-Time Training, Scientific Discovery

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_C9zeOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MELD adapts a hierarchical scientific-discovery system during search. A coordinator assigns a joint portfolio of research directions to scientists with private LoRA adapters, while a frozen executor implements their plans and a shared tree records evaluated candidates. The method alternates coordinator and scientist updates using groups of rollouts from matched snapshots and a smooth potential related to the best observed score. The paper analyzes objective alignment, coordination, specialization, and convergence under explicit assumptions. Experiments on 12 MLGym and 10 MLE-bench tasks report the best mean score on 20 of 22 tasks, with three seeds and matched scored-candidate budgets. A smaller ablation compares adaptation of both levels with coordinator-only, scientist-only, and frozen variants.

Critical strengths
  1. The method addresses an important credit-assignment problem in adaptive research agents: the allocation of research directions and the generation of useful hypotheses can both improve during a run. Matched-snapshot groups make comparisons between sampled portfolios more interpretable, and the separation of private scientist adapters gives the system a concrete mechanism for specialization. The analysis is careful to distinguish the exact sequence-score argument from the biased, one-round implementation surrogate; this distinction makes the theoretical claims easier to assess.
  2. The empirical evaluation is broader than a single application, with a common frozen executor and private evaluators across methods. ARTS* uses the same Qwen3-4B backbone, providing a relevant comparison beyond the frozen Haiku-based baselines. The control-task improvements over ARTS* are substantial, including 93.51 versus 84.53 on MountainCar and 84.97 versus 78.46 on Breakout. The fresh post-update candidate batch in the adaptation ablation also gives useful evidence that enabling updates changes subsequent candidate generation, beyond merely retaining a previously discovered best candidate.
Critical weaknesses

The experiments do not yet isolate the distinctive coordination and specialization mechanisms. The adaptation ablation covers two game tasks and one coordinator–scientist update cycle. It establishes a benefit from enabling both updates in that setting, but does not compare private scientist adapters with a shared adapter, or joint portfolio assignment with independent assignments under matched resources. The N=1 result also changes the number of research units. Consequently, the stronger claims about complementary research directions and private-adapter specialization are supported more by conditional analysis than by direct empirical tests, especially on the control and ML tasks with the largest gains.

Questions for authors
  1. Are the selected artifacts evaluated once on any examples, environment seeds, or opponents that supplied no feedback during search? Please distinguish the data used for policy updates, artifact selection, and final reporting. If there is no independent final evaluation, can the claims be limited accordingly?
  2. Can a matched-resource shared-adapter control and an independent-assignment control be reported on one or two existing tasks with clear gains? These would help determine whether the benefit comes specifically from private specialization and joint coordination.
Reviewer_PLzkOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

MELD introduces a hierarchical test-time training framework for multi-agent scientific discovery, comprising a learned Coordinator, three Scientists with private LoRA adapters, and a frozen Claude Haiku 4.5 Executor. Coordinator and Scientists are trained through alternating GRPO-style updates using a smooth best-of-budget potential, while the Executor implements their research plans.

Evaluation on 22 tasks from MLGym and MLE-bench across three seeds shows that MELD achieves the best results on 20 tasks, with mean and median normalized scores of 0.745 and 0.753, respectively, outperforming the closest matched baseline, ARTS*. The paper also provides theoretical analysis of its potential-based objective, the value of coordination, and convergence, alongside an ablation study of PD/CB adaptation.

Critical strengths

Jointly adapting the team-level assignment policy and the individual research policies is a natural question, and the paper frames it clearly. The evaluation is nice performed. Test labels are held by private evaluators, candidates are matched by scored evaluations rather than wall-clock time, failed calls consume budget, and the aggregate metrics come with bootstrap intervals and a clear normalization (Appendix). The ARTS* comparison is on the same backbone, which is the right baseline for testing the hierarchy.

Evaluation on 22 tasks from MLGym and MLE-bench across three seeds shows that MELD achieves the best results on 20 tasks, with mean and median normalized scores of 0.745 and 0.753,

Critical weaknesses

Whether MELD’s experiments support its claims of recursive self-improvement and reliable performance gains. The ablation tests only one Coordinator–Scientist update cycle, leaving the benefits of repeated adaptation unverified. Moreover, selecting the best candidate using repeated feedback from the same held-out evaluation data risks optimistic performance estimates. Comparisons with most baselines are confounded by differences in Scientist backbone capacity, while many reported gains over ARTS* are marginal and lack task-level variance estimates or paired significance tests.

The PD/CB adaptation ablation is also based on only three seeds, with no uncertainty analysis. The theoretical results rely on strong assumptions that are not verified for LLM adapter training, and key quantities in the convergence bound remain unmeasured. Similarly, the value-of-coordination claim is not directly tested through portfolio-diversity analysis or a matched independent-assignment baseline.

Unclear candidate budget and cycle count, insufficient justification of normalization anchors should be further explained. Overall, the paper presents a promising framework, but stronger multi-cycle experiments, cleaner evaluation protocols, controlled baseline comparisons, statistical testing, and implementation transparency are needed to substantiate its central claims.

Questions for authors

The paper would be substantially strengthened by addressing the following experimental and reproducibility questions.

Can MELD be evaluated over at least two or three Coordinator–Scientist cycles to establish whether performance gains continue with repeated adaptation?

clearly describe final test split, select candidates using validation data only, and quantify how this changes the reported scores?

Provide candidate budget and cycle count, insufficient justification of normalization anchors should be further explained. Can CORAL, AIRA, and MLEvolve be reimplemented using the same Qwen3-4B backbone to isolate the benefits of MELD’s hierarchical architecture, or explain why such comparisons are infeasible?

Report per-task candidate budgets and cycle counts, clarify what “candidates spent (k)” represents in Figure 1, and provide per-task standard deviations and paired statistical tests against ARTS*.

Furthermore, estimate the bias and snapshot-mismatch termsor provide empirical evidence that their effects are small?

To further validate the theoretical value of coordination, portfolio diversity and negative association and compare MELD against a matched independent-assignment Coordinator should be included.

Reviewer_2wnSOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies multi-agent LLM-driven scientific discovery on a single previously unseen executable ML task, and argues that prior work either adapts one task-local research policy (TTT-Discover, ARTS) or coordinates frozen agents through shared memory (CORAL), but never learns the team-level assignment policy and the agent-level research policies together inside one discovery run.

The authors propose MELD. A frozen Qwen3-4B-Instruct-2507 backbone carries four LoRA adapters (r=32, alpha=64, 66M parameters each): one Coordinator adapter and three private Scientist adapters. Each round the Coordinator emits one schema-constrained JSON completion assigning each of N=3 units a parent node from a retrieved shortlist of at most 16 tree nodes plus an abstract research directive; each Scientist turns its assignment into a hypothesis and an execution plan; a shared frozen Claude Haiku 4.5 Executor implements the plan; a private evaluator scores it. Both policy levels are updated during the run with alternating GRPO-style steps on a shared reward, the increment of a log-sum-exp potential Phi_{tau,K} over derivation-tree node utilities, with tau=0.05, G=8 matched-snapshot forks per group (24 scored candidates), leave-one-out advantages, asymmetric clipping [0.8, 1.28] and KL 0.01 to the task-start policy. Adapters reset at task start; nothing transfers across tasks.

Section 5 gives four results: both levels' exact gradients are partial gradients of one objective approximating best-of-budget utility up to tau*log(K+1); a "value of coordination at matched marginals" theorem via the Meilijson-Nadas bound and negative association; a first-order argument that private adapters retain a gradient-dispersion term a shared adapter averages away; and a finite-budget discovery-regret bound decaying geometrically under a Polyak-Lojasiewicz assumption.

Experiments cover 12 MLGym and 10 MLE-bench tasks, three seeds each, 4x A100, about 9 hours per run. AIRA, MLEvolve and CORAL are re-implemented in the authors' harness with frozen Claude Haiku 4.5 as their Scientist, and ARTS* is re-run on MELD's own backbone; Table 1 states that all baseline values are the authors' same-harness re-runs. MELD reports the best point estimate on 20 of 22 tasks and MLGym aggregates of Median 0.753, IQM 0.748, Mean 0.745 and Optimality Gap 0.255 against 0.669, 0.684, 0.694 and 0.306 for ARTS*. An adaptation ablation on Iterated Prisoner's Dilemma and Colonel Blotto compares full MELD against Coordinator-only, Scientist-only and Frozen. The supplementary material contains the implementation only; it ships no run logs, event traces or per-seed results. The contribution is primarily systems and empirical, with a supporting theoretical framing.

Critical strengths

Strength 1: The paper's arithmetic is reproducible, its appendix documents the limits of its own theory more precisely than most submissions document their strengths, and the code release is complete enough that every claim I checked could be verified or refuted. Evidence: All 60 Table 5 cells and all four Figure 2 statistics for all five systems reproduce from Tables 1 and 2 to within rounding. A.3.1 derives exactly where the implemented surrogate departs from the Eq. (5) gradient, including the length-normalization bias term and the fact that dividing by the within-group standard deviation makes the estimator a data-dependent rescaling not covered by part (iv). Remark A.8 gives a numeric case where the best-of-round gain is positive while the potential gain is negative. Remark A.15 states that dispersion does not lower-bound the PL constant. The supplement ships 3,026 lines across 16 files with 25 unit tests, checkpoint and resume, an event log, and mix_update_frozen / worker_update_frozen events that let a reader verify post hoc which ablation arm ran. Why it is consequential: This is what makes the paper assessable, and it is why my criticisms are specific and quantified rather than speculative. It also means the gap between the formalized and the implemented system is recoverable by rewriting Sections 4.1 and 4.3, since the authors plainly know what their code does. A submission that had hidden these caveats, or shipped no code, would be indefensible rather than revisable.

Strength 2: The ablation protocol is designed correctly in principle, and on a minority of tasks MELD's raw scores do sit above the best published values, so there may be a real effect worth isolating. Evidence: B.5 reserves a fresh 24-candidate post-cycle batch, holds policy parameters, discovery state and parent shortlist fixed throughout its collection, excludes it from training, and runs a Frozen arm that keeps the tree search so that weight adaptation is the only varied factor; the arms are independently verifiable from the event log. On raw scores, Breakout 84.97 sits about 1.9 SE above the published ARTS*(4B) 75.95 ± 4.83 and about 0.4 SE above the best published value for that task, MLEvolve 83.28 ± 3.84; RSNA Brain Tumor 0.7129 sits about 1.9 SE above the best published value, ARTS(o3) 0.673 ± .021, and further above AIRA 0.649 ± .014 and MLEvolve 0.656 ± .011; Nomad 2018 0.0501 is below every published value (AIRA 0.062 ± .001) though the differing baseline anchors (0.1086 here against 1.000 in ARTS) make that one hard to read. Why it is consequential: It means the right revision is not a different paper but a much better-instrumented version of this one: the protocol exists, the harness exists, and two or three tasks show a signal that survives comparison against published numbers. If the authors applied the B.5 design to those tasks at adequate seeds, and reported their re-runs next to the published values, the contribution would be assessable on its merits.

Critical weaknesses

Weakness 1: The algorithm that ran differs from the algorithm that was formalized in three specific ways, and each difference removes a property the theory claims rather than merely omitting a detail. Evidence: (a) Fork independence. Section 4.3 says a group "contains G >= 2 independent forks of one snapshot" and Proposition 5.2(iv) claims E[A_hat_g * grad log pi_beta(a_g|x)] = grad J_beta_loc(S) at Theta_old, justified in the text "by conditional independence". The released Coordinator sampling is not independent across forks: mix_diversity_fix=True is the default (config.py:53) and sample_mix_portfolios_batch (trainer.py:334-366) runs up to mix_dedup_max_rounds=2 rounds that replace any fork whose parent-id tuple duplicates an already-seen tuple. The eight forks are therefore draws conditioned on mutual distinctness. That is not the per-response accept-conditioning analyzed in A.3.1, which concerns a judge accepting or rejecting one response; it is a joint constraint across the group, and it is applied in service of precisely the negative association that Theorem 5.3 presents as something coordination should earn through the reward, while Section 4.2 states that assignments are evaluated "through realized candidate utilities, not an explicit diversity bonus". (b) Update count. Table 3 states "Optimizer steps per group: 1" and B.3 repeats that each group is followed by one step, but trainer.py:547 and :610 skip the step entirely when is_collapsed(utilities, 0.02) fires, so the realized M is data-dependent and generally not the 2C that Theorem 5.6 is written in. (c) The action. Eq. (2) and Section 4.1 define the Scientist's action as one (hypothesis, execution plan) pair and Eq. (6) differentiates that action's log-probability. The implementation has the Scientist emit eight hypotheses with self-reported probabilities and selects one by random.choices at trainer.py:410, then trains on the log-probability of the entire eight-item JSON. The differentiated object and the rewarded object are not the same action, the selection factor is nowhere in Eq. (5) or A.3.1, and the eight-fold expansion of per-round exploration is an unreported confound for the N=1 result in Section 6.3, which is the paper's only evidence that multiple units matter. Two smaller mismatches: config.py:15 hardcodes the potential's root term as b0=0.5 whereas Eq. (4) and Lemma A.1 say the root carries s_0 = s(c_0), which matters on Titanic where Table 2's own parenthetical gives a re-measured baseline of 0.943 against a declared 0.766, implying s_0 near 0.94 rather than 0.5; and config.py:48 sets allow_off_shortlist_parents=True, permitting parents outside the 16-node shortlist that B.3 describes as the Coordinator's action space. Why it is consequential: Proposition 5.2 and Theorem 5.6 are the paper's stated justification for treating two separately updated adapters as one optimizing process, and contribution bullet 2 claims a theory of coordination and private adaptation. If the forks are not independent, the unbiasedness claim does not hold for the run that produced Table 1. If the step count is data-dependent, the regret bound's geometric factor is not a function of anything the authors can report. If the trained action is an eight-item list rather than a plan, Section 5 describes a different algorithm than the one that was measured. These are not cosmetic; together they mean the theory is not about the system. What would resolve it: Restate Sections 4.1 and 4.3 to describe what ran, including vs_k and the weighted selection rule, the distinctness resampling and the collapse skip, and add vs_k and per-task K to Table 3. Then either disable mix_diversity_fix and the collapse guard and re-run at least the ablation so the reported numbers come from the process Proposition 5.2(iv) covers, or weaken part (iv) to an approximation and state the bias the distinctness conditioning introduces, in the same spirit A.3.1 already handles length normalization. Add one arm with vs_k=1 and everything else fixed; if the gains survive it the paper's story strengthens, and if they do not, the eight-way proposal is the actual mechanism and should be the contribution.

Weakness 2: The headline empirical claim is measured against the authors' own re-implementations, which diverge from the published results of the same methods by tens of standard errors in both directions, and no run artifacts are released that would let anyone check which measurement is right. Evidence: (a) Re-run fidelity. ARTS (arXiv:2606.21891v2) Table 2 reports ARTS*(4B), the same method MELD re-runs, on the same 22 tasks, as mean ± standard error over 3 runs. MELD's re-run of that method differs from the published values by: Titanic 0.952 vs 0.998 ± .002 (-23 SE); House Prices 0.931 vs 0.940 ± .001 (-9 SE); MountainCar 84.53 vs 91.94 ± 2.83 (-2.6 SE); MetaMaze 75.41 vs 53.00 ± .57 (+39 SE); Colonel Blotto 0.252 vs 0.344 ± .33; Language Modeling 3.672 vs 3.518 ± .276. MELD's re-runs of AIRA and MLEvolve show the same pattern against ARTS's Table 1, for example HMS Brain AIRA 0.7389 vs published 0.550 ± .020 and MLEvolve 0.7286 vs published 0.583 ± .013, and APTOS AIRA 0.8624 vs published 0.922 ± .006. (b) Effect on the headline. MELD's largest margin, MountainCar +8.98, becomes +1.57 (about 0.55 SE) against the published ARTS*(4B) of 91.94 ± 2.83. Recomputing the MLGym aggregate with my validated normalization but substituting published ARTS*(4B) for the re-run, MELD's IQM lead drops from +0.064 to +0.016; using MLGym's own baseline_scores and ARTS's Table 3 anchors as well, the lead over published ARTS*(4B) becomes -0.018 IQM. (c) Anchors. Table 2's anchors do not match the upstream benchmark on five of twelve MLGym tasks. I fetched configs/tasks/.yaml from facebookresearch/MLGym: Fashion-MNIST official 0.7835 against 0.848 here, Language Modeling official 4.672731 against 4.238, MountainCar official 33.7938 against 53.26, Breakout official 48.8172 against 41.93, MetaMaze official 15.7344 against 20.12. B.2 justifies re-measurement for the three control tasks on jax/flax grounds, which is reasonable, but offers no explanation for Fashion-MNIST or Language Modeling. Human-best anchors also differ from ARTS's Table 3 on three tasks (Language Modeling 3.280 vs 3.500, MountainCar 100.0 vs 99.00, MetaMaze 100.0 vs 52.50), and the MetaMaze change is load-bearing: under the 15.73/52.50 anchors MELD's 81.19 and its re-run's 75.41 both clip to z = 1.0 so the task contributes nothing, while under this paper's 20.12/100.0 anchors it contributes +0.072, one of the larger per-task terms in Figure 2. To be fair, the anchor changes are not uniformly self-serving: on Language Modeling this paper's anchors give MELD z = 0.600 where the upstream anchors would give 0.861. (d) Degenerate and saturated rows. On Titanic every system's raw z exceeds 2.4 and clips to 1.0, which B.6 discloses without explaining; Table 2's own re-measured baseline of 0.943 already exceeds the 0.830 human-best anchor, so the anchor sits below the starting solution. On Colonel Blotto all five systems fall in 0.252 to 0.254, while MLGym's own Table 5 (Best Attempt@4) reports -0.248 for the baseline and 0.043, 0.047, 0.576, 0.249 and 0.248 for the five frontier models it evaluates, so the task discriminates strongly upstream and this paper's 0.500 anchor is below a score MLGym published in 2025. Recomputing from Tables 1 and 2, five of twelve MLGym tasks have |MELD - ARTS| <= 0.01 normalized, and the raw margins are MNLI +0.01 accuracy points, Language Modeling +0.002, Battle of the Sexes +0.001, Prisoner's Dilemma +0.002, Colonel Blotto +0.001 and Jigsaw +0.0002 AUC. (e) No dispersion, no artifacts. Table 1 and Table 4 report three-seed means with no standard deviation, standard error or interval anywhere, although the closest prior work reports ± SE for every cell of the same table. The supplement contains source code only: no events.jsonl, no report.json, no per-seed scores. (f) Budget units. B.7 matches systems on scored candidates, but config.py:67 sets executor_max_validates=3, so one scored candidate may consume up to three private-evaluator scorings, and executor.py:37 and :83-85 state that run_python probes are free; Section 3 defines K as the evaluator-call budget. (g) Undisclosed scaffolding bearing on the framing. Trainer.setup() (trainer.py:212-231) makes three Claude Haiku 4.5 calls labelled "baseline audit", "data audit" and "metric audit", concatenates up to 4000 characters and injects it into the Coordinator's state as "TASK AUDIT NOTES" (state.py:29-30); the metric-audit prompt is handed baseline_score and human_best_score, and state.py:26 shows human_best to the Coordinator every round while task.py:32 uses the same value as the reward-normalization denominator. The string "audit" does not appear in the PDF, B.4 lists what the baselines share as "the shared Executor, evaluator, tree bookkeeping, and judge" without mentioning audit notes, and Section 6.1 frames the comparison as testing "whether task-time adaptation lets a compact open-weight Scientist compete with frozen proprietary Scientists". Why it is consequential: Contribution bullet 3 and the abstract rest on "best on 20 of 22". A reader will take that as a claim about standing against the literature, and Table 1 is the only evidence offered for it. But Table 1 measures MELD against re-implementations that differ from the published methods by up to 23 SE downward and 39 SE upward, so it measures MELD against MELD's harness. On that evidence the paper cannot separate "the hierarchy helps" from "this harness favors this system". The aggregate statistics inherit the problem: 62% of MELD's mean advantage over its re-run of ARTS* comes from MountainCar, Breakout and MetaMaze, and MetaMaze's contribution exists only under this paper's anchor choice. Because no run artifacts ship, no reader or AC can audit any of it. This is the difference between an incremental-but-honest systems paper and one whose central claim is unverifiable. What would resolve it: Report, per task and per system, the mean and standard error over at least ten seeds, and place MELD's re-run of each baseline next to that baseline's own published value with an explanation of every gap larger than about 2 SE; where the gap is large, the published value or a corrected re-run should be the comparator. Restore the upstream anchors for Fashion-MNIST and Language Modeling or justify the substitutions, and report Figure 2 under both this paper's anchors and MLGym's so the MetaMaze and Titanic sensitivity is visible. Drop or re-anchor Titanic, and note in B.6 that Optimality Gap is 1 - clipped Mean and therefore not a fourth independent statistic. Extend Table 4 to Breakout, RSNA and one control task with per-seed counts. Ship the events.jsonl and report.json for at least one seed per task, since the code already writes them. State whether the baselines received the audit notes and the human-best value; if they did not, either give them the same information or remove it from MELD and re-run, because Section 6.1's framing depends on the answer. Finally, report evaluator calls and wall-clock per system alongside scored candidates, or set executor_max_validates=1 so the matched unit is the one Section 3 defines.

Questions for authors

(Re: W2) [Potential correction] ARTS's Table 2 reports ARTS*(4B) on these same 22 tasks as mean ± SE over 3 runs. Your re-run of that method differs from it by -23 SE on Titanic (0.952 vs 0.998 ± .002), -9 SE on House Prices (0.931 vs 0.940 ± .001), -2.6 SE on MountainCar (84.53 vs 91.94 ± 2.83) and +39 SE on MetaMaze (75.41 vs 53.00 ± .57). Which differences do you attribute to the executor swap from Gemini 3 Flash to Claude Haiku 4.5, which to matching on scored candidates rather than wall-clock, and which to genuine re-implementation choices? Given that your MountainCar margin over your re-run is +8.98 while the margin over the published ARTS*(4B) is +1.57 against a ± 2.83 standard error, on what basis should the AC read "best on 20 of 22" as a claim about standing against the literature rather than about standing against your harness?

(Re: W2) [Potential correction] Table 2's baselines differ from configs/tasks/*.yaml in facebookresearch/MLGym on five of twelve tasks: Fashion-MNIST 0.848 vs 0.7835, Language Modeling 4.238 vs 4.672731, MountainCar 53.26 vs 33.7938, Breakout 41.93 vs 48.8172, MetaMaze 20.12 vs 15.7344. B.2 explains re-measurement for the control tasks on jax/flax grounds but not for Fashion-MNIST or Language Modeling. Your human-best anchors also differ from ARTS's Table 3 on Language Modeling (3.280 vs 3.500), MountainCar (100.0 vs 99.00) and MetaMaze (100.0 vs 52.50). Under the 15.73/52.50 MetaMaze anchors your 81.19 and your re-run's 75.41 both clip to z = 1.0 and the task contributes nothing to Figure 2, whereas under 20.12/100.0 it contributes +0.072. Can you report Figure 2 under both anchor sets, and say which tasks change rank?

(Re: W2) [Clarification] Trainer.setup() issues three Claude Haiku 4.5 calls whose output is injected into the Coordinator's prompt as "TASK AUDIT NOTES", the third of which is given baseline_score and human_best_score, and state.py:26 shows human_best to the Coordinator every round. None of this appears in the paper. Did AIRA, MLEvolve, CORAL and ARTS* receive the same audit notes and the same human-best value? If they did not, how should I read Section 6.1's claim that this comparison tests whether a compact open-weight Scientist can compete with frozen proprietary Scientists, given that a proprietary model pre-analyzed the task for MELD's Coordinator and supplied the target score that also normalizes its reward?

(Re: W1) [Clarification] Section 4.3 describes the G forks as independent and Proposition 5.2(iv) relies on conditional independence for the leave-one-out baseline, yet mix_diversity_fix (config.py:53, default True) resamples any Coordinator fork whose parent-id tuple duplicates another's, for up to mix_dedup_max_rounds rounds (trainer.py:334-366). Was this active for Tables 1 and 4, and how does conditioning the group on mutual distinctness preserve the mean-zero property of A_hat_g? Separately, Table 3 states one optimizer step per group and Theorem 5.6 counts M = 2C updates, but trainer.py:547 and :610 skip the step when is_collapsed(utilities, 0.02) fires. How many Coordinator and Scientist steps were actually taken across the 22 tasks and three seeds, how often did the collapse guard fire, and what is the realized distribution of C? Since eta <= 1/(4L) and mu <= L force q >= 15/16, C = 1 as in B.5 contracts the initial gap by at most about 6%, so without this I cannot tell whether the q^C term in Eq. (9) does any work at your settings. Two further quantities the paper depends on are never stated: the per-task evaluator-call budget K, which sets Phi_{tau,K} and the tau*log(K+1) slack in Proposition 5.2(i) and Eq. (9), and any run artifact behind Tables 1 and 4, since the supplement ships source only with no events.jsonl or report.json. At tau = 0.05 that slack is 0.19 at K = 45 and 0.27 at K = 240 on a utility scale of [0,1], which is 38 to 54 times the +5.1e-3 coordination gain in your own Remark A.8. What was K per task, and will you add run artifacts for at least one seed per task?

(Re: W1) [Potential correction] Eq. (2) and Section 4.1 define the Scientist's action as one (hypothesis, execution plan) pair, but vs_k=8 (config.py:42) with WORKER_SYSTEM and trainer.py:409-410 has each Scientist emit eight hypotheses with self-reported probabilities and the harness draw one by weighted choice, with the GRPO update applied to the log-probability of the full eight-item response. Which object is the action in Eq. (5), and where does the selection step enter the gradient? Does the Section 6.3 N=1 result hold at vs_k=1, given that at N=1 the eight-way proposal is the only remaining source of within-round diversity?

(Re: W2) [Clarification] Table 4 is the only place the paper isolates joint adaptation from single-level adaptation, and it reports threshold-hit counts over three seeds on two tasks with no dispersion, where the Frozen arm also moves from 2 to 3 hits on Prisoner's Dilemma and you describe the Colonel Blotto effect as "a gain that neither the batch median nor a cumulative best score reveals". Can you report per-seed counts with an interval, and extend the ablation to Breakout, RSNA Brain or one control task where the Table 1 margins are large? Can you add a shared-adapter arm, three units reading one Scientist LoRA, so that Proposition 5.4's dispersion claim D(Z;S) is measured rather than derived?

1 comment
Reviewer_2wnSOpenReview ↗

I raise my score from 1 to 2

Thank you for your detailed response

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.