- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a multi-agent collaboration framework for MLE tasks called Humanize MLE Agents (HMA). The core innovation of the paper is the introduction of a shared workspace-based, two-agent alternating optimization mechanism. This mechanism allows two heterogeneous agents, each using the native Harness, to take turns executing tasks, sets an upper limit on the number of experiments, and resets the context upon switching while preserving existing code, experimental results, and candidate models. Experiments were conducted on the MLE-bench, which comprises 75 Kaggle tasks. Through six agent configurations, experiments with different interruption parameters, and case studies, the authors demonstrated the method’s potential effectiveness in heterogeneous agent collaboration.
Critical strengths
- This paper proposes a concise and general-purpose mechanism that preserves the integrity of the native testing framework while leveraging the complementary behaviors of agents. This design features low integration complexity and has the potential to adapt to different models and native toolchains. Furthermore, the concepts of “testing framework obsolescence” and “long-time-domain deceleration” introduced in this paper reveal practical failure modes faced by many agent-based systems as models evolve or trajectories extend.
- The paper was evaluated on the MLE-bench and covered a variety of different models. The optimal HMA configuration achieved improvement over the strongest single-agent baseline. All six alternating configurations achieved higher medal rates than their corresponding strong single-agent baselines. Furthermore, the paper not only reports final performance but also provides performance trajectories during execution, session-switching behavior, and specific task examples, which help illustrate how shared experimental results can be further leveraged across different agents.
Critical weaknesses
- HMA introduces several factors that may affect performance. However, existing experiments have not fully distinguished the contributions of these factors. For example, the paper lacks control experiments involving multiple context resets for the same model under the same budget. Therefore, it is impossible to determine whether the performance improvement stems from the complementary capabilities of heterogeneous agents or is simply due to context refresh.
- The results in Appendix F.1 indicate that fixed interruptions are not necessarily superior to natural alternation, thereby calling into question the paper’s assertion that “fixed interruptions are the core mechanism driving performance improvements.” Therefore, I recommend including, at a minimum, control experiments involving model restart, heterogeneous model alternation, and context reset, and further elucidating the unique contributions of HMA compared to simple multiple restarts or natural alternation strategies.
- Table 2 presents the mean scores and error results for the single-agent baseline. HMA outperforms the higher-performing single-agent baseline by only 0.9–1.4 percentage points. Therefore, the current results do not indicate whether these differences stem from performance improvements or training-induced variability. It is sugguested to repeat the experiments and report paired confidence intervals at the task level. The paper should also more clearly distinguish between theoretical sufficient conditions and the experimental evidence used to evaluate these conditions.
Questions for authors
Q1. How can we distinguish the performance improvements contributed by different modules? Under the same experimental conditions, could you include feasible control experiments such as restarting the same model, alternating between heterogeneous models, and disabling periodic resets? These experiments would help identify the primary sources of HMA’s benefits.
Q2. What are the actual advantages of the fixed interruption strategy compared to alternating natural terminations? Appendix F.1 shows that the NTA result for GPT→Kimi is 77.3%. Could the authors further report the NTA and fixed-interruption performance of the main model combinations under the same budget, and explain which conditions determine whether fixed interruption yields positive or negative gains? If natural alternation is superior in certain scenarios, should the scope of the paper’s conclusions regarding the advantages of fixed interruption be limited?
Q3. How is the connection established between theoretical assumptions and the actual execution process? Could the authors provide empirical evidence for the collaborative access probability in Assumption 3.2? Furthermore, given that candidate results may degrade during actual operation and that the final review cannot fully preserve historical optimal results, can the theoretical conclusions be appropriately extended or modified under conditions where results are not monotonically preserved?
4 comments
Response to Reviewer ER6M (Continued)
W2 & Q2. When fixed interruptions help under a six-hour budget
The paper attributes the performance gains to alternation between complementary agents. A fixed interruption cap makes those gains less sensitive to which model starts. In the six-hour GPT–Kimi experiments, the cap raises the Kimi-first any-medal rate from 56.0% to 66.7%, while GPT-first NTA achieves 77.3% versus 70.7% with the cap. Across both orders, the cap narrows the gap from 21.3 to 4.0 percentage points and raises the average from 66.7% to 68.7% (Appendix F.1).
Our theory formalizes when handing off helps. Interruption advantage is the value of handing off immediately minus the value of continuing from the same state (Definition A.9). Theorem 3.5 gives a sufficient condition for fixed-cap alternation to exceed the single-agent reference ceiling. The stronger GPT-first NTA result motivates exploring handoff policies that adapt to each model’s strengths and progress.
W3. Effect sizes, run counts, and uncertainty
The performance gain is consistent across independent runs. Across three runs, Opus → GPT exceeds Opus alone by 5.8 percentage points, with an across-run SE of 0.4 and a task-level paired 95% confidence interval of [+1.3, +11.1]. We added repeated runs and task-level paired confidence intervals, and we separate the theoretical conditions from the empirical evidence. The three six-hour runs earned 58/75, 59/75, and 59/75 medals, averaging 78.2% versus 72.4 ± 1.6% for Opus alone. At the task level, Opus → GPT earns a medal more often than Opus /goal on 10 of the 75 tasks and less often on 3, and its difference from the mean of its two constituents is +8.0 points [+3.6, +13.1]. GPT → DeepSeek V4.1 Flash averaged 70.7% (SE 2.0), 6.0 points above the mean of its two constituent /goal baselines (64.7 ± 2.1).
HMA improves over the stronger single-agent baseline by 5.8 percentage points with Opus → GPT, exceeding the cited 0.9–1.4-point gaps. The smaller margins arise in other configurations and show that gains extend across model pairs and starting orders. GPT → Opus exceeds both constituent agents, achieving 73.3% any-medal versus 72.4% for Opus /goal overall, and 86.4% versus 80.3% on Lite tasks. Its smaller gain may reflect the six-hour budget, which leaves limited time for the two models to fully collaborate and allows the weaker starting model, GPT, to dominate early exploration. Across all six configurations in Table 2, HMA exceeds the mean of its two constituent /goal baselines by 3.1–9.6 percentage points. We are adding repeats for the four configurations currently evaluated with one run: GPT → Opus, GPT → Kimi, GPT → GLM, and DeepSeek V4.1 Flash → Kimi. We plan to complete two additional GPT → Kimi runs by November 18 (Anywhere on Earth), bringing the total to three independent runs.
| HMA configuration | Any-medal rate by run | Mean (%) | Across-run SE (pp) |
|---|---|---|---|
| Opus → GPT | 77.3%, 78.7%, 78.7% | 78.2 | 0.4 |
| GPT → DeepSeek V4.1 Flash | 72.0%, 73.3%, 66.7% | 70.7 | 2.0 |
Theoretical conditions versus empirical evidence. Theorem 3.3 gives a sufficient condition, under Assumptions 3.1 and 3.2, for exceeding the single-agent reference ceiling in the long run. Q3 reports the handoff statistics related to Assumption 3.2. The revision will separate these two kinds of statements explicitly.
Response to Reviewer ER6M (Continued)
Q3. Collaborative access and retention of historical best results
We show that the handoff statistics support both steps of Assumption 3.2 in every tested pair, and that the guarantee extends to the returned result, which keeps a medal in 96.7% of runs whose archive holds one. We connect each step of Assumption 3.2 to measurable handoff statistics, and we extend the guarantee to the returned result when candidates are not preserved monotonically. Assumption 3.2 (Cooperative access and use) has two steps: conditional on a cycle starting below the target, the first agent reaches a useful handoff state with probability at least , and from each such state the partner reaches the target beyond reference reachability with probability at least .
The table gives a finite-budget counterpart of both. A post-handoff option counts as an improvement if a valid submission exists at the handoff and the incoming agent produces a submission with a leaderboard percentile strictly above the best recorded before the handoff. For each task, we compare HMA’s any-medal rate with the mean across three /goal runs per constituent model. HMA exceeded the baseline mean in 107 of 450 configuration–task comparisons and fell below it in 34.
| HMA configuration | Post-handoff options improving the best-so-far percentile | Tasks above / below the constituent /goal mean |
|---|---|---|
| Opus → GPT | 37/251 | 18 / 2 |
| GPT → Opus | 59/232 | 15 / 5 |
| GPT → DeepSeek V4.1 Flash | 54/295 | 20 / 6 |
| GPT → Kimi | 52/201 | 20 / 6 |
| GPT → GLM | 39/180 | 17 / 6 |
| DeepSeek V4.1 Flash → Kimi | 35/278 | 17 / 9 |
| Total | 276/1,437 (19.2%) | 107 / 34 of 450 |
We extend the guarantee from the best archived result to the returned result through a conditional selection bound, which covers runs in which the returned candidate falls below the best result found during exploration. The archive preserves the best quality reached after experiments, , even as individual candidates change. Theorem 3.3 applies to archived quality when the archive process satisfies Assumption 3.2, using the same quality measure for the single-agent reference ceiling. For the returned quality , if final review returns a candidate at or above threshold with conditional probability at least given , uniformly over the budgets considered, then
\ge\rho_\tau\Pr(B_{2km}\ge\tau) \ge\rho_\tau\bigl[1-F_0(1-\alpha\beta)^m\bigr], \qquad F_0=\Pr(B_0<\tau).$$ A sufficient condition for the returned result to exceed the single-agent reference ceiling in long-run expected quality is $\rho_\tau\tau > U^*$, with the conditional retention bound holding as the budget grows. In Appendix F.4, the returned result retains any-medal status in 96.7% of six-hour runs whose archive contains a medal-winning result. Given this high retention rate, the main text focuses on the best result found during the run. We will add the above extension in the revision. We believe these new experiments and clarifications address each of your concerns. We would sincerely appreciate it if you would consider raising your score and supporting the paper for acceptance. We are delighted to discuss any further questions.Response to Reviewer ER6M (Continued)
Thank you again for raising the question of how the margins in Table 2 vary across runs.
As promised, we have completed two additional GPT → Kimi runs. The three runs reach 70.7% (Table 2), 70.7%, and 68.0% any-medal, for a mean of 69.8% (SE 0.9 percentage points). The SE is about ten times smaller than the 8.7-point gain over the mean of the two constituents (61.1%), consistent with our finding in W3 that the margins reflect a consistent effect rather than run-to-run variability.
We have not yet heard back after our rebuttal, and we would be grateful to know whether this update, together with our earlier responses, addresses your concerns. We look forward to your response and would be happy to discuss any questions that remain.
Response to Reviewer ER6M
We thank Reviewer ER6M for the detailed comments. We added (i) same-model restart controls for Opus, GPT, and Kimi, (ii) repeated runs for the main configurations, and (iii) task-level paired confidence intervals. The main takeaways are:
/goal; switching to a different model adds a further 2.2 points overall and the entire 15.5-point gain on High-complexity tasks (W1 & Q1)./goal, with a task-level paired 95% confidence interval of [+1.3, +11.1] (W3).We address each point below.
W1 & Q1. Isolating the contributions of HMA components
We show that model switching, not context refresh alone, accounts for HMA’s gain on hard tasks: same-model restarts add 3.6 points over Opus
/goal, and switching to a different model adds a further 2.2 points overall and the entire 15.5-point gain on High-complexity tasks. We follow the evaluation protocol used for Table 2. For the control experiments, restarting the same model is the same-model control, and alternating heterogeneous models is HMA. Disabling periodic context resets corresponds to two existing baselines: for a single model, the/goalrun, in which one native session keeps its context for the full six-hour budget, and for a model pair, natural-termination alternation (NTA), which hands off only when a session ends on its own.Opus → Opus reaches 76.0% any-medal and Opus → GPT reaches 78.2%. Opus → GPT also leads Opus → Opus on Med+ (90.7% versus 86.7%) and mean percentile (89.0 versus 86.3). Each restart uses the same model and native harness, starts a fresh context at natural termination or after five accepted experiments, and carries forward the shared workspace. The experiments use the same 75 MLE-bench tasks, six-hour budget, and algorithm as HMA.
Beyond the same-model controls, the remaining ablations isolate what each further component adds:
Model switching. With the restart procedure, cap, and final review unchanged, swapping the second Opus for GPT lifts the any-medal rate over Opus → Opus by 2.2 points overall and by 15.5 points on High-complexity tasks. Opus → GPT wins medals on five tasks where Opus → Opus does not, three of them High-complexity. Because the workspace is shared, the incoming model can extend what its partner built; on Google QUEST, for instance, GPT and DeepSeek reused stored predictions for target-specific blending (Appendix G.2).
Session cap. The cap gives the second agent more time to contribute within the six-hour budget. Under NTA, the first agent uses 81–83% of the runtime, and with GPT first, Kimi had responded to only 19 of 75 tasks by hour 5.75 (Appendix F.1; see W2 & Q2). The cap mainly affects the opening session. Among first sessions that end before the deadline, 87.2% reach the cap, while 60.3% of later sessions end naturally (Appendix F.3). GPT → Kimi earns 49, 50, and 53 medals at caps of one, three, and five accepted experiments, respectively (Appendix F.2).
Final selection. Holding the exploration trajectories fixed, final review keeps a medal in 319 of the 330 runs that achieved a medal-winning candidate (96.7%), up from 311 (94.2%) before review, a net increase of eight runs (Appendix F.4). Most medal-winning results thus already survive exploration, and final review adds a further safeguard.