acceptodds
Under review as a conference paper at ICLR 2027

Humanize MLE Agents: Multi-Agent Session Alternation Beyond Single-Agent Limits

Abstract

Existing LLM-based agent approaches for long-horizon machine learning engineering (MLE) often use context engineering, implemented through custom harnesses that manage tools, skills, and accumulated experience. However, we find that as single-session execution lengthens, the agent's reasoning intensity declines alongside slowing progress, summarized as **long-horizon deceleration**, and the benefits of these custom harnesses vanish as the underlying model improves, summarized as **harness staleness**. These findings suggest that their benefits may not persist across versions, which motivates a workflow-level approach in which agents build on one another's work across sessions. We model these alternative agent sessions as a *context-reset semi-Markov decision process* (**CR-SMDP**) and establish sufficient conditions for alternation to exceed single-agent reference limits, derive an explicit experiment-budget bound, and characterize when session caps preserve this advantage. Guided by this analysis, we introduce *Humanize MLE Agents* (**HMA**), which alternates two agents on a shared workspace, limits each session to at most experiments and resets the context between sessions, without additional task-specific harness or prior knowledge supplements. Across 75 Kaggle tasks in MLE-bench, **HMA** reaches a **state-of-the-art** any-medal rate of 78.2%, exceeding the strongest open-source baseline by points with a quarter of its time budget. Moreover, among six configurations spanning five heterogeneous agent pairs, HMA improves any-medal rate over the constituent baselines by points on average, demonstrating cross-pair generality.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM agent, machine learning engineering, multi-agent workflow, semi-Markov decision process

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ER6MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a multi-agent collaboration framework for MLE tasks called Humanize MLE Agents (HMA). The core innovation of the paper is the introduction of a shared workspace-based, two-agent alternating optimization mechanism. This mechanism allows two heterogeneous agents, each using the native Harness, to take turns executing tasks, sets an upper limit on the number of experiments, and resets the context upon switching while preserving existing code, experimental results, and candidate models. Experiments were conducted on the MLE-bench, which comprises 75 Kaggle tasks. Through six agent configurations, experiments with different interruption parameters, and case studies, the authors demonstrated the method’s potential effectiveness in heterogeneous agent collaboration.

Critical strengths
  1. This paper proposes a concise and general-purpose mechanism that preserves the integrity of the native testing framework while leveraging the complementary behaviors of agents. This design features low integration complexity and has the potential to adapt to different models and native toolchains. Furthermore, the concepts of “testing framework obsolescence” and “long-time-domain deceleration” introduced in this paper reveal practical failure modes faced by many agent-based systems as models evolve or trajectories extend.
  2. The paper was evaluated on the MLE-bench and covered a variety of different models. The optimal HMA configuration achieved improvement over the strongest single-agent baseline. All six alternating configurations achieved higher medal rates than their corresponding strong single-agent baselines. Furthermore, the paper not only reports final performance but also provides performance trajectories during execution, session-switching behavior, and specific task examples, which help illustrate how shared experimental results can be further leveraged across different agents.
Critical weaknesses
  1. HMA introduces several factors that may affect performance. However, existing experiments have not fully distinguished the contributions of these factors. For example, the paper lacks control experiments involving multiple context resets for the same model under the same budget. Therefore, it is impossible to determine whether the performance improvement stems from the complementary capabilities of heterogeneous agents or is simply due to context refresh.
  2. The results in Appendix F.1 indicate that fixed interruptions are not necessarily superior to natural alternation, thereby calling into question the paper’s assertion that “fixed interruptions are the core mechanism driving performance improvements.” Therefore, I recommend including, at a minimum, control experiments involving model restart, heterogeneous model alternation, and context reset, and further elucidating the unique contributions of HMA compared to simple multiple restarts or natural alternation strategies.
  3. Table 2 presents the mean scores and error results for the single-agent baseline. HMA outperforms the higher-performing single-agent baseline by only 0.9–1.4 percentage points. Therefore, the current results do not indicate whether these differences stem from performance improvements or training-induced variability. It is sugguested to repeat the experiments and report paired confidence intervals at the task level. The paper should also more clearly distinguish between theoretical sufficient conditions and the experimental evidence used to evaluate these conditions.
Questions for authors

Q1. How can we distinguish the performance improvements contributed by different modules? Under the same experimental conditions, could you include feasible control experiments such as restarting the same model, alternating between heterogeneous models, and disabling periodic resets? These experiments would help identify the primary sources of HMA’s benefits.

Q2. What are the actual advantages of the fixed interruption strategy compared to alternating natural terminations? Appendix F.1 shows that the NTA result for GPT→Kimi is 77.3%. Could the authors further report the NTA and fixed-interruption performance of the main model combinations under the same budget, and explain which conditions determine whether fixed interruption yields positive or negative gains? If natural alternation is superior in certain scenarios, should the scope of the paper’s conclusions regarding the advantages of fixed interruption be limited?

Q3. How is the connection established between theoretical assumptions and the actual execution process? Could the authors provide empirical evidence for the collaborative access probability in Assumption 3.2? Furthermore, given that candidate results may degrade during actual operation and that the final review cannot fully preserve historical optimal results, can the theoretical conclusions be appropriately extended or modified under conditions where results are not monotonically preserved?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ER6M

We thank Reviewer ER6M for the detailed comments. We added (i) same-model restart controls for Opus, GPT, and Kimi, (ii) repeated runs for the main configurations, and (iii) task-level paired confidence intervals. The main takeaways are:

  • Model switching, not context refresh alone, drives the gain on hard tasks. Same-model restarts add 3.6 points over Opus /goal; switching to a different model adds a further 2.2 points overall and the entire 15.5-point gain on High-complexity tasks (W1 & Q1).
  • Fixed interruption makes alternation robust to the starting model. It narrows the GPT–Kimi starting-order gap from 21.3 to 4.0 points and raises the average over both orders from 66.7% to 68.7% (W2 & Q2).
  • The performance gain is consistent across independent runs. Over three independent runs, Opus → GPT reaches 78.2% (SE 0.4), 5.8 points above Opus /goal, with a task-level paired 95% confidence interval of [+1.3, +11.1] (W3).
  • The theory connects to measured behavior. Incoming agents improve the best-so-far result after 19.2% of eligible handoffs, and the guarantee extends to the returned result, which keeps a medal in 96.7% of runs whose archive holds one (Q3).

We address each point below.

W1 & Q1. Isolating the contributions of HMA components

We show that model switching, not context refresh alone, accounts for HMA’s gain on hard tasks: same-model restarts add 3.6 points over Opus /goal, and switching to a different model adds a further 2.2 points overall and the entire 15.5-point gain on High-complexity tasks. We follow the evaluation protocol used for Table 2. For the control experiments, restarting the same model is the same-model control, and alternating heterogeneous models is HMA. Disabling periodic context resets corresponds to two existing baselines: for a single model, the /goal run, in which one native session keeps its context for the full six-hour budget, and for a model pair, natural-termination alternation (NTA), which hands off only when a session ends on its own.

Opus → Opus reaches 76.0% any-medal and Opus → GPT reaches 78.2%. Opus → GPT also leads Opus → Opus on Med+ (90.7% versus 86.7%) and mean percentile (89.0 versus 86.3). Each restart uses the same model and native harness, starts a fresh context at natural termination or after five accepted experiments, and carries forward the shared workspace. The experiments use the same 75 MLE-bench tasks, six-hour budget, and algorithm as HMA.

WorkflowLite (22)Medium (38)High (15)All (75)Change in All (pp)
Claude Opus 5 (/goal, mean)80.378.146.772.4
Opus → Opus86.481.646.776.0+3.6
Opus → GPT (HMA, mean)84.880.762.278.2+5.8
GPT-5.6-sol (/goal, mean)80.369.346.768.0
GPT → GPT77.378.946.772.0+4.0
Kimi K3 (/goal, mean)77.345.642.254.2
Kimi → Kimi72.752.640.056.0+1.8

Beyond the same-model controls, the remaining ablations isolate what each further component adds:

  • Model switching: bringing in a model with different strengths strengthens exploration.
  • Session cap: reserving working time for both agents makes the result depend less on which model starts.
  • Final selection: the final review protects medal-winning results.

Model switching. With the restart procedure, cap, and final review unchanged, swapping the second Opus for GPT lifts the any-medal rate over Opus → Opus by 2.2 points overall and by 15.5 points on High-complexity tasks. Opus → GPT wins medals on five tasks where Opus → Opus does not, three of them High-complexity. Because the workspace is shared, the incoming model can extend what its partner built; on Google QUEST, for instance, GPT and DeepSeek reused stored predictions for target-specific blending (Appendix G.2).

Session cap. The cap gives the second agent more time to contribute within the six-hour budget. Under NTA, the first agent uses 81–83% of the runtime, and with GPT first, Kimi had responded to only 19 of 75 tasks by hour 5.75 (Appendix F.1; see W2 & Q2). The cap mainly affects the opening session. Among first sessions that end before the deadline, 87.2% reach the cap, while 60.3% of later sessions end naturally (Appendix F.3). GPT → Kimi earns 49, 50, and 53 medals at caps of one, three, and five accepted experiments, respectively (Appendix F.2).

Final selection. Holding the exploration trajectories fixed, final review keeps a medal in 319 of the 330 runs that achieved a medal-winning candidate (96.7%), up from 311 (94.2%) before review, a net increase of eight runs (Appendix F.4). Most medal-winning results thus already survive exploration, and final review adds a further safeguard.

AuthorsOpenReview ↗

Response to Reviewer ER6M (Continued)

W2 & Q2. When fixed interruptions help under a six-hour budget

The paper attributes the performance gains to alternation between complementary agents. A fixed interruption cap makes those gains less sensitive to which model starts. In the six-hour GPT–Kimi experiments, the cap raises the Kimi-first any-medal rate from 56.0% to 66.7%, while GPT-first NTA achieves 77.3% versus 70.7% with the cap. Across both orders, the cap narrows the gap from 21.3 to 4.0 percentage points and raises the average from 66.7% to 68.7% (Appendix F.1).

Our theory formalizes when handing off helps. Interruption advantage is the value of handing off immediately minus the value of continuing from the same state (Definition A.9). Theorem 3.5 gives a sufficient condition for fixed-cap alternation to exceed the single-agent reference ceiling. The stronger GPT-first NTA result motivates exploring handoff policies that adapt to each model’s strengths and progress.

W3. Effect sizes, run counts, and uncertainty

The performance gain is consistent across independent runs. Across three runs, Opus → GPT exceeds Opus alone by 5.8 percentage points, with an across-run SE of 0.4 and a task-level paired 95% confidence interval of [+1.3, +11.1]. We added repeated runs and task-level paired confidence intervals, and we separate the theoretical conditions from the empirical evidence. The three six-hour runs earned 58/75, 59/75, and 59/75 medals, averaging 78.2% versus 72.4 ± 1.6% for Opus alone. At the task level, Opus → GPT earns a medal more often than Opus /goal on 10 of the 75 tasks and less often on 3, and its difference from the mean of its two constituents is +8.0 points [+3.6, +13.1]. GPT → DeepSeek V4.1 Flash averaged 70.7% (SE 2.0), 6.0 points above the mean of its two constituent /goal baselines (64.7 ± 2.1).

HMA improves over the stronger single-agent baseline by 5.8 percentage points with Opus → GPT, exceeding the cited 0.9–1.4-point gaps. The smaller margins arise in other configurations and show that gains extend across model pairs and starting orders. GPT → Opus exceeds both constituent agents, achieving 73.3% any-medal versus 72.4% for Opus /goal overall, and 86.4% versus 80.3% on Lite tasks. Its smaller gain may reflect the six-hour budget, which leaves limited time for the two models to fully collaborate and allows the weaker starting model, GPT, to dominate early exploration. Across all six configurations in Table 2, HMA exceeds the mean of its two constituent /goal baselines by 3.1–9.6 percentage points. We are adding repeats for the four configurations currently evaluated with one run: GPT → Opus, GPT → Kimi, GPT → GLM, and DeepSeek V4.1 Flash → Kimi. We plan to complete two additional GPT → Kimi runs by November 18 (Anywhere on Earth), bringing the total to three independent runs.

HMA configurationAny-medal rate by runMean (%)Across-run SE (pp)
Opus → GPT77.3%, 78.7%, 78.7%78.20.4
GPT → DeepSeek V4.1 Flash72.0%, 73.3%, 66.7%70.72.0

Theoretical conditions versus empirical evidence. Theorem 3.3 gives a sufficient condition, under Assumptions 3.1 and 3.2, for exceeding the single-agent reference ceiling in the long run. Q3 reports the handoff statistics related to Assumption 3.2. The revision will separate these two kinds of statements explicitly.

AuthorsOpenReview ↗

Response to Reviewer ER6M (Continued)

Q3. Collaborative access and retention of historical best results

We show that the handoff statistics support both steps of Assumption 3.2 in every tested pair, and that the guarantee extends to the returned result, which keeps a medal in 96.7% of runs whose archive holds one. We connect each step of Assumption 3.2 to measurable handoff statistics, and we extend the guarantee to the returned result when candidates are not preserved monotonically. Assumption 3.2 (Cooperative access and use) has two steps: conditional on a cycle starting below the target, the first agent reaches a useful handoff state with probability at least α\alpha, and from each such state the partner reaches the target beyond reference reachability with probability at least β\beta.

The table gives a finite-budget counterpart of both. A post-handoff option counts as an improvement if a valid submission exists at the handoff and the incoming agent produces a submission with a leaderboard percentile strictly above the best recorded before the handoff. For each task, we compare HMA’s any-medal rate with the mean across three /goal runs per constituent model. HMA exceeded the baseline mean in 107 of 450 configuration–task comparisons and fell below it in 34.

HMA configurationPost-handoff options improving the best-so-far percentileTasks above / below the constituent /goal mean
Opus → GPT37/25118 / 2
GPT → Opus59/23215 / 5
GPT → DeepSeek V4.1 Flash54/29520 / 6
GPT → Kimi52/20120 / 6
GPT → GLM39/18017 / 6
DeepSeek V4.1 Flash → Kimi35/27817 / 9
Total276/1,437 (19.2%)107 / 34 of 450

We extend the guarantee from the best archived result to the returned result through a conditional selection bound, which covers runs in which the returned candidate falls below the best result found during exploration. The archive preserves the best quality reached after NN experiments, BNB_N, even as individual candidates change. Theorem 3.3 applies to archived quality when the archive process satisfies Assumption 3.2, using the same quality measure for the single-agent reference ceiling. For the returned quality RNR_N, if final review returns a candidate at or above threshold τ\tau with conditional probability at least ρτ\rho_\tau given BN≥τB_N\ge\tau, uniformly over the budgets considered, then

\ge\rho_\tau\Pr(B_{2km}\ge\tau) \ge\rho_\tau\bigl[1-F_0(1-\alpha\beta)^m\bigr], \qquad F_0=\Pr(B_0<\tau).$$ A sufficient condition for the returned result to exceed the single-agent reference ceiling in long-run expected quality is $\rho_\tau\tau > U^*$, with the conditional retention bound holding as the budget grows. In Appendix F.4, the returned result retains any-medal status in 96.7% of six-hour runs whose archive contains a medal-winning result. Given this high retention rate, the main text focuses on the best result found during the run. We will add the above extension in the revision. We believe these new experiments and clarifications address each of your concerns. We would sincerely appreciate it if you would consider raising your score and supporting the paper for acceptance. We are delighted to discuss any further questions.
AuthorsOpenReview ↗

Response to Reviewer ER6M (Continued)

Thank you again for raising the question of how the margins in Table 2 vary across runs.

As promised, we have completed two additional GPT → Kimi runs. The three runs reach 70.7% (Table 2), 70.7%, and 68.0% any-medal, for a mean of 69.8% (SE 0.9 percentage points). The SE is about ten times smaller than the 8.7-point gain over the mean of the two constituents (61.1%), consistent with our finding in W3 that the margins reflect a consistent effect rather than run-to-run variability.

We have not yet heard back after our rebuttal, and we would be grateful to know whether this update, together with our earlier responses, addresses your concerns. We look forward to your response and would be happy to discuss any questions that remain.

Reviewer_2zubOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a multi-agent workflow for long-horizon machine learning engineering tasks, Humanize MLE Agents (HMA). HMA alternates 2 heterogeneous LLM agents operating on a shared workspace. Each agent performs at most k experiments before handing control to the other, while preserving experimental artifacts and resetting the agent context.

The authors formulate the workflow as a context-reset semi-Markov decision process (CR-SMDP). Experiments on 75 MLE-bench tasks show that HMA achieves a best any-medal rate of 78.2%, outperforming the strongest single-agent baseline by 5.8 percentage points.

Additional experiments investigate different agent pairs, starting orders, interruption caps, and candidate selection.

Critical strengths
  1. The paper explores an important problem in long-horizon MLE agents and proposes a simple workflow that coordinates existing agents without modifying their native harnesses or requiring additional training. This makes the approach practically easy to adopt.
  2. The evaluation covers 75 MLE-bench tasks and several heterogeneous agent configurations. The analyses of starting orders, interruption caps, and candidate selection provide useful empirical insights.
Critical weaknesses

My major concern is that the paper does not distinguish whether the observed gains arise from complementary capabilities of heterogeneous models or simply from restarting agent sessions and exploration. This is particularly important given that model complementarity is a central motivation of the proposed approach. The authors should include stronger baselines, such as homogeneous alternation (e.g., Claude → Claude) and single-agent execution with periodic context resets.

Although the paper formulates the workflow as a CR-SMDP, the actual method relies on a predefined agent pair and a fixed interruption cap of (k=5). It does not adaptively determine when to switch or which agent should take over based on the current task state or exploration progress. This creates a gap between the general theoretical formulation and the actual algorithm.

Theorem 3.3 relies on a strong cooperative-access assumption, but the paper does not establish when or why such complementarity emerges, or how it relates to agent selection and switching decisions.

Questions for authors
  1. Have the authors considered homogeneous alternation (e.g., Claude → Claude) or same-family model alternation? How would these baselines compare with heterogeneous alternation under the same execution method?
  2. The current method uses a predefined agent pair and fixed (k=5) interruption. What determines whether two models are complementary? Could the authors provide an analysis of how agent selection, starting order, and switching frequency affect performance?
  3. Is there a measurable relationship between model complementarity, interruption advantage, and the observed performance gains?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 2zub

We thank Reviewer 2zub for the thoughtful review. We added (i) homogeneous alternation for Opus, GPT, Kimi, and Sonnet 5, (ii) same-family Opus → Sonnet 5 alternation under the same execution method, and (iii) per-task measurements of complementarity. The main takeaways are:

  • Heterogeneous models, not restarts, drive the gain on hard tasks. Same-model restarts add 3.6 points over Opus /goal; a partner from a different family adds a further 2.2 points overall and the entire 15.5-point gain on High-complexity tasks, while a same-family partner does not help (W1 & Q1).
  • HMA is one policy within the CR-SMDP, and its handoffs follow exploration progress. 60.3% of later sessions end naturally before the cap (W2).
  • Complementarity is measurable in every tested pair. Incoming agents improved the best-so-far result after 19.2% of eligible handoffs. HMA’s any-medal rate exceeded the mean of its constituent /goal baselines in 107 of 450 configuration–task comparisons and fell below it in 34 (W3 & Q3).
  • The design choices are grounded in measurements. The choice of partner matters most, starting with the stronger model works better, and k=5k=5 retains 97.83% of observed within-session gains (Q2).

We address each point below.

W1 & Q1. Homogeneous and same-family alternation

We show that the gain on hard tasks comes from heterogeneous models rather than from restarting sessions: same-model restarts add 3.6 points over Opus /goal, switching to a model from a different family adds a further 2.2 points overall and the entire 15.5-point gain on High-complexity tasks, and a same-family partner does not help. We ran homogeneous alternation for Opus, GPT, Kimi, and Sonnet 5 and same-family alternation for Opus → Sonnet 5. All controls follow the HMA workflow and the evaluation protocol used for Table 2. Homogeneous alternation implements single-agent execution with periodic context resets: at each restart, the same model resumes work in the shared workspace with a fresh context. We also plan to add results for same-family alternation between GPT-5.6 Sol and GPT-5.6 Terra by November 18 (Anywhere on Earth).

WorkflowLite (22)Medium (38)High (15)All (75)Change in All (pp)
Claude Opus 5 (/goal, mean)80.378.146.772.4
Opus → Opus (homogeneous)86.481.646.776.0+3.6
Opus → GPT (HMA, mean)84.880.762.278.2+5.8
Opus → Sonnet 5 (same-family)81.873.746.770.7
Sonnet 5 → Sonnet 5 (homogeneous)77.347.440.054.7
GPT-5.6-sol (/goal, mean)80.369.346.768.0
GPT → GPT (homogeneous)77.378.946.772.0+4.0
Kimi K3 (/goal, mean)77.345.642.254.2
Kimi → Kimi (homogeneous)72.752.640.056.0+1.8

Homogeneous alternation. The restart gain holds for every model with a /goal baseline, at +3.6, +4.0, and +1.8 points for Opus, GPT, and Kimi with 3.2 to 6.3 sessions per task, but it varies by model and does not reach the hardest tasks: Opus → Opus matches Opus /goal on High-complexity tasks (46.7%).

Heterogeneous alternation. A partner from a different family supplies the High-complexity gain that restarts alone miss: of the five tasks Opus → GPT gains over Opus → Opus, three are High-complexity, and none of the four it loses is. Opus → GPT also leads Opus → Opus on Med+ (90.7% versus 86.7%) and mean percentile (89.0 versus 86.3), and all six heterogeneous configurations in Table 2 exceed the mean of their two constituent /goal baselines, by 3.1 to 9.6 points.

Same-family alternation. Pairing Sonnet 5 with Opus 5 from the same model family raises Sonnet from 54.7% (Sonnet 5 → Sonnet 5) to 70.7%, but Opus → Sonnet 5 stays below both Opus → Opus (76.0%) and Opus → GPT (78.2%). A same-family partner adds little capability that Opus lacks, so the gain from alternation depends on a partner with complementary strengths rather than on switching models alone.

AuthorsOpenReview ↗

Response to Reviewer 2zub (Continued)

W2. What the CR-SMDP models in the implemented policy

HMA instantiates the fixed-kk interruption policy within the CR-SMDP. The cap sets a maximum of five accepted experiments per session, while the actual handoff can occur earlier if the agent judges that further experiments are unlikely to help. The number of experiments completed before natural termination thus reflects the agent’s assessment of its progress. The policy specifies the model pair and starting agent. A session ends when the agent stops on its own, reaches the cap, or hits the exploration deadline. Appendix F.3 shows that the cap acts mainly on the opening session: 87.2% of first sessions that end before the deadline reach the cap, while 60.3% of later sessions end naturally before reaching it (45.5% across all sessions).

Theorem 3.3 gives a sufficient condition for alternation to exceed the single-agent reference ceiling. At a given experiment budget, Theorem 3.5 shows that the fixed-cap policy exceeds that ceiling if the natural-termination alternation (NTA) margin over the ceiling plus the expected net interruption gain is positive. We adopt fixed-kk interruption for its simplicity and empirical effectiveness. Theorem 3.5 provides a conditional guarantee.

W3 & Q3. Complementarity, handoff improvements, and observed gains

We observe complementarity in every tested pair. Incoming agents improved the best-so-far result after 19.2% of eligible handoffs. HMA’s any-medal rate exceeded the mean of its constituent /goal baselines in 107 of 450 configuration–task comparisons and fell below it in 34. A post-handoff option counts as an improvement if a valid submission exists at the handoff and the incoming agent produces a submission with a leaderboard percentile strictly above the best recorded before the handoff. For each task, we compare HMA’s any-medal rate with the mean across three /goal runs per constituent model.

HMA configurationPost-handoff options improving the best-so-far percentileTasks above / below the constituent /goal mean
Opus → GPT37/25118 / 2
GPT → Opus59/23215 / 5
GPT → DeepSeek V4.1 Flash54/29520 / 6
GPT → Kimi52/20120 / 6
GPT → GLM39/18017 / 6
DeepSeek V4.1 Flash → Kimi35/27817 / 9
Total276/1,437 (19.2%)107 / 34 of 450

When and why complementarity emerges. It appears when a partner can use an inherited artifact that the other model produced but did not fully exploit. On Google QUEST, GPT and DeepSeek reused saved predictions for target-specific blending, and on Smartphone Decimeter, Kimi improved how DeepSeek’s positioning solver handled a signal outage (Appendix G). The same-family control in W1 shows the converse: Opus → Sonnet 5 stays below Opus → Opus, which we attribute to a same-family partner bringing little that Opus lacks.

Relation to the interruption advantage and the gains. The interruption advantage (Section 3.3 and Definition A.9) compares handing off immediately with continuing the current session from the same state. The post-handoff improvement rate is its observed counterpart, counting how often a handoff actually improved the best result. All six configurations show both a positive improvement rate and a net per-task gain over their constituents, consistent with Theorem 3.3, which gives the long-run result under cooperative access and best-result retention.

AuthorsOpenReview ↗

Response to Reviewer 2zub (Continued)

Q2. Agent selection, starting order, and switching frequency

We identify two properties that make models complementary, and our controls show how the three design choices affect performance:

  • Agent selection: the choice of partner matters most.
  • Starting order: starting with the stronger model works better.
  • Switching frequency: k=5k=5 gives the best switching frequency among the caps we tested.

Agent selection. We assess model complementarity along two dimensions. The first is complementary task coverage in standalone /goal runs, where each model succeeds on tasks the other struggles with. The second is the ability to build on a partner’s work after a handoff. This reflects cooperative access and use (Assumption 3.2), as the incoming agent can reuse its partner’s predictions, features, or solvers to improve the shared solution. The controls in W1 show why this matters: with Opus first, a different-family partner reaches 78.2% (Opus → GPT), the same model reaches 76.0% (Opus → Opus), and a same-family partner reaches 70.7% (Opus → Sonnet 5). The table in W3 & Q3 shows how often the incoming agent improves the best-so-far percentile in each pair.

Starting order. Within the six-hour budget, the first agent shapes the workspace and determines how much time its partner has left. With the cap, starting with the stronger model yields higher any-medal rates in both pairs: Opus → GPT reaches 78.2% versus 73.3% for GPT → Opus, and GPT → Kimi reaches 70.7% versus 66.7% for Kimi → GPT. Under natural-termination alternation (NTA), the first model uses about 81–83% of the budget. With GPT first, Kimi had responded to only 19 of 75 tasks by hour 5.75. The GPT-first and Kimi-first NTA runs reach 77.3% and 56.0%, respectively. At k=5k=5, GPT and Kimi use 51.1% and 46.3% of the budget in the GPT-first run. The gap between the two starting orders narrows from 21.3 to 4.0 percentage points, and the average rises from 66.7% to 68.7% (Appendix F.1). With unlimited budget, NTA and fixed-cap alternation reach the same limiting quality when access holds for every target below a common quality ceiling (Corollary A.8).

Switching frequency. We chose the cap using the estimate in Section 3.3 and Appendix B.2, then tested it directly. Early switching limits follow-up experiments, while late switching leaves less time for the partner. Definition A.9 formalizes this trade-off as the interruption advantage, and Corollary 3.6 shows that a finite cap suffices when interruption losses decay with kk. Across six /goal and five NTA configurations, the first 1, 3, and 5 experiments retain 63.15%, 93.35%, and 97.83% of observed gains while using 24.76%, 54.88%, and 73.77% of the experiments, respectively. At k=5k=5, every configuration retains 96.11% to 99.20% of its observed gains (Figure 3 and Appendix B.2). In the six-hour GPT-first ablation, the three caps yield 49, 50, and 53 medals on the same 75 tasks, respectively (Appendix F.2). The cap at k=5k=5 also ends 87.2% of opening sessions that close before the deadline (Appendix F.3), giving the partner an earlier start while retaining nearly all observed within-session gains.

We believe these new experiments and clarifications address each of your concerns. We would sincerely appreciate it if you would consider raising your score and supporting the paper for acceptance. We are glad to discuss any remaining questions.

Reviewer_QneHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether LLM agents can keep making progress on long ML engineering tasks by taking turns. HMA lets two native model–harness pairs share a workspace. It resets the context at each handoff. Each session allows up to five experiments. A final review stage selects candidates. The paper uses a semi-MDP model with context resets. This model gives sufficient conditions for alternation to exceed single-agent reference limits. The study tests six setups with different agents on the 75-task MLE-bench suite. Each setup has a higher reported medal rate than the stronger agent in its pair. The best setup reaches 78.2%, compared with 72.4%. Both use the same six-hour budget.

Critical strengths
  1. The best HMA setup reaches a 78.2% medal rate, compared with 72.4% for the stronger agent in the pair (Table 2). Both use the same hardware and six-hour budget. This is a useful gain. The workflow is simple and offers a practical way to improve strong native agents.

  2. All six setups beat the stronger agent in their pair in reported medal rate. They cover five model pairs, so the gains hold across several pairings.

  3. Appendix F looks at starting order, the experiment cap, and partner participation. There are also examples in Appendix G of agents reusing and improving each other’s artifacts. These details help explain how the workflow works.

  4. With exploration held fixed, medal retention rises from 311/330 to 319/330 after final review. This gives some support for taking another look at archived candidates.

Critical weaknesses
  1. HMA uses context resets, session caps, model switching, and final selection together. It is hard to tell how much of the gain comes from using different agents. A control that restarts the same model under the same workflow would help separate these effects.

  2. Small gaps such as 73.3% versus 72.4% are hard to judge without uncertainty estimates. Table 2 gives no error bars for HMA. Appendix C.3 specifies three repeats, but the actual repeat count and uncertainty are unclear for each HMA setup. Paired comparisons would help show how reliable these differences are.

  3. The guarantee assumes complementary access to useful states. But the paper does not show whether the tested pairs meet this condition. This limits the theory’s value in predicting results.

Questions for authors
  1. HMA resets context, caps sessions, switches models, and selects a final candidate. It is hard to tell how much of the gain comes from using different agents. Testing the same workflow with restarts of a single model would help clarify this.

  2. Table 2 gives no error bars for HMA. Appendix C.3 specifies three repeats, but the actual repeat count and uncertainty are unclear for each setup. This matters when comparing small gaps, such as 73.3% versus 72.4%. Paired comparisons would help show whether these differences are reliable.

4 comments
AuthorsOpenReview ↗

Response to Reviewer QneH

We thank Reviewer QneH for the careful reading and constructive suggestions. We added (i) same-model restart controls for Opus, GPT, and Kimi, (ii) task-level paired comparisons, and (iii) run counts and across-run uncertainty for each HMA configuration. The main takeaways are:

  • Model switching, not restarts alone, drives the gain on hard tasks. Same-model restarts add 3.6 points over Opus /goal; switching to GPT adds a further 2.2 points overall and the entire 15.5-point gain on High-complexity tasks (W1 & Q1).
  • The performance gain is consistent across independent runs. Over three independent runs, Opus → GPT reaches 78.2% (SE 0.4), 5.8 points above Opus /goal (W2 & Q2).
  • Every tested pair exhibits cooperative access. Incoming agents improved the best-so-far result after 19.2% of eligible handoffs. HMA’s any-medal rate exceeded the mean of its constituent /goal baselines in 107 of 450 configuration–task comparisons and fell below it in 34 (W3).

We address each point below.

W1 & Q1. Same-model restart control

We show that model switching, not restarts alone, accounts for HMA’s gain on hard tasks: same-model restarts add 3.6 points over Opus /goal, and switching to GPT adds a further 2.2 points overall and the entire 15.5-point gain on High-complexity tasks. We ran same-model restart controls for Opus, GPT, and Kimi using the evaluation protocol from Table 2. Opus → Opus reaches 76.0% any-medal and Opus → GPT reaches 78.2%. Opus → GPT also leads Opus → Opus on Med+ (90.7% versus 86.7%) and mean percentile (89.0 versus 86.3).

WorkflowLite (22)Medium (38)High (15)All (75)Change in All (pp)
Claude Opus 5 (/goal, mean)80.378.146.772.4
Opus → Opus86.481.646.776.0+3.6
Opus → GPT (HMA, mean)84.880.762.278.2+5.8
GPT-5.6-sol (/goal, mean)80.369.346.768.0
GPT → GPT77.378.946.772.0+4.0
Kimi K3 (/goal, mean)77.345.642.254.2
Kimi → Kimi72.752.640.056.0+1.8

Building on the same-model restart controls above, our ablations separate three further components:

  • Model switching: a partner from a different family improves exploration, especially on hard tasks.
  • Session cap: both agents get time to contribute within the budget.
  • Final selection: medal-winning results found during exploration are preserved.

Model switching. Holding the restart procedure, cap, and final review fixed, Alternating (Opus → GPT) improves over Restarts (Opus → Opus) by 2.2 percentage points overall and 15.5 points on High-complexity tasks. It earns medals on five tasks that Opus → Opus misses, including three High-complexity tasks. The shared workspace lets the incoming model build on its partner’s artifacts. For example, GPT and DeepSeek reused saved predictions for target-specific blending on Google QUEST (Appendix G.2).

Session cap. Fixed interruptions reduce sensitivity to starting order by allocating time for both agents within the six-hour budget. Under NTA, the first model occupies about 81–83% of the runtime; with GPT first, Kimi had responded to only 19 of 75 tasks by hour 5.75, serving as a reviewer rather than an independent collaborator. The cap reduces the any-medal gap between the two starting orders from 21.3 to 4.0 points (Appendix F.1). The cap at k=5k=5 allows sufficient exploration before switching: GPT → Kimi earns 49, 50, and 53 medals at caps of 1, 3, and 5 accepted experiments, respectively (Appendix F.2).

Final selection. With exploration trajectories fixed, final review raises any-medal retention from 311/330 (94.2%) to 319/330 (96.7%) among runs that archived a medal-winning candidate (Appendix F.4). Exploration therefore already retains medal-winning results in most cases, and final review improves their retention further.

AuthorsOpenReview ↗

Response to Reviewer QneH (Continued)

W2 & Q2. Independent runs and paired uncertainty

We show that the main HMA gain is reliable: over three independent runs, Opus → GPT reaches 78.2% with an across-run SE of 0.4 points, and its task-level paired gain over Opus /goal is +5.8 points with a 95% confidence interval of [+1.3, +11.1]. We report the number of runs and the across-run uncertainty for each HMA configuration. Two configurations have three independent runs, with across-run SEs of 0.4 and 2.0 points. We currently have one run for each of four configurations: GPT → Opus, GPT → Kimi, GPT → GLM, and DeepSeek V4.1 Flash → Kimi. We are running additional repeats during the rebuttal period. All six single-agent /goal baselines have three independent repeats, and Table 2 reports their standard errors.

Across three runs, Opus → GPT earned 58/75, 59/75, and 59/75 medals, for a mean of 78.2% with an across-run SE of 0.4 percentage points, 5.8 points above Opus /goal (72.4 ± 1.6). GPT → DeepSeek V4.1 Flash averaged 70.7% (SE 2.0), 6.0 points above the mean of its two constituent /goal baselines (64.7 ± 2.1). At the task level, Opus → GPT earns a medal more often than Opus /goal on 10 of the 75 tasks and less often on 3.

GPT → Opus also exceeds both of its constituent agents, including the stronger Opus /goal: 73.3% versus 72.4% on all 75 tasks and 86.4% versus 80.3% on Lite, and it is 3.1 points above the mean of the two constituents. This configuration mainly shows the generality of the method: HMA applies directly to either starting order of the same pair and to other model pairs. All six configurations in Table 2 exceed the mean of their two constituent /goal baselines across the 75-task benchmark by 3.1 to 9.6 percentage points.

HMA configurationAny-medal rate by runMean (%)Across-run SE (pp)
Opus → GPT77.3%, 78.7%, 78.7%78.20.4
GPT → DeepSeek V4.1 Flash72.0%, 73.3%, 66.7%70.72.0

In the revision, we will state the number of repeats for each configuration and the across-run SEs in Appendix C.3.

W3. Evidence relevant to the cooperative-access assumption

We find evidence of cooperative access in every tested pair. Incoming agents improved the best-so-far leaderboard percentile in 276 of 1,437 eligible post-handoff options (19.2%, ranging from 12.6% to 25.9% across configurations). HMA’s any-medal rate exceeded the mean of its constituent /goal baselines in 107 of 450 configuration–task comparisons and fell below it in 34.

HMA configurationPost-handoff options improving the best-so-far percentileTasks above / below the constituent /goal mean
Opus → GPT37/25118 / 2
GPT → Opus59/23215 / 5
GPT → DeepSeek V4.1 Flash54/29520 / 6
GPT → Kimi52/20120 / 6
GPT → GLM39/18017 / 6
DeepSeek V4.1 Flash → Kimi35/27817 / 9
Total276/1,437 (19.2%)107 / 34 of 450

A post-handoff option counts as an improvement if a valid submission exists at the handoff and the incoming agent produces a submission with a leaderboard percentile strictly above the best recorded before the handoff. For each task, we compare HMA’s any-medal rate with the mean across three /goal runs per constituent model.

On INGV Volcanic Eruption Prediction, a concrete case study, GPT and Opus reused retrieval features and cached predictions across sessions. They improved the best MAE in five of six post-handoff options and reached 34,234, 46.5% below the best of the six single-agent runs (Section 4.4 and Appendix G.1). GPT and DeepSeek reused saved predictions for blending on Google QUEST. Kimi stacked GLM’s predictions on PetFinder and repaired DeepSeek’s inherited solver on Smartphone Decimeter.

In short, the handoff improvements and task-level gains above provide empirical support for the cooperative-access mechanism in Assumption 3.2: one agent creates useful states that its partner can exploit to make further progress. Under Assumptions 3.1 and 3.2, Theorem 3.3 formalizes when this cooperation enables alternation to exceed the single-agent reference ceiling. We will incorporate the above empirical results into the revised manuscript to strengthen the connection between the theoretical assumptions and observed behavior.

We hope these additions address your concerns, and we would be glad to discuss any further questions.

I appreciate the authors' response and their careful answers to my questions. I think my concerns are resolved

Thank you for taking the time to read our rebuttal and for letting us know that it addressed your concerns.

As promised, we have completed two additional GPT → Kimi runs. The three runs reach 70.7% (Table 2), 70.7%, and 68.0% any-medal, for a mean of 69.8% (SE 0.9 percentage points), 8.7 points above the mean of the two constituent /goal baselines (61.1%). GPT → Kimi thus joins Opus → GPT and GPT → DeepSeek V4.1 Flash in showing that the gain is stable across runs, as in our analysis for W2 & Q2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers and the Area Chair. Below we summarize the contributions of HMA and answer the questions shared by several reviewers, together with the experiments added in this rebuttal; detailed responses appear under each review.

Contributions

  • A training-free, plug-and-play way to combine frontier coding agents. HMA alternates two heterogeneous agents, each in its own native harness, over a shared workspace, with a fixed interruption cap and a final review. It requires no fine-tuning and no task-specific design.
  • Theory that explains when alternation helps. Our context-reset semi-Markov decision process (CR-SMDP) gives sufficient conditions under which alternation exceeds the best single agent (Theorem 3.3) and fixed-k interruption retains this advantage (Theorem 3.5 and Corollary 3.6).
  • State-of-the-art results with a quarter of the time budget. On 75 MLE-bench tasks, Claude Opus 5 → GPT-5.6-sol reaches a state-of-the-art any-medal rate of 78.2% in six hours, exceeding the strongest open-source baseline by 8.0 points with a quarter of its time budget and the stronger single agent (72.4%) by 5.8 points. Across six configurations spanning five heterogeneous agent pairs, HMA improves over the mean of its constituent agents by 6.4 points on average.

Common questions

1. Heterogeneous models versus restarts [2zub W1 & Q1; QneH W1 & Q1; ER6M W1 & Q1]

Model switching, not restarts alone, drives the gain on hard tasks. We ran same-model alternation for Opus, GPT, and Kimi under the same workflow as HMA. Restarts help (Opus +3.6, GPT +4.0, and Kimi +1.8 points over their /goal baselines) but do not reach the hardest tasks: Opus → Opus matches Opus /goal on High-complexity tasks (46.7%). Switching to GPT adds a further 2.2 points overall and the entire 15.5-point gain on High-complexity tasks.

2. Repeated runs and uncertainty [QneH W2 & Q2; ER6M W3]

Our main result holds across independent runs. Opus → GPT and GPT → DeepSeek V4.1 Flash were each evaluated in three independent runs. Opus → GPT averaged a 78.2% any-medal rate (SE 0.4), a gain of 5.8 percentage points over Opus /goal (task-level paired 95% CI [+1.3, +11.1]). Across all six configurations, gains over the mean any-medal rate of the two constituent agents range from 3.1 to 9.6 percentage points.

3. Cooperative access and the link between theory and algorithm [2zub W2, W3 & Q3; QneH W3; ER6M Q3]

Every tested pair exhibits the cooperative access in Assumption 3.2. Across the six configurations, incoming agents improved the best-so-far result after 19.2% of eligible handoffs (12.6% to 25.9% per configuration). HMA’s any-medal rate exceeded the mean of its constituent /goal baselines in 107 of 450 configuration–task comparisons and fell below it in 34. Our CR-SMDP theory gives sufficient conditions for alternation to exceed the best single agent (Theorems 3.3 and 3.5); HMA is its practical fixed-k policy, and we extend the guarantee to the returned result (ER6M Q3).

4. Why a fixed interruption cap [2zub Q2; ER6M W2 & Q2]

A fixed cap reduces sensitivity to starting order while leaving room for productive exploration within each session. In the six-hour GPT–Kimi experiments, a cap of five experiments narrows the any-medal gap between the two starting orders from 21.3 to 4.0 percentage points (Appendix F.1). The first five experiments retain 97.83% of the observed within-session gains (Appendix B.2), and k = 5 achieves the highest any-medal rate among the three caps tested with GPT first (Appendix F.2).

These results support the paper’s central claim: a lightweight, training-free alternation of heterogeneous agents extends what a single frontier agent achieves under a fixed budget. We will incorporate all of the above experiments and clarifications into the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.