acceptodds
Under review as a conference paper at ICLR 2027

How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?

Abstract

Recent autonomous machine learning engineering (MLE) agents have made significant progress on public leaderboards. Often motivated by progress stagnation over long-horizon cycles and limited Large Language Model (LLM) primitives, modern MLE agents are deployed on top of increasingly elaborate machinery: multi-agent orchestrators, dedicated retrieval subagents, and more. While such harnesses expand, the use of more primitive but improved coding agents — where LLMs have direct access to the execution environment by using read, write, and bash primitives — have not received enough attention in the field. In this paper we find that, under equal time budget and frontier LLM backbone, open-source state-of-the-art harnesses provide no advantages over a single session of a minimal-harness coding agent baseline, pointing to the backbone as the primary driver for performance. Via a series of large-scale systematic ablation studies, we argue that the machinery layers become redundant in the coding agent setting. We conclude that the effort spent elaborating hand-crafted harnesses around strong models yields poor returns for current MLE benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agents, automl, llm, mle-bench, mle, coding agents, evaluations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PLzSOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how much task-specific orchestration strong coding agents require for autonomous machine learning engineering. The authors introduce Malena, a minimally orchestrated, single-session agent built on OpenCode, and evaluate the effects of execution interfaces, explicit search strategies, autonomy, and multi-agent coordination. They additionally compare Malena against four specialized MLE systems on MLE-bench and evaluate three systems on NatureBench. The results show that direct access to an execution environment provides substantial benefits for capable backbones, while additional orchestration does not consistently improve performance over the single-session baseline. The paper also examines candidate-selection gaps, inference costs, and agent trajectories to investigate potential explanations and trade-offs. Its main empirical contribution is to establish a strong minimal-harness baseline and question whether increasingly elaborate MLE-specific scaffolding provides reliable marginal benefits.

Critical strengths

The paper addresses an important and practically relevant research question. Improvements in agent performance are often accompanied by changes in both backbone models and orchestration frameworks, making their respective contributions difficult to assess. By explicitly investigating the marginal value of harness components, this work addresses a meaningful gap in agent evaluation. Establishing when additional architectural complexity is justified is valuable for both research methodology and practical system development.

The empirical investigation is broad and makes a substantial effort toward comparable evaluation. The study combines component-level ablations with end-to-end comparisons against multiple specialized MLE systems, rather than relying exclusively on simplified implementations of competing approaches. Matching backbones, execution resources, and wall-clock budgets in the principal comparisons strengthens their interpretability. The additional experiments on context limits, tool configurations, and NatureBench broaden the investigation, while the documented compatibility fixes and configuration checks for external systems help address concerns about baseline integration.

The proposed baseline is practically useful and achieves substantial gains in important settings. Malena provides a comparatively simple reference system that future MLE-agent work should consider. Its advantages are not limited to small numerical differences: with GLM 5.2 on the fixed30 task set, it achieves a 62.5% self-selected medal rate, compared with 47.1% for the strongest external system on that metric. The reported paired confidence interval supports a substantial advantage in this setting. These results demonstrate that a capable single-session agent is a serious baseline, even without establishing the paper’s broader interpretation about harness redundancy.

The distinction between candidate generation and candidate selection yields a valuable insight. Reporting both self-selected and oracle performance reveals that producing strong candidates does not necessarily translate into selecting a strong final submission. The near-identical oracle percentiles of Malena and Best-of-N, alongside their noticeably different self-selected results, highlight validation comparability and selection reliability as potentially important bottlenecks. This analysis provides a more informative perspective than reporting final performance alone and motivates concrete directions for future research.

Critical weaknesses
  1. The “post-training ate the harness” interpretation lacks direct causal evidence. The paper does not conduct controlled training experiments demonstrating that coding-agent post-training substitutes for explicit orchestration. Although multiple backbones and model versions are evaluated, these comparisons cannot disentangle post-training effects from differences in pretraining, model capacity, context length, or tool-use capability. Observing that a stronger model performs well with a minimal harness is not sufficient to establish that post-training has internalized the functions previously supplied by that harness. While Appendix G.2 appropriately presents this explanation as a hypothesis, it remains unverified and should not be treated as an established explanation for the empirical results. A more convincing study would compare the same backbone at different stages of agentic post-training under matched harness configurations.

  2. The model coverage is insufficient to support broad conclusions about frontier agents. The evaluation omits contemporaneous frontier models from the GPT and Claude families. This is particularly important because the paper’s own results show that the relative value of a harness depends on the underlying model: specialized scaffolding benefits the weaker Gemma backbone, whereas Malena performs more competitively with stronger backbones. Consequently, the findings cannot be assumed to generalize to other leading models with different tool-use behaviors and training distributions. Including current GPT and Claude models would substantially strengthen the external validity of the conclusions; otherwise, the claims should be explicitly restricted to the evaluated backbones.

  3. The comparisons do not establish the marginal value of mainstream coding-agent harnesses. Although the paper evaluates four specialized MLE systems, it does not compare against widely used general-purpose coding-agent harnesses such as Codex or Claude Code. Moreover, Malena itself is built on OpenCode and therefore already benefits from a substantial coding-agent runtime. The experiments primarily test whether additional MLE-specific orchestration improves upon this runtime, rather than whether harness functionality has become unnecessary. Without comparisons across mature general-purpose harnesses, the study cannot determine how much performance depends on their execution, context-management, and tool-interaction mechanisms, or on model–harness compatibility. Where technically feasible, evaluating the same backbone across such harnesses would help separate model capability from runtime quality. The current evidence supports a narrower claim about the tested MLE-specific additions, not a general conclusion that strong models have absorbed the value of harness design.

  4. The ablations do not cleanly identify the mechanisms responsible for the gains. Despite the extensive ablation suite, several central comparisons change multiple factors simultaneously. In particular, moving from Oneshot-based search to Malena changes the production-run constraint, session continuity, documentation requirements, and tool configuration, while potentially affecting artifact reuse and validation consistency. The resulting gains therefore cannot be attributed specifically to greater autonomy or the removal of explicit search. This ambiguity is especially consequential because Malena and Best-of-N achieve nearly identical oracle percentiles but substantially different self-selected performance, suggesting that candidate selection may explain much of the advantage. A factorial comparison independently controlling search policy, session persistence, artifact access, and validation protocol would provide a substantially stronger account of why Malena works.

Questions for authors

see in weakness

5 comments

We thank the reviewer for the careful and constructive review. We recognise the technical validity of the criticisms raised, and we address each of them below and in the revised manuscript.

Post-training ate the harness

We agree that the post-training explanation is a hypothesis that our experiments cannot establish causally, and the original presentation may have unintentionally put more emphasis on it than we meant: it is a best-effort interpretation of our results that is open to discussion. In the revised manuscript we removed it from the list of contributions, and the conclusion now states clearly that it is a hypothesis and not a claim (see the general response for the wording changes). We defer verification to future work.

Model coverage

Indeed, we cannot make claims about frontier models we have not evaluated, and, as the reviewer suggests, the revision restricts our claims to the evaluated open-source backbones (see the general response). We agree that the value of a harness depends on the backbone, and our results indicate where: the one backbone at which a harness significantly outperforms Malena, Gemma 4 31B, is also the one that gains least from a coding-agent environment (Chat vs. Oneshot, Figure 2), whereas at the three backbones with a large gain, no harness is significantly better than Malena. Kimi K3, the strongest of these, is competitive with GPT-5.5 and Claude Opus 4.8 on coding and agentic benchmarks such as Terminal-Bench and DeepSWE according to its release report. So, we suspect the trend would hold for closed models as well, and we are committed to verifying it with the evaluation below.

Planned: subject to our internal approval process, we commit to adding a full evaluation of at least three harnesses with Gemini 4 Argon in the camera-ready version.

Coding-agent harnesses

The coding-agent environment plays a major role in the success of Malena, and a strong model alone without tools to interact with the environment may not be enough to reach frontier performance, as shown in Figure 2. Our main claims rely on the presence of such a coding-agent environment, but we do not rely on the specific design of OpenCode. Two independent studies (https://harnesstax.github.io) and (https://arxiv.org/abs/2610.04433) report no significant difference in capability between coding-agent harnesses. A systematic comparison of different coding agents is outside the scope of this paper.

Planned: we will execute Malena with Pi Agent before the end of the rebuttal period to validate this stance.

Mechanisms responsible for the gains

We agree that the step from Oneshot-based search to Malena changes several factors at once. This is by design. Our autonomy intervention is precisely letting the agent decide these things itself instead of the harness fixing them. Our claim from this step is therefore about the bundle, not about any single factor. We edited the wording in Section 4.3 to state this explicitly and to avoid attributing the gain to autonomy or search removal specifically. Our other claims compare designs within a fixed iteration type and do not depend on this decomposition. While the exact performance decomposition of the individual factors falls out of the scope of this work, the existing evidence covers several already.

  • Tools: adding the system and jobs tools to Oneshot iterations yields no significant difference (Appendix C.8, Table 14), so tool configuration is unlikely to explain the gap.
  • Validation consistency: a shared baseline workspace (BI, Appendix C.3) reduces Oneshot's selection gap from 5.43 to 2.02, at the cost of a lower oracle ceiling. This is consistent with validation comparability driving the selection gap.
  • Session continuity: capping Malena's context at 64k with repeated compaction lowers its medal rate (57.5% -> 50.0% self-selected; Appendix C.7). The CIs overlap, so this is suggestive only.

Planned: to isolate the remaining factors, we are running targeted ablations where we hold the production-run constraint fixed to account for the other mechanisms. We will update the manuscript when the results are available.

Summary

The review describes our question as important, our evaluation as broad and comparable, and Malena as a baseline that future work should consider, and it rates the paper as informative ("I learned something interesting or useful") with only minor or fixable soundness concerns. The weaknesses it lists concern how we interpret (W1) and scope (W2) the findings, and additional experiments (W3, W4), rather than the findings themselves; the revision narrows our claims accordingly and adds the requested evidence where we can. We therefore find a clear rejection difficult to reconcile with this assessment, and we would respectfully ask the reviewer to revisit the overall rating, and to let us know which points weigh most in it, so that we can address them as fully as possible.

Thank you for the clarification and revisions. My main concern is not the absence of a few optional additional experiments, but the remaining gap between the paper’s central conclusions and its experimental coverage. I recognize Malena’s value as a strong baseline and its advantages in the evaluated settings. However, Section 6 goes further, arguing that elaborating harnesses yields “poor returns” and that “the leverage is in the model and in the runtime it is given, not in the scaffolding built around them.” These statements are substantially stronger than showing that Malena outperforms several tested MLE systems.

The principal comparisons cover selected open-source backbones and four MLE-specific frameworks, but omit Codex and Claude Code paired with their respective contemporaneous model versions—mainstream configurations that are directly relevant to the paper’s practical claims. This is not merely a request to include newer systems released after submission: Codex and Claude Code were already prominent, widely used coding-agent systems during the pre-submission period when the paper would have been developed, and remain so today. Their omission therefore leaves out central practical reference points, rather than peripheral alternatives. Without comparisons against these mainstream model–harness configurations, the broader conclusion that further harness development offers poor returns is not convincing. Kimi’s competitiveness with GPT and Claude on other benchmarks does not resolve this concern: similar benchmark performance does not establish similar responses to orchestration, context management, or tool interfaces. Indeed, the paper’s own finding that harness benefits depend on the backbone makes it particularly important to evaluate representative model–harness combinations directly, rather than extrapolate from the selected open-source backbones.

Moreover, Malena already relies on the runtime capabilities provided by OpenCode. The current experiments therefore primarily assess the marginal value of particular MLE-specific orchestration mechanisms relative to an OpenCode-based baseline, rather than the broader value of mature harness design. Distinguishing the underlying “runtime” from additional “scaffolding” is reasonable, but the resulting conclusions must respect that distinction. Showing limited benefits from the tested additions does not establish that harness development more generally offers poor returns.

If the authors retain these broader practical conclusions, evaluations covering representative leading models and mature coding-agent harnesses are central supporting evidence, rather than merely out-of-scope extensions. I welcome the proposed experiments and would reassess the conclusions based on their results. If such comparisons cannot be completed, the conclusions should be narrowed further, consistently positioning the paper as a baseline and comparative empirical study under the specific models, budgets, and task settings evaluated.

Thank you for the follow-up and for identifying the two Section 6 sentences. We agree that they made the conclusion broader than the abstract, and have corrected them in the manuscript.

Scope. Our question is whether MLE-specific search, orchestration, selection and context-management machinery provides additional performance once a capable coding-agent runtime is available—not whether harnesses in general are unnecessary. Malena relies on OpenCode's runtime. Section 6 now explicitly refers to the poor returns of elaborating MLE-specific harnesses on current MLE benchmarks, and distinguishes the runtime from MLE-specific scaffolding. The conclusion now matches the abstract; the claim remains scoped to the tested models, tasks and budgets.

Evidence. We test the marginal value of these mechanisms in two complementary ways. First, we compare Malena with four released MLE harnesses under matched backbone, hardware and budget. Across 22 frontier-backbone–harness–metric comparisons, 15 significantly favor Malena after Holm correction over all 26 comparisons; none significantly favors an external harness. Second, we ablate MLE-specific interventions within a fixed coding-agent runtime.

The new 12-task held-out evaluation extends the comparison to unseen tasks and two backbones; on DeepSeek V4 Flash, Malena beats MLEvolve on all four metrics after correction (Appendix D.5). We also ran Malena on a second coding-agent backend, Pi: on one backbone and ten tasks, it significantly outperforms Arbor, MLEvolve and ScienceFlow on both metrics, and AiScientist on percentile (Appendix D.2). This is evidence that the comparative result is not unique to OpenCode, although the component ablations remain OpenCode-specific.

Codex and Claude Code. We agree that testing leading proprietary model–harness combinations would strengthen external validity. But those comparisons would not, by themselves, isolate the incremental value of MLE-specific machinery; that requires matching the underlying coding agent when adding the machinery. We do not claim generalization to untested proprietary combinations. Institutional restrictions prevent us from evaluating Codex or Claude Code with their closed models; as noted in the general response, we are committed to a Gemini 4 Argon evaluation in the camera-ready version, subject to approval.

Mechanisms. We also added an ablation between chained one-shot iterations and Malena (Appendix C.8). Carrying the session across links does not by itself change the chain's results, which stay below Malena even though the continued session receives the same prescriptive per-link instruction. Reusing artifacts across iterations recovers most of the gap, mainly on the oracle metrics. One reading is that what helps is not remembering the conversation but being able to build on earlier work products, and that prescriptive per-link instructions may cost something even when the session continues. Other differences between the chains and Malena remain, so we present this as a hypothesis about how autonomy, continuity and artifact reuse combine, not as a causal conclusion; isolating them would require further experiments.

We believe this supports a meaningful practical conclusion: MLE-specific complexity should demonstrate gains over a strong coding-agent baseline rather than be presumed beneficial. We hope the new experiments and corrected scope address your concerns, and would appreciate your reconsideration of the assessment.

Reviewer_uATjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the elaborate machinery of autonomous ML engineering (MLE) harnesses (search trees, multi-agent orchestration, memory management) still helps when the backbone is a strong coding agent. All comparisons hold the backbone, the hardware (1×A100), and the time budget (24h) fixed. The proposed baseline, Malena, is a single long OpenCode session with three small tools: submission registry, system/time status, and background jobs. On about 30 MLE-bench tasks, an ablation ladder re-implements common harness components and shows the following. Moving from a chat interface to a coding-agent environment is by far the largest gain. Tree search over agent iterations, delegation, and parallel workers add no significant improvement. A single autonomous session reaches the highest medal rate. Against four released harnesses (MLEvolve, AiScientist, Arbor, ScienceFlow), Malena matches or beats them on three strong open-weight backbones, for example a 62.5% vs. 47.1% medal rate on GLM 5.2. It is outperformed on the weaker Gemma 4 31B. The authors conclude that the backbone and the coding-agent runtime, not the hand-crafted harness, drive performance on current MLE benchmarks.

Critical strengths
  1. It provides matched comparison of released MLE harnesses. Published MLE-bench numbers usually differ in backbone, hardware, and seeds, so harness gains cannot be attributed. Here every method runs on the same model, the same GPU, and the same budget, with paired-by-task CIs and documented fixes for each external harness. This makes the headline result (e.g., +16.7 pp medal rate over Arbor on GLM 5.2, significant) credible in a way most leaderboard claims are not.

  2. Its ablations locate the gain. Separating interface, search topology, autonomy, and coordination shows that the coding-agent environment accounts for most of the improvement, while search topology barely matters. This tells harness designers where effort pays off.

Critical weaknesses
  1. The claim is broader than the evidence. Harness details clearly matter: adding time nudges alone drops the medal rate from 59.4% to 48.9%. Malena also runs on OpenCode, a full agent runtime. The evidence supports "MLE-specific search adds little on top of a good coding runtime", not "little harness is needed". To improve: narrow the claim.

  2. Two of the four harnesses are not tested on the models they were built for. MLEvolve (Gemini 3.1 Pro) and Arbor (GPT-5.5) were developed on closed models, but the paper runs them only on open-weight models. Porting is fragile: MLEvolve's medal rate on DeepSeek V4 Flash changes from 0% to 25% from the reasoning-effort setting alone. Malena does win on the native model families of AiScientist (GLM) and ScienceFlow (DeepSeek V4 Flash), which supports the paper, but the comparison against the two closed-model harnesses may reflect a backbone mismatch. To improve: compare against MLEvolve and Arbor on at least one closed frontier backbone.

Questions for authors
  1. How does Malena compare with MLEvolve and Arbor on a closed frontier backbone such as Gemini 3.1 Pro or GPT-5.5, the models they were originally developed on?
  2. Were Malena's prompt and tools designed on the evaluation tasks? Can you report results on held-out MLE-bench tasks not used during development?
  3. Will you release Malena (prompts, tools, OpenCode configuration) and the patched harnesses?
4 comments

We thank the reviewer for their time and thoughtful feedback. We respond to each point below, and mark what changed in the revised manuscript.

The claim is broader than the evidence

We agree that harness design is important in MLE systems, and Malena is itself a minimal coding agent harness. We refer to the section of the general response on the breadth of the claim, where we clarify what we claim and list the changes made to the manuscript.

Harnesses not tested on native models

Our access to closed frontier models is restricted by institutional policies (see the general response), so we address this concern in two ways. First, we want to stress that we document our actions carefully: the changes to the harnesses are infrastructural and do not alter their search logic (Appendix D.3), and we report tuning ablations in Appendix D.4. Second, as the reviewer points out, Malena wins on the native model families of AiScientist (GLM) and ScienceFlow (DeepSeek V4 Flash). Moreover, MLEvolve on Gemma 4, a non-native backbone, outperforms Malena, which suggests that our adaptation does not systematically handicap it for a non-native backbone.

On the sensitivity of MLEvolve to reasoning effort with DeepSeek V4 Flash (0% to 25% medal rate), the cause is length: at maximum effort, reasoning is about 5× longer per call, so MLEvolve completes about 3× fewer search nodes in the same budget (18.5 vs. 62.8 per run), while node quality is unchanged (buggy-node rate 61% vs. 62%; Appendix D.4). This is why we use medium effort for MLEvolve there.

Planned: subject to our internal approval process, the camera-ready version will include a reproduction of MLEvolve with Gemini 3.1 Pro in its native 12h setting. Unfortunately, we will not be able to provide Arbor with GPT 5.5. We also plan more DeepSeek reasoning ablations, to check whether the reasoning setting affects the other harnesses.

Malena designed on the evaluation tasks

The default29 set was used during development, and results of earlier versions of Malena on it were known to us when designing the final version. However, the prompt and tool design were not explicitly designed to overfit to this set. NatureBench, where our main claims hold for the backbones tested, was never used during development. Changes in the revised manuscript:

  • The full Malena prompt (Appendix B.2). It is task-agnostic.
  • A held-out evaluation, as requested (Appendix D.5): Malena, Arbor, AiScientist and MLEvolve with Kimi K3 (24h, three seeds) on 12 MLE-bench tasks outside default29 (alaska2 and 11 tasks outside the 30-task split). Malena has the best oracle percentile and oracle medal rate and ties AiScientist on self-selected medal rate, while AiScientist has the best self-selected percentile. In the paired comparison, Malena is not significantly different from AiScientist on any metric, and it is better than MLEvolve on oracle percentile after correction.

Planned: results for DeepSeek V4 Flash on these tasks are coming soon.

Malena release and harness patches

Please refer to the general response.

Summary. Thank you again for the explicit suggestions on how to improve the paper. Our access to closed frontier models is limited, so we cannot complete the closed-model evaluations during the rebuttal period; we commit to adding the MLEvolve reproduction with gemini-3.1-pro to the camera-ready version, and we have otherwise tried our best to fulfil the requests with the held-out evaluation, the released prompt and the documented changes to the harnesses. We are happy to discuss any remaining concerns about the breadth of our claims, and we would be grateful if the reviewer could consider revising their assessment in light of these changes.

As the discussion period is closing, we provide a short update on the items from our response that are now in the revised PDF (all changes are also listed in the second general comment).

  • Narrowing the claim (W1). Section 6 now carries the same MLE-specific qualifiers as the abstract: "effort spent elaborating MLE-specific harnesses on top of a coding agent that is already strong at working autonomously has poor returns on current MLE benchmarks. The leverage is in the model and in the runtime it is given, not in the MLE-specific scaffolding built around them."
  • Dependence on OpenCode (W1). We reran Malena with the Pi agent in place of OpenCode (GLM 5.2, fixed10 task set, four seeds, Appendix D.2, Table 19). After Holm correction, it significantly outperforms Arbor, MLEvolve and ScienceFlow on both self-selected metrics, and AiScientist on percentile. It is not significantly different from the OpenCode backend, although its point estimates are lower.
  • Porting fragility (W2). We repeated the medium vs. max reasoning comparison on DeepSeek V4 Flash for Malena, Arbor and AiScientist (Appendix D.4, Table 21). For no other harness medium reasoning is better, so the 0%→25% swing is specific to MLEvolve's many short calls, which is why we run MLEvolve at medium.
  • Held-out tasks (Q2). Appendix D.5 now also covers DeepSeek V4 Flash. On the 12 held-out tasks, Malena is significantly better than MLEvolve on all four metrics and than Arbor on both percentiles after Holm correction, and it is not significantly different from AiScientist. With Kimi K3 the signs are the same, and only the oracle-percentile advantage over MLEvolve survives correction.
  • Closed backbones (Q1). To complete our earlier answer: although we cannot run Arbor with GPT-5.5, subject to approval the camera-ready version will evaluate Arbor, together with Malena and AiScientist, on Gemini 4 Argon, a closed frontier backbone. We will also include MLEvolve with Gemini 3.1 Pro in its native 12h setting.
  • Code release (Q3). Please refer to the general response.

If these changes address your concerns, we would be grateful if you could reflect them in your assessment. If anything remains unclear, we are happy to clarify in the time left.

Area_Chair_tQKwOpenReview ↗

Does authors' response address your concerns?

Thank you for your review and the follow-up! Could you check the authors’ new response and clarify whether it addresses your original concerns? Many thanks!

AC

Thank you for the thorough responses and additional experiments. The narrowed claim, the Pi-agent replication, and the held-out evaluation address my concerns about scope and task overfitting; on held-out tasks Malena is competitive with, though not better than, AiScientist, which fits the narrowed claim.

My remaining concern is that the planned evaluations and code release are subject to approval, so I cannot assess them yet.

Overall, this is a careful and useful study, and I maintain my score of weak acceptance.

Reviewer_KdmqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper asks how much extra structure a strong coding agent needs for autonomous machine learning engineering. The authors introduce Malena, a single-session agent built on OpenCode. They test several design choices and compare Malena with four existing harnesses using the same backbone model, task hardware, and time budget. They evaluate the systems on MLE-bench and NatureBench.

The results show that a simple, well-designed coding agent can match or beat more complex systems in the tested settings, especially with GLM 5.2. However, they do not fully support the broader claim that complex harnesses are redundant or offer no advantage.

Critical strengths

The paper studies an important question under controlled conditions. Many agent papers change the model, hardware, and harness together, making gains hard to explain. This paper keeps the backbone model, task hardware, and time budget fixed to better measure the value of the harness.

Malena is a strong and useful baseline. With GLM 5.2, Malena reaches a self-selected medal rate of 62.5%, compared with 47.1% for AiScientist, the strongest external baseline in this setting. The paired difference is 15.4 percentage points, with a reported 95% confidence interval of [8.7, 21.9]. This is strong evidence for Malena in this setting. Future MLE-agent papers should include a similar single-session baseline.

The paper separates candidate quality from final submission selection. The self-selected score evaluates the submission chosen by the agent. The oracle score evaluates its best submission on the test set. The gap between them shows that final submission selection is a major problem for several systems.

The authors make useful efforts to improve evaluation quality. They fix four broken or leaky MLE-bench tasks for the main external comparisons and report compatibility fixes, configuration experiments, costs, and failed runs. Most experiments use the highest available reasoning setting, with an exception where a lower setting works better for MLEvolve. These details improve transparency.

Critical weaknesses
  1. The main negative claims are stronger than the statistical evidence. A non-significant difference does not show that two methods are equal. With DeepSeek V4 Flash, Malena's medal-rate difference from Arbor is -5.6 percentage points, with a 95% confidence interval of [-13.3, +2.2]. This interval allows a meaningful disadvantage for Malena and does not show that it is "no worse." The bootstrap keeps the task set fixed and resamples runs within each task. It measures uncertainty on these tasks, not on new tasks. With only 3–4 runs per task in the 24-hour setting, negative results need caution. The paper should also explain how it handles multiple testing.

  2. The fairness of the comparison needs stronger support. Malena was developed on 29 of the 30 main comparison tasks. The external harnesses needed adaptation to new models and local serving infrastructure. The reported fixes and tuning are useful, but they do not show that every adapted harness performs as expected. Some configuration checks are small, and the reasoning setting has a large effect in at least one case.

  3. The results do not easily support a general claim that harnesses are redundant or this could be stronger. Malena is itself a harness, with prompts, context management, resource tools, a submission registry, and a continuation loop. These choices matter: removing the system and jobs tools lowers the medal rate by 8.9 percentage points. The results support careful harness design, even if some extra coordination is not useful.

Questions for authors

none

4 comments

We thank the reviewer for their time and the weaknesses highlighted. We respond to each below, and mark what changed in the revised manuscript.

Statistical evidence

We agree that a non-significant difference does not show that two methods are equal, and that the current phrasing of the submission may not necessarily reflect this. Note that our statistical protocol (in Appendix A.5) already says only that such arms are "statistically indistinguishable at our power level", but the abstract ("provide no advantages"), Sec. 5.1 ("matches or beats") and the Limitations ("no worse than claims are well supported") read non-significance as parity. We added a new submission revision (colour highlighted in blue to mark additions) in which we modify:

  • The wording. Now every parity statement says only what the intervals support. The abstract now reads that Malena significantly outperforms other harnesses in most backbone–harness pairs. The same change is made in the intro, Section 5.1, the conclusion and the Limitations. "No worse than" is removed.
  • A per-comparison account. Section 5.1 now counts outcomes over the 22 frontier comparisons (11 backbone–harness pairs, each on self-selected percentile and self-selected medal rate). After correcting for multiple comparisons, Malena results are significantly better in 15 (all eight at GLM 5.2), the remaining seven are inconclusive, and no harness is significantly better than Malena in any of them. The four with negative point estimates are all DeepSeek V4 Flash against Arbor and AiScientist. At the smaller Gemma 4 31B backbone, MLEvolve is significantly better than Malena on percentile.
  • Multiple testing (new, Appendix D.2). We apply a Holm correction over all 26 MLE-bench comparisons (the 22 frontier ones plus the four at Gemma 4 31B), using two-sided paired-bootstrap p-values from the same protocol. 16 of the 26 are significant: 15 in favour of Malena, including all eight at GLM 5.2, and one in favour of a harness (MLEvolve over Malena on percentile at Gemma 4 31B). Among the 22 frontier comparisons, the one that is significant before correction but not after is DeepSeek V4 Flash vs. ScienceFlow on medal rate (p_Holm = 0.255).
  • Ablations (new, Appendix A.5). We apply the same correction within each ablation table (search strategies, backbones, infrastructure and orchestration), and † marks the comparisons that survive it. The medal-rate advantage over Best-of-N and the medal-rate loss from time nudges remain significant. The percentile advantage over Best-of-N, and the other search strategies on medal rate, do not.
  • Seeds and task set. The 3–4 seeds per task is a cost limit that we state in the Limitations. The intervals hold the task set fixed, as you note; for generalisation to new tasks, see the held-out evaluation under "Fairness of the comparison" below.

Fairness of the comparison

Indeed, Malena was developed on a subset of the tasks on which all harnesses are evaluated. Changes in the revised manuscript:

  • A held-out evaluation on 12 MLE-bench tasks (Appendix D.5; see the response to uATj and the general response).
  • The full Malena prompt (Appendix B.2). It is task-agnostic, and its introduction and guidelines are inspired by the header of MLEvolve's prompt.
  • The changes to the external harnesses (infrastructure only, Appendix D.3) and the tuning ablations (Appendix D.4).

We highlight the case of ScienceFlow, which was designed to be used with DeepSeek V4 Flash: our experiments are the closest a third party could reproduce its results with the available hardware, and Malena still performs better.

Planned: for the fairness of the comparison with MLEvolve, which was developed on Gemini 3.1 Pro, and subject to our internal approval process, the camera-ready version will include a reproduction of MLEvolve with Gemini 3.1 Pro on fixed10 in its native 12h setting (see the general response). We also plan more DeepSeek reasoning ablations, to check whether the reasoning setting affects the other harnesses.

General claim

We acknowledge the pushback and refer to the section of the general response on the breadth of the claim, where we clarify what we claim and list the changes made to the manuscript.

Presentation

In order for us to better prepare for a next manuscript revision, we kindly ask the reviewer to provide specifics regarding the Presentation 1 (Difficult to follow despite careful reading) rating.

If these changes address your concerns, we would be grateful if you would reflect this in your assessment.

Thank you again for the review. We follow up on the two points of our first response that were still planned, both now in the revised manuscript.

Held-out tasks (Appendix D.5). The held-out evaluation on 12 MLE-bench tasks now covers two backbones, Kimi K3 and DeepSeek V4 Flash, with unpaired and paired comparisons (Holm-corrected). With DeepSeek V4 Flash, Malena is significantly better than MLEvolve on all four metrics and than Arbor on both percentiles. With Kimi K3 the pattern is the same, and the paired comparison is conclusive only for the oracle-percentile advantage over MLEvolve. Malena was not tuned on these tasks, so these results address the concern that it was developed on the evaluation tasks.

Reasoning effort (Appendix D.4). For DeepSeek V4 Flash, the backbone where the only four negative point estimates of the main comparison occur, we repeated the max vs. medium reasoning-effort comparison for Malena, Arbor and AiScientist on the 10-task split. No difference is significant on any metric, and the point estimates do not point in a consistent direction. We find no evidence that changing reasoning effort would systematically improve these harnesses on this backbone.

If these changes address your concerns, we would be grateful if you would reflect this in your assessment.

Area_Chair_tQKwOpenReview ↗

Does authors' response address your concerns?

Thank you for your review and the follow-up! Could you check the authors’ new response and clarify whether it addresses your original concerns? Many thanks!

AC

Thank you again for the review. Following up on the fairness of the comparison: our earlier checks of Arbor's configuration were small (8 and 16 runs per alternative configuration, on 2 and 4 tasks). Since Arbor is the harness with the most hyperparameters, we ablated it more thoroughly (Appendix D.4). On GLM 5.2 and the 10-task split, we changed one setting at a time: the maximum tree depth (4 to 2), the executor timeout (4h to 12h) and the convergence early stopping (stop_after 8 to 999). These are Arbor's own defaults from its competition-optimization plugin, not values we chose.

None of the three changes improves Arbor. Reducing the tree depth is indistinguishable from the default on all four metrics (self-selected and oracle percentile and medal rate). The longer executor timeout and the disabled early stopping have lower point estimates than the default, with intervals that partly overlap it. We therefore kept Arbor's default configuration for the comparisons in the paper.

If these changes address your concerns, we would be grateful if you would reflect this in your assessment.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their time. We are glad they found the question important, the matched comparison credible, and Malena a useful baseline, and that all of them rated the paper as informative ("I learned something interesting or useful") and as having only minor or fixable soundness concerns. We take the concerns raised seriously and address them below and in the revised manuscript.

On the main claim ("harnesses are redundant"). All three reviews question how broad our claim is, which gives us a chance to clarify it. We do not claim that harnesses are useless or redundant. As the reviewers point out, Malena is itself a harness. Our question is the one in the title, how much of a harness a strong agent needs, and, as the text makes explicit, how much on top of a general-purpose coding agent. The original submission says so in several places, for example:

  • Sec. 5: "[...] a minimal single-session harness against harnesses that add scaffolding on top of agentic sessions or completions."
  • Sec. 6: "which of the accumulated machinery still earns its keep once the primitive can do these things itself"; "almost none from what is built on top of it".

Our answer is not "none": a thin task interface matters in both directions (as the reviewers note, removing the system and jobs tools lowers the medal rate by 8.9 points, significant before Holm correction only, and periodic time reminders lower it by about 10 which remains significant after the correction), while the MLE-specific machinery we test adds no measurable benefit in the autonomous coding-agent setting. We acknowledge that some of our phrasing could have been confusing, and we changed it in the revised submission. The changes related to the main claim are of three types: we added MLE qualifiers, as some reviewers suggested, to the machinery and harnesses we study; we replaced or qualified absolute wording such as "redundant", "every" and "primary driver", for example with the qualifiers "tested", "open-source" and "MLE-specific"; and we made explicit that the claims concern the autonomous coding-agent setting, and harnesses on top of a coding agent (e.g., in the conclusion). We would also welcome any further specific quotes or places in the manuscript that the reviewers think we should revise to better articulate our actual claim.

On the statistical claims. As Reviewer Kdmq pointed out, we have also made our statistical claims more rigorous: we added a Holm correction for multiple comparisons, applied both to the external-harness comparisons and to our ablations (marked with † in the tables; Appendix A.5), and removed the places where a non-significant difference could be read as parity between methods. The correction leaves our main conclusions unchanged and removes some ablation differences (for example, Malena's percentile advantage over Best-of-N).

On new additions. In the revised manuscript we also added the full Malena prompt for transparency (Appendix B.2) and an evaluation on held-out tasks (Appendix D.5) with Kimi K3. Details about additional planned experiments are in the individual responses.

On closed-source models and code release. Unfortunately, our access to closed frontier models is restricted by institutional policies on licenses and third-party models and tools, and the necessary approvals will take longer than the rebuttal period. Subject to them, the camera-ready version will include a full evaluation of at least three harnesses (Malena, Arbor and AiScientist) with Gemini 4 Argon to account for closed frontier backbones, a reproduction of MLEvolve with Gemini 3.1 Pro on fixed10 (12h runs, its native setting) to evaluate MLEvolve's native model, and the release of the infrastructure-agnostic Malena code, including its tools and OpenCode configuration (the full prompt is already in Appendix B.2). Appendix D.3 now describes the changes to the baseline harnesses, which do not alter their search logic. These changes are infrastructural and specific to our cluster, such as the memory (OOM) monitor and prompt notes on the available RAM, so we cannot release them due to institutional policies.

The revised submission, with all changes highlighted in blue, is published to OpenReview.

Update on the manuscript since our first response. The revised manuscript contains the following further changes. They fulfil the planned items announced in our individual responses (DeepSeek held-out results, Pi Agent, DeepSeek reasoning ablations).

  • Held-out evaluation (Appendix D.5). The held-out tasks are now evaluated with two backbones, Kimi K3 and DeepSeek V4 Flash, in both the unpaired and the paired (Holm-corrected) comparison. The pattern is the same for both: Malena is competitive with or better than the other harnesses. With DeepSeek V4 Flash it is significantly better than MLEvolve on all four metrics, while with Kimi K3 only the oracle-percentile advantage over MLEvolve survives Holm correction.
  • Reasoning effort (Appendix D.4). We compare medium and max reasoning effort on DeepSeek V4 Flash for Malena, Arbor and AiScientist on the 10-task split. No difference is significant on any metric. We find no evidence that changing reasoning effort would systematically improve these harnesses on this backbone.
  • Pi Agent (Appendix D.2). We ran Malena with the Pi agent as the coding-agent backend. Against the external harnesses, the comparison replicates on this second backend (one backbone, ten tasks): Malena with Pi significantly outperforms Arbor, MLEvolve and ScienceFlow on both metrics and AiScientist on percentile, providing evidence that the comparative advantage is not specific to OpenCode.
  • Session continuity and artifact reuse (Appendix C.8). We add a more granular step between chained one-shot iterations and Malena, separating session continuity from artifact reuse across iterations. Continuity alone does not close the gap to Malena, while reusing artifacts recovers most of it; Malena remains highest in point estimates on three of the four metrics.
  • Wording. The practical-reading sentences in Section 6 now carry the same MLE-specific qualifiers as the abstract, bridging the gap between the abstract and the conclusion that we left in the first revision, and a few typos were fixed.

Update on the manuscript since our second response. We added further tuning ablations for Arbor (Appendix D.4) on GLM 5.2 and the 10-task split, changing one setting at a time: maximum tree depth (4 to 2), executor timeout (4h to 12h) and convergence early stopping (stop_after 8 to 999). None of them improves Arbor.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.