- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how much task-specific orchestration strong coding agents require for autonomous machine learning engineering. The authors introduce Malena, a minimally orchestrated, single-session agent built on OpenCode, and evaluate the effects of execution interfaces, explicit search strategies, autonomy, and multi-agent coordination. They additionally compare Malena against four specialized MLE systems on MLE-bench and evaluate three systems on NatureBench. The results show that direct access to an execution environment provides substantial benefits for capable backbones, while additional orchestration does not consistently improve performance over the single-session baseline. The paper also examines candidate-selection gaps, inference costs, and agent trajectories to investigate potential explanations and trade-offs. Its main empirical contribution is to establish a strong minimal-harness baseline and question whether increasingly elaborate MLE-specific scaffolding provides reliable marginal benefits.
Critical strengths
The paper addresses an important and practically relevant research question. Improvements in agent performance are often accompanied by changes in both backbone models and orchestration frameworks, making their respective contributions difficult to assess. By explicitly investigating the marginal value of harness components, this work addresses a meaningful gap in agent evaluation. Establishing when additional architectural complexity is justified is valuable for both research methodology and practical system development.
The empirical investigation is broad and makes a substantial effort toward comparable evaluation. The study combines component-level ablations with end-to-end comparisons against multiple specialized MLE systems, rather than relying exclusively on simplified implementations of competing approaches. Matching backbones, execution resources, and wall-clock budgets in the principal comparisons strengthens their interpretability. The additional experiments on context limits, tool configurations, and NatureBench broaden the investigation, while the documented compatibility fixes and configuration checks for external systems help address concerns about baseline integration.
The proposed baseline is practically useful and achieves substantial gains in important settings. Malena provides a comparatively simple reference system that future MLE-agent work should consider. Its advantages are not limited to small numerical differences: with GLM 5.2 on the fixed30 task set, it achieves a 62.5% self-selected medal rate, compared with 47.1% for the strongest external system on that metric. The reported paired confidence interval supports a substantial advantage in this setting. These results demonstrate that a capable single-session agent is a serious baseline, even without establishing the paper’s broader interpretation about harness redundancy.
The distinction between candidate generation and candidate selection yields a valuable insight. Reporting both self-selected and oracle performance reveals that producing strong candidates does not necessarily translate into selecting a strong final submission. The near-identical oracle percentiles of Malena and Best-of-N, alongside their noticeably different self-selected results, highlight validation comparability and selection reliability as potentially important bottlenecks. This analysis provides a more informative perspective than reporting final performance alone and motivates concrete directions for future research.
Critical weaknesses
-
The “post-training ate the harness” interpretation lacks direct causal evidence. The paper does not conduct controlled training experiments demonstrating that coding-agent post-training substitutes for explicit orchestration. Although multiple backbones and model versions are evaluated, these comparisons cannot disentangle post-training effects from differences in pretraining, model capacity, context length, or tool-use capability. Observing that a stronger model performs well with a minimal harness is not sufficient to establish that post-training has internalized the functions previously supplied by that harness. While Appendix G.2 appropriately presents this explanation as a hypothesis, it remains unverified and should not be treated as an established explanation for the empirical results. A more convincing study would compare the same backbone at different stages of agentic post-training under matched harness configurations.
-
The model coverage is insufficient to support broad conclusions about frontier agents. The evaluation omits contemporaneous frontier models from the GPT and Claude families. This is particularly important because the paper’s own results show that the relative value of a harness depends on the underlying model: specialized scaffolding benefits the weaker Gemma backbone, whereas Malena performs more competitively with stronger backbones. Consequently, the findings cannot be assumed to generalize to other leading models with different tool-use behaviors and training distributions. Including current GPT and Claude models would substantially strengthen the external validity of the conclusions; otherwise, the claims should be explicitly restricted to the evaluated backbones.
-
The comparisons do not establish the marginal value of mainstream coding-agent harnesses. Although the paper evaluates four specialized MLE systems, it does not compare against widely used general-purpose coding-agent harnesses such as Codex or Claude Code. Moreover, Malena itself is built on OpenCode and therefore already benefits from a substantial coding-agent runtime. The experiments primarily test whether additional MLE-specific orchestration improves upon this runtime, rather than whether harness functionality has become unnecessary. Without comparisons across mature general-purpose harnesses, the study cannot determine how much performance depends on their execution, context-management, and tool-interaction mechanisms, or on model–harness compatibility. Where technically feasible, evaluating the same backbone across such harnesses would help separate model capability from runtime quality. The current evidence supports a narrower claim about the tested MLE-specific additions, not a general conclusion that strong models have absorbed the value of harness design.
-
The ablations do not cleanly identify the mechanisms responsible for the gains. Despite the extensive ablation suite, several central comparisons change multiple factors simultaneously. In particular, moving from Oneshot-based search to Malena changes the production-run constraint, session continuity, documentation requirements, and tool configuration, while potentially affecting artifact reuse and validation consistency. The resulting gains therefore cannot be attributed specifically to greater autonomy or the removal of explicit search. This ambiguity is especially consequential because Malena and Best-of-N achieve nearly identical oracle percentiles but substantially different self-selected performance, suggesting that candidate selection may explain much of the advantage. A factorial comparison independently controlling search policy, session persistence, artifact access, and validation protocol would provide a substantially stronger account of why Malena works.
Questions for authors
see in weakness
5 comments
Thank you for the clarification and revisions. My main concern is not the absence of a few optional additional experiments, but the remaining gap between the paper’s central conclusions and its experimental coverage. I recognize Malena’s value as a strong baseline and its advantages in the evaluated settings. However, Section 6 goes further, arguing that elaborating harnesses yields “poor returns” and that “the leverage is in the model and in the runtime it is given, not in the scaffolding built around them.” These statements are substantially stronger than showing that Malena outperforms several tested MLE systems.
The principal comparisons cover selected open-source backbones and four MLE-specific frameworks, but omit Codex and Claude Code paired with their respective contemporaneous model versions—mainstream configurations that are directly relevant to the paper’s practical claims. This is not merely a request to include newer systems released after submission: Codex and Claude Code were already prominent, widely used coding-agent systems during the pre-submission period when the paper would have been developed, and remain so today. Their omission therefore leaves out central practical reference points, rather than peripheral alternatives. Without comparisons against these mainstream model–harness configurations, the broader conclusion that further harness development offers poor returns is not convincing. Kimi’s competitiveness with GPT and Claude on other benchmarks does not resolve this concern: similar benchmark performance does not establish similar responses to orchestration, context management, or tool interfaces. Indeed, the paper’s own finding that harness benefits depend on the backbone makes it particularly important to evaluate representative model–harness combinations directly, rather than extrapolate from the selected open-source backbones.
Moreover, Malena already relies on the runtime capabilities provided by OpenCode. The current experiments therefore primarily assess the marginal value of particular MLE-specific orchestration mechanisms relative to an OpenCode-based baseline, rather than the broader value of mature harness design. Distinguishing the underlying “runtime” from additional “scaffolding” is reasonable, but the resulting conclusions must respect that distinction. Showing limited benefits from the tested additions does not establish that harness development more generally offers poor returns.
If the authors retain these broader practical conclusions, evaluations covering representative leading models and mature coding-agent harnesses are central supporting evidence, rather than merely out-of-scope extensions. I welcome the proposed experiments and would reassess the conclusions based on their results. If such comparisons cannot be completed, the conclusions should be narrowed further, consistently positioning the paper as a baseline and comparative empirical study under the specific models, budgets, and task settings evaluated.
Thank you for the follow-up and for identifying the two Section 6 sentences. We agree that they made the conclusion broader than the abstract, and have corrected them in the manuscript.
Scope. Our question is whether MLE-specific search, orchestration, selection and context-management machinery provides additional performance once a capable coding-agent runtime is available—not whether harnesses in general are unnecessary. Malena relies on OpenCode's runtime. Section 6 now explicitly refers to the poor returns of elaborating MLE-specific harnesses on current MLE benchmarks, and distinguishes the runtime from MLE-specific scaffolding. The conclusion now matches the abstract; the claim remains scoped to the tested models, tasks and budgets.
Evidence. We test the marginal value of these mechanisms in two complementary ways. First, we compare Malena with four released MLE harnesses under matched backbone, hardware and budget. Across 22 frontier-backbone–harness–metric comparisons, 15 significantly favor Malena after Holm correction over all 26 comparisons; none significantly favors an external harness. Second, we ablate MLE-specific interventions within a fixed coding-agent runtime.
The new 12-task held-out evaluation extends the comparison to unseen tasks and two backbones; on DeepSeek V4 Flash, Malena beats MLEvolve on all four metrics after correction (Appendix D.5). We also ran Malena on a second coding-agent backend, Pi: on one backbone and ten tasks, it significantly outperforms Arbor, MLEvolve and ScienceFlow on both metrics, and AiScientist on percentile (Appendix D.2). This is evidence that the comparative result is not unique to OpenCode, although the component ablations remain OpenCode-specific.
Codex and Claude Code. We agree that testing leading proprietary model–harness combinations would strengthen external validity. But those comparisons would not, by themselves, isolate the incremental value of MLE-specific machinery; that requires matching the underlying coding agent when adding the machinery. We do not claim generalization to untested proprietary combinations. Institutional restrictions prevent us from evaluating Codex or Claude Code with their closed models; as noted in the general response, we are committed to a Gemini 4 Argon evaluation in the camera-ready version, subject to approval.
Mechanisms. We also added an ablation between chained one-shot iterations and Malena (Appendix C.8). Carrying the session across links does not by itself change the chain's results, which stay below Malena even though the continued session receives the same prescriptive per-link instruction. Reusing artifacts across iterations recovers most of the gap, mainly on the oracle metrics. One reading is that what helps is not remembering the conversation but being able to build on earlier work products, and that prescriptive per-link instructions may cost something even when the session continues. Other differences between the chains and Malena remain, so we present this as a hypothesis about how autonomy, continuity and artifact reuse combine, not as a causal conclusion; isolating them would require further experiments.
We believe this supports a meaningful practical conclusion: MLE-specific complexity should demonstrate gains over a strong coding-agent baseline rather than be presumed beneficial. We hope the new experiments and corrected scope address your concerns, and would appreciate your reconsideration of the assessment.
We thank the reviewer for the careful and constructive review. We recognise the technical validity of the criticisms raised, and we address each of them below and in the revised manuscript.
Post-training ate the harness
We agree that the post-training explanation is a hypothesis that our experiments cannot establish causally, and the original presentation may have unintentionally put more emphasis on it than we meant: it is a best-effort interpretation of our results that is open to discussion. In the revised manuscript we removed it from the list of contributions, and the conclusion now states clearly that it is a hypothesis and not a claim (see the general response for the wording changes). We defer verification to future work.
Model coverage
Indeed, we cannot make claims about frontier models we have not evaluated, and, as the reviewer suggests, the revision restricts our claims to the evaluated open-source backbones (see the general response). We agree that the value of a harness depends on the backbone, and our results indicate where: the one backbone at which a harness significantly outperforms Malena, Gemma 4 31B, is also the one that gains least from a coding-agent environment (Chat vs. Oneshot, Figure 2), whereas at the three backbones with a large gain, no harness is significantly better than Malena. Kimi K3, the strongest of these, is competitive with GPT-5.5 and Claude Opus 4.8 on coding and agentic benchmarks such as Terminal-Bench and DeepSWE according to its release report. So, we suspect the trend would hold for closed models as well, and we are committed to verifying it with the evaluation below.
Planned: subject to our internal approval process, we commit to adding a full evaluation of at least three harnesses with Gemini 4 Argon in the camera-ready version.
Coding-agent harnesses
The coding-agent environment plays a major role in the success of Malena, and a strong model alone without tools to interact with the environment may not be enough to reach frontier performance, as shown in Figure 2. Our main claims rely on the presence of such a coding-agent environment, but we do not rely on the specific design of OpenCode. Two independent studies (https://harnesstax.github.io) and (https://arxiv.org/abs/2610.04433) report no significant difference in capability between coding-agent harnesses. A systematic comparison of different coding agents is outside the scope of this paper.
Planned: we will execute Malena with Pi Agent before the end of the rebuttal period to validate this stance.
Mechanisms responsible for the gains
We agree that the step from Oneshot-based search to Malena changes several factors at once. This is by design. Our autonomy intervention is precisely letting the agent decide these things itself instead of the harness fixing them. Our claim from this step is therefore about the bundle, not about any single factor. We edited the wording in Section 4.3 to state this explicitly and to avoid attributing the gain to autonomy or search removal specifically. Our other claims compare designs within a fixed iteration type and do not depend on this decomposition. While the exact performance decomposition of the individual factors falls out of the scope of this work, the existing evidence covers several already.
Planned: to isolate the remaining factors, we are running targeted ablations where we hold the production-run constraint fixed to account for the other mechanisms. We will update the manuscript when the results are available.
Summary
The review describes our question as important, our evaluation as broad and comparable, and Malena as a baseline that future work should consider, and it rates the paper as informative ("I learned something interesting or useful") with only minor or fixable soundness concerns. The weaknesses it lists concern how we interpret (W1) and scope (W2) the findings, and additional experiments (W3, W4), rather than the findings themselves; the revision narrows our claims accordingly and adds the requested evidence where we can. We therefore find a clear rejection difficult to reconcile with this assessment, and we would respectfully ask the reviewer to revisit the overall rating, and to let us know which points weigh most in it, so that we can address them as fully as possible.