acceptodds
Under review as a conference paper at ICLR 2027

HASTE: Hierarchical Accumulation of Skills for Transfer-Efficient ML Engineering

Abstract

Agents evaluated on MLE-Bench solve every competition from scratch; whatever an agent learns while engineering one solution is discarded before the next task begins. We present HASTE, an ML-engineering agent that accumulates plain-text skills in a three-tier hierarchy of global, domain, and competition scope, promotes learnings upward between tasks through LLM-driven abstraction, and loads only the tiers relevant to the task at hand. Across three seeds on the 22 MLE-Bench Lite competitions, using Claude Sonnet 4.6 with a CLI backend under a 12-hour cap per attempt, HASTE reaches a medal rate of 62.1% ± 1.5 (SEM); a replication on Claude Opus 4.8 reaches 66.7% ± 1.5, with the same six competitions failing to medal under either model. A multi-pass campaign that keeps each competition's best graded attempt reaches 77.3%, in the leaderboard's top band on a mid-tier model at half the typical budget; because attempts are selected on test outcomes, this figure is an upper bound. The accumulation loop operates end to end: each seed grows a store of 121 to 143 skills from an empty start, the orchestrator promotes learnings across tiers, loaded skills are cited by name in refinement proposals, including 15 cross-competition citations in the logged runs, and revisiting competitions with a matured store recovers 7 of 12 first-attempt failures. A system-level ablation over 8 competitions and 3 seeds places the deployed configuration at 83.3% ± 4.2 against 62.5% with no skills and 50.0% ± 7.2 with an unscoped frozen store; the arms differ in harness budget as well as loading policy, so we read the ablation as comparing configurations rather than isolating loading. All code, the skill store, run logs, submissions, and grading artifacts are included in the supplementary material.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
ML engineering agents, MLE-Bench, LLM agents, hierarchical memory, skill accumulation, multi-agent systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xhKsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents HASTE, an ML-engineering agent for MLE-Bench that accumulates plain-text skills across competitions in a three-tier store organized by scope of applicability: global, domain (tabular, NLP, vision), and competition. An orchestrator routes each competition to a domain specialist, which loads only the tiers matching its scope, runs a profile-prototype-refine-ensemble pipeline, and emits learnings; between rounds an LLM promotes learnings upward through the hierarchy.

The stated contributions are (i) a replicated benchmark result on MLE-Bench Lite, (ii) evidence that the accumulate-promote-reload loop operates end to end in deployment, and (iii) a system-level ablation against no-skill and unscoped-store configurations.

Validation uses MLE-Bench Lite (22 competitions) under a 12-hour cap. Three seeds on Claude Sonnet 4.6 reach 62.1% ± 1.5 medal rate, and three seeds on Claude Opus 4.8 reach 66.7% ± 1.5; a multi-pass campaign selecting each competition's best graded attempt reaches 77.3%. Store growth, promotion counts, self-reported skill citations, and recovery of first-attempt failures on revisit are reported as evidence of the loop. The ablation on 8 competitions and 3 seeds gives 83.3% (tiered), 62.5% (empty), and 50.0% (flat).

Critical strengths
  1. Replicated, selection-free benchmark result with full artifacts. Three seeds under a 12-hour cap, repeated on a second model with an identical failure set, exceed the single-run reporting standard of most leaderboard entries. The archive includes source, the frozen skill store, and scripts that recompute every headline number.
  2. An inspectable knowledge mechanism exercised end to end on real tasks. The plain-text, scope-tagged store can be read and diffed, and production, promotion, reloading, and citation are all logged in deployment rather than demonstrated in isolation. The authors state every confound in their own evidence, which makes the claims easy to evaluate.
Critical weaknesses

1. The ablation does not test the contribution. The tiered arm runs under the production harness (12 h, 20 iterations, learning on) while flat and empty run under a reduced harness (10.5 h, 11 iterations, learning off), and the tiered cells are reused replication seeds rather than a separate experiment. The 83.3 vs 62.5 vs 50.0 comparison therefore measures budget and configuration, not loading policy, and the p = 0.031 result carries no information about scope.

2. The skill store was largely never seen by the model. Each prompt exposed only the first 2,000 characters of the skill block, global tier first; a single skill file is roughly 800 characters. The flat arm thus saw the head of five global entries rather than an unscoped store, and the tiered arm's domain and competition tiers were almost never in context. The paper's central mechanism was mostly inactive in the reported runs, and the Figure 2 caption concedes that whether even the shown skill affected any decision "is not established."

3. No evidence attributes the benchmark result to accumulated skills. The empty configuration was never run on the full 22-competition roster or under the production harness, and the pipeline components (prototype screen, refinement loop, ensemble) are not ablated. All transfer evidence is descriptive and confounded by added attempts, compute, and test-outcome selection, as the authors state; the self-reported citation rate (24 of 317 proposals) is small, and the 62% vs 52% keep rate is not significant. A within-run analysis by roster position, which would estimate transfer without a new harness, is not reported.

4. The 77.3% figure is test-set selection and should not appear on the leaderboard table. Choosing each competition's best graded attempt across a 250-hour campaign is model selection on test outcomes; labeling it an upper bound does not make it a benchmark result, yet the abstract and introduction lead with it as "top band at half the budget." The comparable, selection-free number (62.1%) ties InternAgent and trails R&D-Agent among same-budget entries.

5. Resource and process statistics rest on non-comparable or incomplete data. Table 6 mixes tokens from single runs on different harnesses and dates (tiered from a May campaign, flat and empty from the ablation harness) and uses medal counts from those runs rather than from Table 5, so the "2.7× gap" is not a controlled measurement. Only 35 of 66 seed runs retained refinement logs, promotion counts are stated as lower bounds after out-of-band salvage, and the Opus replication differs in harness hardening and reasoning effort, so neither the loop statistics nor the model comparison is controlled.

Questions for authors
  1. Under the 2,000-character cap, what was the actual skill content exposed in each tiered-arm prompt? Please report, per prompt, how many skills and from which tiers were visible. If domain and competition entries were rarely in context, how do the authors justify attributing any result to the hierarchy?
  2. Was the competition order within each seed fixed or varied? Please report medal rate as a function of roster position, which would estimate within-run transfer without a new harness.
  3. What is the empty-store medal rate on the full 22-competition roster under the production harness (12 h, 20 iterations)? This is the baseline against which the 62.1% result must be read.
  4. In Table 6, which runs supply the tiered token counts, and why do the per-arm medal counts (8/5/5) differ from Table 5 (20/12/15)?
  5. The harness-matched ablation the authors themselves describe in §8: tiered, flat, and empty on one harness with the per-prompt cap lifted, plus a retrieval-based arm (e.g., ExpeL-style top-k) at matched entry count, so that scope can be separated from budget, cardinality, and relevance.
  6. A selection-free multi-pass result: the medal rate of the final pass alone, or of a fresh single pass started from the matured store, in place of the best-of-attempts 77.3%.
Reviewer_QEzwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HASTE, an ML-engineering agent that accumulates plain-text skills across tasks in a three-tier hierarchy (global, domain, and competition scope). An orchestrator assigns competitions to domain specialists and, between rounds, uses an LLM to promote learnings to higher tiers through abstraction. Each specialist loads the tiers whose scope matches the current task (via directory reads rather than retrieval) and uses them to guide prototype selection and iterative refinement.

On the 22 MLE-Bench Lite competitions, with a 12-hour budget per competition, the paper reports medal rates of 62.1% ± 1.5 over three seeds with Claude Sonnet 4.6 and 66.7% ± 1.5 with Claude Opus 4.8. A separate multi-pass campaign that keeps each competition's best graded attempt reaches 77.3%, which the authors present as a test-selected upper bound and place in the top band of the public leaderboard. The paper also documents skill growth, promotion events, and self-reported skill citations, and it compares first attempts with campaign-best attempts (45.5% vs. 77.3%). Finally, a system-level ablation on 8 competitions reports 83.3% for the deployed configuration, 62.5% with no skills, and 50.0% with an unscoped frozen store; the authors note that the arms also differ in harness budget, so the ablation compares configurations rather than isolating the loading policy.

Critical strengths
  • Practical and interpretable framework. The paper presents a simple and interpretable approach to organizing and reusing skills across ML engineering tasks. The framework is implemented and evaluated on realistic tasks, making it a potentially useful systems contribution, although its advantages over simpler alternatives remain unclear.
Critical weaknesses
  1. Inconsistency between the reported evaluation protocol and the supplementary results.

Table 3 describes the three-seed results as "single-pass, selection-free estimates" that are comparable to other leaderboard entries. However, the supplementary documentation (sonnet_multiseed.md and opus_multiseed.md) explicitly describes best-attempt selection across multiple phases and makeup rounds. The released grading reports also indicate that 19 of the 41 Sonnet medals and 11 of the 44 Opus medals come from makeup rounds. In addition, some makeup rounds used different reasoning-effort settings. This raises a substantial concern about the comparability of the reported 62.1% and 66.7% medal rates with single-attempt leaderboard results. Since these figures constitute a central part of the empirical evaluation, I would appreciate clarification of the attempt-selection procedure, the cumulative compute budget per competition, and the performance obtained from only the first eligible attempt.

  1. Insufficient evidence supporting the proposed hierarchical memory mechanism.

Although the proposed three-tier memory architecture is intuitive, the experiments do not adequately isolate its effectiveness. The system-level ablation compares the tiered configuration (12 hours and 20 refinement iterations) against flat and empty configurations (10.5 hours and 11 iterations). Moreover, the tiered results are taken from the multi-round replication campaigns, whereas the flat and empty configurations use single attempts. The supplementary audit_prompt_slice.py further shows that, when scoped and unscoped loading are applied to the same frozen skill store, the first 2,000 characters—the portion actually included in the prompts—are identical across all eight ablation competitions. Although the deployed tiered arm uses a different, accumulating store, these findings make it difficult to attribute the observed performance differences to hierarchical loading itself.

Furthermore, the paper lacks a fully matched no-skill baseline across all 22 competitions and does not isolate the contribution of LLM-based skill promotion. As a result, it remains unclear whether the proposed architecture offers substantial benefits over simpler memory-sharing or retrieval-based alternatives. More controlled component-level ablations would be necessary to establish the effectiveness of the central design choices.

Questions for authors

Please see the Critical Weaknesses above for my main questions and concerns.

Reviewer_3A4SOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduce a method HASDE, which is an ML engineering agent that accumulates plain text skills in a three tier hierarchy of global domain and competition scope, promotes learnings upward between tasks through LLM driven obstruction, and loads only the tiles relevant to the task at hand. Thus HASDA does not need to solve every competition from scratch.

Critical strengths

Accumulating skills across tasks is interesting and useful for ML tasks. The proposed method HASTE can be applied to any LLMs, which is a general method.

Critical weaknesses
  1. Figure 1 is the has architecture but in a very brief way. The figure does not show how the skill hierarchies promoted and accumulated during the task completion.
  2. Section 3 HASTE also is presented in an abstract and brief way without too many method details introduced. This makes me can understand the overall picture of HASTE, but I can't draw a detail picture and concrete architecture of the method.
Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.