- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper presents HASTE, an ML-engineering agent for MLE-Bench that accumulates plain-text skills across competitions in a three-tier store organized by scope of applicability: global, domain (tabular, NLP, vision), and competition. An orchestrator routes each competition to a domain specialist, which loads only the tiers matching its scope, runs a profile-prototype-refine-ensemble pipeline, and emits learnings; between rounds an LLM promotes learnings upward through the hierarchy.
The stated contributions are (i) a replicated benchmark result on MLE-Bench Lite, (ii) evidence that the accumulate-promote-reload loop operates end to end in deployment, and (iii) a system-level ablation against no-skill and unscoped-store configurations.
Validation uses MLE-Bench Lite (22 competitions) under a 12-hour cap. Three seeds on Claude Sonnet 4.6 reach 62.1% ± 1.5 medal rate, and three seeds on Claude Opus 4.8 reach 66.7% ± 1.5; a multi-pass campaign selecting each competition's best graded attempt reaches 77.3%. Store growth, promotion counts, self-reported skill citations, and recovery of first-attempt failures on revisit are reported as evidence of the loop. The ablation on 8 competitions and 3 seeds gives 83.3% (tiered), 62.5% (empty), and 50.0% (flat).
Critical strengths
- Replicated, selection-free benchmark result with full artifacts. Three seeds under a 12-hour cap, repeated on a second model with an identical failure set, exceed the single-run reporting standard of most leaderboard entries. The archive includes source, the frozen skill store, and scripts that recompute every headline number.
- An inspectable knowledge mechanism exercised end to end on real tasks. The plain-text, scope-tagged store can be read and diffed, and production, promotion, reloading, and citation are all logged in deployment rather than demonstrated in isolation. The authors state every confound in their own evidence, which makes the claims easy to evaluate.
Critical weaknesses
1. The ablation does not test the contribution. The tiered arm runs under the production harness (12 h, 20 iterations, learning on) while flat and empty run under a reduced harness (10.5 h, 11 iterations, learning off), and the tiered cells are reused replication seeds rather than a separate experiment. The 83.3 vs 62.5 vs 50.0 comparison therefore measures budget and configuration, not loading policy, and the p = 0.031 result carries no information about scope.
2. The skill store was largely never seen by the model. Each prompt exposed only the first 2,000 characters of the skill block, global tier first; a single skill file is roughly 800 characters. The flat arm thus saw the head of five global entries rather than an unscoped store, and the tiered arm's domain and competition tiers were almost never in context. The paper's central mechanism was mostly inactive in the reported runs, and the Figure 2 caption concedes that whether even the shown skill affected any decision "is not established."
3. No evidence attributes the benchmark result to accumulated skills. The empty configuration was never run on the full 22-competition roster or under the production harness, and the pipeline components (prototype screen, refinement loop, ensemble) are not ablated. All transfer evidence is descriptive and confounded by added attempts, compute, and test-outcome selection, as the authors state; the self-reported citation rate (24 of 317 proposals) is small, and the 62% vs 52% keep rate is not significant. A within-run analysis by roster position, which would estimate transfer without a new harness, is not reported.
4. The 77.3% figure is test-set selection and should not appear on the leaderboard table. Choosing each competition's best graded attempt across a 250-hour campaign is model selection on test outcomes; labeling it an upper bound does not make it a benchmark result, yet the abstract and introduction lead with it as "top band at half the budget." The comparable, selection-free number (62.1%) ties InternAgent and trails R&D-Agent among same-budget entries.
5. Resource and process statistics rest on non-comparable or incomplete data. Table 6 mixes tokens from single runs on different harnesses and dates (tiered from a May campaign, flat and empty from the ablation harness) and uses medal counts from those runs rather than from Table 5, so the "2.7× gap" is not a controlled measurement. Only 35 of 66 seed runs retained refinement logs, promotion counts are stated as lower bounds after out-of-band salvage, and the Opus replication differs in harness hardening and reasoning effort, so neither the loop statistics nor the model comparison is controlled.
Questions for authors
- Under the 2,000-character cap, what was the actual skill content exposed in each tiered-arm prompt? Please report, per prompt, how many skills and from which tiers were visible. If domain and competition entries were rarely in context, how do the authors justify attributing any result to the hierarchy?
- Was the competition order within each seed fixed or varied? Please report medal rate as a function of roster position, which would estimate within-run transfer without a new harness.
- What is the empty-store medal rate on the full 22-competition roster under the production harness (12 h, 20 iterations)? This is the baseline against which the 62.1% result must be read.
- In Table 6, which runs supply the tiered token counts, and why do the per-arm medal counts (8/5/5) differ from Table 5 (20/12/15)?
- The harness-matched ablation the authors themselves describe in §8: tiered, flat, and empty on one harness with the per-prompt cap lifted, plus a retrieval-based arm (e.g., ExpeL-style top-k) at matched entry count, so that scope can be separated from budget, cardinality, and relevance.
- A selection-free multi-pass result: the medal rate of the final pass alone, or of a fresh single pass started from the matured store, in place of the best-of-attempts 77.3%.