acceptodds
Under review as a conference paper at ICLR 2027

AceGRPO: Adaptive Curriculum Enhanced Group Relative Policy Optimization for Autonomous Machine Learning Engineering

Abstract

Autonomous Machine Learning Engineering (MLE) requires agents to perform sustained, iterative optimization over long horizons. While recent LLM-based agents show promise, current prompt-based agents for MLE suffer from behavioral stagnation due to frozen parameters. Although Reinforcement Learning (RL) offers a remedy, applying it to MLE is hindered by prohibitive execution latency and inefficient data selection. Recognizing these challenges, we propose AceGRPO with two core components: (1) Evolving Data Buffer that continuously repurposes execution traces into reusable training tasks, and (2) Adaptive Sampling guided by a Learnability Potential function, which dynamically prioritizes tasks at the agent's learning frontier to maximize learning efficiency. Trained with AceGRPO, Ace-35B substantially improves over its base model, achieving a 100% valid submission rate on MLE-Bench-Lite and approaching or surpassing larger frontier models on key metrics. These results demonstrate the effectiveness of adaptive, execution-grounded training for building capable and reliable long-horizon MLE agents.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large language Model, Reinforcement learning, Self-improvement, Autonomous Machine Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wMMbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AceGRPO trains autonomous MLE agents with an evolving buffer of intermediate execution states and a sampler based on reward dispersion, improvement headroom, and recency. It optimizes single-step code actions rather than full trajectories. Ace-35B is evaluated on MLE-Bench-Lite, and a separate Ace-30B study compares training baselines and component ablations.

Critical strengths
  1. Reuse of execution states is well matched to expensive MLE training. The buffer turns both failures and partial successes into future starting points, and the curriculum is tied to observed execution outcomes.
  2. The held-out-competition evaluation and same-initialization Ace-30B ablations provide useful evidence beyond comparison with an untrained model. Reporting repeated evaluations and first-submission quality also helps characterize the improvement.
Critical weaknesses
  1. The interpretation of raw reward dispersion as a direct gradient-magnitude proxy is not established for standardized GRPO advantages. Away from the numerical stabilizer, rescaling a nonconstant reward group leaves its standardized advantages essentially unchanged. Potential may still be a useful sampling heuristic, but the stated gradient justification needs qualification and validation.
  2. The component study uses Ace-30B rather than the main Ace-35B, and matches update steps rather than measured execution cost. Since tasks have highly unequal runtimes, it does not fully establish that the main-model gain or training efficiency follows specifically from adaptive sampling under an equal cost budget.
Questions for authors
  1. How strongly does potential correlate with measured update utility or gradient statistics after GRPO normalization, compared with variance-only and uniform sampling?
  2. What are the Ace-35B component results and learning curves under equal execution-hours or total GPU-hours?
1 comment

Thank you for the precise review. Below is our point-by-point response.


[W1] Raw reward dispersion as a gradient-magnitude proxy under standardized advantages

We agree and qualify the justification as requested: raw reward dispersion determines whether a group yields a reward-driven gradient, not how large it is. Under the population-standard-deviation convention in Eq. (3), 1G∑i=1GA^i2=σ2(σ+ϵ)2≈1when σ≫ϵ.\frac{1}{G}\sum_{i=1}^{G}\hat A_i^2 = \frac{\sigma^2}{(\sigma+\epsilon)^2} \approx 1 \quad \text{when }\sigma\gg\epsilon.

  • Nonconstant reward groups: away from the stabilizer ϵ\epsilon, larger raw dispersion does not imply larger standardized advantages; the gradient norm also depends on the policy score vectors.
  • Constant reward groups: exactly equal rewards give zero reward-driven gradient (KL or entropy regularization may still produce updates).

We will reword this interpretation wherever it appears and qualify Eq. (5).

[W2 & Q2] Component evidence on a second base model; learning curves and measured cost

Repeating the component ablation on the 35B model was beyond our compute and time budget. The component evidence therefore remains the full Ace-30B study: Table 2 of the paper ablates both components in that setting, and the full run gains on six of the seven metrics of Table 1 over its own base and ties on Silver (Any Medal from 27.27% to 51.52%, HumanRank from 48.75 to 71.14; Table R1), the same pattern as Ace-35B. So the method is effective on both base models, and each components of AceGRPO is effective.

Table R1. The full Ace-30B run against its base at the matched 400-update budget.

MetricQwen3-30B-A3B-2507Ace-30B
HumanRank48.75 ± 3.4571.14 ± 4.08
Any Medal %27.27 ± 6.4351.52 ± 8.57
Above median %50.00 ± 3.7178.79 ± 5.67
Bronze / Silver / Gold %1.52 / 7.58 / 18.1812.12 / 7.58 / 31.82
Valid submissions %84.85 ± 2.14100.00 ± 0.00
  • Learning curves: Figure 5 of the paper gives the training dynamics of the submitted Ace-35B run (valid-submission rates and rewards by task type) over its 400 updates.

  • Component evidence: the effectiveness of the individual components of AceGRPO is shown by the experiments on the 30B model in Appendix B of the paper, and the full 30B run is in Table R1 above. Together with Table 1, this gives the same picture on two base models: each component contributes, and AceGRPO improves the base model on the MLE-Bench-Lite metrics in both settings.


We hope the corrections and the additional results address your concerns. If so, we would be grateful if you would consider adjusting your rating.

Reviewer_SXJ6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an evolving data buffer and adaptive sampling strategy to address prohibitive execution latency and inefficient data selection in long-horizon MLE agents. Ace-35B substantially improves over the base model, demonstrating the effectiveness of the proposed approach.

Critical strengths

The motivation is clear, focusing on key challenges in training long-horizon MLE agents.

The paper provides both performance and training efficiency results, along with additional analyses to support the effectiveness of the proposed method.

Critical weaknesses

On certain datasets, the improvements are relatively limited and appear to fall within the reported variance, raising concerns about the consistency and significance of the gains.

Several existing methods, such as DAPO and other MLE-RL approaches, share similar motivations. More direct comparisons with these methods are needed to better establish the effectiveness and necessity of the proposed approach.

Questions for authors

Are the improvements statistically significant across datasets, particularly where the gains fall within the reported variance?

How does the proposed method compare with existing approaches such as DAPO and other MLE-RL methods that address similar challenges? And from a novelty perspective, could the differences and advantages be further clarified?

2 comments

Thank you for the careful review and for recognizing the motivation and the empirical analyses. Below is our point-by-point response.


[W1 & Q1] Statistical significance and consistency of the gains across datasets

  • Each of the 22 Lite competitions is run with three seeds, the number required by the official MLE-bench leaderboard (at least three).
  • Medal and valid-submission metrics are yes/no outcomes per run, so with 66 runs in total they are coarse: one run changes a rate by 1.52 %, and the gains on Bronze, Silver, Gold and valid submission amount to only two to four runs, which is comparable to the run-to-run variation (the reported variance).
  • The improvement shows on the finer-grained metrics:
    • HumanRank, a continuous score, rises from 66.29 to 77.14, which is significant over the 22 competitions (p = 0.0002)
    • Any Medal rate rises from 48.48% to 62.12% (p = 0.078 in the same test)

Taken together, the two headline metrics, HumanRank and Any Medal gain are statistically significant; the improvement brought by AceGRPO is therefore not explained by run-to-run variation.


[W2 & Q2] Comparison with DAPO and other MLE-RL methods and novelty of AceGRPO

We added an EDB + DAPO-style baseline at matched execution cost: under the same 500 sandbox-hour cap, AceGRPO reaches 71.01 HumanRank and 57.58% Any Medal against DAPO baseline with 62.38 and 43.94% (Table R1).

Setup: both use the same Qwen3.6-35B-A3B initialization and keep the Evolving Data Buffer. The DAPO-style samples states independently of the potential and discards zero-variance groups (reward standard deviation ≤ 10−610^{-6}) after execution. Those executions still count toward the cost.

Table R1. Matched 500 sandbox-hour budget (mean ± standard deviation over three evaluation repeats.

MethodTraining cost sandbox−hourssandbox-hoursUpdate StepsHumanRankAny Medal %Valid submission %
Qwen3.6-35B-A3B——66.29 ± 0.9548.48 ± 2.1496.97 ± 2.14
EDB + DAPO-style500.8013862.38 ± 2.1843.94 ± 2.14 29/6629/6696.97 ± 4.29 64/6664/66
AceGRPO500.0016471.01 ± 3.1457.58 ± 2.14 38/6638/66100.00 ± 0.00 66/6666/66

AceGRPO outperforms DAPO. Both methods are still early in training at this budget (the submitted model uses 400 updates). AceGRPO, however, turns the same 500 sandbox-hours into more training: 164 updates against 138. Learnability Potential is designed for exactly this: a state whose candidates all fail receives zero potential (Eq. (7)) and low sampling priority, so executions are not wasted on tasks the model cannot yet solve. The DAPO-style has no such signal and samples states regardless of their previous outcomes.

Novelty. RL for MLE is limited by execution cost: a single step takes minutes to hours. Step-wise training is a kind of method to save expensive execution cost. But this raises two problems that existing methods leave open:

  1. Where to sample states. Earlier MLE-RL methods such as ML-Agent collect the state pool in advance, at a large upfront execution cost and with limited diversity.
  2. How to sample states. Uniform sampling keeps executing states that are already mastered or out of reach, and DAPO-style sampling discards such zero-signal groups only after executing them; either way execution is wasted.

AceGRPO addresses both:

  • Evolving Data Buffer turns every execution into a new training state, so the pool grows online without a separate collection phase.
  • Adaptive Sampling selects, before execution, the states worth running.

Both components are ablated in Table 2 of the paper. To our knowledge, no prior method handles both under MLE's execution cost.


We hope these clarifications and experiments address your concerns. If so, we would be grateful if you would consider adjusting your rating.

Thanks for the clarification. I prefer to maintain my score.

Reviewer_9rQqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AceGRPO, a reinforcement learning framework for autonomous Machine Learning Engineering (MLE). It addresses the high cost of long-horizon RL through two components: an Evolving Data Buffer that recycles intermediate execution states into reusable training tasks, and Adaptive Sampling that prioritizes states based on reward variance and improvement headroom.

Experiments on MLE-Bench-Lite demonstrate substantial improvements over vanilla GRPO and larger open-source models, with Ace-30B achieving a 51.52% medal rate and 100% valid submission rate.

I think comparisons with closely related methods, particularly existing dynamic sampling approaches are insufficient, making it difficult to assess the benefits of the proposed design over prior alternatives.

Critical strengths

The paper helps mitigate the high cost of long-horizon agentic RL, with the Evolving Data Buffer providing a practical way to reuse intermediate execution states.

AceGRPO substantially outperforms vanilla GRPO, with ablation studies supporting the effectiveness of both proposed components.

Critical weaknesses

Dynamic sampling has been explored in prior work such as DAPO and DSPO. Without comparisons against these approaches or simpler sampling heuristics, the benefits of the proposed Adaptive Sampling strategy remain unclear.

The proposed metric uses reward variance as a proxy for gradient informativeness, but this connection is not sufficiently justified, particularly given GRPO's advantage normalization.

Questions for authors

Could the authors compare AceGRPO against DAPO-style dynamic sampling and simpler variance-based sampling under matched execution budgets?

Could the authors provide performance curves against training wall-clock time to better assess training efficient of the AceGRPO?

2 comments

Thank you for the constructive review. Below is our point-by-point response.


[W1 & Q1]: Comparison with DAPO-style and variance-based sampling at matched execution budgets

We ran the DAPO-style comparison at a matched budget of 500 sandbox-hours: AceGRPO leads on all seven metrics (+13.64% Any Medal, +8.63 HumanRank) and completes 164 policy updates against 138 (Table R1).

Setup: both use the same Qwen3.6-35B-A3B initialization and keep the Evolving Data Buffer. The DAPO-style samples states independently of the potential and discards zero-variance groups (reward standard deviation ≤ 10−610^{-6}) after execution. Those executions still count toward the cost.

Table R1. Matched 500 sandbox-hour budget (mean ± standard deviation over three evaluation repeats.

MetricBase (untrained)EDB + DAPO-styleAceGRPOΔ\Delta
Training cost sandbox−hourssandbox-hours—500.80500.00—
Policy updates—138164+19%
Valid submission %96.97 ± 2.1496.97 ± 4.29 64/6664/66100.00 ± 0.00 66/6666/66+3.03
Above median %65.15 ± 2.1459.09 ± 6.43 39/6639/6660.61 ± 5.67 40/6640/66+1.52
Bronze %1.52 ± 2.146.06 ± 2.14 4/664/667.58 ± 2.14 5/665/66+1.52
Silver %16.67 ± 4.2912.12 ± 2.14 8/668/6619.70 ± 4.29 13/6613/66+7.58
Gold %30.30 ± 2.1425.76 ± 2.14 17/6617/6630.30 ± 2.14 20/6620/66+4.54
Any Medal %48.48 ± 2.1443.94 ± 2.14 29/6629/6657.58 ± 2.14 38/6638/66+13.64
HumanRank66.29 ± 0.9562.38 ± 2.1871.01 ± 3.14+8.63
  • Key difference: when the execution cost is paid. In MLE one rollout is a full ML pipeline run in a sandbox (minutes to hours on GPU nodes), so execution is the cost that limits training.

    • DAPO filters after execution: groups with all-equal rewards are dropped and resampled, although their sandbox time and compute are already spent.
    • AceGRPO decides before execution which state to run.
  • AceGRPO outperforms DAPO-sytle. Both methods are still early in training at this budget (the submitted model uses 400 updates). AceGRPO, however, turns the same 500 sandbox-hours into more training: 164 updates against 138. Learnability Potential is designed for exactly this: a state whose candidates all fail receives zero potential (Eq. (7)) and low sampling priority, so executions are not wasted on tasks the model cannot yet solve. The DAPO-style has no such signal and samples states regardless of their previous outcomes.

Variance-only sampling: we could not run this baseline for the response (each experiment here costs 500 sandbox-hours of training and 3 times 12h-evaluation for 22 tasks), and the DAPO-style filter is itself a variance criterion, applied after execution. A pre-execution sampler that uses dispersion alone remains untested, and we will state this as a limitation.


[W2]: Justification of Learnability Potential under GRPO's advantage normalization

We agree that the paper states this connection too loosely. Stated precisely: under advantage normalization, reward dispersion determines whether a group yields a learning signal, not how large that signal is. The benefit of the sampler as a whole is measured, not derived from this argument (Table R1; Table 2 of the paper). The revision presents Learnability Potential as a sampling heuristic and rewords the proxy for gradient magnitude statements.


[Q2]: Performance curves against training wall-clock time

We have added this curve to the revised paper (Figure 7, Appendix G): the overall training reward of the Ace-35B run, the same series as Figure 5 (Right), plotted against training wall-clock time. The reward roughly doubles within the first three hours (from 0.05 to 0.10) and reaches about 0.20 after 19 hours, roughly half of the 35.5-hour run; it then averages between 0.16 and 0.20 for the rest of the run and ends at 0.21, about four times its initial value.

Held-out performance on the same axis: Any Medal is 48.48% for the base model, 57.58% at a 500 sandbox-hour budget ( Table R1 above) and 62.12% after the full run (HumanRank: 66.29, 71.01 and 77.14; Table 1 of the paper). Two-thirds of the Any-Medal gain is thus obtained with the first 28% of the training budget.


We hope these clarifications and experiments address your concerns. If so, we would be grateful if you would consider adjusting your rating.

Thanks for the additional results and discussion; my main concerns have been largely addressed, and I have raised my rating from 2 to 3. I remain surprised that DAPO-style sampling shows some negative gain at this budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.