- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes PRESAGE, a method for reducing wasted rollouts during GRPO training of code agents. A predictor estimates task difficulty before rollout generation and filters out tasks expected to yield little progress. The method also uses a fractional test reward, with a penalty for regressions, and updates task selection through predictor recalibration and retirement of repeatedly unproductive tasks. Experiments report more informative rollout groups and improved held-out performance compared with uniform GRPO and dynamic sampling.
Critical strengths
(1) The paper targets a concrete source of wasted computation. Detecting uninformative tasks only after generating full agent trajectories can be expensive. Predicting which tasks to sample beforehand is a reasonable way to improve the use of a limited training budget.
(2) The experiments show a substantial benefit from filtering in the tested setting. PRESAGE increases the fraction of sampled groups providing a learning signal from 16.1% to 58.6% and improves held-out performance over uniform GRPO. These results suggest practical value for task pools containing many tasks beyond the current policy’s capability.
(3) The paper investigates the contributions of its components. The reward and filtering comparisons indicate that filtering drives most of the held-out improvement. The authors also acknowledge that the additional benefit of fractional rewards over filtering alone is not statistically established, which helps distinguish supported conclusions from weaker ones.
Critical weaknesses
(1) The evaluation is too narrow to establish a general result for code-agent training. Held-out evaluation covers only 24 tasks, with training drawn from 531 tasks across nine Python repositories and one base policy. Multiple rollouts per task improve score estimation, but do not substitute for a larger and more diverse task set. The current evidence leaves substantial uncertainty about transfer to other repositories, models, and task-difficulty distributions.
(2) The main efficiency comparison does not fully account for predictor construction. Labeling data for the predictor requires 4248 rollouts, compared with 5120 rollouts for policy training. The paper discusses this overhead, but its headline comparison uses equal policy-training rollout budgets rather than equal total budgets. Comparing the combined cost against a baseline trained for the same number of updates also does not establish performance at equal total cost. Since computational efficiency is the central claim, this is a consequential gap.
(3) The predictor’s ability to adapt is limited. The predictor is trained once, and subsequent adaptation uses a monotone calibration function. This can adjust thresholds but cannot change the ordering of tasks when the policy improves unevenly across task types. Permanent retirement can also exclude tasks that become useful later. The experiments do not sufficiently establish robustness to these changes over longer training runs.
(4) The contribution of fractional rewards is weaker than the overall framing suggests. Most held-out gains come from predictive filtering. The improvement from adding fractional rewards on top of filtering is not statistically established, and that comparison also introduces recalibration and retirement. The reward helps expose training-time variation, but its separate contribution to final performance remains unclear.
Questions for authors
(1) What performance does PRESAGE achieve relative to the baselines when predictor labeling, predictor training, and policy training are all charged to the same total budget? Can the baselines use the predictor's setup budget for additional policy training?
(2) How were the 24 held-out tasks selected, and which repositories and difficulty levels do they cover? Is there evidence that the gains persist on a larger held-out set or repositories excluded from predictor and policy training?
(3) Since the initial labeling pass already obtains rollout scores for every training task, how does the learned predictor compare with filtering directly using those observed scores? This would help establish the predictor’s value beyond the initial empirical difficulty estimates.
(4) How often do task difficulties change order during training, and how does the frozen predictor handle this? Have you measured useful tasks incorrectly excluded by the threshold or permanent retirement?
(5) Does the reported resolved rate require all originally passing tests to remain passing? If not, what are the results under that stricter definition, and how do regression rates compare across methods?
(6) What benefit does fractional reward provide when recalibration and retirement are held fixed? The current comparison appears to change all three together.
3 comments
Authors' Response to Reviewer izTL (2/2): Ordering Drift, Strict Resolved, and Graded Reward
Continuing from our first comment: Q4--Q6.
Q4. Ordering drift, the frozen predictor, and wrongly excluded tasks. The mechanism, stated precisely. Each cycle, step 1 re-thresholds the whole pool using the recalibrated score, not the previous cycle's survivors. Recalibration is a monotone map fitted every five cycles; the reviewer is right that it reorders nothing. Three things compensate. Since the threshold is re-applied to the whole pool, a task excluded in cycle comes back as soon as its recalibrated score clears : exclusion is not absorbing. Online retirement is per-task, acts on repeated all-zero draws, and takes several consecutive cycles to trigger. The drift is also slow relative to the run: dead leakage under the frozen filter stays flat at 30% over forty cycles while PRESAGE falls from 32% to 17%. If reordering were the dominant failure mode, the frozen arm would degrade rather than sit flat.
How often the order changes. A cross-cycle rank correlation of true per-task difficulty is the direct measurement of this drift. It requires a per-cycle evaluation of the pool, which sets its cost; it is the first extension we would add with additional compute.
Wrongly excluded tasks. At over the training tasks, 40 tasks are excluded, split evenly between tasks no evaluated policy ever solves and tasks some policy solves, so about 20 learnable tasks are dropped. Their counterfactual contribution is not identifiable from saved runs; what the sweep does establish is the cost curve. The threshold sweep on the training tasks (retained tasks solvable by at least one policy: 52.1%, 52.4%, 56.1%, 60.6%, 72.3% at ) is flat between 0 and 0.1 and steepens after 0.2, which is why we sit at 0.1.
What we defend. We do not claim recalibration tracks arbitrary reordering. The errors that matter are threshold errors on a slowly drifting ordering, and recalibration does fix those: recalibration plus retirement gains pp over neither (, ), either alone recovers most of it (, ), and removing one from the complete method costs and pp, neither measurable. Over much longer runs a periodic full refit of would be needed, and Limitations says so.
Q5. Does resolved require passing tests to keep passing? In the main tables, no. A rollout counts when every originally failing test passes, and the originally passing tests are not checked. The reward penalises regressions (), but the metric does not require their absence. The stricter metric is the one that matters, so we computed it.
Strict resolved (all failing tests repaired and no passing test broken), original pool, 24 tasks, 32 attempts:
| Seed | PRESAGE | Uniform binary | [95% CI] |
|---|---|---|---|
| 42 | 29.4% | 23.8% | pp |
| 1042 | 28.9% | 25.7% | pp |
The gain holds under the stricter metric and is about the size of the paper's resolved, which is the opposite of what a policy farming partial credit would show. Seed 42 survives Holm correction over seven supplementary comparisons (). Dropping the one task whose saved baseline shows only collection errors leaves pp and pp on 23 tasks. Regressions are accounted for through this strict rate rather than through a separate per-method regression table.
Q6. Does the graded reward help with the router held fixed? The original pool does not provide a clean single-variable comparison for this question: the one arm that varies only the reward under the complete router is at cycle 6 of 40. We therefore report the reward's effect where a clean single-variable comparison exists, and state the scope of what that comparison supports.
With sampling held fixed (SWE-bench, uniform). Graded reward raises the share of groups with reward s.d.\ above 0.01 from 46.25% to 74.69% ( pp, ). On the 120 held-out tasks the same pair reads 33.8% versus 31.1%, a pp gap .
Within the complete method (original pool). PRESAGE versus static-filter-with-binary changes reward, recalibration and retirement at once. That comparison is pp (), and we do not attribute it to the reward.
Contribution scope. The reward's demonstrated contribution is to the label source. On pass/fail labels most of the pool shares one target and cannot be fit; on graded labels it reaches AUC 0.963. Its contribution to the final score over and above filtering is not established by our experiments, and the revision reframes the contributions so that the two are not presented as equal.
Authors' Note to Reviewer izTL —Measurements for Both Remaining Open Points
Thank you for the most detailed and probing set of questions we received. We have now provided direct measurements for both remaining open points: the equal-budget comparison extended to 73 updates, and the observed-label filtering arm across 40 full training cycles. We would be very grateful for any updated score or remaining concerns before the discussion period closes.
Authors' Response to Reviewer izTL (1/2): Equal Budget, Held-out Set, Predictor vs. Observed Labels
These are the most specific questions we received. Q1--Q3 are answered here, Q4--Q6 in a second comment.
Q1. Equal total budget. The paper's headline comparison does not answer this, and the reviewer is right to press it. Three accountings follow, from the one most favourable to us to the one least. (1) Pool-level, as reported. The labelling pass costs 4,248 rollouts and yields labels for the whole 531-task pool; shared over the paper's twelve runs that is 354 per run, 6.9% of the 5,120-rollout policy budget. (2) Single run, all charged. 9,368 rollouts for forty updates, against roughly 31,280 for the reactive filter: 3.3× with labelling included. (3) Equal total budget against uniform GRPO. This accounting does not appear in the paper; the natural estimate is that 9,368 rollouts buys uniform GRPO 73 updates rather than 40, which we give as an inference below.
Neither (1) nor (2) settles the question, since both compare against a reactive baseline rather than uniform-at-equal-cost. One inference is available, and it is only that: uniform GRPO shows no measurable held-out movement over the forty updates we ran (30.2% to 31.5%, interval straddling zero; second seed 32.0%) with flat training reward, so its marginal value per update in this pool is close to zero. We offer this as an inference rather than as a substitute for the measurement.
On GPU-time: at matched planned rollout counts, PRESAGE's logged policy-training time is 20.1% (seed 1042) and 19.3% (seed 42) below uniform sampling with graded reward, computed as logged duration times configured GPU count. That is what a practitioner schedules against, but it is not the equal-budget comparison.
Q2. Selection and breadth of the held-out set. The 24 tasks were fixed before any model was trained: 18 from marshmallow, 6 from spectree, taken without reference to difficulty. All 24 are excluded from predictor fitting, cross-validation, calibration, threshold selection and every policy-training draw; the saved sampling records confirm none was ever drawn, and E is fit on the remaining 507. We agree the set is small and its repository composition narrow; the leave-one-repository-out numbers on this pool are correspondingly unstable and we do not lean on them.
Broader evaluation (new). We add SWE-bench with a 120-task test set from four repositories excluded from both policy training and predictor fitting:
PRESAGE is +6.17 pp over uniform binary GRPO [4.77,8.99]. Fitted and validated on disjoint repositories, E reaches AUC 0.915 [0.865,0.965] on the 475-task analysis set and 0.893 [0.832,0.951] on the 355-task training set, exceeding a mean-prediction baseline by 0.418 [0.455,0.479] and 0.410 [0.332,0.587]. Validation error was also used to select checkpoints, so these are validation numbers, not a further untouched test.
Q3. Learned predictor versus filtering on the observed labels. This is the hardest question we were asked, and it splits into three cases.
Where observed scores suffice. For a fixed policy on a fixed, already-labelled pool, ranking by observed mean reward is the Bayes-optimal filter, and a predictor can only lose to it. If that were the deployment setting, E would be unnecessary.
Where they do not. (i) Labels are measured under the initial policy; the router needs the score under the current one, and the ordering drifts: dead leakage under a frozen threshold stays flat at 30% while the recalibrated router falls from 32% to 17%. (ii) A new task or repository has no label at all; the 120 SWE-bench test tasks and the out-of-repository AUC of 0.915 are the evidence that E transfers where labels do not exist. (iii) A single K=8 pass is a noisy per-task estimate, whereas E pools evidence across tasks.
Direct comparison. A single arm filtering on the observed labelling-pass scores at the same pool, budget and reward isolates the predictor's contribution outright. We expect the two to be close early and to diverge as the policy moves, which is the mechanism we claim; we present that as a prediction, not a measurement.