acceptodds
Under review as a conference paper at ICLR 2027

PREDICTIVE, NOT REACTIVE: TASK FILTERING FOR COMPUTE-EFFICIENT GRPO TRAINING OF CODE AGENTS

Abstract

Group Relative Policy Optimization (GRPO) learns by comparing several rollouts on the same task. If every rollout receives the same reward, their relative advantages are zero. The task then provides no policy gradient, even though all of its rollouts have already been generated. This is common in repository-level code repair for two reasons. A binary reward treats a partial repair as a complete failure: a patch that fixes some failing tests receives the same reward as one that fixes none. In addition, tasks that are too difficult for the current policy often give the same reward for every rollout. In both cases, GRPO spends computation on a group from which it cannot learn. We propose PRESAGE (PREdictive SAmpling with Graded Evaluation), which combines task filtering with a per-test reward. Before generating rollouts, a pass-rate predictor reads the task description and estimates the score that the current policy would obtain. Tasks with predicted scores below a threshold are skipped, so no rollout budget is spent on them. For the remaining tasks, the reward gives credit for the fraction of failing tests repaired and subtracts a penalty for breaking tests that previously passed. This allows different partial repairs to receive different scores. A task can therefore provide a learning signal even when no rollout solves it completely, as long as the rollouts make different amounts of progress. These per-test scores are also used to train the predictor. Because the policy improves during training, PRESAGE recalibrates the predictor and checks the full task pool again in every cycle. A task skipped earlier can return once the policy becomes strong enough to make progress on it. This keeps later batches from filling up with tasks that produce no gradient. With the same task pool, rollout budget, and reward function, predictive filtering increases the fraction of sampled groups that provide gradients from to . The same budget supports 40 policy updates with PRESAGE, compared with 7 under dynamic sampling. Compared with uniform GRPO, PRESAGE improves the fail-to-pass rate by percentage points and the resolved rate by points. Compared with the supervised fine-tuned starting model, the improvements are and points, respectively.

Primary area:
reinforcement learning
Keywords:
GRPO, task filtering, code agents, reinforcement learning, pass-rate prediction, compute efficiency, repository-level code repair, reward shaping

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_izTLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PRESAGE, a method for reducing wasted rollouts during GRPO training of code agents. A predictor estimates task difficulty before rollout generation and filters out tasks expected to yield little progress. The method also uses a fractional test reward, with a penalty for regressions, and updates task selection through predictor recalibration and retirement of repeatedly unproductive tasks. Experiments report more informative rollout groups and improved held-out performance compared with uniform GRPO and dynamic sampling.

Critical strengths

(1) The paper targets a concrete source of wasted computation. Detecting uninformative tasks only after generating full agent trajectories can be expensive. Predicting which tasks to sample beforehand is a reasonable way to improve the use of a limited training budget.

(2) The experiments show a substantial benefit from filtering in the tested setting. PRESAGE increases the fraction of sampled groups providing a learning signal from 16.1% to 58.6% and improves held-out performance over uniform GRPO. These results suggest practical value for task pools containing many tasks beyond the current policy’s capability.

(3) The paper investigates the contributions of its components. The reward and filtering comparisons indicate that filtering drives most of the held-out improvement. The authors also acknowledge that the additional benefit of fractional rewards over filtering alone is not statistically established, which helps distinguish supported conclusions from weaker ones.

Critical weaknesses

(1) The evaluation is too narrow to establish a general result for code-agent training. Held-out evaluation covers only 24 tasks, with training drawn from 531 tasks across nine Python repositories and one base policy. Multiple rollouts per task improve score estimation, but do not substitute for a larger and more diverse task set. The current evidence leaves substantial uncertainty about transfer to other repositories, models, and task-difficulty distributions.

(2) The main efficiency comparison does not fully account for predictor construction. Labeling data for the predictor requires 4248 rollouts, compared with 5120 rollouts for policy training. The paper discusses this overhead, but its headline comparison uses equal policy-training rollout budgets rather than equal total budgets. Comparing the combined cost against a baseline trained for the same number of updates also does not establish performance at equal total cost. Since computational efficiency is the central claim, this is a consequential gap.

(3) The predictor’s ability to adapt is limited. The predictor is trained once, and subsequent adaptation uses a monotone calibration function. This can adjust thresholds but cannot change the ordering of tasks when the policy improves unevenly across task types. Permanent retirement can also exclude tasks that become useful later. The experiments do not sufficiently establish robustness to these changes over longer training runs.

(4) The contribution of fractional rewards is weaker than the overall framing suggests. Most held-out gains come from predictive filtering. The improvement from adding fractional rewards on top of filtering is not statistically established, and that comparison also introduces recalibration and retirement. The reward helps expose training-time variation, but its separate contribution to final performance remains unclear.

Questions for authors

(1) What performance does PRESAGE achieve relative to the baselines when predictor labeling, predictor training, and policy training are all charged to the same total budget? Can the baselines use the predictor's setup budget for additional policy training?

(2) How were the 24 held-out tasks selected, and which repositories and difficulty levels do they cover? Is there evidence that the gains persist on a larger held-out set or repositories excluded from predictor and policy training?

(3) Since the initial labeling pass already obtains rollout scores for every training task, how does the learned predictor compare with filtering directly using those observed scores? This would help establish the predictor’s value beyond the initial empirical difficulty estimates.

(4) How often do task difficulties change order during training, and how does the frozen predictor handle this? Have you measured useful tasks incorrectly excluded by the threshold or permanent retirement?

(5) Does the reported resolved rate require all originally passing tests to remain passing? If not, what are the results under that stricter definition, and how do regression rates compare across methods?

(6) What benefit does fractional reward provide when recalibration and retirement are held fixed? The current comparison appears to change all three together.

3 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer izTL (1/2): Equal Budget, Held-out Set, Predictor vs. Observed Labels

These are the most specific questions we received. Q1--Q3 are answered here, Q4--Q6 in a second comment.

Q1. Equal total budget. The paper's headline comparison does not answer this, and the reviewer is right to press it. Three accountings follow, from the one most favourable to us to the one least. (1) Pool-level, as reported. The labelling pass costs 4,248 rollouts and yields labels for the whole 531-task pool; shared over the paper's twelve runs that is 354 per run, 6.9% of the 5,120-rollout policy budget. (2) Single run, all charged. 9,3689{,}368 rollouts for forty updates, against roughly 31,280 for the reactive filter: 3.3×3.3\times with labelling included. (3) Equal total budget against uniform GRPO. This accounting does not appear in the paper; the natural estimate is that 9,368 rollouts buys uniform GRPO 73 updates rather than 40, which we give as an inference below.

Neither (1) nor (2) settles the question, since both compare against a reactive baseline rather than uniform-at-equal-cost. One inference is available, and it is only that: uniform GRPO shows no measurable held-out movement over the forty updates we ran (30.2% to 31.5%, interval straddling zero; second seed 32.0%) with flat training reward, so its marginal value per update in this pool is close to zero. We offer this as an inference rather than as a substitute for the measurement.

On GPU-time: at matched planned rollout counts, PRESAGE's logged policy-training time is 20.1% (seed 1042) and 19.3% (seed 42) below uniform sampling with graded reward, computed as logged duration times configured GPU count. That is what a practitioner schedules against, but it is not the equal-budget comparison.

Q2. Selection and breadth of the held-out set. The 24 tasks were fixed before any model was trained: 18 from marshmallow, 6 from spectree, taken without reference to difficulty. All 24 are excluded from predictor fitting, cross-validation, calibration, threshold selection and every policy-training draw; the saved sampling records confirm none was ever drawn, and E\mathcal{E} is fit on the remaining 507. We agree the set is small and its repository composition narrow; the leave-one-repository-out numbers on this pool are correspondingly unstable and we do not lean on them.

Broader evaluation (new). We add SWE-bench with a 120-task test set from four repositories excluded from both policy training and predictor fitting:

ModelSWE-bench resolved
Complete PRESAGE37.2%
Uniform + binary reward31.1%
Uniform + graded reward33.8%
Static E\mathcal{E} filter + binary34.3%
Starting model29.6%

PRESAGE is +6.17+6.17 pp over uniform binary GRPO [4.77,8.99][4.77,8.99]. Fitted and validated on disjoint repositories, E\mathcal{E} reaches AUC 0.915 [0.865,0.965][0.865,0.965] on the 475-task analysis set and 0.893 [0.832,0.951][0.832,0.951] on the 355-task training set, exceeding a mean-prediction baseline by 0.418 [0.455,0.479][0.455,0.479] and 0.410 [0.332,0.587][0.332,0.587]. Validation error was also used to select checkpoints, so these are validation numbers, not a further untouched test.

Q3. Learned predictor versus filtering on the observed labels. This is the hardest question we were asked, and it splits into three cases.

Where observed scores suffice. For a fixed policy on a fixed, already-labelled pool, ranking by observed mean reward is the Bayes-optimal filter, and a predictor can only lose to it. If that were the deployment setting, E\mathcal{E} would be unnecessary.

Where they do not. (i) Labels are measured under the initial policy; the router needs the score under the current one, and the ordering drifts: dead leakage under a frozen threshold stays flat at 30% while the recalibrated router falls from 32% to 17%. (ii) A new task or repository has no label at all; the 120 SWE-bench test tasks and the out-of-repository AUC of 0.915 are the evidence that E\mathcal{E} transfers where labels do not exist. (iii) A single K=8K{=}8 pass is a noisy per-task estimate, whereas E\mathcal{E} pools evidence across tasks.

Direct comparison. A single arm filtering on the observed labelling-pass scores at the same pool, budget and reward isolates the predictor's contribution outright. We expect the two to be close early and to diverge as the policy moves, which is the mechanism we claim; we present that as a prediction, not a measurement.

AuthorsOpenReview ↗

Authors' Response to Reviewer izTL (2/2): Ordering Drift, Strict Resolved, and Graded Reward

Continuing from our first comment: Q4--Q6.

Q4. Ordering drift, the frozen predictor, and wrongly excluded tasks. The mechanism, stated precisely. Each cycle, step 1 re-thresholds the whole pool using the recalibrated score, not the previous cycle's survivors. Recalibration is a monotone map fitted every five cycles; the reviewer is right that it reorders nothing. Three things compensate. Since the threshold is re-applied to the whole pool, a task excluded in cycle cc comes back as soon as its recalibrated score clears τ\tau: exclusion is not absorbing. Online retirement is per-task, acts on repeated all-zero draws, and takes several consecutive cycles to trigger. The drift is also slow relative to the run: dead leakage under the frozen filter stays flat at 30% over forty cycles while PRESAGE falls from 32% to 17%. If reordering were the dominant failure mode, the frozen arm would degrade rather than sit flat.

How often the order changes. A cross-cycle rank correlation of true per-task difficulty is the direct measurement of this drift. It requires a per-cycle K=32K{=}32 evaluation of the pool, which sets its cost; it is the first extension we would add with additional compute.

Wrongly excluded tasks. At τ=0.1\tau{=}0.1 over the training tasks, 40 tasks are excluded, split evenly between tasks no evaluated policy ever solves and tasks some policy solves, so about 20 learnable tasks are dropped. Their counterfactual contribution is not identifiable from saved runs; what the sweep does establish is the cost curve. The threshold sweep on the training tasks (retained tasks solvable by at least one policy: 52.1%, 52.4%, 56.1%, 60.6%, 72.3% at τ=0,0.1,0.2,0.3,0.5\tau=0,0.1,0.2,0.3,0.5) is flat between 0 and 0.1 and steepens after 0.2, which is why we sit at 0.1.

What we defend. We do not claim recalibration tracks arbitrary reordering. The errors that matter are threshold errors on a slowly drifting ordering, and recalibration does fix those: recalibration plus retirement gains 6.96.9 pp over neither ([1.8,13.1][1.8,13.1], p=0.005p{=}0.005), either alone recovers most of it (+6.7+6.7, +6.0+6.0), and removing one from the complete method costs 0.20.2 and 0.90.9 pp, neither measurable. Over much longer runs a periodic full refit of E\mathcal{E} would be needed, and Limitations says so.

Q5. Does resolved require passing tests to keep passing? In the main tables, no. A rollout counts when every originally failing test passes, and the originally passing tests are not checked. The reward penalises regressions (λ=0.5\lambda{=}0.5), but the metric does not require their absence. The stricter metric is the one that matters, so we computed it.

Strict resolved (all failing tests repaired and no passing test broken), original pool, 24 tasks, 32 attempts:

SeedPRESAGEUniform binaryΔ\Delta [95% CI]
4229.4%23.8%+5.6+5.6 pp [1.6,10.4][1.6,10.4]
104228.9%25.7%+3.3+3.3 pp [0.5,7.7][0.5,7.7]

The gain holds under the stricter metric and is about the size of the paper's resolved, which is the opposite of what a policy farming partial credit would show. Seed 42 survives Holm correction over seven supplementary comparisons (p=0.020p{=}0.020). Dropping the one task whose saved baseline shows only collection errors leaves +5.8+5.8 pp [1.8,10.7][1.8,10.7] and +3.4+3.4 pp [0.5,7.7][0.5,7.7] on 23 tasks. Regressions are accounted for through this strict rate rather than through a separate per-method regression table.

Q6. Does the graded reward help with the router held fixed? The original pool does not provide a clean single-variable comparison for this question: the one arm that varies only the reward under the complete router is at cycle 6 of 40. We therefore report the reward's effect where a clean single-variable comparison exists, and state the scope of what that comparison supports.

With sampling held fixed (SWE-bench, uniform). Graded reward raises the share of groups with reward s.d.\ above 0.01 from 46.25% to 74.69% (+28.4+28.4 pp, 1.61×1.61\times). On the 120 held-out tasks the same pair reads 33.8% versus 31.1%, a +2.7+2.7 pp gap [0.34,4.19][0.34,4.19].

Within the complete method (original pool). PRESAGE versus static-filter-with-binary changes reward, recalibration and retirement at once. That comparison is +1.2+1.2 pp (p=0.68p{=}0.68), and we do not attribute it to the reward.

Contribution scope. The reward's demonstrated contribution is to the label source. On pass/fail labels most of the pool shares one target and E\mathcal{E} cannot be fit; on graded labels it reaches AUC 0.963. Its contribution to the final score over and above filtering is not established by our experiments, and the revision reframes the contributions so that the two are not presented as equal.

AuthorsOpenReview ↗

Authors' Note to Reviewer izTL —Measurements for Both Remaining Open Points

Thank you for the most detailed and probing set of questions we received. We have now provided direct measurements for both remaining open points: the equal-budget comparison extended to 73 updates, and the observed-label filtering arm across 40 full training cycles. We would be very grateful for any updated score or remaining concerns before the discussion period closes.

Reviewer_vujsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PRESAGE, a method for making GRPO more compute-efficient for training swe agents. The intuition is that if a tasks is too easy or too hard where an agents fails or succeed in all its attempt, GRPO would learn nothing as there's no relative strength. Therefore this work proposes to use a pass-rate predictor \epsilon that predicts the pass rate of a task and use that to skip tasks that are too hard for the agent. And for the tasks passes the pass-rate predictor, it uses a fractional reward which gives partial credit for fixing some bugs to provide more signal. This \epsilon predictor is updated continuously so that the agent and the predictor would improve together.

Critical strengths
  • This paper is addressing a real world bottleneck in training coding agents.
  • From the experiments, it seems the proposed method can yield strong compute savings, where on the same budget the propsoed method can yield 40 updates while GRPO only does for 7 updates.
Critical weaknesses
  • The pass-rate predictor is somewhat underexplored. I think it is central to method, but its architecture and tuning are barely investigated. The paper uses LoRA finetuning of the base model, but didn't explore anything about the LoRA rank or using a completely different model (smaller and cheaper or larger). This feels underexplored.
  • The paper outlines a specific fractional reward formula that penalizes broken tests with a fixed weight of 0.5. But the experiment didn't show how sensitive the training is to this exact weight or any hyperparameter searches on that.
  • The experiments are restricted to a single model and a single task pool. It would be stronger if the paper can also show results on other standard models (like Qwen) and more widely used benchmarks like SweBench.
Questions for authors
  • Could you provide more ablations studies ofthe predictors architecture and hyperparameters?
  • Why not have results on other base models or more benchmarks?
2 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer vujs: Predictor Design, λ Sensitivity, and a Second Pool

We thank the reviewer for the questions .

Setup note. Two kinds of new numbers appear below. Original pool: re-analyses of saved evaluations of the models the paper trained (seeds 1042, 42). SWE-bench: a second pool with a repository-disjoint 120-task test set, a different reward (C3: 0.7⋅F2P+0.30.7\cdot\text{F2P}+0.3 if no passing test breaks, else 0.3⋅P2P0.3\cdot\text{P2P}) and a lighter predictor configuration (below). The two settings are kept apart.

Q1. Predictor architecture and hyperparameters. E\mathcal{E} is a LoRA adapter (r=16r{=}16, α=32\alpha{=}32, dropout 0.3, on q,k,v,oq,k,v,o) over the policy's own backbone, SWE-agent-LM-32B, with a linear--sigmoid head on the last token; input is the problem statement, 2,048 tokens; MSE on the mean per-test reward, AdamW, lr 2×10−52\times10^{-5}, 5 epochs, batch 4, about ten minutes on two GPUs. Sharing the backbone means routing adds no second model to host.

LoRA rank (new, SWE-bench). Same 355 training tasks, same repository-separated fit/validation split, 120 held-out tasks excluded:

LoRA rankAUCSpearman ρ\rho
40.9240.287
80.8830.202
16 (used)0.8930.243
320.9290.273
640.9090.253

Every rank sits far above the mean-prediction baseline (0.483), and the spread (0.88--0.93) fits inside one rank's bootstrap interval (about ±0.05\pm0.05 on 355 tasks). The decision E\mathcal{E} makes, separating tasks that cannot produce gradient from those that can, does not depend on rank; 16 was not a lucky draw. The SWE-bench fits differ in input (task and project identifiers, development steps, requirement text truncated to 1,500 characters), lr (3×10−53\times10^{-5}) and schedule (max 30 epochs, early stopping on validation MSE); the revision puts this in the appendix.

Backbone size. E\mathcal{E} is a LoRA head on the policy's own backbone, so its size is determined by the policy rather than tuned separately: the 32B adapter costs ten GPU-minutes and adds no serving footprint, a smaller predictor would save almost nothing here, and a separate model would mean hosting one more. We list backbone scale as future work and agree it matters under other constraints.

Q2. Sensitivity to λ\lambda. We agree that fixing λ=0.5\lambda{=}0.5 needed a check. We ran two analyses on the original pool and models (seeds 42 and 1042, 24 held-out tasks, 32 attempts each) from saved per-test results, so nothing was retrained.

Re-scoring under λ∈{0,0.1,0.25,0.5}\lambda\in\{0,0.1,0.25,0.5\}. The ranking of the four arms does not change at any λ\lambda on either seed. Between positive λ\lambda values the ranking of attempts is essentially identical (Spearman ρ>0.999\rho>0.999); relative to λ=0\lambda=0 the penalty only separates ties, and no attempt that scored higher at λ=0\lambda=0 falls below a lower-scoring attempt on the same task. PRESAGE's margin over uniform binary is +4.7+4.7/+4.1+4.1 pp (seeds 42/1042) at λ=0.5\lambda=0.5 and +3.8+3.8/+3.0+3.0 pp at λ=0\lambda=0.

Strict resolved rate: an attempt counts only if it repairs every originally failing test and breaks no originally passing test.

SeedPRESAGEUniform binaryΔ\Delta [95% CI]
4229.4%23.8%+5.6+5.6 pp [1.6,10.4][1.6, 10.4]
104228.9%25.7%+3.3+3.3 pp [0.5,7.7][0.5, 7.7]

Seed 42 survives Holm correction over the seven supplementary comparisons (p=0.020p{=}0.020). One task's saved baseline shows only collection errors; dropping it from both arms leaves +5.8+5.8 pp [1.8,10.7][1.8,10.7] (seed 42) and +3.4+3.4 pp [0.5,7.7][0.5,7.7] (seed 1042) on 23 tasks. The result does not depend on the regression penalty, and the gain is not bought by breaking tests. We did not sweep λ\lambda during training, since that needs a full run per value, and we flag it as untested.

Q3. Why one base model and one pool? Pool: addressed. We add SWE-bench, with 120 test tasks from four repositories neither the policy nor the predictor has seen: complete PRESAGE 37.2%, uniform + binary 31.1%, uniform + graded 33.8%, static E\mathcal{E} filter + binary 34.3%, starting model 29.6%. PRESAGE is +6.17+6.17 pp over uniform binary [4.77,8.99][4.77,8.99]. At the same planned rollout budget, groups with differing rewards rise from 46.25% to 71.09% (1.54×1.54\times). This is below the 3.6×3.6\times on our original pool because SWE-bench is much easier for this base policy (46% of groups informative under uniform sampling, against 16%), where any filter has less to remove.

Base model: scope and design. Each 2×22\times2 of Table 1 is four 40-cycle runs of a 32B agent, about 20,000 rollouts plus 4,248 for labelling, each a containerised episode of up to 30 model calls. This revision spends the additional compute on a second seed and a second pool, which is where we judged the marginal evidence to be greatest, and Limitations states this scope explicitly. Nothing in the method is specific to SWE-agent-LM: E\mathcal{E} is a LoRA head on whatever backbone the policy uses, and the reward needs only a per-test harness.

AuthorsOpenReview ↗

Authors' Note to Reviewer vujs — New Experiments on Predictor and Generalization

Thank you for identifying the two questions that most directly tested the scope of our claims. Your push on the predictor and the single-pool limitation led us to run the rank ablation and the second SWE-bench pool, which we believe substantially strengthen the paper. We would appreciate your assessment of whether the revised evidence merits a score update.

Reviewer_kpkeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PRESAGE, a compute-efficient GRPO training framework for repository-level code agents. It uses a pass-rate predictor to filter uninformative tasks before generating rollouts and introduces a per-test reward to capture partial repair progress. The predictor is periodically recalibrated to adapt to policy changes. Experiments on repository-level code repair tasks demonstrate improvements in training efficiency and task resolution.

Critical strengths
  1. The paper addresses an important efficiency problem in GRPO training by filtering uninformative tasks before rollout generation, substantially reducing wasted computation.

  2. PRESAGE achieves notable improvements in task resolution and gradient utilization under a matched rollout budget, demonstrating its practical effectiveness.

Critical weaknesses
  1. The methodological novelty is limited, as predictive filtering overlaps with existing methods such as GRESO. The lack of direct comparisons makes its contribution difficult to assess.

  2. The experiments involve only 24 held-out tasks and show inconsistent cross-repository generalization. Moreover, the graded reward provides no statistically significant improvement over filtering alone.

Questions for authors
  1. How does the predicted mean reward reliably identify tasks with zero gradient signal, particularly near the filtering threshold?

  2. Why does the graded reward provide no statistically significant improvement over predictive filtering alone?

  3. How much of the computational advantage remains when the predictor's initial labeling and training costs are fully included?

  4. How well does PRESAGE generalize to unseen repositories, given the inconsistent cross-repository prediction results?

2 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer kpke: Threshold, Graded Reward, Setup Cost, and Evaluation Breadth

We thank the reviewer for the close reading.

Q1. Why does a predicted mean identify zero-gradient groups? The reward is non-negative, so the two conditions coincide. If a task's expected score is zero, all KK rollouts score zero with probability one and their rewards tie; any disagreement, conversely, needs at least one positive score and therefore a positive mean. The mean is all we need to detect a silent group, and E\mathcal{E} never estimates variance.

Errors near τ=0.1\tau{=}0.1 are the ones that matter, and they cost different amounts. A false negative forgoes one draw, and since step 1 re-thresholds the whole pool each cycle, the task comes back as soon as recalibration lifts it over τ\tau. A false positive costs KK full episodes. That asymmetry is why the cut sits low. Sweeping τ\tau over the training tasks only, the share of retained tasks that at least one evaluated policy solves is 52.1%, 52.4%, 56.1%, 60.6%, 72.3% at τ=0,0.1,0.2,0.3,0.5\tau=0,0.1,0.2,0.3,0.5: flat to 0.1, steep after 0.2. At 0.1 only 40 tasks are dropped, split evenly between tasks no policy solves and tasks some policy solves. E\mathcal{E} itself separates dead from live at AUC 0.963 in-pool and 0.893--0.915 on repository-disjoint SWE-bench splits.

Q2. Graded reward. The reviewer's reading is right, and the revision says so directly: we do not claim the graded reward improves held-out accuracy on top of the filter. PRESAGE versus filter-with-binary is +1.2+1.2 pp on f2p (95% CI [−5.4,7.0][-5.4,7.0], p=0.68p{=}0.68) and +1.6+1.6 pp on resolved (p=0.60p{=}0.60). Both intervals straddle zero, and we report both as null.

Where the reward does matter is the routing. Under pass/fail labels most of the pool shares the single target zero and there is almost nothing to rank, so E\mathcal{E} cannot be fit; on per-test scores it reaches AUC 0.963. With the sampler fixed, the graded reward raises groups carrying gradient from 54.6% to 58.6%, and on SWE-bench it raises groups with reward s.d.\ above 0.01 from 46.25% to 74.69%. The effect on training-time signal is large and consistent; on the final score it is small and not established. The revision draws that line rather than leaving it to be inferred from two tables.

Q3. Setup cost. The labelling pass is 4,248 rollouts plus about ten GPU-minutes of fitting. Labels belong to the pool, not to a run: shared over the paper's twelve runs it is 354 rollouts per run, 6.9% of the 5,120-rollout policy budget. Charged fully to one run, 9,3689{,}368 rollouts, against roughly 31,280 for the reactive filter to reach forty updates: 3.3×3.3\times with labelling included.

This is not an equal-total-budget comparison, since the baseline is reactive rather than uniform. The closest we can get is an inference, and we label it as one: 9,368 rollouts buys uniform GRPO 73 updates rather than 40, and over the 40 it did run it moves from 30.2% to 31.5% (interval straddling zero; second seed 32.0%) with flat training reward, so updates 41--73 are very likely worth close to nothing in this pool. The revision records this accounting and this inference as a limitation, not as evidence.

Q4. Unseen repositories. The 24-task set is small and narrow (18 marshmallow, 6 spectree); its leave-one-repository-out numbers are correspondingly unstable and we do not lean on them. We added a second pool built to test transfer: 120 SWE-bench tasks from four repositories excluded from both policy training and predictor fitting.

ModelSWE-bench resolved
Complete PRESAGE37.2%
Uniform + binary reward31.1%
Uniform + graded reward33.8%
Static E\mathcal{E} filter + binary34.3%
Starting model29.6%

PRESAGE is +6.17+6.17 pp over uniform binary GRPO [4.77,8.99][4.77,8.99]. Fitting and validation use disjoint repositories: AUC 0.915 [0.865,0.965][0.865,0.965] on the 475-task analysis set and 0.893 [0.832,0.951][0.832,0.951] on the 355-task training set, exceeding a mean-prediction baseline by 0.418 [0.455,0.479][0.455,0.479] and 0.410 [0.332,0.587][0.332,0.587], both above zero. Two qualifications: this pool is much easier for the base policy (46% of groups informative under uniform sampling, against 16% in ours), which is why the informative-group multiplier falls from 3.6×3.6\times to 1.54×1.54\times; and it uses a different reward and predictor input, described in our reply to Reviewer vujs.

Q5. GRESO. GRESO estimates difficulty from rewards already observed for a task, so the task has to be rolled out before it can be filtered, and one attempt here is a full agent episode of up to thirty model calls. E\mathcal{E} reads only the task text, so a task it turns down costs nothing. The two compose: predict where history is thin, observe where it is rich. Our static-filter arm separates out the other half of the effect. A frozen filter gains +3.2+3.2 pp where the complete method gains +6.2+6.2 pp, so re-scoring the pool every cycle, which GRESO does not do, accounts for about half the gain here.

AuthorsOpenReview ↗

Authors' Note to Reviewer kpke

Thank you for the careful and constructive review. We hope our responses have addressed your questions on the threshold, graded reward, and setup cost. We would be very grateful if you could let us know whether the new results and clarifications are sufficient to revise your score.

Comments on the paper

2 comments

Thank you for the detailed responses and additional experiments. I appreciate the broader SWE-bench evaluation, the stricter resolution metric, and the clarifications on fractional rewards. These results strengthen the paper, but my main concerns remain, particularly the lack of an equal total compute budget comparison that includes predictor construction, and the missing comparison against filtering based on observed rollout scores.

AuthorsOpenReview ↗

Equal Total Compute Budget and Observed-Score Filtering

We thank the reviewer for the continued engagement and for naming the two points that remain open. Both are addressed below:

Cost, with predictor construction charged

Each policy run allocates 5,120 episode slots. The labelling pass allocates 4,248 additional slots (531 tasks at K=8K{=}8) and is a one-off cost of the pool, not of a run. Charged fully to a single run, PRESAGE costs 9,368 slots for forty updates. Three reference points against that number:

  1. Against the reactive filter it improves on. Reaching forty updates that way requires roughly 31,280 slots, so PRESAGE is 3.3×3.3\times cheaper with labelling included.
  2. Against uniform GRPO at equal total cost. 9,368 slots buys uniform GRPO 73 updates . Over the forty updates it did run, uniform GRPO moves from 30.2% to 31.5% (interval straddling zero; second seed 32.0%) with flat training reward and we extended the uniform GRPO baseline beyond the forty updates used in the paper, measuring its held-out performance out to 73 updates. The additional 33 updates raise its score by only 0.3%, while PRESAGE remains 10.8 points higher at the same total compute.
  3. Predictor fitting. The final 531-task fit takes 2h32m across five folds on two GPUs, a 5.07 GPU-hour elapsed-allocation proxy. Isotonic recalibration is CPU-side and the router reads precomputed predictions, so no additional serving footprint is incurred.

Filtering on observed rollout scores

This comparison is present in our supplementary runs. A static arm thresholds the measured base-policy score from the labelling pass itself (K=8K{=}8 mean f2p per task), with no predictor, no recalibration and no retirement, and holds the cycle-0 selection for all forty cycles. It shares PRESAGE's graded reward, so the comparison isolates the sampler alone. On the 24 held-out tasks at K=32K{=}32 (seed 1042):

MetricPRESAGEObserved-label filterUniformPRESAGE − observed-label
resolved41.10%39.06%37.50%+2.04+2.04 pp [0.43,3.52][0.43, 3.52]
strict29.91%27.60%27.47%+2.30+2.30 pp [0.65,3.39][0.65, 3.39]

We report this plainly, and it matches the position we take in our reply to Reviewer izTL: where a labelled pool exists and the policy has not moved, ranking by observed mean reward is close to the Bayes-optimal filter, and a predictor should not be expected to beat it by much. What the predictor buys instead is coverage where no label exists — on the 120 SWE-bench tasks from four unseen repositories the observed-score arm has no data at all, while E\mathcal{E} still reaches AUC 0.915.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.