acceptodds
Under review as a conference paper at ICLR 2027

Breaking the Solver Bottleneck: Training Problem Generators at the Learnable Frontier

Abstract

The limiting resource for training agents via reinforcement learning (RL) is increasingly frontier task supply: valid, solvable tasks just difficult enough to train the current model. As reasoning and agentic models improve, fixed task distributions saturate, while naive synthetic generation yields tasks that are trivial, impossible, or ill-posed. Training a task generator with RL to optimize validity and learnability can address this bottleneck, but direct optimization requires repeated solver rollouts per candidate. For software-engineering (SWE) tasks, a single rollout can take tens of minutes, and solver-in-the-loop generator training needs several such rollouts for every candidate task at every update. We introduce **PROPEL**, a solver-amortized framework for training task generators at the targeted solve rate. PROPEL trains a lightweight activation probe on a one-time labelled corpus of generated tasks and solver outcomes. The probe predicts target-solver pass rate from a frozen generator reference model and serves as a proxy for solve rate during generator optimization, reducing generator evaluation to a single forward pass. Across math, code, and software-engineering at multiple model scales, PROPEL shifts generation toward the targeted solve rate: for coding, tasks generated at the learnable frontier increase from %% for a *Qwen2.5-3B-Instruct* solver and from %% for a *Qwen2.5-7B-Instruct* solver. For SWE, PROPEL increases targeted solve rate generations from %% for *Qwen3.5-27B* on repositories not seen during training of probe and generator.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RL environment generation, Reinforcement learning from feature rewards

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_o2EhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PROPEL, a framework for training task generators using an offline-trained activation probe as a surrogate for expensive solver feedback. The probe predicts whether a generated task falls within a target solver’s desired solve-rate band, using representations from a frozen reference model. The approach extends feature-based rewards to multi-turn SWE trajectories and evaluates difficulty targeting across code induction, mathematics, and software engineering. Results show improved target-band yield on code tasks and held-out SWE repositories, while the mathematics gains are conditional on oracle verification.

Critical strengths
  • The paper addresses a meaningful computational bottleneck. Repeated solver rollouts are particularly expensive for agentic tasks, so amortizing this feedback could make generator training substantially more practical.
Critical weaknesses
  • The reliability of the fixed reward under optimization remains insufficiently characterized. The probe is trained on base-generator outputs, but RL progressively changes this distribution. Freezing the reference model stabilizes the representation function; it does not prevent the generator from exploiting the classifier’s errors. This matters given the SWE probe’s modest balanced accuracy/AUC of 0.594/0.596, increased topic concentration, and the rise in entirely unsolved SWE tasks. Endpoint utility gains are encouraging, but do not establish how long the proxy remains useful. An evolving solver would introduce an additional mismatch, although I recognize that the paper explicitly studies a fixed solver.

  • The methodological advance over existing feature-reward and task-generation methods appears limited, and the validation does not yet establish a compelling practical advantage. The central recipe largely applies RLFR to solver-derived difficulty labels, with trajectory pooling and existing ensemble techniques. Comparisons against a conventional learned reward model and probe-based filtering would help establish what the proposed formulation adds.

Questions for authors
  • How do probe discrimination, high-score precision, and solver-verified utility change across generator checkpoints? Do you have evidence that the proxy remains reliable as optimization moves away from the offline training distribution?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for recognizing the importance of reducing repeated solver rollouts. We first summarize the main points of our rebuttal and then address each concern in detail, including the suggested probe diagnostics and the learned reward model and filtering baselines.

Brief summary of the main points:

  • The probe reward causes the difficulty gain. RL with validity reward alone leaves utility among valid tasks at the Base level, while PROPEL raises it within the first 10 steps and retains it through step 30.
  • Unsolved SWE bugs are valid and solvable. A substantially stronger solver resolves all of them in the same sandboxes and with the same verifier.
  • Trajectory aggregation improves prediction. The original SWE ablation supports the representation used by PROPEL.
  • Both requested baselines are evaluated. PROPEL reaches higher validity than the text reward model at equal or higher utility, and supplies 6.79× more target tasks than filtering Base at a comparable target share among accepted tasks.

1. Probe reliability during optimization (W1 & Q1)

We agree that freezing the reference model does not prevent the generator from exploiting probe errors, which is why every reported utility comes from fresh solver trials.

In Table A we show that the probe reward raises solver-verified utility within the first 10 steps and that this gain is retained through step 30, together with the checkpoint diagnostics you suggested. RL with validity reward alone leaves utility among valid tasks at the Base level, so the difficulty gain comes from the probe reward. The further rise in target tasks per generation comes from further improving validity. On the generator optimized against it, the probe's AUC is at chance from step 10 and high-score precision stays close to the utility of all valid outputs. This is expected once the generator's outputs concentrate where the probe scores highly, and it does not reduce the benefit of training.

Table A. Fixed probe on AZR across generator checkpoints, mean ± SD across three seeds. Step 30 is the checkpoint used by the paper. Steps 10 and 20 use 512 generations per seed and the other rows 1,024.

GeneratorProbe AUCTop-20% precisionUtility among valid tasksTarget tasks per generation
Base0.64 ± 0.0321.18% ± 1.13%12.99% ± 2.27%8.95% ± 1.58%
RL with validity reward, step 300.64 ± 0.0417.85% ± 2.54%11.67% ± 0.61%11.07% ± 0.54%
PROPEL, step 100.49 ± 0.0417.21% ± 5.10%19.43% ± 1.20%15.49% ± 0.69%
PROPEL, step 200.52 ± 0.0322.47% ± 5.04%21.77% ± 2.46%19.73% ± 2.82%
PROPEL, step 300.49 ± 0.0119.80% ± 1.62%21.65% ± 3.14%19.89% ± 2.97%

Top-20% precision is the target-band fraction among the highest-scored 20% of valid tasks. Utilities use independent K=8 solver trials.

How long the proxy remains useful. We report results up to step 30, the checkpoint used by the paper, and do not claim that a single fixed probe remains useful indefinitely. Refreshing the probe to match the evolving generator's distribution is a promising direction for longer horizons, and Appendix I in the paper takes a first step by fitting an auxiliary probe on the original probe's false positives.

Unsolved SWE tasks. The SWE probe provides a valuable training signal for the generator. On repositories unseen by the probe and the generator, utility doubles under fresh solver trials (Table 1 in the paper). The rise in unsolved bugs (Table 7 in the paper) comes from harder bugs that remain valid and solvable. To verify this we ran a substantially stronger solver (Claude Code with Opus 5.5) on the bugs generated by PROPEL that the Qwen solver never fixed, in the same sandboxes and with the same verifier, and it resolved all of them.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2)

2. Novelty and trajectory representation (W2)

The core modification is the reward target. We show that a task's difficulty for a separate solver is decodable from the activations of a frozen copy of the generator, and that optimizing this signal shifts difficulty. For example, a Qwen3.5-4B reference model predicts the outcomes of Qwen2.5-3B and 7B solvers. This removes solver rollouts from generator RL, and PROPEL reaches higher utility than solver-in-the-loop training with fewer solver trials (Table 2 in the paper).

Applying this to task generation raises two problems that we address in our paper. Multi-turn SWE trajectories need a suitable representation, which has a large effect on the probe (Table B). Optimizing a fixed probe can concentrate topics, which we diagnose and partially mitigate with probe ensembles and co-evolution (Section 5 and Appendix I).

In Table B we show that the choice of trajectory representation has a large effect on the probe's performance in SWE. With everything aside from the trajectory representation held fixed, a probe on the final turn gives a lower AUC and a negative correlation with solve rates on held-out repositories. On the other hand, averaging over the latter half of the trajectory raises AUC and gives the strongest held-out correlation among the nine aggregation strategies in our sweep.

Table B. SWE probe by trajectory representation, mean ± SD over three probe seeds.

Trajectory representationCV AUCRepository held-out correlation
Final turn0.543 ± 0.008−0.302 ± 0.131
Average over all turns0.596 ± 0.0060.167 ± 0.114
Average over the latter half, used by PROPEL0.596 ± 0.0020.418 ± 0.041

3. Learned reward model (W2)

In Table C we show that PROPEL reaches higher validity than a fine-tuned text reward model at equal or higher utility, without fine-tuning a reward model. Using the same labeled data we trained the suggested text reward model that reads the generator prompt and generated task and outputs its score. Training this reward model uses LoRA adapters in Qwen3.5-4B and a classification head, while PROPEL fits a small probe on activations of the frozen reference model.

Table C. Learned text reward model and PROPEL on AZR, with the matching single generator seed for both methods.

MethodValidityUtility
Base67.19%8.69%
Learned text reward model81.74%19.24%
PROPEL93.36%22.36%

4. Probe filtering (W2)

In Table D we show that training produces 6.79× higher utility than probe filtering at a comparable target share among accepted tasks. We ranked valid Base outputs by probe score and retained different fractions. A stricter filter raises the target share among the tasks it accepts, but it discards most candidates, so fewer target tasks remain per generation. Retaining the top 20% of valid Base outputs reaches the target share of PROPEL's outputs but a far lower utility. To obtain one target task, this filter needs about 34 generated candidates, against 5 for PROPEL. Training the generator therefore supplies far more useful tasks than filtering a fixed one, and it also improves executable validity (Table 3 in the paper).

Table D. Probe filtering of Base outputs and PROPEL on AZR, mean over three seeds. Utility is target tasks per generation, as in Table C.

MethodTarget share among accepted tasksUtility
Base, all valid outputs12.99%8.95%
Base + probe, top 50% of valid outputs17.82%6.15%
Base + probe, top 20% of valid outputs21.18%2.93%
Base + probe, top 10% of valid outputs24.90%1.73%
PROPEL, all valid outputs21.65%19.89%

Thank you again for your time and valuable feedback. If our clarifications and additional comparisons address the concerns, we would be grateful if you would consider raising your score. We would be happy to answer any further questions.

Thank you for the additional checkpoint results. It improve my confidence that it provides a useful training signal on AZR. I'm still concerning the whether these gains hold on other tasks over the course of training and persists over longer runs as the generator’s output distribution shifts. I will maintain my score as 2.

AuthorsOpenReview ↗

Official Comment by the Authors

Thank you for the follow-up and for engaging with our rebuttal.

We strongly disagree with the conclusion that the gains are shown on AZR only. SWE is probably the strongest result of the paper at the largest scale and here PROPEL doubles utility on repositories unseen by the probe and the generator (Table 1 in the paper).

We also emphasize that the gains do persist over longer runs. We have run training up to step 50 on AZR, which is well beyond the checkpoint used in the paper, and find that target tasks per generation remain high at 18.16% ± 0.20% vs 8.95% ± 1.58% for Base.

We hope this gives you further confidence in our paper and that you might consider this in your final score together with all the other points we have addressed in the rebuttal.

Reviewer_zS1vOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PROPEL, which trains lightweight probes to provide proxy rewards for task generators, reducing reliance on expensive solvers during reinforcement learning. Experimental results demonstrate substantial improvements on code induction and SWE tasks.

Critical strengths

● Addresses a practically important efficiency bottleneck. PROPEL uses lightweight probes to provide proxy rewards for task generators, reducing reliance on expensive online solver calls during reinforcement learning. This design offers clear practical value and a promising approach to lowering the cost of solver-driven task generation. ● Provides empirical evidence of effectiveness across task domains. The method achieves substantial improvements on code induction tasks and increases the utility metric from 9.84% to 20.91% on unseen SWE repositories. These results provide convincing empirical support for PROPEL's effectiveness and strengthen the credibility of the paper's main claims.

Critical weaknesses

The long-term generalization of the fixed probe has not been sufficiently validated. The proposed method relies on a fixed probe to estimate task utility, but the paper does not sufficiently investigate whether the probe can maintain reliable predictive performance as the generator continues to improve and the task distribution shifts during training. If the probe's predictive accuracy declines over time, its utility estimates may become less reliable, potentially undermining the effectiveness of the method over longer training horizons. This limitation raises concerns about the robustness and applicability of the method beyond the experimental settings considered in the paper.

Questions for authors

Given the observed reduction in task diversity, could the authors clarify whether this reduction substantially affects the practical utility of the generated tasks? Is there evidence in the existing experiments that the utility gains do not primarily result from excessive concentration of the task distribution?

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your positive assessment of PROPEL's practical value and empirical results. We address the fixed-probe concern and the diversity question below.

1. Long-term reliability of the fixed probe (W1)

In Table A we show that training with the fixed probe more than doubles target yield, verified by independent solver trials. This is the outcome the probe is used for, i.e., the probe is sufficient to steer the generator toward the target band. We observe that its ranking of the optimized generator's outputs falls to chance during training (Table A), however, this does not reduce the benefit of training. Utility among valid tasks rises within the first 10 steps and is retained through step 30, while the further rise in target tasks per generation comes from improved validity. We argue that the probe as reward needs to move the generator toward target tasks, which we verified with separate solver trials.

Table A. Probe AUC and target yield on AZR across generator checkpoints, mean ± SD across three seeds, with independent K=8 solver trials. Step 30 is the checkpoint used by the paper. Steps 10 and 20 use 512 generations per seed and the other rows 1,024.

GeneratorProbe AUCUtility among valid tasksTarget tasks per generation
Base0.64 ± 0.0312.99% ± 2.27%8.95% ± 1.58%
PROPEL, step 100.49 ± 0.0419.43% ± 1.20%15.49% ± 0.69%
PROPEL, step 200.52 ± 0.0321.77% ± 2.46%19.73% ± 2.82%
PROPEL, step 300.49 ± 0.0121.65% ± 3.14%19.89% ± 2.97%

For longer training horizons, refreshing the probe to match the evolving generator's distribution is a promising direction. Appendix I in the paper takes a first step toward such a refresh by fitting an auxiliary probe on the original probe's false positives from the trained generator, and we leave periodic refresh to future work.

We note that the original fixed probe also generalizes beyond the setting it was trained in. Without retraining, it nearly doubles AZR utility for a Mistral generator (Figure 3 in the paper) and in SWE, PROPEL doubles utility on repositories unseen by both the probe and the generator (Table C).

2. Diversity and practical utility (Q1)

In Table B we show that PROPEL produces more target tasks within a matched topic, so the utility gain does not come from topic concentration alone. We compare all methods on the topic PROPEL generates most, sorting and ordering (task categories of Appendix B). Within this topic PROPEL is above Base and RL with validity reward in every seed. Topic concentration alone would lift Base only to its own rate within this topic.

Table B. Target tasks per generation among sorting and ordering tasks, mean ± SD over three seeds.

MethodTarget tasks per generation
Base11.51% ± 2.13%
RL with validity reward12.29% ± 0.84%
PROPEL20.23% ± 2.95%

In Table C we show that SWE utility doubles on unseen repositories while the diversity metrics change little. The generated AZR programs also remain structurally distinct within the target band, and our ensemble ablation in the paper lowers topic concentration while preserving a utility gain (Table 1 in the paper).

Table C. SWE results on held-out repositories (means ± SE from Table 1 in the paper). OOD utility is normalized over valid bugs with usable solver evaluations.

MethodOOD utilitySelf-BLEUDistinct-3Top-edited-file share
Base9.84%0.8110.0850.605
PROPEL20.91% ± 1.31%0.862 ± 0.0360.107 ± 0.0150.650 ± 0.068

Thank you again for your time and constructive feedback. If these additional analyses address your concerns about probe reliability and task diversity, we would be grateful if you would consider raising your score. We would be happy to answer any further questions.

Reviewer_1hkGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PROPEL, which trains task generators using activation probes learned from offline solver outcomes. Features from a frozen reference model provide a reward that replaces repeated solver evaluations during generator RL. Experiments in mathematics, code induction, and software engineering examine difficulty targeting, computational savings, and transfer. The strongest evidence comes from controlled code induction experiments, while mathematical verification and reduced task coverage limit the broader interpretation of the results.

Critical strengths
  1. The method addresses a concrete bottleneck in training task generators. Reusing an offline probe removes repeated solver calls from the optimization loop, and the AZR comparison includes the offline labeling cost when assessing solver usage. This provides credible support for the intended amortization benefit.

  2. The AZR validity control and alternative target bands demonstrate that the reward can steer difficulty beyond improving executability. Evaluation on SWE repositories excluded from both probe training and generator training adds useful evidence of transfer. These are substantive validations of the proposed application.

Critical weaknesses
  1. The technical novelty is limited. The core reward mechanism follows RLFR, with the main additions being task utility prediction and the extension to SWE trajectories.

  2. Training usefulness remains insufficiently established. Utility is defined through a fixed solver’s success band, leaving its connection to subsequent learning unvalidated.

  3. Task coverage narrows under optimization. Structural uniqueness does not resolve the increased topic concentration and reduced representation coverage, which the proposed mitigations only partially address.

  4. The math gains depend on oracle filtering. In the highlighted settings, higher conditional utility brings little or no improvement in verified target task yield per generation.

  5. The SWE results do not clearly isolate difficulty targeting from improved bug validity. The utility denominator is also insufficiently explicit in the main text.

  6. The cost comparison includes offline labeling but primarily measures solver calls in AZR. The main results do not establish the overall computational advantage of the complete pipeline in SWE.

Questions for authors
  1. Which modification to RLFR is essential here, and what evidence establishes the benefit of the SWE trajectory representation?

  2. What supports interpreting the selected success band as training usefulness for these generated tasks?

  3. Does coverage still narrow when comparing only valid tasks within the target success band?

  4. Does the SWE utility denominator include all generations, valid bugs, or only bugs with usable solver evaluations?

  5. How much of the SWE improvement remains after accounting for changes in bug validity?

  6. What is the overall computational saving after including probe fitting, reference model inference, and validity checking, particularly in SWE?

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1)

Thank you for your feedback and for recognizing the computational motivation, the AZR ablations and the evidence of transfer to unseen SWE repositories.

We briefly summarize the main points of our rebuttal and then respond to each raised weakness and question in more detail below. Briefly summarised our main points are:

  • The original ablation (see Table A) shows improved difficulty prediction when pooling across turns, so trajectory representation matters.
  • Utility gains persist when accounting for validity gains on SWE and when comparing within a matched topic on AZR.
  • We analyze solver training on AZR and show that tasks in the target band provide the most learning signal and improve fastest (see Table B).
  • We report online cost savings and estimates of total computation that include the fitting corpus and solver labels. PROPEL is clearly ahead on all of them.

1. Novelty and trajectory representation (W1 & Q1)

The core modification is the reward target. In our setting, the activations of a frozen reference copy of the generator predict how a separate solver will perform on the generated task. We show that the difficulty of a task for a target solver is decodable from the hidden states of this frozen reference model, for example, with a Qwen3.5-4B reference model predicting the outcomes of Qwen2.5-3B and 7B solvers, and that optimizing this signal shifts difficulty.

In addition to the novelty of the reward signal itself, SWE tasks are multi-turn agent trajectories that require a trajectory representation and in PROPEL optimizing a fixed probe can concentrate task topics, which we diagnose and partially mitigate with probe ensembles and co-evolution (Section 5 and Appendix I).

In Table A we show that the choice of trajectory representation has a large effect on the probe's performance in SWE. With everything aside from the trajectory representation held fixed, a probe on the final turn gives a lower AUC and a negative correlation with solve rates on held-out repositories. On the other hand, averaging over the latter half of the trajectory raises AUC and gives the strongest held-out correlation among the nine aggregation strategies in our sweep.

Table A. SWE probe by trajectory representation, mean ± SD over three probe seeds.

Trajectory representationCV AUCRepository held-out correlation
Final turn0.543 ± 0.008−0.302 ± 0.131
Average over all turns0.596 ± 0.0060.167 ± 0.114
Average over the latter half, used by PROPEL0.596 ± 0.0020.418 ± 0.041

2. Training usefulness of the target band (W2 & Q2)

The target band contains the tasks a solver learns most from, and we have run solver training on our generated tasks to confirm this.

During RL training, a solver learns only from tasks with non-zero group variance. This is also the justification for the proposer reward in Absolute Zero and Self-play SWE-RL (see Section 3.2). The analysis in Self-play SWE-RL places the optimal solve rate near 0.2 for eight attempts, which our band of 1–3 successes in 8 attempts brackets.

We tested this by training the Qwen2.5-3B solver with GRPO on PROPEL tasks and grouping the tasks by their solve count before training. In Table B we observe that indeed target band tasks provide the strongest learning signal and the largest performance gains during training.

Table B. Solver training on PROPEL tasks by task difficulty.

Solve count before training (of 8)TasksLearning signalAccuracy gain
023473.7%+6.4
1–3 (target band)7080.2%+11.4
4–75161.1%+9.0
81423.8%−0.2

Learning signal is the share of task presentations whose eight rollouts receive different rewards. The training reward is the fraction of hidden examples solved, so tasks never fully solved still carry signal. Accuracy gain is the trend in accuracy on hidden examples, in points per 100 updates. Values average two seeds, and the target band ranks first on both in each. This experiment directly connects the target band of the fixed solver to subsequent learning. In Table C we show that PROPEL increases the supply of these tasks beyond the gain from optimizing validity only.

Table C. Target tasks per generation for the band of 1–3 successes in 8 attempts, mean ± SD over three seeds.

MethodTarget tasks per generation
RL with validity reward11.1% ± 0.5%
PROPEL19.9% ± 3.0%
AuthorsOpenReview ↗

Official Comment by Authors (Part 2)

3. Diversity and task coverage (W3 & Q3)

In Table D we show that structural diversity is preserved within the target band, while coverage stays as narrow as among all valid tasks. Nearly all PROPEL programs have a unique structure, within the target band and among all valid tasks. Conditioning tasks on the target band does not change the coverage over the base distribution meaningfully. The narrower coverage of PROPEL is already stated as a limitation in the paper.

Table D. Diversity of PROPEL tasks (1–3 successes in 8 attempts).

Subset of PROPEL tasksUnique abstract program structuresCoverage (PRDC recall)
All valid tasks99.7%0.361
Tasks within the target band99.8%0.378

In Table E we show that PROPEL produces more target tasks within a matched topic, so the utility gain does not come from topic concentration alone. We compare all methods on the topic PROPEL generates most, sorting and ordering (task categories of Appendix B). Within this topic PROPEL is above Base and RL with validity reward in every seed. Topic concentration alone would lift Base only to its rate within this topic.

Table E. Target tasks per generation among sorting and ordering tasks, mean ± SD over three seeds.

MethodTarget tasks per generation
Base11.51% ± 2.13%
RL with validity reward12.29% ± 0.84%
PROPEL20.23% ± 2.95%

In Table F we show that the probe ensemble in the paper reduces topic concentration while retaining a utility gain. Optimizing the minimum reward across two probes (WCO) lowers the largest topic share while keeping utility well above Base and the co-evolution variant in Appendix I yields the largest topic share to the Base level while retaining a utility gain over Base. These mitigations are partial, as we already note in the paper, but nevertheless provide a promising path forward. We also note that loss of diversity under reward optimization is a general problem in RL, and that methods developed to mitigate it can be directly applied to PROPEL.

Table F. AZR with the 7B solver. Utility corresponds to target tasks per generation.

MethodLargest topic shareUtility
Base29.3%5.27%
PROPEL67.4%12.55%
PROPEL with probe ensemble, WCO53.5%9.81%

4. Math verification (W4)

We emphasize that the math result does not call the method into question. The probe amortizes solver feedback on difficulty, and it does so in math as well. Utility among verified tasks rises by approximately 1.7× at both solver sizes. The probe is not meant to supply the task validity signal, which is why yield per generation under the strict oracle does not improve (Section 5). It is important to note that Math does not have the same validity check during training, and the strict oracle is applied after generation only, outside the RL objective (see Sections 4.2 and 5, Appendix A). In AZR and SWE, executable validity checks are part of training, and both validity and utility improve substantially throughout. The math result therefore highlights the importance of validity supervision alongside difficulty optimization, which is how PROPEL is trained wherever that signal exists.

5. SWE validity and utility (W5 & Q4–Q5)

In Table G we show that both validity and utility improve in SWE, with utility already measured among valid bugs. The denominator of utility is valid bugs with usable solver evaluations. Among these bugs the target-band fraction more than doubles, so the utility gain is net of validity.

Table G. SWE results on held-out repositories (Table 1 in the paper). Validity is valid bugs per generation. PROPEL values are mean ± SE over two seeds.

MethodValidityOOD utility
Base13.27%9.84%
PROPEL35.18% ± 0.64%20.91% ± 1.31%

In AZR the ablation we ran separates validity from difficulty directly. RL with validity reward reaches the highest validity but a much lower target yield than PROPEL (Table C).

AuthorsOpenReview ↗

Official Comment by Authors (Part 3)

6. Total SWE computation (W6 & Q6)

In Table H we show that PROPEL uses 3.50× fewer GPU-hours on a matched SWE update, measured end to end, and that the advantage remains once all setup costs are included. The measurement covers generation, validity checks, reference-model and probe scoring, optimization and allocation overhead. The matched update is the first of training. PROPEL's full 30-update run takes about 348 GPU-hours, about the cost of four SIL updates. For the total, we estimate model computation over 30 updates, charging PROPEL for the probe training corpus, solver labels, activation extraction, probe fitting, reference scoring and generator training, and SIL for solver feedback on the same tasks. SIL's solver cost grows as validity improves. PROPEL requires 59.77% less estimated computation and recovers its setup cost at update 13.

Table H. SWE computation for SIL and PROPEL.

QuantitySILPROPELBasis
GPU-hours for the first update83.0223.69Measured
Total model computation with setup, 30 updates (10¹⁸ FLOPs)637.43256.45Estimated

Thank you again for your time and feedback on our work. If our clarifications and additional results address your concerns, we would be grateful if you would consider raising your score. We would be happy to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Author response

We thank the reviewers for their feedback. We provide a summary of the additional experiments and clarifications that resulted from the discussion phase. Please refer to the individual responses for details.

  • The utility gain holds throughout training and comes from the probe reward (zS1v, o2Eh). We added independent solver trials at every generator checkpoint, together with the requested probe diagnostics. PROPEL’s yield of target tasks rises to about twice that of the base generator. RL with validity reward alone does not increase utility among valid tasks, so the gain comes from the probe reward.
  • PROPEL’s gains do not come from topic concentration alone (1hkG, zS1v). We added a comparison within a matched topic, where PROPEL produces more target tasks than the base model and RL with validity reward.
  • The target band contains the tasks a solver learns most from (1hkG). We added solver training on PROPEL tasks, in which target-band tasks give the strongest learning signal and the largest accuracy gains.
  • Our trajectory representation on SWE matters for probe performance (1hkG, o2Eh). Our ablation over trajectory representations shows that a probe on the final turn correlates negatively with solve rates on held-out repositories, while averaging over the latter half of the trajectory gives the strongest held-out correlation.
  • We added probe-based filtering and a text-based reward model as baselines (o2Eh). PROPEL reaches higher validity than a learned text reward model at equal or higher utility, and 6.79× the utility of probe-based filtering at a comparable target share among accepted tasks.
  • SWE, our largest-scale result (all reviewers). PROPEL doubles utility on repositories unseen by the probe and the generator. We clarify that this is measured at the end of the complete training run and among valid bugs, so the gain is on top of the validity improvement. We ran a check with a substantially stronger solver, which resolves all bugs the original solver never fixes, so the hardest bugs are valid and solvable. We added an end-to-end cost measurement, in which PROPEL uses 3.50× fewer GPU-hours on a matched SIL SWE update, and the advantage remains once all costs are accounted for.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.