- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces PROPEL, a framework for training task generators using an offline-trained activation probe as a surrogate for expensive solver feedback. The probe predicts whether a generated task falls within a target solver’s desired solve-rate band, using representations from a frozen reference model. The approach extends feature-based rewards to multi-turn SWE trajectories and evaluates difficulty targeting across code induction, mathematics, and software engineering. Results show improved target-band yield on code tasks and held-out SWE repositories, while the mathematics gains are conditional on oracle verification.
Critical strengths
- The paper addresses a meaningful computational bottleneck. Repeated solver rollouts are particularly expensive for agentic tasks, so amortizing this feedback could make generator training substantially more practical.
Critical weaknesses
-
The reliability of the fixed reward under optimization remains insufficiently characterized. The probe is trained on base-generator outputs, but RL progressively changes this distribution. Freezing the reference model stabilizes the representation function; it does not prevent the generator from exploiting the classifier’s errors. This matters given the SWE probe’s modest balanced accuracy/AUC of 0.594/0.596, increased topic concentration, and the rise in entirely unsolved SWE tasks. Endpoint utility gains are encouraging, but do not establish how long the proxy remains useful. An evolving solver would introduce an additional mismatch, although I recognize that the paper explicitly studies a fixed solver.
-
The methodological advance over existing feature-reward and task-generation methods appears limited, and the validation does not yet establish a compelling practical advantage. The central recipe largely applies RLFR to solver-derived difficulty labels, with trajectory pooling and existing ensemble techniques. Comparisons against a conventional learned reward model and probe-based filtering would help establish what the proposed formulation adds.
Questions for authors
- How do probe discrimination, high-score precision, and solver-verified utility change across generator checkpoints? Do you have evidence that the proxy remains reliable as optimization moves away from the offline training distribution?
4 comments
Official Comment by Authors (Part 2)
2. Novelty and trajectory representation (W2)
The core modification is the reward target. We show that a task's difficulty for a separate solver is decodable from the activations of a frozen copy of the generator, and that optimizing this signal shifts difficulty. For example, a Qwen3.5-4B reference model predicts the outcomes of Qwen2.5-3B and 7B solvers. This removes solver rollouts from generator RL, and PROPEL reaches higher utility than solver-in-the-loop training with fewer solver trials (Table 2 in the paper).
Applying this to task generation raises two problems that we address in our paper. Multi-turn SWE trajectories need a suitable representation, which has a large effect on the probe (Table B). Optimizing a fixed probe can concentrate topics, which we diagnose and partially mitigate with probe ensembles and co-evolution (Section 5 and Appendix I).
In Table B we show that the choice of trajectory representation has a large effect on the probe's performance in SWE. With everything aside from the trajectory representation held fixed, a probe on the final turn gives a lower AUC and a negative correlation with solve rates on held-out repositories. On the other hand, averaging over the latter half of the trajectory raises AUC and gives the strongest held-out correlation among the nine aggregation strategies in our sweep.
Table B. SWE probe by trajectory representation, mean ± SD over three probe seeds.
| Trajectory representation | CV AUC | Repository held-out correlation |
|---|---|---|
| Final turn | 0.543 ± 0.008 | −0.302 ± 0.131 |
| Average over all turns | 0.596 ± 0.006 | 0.167 ± 0.114 |
| Average over the latter half, used by PROPEL | 0.596 ± 0.002 | 0.418 ± 0.041 |
3. Learned reward model (W2)
In Table C we show that PROPEL reaches higher validity than a fine-tuned text reward model at equal or higher utility, without fine-tuning a reward model. Using the same labeled data we trained the suggested text reward model that reads the generator prompt and generated task and outputs its score. Training this reward model uses LoRA adapters in Qwen3.5-4B and a classification head, while PROPEL fits a small probe on activations of the frozen reference model.
Table C. Learned text reward model and PROPEL on AZR, with the matching single generator seed for both methods.
| Method | Validity | Utility |
|---|---|---|
| Base | 67.19% | 8.69% |
| Learned text reward model | 81.74% | 19.24% |
| PROPEL | 93.36% | 22.36% |
4. Probe filtering (W2)
In Table D we show that training produces 6.79× higher utility than probe filtering at a comparable target share among accepted tasks. We ranked valid Base outputs by probe score and retained different fractions. A stricter filter raises the target share among the tasks it accepts, but it discards most candidates, so fewer target tasks remain per generation. Retaining the top 20% of valid Base outputs reaches the target share of PROPEL's outputs but a far lower utility. To obtain one target task, this filter needs about 34 generated candidates, against 5 for PROPEL. Training the generator therefore supplies far more useful tasks than filtering a fixed one, and it also improves executable validity (Table 3 in the paper).
Table D. Probe filtering of Base outputs and PROPEL on AZR, mean over three seeds. Utility is target tasks per generation, as in Table C.
| Method | Target share among accepted tasks | Utility |
|---|---|---|
| Base, all valid outputs | 12.99% | 8.95% |
| Base + probe, top 50% of valid outputs | 17.82% | 6.15% |
| Base + probe, top 20% of valid outputs | 21.18% | 2.93% |
| Base + probe, top 10% of valid outputs | 24.90% | 1.73% |
| PROPEL, all valid outputs | 21.65% | 19.89% |
Thank you again for your time and valuable feedback. If our clarifications and additional comparisons address the concerns, we would be grateful if you would consider raising your score. We would be happy to answer any further questions.
Thank you for the additional checkpoint results. It improve my confidence that it provides a useful training signal on AZR. I'm still concerning the whether these gains hold on other tasks over the course of training and persists over longer runs as the generator’s output distribution shifts. I will maintain my score as 2.
Official Comment by the Authors
Thank you for the follow-up and for engaging with our rebuttal.
We strongly disagree with the conclusion that the gains are shown on AZR only. SWE is probably the strongest result of the paper at the largest scale and here PROPEL doubles utility on repositories unseen by the probe and the generator (Table 1 in the paper).
We also emphasize that the gains do persist over longer runs. We have run training up to step 50 on AZR, which is well beyond the checkpoint used in the paper, and find that target tasks per generation remain high at 18.16% ± 0.20% vs 8.95% ± 1.58% for Base.
We hope this gives you further confidence in our paper and that you might consider this in your final score together with all the other points we have addressed in the rebuttal.
Official Comment by Authors
Thank you for recognizing the importance of reducing repeated solver rollouts. We first summarize the main points of our rebuttal and then address each concern in detail, including the suggested probe diagnostics and the learned reward model and filtering baselines.
Brief summary of the main points:
1. Probe reliability during optimization (W1 & Q1)
We agree that freezing the reference model does not prevent the generator from exploiting probe errors, which is why every reported utility comes from fresh solver trials.
In Table A we show that the probe reward raises solver-verified utility within the first 10 steps and that this gain is retained through step 30, together with the checkpoint diagnostics you suggested. RL with validity reward alone leaves utility among valid tasks at the Base level, so the difficulty gain comes from the probe reward. The further rise in target tasks per generation comes from further improving validity. On the generator optimized against it, the probe's AUC is at chance from step 10 and high-score precision stays close to the utility of all valid outputs. This is expected once the generator's outputs concentrate where the probe scores highly, and it does not reduce the benefit of training.
Table A. Fixed probe on AZR across generator checkpoints, mean ± SD across three seeds. Step 30 is the checkpoint used by the paper. Steps 10 and 20 use 512 generations per seed and the other rows 1,024.
Top-20% precision is the target-band fraction among the highest-scored 20% of valid tasks. Utilities use independent K=8 solver trials.
How long the proxy remains useful. We report results up to step 30, the checkpoint used by the paper, and do not claim that a single fixed probe remains useful indefinitely. Refreshing the probe to match the evolving generator's distribution is a promising direction for longer horizons, and Appendix I in the paper takes a first step by fitting an auxiliary probe on the original probe's false positives.
Unsolved SWE tasks. The SWE probe provides a valuable training signal for the generator. On repositories unseen by the probe and the generator, utility doubles under fresh solver trials (Table 1 in the paper). The rise in unsolved bugs (Table 7 in the paper) comes from harder bugs that remain valid and solvable. To verify this we ran a substantially stronger solver (Claude Code with Opus 5.5) on the bugs generated by PROPEL that the Qwen solver never fixed, in the same sandboxes and with the same verifier, and it resolved all of them.