- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies reward design in solver–auditor systems, focusing on the relationship between discouraging solver errors and maintaining auditing incentives. Motivated by a simplified game, it uses an outer-loop bandit controller to select reward profiles for co-training based on principal value (PV). Code-generation experiments show fewer incorrect attempts and improved in-distribution PV, while downstream utility gains remain unestablished.
Critical strengths
- The problem formulation is valuable. The paper considers how to maintain effective oversight while constraining the generator, and clearly distinguishes training rewards from the system’s ultimate utility.
- The experiments provide encouraging preliminary evidence. Improved in-distribution PV and evidence of selective abstention support reward design as an approach to learning when to answer and when to abstain.
Critical weaknesses
- It remains unclear whether the method achieves a better risk–coverage tradeoff. The reported hallucination rate counts incorrect attempts among all samples, so it depends on answer coverage. The evidence for selective abstention and improved in-distribution overall pass rate means the gains cannot simply be dismissed as more abstention. However, without selective-answering baselines at matched coverage, the advantage over simple confidence filtering remains unclear. On LiveCodeBench, fewer incorrect attempts do not translate into higher overall pass rate or proxy PV than fixed-binary, limiting the scope of the conclusions.
- The independent contribution of feedback-driven reward selection remains unclear. Comparisons with static rewards do not sufficiently distinguish benefits from changing rewards during training from those of effective feedback-driven selection. Random or fixed-cycle schedules over the same reward library would provide direct and important controls. The single-seed main comparisons also limit confidence in a consistent advantage.
Questions for authors
- Regarding W1: Does the method retain lower error rates or higher PV than confidence-filtered baselines at matched answer coverage, with filtering rules calibrated on separate data?
- Regarding W2: Can random or fixed-cycle schedules over the same reward library achieve similar results under matched training budgets? Please also report repeated training results if available to clarify the consistency of the gains.
- Additional clarification: The cross-checkpoint analysis provides some evidence about the auditor’s contribution. Could the authors further clarify the contribution of auditor learning and how the single-assert auditing format limits generalization to more complex tasks?