- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper provides a study on whether using fraction of passing test provide more useful RL reward than binary all-or-nothing score. The authors constructed a 42 tasks evaluation set, and run controlled comparison on how two ways of scoring provide different best-of-n selection. Increasing candidate set size to 16 still yields distinction in best-of-n selection in only 1 task. A separate study reveals that pass rate scoring may not reflect meaningful check against the task requirement.
Critical strengths
- The controlled comparison of candidate sets in Section 4 provides evidence for the relationship between rollout size and the difference between binary and pass-rate rewards.
Critical weaknesses
- The presentation is poor and very hard to understand. The paper title is about "agentic RL using execution-derived rewards". However, the main experiment in Section 4 reveals that pass-rate rewards require a very large candidate set to produce different "best-of-n" selections and may be noisy because of test quality. The example in Figure 1 is confusing for agentic RL: all c1/c2/c3 will get a reward and have different advantage in RL since their score is not equal. I don't find the relationship between "selecting the best one" (which is more like SFT reject sampling or test-time scaling selection) and agentic RL. Furthermore, there is no RL training in the experiment, and the paper mainly studies best-of-n selection. It is difficult to understand how we should choose the reward for agentic RL.
- Code inspection shows that gen_model_repair.py (lines 83–99) constructs the buggy prompt directly from the reference checkout without applying defect_patch.diff, which generate_candidates.py (lines 123–130) uses to create the defective version. Thus, the intended bug is never injected into the model’s input in the released generation path. This makes the conclusions in Section 4 questionable.
- The evidence is not strong enough to support the claim. The size of the task set is limited, and details about both the construction of the tasks and the tasks themselves are absent.
Questions for authors
- What is the relationship between best-of-n selection under different reward rules and agentic coding RL? What is the methodology for choosing the reward type for RL training, and how does this choice affect the results? How should we understand Sections 5–7?
- In Section 4, please check the task quality. I found that the bug is never inserted into the repo, so the agent has no injected bug to fix. Why would this set of 42 tasks be representative of RL training data?