- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
CodeLift improves reward design for reinforcement learning of software engineering agents. It groups tests and patches by their execution outcomes, then dynamically weights test groups to reduce biases in raw pass-rate rewards. It also scores intermediate repository snapshots to provide credit within a rollout. Experiments on SWE benchmarks show improvements over binary and pass-rate-based rewards.
Critical strengths
- The paper studies a meaningful limitation of test-based rewards. Figure 2 provides empirical support for the test-suite imbalances motivating the method.
- The approach uses execution results to provide denser supervision, and the reported gains are supported by component ablations and training analyses. Rewarding useful intermediate work is particularly relevant to long-horizon SWE tasks.
Critical weaknesses
- The reliability of behavior-based grouping is unclear with only a small number of sampled rollouts. Tests covering different functionality may happen to produce identical outcomes and be merged. The paper does not establish how sensitive the resulting rewards are to the rollout count or sampling variation.
- The credit assignment in snapshot slicing needs stronger justification. Comparing an intermediate state's score with other rollouts' final scores measures its relative quality, but does not directly measure the contribution of the preceding actions. A regressive slice can still receive positive advantage, while a useful intermediate step can receive negative advantage. The slicing ablation shows an overall benefit, but does not resolve this concern about the chosen credit rule.
Questions for authors
- Why use other rollouts' final scores as the reference for intermediate snapshots? Have you compared this rule with a simple snapshot-to-snapshot progress signal, particularly on trajectories containing temporary regressions?
- How stable are the test groups and rewards across different rollout samples? A sensitivity analysis over the number of rollouts would help assess whether accidental grouping affects learning.
3 comments
Thank you for the clarification. Your response addressed some of my concerns. On balance, I will keep my original score.
Thank you for your support of our work! Your suggestions (e.g., sensitivity) are helpful for improving our work and are adopted in the manuscript! We really appreciate it!
Q1: Other rollouts' final scores as the reference for snapshots
We agree that a state better than the average final state is not necessarily better than the previous state. This is a deliberate choice: CodeLift does credit assignment, not process reward. It spreads the final reward back to earlier steps, instead of scoring each step on its own. We made this choice for two reasons:
Given the reasons above, we select the outcome-baseline credit assignment as final method rather than the snapshot-to-snapshot process method.
Q2: Sensitivity to the number of rollouts
Thank you for raising this. Since retraining with different rollout (2-7) in SWE Agentic RL is heavy, we reuse the 8-rollout training data offline for analysis: For each N from 2 to 7, we take every subset of N rollouts (C(8, N) in total), recompute the groups and rewards, and average three measures:
Even with only 2 rollouts, two tests in the same group disagree on 2.52% of the unsampled rollouts. So accidental grouping rarely merges tests that behave differently on new patches. Across all N, the final advantage changes sign in at most 2.21% of cases. To check accidental grouping alone, we fix the same N patches and compare two options: regroup the tests, or keep the test groups from all 8 rollouts. The mean reward difference is only 0.0102 at N = 4 and 0.0027 at N = 7. So accidental grouping has little effect on rewards. We find this analysis useful and have added it to Appendix C.6.