- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Behavior Self-Reward (BSR), which utilizes the current policy model to compare trajectories – all sharing the same software task and verification outcome – thereby generating supplementary learning signals for cases where binary rewards fail to provide differentiation. A posterior “judge” model processes compressed action-observation sequences alongside reference patches for the training tasks; notably, the “actor” model does not access these reference patches during task execution or testing. Pairwise preferences are aggregated via a weighted Bradley-Terry model and subsequently transformed into auxiliary rewards through a pipeline involving normalization, dead-zone application, soft mapping, centering, and magnitude constraints. The magnitude constraints ensure that trajectories with successful outcomes consistently receive higher rewards than those that fail.
The primary configuration focuses on refining the “failure” group and is integrated into the LOOP optimization framework. The authors trained Qwen3-14B and 32B models on a subset of R2E-Gym and evaluated them on SWE-bench Verified, Live Lite, and Terminal-Bench 2.1. For the 14B model, with a budget of 400 steps, the average accuracy on the Verified benchmark improved from 32.1% to 34.7%. Ablation studies – covering grouping strategies, reward mapping, and the impact of randomized rewards – validate the utility of the proposed signals. Training time for the 14B model increased by approximately 19.86%, while the inference (testing) phase requires no judge model.
Critical strengths
- To address the lack of discriminative power in binary rewards within a group, the method incorporates explicit constraints. The condition ensures that successful trajectories receive higher rewards than failed ones. By reusing the existing policy rather than training a separate reward model, the approach offers significant integration value. While this guarantees the correct algebraic ordering, it does not automatically ensure that the auxiliary signal points in the optimal direction.
- Performance gains are observed across two model scales and multiple evaluation settings. Since random auxiliary rewards fail to replicate the gains achieved by BSR, the existing baselines confirm that the rewards indeed contain useful information. The paper also reports an additional training time overhead of approximately 19.86% and provides cost context by comparing 270 steps of BSR against a 400-step baseline.
Critical weaknesses
- The independent value of information regarding the behavioral process has yet to be fully recognized, which complicates the assessment of innovation. When an evaluator (judge) is presented with both a behavioral summary and a reference patch, the observed performance gains might stem from the judge’s ability to identify patches that closely resemble the reference solution, rather than from the full debugging process itself. While stochastic rewards can filter out arbitrary noise, they do not necessarily preclude the model from achieving higher scores through simpler, less sophisticated strategies under the same supervisory signal. This paper cites research on hybrid reward schemes (such as HERO); however, any unique incremental contribution should be attributed specifically to the pairwise evaluation of behaviors within the context of the current policy, rather than being broadly ascribed to a combination of “binary rewards plus dense signals.” Furthermore, the use of reference patches during training does not automatically constitute test-set leakage.
- The reporting methodology and robustness regarding marginal performance gains require clarification. In Table 1, a figure of 33.2% is reported for the 14B model under a 270-step budget; however, Figure 2 marks this same value at a peak occurring before the budget is exhausted, with subsequent performance dropping to lower levels. This discrepancy could be due to “cherry-picking” the best result within the budget or an issue with how the data is plotted – it does not necessarily imply that the data itself is incorrect. The standard deviation derived from three evaluation runs reflects only the sampling variance at fixed checkpoints and fails to account for the inherent stochasticity of the training process. The reported gains of 1.1-2.6 percentage points in “Verified” performance require further substantiation through consistent selection of evaluation checkpoints, task-level pairwise comparisons, and a more clearly defined framework for assessing computational costs.
Questions for authors
- For each row in Table 1, please provide the checkpoint ID, the final step count, and the selection criterion used (i.e., a fixed endpoint or the best-performing checkpoint within the budget). Was the “Verified” test set performance used in the selection process? Were the same criteria applied to the baseline/BSR results? Furthermore, were the same selected models used for both “Live Lite” and “Terminal” settings?
- Keeping the judge, reference patches, trajectories, reward mapping, and budget constant, compare the “full behavioral summary” against the “final patch/terminal state.” What is the incremental value of the full process? Additionally, conducting a controlled experiment – either by removing the reference patches or replacing them with a simpler patch-similarity-based reward – would help further clarify the extent to which the approach relies on answer-based supervision.