acceptodds
Under review as a conference paper at ICLR 2027

Beyond Binary Verifiable Rewards: Behavioral Self-Rewarding for SWE Agents

Abstract

Reinforcement learning with verifiable rewards (RLVR) provides reliable supervision for software engineering agents through executable tests, but binary rewards fail to distinguish trajectories with substantially different debugging quality. We investigate whether the current policy can judge its own trajectories and use these judgments to improve task performance without a separately trained reward model. To this end, we introduce Behavioral Self-Rewarding (BSR), a reward-refinement method that reuses the policy to compare observable debugging behavior among trajectories from the same task with the same verifier outcome. BSR aggregates pairwise preferences into task-local scores and converts them into bounded auxiliary rewards, introducing reward contrast while preserving the verifier-defined pass/fail ordering. This provides additional supervision from already-collected trajectories without requiring new environment interactions. Experiments on SWE-bench Verified and SWE-bench-Live show consistent improvements across the evaluated Qwen3-14B and Qwen3-32B configurations, supporting using the policy’s own comparative judgments as a complementary learning signal for training software engineering agents.

Primary area:
reinforcement learning
Keywords:
Software Engineering Agents, LLM Post-Training, Multi-Turn RL

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Est2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Behavior Self-Reward (BSR), which utilizes the current policy model to compare trajectories – all sharing the same software task and verification outcome – thereby generating supplementary learning signals for cases where binary rewards fail to provide differentiation. A posterior “judge” model processes compressed action-observation sequences alongside reference patches for the training tasks; notably, the “actor” model does not access these reference patches during task execution or testing. Pairwise preferences are aggregated via a weighted Bradley-Terry model and subsequently transformed into auxiliary rewards through a pipeline involving normalization, dead-zone application, soft mapping, centering, and magnitude constraints. The magnitude constraints ensure that trajectories with successful outcomes consistently receive higher rewards than those that fail.

The primary configuration focuses on refining the “failure” group and is integrated into the LOOP optimization framework. The authors trained Qwen3-14B and 32B models on a subset of R2E-Gym and evaluated them on SWE-bench Verified, Live Lite, and Terminal-Bench 2.1. For the 14B model, with a budget of 400 steps, the average accuracy on the Verified benchmark improved from 32.1% to 34.7%. Ablation studies – covering grouping strategies, reward mapping, and the impact of randomized rewards – validate the utility of the proposed signals. Training time for the 14B model increased by approximately 19.86%, while the inference (testing) phase requires no judge model.

Critical strengths
  1. To address the lack of discriminative power in binary rewards within a group, the method incorporates explicit constraints. The condition a0+a1<1a_0 + a_1 < 1 ensures that successful trajectories receive higher rewards than failed ones. By reusing the existing policy rather than training a separate reward model, the approach offers significant integration value. While this guarantees the correct algebraic ordering, it does not automatically ensure that the auxiliary signal points in the optimal direction.
  2. Performance gains are observed across two model scales and multiple evaluation settings. Since random auxiliary rewards fail to replicate the gains achieved by BSR, the existing baselines confirm that the rewards indeed contain useful information. The paper also reports an additional training time overhead of approximately 19.86% and provides cost context by comparing 270 steps of BSR against a 400-step baseline.
Critical weaknesses
  1. The independent value of information regarding the behavioral process has yet to be fully recognized, which complicates the assessment of innovation. When an evaluator (judge) is presented with both a behavioral summary and a reference patch, the observed performance gains might stem from the judge’s ability to identify patches that closely resemble the reference solution, rather than from the full debugging process itself. While stochastic rewards can filter out arbitrary noise, they do not necessarily preclude the model from achieving higher scores through simpler, less sophisticated strategies under the same supervisory signal. This paper cites research on hybrid reward schemes (such as HERO); however, any unique incremental contribution should be attributed specifically to the pairwise evaluation of behaviors within the context of the current policy, rather than being broadly ascribed to a combination of “binary rewards plus dense signals.” Furthermore, the use of reference patches during training does not automatically constitute test-set leakage.
  2. The reporting methodology and robustness regarding marginal performance gains require clarification. In Table 1, a figure of 33.2% is reported for the 14B model under a 270-step budget; however, Figure 2 marks this same value at a peak occurring before the budget is exhausted, with subsequent performance dropping to lower levels. This discrepancy could be due to “cherry-picking” the best result within the budget or an issue with how the data is plotted – it does not necessarily imply that the data itself is incorrect. The standard deviation derived from three evaluation runs reflects only the sampling variance at fixed checkpoints and fails to account for the inherent stochasticity of the training process. The reported gains of 1.1-2.6 percentage points in “Verified” performance require further substantiation through consistent selection of evaluation checkpoints, task-level pairwise comparisons, and a more clearly defined framework for assessing computational costs.
Questions for authors
  1. For each row in Table 1, please provide the checkpoint ID, the final step count, and the selection criterion used (i.e., a fixed endpoint or the best-performing checkpoint within the budget). Was the “Verified” test set performance used in the selection process? Were the same criteria applied to the baseline/BSR results? Furthermore, were the same selected models used for both “Live Lite” and “Terminal” settings?
  2. Keeping the judge, reference patches, trajectories, reward mapping, and budget constant, compare the “full behavioral summary” against the “final patch/terminal state.” What is the incremental value of the full process? Additionally, conducting a controlled experiment – either by removing the reference patches or replacing them with a simpler patch-similarity-based reward – would help further clarify the extent to which the approach relies on answer-based supervision.
Reviewer_ojmvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

With binary test rewards, all-fail rollout groups give zero advantage in GRPO/LOOP, so a lot of expensive SWE rollouts teach nothing. BSR reuses the current policy as a judge: within each task, trajectories with the same verifier outcome are compared pairwise (the judge sees the issue, compressed action/observation traces, and the reference patch), the preferences are aggregated with a weighted Bradley-Terry model, and the scores are mapped through a deadzone/tanh/centering step into a small bounded bonus. The bound keeps every passing trajectory above every failing one. In the main setting only the fail bucket is refined (a0=0.15). On Qwen3-14B and 32B trained on R2E-Gym, BSR improves SWE-bench Verified (e.g. 31.0 to 33.2, 32.1 to 34.7), SWE-bench-Live Lite, and Terminal-Bench 2.1. Ablations cover which buckets are refined, the reward mapping, and a random-reward control.

Critical strengths

The problem is real and the fix is practical. All-fail groups are common in SWE agent RL and rollouts are expensive, so recovering signal from them without training a separate reward model, at about 20% extra wall-clock, is useful. The design is careful (comparisons only within the same outcome bucket, a bound that preserves pass > fail ordering, Swiss scheduling to reduce comparisons). Every reported comparison goes in the same direction: average accuracy and Pass@3 across two model sizes, two budgets, and three benchmarks. Individual gains are small, but the consistency counts for something given how costly these runs are.

Critical weaknesses
  1. The obvious non-binary baselines are missing. The title is about going beyond binary verifiable rewards, but BSR is never compared with the simplest dense verifiable signals: the fraction of target tests passed (available in R2E-Gym) and patch similarity to the reference patch (as in SWE-RL), which uses the same information the judge already gets. A heuristic reward built from the statistics that are fed to the judge (num_bash_runs_after_last_edit, num_repeated_actions, num_edits_without_prior_view) is also needed, since the rubric explicitly tells the judge to weight them. The random-reward control only shows that some signal is needed, not that an LLM judgment is needed. If a partial-pass-rate reward matches BSR, the case for self-judging is much weaker.

  2. There is little evidence that the judge's preferences are valid, and some risk of drift. The only check is agreement with Kimi-K2.5 (macro-F1 around 0.6-0.7, decreasing later in training), which measures agreement between two LLM judges, not correctness. The decline could mean later pairs are harder, that the judge degrades, or that the policy is drifting toward behaviors its own judge likes. The rubric rewards stats that are easy to inflate, and the example in Fig. 12 credits an agent-written reproduction script as verification. Within 270-400 steps the bounded bonus probably limits the damage, but the paper doesn't show where the gain actually comes from.

Questions for authors
  1. Please compare against (a) a partial test pass-rate reward and (b) a patch-similarity reward for failed trajectories, under the same bounded mapping and training budget.
  2. Judge validity: does the judge's preference among failed trajectories agree with partial test pass rate or with whether the gold-patch files were edited? What is the position-swap consistency, and how often are preferences cyclic?
  3. To explain the agreement drop: evaluate an early and a late judge checkpoint on a fixed set of early pairs and a fixed set of late pairs (2x2). This is offline and cheap.
  4. Does a better judge give a bigger gain?
Reviewer_dUxtOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Behavioral Self-Rewarding (BSR), which uses the current policy to compare same-task trajectories with the same verifier outcome, conditioned on their debugging behavior and the training task’s reference patch. The preferences are converted into bounded auxiliary rewards that preserve the pass/fail ordering, with reported improvements over verifier-only RL on SWE-bench Verified, SWE-bench-Live Lite, and Terminal-Bench.

Critical strengths

The motivation is practical. Failed SWE trajectories can contain useful investigation and partial progress that binary rewards do not distinguish. Comparing trajectories within the same task and outcome bucket gives the judge a restricted role, while the reward bounds preserve the verifier’s pass/fail ordering.

The reported improvements are consistent across the evaluated model sizes and training budgets. On SWE-bench Verified, average accuracy improves from 31.0% to 33.2% for 14B under the 270-step budget, from 32.1% to 34.7% under the 400-step budget, and from 34.4% to 35.5% for 32B. Improvements on SWE-bench-Live Lite also suggest that the benefit is not limited to one benchmark.

The random-reward and outcome-bucket ablations are useful. Random rewards with the same amplitude range do not reproduce the gain, while failure-only refinement achieves comparable performance to refining both buckets with fewer comparisons. The method also avoids a separate reward-model deployment and requires no judge calls during evaluation.

Critical weaknesses

The contribution of behavioral judgment is not separated from the additional reference-patch supervision. Section 3 gives the judge the training task’s reference patch, whereas the baseline receives only binary rewards. The random-reward control shows that arbitrary reward variation is insufficient, but does not distinguish assessing debugging behavior from recognizing progress toward the reference solution. I would like to see whether the behavior-based comparison still adds value over a simpler reference-based progress signal, and whether it remains useful without the reference patch.

The checkpoint-selection protocol is unclear and affects the reported improvement. Table 1 reports 34.7% versus 32.1% for the 14B/400-step setting, but these correspond to earlier high points in Figure 4; at step 400, the plotted accuracies are 32.2% and 31.3%. Appendix B.1 describes periodic evaluation but does not explain how the reported checkpoints are selected or whether the same checkpoints are used across benchmarks. The reported standard deviations come from repeated evaluations of the same checkpoint, rather than independent training runs, so they do not establish training stability.

The reward mapping can amplify weak preferences in small outcome buckets. The prompts require a winner, and for a bucket with only two trajectories, Equations 8–13 produce approximately the full positive and negative reward amplitudes under the main configuration, even for a slight preference. Appendix B.5 provides an external-judge comparison, but uses newly sampled pairs at each checkpoint and does not establish consistency under repeated judgments or presentation-order changes. It remains unclear how often a weak or unstable comparison becomes a substantial training signal.

Questions for authors

How does BSR perform without giving the judge the reference patch? When the reference is available, does evaluating the debugging trajectory outperform a simpler reference-based progress judgment?

Which checkpoints are reported in Table 1, and what rule selects them? Please clarify whether selection uses SWE-bench Verified results, whether checkpoints are shared across evaluation benchmarks, and how many independent training runs were conducted.

How often do outcome buckets contain only two trajectories? How consistent are the preferences under repeated judgments or swapping trajectory order, and does allowing ties or filtering inconsistent comparisons preserve the improvement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.