acceptodds
Under review as a conference paper at ICLR 2027

Beyond Binary Tests: Auditing Execution-Derived Rewards for Repository-Level Agent RL

Abstract

Repository-level coding agents are attractive targets for reinforcement learning because software can be executed, yet training commonly reduces that evidence to a single test-pass bit. We study two limitations of this paradigm: verifier trustworthiness, because passing tests need not imply that the intended task was solved, and reward starvation, because uniformly failing rollout groups provide no within-group learning signal. We audit execution-derived rewards for selectively certified repository tasks by separating two choices: which executable evidence the environment collects and how that evidence is aggregated. Holding hidden reference executions fixed, we characterize exactly when strict all-pass and fractional aggregation can select differently, then test semantic validity and held-out utility. Across 42 prospectively evaluated model-repair tasks, raw selection opportunity is 0/42 at the predeclared endpoint and 1/42 at and . In 288 natural agent patches, the sole raw opportunity among 18 comparable pools disappears under repository-native semantics. Controlled studies exhibit positive held-out differences on some raw-opportunity pools, but gains are concentrated; a task-disjoint replication has 2/16 raw opportunities and an average-effect interval including zero. Complementary historical real-agent data show that an earlier execution-derived signal recovers variation in 19/22 terminal-binary-flat rollout groups versus 8/22 for repository-test fraction. A frozen adversarial campaign withholds full credit from 21/22 qualifying visible-verifier exploits, while a separate optimizer-interface check verifies a nondegenerate optimizer signal without supporting a causal policy comparison. These results motivate selective executable supervision: determine whether the bottleneck lies in reward semantics, aggregation, evidence construction, or candidate generation rather than forcing every task into a binary or pseudo-dense reward template.

Primary area:
reinforcement learning
Keywords:
repository-level agents, reinforcement learning, executable rewards, reward aggregation, verifier reliability, reward starvation, software engineering agents, code agents

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_seYZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper provides a study on whether using fraction of passing test provide more useful RL reward than binary all-or-nothing score. The authors constructed a 42 tasks evaluation set, and run controlled comparison on how two ways of scoring provide different best-of-n selection. Increasing candidate set size to 16 still yields distinction in best-of-n selection in only 1 task. A separate study reveals that pass rate scoring may not reflect meaningful check against the task requirement.

Critical strengths
  • The controlled comparison of candidate sets in Section 4 provides evidence for the relationship between rollout size and the difference between binary and pass-rate rewards.
Critical weaknesses
  1. The presentation is poor and very hard to understand. The paper title is about "agentic RL using execution-derived rewards". However, the main experiment in Section 4 reveals that pass-rate rewards require a very large candidate set to produce different "best-of-n" selections and may be noisy because of test quality. The example in Figure 1 is confusing for agentic RL: all c1/c2/c3 will get a reward and have different advantage in RL since their score is not equal. I don't find the relationship between "selecting the best one" (which is more like SFT reject sampling or test-time scaling selection) and agentic RL. Furthermore, there is no RL training in the experiment, and the paper mainly studies best-of-n selection. It is difficult to understand how we should choose the reward for agentic RL.
  2. Code inspection shows that gen_model_repair.py (lines 83–99) constructs the buggy prompt directly from the reference checkout without applying defect_patch.diff, which generate_candidates.py (lines 123–130) uses to create the defective version. Thus, the intended bug is never injected into the model’s input in the released generation path. This makes the conclusions in Section 4 questionable.
  3. The evidence is not strong enough to support the claim. The size of the task set is limited, and details about both the construction of the tasks and the tasks themselves are absent.
Questions for authors
  1. What is the relationship between best-of-n selection under different reward rules and agentic coding RL? What is the methodology for choosing the reward type for RL training, and how does this choice affect the results? How should we understand Sections 5–7?
  2. In Section 4, please check the task quality. I found that the bug is never inserted into the repo, so the agent has no injected bug to fix. Why would this set of 42 tasks be representative of RL training data?
Reviewer_8P2HOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper audits execution-derived rewards by separating evidence collection from reward aggregation. For strict all-pass and fractional scoring on identical hidden calls, it characterizes when their top-scoring candidate sets differ and decomposes average held-out selection gain into opportunity prevalence and conditional gain. A prospective model-repair study finds no opportunities on 42 tasks at K=4 and one at K=8/16. In natural agent patches, the only raw opportunity among 18 comparable pools disappears under repository-native semantics. Controlled mutation studies show concentrated positive held-out differences, while separate historical studies examine signal recovery and verifier exploits. The contribution is an offline diagnostic and empirical audit; it does not establish improved RL policy quality.

Critical strengths
  1. The matched-evidence comparison isolates a useful mechanism. Holding probes and candidates fixed avoids attributing the benefit of additional hidden evidence to fractional aggregation. The prospective result is informative even though the gain is zero: under this selector, the observed pools leave essentially no room for aggregation alone to help. The manuscript correctly distinguishes this finite-set statement from a population bound and from RL optimization behavior.

  2. The semantic audit identifies a concrete failure mode in reward evaluation. In Xarray, numerical spread comes from representation ordering despite repository-native equivalence. This example makes a consequential point: reward variation can be manufactured by the comparator. I also appreciate the explicit disclosure of protocol deviations, missing historical records, and the limitations of the optimizer comparison.

Critical weaknesses
  1. The empirical audit has limited coverage and does not complete its own validation chain. The prospective study retains 42 of 82 tasks under one model/prompt configuration. Only 18 natural-agent pools support the aggregation comparison, and none demonstrates a semantically validated opportunity with a standardized held-out gain (§6, §8). In the fresh controlled study, the two beneficial raw opportunities remain semantically unvalidated, the predeclared developer-test endpoint is absent, and the average-effect interval includes zero (§7.2, Appendix D). A and B share a generation mechanism, leaving shared behavioral blind spots possible. The original controlled gain comes entirely from four of 15 tasks; candidate-level selector records are unavailable, and the historical ingest_rsa discrepancy remains unexplained (Appendix C). Harvesting also certifies only 16/87 attempted natural tasks, all in Django. The negative observations are useful, but the small task populations, incomplete positive validation, and restricted coverage leave the empirical basis too thin for a strong general diagnostic contribution.

  2. The connection to RL exploration and policy learning is insufficiently evaluated. The optimizer check has only seven and nine valid steps in its two arms, uses separately trained SFT checkpoints, and suffers different infrastructure exclusions (Appendix F). Nonzero loss establishes that a reward can enter the optimizer; it does not establish better exploration or learning. Moreover, this check uses a prefix/fraction composite score, whereas the main analysis concerns pure fractional aggregation. The K=4/8/16 sweep expands offline repair pools under fixed probes; it does not evaluate how rollout-group size, sampling temperature, or reward normalization affect learning. There is no controlled comparison of learning curves, stability across seeds, or held-out policy solve rates under matched training and inference budgets. This matters because identical top sets can coexist with different RL updates, so the selection identity does not characterize the training effect. I recognize the authors explicitly frame the work as an audit. Even under that framing, the transfer from the offline measurements to actionable conclusions about agent RL remains largely untested, and the limited offline evidence does not compensate for this gap.

Questions for authors
  1. Validation request: Can you complete the repository-appropriate semantic audits for the two fresh opportunities, mbstrdecoder and geomet, and report their selection differences on the predeclared developer-test endpoint? This would directly affect whether I regard them as validated beneficial orderings rather than raw score opportunities.

  2. Diagnostic clarification: Using the already generated A32 evidence, do the nine initially no-full-pass/flat prospective pools remain flat, and does the opportunity anatomy change? What do the retained calls reveal about coverage of the defective behavior? This would help distinguish a candidate bottleneck from insufficient probe resolution without changing the primary A16 result.

  3. RL evidence request: Is a comparison available that starts from the same checkpoint and varies only hidden binary versus hidden fractional reward, with matched training budgets, repeated seeds, and held-out policy evaluation? Even a limited-budget comparison, accompanied by a small rollout-group-size or temperature sensitivity study, would materially affect my assessment. The existing optimizer-interface check cannot answer whether the diagnostic predicts useful exploration or learning behavior.

Reviewer_sovSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies execution-derived rewards for repository-level coding agents, focusing on how strict binary rewards and fractional pass-rate rewards affect candidate selection when candidate code and execution evidence are held fixed. The authors characterize when the two reward formulations lead to different selections and use model-generated repairs, natural agent patches, and controlled code mutations to examine whether these differences are semantically meaningful and improve held-out performance.

Critical strengths

The experiments demonstrate that, in some candidate pools, different reward aggregation schemes lead to different candidate selections and can improve the selected code’s held-out performance. The paper also distinguishes insufficient execution evidence, information loss through reward aggregation, and semantically meaningless score differences. These distinctions suggest potential improvements, including more semantically faithful verifiers, richer execution checks, and revised reward aggregation.

Critical weaknesses

The paper does not systematically trace the chain from reward changes to advantage changes, policy updates, and final model performance. Its analysis remains primarily at the level of scoring and selecting candidates from fixed pools. Changes in candidate selection do not directly establish improvements in the trained policy, while an unchanged top-ranked candidate does not imply an unchanged training signal. Without controlled end-to-end reinforcement learning experiments, the paper offers limited insight into how reward formulations affect learning dynamics and final performance, and its proposed improvement directions remain insufficiently validated.

Questions for authors
  1. How do reward changes affect actual policy updates through advantages? It would be helpful to compare the advantage distributions and resulting policy updates under binary and fractional rewards, using the same initial model, candidate trajectories, and execution evidence. This would establish whether differences in candidate scores translate into meaningful learning signals.

  2. Do these policy updates improve final model performance? Controlled end-to-end RL experiments with the same initial checkpoint, matched training budgets, and multiple random seeds would help answer this question. Evaluating final success rates on held-out tasks would complete the empirical chain from reward changes to advantage changes, policy updates, and final model performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.