acceptodds
Under review as a conference paper at ICLR 2027

SWE-AaaV: Building Adaptive Executable Evidence for Coding Agent Rewards

Abstract

Large language models have made rapid progress on software engineering tasks, where pre-defined unit tests are widely used as verifiable rewards for training coding agents. However, limited coverage and implementation-specific assertions can cause them to accept incorrect solutions or reject valid ones, and such misclassifications are difficult to eliminate at scale. In this paper, we propose SWE-AaaV, an Agent-as-a-Verifier framework that constructs adaptive executable evidence for coding agent rewards. Rather than directly judging patch correctness, it generates paired test scripts conditioned on the candidate and reference solutions. To reduce false negatives, the paired scripts accommodate implementation differences while testing the same problem-scoped behavior. To uncover false positives, the verifier agent uses white-box differential testing to find issue-relevant failures and regressions, with the reference solution serving as a behavioral oracle. A deterministic harness checks reference validity, base consistency, and resolved evidence, returning feedback for test revision. For validated evidence, execution on the candidate determines the final reward. To make this candidate-conditioned verification practical for reinforcement learning (RL), we integrate it with asynchronous reward computation and hybrid scheduling. In offline evaluation, SWE-AaaV improves verifier AUC by 4.55-5.98 points over the strongest model-based baselines. When used to provide RL rewards, SWE-AaaV improves SWE-bench Verified performance by 1.8 points over standard unit-test rewards, with only a 15% increase in wall-clock training time. These results demonstrate a practical approach to evaluation-time scaling for coding agent rewards.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Coding Agents, Agentic Verification, Reinforcement Learning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AdKMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SWE-AaaV, an adaptive verification framework for rewarding coding agents when fixed unit tests provide incomplete or overly restrictive assessments of candidate patches. A verification agent constructs paired executable tests conditioned on candidate and reference implementations, while a deterministic harness checks their consistency and evidence of issue resolution. Asynchronous verification and a windowed FIFO mechanism integrate these rewards into reinforcement learning. Evaluation covers 618 candidate patches from SWE-bench Verified, using unit-test outcomes as proxy labels alongside sampled human adjudication and ablations. With stronger verifier backbones, the framework reports AUC improvements of 4.55-5.98 points over model-based baselines. In RL experiments, SWE-AaaV improves SWE-bench Verified performance by 1.8 points over standard unit-test rewards, with ~15% additional training wall-clock training time.

Critical strengths
  1. The experimental design is thorough and comprehensive, covering offline verifier evaluation, ablation studies, and RL results.

  2. Efficiency is well taken into account. The asynchronous reward computation alleviates the slowdown introduced by the complex verification system, showing that the authors have carefully considered the potential challenges of deploying this work in large-scale RL training.

Critical weaknesses
  1. The more accurate reward does not yield a clear performance gain in the whole RL pipeline. In Figure 4(a), the accuracy curves of SWE-AaaV and the Test Verifier are nearly identical over the first 500 RL steps, and only in the final 200 steps does a mere 1.8 percentage gap emerge. This inevitably raises doubts about the importance of the reward system claimed in this paper — perhaps the less precise reward provided by the Test Verifier is already sufficient?

  2. Lack of novelty. Prior work has already explored using agents as verifiers in RL training, yet this paper lacks comparative experiments against such similar methods [1]. Furthermore, regarding asynchronous reward computation, many mature implementations already exist in industry, e.g., the fully async mode of verl [2]; the authors should supplement experiments demonstrating the performance advantages of their proposed method over these existing implementations.

[1] The Verification Horizon: No Silver Bullet for Coding Agent Rewards

[2] https://verl.readthedocs.io/en/latest/advance/fully_async.html

Questions for authors
  1. In the full RL pipeline, could you roughly estimate how many samples' correctness is misjudged by the Test Verifier but correctly judged by SWE-AaaV? This could be estimated via manual sampling or LLM-as-a-judge.
4 comments

Author Response Overview: Thank you for recognizing the breadth of our experiments and training-efficiency work. Below, we add training-time reward adjudication and clarify the relationship to prior verifiers and asynchronous systems.

W1 & C1: Downstream RL Gains and Reward Corrections

TL;DR: Estimated reward-correction rates increase with curriculum difficulty: 8.2% -> 12.0% -> 20.6% across the three intervals. RL gains remain 1.6–2.4 pp across cross-comparisons of two independent experiments.

To avoid additional test-execution overhead affecting the efficiency measurements, we did not run Test Verifier during SWE-AaaV RL. We instead conducted a post-hoc audit across three curriculum intervals of increasing difficulty. For each interval, we sampled 500 rollouts and ran Test Verifier on their candidate patches offline. We then sampled 10 cases from each disagreement direction within that interval for manual adjudication.

Table R1. Training disagreement rates and SWE-AaaV correctness in audited disagreements. Verdict order: SWE-AaaV / Test Verifier.

Training stepsDisagreement rateFail/PassPass/FailDisagreement correctness
[0, 250)16.4%60.0% (6/10)40.0% (4/10)50%
[250, 500)21.8%60.0% (6/10)50.0% (5/10)55%
[500, 700]29.4%70.0% (7/10)70.0% (7/10)70%
Average22.5%63.3%53.3%58.3%

Results: Using direction-balanced audit accuracy, the estimated reward-correction rates of 8.2%, 12.0%, and 20.6%, showing a positive association with curriculum difficulty. In the final interval, manual adjudication favors SWE-AaaV in 7/10 cases in each direction. This pattern is also consistent with the late-stage RL gains. A plausible explanation is that complex issues make correct and comprehensive predefined tests harder to construct, while candidate-conditioned verification can target the particular patch. Moreover, an independent RL repetition yields 48.0% versus 50.2%, alongside the original 47.8% versus 49.6%. The two results per method imply a 1.6–2.4 pp gain across cross-comparisons.

W2: Novelty Relative to Prior Agentic Verifiers and Asynchronous Integration

TL;DR: Our distinction is harness-validated paired executable evidence as RL rewards. We build on an existing asynchronous framework, with integration reducing relative step time from 2.00× to 1.15×. Our contributions are the SWE-AaaV framework and its efficient integration into RL.

Prior verifiers. In [1], the SWE quality judge assesses task–test alignment for training-data filtering, while the interactive and long-horizon evaluators are not applicable to SWE RL. Dockerless [2] uses read-only repository exploration to judge patches, but we could not locate a public release of its trained verifier. SWE-AaaV's distinguishing feature is paired executable evidence checked by a deterministic harness, with the validated outcome used directly as RL rewards.

Asynchronous integration. We build on an existing asynchronous RL framework and integrate asynchronous reward computation and windowed FIFO to support agentic verification. The systems contribution lies in adapting and evaluating these mechanisms for agentic rewards, whose multi-turn verification creates a substantially greater scheduling burden than a single-response LLM judge. Figure 4(b) quantifies the benefit of this integration within our implementation: relative step time falls from 2.00× to 1.40× with asynchronous rewards, then to 1.15× with windowed FIFO.

[1] Qwen Team. The verification horizon: No silver bullet for coding agent rewards. 2026.

[2] Wenhao Zeng, et al. Dockerless: Environment-free program verifier for coding agents. 2026.

Thank you for the response. The 1.6–2.4 pp improvement from RL is indeed relatively modest, and integrating asynchronous rewards into an existing RL framework appears to contribute more to the practical implementation of RL systems than to methodological novelty. While I acknowledge the authors' substantial contributions to RL infra, I think the academic novelty of the work remains somewhat limited. Therefore, I maintain my original score.

Thank you for engaging in this discussion! Our previous response focused too heavily on RL infra and did not sufficiently highlight the core verification mechanism.

We distinguish two contributions:

  • Methodology: generate candidate-conditioned paired tests, validate their evidence through mandatory execution checks, and determine rewards from candidate execution.
  • Engineering: integrate this costly verification process into an existing asynchronous RL framework.

Experiment results. 1. The 1.6-2.4 pp gain is an additional improvement over the best performance achieved with test verifier. Relative to the initial policy, SWE-AaaV achieves an absolute RL improvement of up to 10.6 points. 2. The ablations also support the methodological contribution: "w/o validation" retains the same information and execution access, providing a general baseline of agentic verifier. Removing this mandatory validation reduces AUC by 7.89 points on Qwen and 3.11 points on DeepSeek.

To our knowledge, SWE-AaaV is the first framework to separate paired executable evidence generation from mandatory evidence validation for coding agent rewards. We respectfully ask you to reconsider the methodological novelty and contribution.

Dear Reviewer AdKM,

Thank you again for your time and for continuing the discussion. As the author discussion period ends on November 18 (AoE), we briefly summarize our responses to your concerns below.

  1. Reward corrections during training. Manual adjudication estimates that SWE-AaaV corrected the test verifier on 8.2% of rollouts in the easiest training interval and 20.6% in the hardest. An independent RL run also reproduced the gain over the test verifier (+2.2 pp).

  2. Methodological contribution. Our latest reply explains how SWE-AaaV generates paired tests for each candidate patch and validates the evidence before assigning a reward. With reference and execution access held fixed, removing validation lowered AUC by 7.89 points on Qwen and 3.11 points on DeepSeek.

If you have a moment before the discussion closes, we would appreciate your thoughts on whether this clarifies the methodological contribution.

Reviewer_U87VOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SWE-AaaV, an Agent-as-a-Verifier framework for constructing adaptive executable evidence for coding-agent rewards. Instead of directly predicting whether a candidate patch is correct, SWE-AaaV uses a verifier agent to inspect the issue, base repository, reference solution, and candidate solution, and generate paired test scripts tailored to the reference and candidate implementations. A deterministic execution harness then validates the generated evidence through reference validity, base consistency, and resolved-evidence checks before using the candidate-side execution result as the final reward. The paper further integrates this relatively expensive candidate-conditioned verification into RL through asynchronous reward computation and windowed FIFO scheduling. Experiments report improvements over model-based verifiers under unit-test proxy labels and show a 1.8-point improvement on SWE-bench Verified when SWE-AaaV is used as the training reward.

Critical strengths
  1. Clear and compelling formulation. The paper provides an intuitive framing of verifier agents as constructors of executable evidence. The separation between flexible evidence construction and deterministic evidence acceptance is conceptually clean and well motivated.

  2. Strong presentation. The paper is very well written and easy to follow. The progression from the limitations of predefined test suites, to paired adaptive tests, execution-based validation, and finally training-time integration is particularly smooth. The notation and figures also make the overall framework easy to understand.

  3. Practical consideration of training-time verification. The paper goes beyond offline verifier evaluation and studies how expensive agentic verification can be incorporated into RL. The asynchronous reward computation and scheduling optimizations make the proposed verifier substantially more practical as a training-time reward.

Critical weaknesses
  1. Tables 1 and 2 use the outcomes of the predefined test verifier VtV_t as proxy labels for computing AUC, FNR, and FPR. However, a central motivation of the paper is precisely that VtV_t can produce both false positives and false negatives and therefore differs from the ideal correctness oracle V∗V^*. Consequently, a correct disagreement with VtV_t is counted as an error in these tables. Figure 3 partially addresses this concern through manual adjudication and indeed shows that some disagreements are cases where SWE-AaaV is correct and VtV_t is wrong. However, only a small sampled subset of disagreements is adjudicated, and the paper explicitly notes that this analysis cannot estimate the overall error rates of either verifier. Moreover, important details of the manual adjudication protocol, such as annotator expertise, blindness to verifier predictions, annotation guidelines, and pre-discussion agreement, are not reported. Thus, the reported AUC improvements are more accurately improvements against unit-test proxy labels rather than demonstrated improvements in discrimination of true patch correctness.

  2. Section 4.3 presents adaptive white-box differential testing as a central component, where Reference-Candidate code differences guide behavioral hypotheses and targeted test generation. However, unlike adaptation and validation, this mechanism is not independently ablated. The w/o reference variant simultaneously removes the behavioral oracle, Reference-Candidate comparison, and other reference-grounded information, and therefore does not isolate the contribution of differential testing itself. This attribution issue is particularly relevant given the strong backbone dependence visible in the results. In the manually adjudicated disagreements in Figure 3, Qwen3.6-35B-A3B frequently loses to the predefined test verifier, especially when SWE-AaaV accepts a candidate that the test verifier rejects, whereas DeepSeek-V4.1-Flash shows substantially stronger results in both disagreement directions. This leaves it unclear how much of the gain should be attributed to the proposed differential-testing framework itself versus the capability of a sufficiently strong verifier backbone to reason over the issue, reference, candidate, and execution environment.

Questions for authors
  1. Can the authors provide an evaluation against an independent correctness ground truth rather than VtV_t? For example, could a sufficiently large subset of candidates be independently adjudicated and then used to report AUC/FNR/FPR for SWE-AaaV, VtV_t, and the other verifier baselines under the same human correctness labels? It would also be helpful to clarify the current adjudication protocol, including what information was visible to annotators, whether they were blinded to verifier predictions, the annotation guidelines, and inter-annotator agreement before discussion.

  2. Can the authors better isolate the contribution of adaptive white-box differential testing and characterize its dependence on verifier capability? For example, an ablation could retain access to the reference solution and the same execution environment while removing explicit Reference-Candidate differential guidance or delta-driven test planning. This would help distinguish the benefit of differential testing from the more general benefit of providing a strong agent with privileged reference information and execution access. Given the large difference between Qwen3.6-35B-A3B and DeepSeek-V4.1-Flash in Figure 3, it would also be useful to clarify what level of verifier capability is required for SWE-AaaV to reliably outperform predefined tests.

4 comments

Author Response Overview: Thank you for recognizing the clear formulation, presentation, and practical training integration. Below, we add human correctness evaluation and clarify the annotation procedure, differential-testing contribution, and backbone dependence.

W1 & C1: Evaluation Beyond Unit-Test Proxy Labels

TL;DR: Based on more manual auditing, the estimated results support Qwen SWE-AaaV < (benchmark-level) Test Verifier < DeepSeek SWE-AaaV. We also clarify the current adjudication protocol and report 88.1% pre-discussion agreement.

SWE-bench Verified's human-screened tests by OpenAI provide a common proxy for comparing methods. To evaluate beyond that proxy, we supplement the disagreement audit with 20 examples from each agreement category (Pass/Pass and Fail/Fail) for each backbone. The shared verdict is judged correct in 18/20 cases in each category for Qwen3.6-35B-A3B, and 19/20 cases in each category for DeepSeek-V4.1-Flash. For each backbone, we weight the four audited categories by their frequencies to estimate correctness over all samples. Under these audit-based estimates, SWE-AaaV underperforms Test Verifier with Qwen3.6-35B-A3B but outperforms it with DeepSeek-V4.1-Flash.

Table R2. Offline verifier performance (%) estimated from manual audits weighted by category frequency.

Backbone comparisonVerifierAUCFNRFPR
Qwen3.6-35B-A3BTest Verifier86.3611.9415.34
Qwen3.6-35B-A3BSWE-AaaV76.9215.5130.65
DeepSeek-V4.1-FlashTest Verifier84.3015.5715.82
DeepSeek-V4.1-FlashSWE-AaaV92.306.259.14

Annotation procedure. Both annotators have over five years of Python testing experience. Annotators receive the executable container, the issue, and the repository with base/candidate/reference commits, together with the benchmark test patch and generated scripts. These testing materials are provided without identifying their source and serve only as auxiliary evidence, not authoritative correctness labels. Claude Code with Claude Opus 4.8 is available for investigation. Each annotator independently establishes the issue's scope, checks the candidate's behavior, and runs additional tests when needed. The decision is whether the candidate resolves the issue without relevant regressions. In the annotation process, pre-discussion agreement was 88.1%.

W2 & C2: Differential Testing and Backbone Dependence

TL;DR: New ablation experiments show that differential guidance is useful for SWE-AaaV, particularly for false positives. The audit-based estimates show that DeepSeek-V4.1-Flash has outperformed the predefined tests, but a general capability threshold cannot be established.

We add an ablation removing explicit differential-testing guidance while keeping the available information unchanged. The model can therefore still perform differential testing spontaneously. Removing explicit guidance increases FPR for both backbones, indicating that the guidance helps identify defects and reject incorrect patches.

Table R3. Differential-guidance ablation results (%).

BackboneVariantAUCFNRFPR
Qwen3.6-35B-A3BSWE-AaaV78.5615.1327.75
Qwen3.6-35B-A3Bw/o guidance75.2016.4233.17
DeepSeek-V4.1-FlashSWE-AaaV85.1113.5916.18
DeepSeek-V4.1-Flashw/o guidance83.9814.1617.88

Backbone dependence. SWE-AaaV specifies how evidence is constructed and validated, and finding informative evidence still depends on the backbone model. Stronger models may recover useful verification strategies even when explicit guidance is removed. The ablation results show that the harness contributes under both backbones, while outperforming curated tests depends on verifier capability. Our results do not establish a general capability threshold for backbones.

Dear Reviewer U87V,

Thank you again for your time and thoughtful review. As the author discussion period ends on November 18 (AoE), we briefly summarize our responses to your concerns below.

  1. Human evaluation. We expanded the manual audit to include cases where both verifiers agreed. Our weighted estimates place SWE-AaaV above the test verifier with DeepSeek and below it with Qwen. In the annotation process, pre-discussion agreement was 88.1%.

  2. Differential-testing guidance. We removed the explicit guidance while keeping the same framework. FPR rose for both Qwen and DeepSeek, suggesting that the guidance helps both models catch incorrect patches.

If you have a moment before the discussion closes, we would be grateful to hear whether these results address your concerns.

Thank you for the detailed responses. The authors have addressed some of my concerns. I remain positive on this paper and keep my current rating.

Thank you for your encouraging response! We are glad that our responses help address your concerns, and we appreciate the time and effort during this busy period.

Reviewer_tyd6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SWE-AaaV, an agentic verifier that generates paired tests for reference and candidate implementations, with a deterministic harness validating the resulting evidence. The paper evaluates verification accuracy and its use as an RL reward, reporting improved offline AUC with stronger verifier backbones and a best observed SWE-bench Verified resolved rate of 49.6%, compared with 47.8% using unit-test rewards.

Critical strengths

The separation between test generation and evidence validation is a useful design. Reference validity, base consistency, and fail-to-pass checks constrain the generated tests before they determine rewards. Table 2 supports the importance of these checks: removing mandatory validation reduces Qwen3.6-35B-A3B’s AUC from 78.56 to 70.67, even though the agent can still execute code.

The paper goes beyond measuring agreement with existing tests. With DeepSeek-V4.1-Flash, manual adjudication supports SWE-AaaV in 17/20 cases where it rejects a test-passing patch and 14/20 cases where it accepts a test-failing patch (Figure 3). These results provide useful evidence that executable verification can identify mistakes missed by the original tests.

Critical weaknesses

The reward reliability is less convincing for the verifier actually used in RL. For Qwen3.6-35B-A3B, only 3 of the 20 inspected patches accepted by SWE-AaaV but rejected by the unit tests are judged correct; the others contain unresolved behavior or regressions (Figure 3 and Section 7.1). This is a selected disagreement subset rather than an overall error rate, but it directly concerns the positive rewards introduced by replacing the original tests. The stronger DeepSeek results do not resolve this concern for the Qwen verifier used during training.

The reward assignment at the turn limit needs clarification. Section 4.3 requires evidence to pass harness validation, but Section 6 states that candidate execution is prioritized for reward assignment when the round limit is reached. It is unclear whether this fallback still requires reference validity, base consistency, and resolved evidence, and Appendix A only describes the normal finish path. This matters because the claimed reliability comes from these checks, rather than candidate execution alone.

The downstream RL result is promising, but the evidence for a reliable improvement is limited. Figure 4 compares best observed checkpoints with a 1.8-point difference, without independent training repetitions or uncertainty estimates; the four repetitions in Appendix B concern offline verifier evaluation. The practical cost is also difficult to assess given roughly 50 verifier turns per trajectory and a separate 32-GPU verifier pool. Reporting total GPU-hours alongside the 15% wall-clock overhead would help establish whether the gain justifies replacing unit-test rewards.

Questions for authors

For the Qwen verifier used in RL, can the authors manually evaluate a sample of positively rewarded training rollouts, particularly those rejected by the original tests? How often do these disagreements occur during training, and how often is the verifier correct?

What reward is assigned when the verifier reaches the turn limit? Do all positive rewards still require the full harness validation, and how frequently is this fallback used during RL?

Can the authors report independent RL repetitions under a fixed checkpoint-selection rule, together with total GPU-hours for both reward settings?

2 comments

Author Response Overview: Thank you for recognizing our evidence-validation design, ablations, and manual analysis. Below, we clarify training-time reliability, turn-limit handling, and the robustness and cost of the RL gains.

W1 & C1: Reliability of the Qwen Verifier During RL

TL;DR: The verifiers disagreed on 22.5% of the sampled training rollouts, and Qwen was judged correct in 58.3% of the manually audited disagreements. Among these, Qwen correctly accepted 16 out of 30 patches rejected by the test verifiers, including 7 out of 10 patches within the hardest interval.

SWE-bench Verified uses human-screened tests by OpenAI, whereas the SWE-rebench training tasks are automatically collected. This quality gap matters: SWE-AaaV with Qwen3.6-35B-A3B can improve training rewards even though the curated benchmark tests outperform it. To avoid additional test-execution overhead affecting the efficiency measurements, we did not run Test Verifier during SWE-AaaV RL. We instead conducted a post-hoc audit across three curriculum intervals of increasing difficulty. For each interval, we sampled 500 rollouts and ran Test Verifier on their candidate patches offline. We then sampled 10 cases from each disagreement direction within that interval for manual adjudication.

Table R1. Training disagreement rates and SWE-AaaV correctness in audited disagreements. Verdict order: SWE-AaaV / Test Verifier.

Training stepsDisagreement rateFail/PassPass/FailDisagreement correctness
[0, 250)16.4%60.0% (6/10)40.0% (4/10)50%
[250, 500)21.8%60.0% (6/10)50.0% (5/10)55%
[500, 700]29.4%70.0% (7/10)70.0% (7/10)70%
Average22.5%63.3%53.3%58.3%

Results: Disagreements become more frequent in harder curriculum intervals. In the final interval, manual adjudication favors SWE-AaaV in 7/10 cases in each direction. A plausible explanation is that complex issues make correct and comprehensive predefined tests harder to construct, while candidate-conditioned verification can target the particular patch.

W2 & C2: Reward Assignment at the Turn Limit

TL;DR: Rewards at the turn limit still require the full harness validation. 4.03% of rollouts are excluded from RL updates due to invalid verification.

Reaching the turn limit does not bypass validation. Existing scripts still undergo the harness checks in Equation (8), allowing a completed test to be used even if the agent has not called the finish action. Invalid verification counts as incorrect in offline evaluation. During RL, the corresponding rollout is excluded from its group, and we do not guarantee a fixed effective group size. The invalid rates are 5.62% for Qwen3.6-35B-A3B and 1.62% for DeepSeek-V4.1-Flash in offline evaluation, and 4.03% during RL.

W3 & C3: Robustness and Computational Cost of RL Gains

TL;DR: The gain persists at 1.6–2.4 pp across two independent RL experiments, with 15% more time per step and 68% more total allocated GPU-hours.

Both offline evaluation and RL checkpoint evaluation average four samples. We conduct one additional independent training run per reward setting: Test Verifier reaches 48.0%, and SWE-AaaV 50.2%, compared with 47.8% and 49.6% originally. Across the two results per method, the four cross-comparisons give an improvement of 1.6–2.4 pp.

Figure 4(b) reports 7.2 versus 8.3 minutes per RL step (+15%). Including validation every 25 steps, total durations are 103 versus 116 hours. Training and rollout inference each use 32 GPUs, and SWE-AaaV adds 32 GPUs for verification. Total allocated GPU-hours (including idle time) are 6,613 versus 11,129 (1.68×). Asynchronous integration limits the wall-clock overhead, while the verifier's additional inference still incurs compute cost.

Dear Reviewer tyd6,

Thank you again for your time and thoughtful review. As the author discussion period ends on November 18 (AoE), we briefly summarize our responses to your concerns below.

  1. Reward reliability. We manually adjudicated 60 RL rollouts on which Qwen and the test verifier disagreed. Of the 30 patches that Qwen accepted and the test verifier rejected, 16 were judged correct, including 7 of 10 in the hardest interval.

  2. Turn-limit handling. Even at the turn limit, a positive reward requires full validation. During training, 4.03% of rollouts were excluded from RL updates due to invalid verification.

  3. RL gains and cost. An independent run reproduced the gain over the test verifier (+2.2 pp). SWE-AaaV used 68% more allocated GPU-hours and took 15% longer per training step.

Before the discussion closes, we would be grateful to hear whether these results address your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion

We thank the Area Chairs and reviewers for their constructive feedback. We believe that all core concerns have been resolved through supplementary experiments and the revised submission. Here is a summary of the discussion phase:

Strengths of the paper: Reviewers highlight the methodological novelty, especially the separate evidence validation (tyd6, U87V); the breadth of the experiments (tyd6, AdKM); and the practical integration into RL training (U87V, AdKM).

Core Concerns Addressed:

  • Correctness beyond unit-test labels (U87V). We expanded the human audit to include cases where both verifiers agreed. The weighted estimates place DeepSeek above the benchmark tests and Qwen below them (Table R2). We also documented the annotation guidelines and reported 88.1% agreement.

  • Reliability of training rewards (tyd6, AdKM). Human review confirmed 16 of 30 Qwen's positive rewards rejected by the original tests, including 7 of 10 in the hardest interval (Table R1). We clarified that rewards require full validation even at the turn limit.

  • Methodological contribution (AdKM). With identical reference and execution access, removing mandatory validation lowered AUC on both models. A new ablation showed that removing differential testing guidance increased false positives (Tables 2 and R3).

  • RL gains and cost (tyd6, AdKM). An independent run reproduced the gain over the test verifier (+2.2 pp, compared with +1.8 originally). Training took 15% longer per step and used 68% more allocated GPU hours.

We believe that SWE-AaaV and the experimental findings provide significant insights into evaluation-time scaling for coding agent rewards. We sincerely hope that the Area Chairs and reviewers will give this further consideration. Thank you again for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.