acceptodds
Under review as a conference paper at ICLR 2027

Patches Hide Tests to Write: Deriving Trainable Test-Generation Tasks from Existing SWE Environments

12 views

Abstract

Reinforcement learning for repository-level coding agents is limited by the supply of executable tasks, and the work that enlarges it either curates more real issues or manufactures new defects. An issue-resolution instance carries four artifacts: the problem description, the repository snapshot in which the issue is present, the reference patch that fixes it, and the reference tests that the fix turns from failing to passing. Issue resolution withholds the patch and judges with the tests. Exchange the two roles and the same instance defines a second task, writing a test that reproduces the reported behaviour, inside the environment the instance already ships and at no annotation cost. Fail-to-pass status alone is satisfied by tests that exercise no behaviour, such as one that checks whether a line of the buggy source is absent. We therefore require a submitted test to fail both on the buggy snapshot and on snapshots in which the reference patch has been damaged, and to pass both on the repaired snapshot and on versions of it whose source text has been rewritten without changing behaviour. Applied to three collections of instances that differ in where their defects and tests come from, the conversion produces 30,605 fail-to-pass test-generation tasks, of which 23,520 pass an audit of their verifiers. Forty steps of joint reinforcement learning on the two tasks, which draw about 480 tasks in total, raise Qwen3.5-9B from 48.2% to 55.0% on SWE-bench Verified and from 38.5% to 44.5% on TDD-Bench Verified, ahead of training on either task alone. We release SWE-Flip, the conversion pipeline together with the tasks it produced.

Primary area:
datasets and benchmarks
Keywords:
code agents, test generation, training environments, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ubRtOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes using test generation as an additional training task alongside issue resolution. The paper proposes a method to convert existing tasks into test-generation tasks and shows that training for both test generation and task resolution increases the overall performance.

Critical strengths
  1. The joint training on issue resolution and test generation performs better.
  2. The dataset of generated problems may be useful in the future to others seeking to train SWE agents.
Critical weaknesses
  1. The verification pipeline for the test generation task is under-explained. See the questions section.
  2. Table 2 shows that training on issue resolution on SWE-Bench-Verified hardly increases performance, and for SWE-Bench Pro and TDD-Bench Verified, it decreases. Why was it decided to train for only 40 steps? It seems that something may have gone wrong during training if there is such a small increase in performance.
Questions for authors
  1. If the issue becomes the prompt, how does the agent know how to generate tests for the entire codebase and not only tests relevant to that issue?
  2. How do we know that the tests being generated are semantically interesting and relevant? In a naive case, couldn't the agent generate tasks that simply check for syntax differences between the buggy snapshot and the correct one?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ubRt

We thank the reviewer for the review. We hope the clarifications below resolve the reviewer's concerns.

Q1. How does the agent know which tests to write?

The agent is asked to test only the reported issue, and only a test of that issue can be rewarded. The prompt is the issue together with one instruction: write a single test file, at a fixed path, that fails on the current repository because of the reported problem and passes once it is fixed. The verifier then judges the test only on snapshots that concern this issue: the buggy and repaired snapshots, damaged versions of the patch that fixed this issue, and rewrites of the files that patch changed. A test of unrelated code passes on the buggy snapshot and receives no reward. The prompt template is INSTRUCTION_TMPL in filtering/admission/f2p_env.py of the supplementary material, and each of the 20 sample tasks there includes its filled-in prompt as instruction.md.

A task is kept only when the issue and the patch concern the same behaviour (Section 3.2, Appendix B), so the issue describes the behaviour the verifier checks. An instance whose patch also contains changes the issue does not mention is excluded before any container is run, since a test written from the issue could not be expected to detect them.

Q2. Can a test that only checks syntax differences be rewarded?

No. Plain fail-to-pass would reward such a test, for example one that asserts a line of the buggy source is absent, because it fails before the fix and passes after it. Equation (4) adds two checks that such a test cannot pass.

First, the test must also fail when part of the fix is removed. We build damaged versions of the reference patch, each undoing or breaking one part of it, and keep a damaged version only if the project's own tests fail on it (N\mathcal{N} in Equation (4)). A test that still passes on a damaged version does not check the behaviour the fix changes, so it gets no reward. A rewarded test therefore checks the behaviour the issue reports.

Second, the test must still pass when the fixed code is rewritten without changing what it does (P\mathcal{P}). The rewrites add spaces between tokens, split string literals and insert pass statements. A test that looks for a piece of source text fails on them, because the text has changed while the behaviour has not. A test that hashes the syntax tree of the file also fails, because the inserted pass statements change the hash. Figure 2 shows both kinds of shortcut on one instance.

We also checked this in practice. We wrote 10 to 15 kinds of shortcut test and ran each on 100 tasks per source. The verifier refused 3,824 of the 3,833 runs and still rewarded the project's own tests on every task. During RL, no rewarded test was a shortcut: we read every rewarded test that resembled one (Table 7). Finally, GLM-5.3 wrote 23,315 tests that plain fail-to-pass would reward. Equation (4) rejects 3,230 of them (13.9%), almost all because they still pass when part of the fix is removed.

AuthorsOpenReview ↗

Response to Reviewer ubRt 2

Response to Reviewer ubRt (2/2)

W1. The verification pipeline is under-explained

The verifier gives the agent's test a reward of 1 only if every step below holds (Section 3.3, Appendix C).

  1. The agent's test runs on the buggy snapshot and must fail through a failing assertion, not through an import or collection error.
  2. It runs on the repaired snapshot and must pass.
  3. It runs on each damaged version in N\mathcal{N} and must fail, and on each behaviour-preserving rewrite in P\mathcal{P} and must pass.
  4. The trials run in a random order per task, with writable state cleared between trials. On trials the test cannot tell apart, the verifier runs the test twice and requires the same verdict both times.
  5. The test must execute at least one line the patch changed, as recorded by an execution tracer.

The supplementary material contains the verifier _gate1b_scaleswe.py, the code that generates the task files and 20 tasks drawn during training, each with its verifier.

W2. The resolution-only model and the length of training

The resolution-only run trained normally: its mean training reward rose from 0.34 in steps 1–10 to 0.45 in steps 11–20 and stayed there.

Its reward depends only on the final patch, so it learned to submit sooner and check its own work less, and its scores fall where self-checking matters. On SWE-bench Verified it ends 41.4% of instances without running any test, against 25.1% for the base model (Table 5), and its score is unchanged (48.2% to 48.4%). On SWE-bench Pro, where its score drops most, it submits without running any test on 36% of instances, against 21%. On TDD-Bench Verified, 59 of the 70 instances it loses to the base model have a test that still fails after the reference fix, so the test does not reproduce the issue. Joint training also rewards writing tests, and the joint model tests more and scores higher on all three benchmarks.

We trained all three models for 40 steps with the same hyperparameters, so they are compared at the same training budget. We did not train the resolution-only model longer. We did not train the resolution-only model longer. Its reward would still score only the patch, so more steps give it no reason to test more, and we expect the same trend.

Reviewer_ak7NOpenReview ↗
Rating
3(−1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the shortage of trainable tasks for RL on repository-level coding agents. An issue-resolution instance has four parts: a problem description, a buggy snapshot, a reference patch, and reference tests. The authors swap the roles of the patch and the tests. The agent writes a fail-to-pass (F2P) test from the issue, and the reference patch grades it, so no new annotation is needed. To stop shortcut tests that check no behaviour (for example, a test that checks whether a source line contains some string), the authors strengthen the verifier (Eq. 4): the test must fail on the buggy snapshot and on several damaged versions of the reference patch, and must pass on the fixed snapshot and on several semantically equivalent source rewrites. The authors convert 30,605 tasks from Scale-SWE, SWE-smith-py, and R2E-Gym, of which 23,520 pass admission. With 40 steps of joint GRPO, Qwen3.5-9B improves from 48.2% to 55.0% on SWE-bench Verified and from 38.5% to 44.5% on TDD-Bench Verified, and both results are higher than training on either task alone.

Critical strengths
  1. The problem is defined carefully. Eq. 1 decides out(t, S) from the tests that actually ran, without relying on the exit status, and it states how skips and import errors are handled. These details matter for preventing reward hacking.
  2. The verifier design is simple and practical. Using the reference tests to decide which damaged patches to keep (Section 3.3) means the N family does not need hand-crafted mutants. This is a clever choice.
  3. The analysis of the low R2E-Gym admission rate is insightful. For 2,169 tasks, the reference tests are insensitive to local damage. This shows that the value of a data source depends on the strength of its reference tests, and task count alone does not determine it.
  4. Figure 2 gives a concrete example of two kinds of shortcut and shows which added snapshot rejects each one. It is easy to follow.
  5. The paper handles information leakage with care: it removes git history, disables the network, mounts graded artifacts only at grading time, fixes the submission path, cleans writable state between trials, and randomizes trial order (Section 3.3, Appendix A).
  6. The counts are transparent and consistent. I checked the numbers in Table 1, Section 4, Appendix B, and Tables 4, 6, and 7, and they match.
  7. The authors are honest about limitations. For example, Appendix D states that the 3/89 difference on Terminal-Bench is not significant, and Appendix C states that samples from zero-variance groups were not kept.
Critical weaknesses

Post-rebuttal update. I thank the authors for the detailed response, the repeated evaluations with McNemar tests, the shortcut probes on all three sources and the repository-level overlap check. However, my main concern remains. The paper's central technical contribution is the strengthened verifier (Eq. 4), yet there is still no RL run that uses the plain fail-to-pass reward under the same tasks and hyperparameters. The new analysis shows that Eq. 4 rejects 13.9% of GLM-5.3 tests, which shows it is stricter, but not that it improves training. The evidence therefore does not isolate how much of the joint-training gain comes from the verifier. In addition, the response clarifies that the set described as "hand-labelled" for the single-behaviour filter was in fact labelled by a model; this needs to be corrected. I am lowering my score from 4 to 3.


Key reasons for my recommendation: (1) The main RL result rests on a single run of 40 steps with no variance estimate, and the single-task baseline behaves abnormally. (2) The strengthened verifier is the core technical contribution, but there is no RL ablation showing that it helps training and no measurement of how often it wrongly rejects honest tests.

1. The core RL result has no variance estimate, and the single-task baseline may be weak.

  • What the paper does. Each of the four models in Table 2 is trained once and evaluated once. The paper does not state the number of seeds, the evaluation temperature, or the number of repeats. SWE-bench Verified has 500 tasks, so near 50% the standard error of one evaluation is about 2.2 points. TDD-Bench Verified has 449 tasks, so the standard error is about 2.3 points. On SWE-bench Pro, joint training gains only 1.0 point over the base model (26.0 vs 25.0). Test-generation-only training gains only 1.8 points on TDD-Bench. These gaps are within noise.
  • The baseline looks abnormal. The resolution-only model trains for 40 steps with a learning rate of only 1e-6, yet SWE-bench Pro drops from 25.0 to 20.8 (-4.2) and TDD-Bench drops from 38.5 to 35.0, while SWE-bench Verified is nearly unchanged (48.4). Prior work usually reports gains from resolution RL with execution reward. This suggests a problem in the resolution-only setup, such as unstable training, an unsuitable patch-size filter (1,287–3,982 bytes), or hyperparameters tuned for the joint run. The claim that joint training beats either single task depends on this baseline.
  • Why it matters. This is the main empirical claim of the paper. The abstract and conclusion both repeat these numbers.
  • What would fix it. (a) Run at least 3 seeds per setting and report mean and standard deviation. (b) Run paired tests (McNemar or paired bootstrap) between models in Table 2. (c) Show reward curves and evaluation curves over training steps for all three trained models. (d) Train resolution-only longer or tune its hyperparameters, to show the drop is not a tuning artifact. (e) State the evaluation decoding settings and number of repeats.

2. No ablation isolates the effect of the strengthened verifier on training.

  • What the paper does. The method rests on Eq. 4 being stricter than plain F2P (Eq. 2). Section 5 has no run trained with plain F2P reward. Table 7 (Appendix C) shows that shortcut-like submissions are only about 3.6% of training submissions ((23+12+29) / 1,790). After manual reading, the authors judge that 0 of the 27 rewarded submissions lack behavioural content. This suggests that plain F2P reward may be enough at 40 steps with a 9B model.
  • Why it matters. If removing the N and P families gives similar results, the main technical contribution does not affect the reported results. If the verifier does help, an experiment should show it. The verifier also costs m+n+2 runs plus repeat runs per submission, and the paper does not report this cost.
  • What would fix it. (a) Add an RL control with the same tasks and hyperparameters, with reward replaced by plain F2P (Eq. 2). Compare all Table 2 metrics and the shortcut rate. (b) Remove the N family and the P family separately to show the effect of each. (c) Report the average m and n per task and the verifier wall-clock time.

3. The false-negative rate of the verifier (wrongly rejecting honest tests) is not measured.

  • What the paper does. Eq. 4 requires the test to fail on all damaged patches. The reference tests decide which damaged patches are kept (Section 3.3). The "honest probe" in Appendix C uses the reference tests themselves. Since the N family is filtered by those same tests, this probe passes by construction and says nothing about other honest tests.
  • Why it matters. An honest test that reproduces only the main symptom in the issue may not catch the reversal of another hunk in the reference patch (Appendix B allows up to 4 hunks). Such a test receives 0 reward. This pushes the model to copy the coverage of the reference tests, which may differ from the behaviour described in the issue. Section 3.2 itself admits that a mismatch between the patch and the description makes the reward "misleading". The single-behaviour filter relies on heuristics such as hunk count and the number of sections in the description (Appendix B). The paper mentions a hand-labelled set but does not give its size or labelling procedure.
  • What would fix it. (a) Take tests from GLM-5.3 or another strong model that pass plain F2P, and report the fraction that Eq. 4 rejects. Manually check a sample of the rejected tests to see whether they truly lack behavioural content. (b) Report the size of the hand-labelled set, the number of annotators, and their agreement. (c) Discuss requiring failure on only a subset of damaged patches, or giving partial credit per hunk.

4. The shortcut probe is small and narrow.

  • What the paper does. Table 6 has only 86 probes, 13–16 per category, all from Scale-SWE, and all fixed shortcuts written by the authors. SWE-smith-py and R2E-Gym are not probed.
  • Why it matters. A 100% rejection rate shows only that the verifier blocks the attacks its designers anticipated. It says little about adaptive attackers or new shortcuts that may appear in long RL runs. Page 2 says admission audits shortcuts "on a sample". 86 probes against 23,520 tasks is too few to support the claim that the verifier is audited.
  • What would fix it. Extend the probe to all three sources, with at least several hundred tasks each. Add LLM-generated adversarial shortcuts, for example by prompting a model to "write a test that checks no behaviour but still gets reward". Report the fraction that passes the verifier.

5. Contamination is checked only at the instance level.

  • What the paper does. Section 5.1 states only that the 480 RL training tasks "share no instance" with the evaluation sets. Scale-SWE covers 888 repositories and likely overlaps with the 12 repositories in SWE-bench Verified (such as django and sympy). Similar issues from the same repository may also appear. The SFT part (Table 3) has a repository-level exclusion analysis; the RL part does not.
  • Why it matters. Training on the same repositories makes the model familiar with their code structure and may inflate SWE-bench Verified and TDD-Bench Verified scores. The effect may be larger for the joint model, since its test-generation task has the same form as TDD-Bench.
  • What would fix it. List the repositories shared between RL training tasks and the evaluation sets, and report Table 2 with those repositories removed, as in Table 3. Explain how the "4 shared instances" in Table 3 were identified (instance id, commit, or issue-text similarity).

6. The behaviour analysis in Table 5 does not support the causal explanation.

  • What the paper does. Section 5.2 explains the resolution-only drop by reduced testing behaviour. But the test-generation-only model is also worse on all four testing metrics in Table 5 (commands per turn 0.54, tests written under tests/ 5.8%, median test runs 7, no test runs 27.9%), and it still gains 2.2 points over the base model on SWE-bench Verified. This contradicts the explanation that more testing leads to more resolved tasks. Also, the evaluation turn limit is 250 while the training limit is 80. The base model hits the 250-turn limit on 28.7% of tasks and the joint model on 14.8%. Part of the gain may come from fewer truncations.
  • Coverage reporting. Section 5.2 says the joint model's tests "execute the lines the reference patch changed about equally well", but the lower quartiles reported right after are 100% vs 50%, which differ clearly. The authors count missing coverage reports as 0, while the official TDD-Bench implementation counts them as full coverage. The paper should state which convention the "TDD score" in Table 2 uses.
  • What would fix it. Report resolved rate split by whether the run hit the turn limit, or re-evaluate with an 80-turn limit. Remove or rewrite the causal claims. Use one coverage convention throughout.

7. Task selection and sampling details are unclear, which affects reproducibility and fairness.

  • The training pool "drop[s] the tasks that earlier runs solved in no trajectory at all". Which runs are these, and which model produced them? If early joint runs were used, the three models do not see the same training distribution.
  • The pools differ in size (1,329 resolution tasks and 973 test-generation tasks), which the authors say "equalises what reaches the update". Pool size affects the ratio only under uniform sampling from the union. Please state the per-step sampling ratio of the two task types in joint training, and the actual ratio of groups that reach the update.
  • The reward is not strictly binary. Table 7 and Appendix D mention a "deduction for not following the submission protocol", but neither Eq. 4 nor Section 5 defines it. Please give the full reward definition.
  • Please confirm whether the resolution training tasks come from the 4,677 admitted Scale-SWE resolution tasks in Section 4.2.
  • Missing details: max response length, evaluation temperature, tool list per trajectory, the prompt template for test-generation tasks, and an anonymous link to code and data.

8. No empirical comparison with directly related work, and the SFT experiment lacks controls.

  • SWE-Tester (Soni et al., 2026) already provides 41K reproduction instances and trains open models. SWE-Playground trains jointly on resolution and reproduction. The paper cites both in related work but does not compare numbers on TDD-Bench Verified or compare data quality. Since converting issue-resolution instances into test generation has been done before (SWT-Bench and TDD-Bench Verified for evaluation; SWE-Playground and SWE-Tester for training), the novelty lies mainly in the strengthened verifier, which makes Weaknesses 2 and 3 more important.
  • The SFT gain in Section 5.3 (26.1% to 73.3%) comes mostly from distilling a strong teacher (GLM-5.3, 87.8%). To show the value of using the verifier to filter trajectories, two controls are needed at the same token budget: the same GLM-5.3 trajectories filtered by plain F2P (or not filtered), and SWE-Tester data.
  • Table 3 uses the claude-code scaffold (base 26.1%), and Table 2 uses mini-swe-agent (base 38.5%). The two tables cannot be compared directly. I suggest reporting the RL and SFT models under the same scaffold.

Minor issues

  1. Damaged patches are mutants in the sense of mutation testing, and "the test must kill the mutant" corresponds to mutation score. Please cite classic mutation testing work (e.g., the survey by Jia & Harman) and recent work that uses mutation score to evaluate LLM test generation. The semantically equivalent rewrites relate to the literature on semantic-preserving transformations and metamorphic testing.
  2. The definition of rtest in Eq. 4 does not include the condition at the end of Section 3.3 that the same test is run twice with consistent results. Please add it to the definition or state that it is an extra condition.
  3. The patch-size range 1,287–3,982 bytes is very specific. Please explain how it was chosen. 1,287 is also the number of Scale-SWE tasks that fail admission (Appendix B); please check that this is not a typo.
  4. Section 4.2 says 4,677 of the 19,473 Scale-SWE instances admit a resolution task. This number is not in Table 1, and its relation to the RL training pool is not stated.
  5. In Table 5, "Tasks with a test written under tests/" counts tests in the tests/ directory. Is the fixed submission path for test-generation tasks also under tests/? If not, this row may undercount testing by the test-generation-only model.
  6. The Terminal-Bench result in Appendix D (18/89 vs 21/89) is not significant, as the authors note. Please state the claim that the conversion does not depend on tasks being repository issues more cautiously.
Questions for authors
  1. How many seeds were used for each model in Table 2? How many times was each benchmark evaluated, and with what decoding settings?
  2. How does a joint model trained with plain F2P reward (Eq. 2) perform on Table 2?
  3. How many strong-model tests pass plain F2P but are rejected by Eq. 4? How many of these are judged honest by manual inspection?
  4. How do you explain the 4.2-point drop of the resolution-only model on SWE-bench Pro? Did you try other learning rates or longer training?
  5. Which "earlier runs" were used to filter the training pool, and which model produced them?
  6. In joint training, what is the sampling ratio of the two task types, and what ratio reaches the update?
  7. Which repositories do the RL training tasks share with SWE-bench Verified? How does Table 2 change when those repositories are removed?
  8. What is the exact rule for the "deduction for not following the submission protocol", and how often did it trigger during training?
  9. How large is the hand-labelled set for the single-behaviour filter, who labelled it, and what was the agreement?
  10. On average, how many damaged patches and rewrites does each task have, and how long does one verifier call take?
  11. How do the converted SWE-smith-py and R2E-Gym tasks perform when used for RL? Does adding them change the conclusion about joint training?

Suggestions (not affecting my score):

  • Split the "Repositories" column in Table 1 ("953 / 888") into separate converted and admitted columns.
  • Use the same terms for the hardening checks in Figure 1 and the N/P families in the text.
  • Please check the Rajan (2026) figure on page 2 ("an audit of 49 tasks ... finds that 28.5%"). 14/49 = 28.57%, which rounds to 28.6%, and no integer count out of 49 gives 28.5%.
6 comments
AuthorsOpenReview ↗

Response to Reviewer ak7N (1/5)

We thank the reviewer for the detailed review and for the positive assessment of our work. Below we answer each weakness in turn, and we hope the answers resolve the reviewer's concerns.

W1. Single run, no variance estimate, and the resolution-only baseline

(a) Seeds and variance. Each model in Table 2 comes from one training run. We did not train with several seeds. Instead, we evaluated every checkpoint two more times under the same settings, so the figures below measure the variance of evaluation rather than of training. Mean ± standard deviation over the three evaluations:

BenchmarkBaseIssue resolution onlyTest generation onlyJoint
SWE-bench Verified48.2 ± 0.948.5 ± 1.151.0 ± 0.654.1 ± 0.9
SWE-bench Pro25.0 ± 0.320.5 ± 0.424.0 ± 0.826.8 ± 1.0
TDD-Bench Verified38.2 ± 0.835.0 ± 0.239.7 ± 0.643.7 ± 1.2

The joint model is the best of the four in every evaluation on every benchmark.

(b) Paired tests. We ran the exact McNemar test on all three evaluations together, pairing the joint model with each other model on the same instance in the same evaluation. The joint model is significantly better (p<0.05p < 0.05) in 8 of the 9 comparisons. The pp-values are:

Joint model vs.SWE-bench VerifiedSWE-bench ProTDD-Bench Verified
Base<10−5< 10^{-5}0.055<10−3< 10^{-3}
Issue resolution only<10−5< 10^{-5}<10−11< 10^{-11}<10−8< 10^{-8}
Test generation only0.0060.0020.007

The exception is the base model on SWE-bench Pro (p=0.055p = 0.055).

(c) Training curves. Mean training reward per 10 steps:

RunSteps 1–1011–2021–3031–40
Joint0.330.410.380.51
Issue resolution only0.340.450.440.44
Test generation only0.350.490.460.52

The training reward rises in all three runs. The resolution-only run rises from 0.34 to 0.45 within 20 steps and then stays flat.

(d) The resolution-only baseline. Its decline holds in all three evaluations: it scores below the base model on SWE-bench Pro (20.5 against 25.0 on average) and on TDD-Bench Verified (35.0 against 38.2). Its reward depends only on the patch, so it learned to submit sooner and check its own work less, and its scores fall where self-checking matters. On SWE-bench Verified it ends 41.4% of instances without running any test, against 25.1% for the base model (Table 5), and its score is unchanged. On SWE-bench Pro, where its score drops most, it submits without running any test on 36% of instances, against 21%. On TDD-Bench Verified, 59 of the 70 instances it loses to the base model have a test that still fails after the reference fix, so the test does not reproduce the issue. The patch-size filter the reviewer mentions is explained in W7.

We did not try other learning rates or a longer run. The three runs share every hyperparameter (learning rate 10−610^{-6}, group size, batch size and 40 steps) and differ only in the tasks they draw, and none was tuned for the joint run. We expect the same trend with a different learning rate or a longer run, because the reward would still score only the patch and give no reason to test.

(e) Decoding settings and number of evaluations. Now we evaluated each checkpoint three times, each evaluation makes one attempt per instance and samples Qwen3.5-9B with its default sampling parameters (temperature 1.0, top-p 0.95, top-k 20). An attempt is limited at 250 steps, a context of 262144 tokens and 10800 seconds for timeout. We will state this in Section 5.1. Table 2 reports the first of the three evaluations in (a).

AuthorsOpenReview ↗

Response to Reviewer ak7N (2/5)

W2. No ablation of the strengthened verifier

(a) RL with the plain fail-to-pass reward. We did not run RL with the plain fail-to-pass reward. We instead measured how Equation (4) changes the reward on tests written by an independent model. We ran GLM-5.3 on the converted test-generation tasks of all three sources and kept its tests that pass plain fail-to-pass (Equation 2), which plain fail-to-pass would reward. The table counts these tests per source and how many of them Equation (4) rejects:

SourceGLM-5.3 tests passing plain F2PRejected by Eq. (4)
R2E-Gym6,850779 (11.4%)
Scale-SWE6,672455 (6.8%)
SWE-smith-py9,7931,996 (20.4%)
All23,3153,230 (13.9%)

Equation (4) rejects 3,230 of the 23,315 tests (13.9%). The rejection rate rises with the number of damaged versions per task (W2(c)): 6.8% on Scale-SWE (mean mm = 1.6), 11.4% on R2E-Gym (2.4) and 20.4% on SWE-smith-py (10.0). Overall, under a plain fail-to-pass reward, about one rewarded test in seven would be one that cannot detect a missing part of the fix (W3).

(b) Effect of each family. N\mathcal{N} rejects 3,187 of the 3,230 tests (98.7%): each of them still passes on a damaged version of the reference patch, so these tests are removed by N\mathcal{N}. P\mathcal{P} targets a different shortcut, tests that depend on the source text, and Table 6 and the probes of W4 test it directly. We did not train with either family removed.

(c) Size of N\mathcal{N} and P\mathcal{P}, and cost. Over the admitted tasks of each source:

SourceDamaged versions mm per taskRewrites nn per taskVerifier time per submission (median / 90th percentile)
Scale-SWEmean 1.6, range 1–4226 s / 64 s
R2E-Gymmean 2.4, range 1–62142 s / 914 s
SWE-smith-pymean 10.0, median 8, range 1–30185 s / 425 s

On Scale-SWE, which we use for RL, mm is the number of hunks of the reference patch whose reversal the reference tests detect. Of the Scale-SWE tasks, 58%, 30%, 10% and 3% have m=1,…,4m = 1, \dots, 4. On the other two sources the damaged versions come from mutation operators applied to the changed lines. Times are measured on our evaluation cluster with the reference tests as the submission. On Scale-SWE a verifier call takes a median of 26 s.

W3. False negatives of the verifier

(a) Tests from a strong model. The tests that Equation (4) rejects check only part of the fix, so rejecting them is reasonable. The probe in Appendix C, which submits the instance's own reference tests, passes by construction, as the reviewer notes. We therefore checked the tests that Equation (4) rejects among those written by GLM-5.3.

Of the GLM-5.3 tests that Equation (4) rejects in W2, 98.7% still pass when one part of the reference patch is removed, while the project's own tests fail.

For example, an issue reports that yamllint's truthy rule swaps the YAML 1.1 and 1.2 value sets. The submitted tests only check that valid values raise no problem. None checks that a wrong value is flagged, so the tests still pass when the rule skips the check entirely.

(b) Labelled set for the single-behaviour filter. The set has 68 Scale-SWE instances, each labelled once by a model as a single behaviour (39), several unrelated changes (21) or not testable (8). The thresholds of Appendix B are the loosest at which no instance of the second kind is kept. "Hand-labelled" in the submission is inaccurate. We will correct it and report the set and its labels in the appendix.

(c) Partial credit. We will discuss partial credit per damaged version in Section 3.3.

AuthorsOpenReview ↗

Response to Reviewer ak7N (3/5)

W4. Scope of the shortcut probes

We did not prompt a model to write shortcuts, and we did not extend the probes to several hundred tasks per source.

Probes on all three sources. We ran the shortcut probes on 100 admitted tasks of each source, one run per task and shortcut, against the verifier released with each task:

SourceShortcut kindsRunsRefusedReference tests rewarded
Scale-SWE141,3951,387100 / 100
R2E-Gym151,4411,441100 / 100
SWE-smith-py10997996100 / 100

Across the three sources the verifier refuses 3,824 of 3,833 shortcut runs and rewards the reference tests on every task, so it blocks the shortcuts without refusing correct tests.

The shortcuts cover the families of Table 6: asserting nothing, checking the buggy or the fixed source text, hashing the syntax tree of the modified file (with or without the inserted no-op statements removed), and reading state left by an earlier trial. They add shortcuts that count trials, change their answer on the second run on the same snapshot, read the log the verifier writes, or write a fake record of which lines executed.

Shortcuts arising in training. No rewarded submission in training is a shortcut. Table 7 (Appendix C) classifies the 1,790 test-generation submissions collected during the two RL runs, and 1,728 (96.5%) match none of the shortcut patterns. We read each of the 26 rewarded submissions that match one (27 entries in Table 7, since one matches two patterns), and each tests the behaviour, for example a test of a code formatter that asserts the formatter's output text. GLM-5.3 also wrote shortcuts without being asked: 17 of its tests that pass plain fail-to-pass read the environment or disable the execution tracer, and the verifier refuses all 17.

W5. Contamination at the repository level

The joint and test-generation-only runs share no repository with any evaluation set, so removing shared repositories leaves their columns of Table 2 unchanged. We compared the tasks each run drew in its 40 steps with the three evaluation sets, by owner/repo and by instance:

RunTasks drawnRepositories shared with SWE-bench Verified / TDD-Bench Verified / SWE-bench ProInstances shared
Joint4800 / 0 / 00
Test generation only4800 / 0 / 00
Issue resolution only4921 (pallets/flask) / 1 / 00

The only shared repository is pallets/flask, drawn by the resolution-only run (pallets_flask_pr3684). The benchmark instance there is pallets__flask-5014, a different pull request.

The four instances of Table 3 that the fine-tuning data shares with TDD-Bench Verified were identified at the instance level, by matching repository and pull request against the 449 benchmark instances.

W6. The behaviour analysis of Table 5

Turn limit. On the SWE-bench Verified evaluation reported in Table 2, we split the trajectories by whether they reached the 250-turn limit:

ModelReaches the limitResolved, trajectories below the limit
Base28.7%61.5%
Issue resolution only13.4%54.0%
Test generation only18.2%59.1%
Joint14.8%62.7%

The resolution-only model reaches the limit least often of all four models, yet its resolved rate does not rise, so fewer truncations do not by themselves raise the score.

Test-generation-only model. We will phrase Section 5.2 as an association between the behaviour counts and the resolved rates.

Coverage. The TDD score in Table 2 uses the same convention as the coverage analysis of Section 5.2 and counts an instance without a coverage report as covering nothing. The convention does not change any entry of Table 2. On every instance without a report (4 to 15 per model and evaluation), the test does not reach fail-to-pass, so it scores 0 under either convention. We will state this next to Table 2.

Tests that reach fail-to-pass and tests that do not execute the changed lines about equally well overall: the median share executed is 100% both for tests that reach fail-to-pass and for those that do not. They differ only in the lower quartile, 100% against 50%.

Reviewer_cxMtOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper attempts to resolve the issue of “reward hacking” in the SWE-bench evaluation by adopting a new perspective. The authors represent an issue-resolution instance as four sets: the problem statement, the buggy snapshot, the reference patch, and the reference tests. Generally, the standard strategy involves hiding the reference patch and using the reference tests for scoring. In contrast, the authors propose a new approach in which the reference tests are hidden and scoring is performed using the reference patch. As a result, they aim to avoid the high cost of additional manual annotation and resolve existing supervision issues by reframing the learning task. The authors demonstrate the effectiveness of their proposed method across tens of thousands of tasks.

Critical strengths
  • I find the core idea of this paper very interesting. (I am aware that there is a leap in logic here.) The framework in which issue resolution and test generation work together or in opposition to audit a certain type of “reward hack” in the SWE environment feels, in a way, like the paradigm shift I experienced when I first read a paper on GANs. While this idea is straightforward, it appears to be a novel approach based on my research.

  • The topic of this paper timely addresses an inherent problem in recent agent-based coding and has the potential to capture the interest of a wide and diverse readership. By clarifying the fundamental challenges of F2P and proposing to alter the very nature of reward allocation through behavior-preserving rewrites, the paper successfully appeals to readers as a valid and promising approach.

  • The empirical experiments are large-scale and constitute a thorough report that also identifies cases where the proposed method does not perform well. (With the exception of the points I would like to ask the authors about in the “Weaknesses” section) the experimental design and results are highly insightful. In particular, it is very interesting that the method of joint training these two tasks—issue resolution and test generation—clearly yields better results than training them separately.

Critical weaknesses
  • While the conceptual idea is very compelling, the implementation feels somewhat heuristic. I understand that the experiments demonstrate the verifier can significantly suppress obvious shortcuts, but it is difficult to discern from the paper to what extent the theoretical guarantees have been realized (or whether these guarantees are left as future work). In particular, I do not fully understand the mechanism described in Section 3.3, specifically, whether it truly refines behavior-preserving tests as intended by the authors’ proposed concept.

  • It does not appear that the method has completely broken free from its dependence on the quality of reference tests. It seems that the criteria for judging the success or failure of poor-quality patches ultimately rely on the reference tests. (This may be related to the section the authors label as “future work” at the end of Section 6.)

Questions for authors

I thoroughly enjoyed reading this paper, which shares a very elegant idea on a highly timely topic. Thank you. While I find the ideas in this paper conceptually very elegant, I am still struggling to fully grasp how the implementation—particularly the methodology in Section 3.3—actually realizes those ideas. I would greatly appreciate any clarification from the authors to help me understand.

My first question concerns how the “damaged versions” P_{1},\dots, P_{m} of the reference patch, mentioned around line 250, are generated. For example, do the authors explicitly mention anywhere how mm is set, or to what extent P1,…,PmP_{1}, \dots, P_{m} cover a certain hypothesis space? (I skimmed through the supplementary materials but was unable to fully understand this.) Do these originate from Qwen3.5-9B with GRPO?

My next question concerns whether the form of Equation (4) truly allows for the selection of only behavior-preserving tests. Based on the authors’ explanation, I understand that the standard F2P reward contains a shortcut, and that by conceptually introducing “damaged fixes” and behavior-preserving rewrites (as in the proposed method) it seems possible to suppress this obvious shortcut. On the other hand, it seems to me that the discrimination condition in Equation (4) merely applies the verifier to the generated variants (in some way? As mentioned in my previous question, perhaps I haven’t fully grasped this part) and does not include a mechanism to explicitly guide them toward a behavior-preserving direction. Is “behavior-preserving rewrites” an ideal concept? Or are the authors arguing that this is already achieved in the form of Equation (4)?

1 comment
AuthorsOpenReview ↗

Response to Reviewer cxMt

We thank the reviewer for the careful reading and for the positive assessment of our work. Both questions concern details that Section 3.3 states too briefly. We answer them below and hope the answers resolve them. We will also add these details to Sec. 3.3.

Q1. How are the damaged versions P1,…,PmP_1,\dots,P_m generated, and how is mm set?

A fixed program in the verifier builds them. No model is involved, so they do not come from Qwen3.5-9B. On Scale-SWE the program reverts one hunk of the reference patch at a time. On R2E-Gym and SWE-smith-py it applies mutation operators to the changed lines. A damaged version is kept only if the instance's reference tests fail on it, which shows that it changes behaviour.

mm is the number of damaged versions kept. It varies by task. Every admitted task has m≥1m \ge 1 (Sec. 4.2), and on Scale-SWE the limit of four revertible hunks at conversion (Appendix B) caps mm at 4. The table gives mm on the admitted tasks.

SourceHow damaged versions are builtmm per task
Scale-SWErevert one hunk of the reference patchmean 1.6, range 1–4
R2E-Gymmutation operators on the changed linesmean 2.4, range 1–6
SWE-smith-pymutation operators on the changed linesmean 10.0, range 1–30

The mutation operators flip a comparison or boolean operator, change an arithmetic operator or a constant, replace a returned value with None, delete a statement, or swap one name for another.

The damaged versions cover the code that the reference patch changes, and within that code only the damage that the reference tests detect, which is the behaviour the issue is about. They are not meant to cover every possible defect.

Q2. Does Equation (4) select behaviour-preserving tests, and are behaviour-preserving rewrites an ideal concept?

Equation (4) is the reward, and reinforcement learning on this reward guides the model toward tests that satisfy it. A test is rewarded only if

  1. it fails on the buggy snapshot SS and on every damaged version S⊕PiS \oplus P_i, so its outcome depends on the behaviour the fix changes, and
  2. it passes on the repaired snapshot S⋆S^\star and on every rewrite ρj(S⋆)\rho_j(S^\star), so its outcome does not depend on the source text.

Each shortcut in Fig. 2 violates one of the two conditions, so neither is rewarded.

The behaviour-preserving rewrites are implemented, and they rewrite the repaired source rather than the tests. The verifier inserts spaces between tokens, splits string literals into adjacent literals, and inserts no-op statements. It uses a rewrite only if the rewritten files parse to the same syntax tree as the originals, ignoring the inserted no-op statements.

W1. The implementation feels heuristic

The guarantee is realised in the current verifier. Eq. (4) is the guarantee the method makes, and the verifier enforces it on every submission: a test is rewarded only if it fails on every snapshot in N\mathcal{N} and passes on every snapshot in P\mathcal{P}. We measured the shortcuts this check refuses in two ways (details in our response to Reviewer ak7N, W4).

  • Shortcuts we wrote. We ran 10 to 15 kinds of shortcut on 100 admitted tasks of each source, one run per task and kind. The verifier refused 1,387 of 1,395 runs on Scale-SWE, 1,441 of 1,441 on R2E-Gym and 996 of 997 on SWE-smith-py. It rewarded the reference tests of the same tasks on 100 of 100 in each source, so it blocks the shortcuts without refusing correct tests.
  • Tests wrote by the trained model. No rewarded submission in training is a shortcut. Tab. 7 (Appendix C) classifies the 1,790 test-generation submissions collected during reinforcement learning, and 1,728 (96.5%) match none of the shortcut patterns. We read each of the 26 rewarded submissions that match a pattern, and each one tests the behaviour.

W2. Dependence on the quality of the reference tests

The reference tests decide which damaged versions enter N\mathcal{N}, but they never judge the agent's test, and weak reference tests remove a task instead of misleading its reward.

  • In test generation, the reference tests are used only when the task is built. They decide which damaged versions belong in N\mathcal{N}, and they are never compared with the agent's test. The agent's test is judged solely by its own outcomes on the snapshots of N\mathcal{N} and P\mathcal{P}.
  • Weak reference tests remove a task instead of giving it a misleading reward. If no damaged version makes the reference tests fail, the damaged-version condition of Eq. (4) cannot be checked, so the task is not admitted. On R2E-Gym this rule removes 2,169 tasks (Sec. 4.2). Weak reference tests therefore reduce how many tasks a source contributes, not the accuracy of the reward on the tasks it keeps.

Future work at the end of Sec. 6 is another question: using the agent's own tests to check its patches at inferenc. And tab. 4 already shows that the two capabilities could rise together.

Comments on the paper

1 comment

We thank the three reviewers for their careful evaluation and constructive suggestions. The reviewers found the idea novel and timely (cxMt), the verifier simple and practical (ak7N), and the joint training and released tasks useful to others (ubRt). The main questions and our answers are:

  1. Is the joint-training gain reliable? (ak7N) We evaluated every checkpoint of Table 2 three times. The joint model is the best of the four in every evaluation on every benchmark, and paired McNemar tests confirm its lead in 8 of the 9 comparisons.
  2. Does Equation (4) reject the right tests? (ak7N, cxMt) Of the GLM-5.3 tests that plain fail-to-pass rewards, Equation (4) rejects 13.9%, and 98.7% of these still pass when part of the reference patch is removed. Equation (4) therefore stops the model from being rewarded for tests that miss part of the fix.
  3. Are shortcuts refused on all sources? (ak7N, cxMt, ubRt) On 100 admitted tasks of each source, the verifier refuses 3,824 of 3,833 shortcut runs and rewards the reference tests on all 300 tasks.
  4. How are the damaged versions and rewrites built? (cxMt, ubRt) A fixed program in the verifier builds them, with no model involved, and keeps a damaged version only if the reference tests fail on it.
  5. Contamination. (ak7N) The runs that train test generation share no repository with any evaluation set.

The revised paper (changes in blue) adds the full reward, how the damaged versions are built, the evaluation settings, the test-generation prompt, and related work on mutation testing, metamorphic testing and coverage-based rewards.

We appreciate the reviewers’ feedback, which has strengthened the experimental evidence and presentation of this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.