Post-rebuttal update. I thank the authors for the detailed response, the repeated evaluations with McNemar tests, the shortcut probes on all three sources and the repository-level overlap check. However, my main concern remains. The paper's central technical contribution is the strengthened verifier (Eq. 4), yet there is still no RL run that uses the plain fail-to-pass reward under the same tasks and hyperparameters. The new analysis shows that Eq. 4 rejects 13.9% of GLM-5.3 tests, which shows it is stricter, but not that it improves training. The evidence therefore does not isolate how much of the joint-training gain comes from the verifier. In addition, the response clarifies that the set described as "hand-labelled" for the single-behaviour filter was in fact labelled by a model; this needs to be corrected. I am lowering my score from 4 to 3.
Key reasons for my recommendation: (1) The main RL result rests on a single run of 40 steps with no variance estimate, and the single-task baseline behaves abnormally. (2) The strengthened verifier is the core technical contribution, but there is no RL ablation showing that it helps training and no measurement of how often it wrongly rejects honest tests.
1. The core RL result has no variance estimate, and the single-task baseline may be weak.
- What the paper does. Each of the four models in Table 2 is trained once and evaluated once. The paper does not state the number of seeds, the evaluation temperature, or the number of repeats. SWE-bench Verified has 500 tasks, so near 50% the standard error of one evaluation is about 2.2 points. TDD-Bench Verified has 449 tasks, so the standard error is about 2.3 points. On SWE-bench Pro, joint training gains only 1.0 point over the base model (26.0 vs 25.0). Test-generation-only training gains only 1.8 points on TDD-Bench. These gaps are within noise.
- The baseline looks abnormal. The resolution-only model trains for 40 steps with a learning rate of only 1e-6, yet SWE-bench Pro drops from 25.0 to 20.8 (-4.2) and TDD-Bench drops from 38.5 to 35.0, while SWE-bench Verified is nearly unchanged (48.4). Prior work usually reports gains from resolution RL with execution reward. This suggests a problem in the resolution-only setup, such as unstable training, an unsuitable patch-size filter (1,287–3,982 bytes), or hyperparameters tuned for the joint run. The claim that joint training beats either single task depends on this baseline.
- Why it matters. This is the main empirical claim of the paper. The abstract and conclusion both repeat these numbers.
- What would fix it. (a) Run at least 3 seeds per setting and report mean and standard deviation. (b) Run paired tests (McNemar or paired bootstrap) between models in Table 2. (c) Show reward curves and evaluation curves over training steps for all three trained models. (d) Train resolution-only longer or tune its hyperparameters, to show the drop is not a tuning artifact. (e) State the evaluation decoding settings and number of repeats.
2. No ablation isolates the effect of the strengthened verifier on training.
- What the paper does. The method rests on Eq. 4 being stricter than plain F2P (Eq. 2). Section 5 has no run trained with plain F2P reward. Table 7 (Appendix C) shows that shortcut-like submissions are only about 3.6% of training submissions ((23+12+29) / 1,790). After manual reading, the authors judge that 0 of the 27 rewarded submissions lack behavioural content. This suggests that plain F2P reward may be enough at 40 steps with a 9B model.
- Why it matters. If removing the N and P families gives similar results, the main technical contribution does not affect the reported results. If the verifier does help, an experiment should show it. The verifier also costs m+n+2 runs plus repeat runs per submission, and the paper does not report this cost.
- What would fix it. (a) Add an RL control with the same tasks and hyperparameters, with reward replaced by plain F2P (Eq. 2). Compare all Table 2 metrics and the shortcut rate. (b) Remove the N family and the P family separately to show the effect of each. (c) Report the average m and n per task and the verifier wall-clock time.
3. The false-negative rate of the verifier (wrongly rejecting honest tests) is not measured.
- What the paper does. Eq. 4 requires the test to fail on all damaged patches. The reference tests decide which damaged patches are kept (Section 3.3). The "honest probe" in Appendix C uses the reference tests themselves. Since the N family is filtered by those same tests, this probe passes by construction and says nothing about other honest tests.
- Why it matters. An honest test that reproduces only the main symptom in the issue may not catch the reversal of another hunk in the reference patch (Appendix B allows up to 4 hunks). Such a test receives 0 reward. This pushes the model to copy the coverage of the reference tests, which may differ from the behaviour described in the issue. Section 3.2 itself admits that a mismatch between the patch and the description makes the reward "misleading". The single-behaviour filter relies on heuristics such as hunk count and the number of sections in the description (Appendix B). The paper mentions a hand-labelled set but does not give its size or labelling procedure.
- What would fix it. (a) Take tests from GLM-5.3 or another strong model that pass plain F2P, and report the fraction that Eq. 4 rejects. Manually check a sample of the rejected tests to see whether they truly lack behavioural content. (b) Report the size of the hand-labelled set, the number of annotators, and their agreement. (c) Discuss requiring failure on only a subset of damaged patches, or giving partial credit per hunk.
4. The shortcut probe is small and narrow.
- What the paper does. Table 6 has only 86 probes, 13–16 per category, all from Scale-SWE, and all fixed shortcuts written by the authors. SWE-smith-py and R2E-Gym are not probed.
- Why it matters. A 100% rejection rate shows only that the verifier blocks the attacks its designers anticipated. It says little about adaptive attackers or new shortcuts that may appear in long RL runs. Page 2 says admission audits shortcuts "on a sample". 86 probes against 23,520 tasks is too few to support the claim that the verifier is audited.
- What would fix it. Extend the probe to all three sources, with at least several hundred tasks each. Add LLM-generated adversarial shortcuts, for example by prompting a model to "write a test that checks no behaviour but still gets reward". Report the fraction that passes the verifier.
5. Contamination is checked only at the instance level.
- What the paper does. Section 5.1 states only that the 480 RL training tasks "share no instance" with the evaluation sets. Scale-SWE covers 888 repositories and likely overlaps with the 12 repositories in SWE-bench Verified (such as django and sympy). Similar issues from the same repository may also appear. The SFT part (Table 3) has a repository-level exclusion analysis; the RL part does not.
- Why it matters. Training on the same repositories makes the model familiar with their code structure and may inflate SWE-bench Verified and TDD-Bench Verified scores. The effect may be larger for the joint model, since its test-generation task has the same form as TDD-Bench.
- What would fix it. List the repositories shared between RL training tasks and the evaluation sets, and report Table 2 with those repositories removed, as in Table 3. Explain how the "4 shared instances" in Table 3 were identified (instance id, commit, or issue-text similarity).
6. The behaviour analysis in Table 5 does not support the causal explanation.
- What the paper does. Section 5.2 explains the resolution-only drop by reduced testing behaviour. But the test-generation-only model is also worse on all four testing metrics in Table 5 (commands per turn 0.54, tests written under tests/ 5.8%, median test runs 7, no test runs 27.9%), and it still gains 2.2 points over the base model on SWE-bench Verified. This contradicts the explanation that more testing leads to more resolved tasks. Also, the evaluation turn limit is 250 while the training limit is 80. The base model hits the 250-turn limit on 28.7% of tasks and the joint model on 14.8%. Part of the gain may come from fewer truncations.
- Coverage reporting. Section 5.2 says the joint model's tests "execute the lines the reference patch changed about equally well", but the lower quartiles reported right after are 100% vs 50%, which differ clearly. The authors count missing coverage reports as 0, while the official TDD-Bench implementation counts them as full coverage. The paper should state which convention the "TDD score" in Table 2 uses.
- What would fix it. Report resolved rate split by whether the run hit the turn limit, or re-evaluate with an 80-turn limit. Remove or rewrite the causal claims. Use one coverage convention throughout.
7. Task selection and sampling details are unclear, which affects reproducibility and fairness.
- The training pool "drop[s] the tasks that earlier runs solved in no trajectory at all". Which runs are these, and which model produced them? If early joint runs were used, the three models do not see the same training distribution.
- The pools differ in size (1,329 resolution tasks and 973 test-generation tasks), which the authors say "equalises what reaches the update". Pool size affects the ratio only under uniform sampling from the union. Please state the per-step sampling ratio of the two task types in joint training, and the actual ratio of groups that reach the update.
- The reward is not strictly binary. Table 7 and Appendix D mention a "deduction for not following the submission protocol", but neither Eq. 4 nor Section 5 defines it. Please give the full reward definition.
- Please confirm whether the resolution training tasks come from the 4,677 admitted Scale-SWE resolution tasks in Section 4.2.
- Missing details: max response length, evaluation temperature, tool list per trajectory, the prompt template for test-generation tasks, and an anonymous link to code and data.
8. No empirical comparison with directly related work, and the SFT experiment lacks controls.
- SWE-Tester (Soni et al., 2026) already provides 41K reproduction instances and trains open models. SWE-Playground trains jointly on resolution and reproduction. The paper cites both in related work but does not compare numbers on TDD-Bench Verified or compare data quality. Since converting issue-resolution instances into test generation has been done before (SWT-Bench and TDD-Bench Verified for evaluation; SWE-Playground and SWE-Tester for training), the novelty lies mainly in the strengthened verifier, which makes Weaknesses 2 and 3 more important.
- The SFT gain in Section 5.3 (26.1% to 73.3%) comes mostly from distilling a strong teacher (GLM-5.3, 87.8%). To show the value of using the verifier to filter trajectories, two controls are needed at the same token budget: the same GLM-5.3 trajectories filtered by plain F2P (or not filtered), and SWE-Tester data.
- Table 3 uses the claude-code scaffold (base 26.1%), and Table 2 uses mini-swe-agent (base 38.5%). The two tables cannot be compared directly. I suggest reporting the RL and SFT models under the same scaffold.
Minor issues
- Damaged patches are mutants in the sense of mutation testing, and "the test must kill the mutant" corresponds to mutation score. Please cite classic mutation testing work (e.g., the survey by Jia & Harman) and recent work that uses mutation score to evaluate LLM test generation. The semantically equivalent rewrites relate to the literature on semantic-preserving transformations and metamorphic testing.
- The definition of rtest in Eq. 4 does not include the condition at the end of Section 3.3 that the same test is run twice with consistent results. Please add it to the definition or state that it is an extra condition.
- The patch-size range 1,287–3,982 bytes is very specific. Please explain how it was chosen. 1,287 is also the number of Scale-SWE tasks that fail admission (Appendix B); please check that this is not a typo.
- Section 4.2 says 4,677 of the 19,473 Scale-SWE instances admit a resolution task. This number is not in Table 1, and its relation to the RL training pool is not stated.
- In Table 5, "Tasks with a test written under tests/" counts tests in the tests/ directory. Is the fixed submission path for test-generation tasks also under tests/? If not, this row may undercount testing by the test-generation-only model.
- The Terminal-Bench result in Appendix D (18/89 vs 21/89) is not significant, as the authors note. Please state the claim that the conversion does not depend on tasks being repository issues more cautiously.
Response to Reviewer ubRt
We thank the reviewer for the review. We hope the clarifications below resolve the reviewer's concerns.
Q1. How does the agent know which tests to write?
The agent is asked to test only the reported issue, and only a test of that issue can be rewarded. The prompt is the issue together with one instruction: write a single test file, at a fixed path, that fails on the current repository because of the reported problem and passes once it is fixed. The verifier then judges the test only on snapshots that concern this issue: the buggy and repaired snapshots, damaged versions of the patch that fixed this issue, and rewrites of the files that patch changed. A test of unrelated code passes on the buggy snapshot and receives no reward. The prompt template is INSTRUCTION_TMPL in filtering/admission/f2p_env.py of the supplementary material, and each of the 20 sample tasks there includes its filled-in prompt as instruction.md.
A task is kept only when the issue and the patch concern the same behaviour (Section 3.2, Appendix B), so the issue describes the behaviour the verifier checks. An instance whose patch also contains changes the issue does not mention is excluded before any container is run, since a test written from the issue could not be expected to detect them.
Q2. Can a test that only checks syntax differences be rewarded?
No. Plain fail-to-pass would reward such a test, for example one that asserts a line of the buggy source is absent, because it fails before the fix and passes after it. Equation (4) adds two checks that such a test cannot pass.
First, the test must also fail when part of the fix is removed. We build damaged versions of the reference patch, each undoing or breaking one part of it, and keep a damaged version only if the project's own tests fail on it (N in Equation (4)). A test that still passes on a damaged version does not check the behaviour the fix changes, so it gets no reward. A rewarded test therefore checks the behaviour the issue reports.
Second, the test must still pass when the fixed code is rewritten without changing what it does (P). The rewrites add spaces between tokens, split string literals and insert
passstatements. A test that looks for a piece of source text fails on them, because the text has changed while the behaviour has not. A test that hashes the syntax tree of the file also fails, because the insertedpassstatements change the hash. Figure 2 shows both kinds of shortcut on one instance.We also checked this in practice. We wrote 10 to 15 kinds of shortcut test and ran each on 100 tasks per source. The verifier refused 3,824 of the 3,833 runs and still rewarded the project's own tests on every task. During RL, no rewarded test was a shortcut: we read every rewarded test that resembled one (Table 7). Finally, GLM-5.3 wrote 23,315 tests that plain fail-to-pass would reward. Equation (4) rejects 3,230 of them (13.9%), almost all because they still pass when part of the fix is removed.