acceptodds
Under review as a conference paper at ICLR 2027

Fix-Test Collapse: Diagnosing and Resolving Self-Confirming Verification in Code Agents

Abstract

Self-testing lets code agents inspect execution feedback and repair errors, yet successful local validation can precede failure on held-out tests. We call this failure Fix–Test Collapse and distinguish tests that miss bugs from misinterpretation of outputs that expose them. We propose Falsification-Guided Self-Testing, which combines protocol SFT, action-level reinforcement learning, and trajectory rewards under a single-trajectory interface that withholds reference outputs and hidden verdicts from the policy. Offline scorers reward diagnostic tests on contest tasks, and an issue-grounded judge, updated with on-policy hard negatives, supplies the diagnostic signal on repository tasks. Across three model families, pass rates improve by up to 34.7% on CodeContests and 19.5% on LiveCodeBench relative to untrained tool-enabled policies, and Qwen3.5-35B reaches 48.3% resolution on 731 SWE-bench Pro tasks and 28.3% on the complete 113-task DeepSWE split. In a matched control starting from an identical SFT checkpoint, falsification RL outperforms final-only RL on all three paired training seeds, by 7.7 points on Pro and 10.6 points on DeepSWE on average. Training also raises self-test coverage and lowers collapse among locally validated submissions; this conditional reliability measure is distinct from an all-instance failure rate.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Code Agents, Self-Testing, Self-Verification, Large Language Model, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_oJAEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies agents that accept their own local verification but fail held-out evaluation. It separates weak test inputs from misinterpretation of outputs that expose an error, and proposes a training curriculum combining protocol SFT, action-level RL, and full-trajectory RL. Contest tasks use offline reference executions to reward diagnostic inputs, while repository tasks use a dynamically updated rubric judge. Reference outputs and hidden verdicts are withheld from the acting policy. The extended Qwen configuration reaches 48.3% on 731 SWE-bench Pro tasks and 32/113 on DeepSWE. A separate, stronger causal control starts both arms from the same SFT checkpoint and reports mean improvements of 7.7 and 10.6 percentage points over final-only RL across three paired training seeds.

Critical strengths
  1. The distinction between producing a diagnostic test and correctly interpreting its output is useful. The operational definition of collapse also carefully separates conditional error among locally validated submissions from all-instance failure and unverified submission.
  2. Offline references and final evaluators supply training supervision without being exposed as online feedback, and the repository initial-patch boundary is fixed independently of when the agent chooses to test. These details make the setting substantially easier to interpret.
Critical weaknesses
  1. The dynamic judge's contribution and trajectory-level reliability remain incompletely isolated. The audits show accepted-probe precision of 80.6% before updates and 94.3% immediately afterward, but each boundary samples only 20 accepted probes, and the final pre-update sample has 15 valid probes out of 20. These results support mitigation, as the authors state, rather than sustained calibration. Since D_agent is positive if any probe is accepted, probe-level precision does not directly establish how often an entire rewarded trajectory contains genuinely diagnostic evidence. A fixed-judge versus dynamic-judge control, along with an audit of the trajectory-level diagnostic event and an estimate of false negatives on current policy probes, would clarify the value of adaptation.

  2. The most convincing treatment comparison establishes a package effect. Table 3 changes initial-correctness shaping, diagnostic reward, the collapse penalty, and the associated judge machinery together. Appendix H offers useful component ablations, but repository ablations use the earlier budget on SWE-bench Verified rather than the Pro/DeepSWE post-SFT setting carrying the strongest claims. A focused ablation in that setting would strengthen attribution. Likewise, rewarding diagnostic discovery followed by successful repair does not show that the agent used the diagnostic trace; a trace-redaction or replacement intervention would distinguish evidence-driven repair from extra testing accompanying better repair.

Questions for authors
  1. Can the authors provide the complete verification/correctness contingency counts and validate the extraction of local-validation status?
  2. How often is the trajectory-level diagnostic bonus supported by at least one genuinely diagnostic probe, and what is the effect of freezing the judge?
  3. Do the diagnostic bonus and collapse penalty retain their benefits individually under the matched Pro/DeepSWE post-SFT protocol?
  4. What are the exact task manifests, near-duplicate exclusions, and additional judge/teacher costs?
3 comments

Thank you for the careful review — the weaknesses identify real gaps, addressed below with new matched experiments. This comment answers Q1 and Q4; Q2 and Q3, which share one set of matched experiments, follow in the next comment.

  • R1 — Contingency counts and L extraction (Q1)

Let n11, n10, n01, n00 count (L,CK)=(1,1),(1,0),(0,1),(0,0); then L=n11+n10, E=n10, U=n00. Every condition samples N=100 trajectories, so L/N, E/N, U/N in percent equal n11+n10, n10, n00; success is (n11+n01)/N; E/L is shown rounded to one decimal. Subsamples carry sampling error vs. the full tables, widest in small-L cells.

R1a (Table 4; original 3,300/condition), CodeContests; Stage 2 = S1 SFT + RL:

Conditionn11n10n01n00E/L(%)Succ(%)
Qwen/Official37553570.058
Qwen/+S1243836261.360
Qwen/+S2622115225.377
GPT-OSS/Official84454333.353
GPT-OSS/+S1502042628.654
GPT-OSS/+S2601532220.063
Gemma/Official4914142322.263
Gemma/+S15911121815.771
Gemma/+S27583149.678

R1b (Tables 14/16), SWE-bench Verified; full reward and no penalty are β=0.3 and β=0:

Conditionn11n10n01n00E/L(%)Succ(%)
Full reward (β=0.3)612411428.262
No diagnostic reward4430141240.558
No collapse penalty (β=0)453314842.359
β=0.158277831.865
β=0.5592351328.064

From Stage 1 to Stage 2, coverage rises (Qwen 62→83%, GPT-OSS 70→75%, Gemma 70→83%), L=1 failures fall (38→21, 20→15, 11→8), and unverified failures are 2→2, 26→22, 18→14. Fewer falsely validated failures accompany broader local validation. Against Official, Qwen and GPT-OSS still show higher E/N despite lower E/L as coverage expands. On Verified, full vs. no collapse penalty: 9 fewer collapsed failures, 6 more unverified, 3 more successes (62% vs. 59%); vs. no diagnostic: 4 more successes (62% vs. 58%). At β=0.5, coverage falls to 82, collapsed failures to 23, success rises to 64; β=0.1 reaches 65 — this sample does not rank β=0.3 highest; the full Table 16 does (65.4).

Labeling L. L(τ)=1 requires a qualifying local check on the final submitted candidate plus an explicit submission. A qualifying check concerns issue-related behavior and executes validly — skips, collection/infrastructure failures, and unrelated/inconclusive checks never qualify (where present, the run's judge supplies relevance and execution_validity) — and is rerun if the candidate changes. Other rubric fields enter D_agent, not L, so a valid test that misses the bug can still qualify; timeout auto-captures and unchecked submissions give L=0. Contest acceptance is the explicit SUBMIT after valid local execution (Listing F.3); "pass" is the agent's own reading, so L=1 can coexist with a wrong answer. L follows each run's own labeling; final resolution uses the common held-out evaluator. On a new 300-trajectory sample, extractor–human agreement is 95.7% (TP/FP/FN/TN 251/9/4/36), inter-annotator 97.3% pre-discussion; 12 ambiguous cases ended 8/4.

  • R4 — Manifests, exact-match exclusions, and judge/teacher costs (Q4)

Code and data are released upon acceptance, including a task manifest for every training and evaluation set (exact task IDs, dataset/environment/evaluator versions, SHA-256 checksums): repository training/validation (4,000/200 tasks, 11 repositories, R2E-Gym-V1), per-seed Table 3 schedules, the 731-task Pro and official 113-task DeepSWE sets with base commits, contest training (TACO, CodeContests) with source revisions and preprocessing versions, and the CodeContests/TACO-Verified/LiveCodeBench releases with date ranges and retained IDs. LiveCodeBench uses release_v6 (1,055 problems, May 2023–April 2025). The TACO-Verified diagnostic evaluation covers all 12,898 problems in the public likaixin/TACO-verified corpus.

Exclusion removes exact task-ID matches and, after removing spaces, exact issue/problem-text matches; sharing a repository alone is allowed — the rule is exact-match by design. The judge-distillation pool is split by issue (3,600/400).

Costs. Beyond the matched 16,000 H200 GPU-hour policy allocation, each full-reward run adds 160 GPU-hours of local judge inference and 80 of judge-update optimization (240 total, 1.5%) and 76,000 newly teacher-labeled probes (246.0M input / 12.2M output tokens); the final-only arm adds none. One-time: the 4,000-probe teacher-labeled judge pool (3,600/400, six rubric fields per probe) and 40 GPU-hours of initial judge training; the new rebuttal audits took about 40 person-hours. For reference, the audited 400-update extended continuation added 50,776 teacher-labeled probes (judge updates: 400 new hard negatives + 400 replayed positives each); the earlier pipeline and the continuation used ≈8,512 and 10,752 GPU-hours. Billed API charges, retry/cache accounting, and these budget figures will be added in the revision.

  • R2 — Trajectory-level support, the frozen-judge control, and false negatives (W2/Q2)

As the review notes, accepted-probe precision differs from whether a bonus-eligible trajectory holds ≥1 genuinely diagnostic probe. We audited 50 bonus-eligible trajectories from 50 distinct issues (25 Pro + 25 DeepSWE, one per issue; final extended-continuation Qwen3.5-35B checkpoint, 200+400 updates, Table 2). These are instrumented evaluation rollouts: the run's final judge version scores probes at generation, logged only, never shown to the policy; evaluation stays judge-free as in the paper; the audit conditions on the recorded eligibility fields (C_0=0, C_K=1, D_agent=1); C_K comes from the benchmark evaluator, C_0 from the training-time frozen-initial-patch protocol. A probe counts only if it runs validly, checks a concrete issue requirement, and exposes an issue-specific violation a correct fix should resolve. Genuine evidence supports 45/50 audited trajectories (95% Wilson CI 78.6–95.7%); 3 all-false-positive, 2 unresolved, all in the denominator. By probe count: 14/14 at 1–3, 18/21 at 4–7, 13/15 at ≥8.

Frozen vs. dynamic judge. Both arms share the protocol-SFT Qwen3.5-35B start, task schedules, budgets, and Eq. 5 coefficients (α=0.6, λ=0.2, β=0.3); the dynamic arm updates the initial Qwen3.5-4B judge after steps 60/120/170 (400 new hard negatives + 400 replayed positives each; rollouts never rescored). Cells: seed-42/43/44 rates (731 Pro / 113 DeepSWE tasks); bold means and differences computed from counts, truncated to one decimal; the two R3 ablations included:

ArmPro s42/s43/s44 (mean)DeepSWE s42/s43/s44 (mean)
Frozen judge38.3; 35.9; 38.0; 37.416.8; 15.9; 17.6; 16.8
Full (dynamic judge)41.7; 39.5; 39.9; 40.419.4; 18.5; 19.4; 19.1
No diagnostic36.3; 35.7; 38.4; 36.816.8; 13.2; 14.1; 14.7
No collapse penalty31.8; 35.2; 31.7; 32.914.1; 11.5; 15.9; 13.8

Dynamic updating beats the frozen judge by +2.9 (Pro) / +2.3 (DeepSWE) points, positive for every seed (DeepSWE: 2–3 tasks/seed). Local-validation counts (seed-42 step-200 checkpoints, 100 per row; n11,n10,n00 count (L,CK)=(1,1),(1,0),(0,0), n01 the rest; E/L rounded):

Conditionn11n10n01n00E/L(%)Succ(%)
Pro/Frozen315071261.738
Pro/Full reward384731255.341
Pro/No diagnostic325151261.437
Pro/No penalty25598870.233
DeepSWE/Frozen146242081.618
DeepSWE/Full reward215902073.821
DeepSWE/No diagnostic146521982.316
DeepSWE/No penalty136512183.314

The dynamic arm has wider labeled coverage, 3 fewer L=1 failures per 100 on each benchmark, and unchanged unverified counts.

False negatives. The trajectory audit cannot see probes the judge rejects, so we also labeled 200 current-policy probes per benchmark, judge-negative included. Judge precision/recall: Pro 91.5/86.0 (TP/FP/FN/TN 86/8/14/92), DeepSWE 89.0/83.0 (73/9/15/103); pooled recall 84.6%. The missed 14/100 and 15/88 diagnostic probes quantify evidence the judge rejects.

  • R3 — Components under the matched setting, and evidence use (W3/Q3)

As W3 notes, Table 3 establishes a package effect (Pro 47.4% vs. 39.7%, DeepSWE 26.5% vs. 15.9% at 600 updates). The 200-update ablations (λ=0 or β=0; everything else as in the judge control, same seeds) are the ablation rows above, with per-seed resolution and seed-42 counts. Removing the diagnostic term costs 3.5 (Pro) / 4.4 (DeepSWE) points on the means, the collapse penalty 7.4 / 5.3; full reward beats each ablation for every seed on both benchmarks. Behaviorally, full reward has the lowest labeled conditional collapse, widest coverage, and highest sample success; versus no-penalty, 12 fewer collapsed vs 4 more unverified on Pro, 6 fewer collapsed and 1 fewer unverified on DeepSWE.

Trace redaction/replacement. We ran the proposed intervention from the same incorrect candidate — policy, prior history, environment state, and remaining generation/tool budgets fixed — 60 prefixes from 60 distinct issues (30 Pro + 30 DeepSWE) × 3 paired continuation seeds, all four conditions per pair (sham: neutral reformatting, byte-for-byte recoverable, exit status kept, no hint):

ConditionSuccess%
Intact112/18062.2
Redacted92/18051.1
Replaced88/18048.9
Sham107/18059.4

Removing or replacing the evidence costs 11.1 and 13.3 points; reformatting alone costs 2.8. Issue-level paired 95% intervals: intact−redacted [2.1, 19.8], intact−replaced [4.5, 22.3], intact−sham [−6.0, 9.5] — the first two exclude zero, the sham interval spans it. This supports use of the diagnostic trace beyond rewarded co-occurrence, though it does not assign the entire Table 3 gain to this mechanism.

Thank you for running the additional controls. The frozen-judge comparison and the matched reward ablations address my concerns about what drives the gains. I keep positive on this paper.

Reviewer_e7MtOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a failure in code agents that test their own work. The agent believes its local validation passed and submits, but the answer fails on a held-out evaluator. The authors name this Fix–Test Collapse (Eq. 1) and split it into two causes: weak test inputs that do not expose the bug, and judgment errors where the output exposes the bug but the model misreads it. The proposed method, Falsification-Guided Self-Testing, has three stages: protocol SFT, action-level RL (Phase A), and end-to-end trajectory RL (Phase B). On competition problems, the reward uses diagnostic test inputs (DTI, Eq. 2) computed offline with a reference program. On repository tasks, the reward comes from a Qwen3.5-4B rubric judge distilled from Claude Opus 4.6 and updated with on-policy hard negatives, plus a collapse penalty (Eq. 5). The main results are gains of up to 19.0 and 14.2 points over Official on CodeContests and LiveCodeBench, 48.3% on 731 SWE-bench Pro tasks and 28.3% on 113 DeepSWE tasks for Qwen3.5-35B, and gains of 7.7 and 10.6 points over final-only RL in a matched post-SFT control with 3 training seeds (Table 3).

Critical strengths
  1. The problem is well defined. Eq. (1) defines an observable event. The main text and Appendix G consistently separate the conditional rate P(C_K=0|L=1) from the all-instance incidence P(E=1). This avoids a common metric confusion.
  2. The information boundary between policy and scorer is careful. Reference outputs, hidden verdicts, and judge labels never enter the policy context (Listing C.1, Section 3.3). The DTI definition in Eq. (2) requires the input validator to pass and the reference to run normally, which removes spurious reward from invalid inputs (Appendix C).
  3. The definition of C_0 handles reward timing. Section 3.3 defines repository C_0 with a frozen initial patch and explains that using the "first validation command" as the boundary would create a 0.4 reward gap. This level of analysis is uncommon.
  4. The post-SFT control in Table 3 uses 3 paired training seeds, and the gain is positive for every seed.
  5. The dynamic judge has a per-window blind human audit (Table 12). The paper reports that reward gaming cases increase during training. The authors report this negative signal openly.
  6. The numbers are internally consistent. I checked most numbers in the text against the tables and found only a few points that need clarification.
  7. The authors are careful about the scope of their claims. They state in several places when a result does not establish causality or cannot rule out other explanations.
  8. The diagnosis of collapse into weak-input and judgment failures is a useful new framing. It gives the community a concrete way to measure why self-testing fails (Table 4).
Critical weaknesses

Key reasons for my recommendation: (1) the main behavioral metric, conditional collapse, can be lowered by doing less local validation, and the paper does not report the numbers needed to rule this out; (2) the repository gains are not attributed to the falsification mechanism, and the falsification arm receives extra supervision from Claude Opus 4.6.

1. The collapse metric can be reduced by skipping local validation. The core behavioral evidence is the drop in conditional collapse (Tables 4, 13, 14, 16). Its denominator is the set of submissions with L=1. The −βE term in Eq. (5) only penalizes failures with L=1, so the policy can avoid the penalty by running less post-patch validation. The Limitations section acknowledges this but gives no data. The existing numbers suggest the risk is real:

  • In Table 14, removing the collapse penalty raises collapse from 27.2 to 39.8, while the resolved rate only moves from 65.4 to 63.7. A large change in collapse with almost no change in correctness fits the explanation that the penalty moves failures from L=1 to L=0.
  • Table 2 pass rates and Table 4 conditional collapse give an upper bound on P(L). For Qwen Stage 2, pass is 73.8 and collapse is 30.5, so P(L) ≤ (1−0.738)/0.305 ≈ 0.86. The self-test invocation rate is 98.2%. So at least about 12% of trajectories call self-tests but do not count as local validation. The outcomes of these trajectories are not reported.
  • On repository tasks, L(τ) depends on whether the agent "treats a validation signal as sufficient" (Appendix G). The paper does not say how this is decided at training and evaluation time (rules, an LLM, or humans). If an LLM decides it, the policy could also exploit that decision.

I suggest reporting P(L), the all-instance incidence ΣE/N, and the failure rate with L=0 for every row of Tables 4, 13, 14, and 16. Please also describe how repository L(τ) is computed and give its agreement rate with human labels.

2. Repository gains are not attributed to falsification, and the falsification arm has extra Claude supervision. The control arm in Tables 2 and 3 is final-only RL (2C_K−1). The Ours arm differs from it in several ways: the αC_0 term, the λ diagnostic reward, the −βE penalty, and a judge distilled from Claude and continually updated with Claude labels during training. Appendix E shows that about 6,300 probes per 50-step window are labeled by Claude, 50,776 in total, plus human review. The final-only arm "omits judge calls". The two arms are matched in policy update steps, but they are not matched in external supervision. Part of the gain may come from distilling Claude's knowledge of what makes a good test. The current experiments cannot separate this from the effect of falsification.

Section 4.2 states that Table 3 does not separate the reward terms and points to Appendix H. The repository ablation in Appendix H (Table 14) has these limits:

  • It uses an earlier checkpoint (Verified 65.4), not the extended checkpoint used for the main results.
  • It is run only on SWE-bench Verified, while Pro and DeepSWE are the main evaluations.
  • It is a single run with no standard deviation. Table 6 shows a run-to-run standard deviation of about 1.4 to 2.3 on Verified. The gap of 1.7 between 65.4 and 63.7 in Table 14, and the β sweep range of 63.7 to 65.4 in Table 16, are both within this noise.
  • In Table 14, "w/o diagnostic judge reward" scores 61.6, below Official at 63.2 in Table 6. This suggests the rest of the training pipeline may hurt performance. This needs an explanation.

There is also no downstream comparison of the dynamic judge against a static judge. The dynamic judge is a claimed contribution, but the only evidence is judge accuracy (Table 12). There is no evidence on policy performance.

I suggest adding these arms in the Table 3 setting (at least Qwen, 3 seeds), evaluated on Pro and DeepSWE: (a) without the λ diagnostic term; (b) without −βE; (c) without αC_0; (d) a static judge that is not updated; (e) a control that uses the same amount of Claude labeling but does not reward falsification, for example Claude judgments of patch quality as a dense reward.

3. Statistical uncertainty is under-reported, and some intervals need checking.

  • The repository results in Table 2 (Pro, DeepSWE) use one training seed and one evaluation run. GPT-OSS and Gemma have only point estimates. DeepSWE has 113 tasks, so one task equals 0.88 points. For Gemma, Final-only and Ours differ by 27 versus 18 tasks with no paired test.
  • Table 3 uses only evaluation seed 0 for each training seed, so evaluation noise is not estimated. The gains across the 3 seeds are very close (Pro 7.3 to 8.2; DeepSWE exactly 11, 12, and 13 tasks). This is unusual in agent RL. Please report absolute resolved rates per seed and use several evaluation seeds.
  • The competition results in Table 2 are said to be means over three runs, but only Qwen has standard deviations (Table 11). The margins of GPT-OSS and Gemma over Revisit Self-Debugging are only 2.2 to 3.0 points on 165 CodeContests problems. These need standard deviations or paired tests. The competition results also use only one training seed.
  • The three 95% intervals in Table 7 have identical half-widths of ±3.4 ([3.2, 10.0], [1.8, 8.6], [2.6, 9.4]) and are exactly symmetric around the point estimates. The three models have different numbers of discordant tasks, so a task-level percentile bootstrap is unlikely to give identical half-widths. Table 6 also has three runs, and the paper does not say how the bootstrap pairs them. Please give the raw paired counts and the code.
  • The ablations in Tables 13, 15, and 16 are single runs without standard deviations. In Table 15, the gap between λ=0.1 and λ=0.4 (71.9 vs. 72.6) is of the same size as the standard deviation in Table 11 (about 0.8).

I suggest adding standard deviations or paired intervals to all main tables, at least 3 training seeds for the main repository results, and a description of how Table 7 was computed.

4. Data contamination is not ruled out.

  • Competition training uses TACO and CodeContests training problems (Section 4.1). TACO collects problems from CodeContests, Codeforces, and other sources, so it may overlap with the CodeContests test set. No deduplication procedure is described.
  • The diagnostic study in Section 2 uses TACO-Verified. Its relation to the training set is also not described.
  • The LiveCodeBench version and problem date range are not given. Qwen3.5, GPT-OSS, and Gemma4 are recent models. If the date range is before their training cutoffs, the base models may have seen the problems.
  • Repository training uses 11 repositories from R2E-Gym. The authors only exclude identical task IDs and normalized issues, so shared repositories are allowed (Section 4.1). The paper does not list which repositories are shared with SWE-bench Verified and Pro.
  • Claude Opus 4.6 serves as judge teacher and audit labeler. Its training data may contain the issues and fixes from these public benchmarks.

I suggest giving the TACO/CodeContests deduplication method and overlap statistics, the LiveCodeBench version and date range, and the list of overlapping repositories, with Verified and Pro results split by shared versus non-shared repositories.

5. Baselines are not fully fair or complete.

  • On competition tasks, Ours uses thousands of GPU hours of training (about 8,512 H200 GPU-hours for Qwen, Appendix G). Trigger, CodeT, Self-Edit, and Revisit Self-Debugging are inference-time methods. The only trained baseline, ReVeal, is not described as retrained on the same base model, data, and budget. The most direct trained control is final-only RL with the same budget. It appears only once, for Qwen in Table 13 (62.7), and is not in Table 2.
  • CodeT still beats Ours on Qwen (75.4 vs. 73.8; 88.5 vs. 87.2). The authors explain that this is a call-count comparison. It would help to add best-of-k or CodeT-style reranking on top of Ours to see whether the two are complementary.
  • Self-Edit and Revisit are adaptations, which the authors acknowledge are not the original methods (Appendix D.1). This weakens the comparison.
  • On repository tasks, Table 2 has only Official and Final-only RL. Adv. Prompt appears only in one sentence in Appendix B (38.0). The repository version of ReVeal on Verified is not described, and it scores below Official (Table 6). There is no comparison with published open-source repository RL agents of similar size, such as agents trained on SWE-Gym or R2E-Gym.

6. The oracle condition in the motivating study (Section 2, Table 1) leaks correctness information. The "After Oracle Benchmark Inputs" condition in Table 1 gives benchmark failure inputs only to problems the model got wrong. Even without reference outputs, the fact that extra inputs are given tells the model its answer is wrong. So "Recovered by Oracle" mixes two signals: better test inputs and an implicit wrong-answer verdict. This weakens the conclusion that verification is the bottleneck. Please show the prompt the model sees in the oracle condition. I suggest a control that gives the same number of benchmark inputs to all problems, including correct ones, or mixes in non-failing inputs.

In addition, the "Collapse Rate" in Figure 1(a) has no stated denominator. Its values (for example GPT-Low 19.5%/32.3%) do not match the collapse incidence in Table 1 (22.3%). The text does not explain what conclusion the patch similarity in Figure 1(b) supports.

7. Evidence for repository generalization is concentrated in Python. R2E-Gym contains only Python repositories. Table 8 shows gains of 15.4 points on Python, 8.9 on Go, 9.1 on JavaScript, and 0.0 on TypeScript (only 20 tasks). This pattern fits the explanation that gains are largest in the training language. Contribution (iii) mentions "limits of repository generalization", but the main text does not discuss this pattern. The language breakdown of DeepSWE is not reported. Please give it and discuss where the non-Python gains come from.

Minor issues

  1. The abstract reports "up to 34.7% on CodeContests and 19.5% on LiveCodeBench". These are relative gains (73.8/54.8, 87.2/73.0), while the main text uses absolute points (19.0, 14.2). I suggest using points in the abstract.
  2. Section 2 says "Table 4 reports the aggregate decomposition", but that paragraph discusses TACO-Verified and Table 4 is on CodeContests.
  3. In Eq. (3), a trajectory with C_0=1 and C_K=0 gets −0.4, which is higher than −1 for C_0=0 and C_K=0. So a trajectory that was right and then broke the answer is penalized less. Please state whether this is intended.
  4. In Table 13, the Initial value of the "Final correctness only" row is 54.8, the same as the final pass rate of Official in Table 2. Please confirm this is not a copy error.
  5. The frozen initial patch protocol for repository C_0 is described only for training rollouts. Please state whether evaluation uses the same protocol. If not, training and test conditions differ.
  6. The Reproducibility Statement does not say whether code, judge weights, Claude labels, and audit data will be released. The judge is a core part of the method, so this matters.
Questions for authors
  1. For each row of Tables 4, 13, 14, and 16, what are P(L), ΣE/N, and the failure rate with L=0? Does the share of submissions without local validation increase after Stage 2?
  2. How is repository L(τ) (the agent "treats as sufficient") decided at training and evaluation time? What is its agreement rate with human labels?
  3. Apart from the reward function, how much external supervision (Claude labels, human review) does the falsification arm receive beyond the final-only arm? Is there a control with the same amount of Claude labeling?
  4. Are there downstream Pro/DeepSWE results for a static judge that is not updated?
  5. How were the Table 7 intervals computed, given their identical half-widths? How are the three runs paired?
  6. What is the absolute resolved rate for each seed in Table 3? How much do the gains change with different evaluation seeds?
  7. How were TACO training problems deduplicated against the CodeContests test set? Which LiveCodeBench version and date range were used?
  8. Which R2E-Gym training repositories overlap with SWE-bench Verified and Pro? Do the gains hold when results are split by overlap?
  9. What prompt does the model see in the Table 1 oracle condition? Is it told that the inputs are failure inputs?
  10. Was ReVeal retrained on the same base model and training data? How was it implemented for repository tasks?
  11. Why does "w/o diagnostic judge reward" in Table 14 (61.6) score below Official (63.2)?
  12. In Section 5.2 / Table 5, does the external tester only generate inputs, or does it also judge outputs? Is the "Trained Qwen3.5-35B" tester the Stage 2 model? If so, it has also learned judgment, and the comparison involves more than the tester.
  13. Section 3.3 says hard negatives are "reviewed", while Appendix E says they are selected from judge-accepted probes that Claude rejected. Are humans involved, and what is the selection rule?

Suggestions (not affecting my score)

  • Competition and repository results use different checkpoints (Table 2 caption), so there is no single final model. Please state this in the abstract or contributions.
  • The text uses "earlier" and "extended" checkpoints and phrases like "earlier Pro language breakdown". Please use consistent names and explain the two checkpoints in one place.
  • Many hedging sentences in the main text ("does not identify", "does not establish") are each reasonable but together reduce readability. I suggest collecting them in the Limitations section.
  • Table 15 text says "Lowering α reduces initial correctness", but the table has no Initial column.
  • Consider discussing verifier training in SWE-Gym/R2E-Gym, repository RL methods such as SWE-RL, and work on reward hacking of LLM-as-judge in test-time verification.
4 comments

Thanks for the detailed review.

  • R1 — Verification counts and L labeling (W1/Q1/Q2/M2)

Let n11,n10,n00 count (L,CK)=(1,1),(1,0),(0,0); n01=100−n11−n10−n00. Each row samples N=100 trajectories (full CodeContests: 3,300/condition); in percent: E/N=n10, U/N=n00, success =n11+n01, P(L)=n11+n10, E/L=n10/(n11+n10) (rounded); subsamples carry sampling error, widest at small L.

R1 counts — CodeContests: Table 4 (Stage 2 = S1 SFT + RL); SV: Verified, Tables 14/16:

Conditionn11n10n00
Qwen/Official3735
Qwen/+S124382
Qwen/+S262212
GPT-OSS/Official8443
GPT-OSS/+S1502026
GPT-OSS/+S2601522
Gemma/Official491423
Gemma/+S1591118
Gemma/+S275814
SV Full (β=0.3)612414
SV No diagnostic443012
SV No penalty (β=0)45338
SV β=0.158278
SV β=0.5592313

W1 rightly asks whether lower E/L reflects less validation; stagewise, samples show the opposite. Q1's last question: unverified shares fall S1→S2 (Qwen/GPT-OSS/Gemma 38→17, 30→25, 30→17), as do L=1 failures; unverified failures 2→2, 26→22, 18→14. On Verified, full vs. no penalty: 9 fewer collapsed, 6 more unverified, 3 more successes (62 vs. 59); vs. no diagnostic: 4 more (62 vs. 58). β=0.5: coverage 82, collapsed 23, success 64; β=0.1: 65 — this sample does not rank β=0.3 highest; full Table 16 does (65.4).

Table 13 — sampled counts (N=100 each):

Reward configurationLEUE/L(%)Succ(%)
Full objective8321225.377
w/o diagnostic6431248.467
expected-output6233453.263
w/o initial-correctness7631540.864
w/o Phase-A repair shaping7333345.264
Final correctness only5437468.559

Versus final-only: coverage 54→83, collapsed 37→21, unverified 4→2, success 59→77; full tops every ablation.

Labeling L (Q2). L(τ)=1 requires a qualifying local check on the final submitted candidate plus an explicit submission (contests: explicit SUBMIT after valid local execution, Listing F.3). A qualifying check concerns issue-related behavior and executes validly — skips, collection/infrastructure failures, and unrelated/inconclusive checks never qualify (the run's judge, where present, supplies relevance/execution_validity) — and is rerun if the candidate changes. Other rubric fields enter D_agent only; timeout auto-captures and unchecked submissions give L=0. L follows each run's own labeling; final resolution uses the held-out evaluator. On 300 trajectories, extractor–human agreement is 95.7% (TP/FP/FN/TN 251/9/4/36); inter-annotator 97.3% pre-discussion.

W1's bound mixes Table 2's three-run average with the separate 165×20 Table 4 sample, so it cannot count invoked-but-unvalidated runs; the Section 2 dataset reference will be fixed (M2).

  • R2 — Components, judge updates, and extra supervision (W2/Q3/Q4)

Table 3 tests the combined package at 600 post-SFT updates (Pro 47.4 vs. 39.7, DeepSWE 26.5 vs. 15.9). Matched 200-update controls (seeds 42/43/44) share post-SFT init, schedules, budgets: full reward α=0.6/λ=0.2/β=0.3; ablations λ=0 or β=0; the frozen arm keeps the initial Qwen3.5-4B judge (diagnostic reward retained); the dynamic updates it at steps 60/120/170. Mean step-200 resolution (Pro/DeepSWE; truncated from counts): frozen 37.4/16.8, full (dynamic) 40.4/19.1, no diagnostic 36.8/14.7, no penalty 32.9/13.8 (per-seed in comment 2); full leads on every seed and benchmark (Q4).

Judge-signal audits. Instrumented evaluation rollouts: the run's final judge version scores probes at generation, logged only, never shown to the policy; evaluation stays judge-free as in the paper; eligibility: recorded C_0=0, C_K=1, D_agent=1, C_K from the benchmark evaluator. At the final extended-continuation checkpoint (200+400 updates, Table 2), 45/50 bonus-eligible trajectories from 50 issues (25/25 Pro/DeepSWE) hold genuine diagnostic evidence (95% Wilson CI 78.6–95.7; 3 all-FP, 2 unresolved; 14/14, 18/21, 13/15 by probe count). Current-policy probes, rejected included: precision/recall Pro 91.5/86.0 (TP/FP/FN/TN 86/8/14/92), DeepSWE 89.0/83.0 (73/9/15/103).

Supervision ledger (Q3). Per 600-update run, beyond the matched 16,000 H200-hour policy budget, full reward adds 240 judge GPU-hours (1.5%) and 76,000 teacher-labeled probes; final-only, none. One-time: the 4,000-probe pool (3,600/400) and 40 judge-training GPU-hours; the 50,776 labels cover the audited 400-update continuation only. Both arms share the initial distilled judge; the 2.9/2.3-point gap measures dynamic updates' incremental benefit, ongoing teacher labels included. Removing the diagnostic term costs 3.5/4.4 points, the penalty 7.4/5.3; the penalty reuses the run's judge outputs for L, adding no teacher labels. Component effects are conditional on the rest of the setup; the frozen control neither isolates the shared initial distillation nor equalizes teacher supervision vs. final-only RL.

  • R3 — Statistical uncertainty, part I

Meaning of runs and SD. In the contest three-run summaries (Tables 2, 11), Ours' three runs are independent training seeds (42/43/44), each evaluated once at seed 0; ReVeal is one retrained checkpoint per base model, evaluated three times like the inference-only baselines. On Verified (Tables 6, 7), Ours has one training seed, so every method's three runs are repeated evaluations of a fixed checkpoint. On CodeContests, each evaluation samples three trajectories per problem and averages their success outcomes (495 per run on 165 problems); these within-run samples are neither runs nor seeds. Appendix G's "one sample" is the per-call decoding setting (no best-of-n selection); the three trajectories are independent repetitions within a run. The ablation and sweep rows follow the same two protocols: Tables 13 and 15 report three-training-seed means — hence the full-objective row repeats Table 2's 73.8 — and Tables 14 and 16 report three-evaluation means of the single repository training seed, as in Table 6. The captions omitted these protocols, so W3's single-run reading was natural; the actual protocols are as above. Mean ± SD is the sample SD over the three run rates (ddof=1). The captions of Tables 2, 6, 11, and 13–16 and Appendix G will state these definitions and the CodeContests sampling setting.

Per-seed gains of full reward over each 200-update arm (sample SDs, not CIs; floor(10x)/10 display of full-precision values — recomputing from displayed values can differ by 0.1):

Full − armPro (mean±SD)DeepSWE (mean±SD)
frozen judge3.4/3.5/1.9 (2.9±0.9)2.6/2.6/1.7 (2.3±0.5)
no diagnostic5.3/3.8/1.5 (3.5±1.9)2.6/5.3/5.3 (4.4±1.5)
no penalty9.8/4.2/8.2 (7.4±2.8)5.3/7.0/3.5 (5.3±1.7)

Table 3 per-seed results (Q6; step-600 checkpoints; eval seed 0; 731/113 tasks; two-decimal truncation):

SeedPro final-onlyPro fullDeepSWE final-onlyDeepSWE full
4239.2646.5115.0424.77
4340.0848.2916.8127.43
4439.6747.3315.9227.43

These reproduce Table 3's rounded means and gains; the DeepSWE gains are 11/12/13 tasks, positive for every training seed at evaluation seed 0. Tables 6/11's SDs describe their own protocols, not how Table 3's gains move with the evaluation seed; the next check addresses that.

Additional evaluation seeds (Q6). We re-evaluated the same step-600 Table 3 checkpoints (both arms) at evaluation seeds 1 and 2, task sets and budgets fixed, evaluation seeds shared between arms (seed 0 above); entries are success counts:

Training seedEvaluation seedPro final-onlyPro fullDeepSWE final-onlyDeepSWE full
421291/731338/73119/11329/113
422280/731343/73116/11328/113
431288/731342/73120/11327/113
432298/731350/73118/11333/113
441286/731342/73121/11330/113
442294/731349/73117/11331/113

Mean gains at evaluation seeds 0/1/2 are 7.7/7.2/7.8 pp on Pro and 10.6/7.7/12.1 pp on DeepSWE (rounded); full reward wins for every training/evaluation seed pair.

Table 7 (Q5). Paired counts per run (N=500); three direct evaluation runs per fixed checkpoint (Ours: the single-training-seed extended one); run labels index the paired records:

ModelRunBothOnly OursOnly OfficialNeither
Qwen12865621137
Qwen22955422129
Qwen32995725119
GPT-OSS12175426203
GPT-OSS22255328194
GPT-OSS32325732179
Gemma11915623230
Gemma22025422222
Gemma32115226211

Per-run rates: (Both+Only Ours)/500 and (Both+Only Official)/500; statistics use full precision, displayed floor(10x)/10, recovering Table 6. Pairing keeps task identity: per-task Ours−Official outcomes are averaged and the 500 task vectors resampled (10,000 replicates, RNG seed 0/model, 2.5/97.5 percentiles, linear interpolation). Pre-display endpoints [3.265, 10.0667], [1.800, 8.6667], [2.600, 9.400] share displayed half-width 3.4; Gemma is symmetric around 6.0, not 6.0667. Positive replicates: 9999/9987/9998 of 10,000 (99.87%→99.8%). With one training seed, the bootstrap reflects task-level variability only. Paired records and code: released on acceptance.

Q11: 61.6 and 63.2 are three-evaluation means of the single earlier-budget run and the untrained policy; given Table 6's run SDs (1.7–2.3), the 1.6-point gap is within evaluation noise rather than evidence that the pipeline hurts. The term's contribution rests on the matched ablations. The 200-update budget went to the contested terms (λ, β); α is evidenced by the contest ablation (initial 58.6→51.4, final 73.8→69.2) and the §3.3 boundary analysis. M4: the two 54.8 entries differ — Table 13's Initial is the trained final-only arm's first candidate, Table 2's Official Pass the untrained policy's final submission. S4: the initial-correctness sentence will be dropped (Table 15 has no Initial column).

  • R3 — Statistical uncertainty, part II: per-run counts and paired comparisons (W3)

Per-run success counts (W3). Run definitions as in comment 2 — CodeContests: three trajectories per problem, 495 per run; LiveCodeBench: one per problem, 1,055. Mean and sample SD (ddof=1) from full-precision run rates, rounded to one decimal:

ModelBenchmarkMethodRun 1Run 2Run 3Mean ± sample SD (%)
GPT-OSSCodeContestsOfficial261/495249/495251/49551.2 ± 1.3
GPT-OSSCodeContestsRevisit284/495297/495310/49560.0 ± 2.6
GPT-OSSCodeContestsOurs300/495310/495326/49563.0 ± 2.6
GemmaCodeContestsOfficial348/495332/495329/49567.9 ± 2.1
GemmaCodeContestsRevisit356/495344/495345/49570.4 ± 1.3
GemmaCodeContestsOurs370/495366/495348/49573.0 ± 2.4
GPT-OSSLiveCodeBenchOfficial723/1055741/1055724/105569.1 ± 1.0
GPT-OSSLiveCodeBenchRevisit793/1055804/1055807/105576.0 ± 0.7
GPT-OSSLiveCodeBenchOurs841/1055858/1055852/105580.6 ± 0.8
GemmaLiveCodeBenchOfficial877/1055890/1055886/105583.8 ± 0.6
GemmaLiveCodeBenchRevisit899/1055911/1055912/105586.0 ± 0.7
GemmaLiveCodeBenchOurs929/1055942/1055940/105588.8 ± 0.7

Repository paired comparisons (W3). Measured paired outcomes against the stated baseline — Official for GPT-OSS/Pro (Table 2's headline gain), the matched final-only control elsewhere (Gemma/DeepSWE is the 27-versus-18 comparison W3 cites); gains (Only Ours − Only baseline)/N, truncated; 95% task-bootstrap intervals (10,000 replicates, seed 0 per row; endpoints rounded); single training seed.

ModelBenchmarkBaselineNBothOnly OursOnly baselineNeitherGain (pp)95% CI (pp)
GPT-OSSProOfficial73118010034417+9.0[6.0, 12.0]
GPT-OSSDeepSWEFinal-only113915683+7.9[0.0, 15.9]
GemmaProFinal-only7312178334397+6.7[3.8, 9.6]
GemmaDeepSWEFinal-only1131017878+7.9[-0.9, 16.8]

Table 2 consistency. The marginals reproduce Table 2 (GPT-OSS Pro 280 vs. 214, DeepSWE 24 vs. 15; Gemma Pro 300 vs. 251, DeepSWE 27 vs. 18); Table 2 rounds to one decimal, so three of these rates sit 0.1 above their truncated values. The Pro intervals exclude zero; GPT-OSS/DeepSWE touches zero at the lower endpoint and Gemma/DeepSWE spans zero.

  • R4 — Baselines and interfaces (W5/Q10/Q12/Q13/M3/M5/S5)

The trained contest control is Table 13's final-only arm, 62.7% vs. 73.8% (to be foregrounded). Inference-time baselines' call budgets exclude training cost; Appendix G's ≈8,512 H200 GPU-hours cover the full earlier pipeline (SFT, RL, repository execution, evaluation), contest training being one part. CodeT stays higher on Qwen (75.4/88.5 vs. 73.8/87.2) at its 20-call budget; reranking on the trained policy is untested, plausibly complementary. Self-Edit/Revisit keep adaptation labels; Adv. Prompt's 38.0% is the earlier-checkpoint comparison. ReVeal (Q10) was retrained per base model with its published method and data from its released pipeline — see its TACO preprocessing script building files from BAAI/TACO — (Tables 2, 6; repository rows share the OpenHands protocol); the fully matched control (base, data, budget) remains the final-only arm. SWE-Gym/R2E-Gym-trained agents differ in scaffold, base model, and training data, so side-by-side numbers are not controlled comparisons; S5 will discuss these plus SWE-RL, EvilGenie, CHERRL. The judge scores probe-level issue violations with repair independently evaluated; the audits and frozen-judge control address, not eliminate, reward exploitation.

Table 5 (Q12): Listing F.2 requests only stdin inputs; the executor returns actual outputs without verdicts; Listing F.3 asks for interpretation and a FIX/MORE TESTS/SUBMIT choice. The tester model handles both prompts in each coder block, and Trained Qwen3.5-35B is the fully trained (Stage 2) checkpoint — as the review notes, substitution changes input generation and output judgment together; only the coder's repair stays fixed; the caption will state both. Q13: humans enter only at screening: an eligible hard negative is a judge-accepted probe whose six-field Claude annotation aggregates to negative (415–468 candidates per audited window; each update takes 400, plus 400 replayed positives, no extra teacher requests); the selection rule and screening protocol will be specified. M5: evaluation uses the training-time frozen-initial-patch protocol (C_0 recorded identically); the revision will state this.

M3: intended — with α=0.6, (C0,CK)=(1,0) gets −0.4 versus −1 for (0,0): the term rewards first-candidate correctness even when repair fails, a preserved correct candidate earns 1.6, and the bonus vanishes on final failure. Phase B optimizes Eq. (3) only (reference comparisons stay offline); the collapse penalty belongs to Eq. (5); both will be stated.

  • R5 — Data overlap and repository generalization (W4/W7/Q7/Q8)

Exclusion removes exact task-ID and space-stripped exact issue/problem-text matches; sharing a repository alone is allowed. To examine transfer beyond the Python training repositories, we report Pro gains on Go/JavaScript (below), non-Python DeepSWE successes rising 5/78→20/78, and LiveCodeBench gains from the same contest checkpoints (+14.2/+11.5/+5.0) — supporting cross-language and cross-benchmark transfer without establishing absence of paraphrase or shared-solution overlap.

Provenance (Q7/Q8): LiveCodeBench release_v6 (1,055 problems, May 2023–Apr 2025; version list). All compared methods, Official and baselines included, share the same base models, so pretraining exposure is common across arms; it can inflate absolute rates but is held constant in each comparison. The public DeepSWE manifest pins 113 task IDs (revision 6d6f134…, five languages); the revision adds evaluated snapshots and per-task language labels; the training-repository list follows, and shared-repository Verified/Pro splits follow once per-task intersections are compiled; manifests ship at release. Teacher: Claude labels only probe quality; judge inputs exclude gold patches and hidden verdicts, final resolution comes solely from the benchmark evaluator; probe-quality judgments are audited against blinded human labels (Table 12) and static splits (Table 17).

W7: Table 8's Pro breakdown (task-level paired intervals; 10,000 replicates, seed 0, one training seed): Python +15.4 [9.8, 21.1] (N=266), Go +8.9 [4.3, 13.6] (280), JavaScript +9.1 [3.0, 15.8] (165), TypeScript 0.0 [−25.0, 25.0] (20). Python's gain is largest; transfer could tap base-model capability in other languages and difficulty could matter; mechanisms are not isolated. The requested DeepSWE breakdown (extended Table 2 checkpoint; Official→Ours, pp gains, truncated): Python 5/35→12/35 (+20.0), Go 2/29→9/29 (+24.1), TypeScript 2/27→6/27 (+14.8), JavaScript 1/14→4/14 (+21.4), Rust 0/8→1/8 (+12.5); totals match Table 2 (10/113→32/113; +19.4 truncated, 19.5 from Table 2's displayed rates). Non-Python successes rise 5/78→20/78 (6.4%→25.6%). Small groups (Rust: 8) warrant caution.

  • R6 — Oracle inputs, evidence use, and Figure 1 (W6/Q9)

The Table 1/Figure 1 diagnostic study evaluates all 12,898 TACO-verified problems with untrained policies, so overlap with the later training set does not bear on these observations. Table 1 is motivating: Oracle assistance recovers 44.9–73.0% of collapsed cases; separating the input's diagnostic content from a possible selection cue would need the suggested uniform-input control.

Oracle prompt (Q9). Benchmark failure inputs replace the self-generated ones; the candidate runs on them and the observations enter Listing F.3 (problem, code, {execution_results}): "Analyze whether the outputs are correct based on the problem specification," then FIX/MORE TESTS/SUBMIT. F.3's "If you found incorrect output(s)" is a conditional instruction, not a supplied verdict; the Oracle condition neither marks inputs as failures nor supplies reference outputs or correctness labels.

A repository trace-redaction intervention — same incorrect candidate; policy, prior history, environment state, and remaining budgets fixed — compares intact diagnostic output, its redaction, a format-matched non-diagnostic replacement, and a neutral-reformatting sham (no hint); 60 issues × 3 paired seeds (180/condition), in order: 62.2%/51.1%/48.9%/59.4%; paired 95% intervals [2.1, 19.8] and [4.5, 22.3] exclude zero, sham [−6.0, 9.5] spans it.

Figure 1 (W6). iter0 is the initial candidate before any repair; iter1+ covers later repair iterations — stages, not test-call counts. Figure 1(a) is stage-conditioned, Table 1 is the all-instance fraction; the caption will state each stage's population and repeated-observation rule. Figure 1(b): normalized token-level Levenshtein similarity between consecutive patches among collapse samples (n<5 hollow; 4/2-sample models omitted) — edit similarity, not equivalence or causation.

  • R7 — Presentation, clarifications, and reproducibility (M1/M6/S1–S3)

The abstract will use absolute gains (+19.0 CodeContests, +14.2 LiveCodeBench; 34.7%/19.5% are relative) and note the different contest/repository checkpoints (S1), named consistently (S2): contest policy; earlier repository policy (step 200); extended continuation (+400; Table 2); post-SFT comparison (600; Table 3); rebuttal controls (200). Hedging consolidates in the Limitations (S3). M6: code and data are released upon acceptance — training/evaluation/scoring code, configs and prompts, task manifests, teacher labels, and audit annotations; judge weights are excluded, but the released label pool, recipe, and update protocol allow retraining.

Reviewer_13YeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies Fix–Test Collapse, where a code agent considers its local verification successful but submits code that fails hidden tests. It distinguishes tests that miss bugs from incorrect interpretations of outputs that expose them, and proposes a training curriculum combining SFT, reinforcement learning over individual actions, and optimization of complete trajectories. Diagnostic rewards come from reference executions for contest problems and an issue based judge for repository tasks. Experiments across three model families show improvements in correctness and verification reliability. Most notably, training from the same SFT checkpoint improves over final success rewards alone by 7.7 percentage points on SWE-bench Pro and 10.6 points on DeepSWE, averaged across three paired training seeds.

Critical strengths
  1. The controlled training comparison provides convincing evidence for the method. Table 3 starts both conditions from the same SFT checkpoint and matches the policy and environment budgets across three paired training seeds. Improvements are positive for every seed, ranging from 7.3 to 8.2 percentage points on Pro and from 9.7 to 11.5 points on DeepSWE. This directly supports the value of the proposed reward scheme beyond additional training with final success supervision.

  2. Separating weak test inputs from incorrect output judgments identifies distinct bottlenecks that overall success rates would obscure. The method then addresses test generation, interpretation, repair, and stopping decisions through targeted training. This connection between diagnosis and intervention makes the paper useful for both understanding verification failures and designing better agents.

Critical weaknesses
  1. Changes in verification coverage are not fully reported. Tables 4 and 14 provide test invocation rates and conditional collapse rates, but do not give the corresponding local validation coverage and all-instance collapse counts. This makes it difficult to assess how much of the conditional improvement reflects better verification versus a change in which submissions the agent validates. The distinction matters because the collapse penalty applies only to failures that the agent has locally accepted.

  2. The downstream benefit of updating the judge is not directly isolated. The audits show that accepted probes are more often valid immediately after an update, which supports the calibration procedure. However, there is no matched comparison between a frozen judge and an updated judge using the same reward scheme. Removing the diagnostic reward in Table 14 answers a different question. Since dynamic updating is a central part of the repository method, its contribution to final repair performance deserves clearer evidence.

Questions for authors
  1. Could you provide local validation coverage (P(L=1)), all-instance collapse incidence (P(E=1)), and unverified failure incidence (P(C_K=0,L=0)), with the underlying counts? The full reward and no collapse penalty conditions in Table 14 would be particularly useful. These results would clarify whether the penalty improves verification reliability while preserving coverage.

  2. Have you compared a frozen judge with the dynamically updated judge while keeping the policy initialization, reward coefficients, and training budget fixed? If such results are available, how do final resolution and verification reliability differ? This comparison would materially affect my assessment of the dynamic judge’s contribution.

1 comment

Thank you for the careful review — the requested matched judge comparison and sampled event counts follow.

  • R1 — Validation coverage and failure composition (W1/Q1)

Per condition (Tables 4 and 14) we sampled 100 trajectories: L locally validated submissions, E those failing the held-out evaluator, U failures without qualifying validation. These estimate Q1's three probabilities; with N=100 counts equal L/N, E/N, U/N in percent, success is (N−E−U)/N, joint counts (L−E, E, N−L−U, U), E/L rounded; subsamples carry sampling error vs. the full tables, widest in small-L cells. Self-test invocation is a weaker event than qualifying validation.

R1a (Table 4; original 3,300/condition), CodeContests; Stage 2 = S1 SFT + RL:

ModelTrainLEUE/L(%)Succ(%)
QwenOfficial1073570.058
Qwen+S16238261.360
Qwen+S28321225.377
GPT-OSSOfficial1244333.353
GPT-OSS+S170202628.654
GPT-OSS+S275152220.063
GemmaOfficial63142322.263
Gemma+S170111815.771
Gemma+S2838149.678

R1b (Table 14), SWE-bench Verified, Qwen3.5-35B:

ConfigLEUE/L(%)Succ(%)
Full reward85241428.262
No diagnostic reward74301240.558
No collapse penalty7833842.359

From Stage 1 to Stage 2, coverage rises (Qwen 62→83%, GPT-OSS 70→75%, Gemma 70→83%), L=1 failures fall (38→21, 20→15, 11→8), and unverified failures are 2→2, 26→22, 18→14. Against Official, Qwen and GPT-OSS still show higher E/N despite lower E/L as coverage expands.

On the comparison Q1 highlights: under the respective judges, the collapse penalty brings higher coverage (78%→85%), fewer collapsed failures (33→24), and lower conditional collapse (42.3%→28.2%); unverified failures rise 8→14 — part of the E drop shifts to L=0, a risk the Limitations note — while successes rise 59→62.

Labeling L. L(τ)=1 requires a qualifying local check on the final submitted candidate plus an explicit submission. A qualifying check concerns issue-related behavior and executes validly — skips, collection/infrastructure failures, and unrelated/inconclusive checks never qualify (where present, the run's judge supplies relevance and execution_validity) — and is rerun if the candidate changes. Other rubric fields enter D_agent, not L, so a valid test that misses the bug can still qualify; timeout auto-captures and unchecked submissions give L=0. Contest acceptance is the explicit SUBMIT after valid local execution (Listing F.3); "pass" is the agent's own reading, so L=1 can coexist with a wrong answer. L follows each run's own labeling; final resolution uses the common held-out evaluator. On a new 300-trajectory sample, extractor–human agreement is 95.7% (TP/FP/FN/TN 251/9/4/36), inter-annotator 97.3% pre-discussion; 12 ambiguous cases ended 8/4.

  • R2 — Frozen vs. dynamically updated judge (W2/Q2)

Both arms start from the same protocol-SFT Qwen3.5-35B checkpoint with matched task schedules and policy/environment budgets, use Eq. 5 defaults (α=0.6, λ=0.2, β=0.3), and run 200 policy updates. The frozen arm keeps the initial Qwen3.5-4B rubric judge; the dynamic arm updates it after steps 60/120/170 (Sec. 3.3, App. E), j0–j3 covering 1–60/61–120/121–170/171–200; each update adds 400 new hard negatives and 400 replayed positives, and rollouts are never rescored. Only the update procedure varies; both arms keep the diagnostic reward — as the review notes, the Table 14 ablation answers a different question. Held-out resolution at the final step-200 checkpoints (731 Pro / 113 DeepSWE tasks; full-precision rates and gains truncated to one decimal):

SeedPro frozenPro dynamicΔ(pp)DeepSWE frozenDeepSWE dynamicΔ(pp)
4238.341.7+3.416.819.4+2.6
4335.939.5+3.515.918.5+2.6
4438.039.9+1.917.619.4+1.7
Mean37.440.4+2.916.819.1+2.3

Dynamic updates are intended to keep the judge calibrated to the probes the evolving policy produces. Within these 200 updates they improve mean held-out resolution by 2.9 (Pro) and 2.3 (DeepSWE) points, positive for every seed on both benchmarks; gains are computed before truncation, not by subtracting the displayed means.

Verification behavior. 100 sampled trajectories per benchmark and judge condition, seed-42 step-200 checkpoints (E/L rounded):

BenchmarkJudgeLEUE/L(%)Succ(%)
ProFrozen81501261.738
ProDynamic85471255.341
DeepSWEFrozen76622081.618
DeepSWEDynamic80592073.821

The dynamic arm has higher labeled coverage on both benchmarks, 3 fewer L=1 failures per 100, unchanged unverified counts, and lower conditional collapse; the resolution table separately shows a positive repair gain under the common held-out evaluator. These counts and the L protocol will go into the appendix.

Reviewer_hKN5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper defines Fix–Test Collapse as submission after successful local validation followed by failure on the held-out evaluator, and distinguishes weak tests from output-judgment errors. Falsification-guided training combines protocol SFT, action-level RL, and trajectory rewards. Repository training uses a dynamically updated diagnostic judge, while final resolution is evaluated independently by the held-out benchmark. In the matched post-SFT comparison, both arms receive 600 repository updates with matched policy/environment budgets; the full training package improves Pro by 7.7 percentage points and DeepSWE by 10.6 points on average, with positive increments across three paired training seeds (Table 3).

Critical strengths
  1. The matched post-SFT control is a strong part of the evidence. Starting from the same protocol-SFT checkpoint, both arms receive 600 repository updates under matched policy and environment budgets. The full falsification-training package improves Pro resolution by 7.7 percentage points and DeepSWE resolution by 10.6 points on average, with positive increments for all three paired training seeds (Table 3). This supports the effectiveness of the combined reward and judge package, while leaving its individual components to be assessed separately.

  2. The paper carefully separates self-test invocation, local-validation coverage, and conditional collapse, and gives explicit final-candidate attribution rules and an offline/policy-visible information boundary. The supplied judge prompts, static confusion counts, and blinded on-policy audits make the approximate repository reward substantially more inspectable. The audits also expose residual reward gaming rather than treating judge agreement as ground truth.

Critical weaknesses
  1. The on-policy audits assess accepted probes, whereas the diagnostic reward uses the maximum verdict over all probes in a trajectory (Appendix G). These probe-level results do not establish how often a rewarded trajectory lacks any genuinely diagnostic evidence, particularly as the number of probes changes. The small, single-seed audits and the absence of a fixed-judge control leave the trajectory-level reliability and the incremental contribution of dynamic updates uncertain. This limits the interpretation of the training signal; final resolution is evaluated independently by the held-out benchmark, without invoking the judge.

  2. The reduction in conditional collapse is harder to interpret without local-validation coverage and unverified-failure counts for the same policies. Moreover, the diagnostic bonus requires the co-occurrence of discovered evidence and successful repair, so it does not establish that the repair used that evidence, as the authors acknowledge. Complete event counts and a matched trace-redaction or evidence-replacement intervention would clarify these verification and mechanism claims. These gaps do not by themselves invalidate the independently evaluated resolution gains.

Questions for authors
  1. For the late-stage training policies, can you audit a held-out sample of trajectories receiving the diagnostic bonus and report how often at least one judge-positive probe is genuinely diagnostic, including the fraction whose positive probes are all false positives? Please stratify by probe count and account for repeated issues. Do you have a matched fixed-judge control, or other evidence separating the contribution of dynamic updates from the rest of the reward package?

  2. Alongside Tables 4, 14, and 16, can you report N, L, E, local-validation coverage L/N, all-instance collapse incidence E/N, and the number of unverified failures for each condition? Please also specify how L is labeled from the trajectory and how ambiguous cases are checked. These counts would show how much of the change in E/L comes from fewer falsely validated failures versus a changed population of locally validated submissions.

  3. Have you evaluated the trace-redaction intervention proposed in the Limitations, or a matched evidence-replacement control, starting from the same incorrect candidate and keeping the remaining repair budget fixed? Reporting the change in repair success when the diagnostic output is removed or replaced would help distinguish actual use of falsifying evidence from the co-occurrence rewarded by Eq. (3). If this experiment is unavailable, which mechanism claims should be limited to association?

1 comment

Thank you for the review — the requested audits and controls follow.

  • R1 — Trajectory-level reliability and dynamic judge (W1/Q1)

We audited 50 bonus-eligible trajectories from 50 distinct issues (25 Pro + 25 DeepSWE) at the final extended-continuation Qwen3.5-35B checkpoint (200+400 updates; Table 2); the bonus event is the probe-max D_agent (Eq. 5). These are instrumented evaluation rollouts: the run's final judge version scores probes at generation, logged only, never shown to the policy; evaluation stays judge-free as in the paper; eligibility uses the recorded fields C_0=0, C_K=1, D_agent=1; C_K comes from the benchmark evaluator. Genuinely diagnostic (App. F.2): runs validly, checks a concrete issue requirement, and exposes an issue-specific violation a correct fix should resolve. A trajectory is supported if ≥1 judge-positive probe is genuinely diagnostic, all-false-positive if none is, else unresolved; all stay in the denominator:

ProbesAuditedSupported95% Wilson CI (%)All-FPUnresolved
1–31414 (100.0%)78.5–100.000
4–72118 (85.7%)65.4–95.021
≥81513 (86.7%)62.1–96.311
All5045 (90.0%)78.6–95.732

Fixed-judge control. Matched post-SFT init, rewards, schedules, budgets (seeds 42/43/44); 200 early-schedule steps (dynamic-arm judge updates 60/120/170). Mean held-out resolution, dynamic/frozen: Pro 40.4/37.4%, DeepSWE 19.1/16.8% (+2.9/+2.3 pp, pre-truncation values).

  • R2 — Event counts for Tables 4/14/16; labeling L (W2/Q2)

Per condition we sampled 100 trajectories: L locally validated submissions, E those failing held-out, U failures without qualifying validation; with N=100 these are L/N, E/N, U/N in percent (E/L rounded). Subsamples carry sampling error vs. the full tables, widest in small-L cells.

R2a (Table 4; original 3,300/condition), CodeContests; Stage 2 = S1 SFT + RL:

ModelTrainLEUE/L(%)Succ(%)
QwenOfficial1073570.058
Qwen+S16238261.360
Qwen+S28321225.377
GPT-OSSOfficial1244333.353
GPT-OSS+S170202628.654
GPT-OSS+S275152220.063
GemmaOfficial63142322.263
Gemma+S170111815.771
Gemma+S2838149.678

R2b (Tables 14/16), SWE-bench Verified:

ConfigLEUE/L(%)Succ(%)
Full reward (β=0.3)85241428.262
No diagnostic reward74301240.558
No collapse penalty (β=0)7833842.359
β=0.18527831.865
β=0.582231328.064

From Stage 1 to Stage 2, coverage rises (Qwen 62→83%, GPT-OSS 70→75%, Gemma 70→83%), L=1 failures fall (38→21, 20→15, 11→8), and unverified failures are 2→2, 26→22, 18→14. Against Official, Qwen and GPT-OSS still show higher E/N despite lower E/L as coverage expands. On Verified, full vs. no collapse penalty: 9 fewer collapsed failures, 6 more unverified, 3 more successes (62% vs. 59%); vs. no diagnostic: 4 more successes (62% vs. 58%). β=0.5: coverage 82, collapsed 23, success 64; β=0.1: 65 — this sample does not rank β=0.3 highest; the full Table 16 does (65.4).

Labeling L. L(τ)=1 requires a qualifying local check on the final submitted candidate plus an explicit submission. A qualifying check concerns issue-related behavior and executes validly — skips, collection/infrastructure failures, and unrelated/inconclusive checks never qualify (where present, the run's judge supplies relevance and execution_validity) — and is rerun if the candidate changes. Other rubric fields enter D_agent, not L, so a valid test that misses the bug can still qualify; timeout auto-captures and unchecked submissions give L=0. Contest acceptance is the explicit SUBMIT after valid local execution (Listing F.3); "pass" is the agent's own reading, so L=1 can coexist with a wrong answer. L follows each run's own labeling; final resolution uses the common held-out evaluator. On a new 300-trajectory sample, extractor–human agreement is 95.7% (TP/FP/FN/TN 251/9/4/36), inter-annotator 97.3% pre-discussion; 12 ambiguous cases ended 8/4.

  • R3 — Redaction/replacement intervention (W2/Q3)

From the same incorrect candidate — policy, prior history, environment state, and remaining generation/tool budgets fixed — we compared intact diagnostic output, redaction, a format-matched non-diagnostic replacement, and a sham (neutral reformatting; byte-for-byte recoverable, no hint, exit status preserved) — 60 prefixes from 60 issues (30 Pro + 30 DeepSWE) × 3 paired seeds, all four conditions per pair:

ConditionSuccess%
Intact112/18062.2
Redacted92/18051.1
Replaced88/18048.9
Sham107/18059.4

Redaction costs 11.1 points, replacement 13.3, reformatting 2.8. Issue-level paired 95% intervals: intact−redacted [2.1, 19.8], intact−replaced [4.5, 22.3], intact−sham [−6.0, 9.5] — the first two exclude zero, the sham interval spans it. This supports a contribution of diagnostic content beyond rewarded co-occurrence.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We first apologize to the reviewers and the AC for the late response: several of the requested experiments required long training runs, and we waited to post verified numbers.

We thank all reviewers for their careful and constructive reviews, and we are encouraged by what they found valuable: the Fix–Test Collapse framing and the weak-input/judgment-error decomposition (13Ye, e7Mt, oJAE), the metric definitions and the policy/scorer information boundary (hKN5, e7Mt, oJAE), the matched post-SFT control with positive gains on all three paired seeds (hKN5, 13Ye, e7Mt), and the audits that report residual reward gaming openly (hKN5, e7Mt).

The reviews converge on four shared concerns, each now addressed with new matched experiments:

1. Could lower conditional collapse come from doing less local validation? (hKN5 W2, 13Ye W1, e7Mt W1, oJAE Q1.) We report complete (L, C_K) contingency counts for every condition of Tables 4, 13, 14, and 16. Validation coverage rises with training while locally validated failures fall, so the conditional improvement is not a shrinking-denominator effect; the one adverse shift (more unverified failures under the penalty on Verified) is reported as-is. The L-extraction protocol is specified and agrees with human labels on 95.7% of 300 trajectories.

2. Trajectory-level reliability of the diagnostic reward, and the dynamic judge's contribution. (hKN5 W1, 13Ye W2, e7Mt W2, oJAE W2.) At the final checkpoint, 45/50 audited bonus-eligible trajectories contain genuinely diagnostic evidence (90.0%; Wilson 95% CI 78.6–95.7%). A new frozen-vs-dynamic judge control shows dynamic updating adds +2.9 (Pro) / +2.3 (DeepSWE) points, positive for every seed; judge recall on current-policy probes is 86.0/83.0%.

3. Does the agent actually use the falsifying evidence? (hKN5 Q3, oJAE W3.) The requested trace-redaction intervention (60 issues × 3 seeds × 4 conditions, matched prefixes and budgets) shows redacting or replacing the diagnostic output costs 11.1/13.3 points versus 2.8 for a formatting-only sham; the redaction and replacement paired intervals exclude zero, the sham interval spans it.

4. Component attribution and statistical uncertainty. (e7Mt W2/W3, oJAE W3/Q3.) Under matched 200-update Pro/DeepSWE ablations, removing the diagnostic term costs 3.5/4.4 points and the collapse penalty 7.4/5.3, with full reward ahead for every seed. We clarified the run/seed protocol of every table, reported per-run and per-seed counts, and re-evaluated Table 3 at two further evaluation seeds (mean gains 7.7/7.2/7.8 on Pro, 10.6/7.7/12.1 on DeepSWE; full reward wins every pairing); Table 7's paired counts and bootstrap procedure are given in full.

We also specified data provenance and the falsification arm's supervision/cost ledger (e7Mt W4, oJAE Q4); task manifests with checksums ship with the code-and-data release on acceptance.

All new experiments and counts will be incorporated into the revision, together with the requested presentation fixes. For each reviewer's individual questions, please see our separate point-by-point responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.