- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies agents that accept their own local verification but fail held-out evaluation. It separates weak test inputs from misinterpretation of outputs that expose an error, and proposes a training curriculum combining protocol SFT, action-level RL, and full-trajectory RL. Contest tasks use offline reference executions to reward diagnostic inputs, while repository tasks use a dynamically updated rubric judge. Reference outputs and hidden verdicts are withheld from the acting policy. The extended Qwen configuration reaches 48.3% on 731 SWE-bench Pro tasks and 32/113 on DeepSWE. A separate, stronger causal control starts both arms from the same SFT checkpoint and reports mean improvements of 7.7 and 10.6 percentage points over final-only RL across three paired training seeds.
Critical strengths
- The distinction between producing a diagnostic test and correctly interpreting its output is useful. The operational definition of collapse also carefully separates conditional error among locally validated submissions from all-instance failure and unverified submission.
- Offline references and final evaluators supply training supervision without being exposed as online feedback, and the repository initial-patch boundary is fixed independently of when the agent chooses to test. These details make the setting substantially easier to interpret.
Critical weaknesses
-
The dynamic judge's contribution and trajectory-level reliability remain incompletely isolated. The audits show accepted-probe precision of 80.6% before updates and 94.3% immediately afterward, but each boundary samples only 20 accepted probes, and the final pre-update sample has 15 valid probes out of 20. These results support mitigation, as the authors state, rather than sustained calibration. Since D_agent is positive if any probe is accepted, probe-level precision does not directly establish how often an entire rewarded trajectory contains genuinely diagnostic evidence. A fixed-judge versus dynamic-judge control, along with an audit of the trajectory-level diagnostic event and an estimate of false negatives on current policy probes, would clarify the value of adaptation.
-
The most convincing treatment comparison establishes a package effect. Table 3 changes initial-correctness shaping, diagnostic reward, the collapse penalty, and the associated judge machinery together. Appendix H offers useful component ablations, but repository ablations use the earlier budget on SWE-bench Verified rather than the Pro/DeepSWE post-SFT setting carrying the strongest claims. A focused ablation in that setting would strengthen attribution. Likewise, rewarding diagnostic discovery followed by successful repair does not show that the agent used the diagnostic trace; a trace-redaction or replacement intervention would distinguish evidence-driven repair from extra testing accompanying better repair.
Questions for authors
- Can the authors provide the complete verification/correctness contingency counts and validate the extraction of local-validation status?
- How often is the trajectory-level diagnostic bonus supported by at least one genuinely diagnostic probe, and what is the effect of freezing the judge?
- Do the diagnostic bonus and collapse penalty retain their benefits individually under the matched Pro/DeepSWE post-SFT protocol?
- What are the exact task manifests, near-duplicate exclusions, and additional judge/teacher costs?
3 comments
- R2 — Trajectory-level support, the frozen-judge control, and false negatives (W2/Q2)
As the review notes, accepted-probe precision differs from whether a bonus-eligible trajectory holds ≥1 genuinely diagnostic probe. We audited 50 bonus-eligible trajectories from 50 distinct issues (25 Pro + 25 DeepSWE, one per issue; final extended-continuation Qwen3.5-35B checkpoint, 200+400 updates, Table 2). These are instrumented evaluation rollouts: the run's final judge version scores probes at generation, logged only, never shown to the policy; evaluation stays judge-free as in the paper; the audit conditions on the recorded eligibility fields (C_0=0, C_K=1, D_agent=1); C_K comes from the benchmark evaluator, C_0 from the training-time frozen-initial-patch protocol. A probe counts only if it runs validly, checks a concrete issue requirement, and exposes an issue-specific violation a correct fix should resolve. Genuine evidence supports 45/50 audited trajectories (95% Wilson CI 78.6–95.7%); 3 all-false-positive, 2 unresolved, all in the denominator. By probe count: 14/14 at 1–3, 18/21 at 4–7, 13/15 at ≥8.
Frozen vs. dynamic judge. Both arms share the protocol-SFT Qwen3.5-35B start, task schedules, budgets, and Eq. 5 coefficients (α=0.6, λ=0.2, β=0.3); the dynamic arm updates the initial Qwen3.5-4B judge after steps 60/120/170 (400 new hard negatives + 400 replayed positives each; rollouts never rescored). Cells: seed-42/43/44 rates (731 Pro / 113 DeepSWE tasks); bold means and differences computed from counts, truncated to one decimal; the two R3 ablations included:
| Arm | Pro s42/s43/s44 (mean) | DeepSWE s42/s43/s44 (mean) |
|---|---|---|
| Frozen judge | 38.3; 35.9; 38.0; 37.4 | 16.8; 15.9; 17.6; 16.8 |
| Full (dynamic judge) | 41.7; 39.5; 39.9; 40.4 | 19.4; 18.5; 19.4; 19.1 |
| No diagnostic | 36.3; 35.7; 38.4; 36.8 | 16.8; 13.2; 14.1; 14.7 |
| No collapse penalty | 31.8; 35.2; 31.7; 32.9 | 14.1; 11.5; 15.9; 13.8 |
Dynamic updating beats the frozen judge by +2.9 (Pro) / +2.3 (DeepSWE) points, positive for every seed (DeepSWE: 2–3 tasks/seed). Local-validation counts (seed-42 step-200 checkpoints, 100 per row; n11,n10,n00 count (L,CK)=(1,1),(1,0),(0,0), n01 the rest; E/L rounded):
| Condition | n11 | n10 | n01 | n00 | E/L(%) | Succ(%) |
|---|---|---|---|---|---|---|
| Pro/Frozen | 31 | 50 | 7 | 12 | 61.7 | 38 |
| Pro/Full reward | 38 | 47 | 3 | 12 | 55.3 | 41 |
| Pro/No diagnostic | 32 | 51 | 5 | 12 | 61.4 | 37 |
| Pro/No penalty | 25 | 59 | 8 | 8 | 70.2 | 33 |
| DeepSWE/Frozen | 14 | 62 | 4 | 20 | 81.6 | 18 |
| DeepSWE/Full reward | 21 | 59 | 0 | 20 | 73.8 | 21 |
| DeepSWE/No diagnostic | 14 | 65 | 2 | 19 | 82.3 | 16 |
| DeepSWE/No penalty | 13 | 65 | 1 | 21 | 83.3 | 14 |
The dynamic arm has wider labeled coverage, 3 fewer L=1 failures per 100 on each benchmark, and unchanged unverified counts.
False negatives. The trajectory audit cannot see probes the judge rejects, so we also labeled 200 current-policy probes per benchmark, judge-negative included. Judge precision/recall: Pro 91.5/86.0 (TP/FP/FN/TN 86/8/14/92), DeepSWE 89.0/83.0 (73/9/15/103); pooled recall 84.6%. The missed 14/100 and 15/88 diagnostic probes quantify evidence the judge rejects.
- R3 — Components under the matched setting, and evidence use (W3/Q3)
As W3 notes, Table 3 establishes a package effect (Pro 47.4% vs. 39.7%, DeepSWE 26.5% vs. 15.9% at 600 updates). The 200-update ablations (λ=0 or β=0; everything else as in the judge control, same seeds) are the ablation rows above, with per-seed resolution and seed-42 counts. Removing the diagnostic term costs 3.5 (Pro) / 4.4 (DeepSWE) points on the means, the collapse penalty 7.4 / 5.3; full reward beats each ablation for every seed on both benchmarks. Behaviorally, full reward has the lowest labeled conditional collapse, widest coverage, and highest sample success; versus no-penalty, 12 fewer collapsed vs 4 more unverified on Pro, 6 fewer collapsed and 1 fewer unverified on DeepSWE.
Trace redaction/replacement. We ran the proposed intervention from the same incorrect candidate — policy, prior history, environment state, and remaining generation/tool budgets fixed — 60 prefixes from 60 distinct issues (30 Pro + 30 DeepSWE) × 3 paired continuation seeds, all four conditions per pair (sham: neutral reformatting, byte-for-byte recoverable, exit status kept, no hint):
| Condition | Success | % |
|---|---|---|
| Intact | 112/180 | 62.2 |
| Redacted | 92/180 | 51.1 |
| Replaced | 88/180 | 48.9 |
| Sham | 107/180 | 59.4 |
Removing or replacing the evidence costs 11.1 and 13.3 points; reformatting alone costs 2.8. Issue-level paired 95% intervals: intact−redacted [2.1, 19.8], intact−replaced [4.5, 22.3], intact−sham [−6.0, 9.5] — the first two exclude zero, the sham interval spans it. This supports use of the diagnostic trace beyond rewarded co-occurrence, though it does not assign the entire Table 3 gain to this mechanism.
Thank you for running the additional controls. The frozen-judge comparison and the matched reward ablations address my concerns about what drives the gains. I keep positive on this paper.
Thank you for the careful review — the weaknesses identify real gaps, addressed below with new matched experiments. This comment answers Q1 and Q4; Q2 and Q3, which share one set of matched experiments, follow in the next comment.
Let
n11, n10, n01, n00count(L,CK)=(1,1),(1,0),(0,1),(0,0); thenL=n11+n10,E=n10,U=n00. Every condition samples N=100 trajectories, soL/N,E/N,U/Nin percent equaln11+n10,n10,n00; success is(n11+n01)/N; E/L is shown rounded to one decimal. Subsamples carry sampling error vs. the full tables, widest in small-L cells.R1a (Table 4; original 3,300/condition), CodeContests; Stage 2 = S1 SFT + RL:
R1b (Tables 14/16), SWE-bench Verified; full reward and no penalty are β=0.3 and β=0:
From Stage 1 to Stage 2, coverage rises (Qwen 62→83%, GPT-OSS 70→75%, Gemma 70→83%), L=1 failures fall (38→21, 20→15, 11→8), and unverified failures are 2→2, 26→22, 18→14. Fewer falsely validated failures accompany broader local validation. Against Official, Qwen and GPT-OSS still show higher E/N despite lower E/L as coverage expands. On Verified, full vs. no collapse penalty: 9 fewer collapsed failures, 6 more unverified, 3 more successes (62% vs. 59%); vs. no diagnostic: 4 more successes (62% vs. 58%). At β=0.5, coverage falls to 82, collapsed failures to 23, success rises to 64; β=0.1 reaches 65 — this sample does not rank β=0.3 highest; the full Table 16 does (65.4).
Labeling L.
L(τ)=1requires a qualifying local check on the final submitted candidate plus an explicit submission. A qualifying check concerns issue-related behavior and executes validly — skips, collection/infrastructure failures, and unrelated/inconclusive checks never qualify (where present, the run's judge suppliesrelevanceandexecution_validity) — and is rerun if the candidate changes. Other rubric fields enterD_agent, not L, so a valid test that misses the bug can still qualify; timeout auto-captures and unchecked submissions give L=0. Contest acceptance is the explicit SUBMIT after valid local execution (Listing F.3); "pass" is the agent's own reading, so L=1 can coexist with a wrong answer. L follows each run's own labeling; final resolution uses the common held-out evaluator. On a new 300-trajectory sample, extractor–human agreement is 95.7% (TP/FP/FN/TN 251/9/4/36), inter-annotator 97.3% pre-discussion; 12 ambiguous cases ended 8/4.Code and data are released upon acceptance, including a task manifest for every training and evaluation set (exact task IDs, dataset/environment/evaluator versions, SHA-256 checksums): repository training/validation (4,000/200 tasks, 11 repositories, R2E-Gym-V1), per-seed Table 3 schedules, the 731-task Pro and official 113-task DeepSWE sets with base commits, contest training (TACO, CodeContests) with source revisions and preprocessing versions, and the CodeContests/TACO-Verified/LiveCodeBench releases with date ranges and retained IDs. LiveCodeBench uses
release_v6(1,055 problems, May 2023–April 2025). The TACO-Verified diagnostic evaluation covers all 12,898 problems in the public likaixin/TACO-verified corpus.Exclusion removes exact task-ID matches and, after removing spaces, exact issue/problem-text matches; sharing a repository alone is allowed — the rule is exact-match by design. The judge-distillation pool is split by issue (3,600/400).
Costs. Beyond the matched 16,000 H200 GPU-hour policy allocation, each full-reward run adds 160 GPU-hours of local judge inference and 80 of judge-update optimization (240 total, 1.5%) and 76,000 newly teacher-labeled probes (246.0M input / 12.2M output tokens); the final-only arm adds none. One-time: the 4,000-probe teacher-labeled judge pool (3,600/400, six rubric fields per probe) and 40 GPU-hours of initial judge training; the new rebuttal audits took about 40 person-hours. For reference, the audited 400-update extended continuation added 50,776 teacher-labeled probes (judge updates: 400 new hard negatives + 400 replayed positives each); the earlier pipeline and the continuation used ≈8,512 and 10,752 GPU-hours. Billed API charges, retry/cache accounting, and these budget figures will be added in the revision.