- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a way to evaluate base checkpoints for agentic coding before post-training. Successful trajectories from frontier agents on DeepSWE, SWE-Pro and SWE-bench Verified are replayed, and the task's tests are run after every code-changing step to find the decisive step: the first step whose cumulative patch passes. Three probes are built at that step: Decisive-Action BPB (byte-normalized likelihood of the recorded action, with tool-call formatting masked), Patch MCQ (choosing the recorded action among alternatives that the same verifier rejects, read from answer-letter probabilities under cyclic rotation), and prefix-conditioned pass@K (the base model takes over at the decisive step through a prompt bridge and the verifier scores its continuations). Across ten public base checkpoints, the DeepSWE-derived probes reach Spearman ρ of 0.964, 0.867 and 0.951 with the public post-trained SWE-bench Verified pass@1, above bounded code benchmarks (best: RepoBench XFirst, 0.830), and a controlled SFT run on three checkpoints preserves the ordering.
Critical strengths
-
Clean, reusable construction. The verifier-certified decisive step turns an agentic benchmark into a base-model evaluation without requiring the base model to run the harness from scratch, where end-to-end evaluation fails (Table 8: five of six checkpoints score 0%).
-
Careful item design. Distractors are rejected by the same verifier, motivated by an audit finding 43.2% of APTBench SWE MCQs invalid, and the limits of n = 10 are stated openly.
Critical weaknesses
W1. Evidence for selecting new base checkpoints is limited.
The main results and the ablations used to justify the probe settings all use the same ten models, with no separate model set for validation. Their downstream scores also come from different post-training recipes and harnesses. The controlled SFT experiment helps, but covers only three models, and the DeepSWE-based BPB and Patch MCQ each misorder one pair (Table 16). It remains unclear whether a fixed probe can reliably rank new base checkpoints under the same post-training setup. Testing a fixed configuration on additional models, with uncertainty estimates for the ranking comparisons, would help establish this.
W2. The benefit over simpler likelihood scores is unclear.
All-step BPB already reaches ρ = 0.915, compared with 0.964 for decisive-step BPB (Fig. 2), and no uncertainty estimates are provided for this difference. The paper also lacks comparisons with BPB on generic code or on reference patches given only the issue and pre-edit files. Without these baselines, it is unclear how much the agentic context and decisive-step selection contribute beyond general coding ability. This matters because collecting trajectories and replaying them through the verifier adds work that simpler likelihood scores would avoid.
Questions for authors
Q1: Can you test a fixed configuration for each probe on new base checkpoints, using the same post-training recipe and evaluation harness? The configuration should be chosen without using these checkpoints’ downstream results.
Q2: How stable is the improvement from all-step BPB (ρ = 0.915) to decisive-step BPB (ρ = 0.964)? Since both correlations use the same ten models, could you report a paired confidence interval for the difference?
Q3: How do the probes compare with BPB on a generic code corpus, or BPB on the reference patch given only the issue and pre-edit files? These comparisons would help determine whether the trajectory context and decisive-step selection provide additional predictive value.
3 comments
Response to Reviewer aiaK (Part 2/2)
Q3: How do the probes compare with BPB on generic code, or on the reference patch given the issue and pre-edit files?
A3:
We added both baselines on the same ten checkpoints:
| Likelihood score | Tasks | ρ, SWE-V | Fisher-z 95% CI | Pairs correct | ρ, SWE-Multi | ρ, Terminal-Bench |
|---|---|---|---|---|---|---|
| BPB on generic Python code | — | 0.891 | [0.57, 0.98] | 40/45 | 0.818 | 0.553 |
| BPB on reference patch given issue + pre-edit files | SWE-V | 0.867 | [0.49, 0.97] | 39/45 | 0.782 | 0.565 |
| All-step BPB | SWE-V | 0.855 | [0.45, 0.97] | 39/45 | 0.915 | 0.754 |
| Decisive-Action BPB (ours) | SWE-V | 0.915 | [0.65, 0.98] | 41/45 | 0.964 | 0.809 |
| BPB on reference patch given issue + pre-edit files | DeepSWE | 0.915 | [0.65, 0.98] | 40/45 | 0.879 | 0.596 |
| All-step BPB | DeepSWE | 0.915 | [0.65, 0.98] | 41/45 | 0.964 | 0.809 |
| Decisive-Action BPB (ours) | DeepSWE | 0.964 | [0.84, 0.99] | 42/45 | 0.976 | 0.833 |
Spearman ρ with post-trained pass@1. SWE-V: SWE-bench Verified; SWE-Multi: SWE-bench Multilingual. Pairs correct is against SWE-V. Reference-patch BPB on DeepSWE uses 113 of the 120 tasks.
Both baselines are strong on SWE-bench Verified. Generic code BPB reaches ρ = 0.891, and Decisive-Action BPB leads it by only 0.024 (paired 95% CI [−0.33, 0.43]). The gap opens on the other targets. On SWE-bench Multilingual, the two baselines reach 0.818 and 0.782, against 0.964 for Decisive-Action BPB. On Terminal-Bench 2.1, they reach 0.553 and 0.565, against 0.809. Both baseline intervals include zero, whereas ours does not ([0.33, 0.96]). All-step BPB also holds up on these targets. The trajectory context, rather than decisive-step selection alone, therefore drives this robustness.
The two baselines rank the checkpoints almost identically (ρ = 0.988 between them). Decisive-Action BPB correlates with generic code BPB at only 0.758. For example, GLM-4.5-Air compresses generic code better than Nemotron Ultra, Qwen 3.5 35B and Nemotron Super, but post-trains below all three. Both baselines misorder these pairs, while Decisive-Action BPB orders all three correctly.
In short, general code likelihood carries much of the signal for SWE-style targets. Trajectory context makes the score more robust across task families. The trajectory construction also enables the verifier-grounded Patch MCQ and prefix pass@K. The paired interval on SWE-bench Verified includes zero, so we present this as a consistent direction rather than a significant gain. We will add both baselines to Table 7.
We will incorporate these analyses into the revision.
Dear Reviewer aiaK,
Thank you again for your careful review. Your two main concerns were whether a fixed probe can rank new base checkpoints (W1) and whether the probes add value over simpler likelihood scores (W2). In brief:
- New checkpoints (W1): We applied the frozen configuration, unchanged, to Super-v3.5-Base, released after our submission. Decisive-Action BPB and Patch MCQ place it at its true rank (fifth of eleven), and ρ stays at 0.945 (Fisher-z 95% CI [0.78, 0.99]). Re-selecting the configuration on every nine-checkpoint subset recovers the reported Patch MCQ setting in 9 of 10 folds. This checkpoint has its own post-training recipe, so it tests generalization rather than the controlled setting, and one checkpoint is limited evidence.
- Simpler likelihood scores (W2): As requested, we added BPB on generic code and on the reference patch given the issue and pre-edit files, and a paired interval for the all-step gap. On SWE-bench Verified these baselines are strong and the gaps are not significant at n = 10. The advantage appears on other targets: on Terminal-Bench 2.1, Decisive-Action BPB reaches 0.833 versus 0.553–0.596 for the baselines.
Could you let us know whether this addresses your concerns? If a further analysis would help, for example additional held-out checkpoints or another baseline, we would be glad to run it before the discussion closes.
Response to Reviewer aiaK (Part 1/2)
We thank Reviewer aiaK for the accurate summary and constructive questions. We will also streamline the presentation of the method and appendices.
Q1: If each probe's configuration is fixed in advance, does it still rank new base checkpoints correctly when they all receive the same post-training recipe and evaluation harness?
A1:
We agree that this is the most direct test. We applied the frozen configuration, unchanged, to Nemotron-3.5-Super-VL-120B-A12B-Base (Super-v3.5-Base). This checkpoint was released after our submission, so it played no part in any configuration choice. Its post-trained model reaches 71.1% on SWE-bench Verified, fifth of the eleven checkpoints.
Frozen configuration applied to Super-v3.5-Base (true rank 5 of 11 on post-trained SWE-bench Verified). Spearman ρ with post-trained pass@1.
Decisive-Action BPB and Patch MCQ both place Super-v3.5 at its true rank. Decisive-Action BPB misorders two pairs, with Kimi K2 (71.3%) and Nemotron Ultra (70.7%). Both are within 0.4 points of Super-v3.5. Adding the checkpoint changes ρ only slightly. With eleven checkpoints, the 95% Fisher-z interval is [0.78, 0.99] for Decisive-Action BPB and [0.67, 0.98] for prefix pass@32. Super-v3.5 is post-trained with its own recipe. This therefore tests generalization to an unseen checkpoint rather than the controlled setting, and one checkpoint is limited evidence.
We also checked whether the configuration choices depend on the original ten checkpoints. Several choices are fixed by construction. The decisive step is defined by the task verifier, and cyclic option rotation is a standard debiasing procedure (Zheng et al., 2024). For the choices we ablated, we re-selected the configuration on every nine-checkpoint subset. The reported Patch MCQ configuration (joint rotated scoring with reasoning) is selected in 9 of 10 folds on DeepSWE and 8 of 10 on SWE-bench Verified. Per-option scoring is never selected. For Decisive-Action BPB, the choices matter only modestly. Masking tool-call formatting moves ρ from 0.95 to 0.96, and decisive-step selection moves it from 0.915 to 0.964. We will fix one configuration per probe for all results, including K = 32 for prefix pass@K. We will also correct the K = 16 figure quoted in the introduction.
On Table 16, BPB and Patch MCQ misorder the same pair: Nemotron Super vs. Qwen 3.5 35B (post-SFT 53.1 vs. 61.2). The two are essentially tied on both probes. Their BPB values are 0.1221 and 0.1222. Their MCQ accuracies (41.2% vs. 38.8%) differ by about one standard error on 756 questions. Prefix pass@32 orders all three checkpoints correctly on every trajectory source. So do all three probes on SWE-Pro and SWE-bench Verified. We will move Table 16 into the main text.
Q2: How stable is the improvement from all-step BPB (ρ = 0.915) to decisive-step BPB (ρ = 0.964)? Could you report a paired confidence interval?
A2:
We ran a paired bootstrap over the ten checkpoints (10,000 resamples). Each resample recomputes both correlations:
Spearman ρ with post-trained SWE-bench Verified pass@1; Δρ is computed from unrounded values. Pairs correct is out of 45.
On DeepSWE, the interval includes zero, so we cannot claim significance at n = 10. The gap amounts to a few rank swaps. Decisive-step selection fixes DeepSeek V4 Flash vs. Nemotron Ultra and DeepSeek V4 Pro vs. Hy3. It introduces one error, Nemotron Super vs. Qwen 3.5 35B. On SWE-bench Verified, every pair misordered by decisive-step BPB is also misordered by all-step BPB. No resample therefore favors all-step BPB. The two corpora share the same checkpoints and are not independent replications. We will report both intervals and correct Figure 2b.
The replay cost is shared across probes. Replay also defines the golden action and distractors for Patch MCQ and the fork point for prefix pass@K. It is paid once per corpus (Figure 8). Per checkpoint, Decisive-Action BPB costs one forward pass per item.
References