acceptodds
Under review as a conference paper at ICLR 2027

Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models

Abstract

How can we predict which base checkpoint is worth an expensive round of agentic post-training? End-to-end pass@K tests whether successful behavior already appears in a base model’s distribution, but it is a poor fit for agentic coding: many base checkpoints cannot reliably produce the well-formed tool invocation required to complete a task end-to-end. Single-shot or short-horizon tasks avoid these tool-calling failures by collapsing a multi-turn interaction into a fixed prompt and a single patch, but they sidestep the core capability we care about: maintaining coherent state over many tool-using turns as the repository evolves. To bridge this gap, we treat successful post-trained agent trajectories as a lookahead signal of base-model potential. Replaying each trajectory and rerunning tests after every code-changing turn identifies the decisive turn: the first turn whose cumulative patch flips the repository from failing to passing, certifying that the recorded action solves the task given the prior context. Motivated by a coverage principle for agentic traces, we build three screens at this position that do not require a base checkpoint to drive the harness from a cold start: (i) Decisive-Action BPB (bits per byte) measures the probability mass on the certified action, (ii) Patch MCQ tests the checkpoint’s choice between that action and alternatives rejected by the same verifier, and (iii) prefix-conditioned pass@K evaluates support for functionally-correct generations and credits any continuation that the tests accept. Across ten pairs of public base and post-trained models, all three screens rank the cohort in close agreement with post-trained SWE-bench Verified pass@1. As our methods need only a benchmark’s successful trajectories and its verifier, they can be applied to turn future agentic coding benchmarks into base-model evaluations

Primary area:
datasets and benchmarks
Keywords:
large language models, agentic coding, model evaluation, checkpoint selection, LLM agents, post-training, benchmark correlation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aiaKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a way to evaluate base checkpoints for agentic coding before post-training. Successful trajectories from frontier agents on DeepSWE, SWE-Pro and SWE-bench Verified are replayed, and the task's tests are run after every code-changing step to find the decisive step: the first step whose cumulative patch passes. Three probes are built at that step: Decisive-Action BPB (byte-normalized likelihood of the recorded action, with tool-call formatting masked), Patch MCQ (choosing the recorded action among alternatives that the same verifier rejects, read from answer-letter probabilities under cyclic rotation), and prefix-conditioned pass@K (the base model takes over at the decisive step through a prompt bridge and the verifier scores its continuations). Across ten public base checkpoints, the DeepSWE-derived probes reach Spearman ρ of 0.964, 0.867 and 0.951 with the public post-trained SWE-bench Verified pass@1, above bounded code benchmarks (best: RepoBench XFirst, 0.830), and a controlled SFT run on three checkpoints preserves the ordering.

Critical strengths
  1. Clean, reusable construction. The verifier-certified decisive step turns an agentic benchmark into a base-model evaluation without requiring the base model to run the harness from scratch, where end-to-end evaluation fails (Table 8: five of six checkpoints score 0%).

  2. Careful item design. Distractors are rejected by the same verifier, motivated by an audit finding 43.2% of APTBench SWE MCQs invalid, and the limits of n = 10 are stated openly.

Critical weaknesses

W1. Evidence for selecting new base checkpoints is limited.

The main results and the ablations used to justify the probe settings all use the same ten models, with no separate model set for validation. Their downstream scores also come from different post-training recipes and harnesses. The controlled SFT experiment helps, but covers only three models, and the DeepSWE-based BPB and Patch MCQ each misorder one pair (Table 16). It remains unclear whether a fixed probe can reliably rank new base checkpoints under the same post-training setup. Testing a fixed configuration on additional models, with uncertainty estimates for the ranking comparisons, would help establish this.

W2. The benefit over simpler likelihood scores is unclear.

All-step BPB already reaches ρ = 0.915, compared with 0.964 for decisive-step BPB (Fig. 2), and no uncertainty estimates are provided for this difference. The paper also lacks comparisons with BPB on generic code or on reference patches given only the issue and pre-edit files. Without these baselines, it is unclear how much the agentic context and decisive-step selection contribute beyond general coding ability. This matters because collecting trajectories and replaying them through the verifier adds work that simpler likelihood scores would avoid.

Questions for authors

Q1: Can you test a fixed configuration for each probe on new base checkpoints, using the same post-training recipe and evaluation harness? The configuration should be chosen without using these checkpoints’ downstream results.

Q2: How stable is the improvement from all-step BPB (ρ = 0.915) to decisive-step BPB (ρ = 0.964)? Since both correlations use the same ten models, could you report a paired confidence interval for the difference?

Q3: How do the probes compare with BPB on a generic code corpus, or BPB on the reference patch given only the issue and pre-edit files? These comparisons would help determine whether the trajectory context and decisive-step selection provide additional predictive value.

3 comments
AuthorsOpenReview ↗

Response to Reviewer aiaK (Part 1/2)

We thank Reviewer aiaK for the accurate summary and constructive questions. We will also streamline the presentation of the method and appendices.

Q1: If each probe's configuration is fixed in advance, does it still rank new base checkpoints correctly when they all receive the same post-training recipe and evaluation harness?

A1:

We agree that this is the most direct test. We applied the frozen configuration, unchanged, to Nemotron-3.5-Super-VL-120B-A12B-Base (Super-v3.5-Base). This checkpoint was released after our submission, so it played no part in any configuration choice. Its post-trained model reaches 71.1% on SWE-bench Verified, fifth of the eleven checkpoints.

Probe (DeepSWE source)Predicted rank of Super-v3.5Pairs with Super-v3.5 correctρ, SWE-V (10 ckpts)ρ, SWE-V (11 ckpts)ρ, Terminal-Bench (11 ckpts)
Decisive-Action BPB58/100.9640.9450.866
Patch MCQ58/100.8670.8550.829
Prefix pass@3238/100.9510.9130.934
Generic code BPB (baseline)78/100.8910.8730.538

Frozen configuration applied to Super-v3.5-Base (true rank 5 of 11 on post-trained SWE-bench Verified). Spearman ρ with post-trained pass@1.

Decisive-Action BPB and Patch MCQ both place Super-v3.5 at its true rank. Decisive-Action BPB misorders two pairs, with Kimi K2 (71.3%) and Nemotron Ultra (70.7%). Both are within 0.4 points of Super-v3.5. Adding the checkpoint changes ρ only slightly. With eleven checkpoints, the 95% Fisher-z interval is [0.78, 0.99] for Decisive-Action BPB and [0.67, 0.98] for prefix pass@32. Super-v3.5 is post-trained with its own recipe. This therefore tests generalization to an unseen checkpoint rather than the controlled setting, and one checkpoint is limited evidence.

We also checked whether the configuration choices depend on the original ten checkpoints. Several choices are fixed by construction. The decisive step is defined by the task verifier, and cyclic option rotation is a standard debiasing procedure (Zheng et al., 2024). For the choices we ablated, we re-selected the configuration on every nine-checkpoint subset. The reported Patch MCQ configuration (joint rotated scoring with reasoning) is selected in 9 of 10 folds on DeepSWE and 8 of 10 on SWE-bench Verified. Per-option scoring is never selected. For Decisive-Action BPB, the choices matter only modestly. Masking tool-call formatting moves ρ from 0.95 to 0.96, and decisive-step selection moves it from 0.915 to 0.964. We will fix one configuration per probe for all results, including K = 32 for prefix pass@K. We will also correct the K = 16 figure quoted in the introduction.

On Table 16, BPB and Patch MCQ misorder the same pair: Nemotron Super vs. Qwen 3.5 35B (post-SFT 53.1 vs. 61.2). The two are essentially tied on both probes. Their BPB values are 0.1221 and 0.1222. Their MCQ accuracies (41.2% vs. 38.8%) differ by about one standard error on 756 questions. Prefix pass@32 orders all three checkpoints correctly on every trajectory source. So do all three probes on SWE-Pro and SWE-bench Verified. We will move Table 16 into the main text.

Q2: How stable is the improvement from all-step BPB (ρ = 0.915) to decisive-step BPB (ρ = 0.964)? Could you report a paired confidence interval?

A2:

We ran a paired bootstrap over the ten checkpoints (10,000 resamples). Each resample recomputes both correlations:

TrajectoriesAll-step ρDecisive-step ρΔρPaired 95% CIΔρ > 0Δρ ≥ 0Pairs correct
DeepSWE0.9150.964+0.048[−0.13, 0.30]58%77%41 → 42
SWE-bench Verified0.8550.915+0.061[0.00, 0.30]68%100%39 → 41

Spearman ρ with post-trained SWE-bench Verified pass@1; Δρ is computed from unrounded values. Pairs correct is out of 45.

On DeepSWE, the interval includes zero, so we cannot claim significance at n = 10. The gap amounts to a few rank swaps. Decisive-step selection fixes DeepSeek V4 Flash vs. Nemotron Ultra and DeepSeek V4 Pro vs. Hy3. It introduces one error, Nemotron Super vs. Qwen 3.5 35B. On SWE-bench Verified, every pair misordered by decisive-step BPB is also misordered by all-step BPB. No resample therefore favors all-step BPB. The two corpora share the same checkpoints and are not independent replications. We will report both intervals and correct Figure 2b.

The replay cost is shared across probes. Replay also defines the golden action and distractors for Patch MCQ and the fork point for prefix pass@K. It is paid once per corpus (Figure 8). Per checkpoint, Decisive-Action BPB costs one forward pass per item.

References

  • Zheng et al., 2024. Large Language Models Are Not Robust Multiple Choice Selectors. ICLR 2024.
AuthorsOpenReview ↗

Response to Reviewer aiaK (Part 2/2)

Q3: How do the probes compare with BPB on generic code, or on the reference patch given the issue and pre-edit files?

A3:

We added both baselines on the same ten checkpoints:

Likelihood scoreTasksρ, SWE-VFisher-z 95% CIPairs correctρ, SWE-Multiρ, Terminal-Bench
BPB on generic Python code—0.891[0.57, 0.98]40/450.8180.553
BPB on reference patch given issue + pre-edit filesSWE-V0.867[0.49, 0.97]39/450.7820.565
All-step BPBSWE-V0.855[0.45, 0.97]39/450.9150.754
Decisive-Action BPB (ours)SWE-V0.915[0.65, 0.98]41/450.9640.809
BPB on reference patch given issue + pre-edit filesDeepSWE0.915[0.65, 0.98]40/450.8790.596
All-step BPBDeepSWE0.915[0.65, 0.98]41/450.9640.809
Decisive-Action BPB (ours)DeepSWE0.964[0.84, 0.99]42/450.9760.833

Spearman ρ with post-trained pass@1. SWE-V: SWE-bench Verified; SWE-Multi: SWE-bench Multilingual. Pairs correct is against SWE-V. Reference-patch BPB on DeepSWE uses 113 of the 120 tasks.

Both baselines are strong on SWE-bench Verified. Generic code BPB reaches ρ = 0.891, and Decisive-Action BPB leads it by only 0.024 (paired 95% CI [−0.33, 0.43]). The gap opens on the other targets. On SWE-bench Multilingual, the two baselines reach 0.818 and 0.782, against 0.964 for Decisive-Action BPB. On Terminal-Bench 2.1, they reach 0.553 and 0.565, against 0.809. Both baseline intervals include zero, whereas ours does not ([0.33, 0.96]). All-step BPB also holds up on these targets. The trajectory context, rather than decisive-step selection alone, therefore drives this robustness.

The two baselines rank the checkpoints almost identically (ρ = 0.988 between them). Decisive-Action BPB correlates with generic code BPB at only 0.758. For example, GLM-4.5-Air compresses generic code better than Nemotron Ultra, Qwen 3.5 35B and Nemotron Super, but post-trains below all three. Both baselines misorder these pairs, while Decisive-Action BPB orders all three correctly.

In short, general code likelihood carries much of the signal for SWE-style targets. Trajectory context makes the score more robust across task families. The trajectory construction also enables the verifier-grounded Patch MCQ and prefix pass@K. The paired interval on SWE-bench Verified includes zero, so we present this as a consistent direction rather than a significant gain. We will add both baselines to Table 7.

We will incorporate these analyses into the revision.

Dear Reviewer aiaK,

Thank you again for your careful review. Your two main concerns were whether a fixed probe can rank new base checkpoints (W1) and whether the probes add value over simpler likelihood scores (W2). In brief:

  • New checkpoints (W1): We applied the frozen configuration, unchanged, to Super-v3.5-Base, released after our submission. Decisive-Action BPB and Patch MCQ place it at its true rank (fifth of eleven), and ρ stays at 0.945 (Fisher-z 95% CI [0.78, 0.99]). Re-selecting the configuration on every nine-checkpoint subset recovers the reported Patch MCQ setting in 9 of 10 folds. This checkpoint has its own post-training recipe, so it tests generalization rather than the controlled setting, and one checkpoint is limited evidence.
  • Simpler likelihood scores (W2): As requested, we added BPB on generic code and on the reference patch given the issue and pre-edit files, and a paired interval for the all-step gap. On SWE-bench Verified these baselines are strong and the gaps are not significant at n = 10. The advantage appears on other targets: on Terminal-Bench 2.1, Decisive-Action BPB reaches 0.833 versus 0.553–0.596 for the baselines.

Could you let us know whether this addresses your concerns? If a further analysis would help, for example additional held-out checkpoints or another baseline, we would be glad to run it before the discussion closes.

Reviewer_FzueOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present a framework for predicting the post-training agentic coding performance of base language models without requiring them to execute end-to-end agentic tasks. The authors leverage successful trajectories collected from post-trained coding agents and identify the decisive step, defined as the earliest step at which the cumulative code changes satisfy the task verifier. Building on this concept, they propose three evaluation metrics: (1) Decisive-Action BPB, which measures the likelihood of the successful action; (2) Patch MCQ, which assesses the model's ability to distinguish successful patches from unsuccessful alternatives; and (3) prefix-conditioned pass@K, which estimates the probability of generating a successful continuation. Experiments across ten base model checkpoints demonstrate strong correlations between these metrics and the downstream performance of their corresponding post-trained models on SWE-bench Verified and other coding benchmarks. The authors further conduct ablation studies and controlled supervised fine-tuning experiments to validate the effectiveness of the proposed evaluation framework.

Critical strengths
  • Predicting downstream agentic capabilities before costly post-training is an important and practically relevant problem. The proposed framework provides a way to assess base model checkpoints that are not yet capable of reliably completing end-to-end agentic workflows, potentially reducing the computational cost of base-model selection.

  • The proposed metrics demonstrate strong correlations with post-training performance across ten model checkpoints and multiple benchmark settings. The paper further provides comparisons with existing coding benchmarks, ablation studies, and controlled fine-tuning experiments. Together, these results offer promising empirical support for the effectiveness of the proposed trajectory-based evaluation approach.

Critical weaknesses
  • The framework focuses exclusively on the decisive step, where the accumulated trajectory first produces a correct solution. However, successful agentic behavior also requires intermediate capabilities, such as planning, tool calling, interpreting observations, and adapting actions based on environmental feedback. By providing the preceding trajectory and evaluating only the final decisive action, the proposed metrics may primarily measure the model's ability to recognize or generate correct code patches given an already constructed context, rather than its ability to execute long-horizon agentic workflows. Although the paper compares decisive-step BPB against all-step BPB, this does not establish that intermediate steps are uninformative or that the decisive step sufficiently captures agentic capability. Furthermore, the applicability of this approach beyond coding environments with executable verifiers remains unclear, particularly for general-purpose agents involving tool calls, knowledge bases, and user interactions. These concerns limit the interpretation and generality of the proposed framework.

  • Insufficient justification for the evaluation metrics and interpretation of the experimental correlations. The paper introduces three metrics, but their necessity and complementary contributions are not sufficiently established. All three evaluate closely related aspects of the model's distribution at the same decisive step, and their strong correlations with downstream performance raise the question of whether a single metric would suffice. More importantly, the extremely high correlations between base-model metrics and post-trained performance are surprising given that the evaluated models undergo different post-training procedures. It is unclear to what extent these correlations reflect the intrinsic predictive power of the proposed metrics, as opposed to differences in model scale or post-training resources. Although the paper includes a controlled fine-tuning experiment, it covers only three checkpoints, which is insufficient to fully disentangle these factors. Consequently, the evidence does not yet establish how reliably the framework predicts post-training performance across different training recipes or among base models of similar capability.

Questions for authors
  • Why is the decisive step considered the most appropriate point for evaluating agentic potential? Intermediate steps may involve important capabilities, such as planning, tool selection, reasoning over observations, and state tracking, which may not be adequately captured by the final code-changing action. Could the authors evaluate different categories of intermediate steps or investigate whether combining their scores with decisive-step metrics provides complementary predictive information? Such an analysis would also help clarify whether the proposed framework primarily measures coding proficiency or broader agentic capabilities.

  • What distinct aspects of model capability are captured by Decisive-Action BPB, Patch MCQ, and prefix-conditioned pass@K? Given that all three metrics exhibit similarly strong correlations with post-training performance, it would be valuable to report their pairwise correlations and investigate whether combining them yields additional predictive value. If a single metric achieves comparable predictive performance, what justifies the additional complexity introduced by the other two?

  • The reported correlations between base-model evaluation metrics and post-trained agentic performance are remarkably high, despite variations in post-training recipes across models. Could the authors provide further insight into why base-model performance predicts post-training outcomes so accurately? In particular, to what extent are these correlations explained by underlying model capabilities, model size, or differences in post-training resources? Additional controlled post-training experiments across a larger set of checkpoints, or analyses that account for these potential confounding factors, would help establish whether the proposed metrics genuinely capture post-training potential rather than simply serving as proxies for overall model strength.

  • How could the proposed framework be extended to more general agentic tasks involving tool use, knowledge-base retrieval, or user interactions, where success may depend on multiple intermediate actions and reliable step-level verification may not be available? Furthermore, APTBench already incorporates answer discrimination and supports a broader range of agentic evaluation scenarios. Beyond leveraging verifier-certified decisive actions, what are the key methodological advantages of the proposed framework over existing approaches such as APTBench, particularly in settings where action correctness can also be reliably verified?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Fzue (Part 1/2)

We thank Reviewer Fzue for the thoughtful review and address each question below.

Q1: Why the decisive step, and do the probes measure coding proficiency or broader agentic capability?

A1: At T⋆T^{\star}, the checkpoint conditions on the full agentic history. This includes the task statement, tool definitions, earlier commands and their outputs, test results, and the repository state left by earlier edits. On DeepSWE, this prefix averages 18.6 tool calls (median 17) and about 42K tokens (median about 34K). The probes therefore test whether a model can act correctly on that state, not isolated patch writing. Consistent with this, isolated coding benchmarks track post-trained SWE-bench Verified poorly on the same ten checkpoints. HumanEval reaches ρ = −0.394, MBPP 0.091, LiveCodeBench 0.455, CRUXEval-O 0.758 and RepoBench XFirst 0.830. Our probes reach 0.867–0.964.

We focus on the decisive step for two reasons. First, it is the only step whose action is certified correct. Each intermediate action is one of many valid choices, so its likelihood partly measures imitation of the trajectory generator's style. Second, base checkpoints cannot yet carry out the intermediate phase themselves. In a matched control where prefix pass@32 forks five steps before the decisive step, nine of ten checkpoints resolve no tasks and ρ falls to −0.06.

We do not claim that intermediate steps are uninformative. All-step BPB reaches ρ = 0.915, versus 0.964 for the decisive step, and at n = 10 this gap is not significant (paired 95% CI [−0.13, 0.30]). The decisive step is therefore a principled choice rather than a statistically superior one. A breakdown by intermediate-step category (e.g., exploration vs. test runs) is a natural extension that we will add.

We therefore agree with the reviewer's scoping. The probes measure the capacity to act correctly within an agentic context. They do not measure the ability to construct that context through planning and exploration. We will state this explicitly.

Q2: What do the three probes capture, how correlated are they, and does combining them help?

A2: Pairwise Spearman correlations across the ten checkpoints (DeepSWE source):

Probe pairSpearman ρ
Decisive-Action BPB vs. Patch MCQ0.964
Decisive-Action BPB vs. prefix pass@320.939
Patch MCQ vs. prefix pass@320.878

The probes are strongly correlated. Simple rank averaging does not help: averaging all three gives ρ = 0.964, the same as Decisive-Action BPB alone. A fitted combination of BPB and pass@32 does help. In leave-one-out linear prediction (fit on nine checkpoints, predict the tenth), BPB alone predicts held-out SWE-bench Verified within 3.9 points (mean absolute error), versus 12.0 for the cohort mean. Adding pass@32 lowers the error to 2.7 points, and on Terminal-Bench 2.1 from 12.1 to 7.7 (bootstrap 95% CI of the gain [0.8, 7.8]). Adding Patch MCQ does not help. This pair was selected post hoc at n = 10, so we treat the gain as suggestive. The probes also play distinct roles:

  • They ask different questions. BPB measures probability mass on the certified action. Patch MCQ tests whether the model can tell it apart from verifier-rejected near-misses, which likelihood alone cannot. Pass@K tests whether the model can generate a passing action itself.
  • They fail differently. BPB depends on the tokenizer and chat template, Patch MCQ on answer-position bias, and pass@K on the harness and sampling. Their agreement indicates that the ranking is not an artifact of any one of them.
  • They differ in cost and transfer. BPB needs one forward pass per item and no environment. Pass@K needs K verified rollouts per item but transfers better to Terminal-Bench 2.1 (ρ = 0.930 vs. 0.833).

We recommend Decisive-Action BPB as the default screen, adding prefix pass@K when rollouts are affordable or the target goes beyond repository-level issue resolution. Patch MCQ gives an interpretable discrimination score. We will add this guidance and the analysis above to the paper.

AuthorsOpenReview ↗

Response to Reviewer Fzue (Part 2/2)

Q3: Why do base-model metrics predict post-training outcomes, and do model size or post-training resources explain the correlations?

A3:

Our motivation is the coverage principle (Chen et al., 2026). Post-training, and RLVR in particular, mainly sharpens behaviors that already have probability mass under the base model (Yue et al., 2026; Wu et al., 2025). The probes measure that mass at verifier-certified decision points. We present our results as consistent with this account, not as proof of it.

Model size. Size is a real confound. Log total parameters (publisher counts) alone give ρ = 0.842 with post-trained SWE-bench Verified, below all three probes (0.867–0.964). The gap widens on Terminal-Bench 2.1, where size reaches only 0.505 versus 0.796–0.930 for the probes. In leave-one-out prediction of SWE-bench Verified, a linear fit on size misses the held-out checkpoint by 8.2 points on average, versus 3.9 for Decisive-Action BPB. Patch MCQ is the most size-sensitive probe. Two cases show the probes are not size proxies. First, Nemotron Nano, Qwen 3.5 35B and Gemma 4 26B (26–35B total, 3–4B active) span 38.8–69.2% post-trained, and all three probes order them correctly. Second, Kimi K2 (about 1T) and Nemotron Ultra (550B) post-train below the smaller DeepSeek V4 Flash (284B) and Hy3 (295B). Decisive-Action BPB ranks them accordingly.

Overall model strength. General coding ability does not explain the results. BPB on generic Python reaches ρ = 0.891 on SWE-bench Verified, but only 0.553 on Terminal-Bench 2.1, against 0.833 for Decisive-Action BPB (DeepSWE source). The two scores correlate at only 0.818 across checkpoints.

Post-training resources. These are unobservable for public models, so we cannot rule them out. The controlled SFT study (Section 4.4) fixes the recipe (same data, 16.8B tokens, same harness). Post-SFT scores are 28.5, 53.1 and 61.2 (gaps of ~8.2 and 2.6 standard errors). Prefix pass@32 recovers this order on every trajectory source. On DeepSWE, BPB and Patch MCQ swap only the closer, essentially tied pair. Three checkpoints are supporting, not conclusive, evidence; we will state this. As a further test, we applied the frozen configuration to Nemotron-3.5-Super-VL-120B-A12B-Base, released after submission. Decisive-Action BPB places it at its true rank (fifth of eleven), and ρ stays at 0.945.

Uncertainty. The 95% Fisher-z intervals are [0.84, 0.99] for Decisive-Action BPB, [0.49, 0.97] for Patch MCQ and [0.79, 0.99] for prefix pass@32. Leave-one-out ρ stays at or above 0.82. BPB misorders three pairs; two differ by ≤1.0 point downstream, below the ~2.9-point standard error of a difference on 500 tasks.

Q4: How can the framework extend beyond code without step-level verification, and how does it differ from APTBench?

A4:

The framework does not need step-level verification. Decisive-step discovery applies the task's own outcome verifier to replayed intermediate states. It requires only (i) successful trajectories, (ii) an outcome verifier and (iii) replayable, restorable environment state. None is code-specific. Terminal-Bench-style tasks test the final system state. τ-bench verifies the final database state, and recorded user turns can be replayed for the prefix (pass@K also needs a user simulator). Without any verifier, only Decisive-Action BPB on successful trajectories remains. Model-based judges (e.g., RuDE; Li et al., 2026) could substitute, at the cost of execution-grounded validity. We will state this scope explicitly.

We do not claim higher correlation than APTBench, and APTBench covers a broader range of agentic scenarios. On our cohort, APTBench-SWE reaches ρ = 0.879 and APTBench-DR 0.903, inseparable from our probes at n = 10. The differences are methodological:

  • Valid answers. 727 of 1,682 APTBench SWE MCQs (43.2%) have a designated answer that is not uniquely correct (Appendix A). Our correct option is certified by a fail-to-pass verifier flip. Every distractor is certified to fail.
  • Functional scoring. APTBench scores action generation by exact match or ROUGE. Prefix pass@K credits any continuation the verifier accepts.
  • Real context. Our items use actual trajectory prefixes and restorable states. Every option is an action an agent actually produced.
  • Renewable construction. Any benchmark with successful trajectories and a verifier converts automatically, so new, less contaminated benchmarks can be adopted as they appear.

We will incorporate these clarifications and analyses into the revision.

References

  • Chen et al., 2026. The Coverage Principle: How Pre-training Enables Post-training. ICLR 2026.
  • Li et al., 2026. On Predicting the Post-Training Potential of Pre-trained LLMs. arXiv:2605.11978.
  • Wu et al., 2025. The Invisible Leash: Why RLVR May or May Not Escape Its Origin. arXiv:2507.14843.
  • Yue et al., 2026. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? NeurIPS 2026.

Dear Reviewer Fzue,

Thank you again for your thoughtful review. Your two main concerns were interpretation (whether the decisive step measures agentic capability or only patch recognition) and generality beyond code. In brief:

  • Interpretation: We agree with your scoping. The probes measure acting correctly within an agentic context (on average 18.6 prior tool calls and ~42K tokens), not constructing that context. We will state this explicitly and no longer claim that intermediate steps are uninformative.
  • Generality: Decisive-step discovery needs only an outcome verifier and replayable state, not step-level verification; we outlined how this applies to Terminal-Bench and τ-bench.

Could you let us know whether this addresses your concerns, or whether a specific analysis would help? We would be glad to run it before the discussion closes.

Reviewer_5hFSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes three probes for predicting coding-agent performance after post-training from base checkpoints. Successful agent trajectories are replayed to identify the first code-changing step that passes the task’s tests. At this point, the probes measure the likelihood of the successful action, discrimination between verified and rejected alternatives, and success under sampled continuations. Validation includes ten base/post-trained model pairs, three trajectory sources, correlations with three downstream benchmarks, protocol ablations, and a controlled SFT experiment on three models.

Critical strengths
  1. Practically relevant objective. Selecting promising base checkpoints before expensive agentic post-training is an important problem. The static probes provide a potentially economical screening mechanism once the trajectory corpus has been constructed.

  2. Execution-grounded evaluation. Using the task’s verifier to validate successful actions and reject distractors gives the probes a concrete correctness criterion and reduces reliance on subjective answer annotation.

  3. Multiple complementary validations. The study combines likelihood, discrimination, and generation probes with cross-benchmark correlations, ablations, and a shared-recipe SFT experiment. This provides useful triangulation beyond a single proxy metric.

Critical weaknesses
  1. Predictive validity remains insufficiently established. The main evidence consists of retrospective correlations across ten heterogeneous model families, whose post-training recipes and evaluation harnesses differ. The controlled SFT study covers only three models, and some probe rankings depend on the trajectory source. Without uncertainty estimates or within-run checkpoint validation, the results do not yet establish reliable checkpoint selection.

  2. The decisive-step guarantee is not justified by the implemented search. Appendix D states that all reported steps were found by bisection, assuming that test success is monotonic along a trajectory. Later edits can invalidate an earlier passing patch, so bisection does not generally identify the earliest passing step. No exhaustive-scan audit is reported, leaving the framework’s defining selection criterion unverified.

  3. The benefit of decisive-step conditioning is not adequately isolated. The continuation probe uses an expert prefix, explicit edit-directed steering, action prefilling, and a three-step generation budget. The cold-start comparison does not provide matched controls for these adaptations. Consequently, the evidence does not clearly separate the value of decisive-step context from protocol assistance and proximity to an existing solution.

Questions for authors
  1. Does the proposed ranking remain reliable among checkpoints from the same pretraining run under identical post-training? Can uncertainty estimates establish the reported improvements over all-step BPB and existing proxies?

  2. Have bisection-selected steps been checked against exhaustive scans? How frequently do trajectories violate monotonicity, and would correcting the selected steps change the results?

  3. Under matched steering, action prefilling, and generation budgets, how do early or randomly selected trajectory prefixes compare with decisive-step prefixes?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5hFS (Part 1/2)

We thank Reviewer 5hFS for the careful review. We agree that our results are checkpoint-ranking evidence, not a calibrated guarantee of post-training performance. We have run the requested analyses below.

Q1: Do uncertainty estimates support the improvements over all-step BPB and existing proxies?

A1:

For each probe (DeepSWE source), the table reports Spearman ρ with post-trained SWE-bench Verified pass@1. It also gives a 95% Fisher-z interval (SE = 1.06/√(n−3)) and a 10,000-resample bootstrap over checkpoints. The last two columns give the leave-one-out range and the checkpoint pairs ordered correctly, out of 45.

ProbeρFisher-z 95% CIBootstrap 95% CILeave-one-out ρPairs correct
Decisive-Action BPB0.964[0.84, 0.99][0.79, 1.00]0.95–0.9742/45
Patch MCQ (think)0.867[0.49, 0.97][0.47, 1.00]0.82–0.9039/45
Prefix pass@320.951[0.79, 0.99][0.69, 1.00]0.93–0.9941/45

Bootstrap upper bounds of 1.00 reflect duplicate checkpoints within resamples; we treat the Fisher-z interval as primary.

No single checkpoint drives the results: leave-one-out ρ stays at or above 0.82 for every probe. Most misorderings fall within benchmark noise. On 500 tasks, the standard error of a difference between two models is about 2.9 points. Two of the three pairs misordered by Decisive-Action BPB differ by at most 1.0 point (Nemotron Ultra vs. Kimi K2; DeepSeek V4 Flash vs. Hy3).

The intervals clearly separate Decisive-Action BPB from function-synthesis benchmarks. The upper bounds for MBPP (0.70) and HumanEval (0.35) lie below our lower bound of 0.84. They do not separate it from all-step BPB (0.915, [0.65, 0.98]), RepoBench XFirst (0.830, [0.38, 0.96]) or APTBench-SWE (0.879, [0.53, 0.97]). Against all-step BPB, a paired bootstrap gives Δρ = +0.048 (95% CI [−0.13, 0.30]). We therefore do not claim this gain is significant at n = 10. Decisive-step selection fixes two orderings, placing DeepSeek V4 Flash above Nemotron Ultra and DeepSeek V4 Pro above Hy3. It introduces one misordering, placing Nemotron Super above Qwen 3.5 35B. The net result is 42 rather than 41 of 45 pairs correct. We will revise the text and Figure 2b accordingly.

Q2: Does the ranking hold for checkpoints from the same pretraining run under identical post-training?

A2:

We agree that correlations across model families cannot fully separate base-checkpoint quality from recipe effects. Our closest test is the controlled SFT study (Section 4.4). It applies an identical recipe (same data, 16.8B tokens, same harness and evaluation) to three base checkpoints. They reach 28.5, 53.1 and 61.2 on SWE-bench Verified, gaps of about 8.2 and 2.6 standard errors. Prefix pass@32 recovers this order on every trajectory source, and all three probes do so on SWE-Pro and SWE-bench Verified. On DeepSWE, BPB and Patch MCQ swap the closer pair, whose BPB values are essentially tied (0.1221 vs. 0.1222). On the full ten-checkpoint cohort, ρ for Decisive-Action BPB stays between 0.915 and 0.964 across the three trajectory sources. Three models are supporting rather than conclusive evidence: a random ranking of three is correct with probability 1/6. We will say so. As a weaker within-family check, we also tested Super-v3.5-Base, released after our submission. All three probes rank it above Nemotron Super, matching their post-trained scores (71.1% vs. 60.5%).

Checkpoints from a single pretraining run would be the most direct test, but they are out of reach for the rebuttal. Each controlled pretraining run requires a multi-trillion-token corpus. Most models in our cohort do not release intermediate checkpoints. Each checkpoint would also need its own full post-training run to provide a downstream target. We will state this limitation and list it as future work.

AuthorsOpenReview ↗

Response to Reviewer 5hFS (Part 2/2)

Q3: Does bisection find the earliest passing step when verifier success is non-monotonic?

A3:

We exhaustively scanned all 120 DeepSWE trajectories. We graded the cumulative patch at every code-changing step (2,543 verifier runs). 116 of 120 trajectories (96.7%) are monotonic. Bisection finds the true earliest passing step in 117 of 120 (97.5%).

Non-monotonic trajectoryBisection stepTrue earliest stepOutcome by step (1 = pass)
go-git-worktree-merge-conflicts2118…00011011111
wazero-multi-module-snapshots2922…0110000111111111111
yjs-map-conflict-detection1813…011000111
yaegi-go-embed-directives1616…01101111

In every violation, the patch passes briefly, the next edit breaks it, and it passes again later. Bisection then skips the brief pass and returns a later passing step, 3 to 7 steps late. One step timed out twice in the grader (about 6,300 s each), and we count it as failing. Excluding that trajectory instead gives 115 of 119 and 116 of 119. Only 3 of the 120 decisive steps change, so the correction affects 2.5% of items. Bisection nonetheless does not guarantee the stated definition. We will switch to exhaustive scanning and audit the SWE-Pro and SWE-bench Verified trajectories in the same way. We will then recompute all three probes and report the audit in Appendix D.

Q4: Under matched steering, prefilling and budget, how do early prefixes compare with decisive-step prefixes?

A4:

We ran a matched control on the same 120 DeepSWE tasks and ten checkpoints. The edit-directed steer, action prefilling, sampling temperature and three-step generation budget are identical. Only the fork point moves, from the decisive step T⋆T^{\star} to T⋆T^{\star}−5. Nine of the ten checkpoints then resolve no tasks. Only Qwen 3.5 35B scores (8.5% pass@32), and ρ with post-trained SWE-bench Verified falls from 0.951 to −0.06. This mirrors cold-start evaluation (Table 8), where Qwen 3.5 35B is also the only non-zero model. We attribute this to familiarity with agent-formatted interaction rather than coding ability: its post-trained model ranks sixth of ten. The protocol alone therefore does not produce a ranking; the decisive-step context does.

Forking at the decisive step does not make the task trivial. DeepSWE pass@32 spans 2.5% (Nemotron Nano) to 45.8% (DeepSeek V4 Pro), and this spread carries the ranking. Decisive-Action BPB and Patch MCQ involve no steering, prefilling or sampling. Yet they rank the checkpoints about as well as prefix pass@K (ρ = 0.964 and 0.867, vs. 0.951). We will add this control to Section 4.5.

We will incorporate all of these analyses into the revision.

Dear Reviewer 5hFS,

Thank you again for your careful review. Your three main concerns were predictive validity, the bisection-based decisive-step search, and isolating the value of decisive-step context. In brief:

  • Predictive validity: We added Fisher-z and bootstrap intervals for every probe (Decisive-Action BPB: ρ = 0.964, 95% CI [0.84, 0.99]) and leave-one-out ranges (ρ ≥ 0.82 for every probe). The intervals separate our probes from function-synthesis benchmarks but not from all-step BPB or APTBench, and we no longer claim a significant gain over all-step BPB (paired CI [−0.13, 0.30]). Checkpoints from a single pretraining run remain out of reach; we state this as a limitation.
  • Bisection: We exhaustively scanned all 120 DeepSWE trajectories (2,543 verifier runs). 116 of 120 are monotonic, and bisection finds the true earliest passing step in 117 of 120, so 2.5% of items change. We will switch to exhaustive scanning and audit the other two sources.
  • Decisive-step context: Under identical steering, prefilling, temperature and generation budget, moving the fork point five steps earlier leaves nine of ten checkpoints resolving no tasks, and ρ falls from 0.951 to −0.06. The protocol alone does not produce a ranking; the decisive-step context does.

Could you let us know whether this addresses your concerns? If a further analysis would help, for example the randomly selected prefixes you suggested, we would be glad to run it before the discussion closes.

Comments on the paper

1 comment

We thank the reviewer for the follow-up and address the remaining concerns below.

Q1: The bisection part is solved for me. The full scan on DeepSWE is what I wanted, and only 3 of 120 changed. Please also check SWE-Pro and SWE-bench Verified in the revision.

A1: We ran the same exhaustive scan on both corpora. On SWE-bench Verified, all 98 trajectories are monotonic, so bisection already returns the true earliest passing step. On SWE-Pro, 94 of 97 trajectories are monotonic. We will use the exhaustive scan for all three corpora, recompute all probes, and report the full audit in Appendix D.

Q2: The intervals help, but they make me less sure about the decisive step. On SWE-bench Verified, it is not better than all-step BPB or generic code BPB.

A2: On SWE-bench Verified, Decisive-Action BPB (last row of the table below) reaches ρ = 0.915. This is better than both all-step BPB (0.855) and BPB on generic Python code (0.891). The advantage of decisive-step BPB is even clearer on SWE-bench Multilingual and Terminal-Bench 2.1:

Likelihood score (SWE-V trajectories)ρ, SWE-Vρ, SWE-Multiρ, Terminal-Bench
BPB on generic Python code0.8910.8180.553
BPB on reference patch given issue + pre-edit files0.8670.7820.565
All-step BPB0.8550.9150.754
Decisive-Action BPB (ours)0.9150.9640.809

Q3: Also, BPB gets Nemotron Super vs. Qwen 3.5 35B wrong, and they are about 8 points apart, so this is not just noise.

A3: Our BPB does not get Nemotron Super vs. Qwen 3.5 35B wrong. As shown in Table 1 of the manuscript, Qwen-3.5-35B-A3B-Base reaches 0.092 BPB, lower than the 0.100 of Nemotron-3-Super-Base. For BPB, lower is better, so the probe correctly ranks Qwen 3.5 35B above Nemotron Super, matching their post-SFT scores (61.2 vs. 53.1).

Q4: The T⋆−5 test does not really answer my question. The model is told the investigation is done when it is not, and it only has 3 steps, so almost all models get 0. To me this shows the probe only works when the model is very close to the answer. Testing at T⋆−1, T⋆−2 and T⋆−3 would help more.

A4: We agree with the reviewer's reading and will state it in the paper. Prefix pass@K separates checkpoints only at or very near the decisive step. This is its intended scope: it asks whether a model can take the decisive action given an agentic context, not whether it can finish the investigation itself. We also agree that the edit-directed steer does not fit earlier fork points, so T⋆−5T^{\star}-5 is not a clean test of prefix location.

Following the suggestion, we ran T⋆−3T^{\star}-3 with the same steer, prefilling, temperature and three-step budget. Eight checkpoints have finished so far, and the other two are still running:

CheckpointPost-trained SWE-V (%)pass@16 at T⋆T^{\star} (%)pass@16 at T⋆−3T^{\star}-3 (%)
DeepSeek V4 Flash79.033.35.8
Hy378.020.83.3
Kimi K271.316.73.3
Nemotron Ultra70.717.55.0
Nemotron Super60.510.83.3
GLM-4.5-Air57.69.23.3
Gemma 4 26B52.06.72.5
Nemotron Nano38.82.52.5

At T⋆−3T^{\star}-3, the probe can no longer separate the checkpoints. All eight solve between three and seven of the 120 tasks. Hy3, Kimi K2, Nemotron Super and GLM-4.5-Air each solve exactly four, although their post-trained scores span 57.6–78.0%. Gemma 4 26B and Nemotron Nano also tie. Of the 28 checkpoint pairs, 27 are ordered correctly at T⋆T^{\star}, but only 19 at T⋆−3T^{\star}-3, with 7 exact ties.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.