- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Contextual Early Reward (CER), an LLM-judge method that predicts the terminal (test-based) reward of an unfinished coding-agent trajectory from its prefix. For a group of prefixes sampled for the same task, CER generates one shared set of weighted, task- and stage-specific rubrics, and scores every prefix with criterion-level evidence. For TTS the scores only need to preserve the within-group order (Eq. 2). For RL the paper adds a batch-stability condition (App. C.3). The paper studies two uses:
- Test-time scaling (TTS). The policy model generates its own rubrics, guided by an experience bank that GPT-5.6 Sol distills from verified historical groups (BM25 retrieval, leave-one-out). The scores drive a beam-style search: expansion every 20 steps, regression stopping, and tie-breaking rubrics. On SWE-bench Verified, CER@60 reaches 67.6% RM@8 on Nemotron 3 Ultra and 71.4% on Qwen 3.6 27B. The strongest reranker of 8 complete trajectories reaches 63.4% and 69.4%. On Nemotron, CER@20 already exceeds that reranker by 1.4 pp while using 15.3% of its tokens.
- RL. GPT-5.6 Sol builds task-specific rubrics offline from verified historical rollouts, and their weights are fitted to pairwise agreement with the verifier. Online rollouts of Qwen 3.5 9B are cut at 40 turns, and GPT-5.6 Luna scores them to give group-relative advantages in the TMax/DPPO objective. Over three best-effort repository-separated folds (250 train / 50 val / 200 test tasks), CER reaches a 51.6% test resolved rate, compared with 49.7% for TMax and 49.8% for TMax-40. At the matched 384-step checkpoint, CER uses 52.7% fewer online policy-and-judge tokens than TMax.
Further analyses cover experience transfer across models, rubric–verifier agreement (60.9%), a rubric taxonomy, component ablations on a 100-task subset, and a negative result for using CER as a process reward.
Critical strengths
S1. A relevant problem and a sensible, interpretable design with some controlled evidence. Long-horizon SWE agents are expensive to roll out. The paper's design scores a group of sibling prefixes under one shared rubric set and requires evidence for each criterion, which is a reasonable choice. The ablation supports it: removing sibling comparison costs 3.0–4.0 pp (Table 8). The design also sharply lowers the top-tie rate relative to the adaptive-rubric judges: Agentic Rubrics ties on 57.2% / 88.8% of groups and Adaptive Rubrics on 56.8% / 31.4%, against 11.8% / 3.0% for CER@20.
The closest-to-controlled TTS comparison is CER@20 against OpenHands Critic@20. CER@20 is a single round with no beam parents or regression stopping, though tie-breaking is still active. Like OpenHands Critic@20, it scores 20-step prefixes and selects one. CER@20 reaches 64.8% / 68.6% RM@8, against 57.8% / 64.2% (Table 1). This is the main reason I see value in the work.
S2. Candid reporting of negative evidence. The paper reports results that limit its own scope. These include the moderate online agreement with the verifier (Sec. 6.2), the fact that scores are not comparable across cutoffs and so cannot serve as a process reward (App. F.5), and the weaker results at short RL budgets (App. F.6). This makes it possible to judge what the method actually delivers.
Critical weaknesses
W1. The evidence does not establish the central claim that CER predicts the terminal reward early, and the main comparison lacks the most relevant baselines.
(a) Early prediction is not measured where the headline claims are made, and the direct measurements that exist are modest. The headline results are TTS results. For TTS, the paper never reports how well CER scores at a fixed cutoff rank the terminal outcomes (e.g. AUROC or within-group pairwise accuracy). It also never compares this against the baseline scorers on the same prefixes; only the downstream RM@k is reported.
Sec. 4.3 and App. F.1 give two reasons:
- a prefix's value is latent, and a single completion is only one Bernoulli sample of it;
- Acc_pair is not comparable, because CER's beam groups differ from the baselines' candidate sets.
Neither reason prevents scoring the same fixed prefixes with CER and with the baselines, for example the first t steps of the 8 shared rollouts. Label noise then pulls all scorers toward chance alike, so the comparison remains valid. On a subset, V(h_t) could also be estimated directly with several completions per prefix.
The direct measurements the paper does report concern the RL judge, and they are modest:
- online pairwise agreement with the verifier is 60.9% (53.9–70.1% over training; ties count as incorrect; Sec. 6.2);
- offline held-out accuracy after weight fitting is 72.88%. This is on a pair-level split in which held-out pairs share trajectories with the fitting pairs (App. D), so it is likely optimistic.
F.5 additionally shows that the scores are not comparable across cutoffs. In that single-rollout analysis with the Nemotron judge, the mean score rises from 0.431 to 0.852 and the BCE from 0.755 to 3.514 between steps 10 and 100. CER is therefore a within-cutoff ranker rather than a predictor of the terminal reward. This is narrower than the title and abstract suggest.
(b) The gain from being early is small, and the headline numbers are not early.
- At @20, CER exceeds the strongest complete-trajectory reranker by 1.4 pp on Nemotron, at 15% of its tokens. On Qwen it falls 0.8 pp below that reranker (68.6 vs. 69.4 RM@8), at about 35% of its tokens.
- The headline +4.2 / +2.0 pp come from @60. There, only 188/500 Nemotron tasks and 17/500 Qwen tasks still have all candidates unsubmitted (Table 6), so for most tasks the comparison involves finished trajectories. On the fully unsubmitted subsets, CER leads Adaptive Rubrics by about 2 pp at every Nemotron budget and on Qwen @40 (98 tasks), but trails it on Qwen @20 (63.7 vs. 64.8).
- In RL, the 40-turn cutoff covers roughly two thirds of an average TMax trajectory (60.2 turns), and 27.1% of CER rollouts have already submitted within it. On fold 0, budgets of 10 and 20 turns fall below TMax (48.1 / 48.3% vs. 49.7%), and 30 turns only matches it (49.8%) (F.6, single runs). The RL benefit appears only once the prefix contains editing and validation (median entry steps 22 and 28, F.6), so it is not particularly early.
- None of the TTS numbers has a confidence interval (App. F.1). Differences of 1–2 pp on 500 tasks from a single run are within sampling noise.
(c) Beyond @20, the gains are confounded with the search procedure, and the most relevant baselines are missing. From @40 on, CER runs inside its own multi-round beam search: beam parents, regression stopping and freshly sampled continuations. Tie-breaking rubrics are active from @20. No baseline uses search. OpenHands Critic@k scores the first k steps of the same 8 independently sampled rollouts, and the other baselines score the complete trajectories (App. E.1).
Table 8 indicates that both the scorer and the search contribute: removing sibling comparison costs 4.0 / 3.0 pp (Nemotron / Qwen), and removing beam parents costs 2.3 pp. However, no baseline scorer is ever placed inside the same search. Table 1 therefore cannot show how much of the @40–@80 gain comes from the early reward itself.
Several methods already prune or guide partial generations with a reward or value signal. They are neither compared against nor cited:
- Speculative Rejection (Sun et al., Fast Best-of-N Decoding via Speculative Rejection, NeurIPS 2024). During Best-of-N decoding, it scores partial generations with a reward model and halts the lowest-scoring fraction, so compute goes to the promising candidates. This is the direct analogue of CER's TTS use. An agent-step version (score every 20 steps, halt the bottom fraction) with SWE-RM, OpenHands Critic or Adaptive Rubrics as the partial-trajectory scorer, run under the same token budget, would show whether existing scorers already gain as much as CER once they are given a pruning search. Together with Q1(b), it would separate CER's scorer from its search.
- SWE-Search (Antoniades et al., SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement, ICLR 2025). It runs MCTS over a software agent, guided by a prompted LLM value agent. The value agent scores intermediate states with a numerical value and a written explanation, and a discriminator agent selects among the finished solutions. This is training-free LLM value estimation on unfinished SWE trajectories combined with search, which is close to CER's TTS setting. The differences are that SWE-Search scores single MCTS nodes on an absolute scale with fixed, state-specific value prompts, and has no adaptive rubrics, sibling comparison or experience bank. Its value prompt is also an obvious alternative scorer for Algorithm 1.
- SWE-TRACE (Han et al., SWE-TRACE: Optimizing Long-Horizon SWE Agents Through Rubric Process Reward Models and Heuristic Test-Time Scaling, arXiv:2604.14820, 2026). It generates issue-specific weighted rubrics and scores provisional partial trajectories with a learned rubric evaluator, which it uses to prune candidate actions at test time on SWE-bench Verified. The same rubric PRM also shapes GRPO rewards. It is an arXiv preprint, so I do not expect a full reproduction. However, the overlap with both of CER's uses (rubric-scored prefixes for TTS and rubric rewards for RL on SWE-bench Verified) should at least be discussed, and ideally its prefix scorer should be compared.
Without these comparisons, it is unclear whether CER improves over existing prefix-level value estimation plus search. It may mainly improve over rerankers that are given no search at all.
W2. The RL improvement is not statistically convincing, and the token savings do not show that early, rubric-rewarded training is sufficient or what it costs relative to the full-rollout scheme.
(a) The improvement is small relative to the reported uncertainty, and that uncertainty is incomplete.
- CER and TMax reach 51.6 ± 0.9 and 49.7 ± 1.1 (Table 2). On fold 1 the intervals overlap heavily (49.8 ± 2.1 vs. 49.1 ± 2.7), and the paper itself calls the gain "a small margin".
- The intervals cover only evaluation sampling from one training run per method and fold (Sec. 4.3), not training variance. In long-horizon agent RL, training variance can easily be as large as the 1.9 pp gap.
- The 3.5 pp gap at step 384 is measured on the 50-task validation split, which is also used for checkpoint selection.
- The setting is narrow: 250 training tasks, on which TMax saturates within 128–256 steps (Sec. 5.2).
- CER also draws on rubrics that GPT-5.6 Sol distills from verified rollouts of the training tasks. TMax gets no such teacher.
(b) The token saving is not established for the reported results, and it is unclear when it is free.
- The 52.7% figure is not measured at the checkpoints behind Table 2. It is measured at the matched 384-step checkpoint (validation). The test numbers in Table 2 come from each method's best-validation checkpoint. TMax peaks by 128–256 steps, while CER is extended to 512 steps, which costs about as much as TMax's 256 steps (Sec. 5.2). At the checkpoints actually compared in Table 2, the saving could therefore be much smaller, or absent.
- Only online tokens are counted. The offline stage consists of rolling out and verifying historical groups for every training task, then running GPT-5.6 Sol rubric synthesis with two refinement passes. It appears only as a formula (Eq. 5) and is never quantified. The RL rubrics are task-specific, so this cost is paid per training task and cannot be amortized across tasks as the Limitations section suggests.
- Short prefixes are not shown to be sufficient. TMax-40 ≈ TMax shows that restricting gradients to the first 40 turns costs nothing when the reward is still computed on full rollouts. In CER, the reward itself is judged from the 40-turn prefix. The only evidence on how far this can be pushed is F.6, where performance drops at shorter cutoffs: 48.1 / 48.3 / 49.8% at 10 / 20 / 30 turns, against 52.7% at 40. F.6 reports no token counts, so the trade-off between tokens and performance across cutoffs is not characterized. There is also no breakdown for tasks that need more than 40 turns. Apart from the full-rollout length trend in Fig. 5(d), the paper does not analyze behavior after turn 40.
- In 40.5% of the groups CER trains on, all eight full trajectories have the same binary resolution (F.4). In these groups binary resolution gives no signal, so the update rests entirely on the judge's within-group ordering. The paper does not report whether the passed-test rate (the RL verifier reward, App. E.1) still differs within these groups. Nor does it say whether such pairs enter the 60.9% agreement in Fig. 4. There is therefore no evidence that the rubric signal in these groups is more than noise. The paper's own batch-stability condition (App. C.3: no systematic directional bias across groups) is never checked for the RL judge (GPT-5.6 Luna). F.5 shows a length-dependent score drift for a different judge, which suggests that such biases can occur.
Questions for authors
Q1–Q5 could change my assessment. Q6 is a clarification.
Q1. Baselines and the search confound. (a) Can you compare against Speculative Rejection (agent-step version) and SWE-Search at matched token budgets, and discuss SWE-TRACE (ideally comparing its prefix scorer with CER's)? (b) Can you plug the existing scorers (SWE-RM, OpenHands Critic, Adaptive Rubrics) into Algorithm 1 in place of CER's rubric score? (c) CER@20 is effectively a single-round selection. For k ∈ {40, 60, 80}, can you report CER without search, scoring the first k steps of the same 8 shared rollouts and selecting one, as for OpenHands Critic@k? If CER's scorer still wins inside the same search, and also without search at larger k, W1(c) would be largely resolved.
Q2. Direct evidence for early prediction in TTS. (a) At fixed cutoffs t ∈ {20, 40, 60}, can you report AUROC and within-group pairwise accuracy against terminal outcomes, for CER and for the baselines, on the same prefixes (e.g. the first t steps of the 8 shared rollouts)? (b) On a subset, can you estimate V(h_t) with 4–8 completions per prefix and report the same metrics? (c) Table 6 gives the task-level counts. What fraction of the individual prefixes at each cutoff is already finished, and how do the metrics in (a) change when those prefixes are excluded? (d) The 72.88% held-out accuracy (App. D) uses a pair-level split with shared trajectories. What is the accuracy with a trajectory-level split?
Q3. TTS variance. How much does RM@8 vary across repeated TTS runs with different sampling seeds, for example on the 100-task ablation subset? Are the 1–2 pp differences in Table 1 and the 0.5–4 pp differences in Table 8 larger than this variation?
Q4. RL significance and the source of the gain. (a) Can you report several independent training seeds per fold, or at least for one fold, together with a paired test? (b) TMax-40 separates update truncation from the full-rollout baseline, but it still uses full-rollout verifier rewards. Can you add the missing cell of the design: the same GPT-5.6 rubric reward judged on full, untruncated trajectories? It would show whether the gain comes from the rubric reward or from judging early. It would also show how much performance the early cutoff costs relative to the full-rollout scheme.
Q5. Cost and sufficiency of early training. (a) Which checkpoint (training step) is behind each Table 2 entry, and what are its cumulative online tokens? What is the saving relative to TMax at those checkpoints? (b) What is the total cost including the offline stage (historical rollouts with verification, GPT-5.6 Sol rubric generation and refinement)? Please give it in tokens and, ideally, also in GPU-hours, API cost and wall-clock time. How does CER compare with TMax once this cost is included? (c) What are the token costs of the 10/20/30-turn runs in F.6? With them, resolved rate could be plotted against total tokens across cutoffs. (d) How do the resolved rates of CER and TMax compare when split by the number of turns a trajectory needs (≤ 40 vs. > 40)? (e) In the groups with identical binary resolution (40.5% of CER's training groups), does the passed-test rate still differ? What is the rubric–verifier agreement restricted to these groups?
Q6. Clarification. (a) In Table 6, does "every candidate trajectory" refer to the 8 shared baseline rollouts or to CER's own prefixes? (b) How is RM@4 defined for CER, given that CER returns one completed trajectory per budget?
Thanks for the detailed rebuttal. W1: Thanks for running the extra experiments. The direct pairwise-accuracy numbers answer what I was asking, and CER does look better than the other value models, especially in the pooled column. W2/W3: Thanks for the explanations, these clear up my concerns.