acceptodds
Under review as a conference paper at ICLR 2027

Before the Rollout Ends: Early Terminal Reward Prediction for Long-horizon Coding Agents

Abstract

Long-horizon coding agents receive verifiable rewards only after completing expensive sequences of tool calls. This increases inference cost, amplifies early wrong hypotheses, and can lead to sparse terminal reward and unstable training. We introduce Contextual Early Reward (CER), which predicts terminal reward through behavioral evidence in a trajectory prefix. CER synthesizes adaptive rubrics specific to the current task and stage through experiences summarized from related historical tasks. In test-time scaling on SWE-bench Verified, CER improves RM@8 over the strongest baseline by 4.2 percentage points (pp) on Nemotron 3 Ultra and 2.0 pp on Qwen 3.6 27B; on Nemotron, it takes only 15.2% tokens to match the best baseline performance. In RL training experiments, CER exceeds full-rollout TMax by 1.9 pp while using 52.7% fewer online policy-and-judge tokens. Together, CER provides an interpretable, efficient, and dense evaluation method for long-horizon coding agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long Horizon Agent, Evaluation, Early Prediction, Coding Agent, Test-time Scaling, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_EuLYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Contextual Early Reward (CER), an LLM-judge method that predicts the terminal (test-based) reward of an unfinished coding-agent trajectory from its prefix. For a group of prefixes sampled for the same task, CER generates one shared set of weighted, task- and stage-specific rubrics, and scores every prefix with criterion-level evidence. For TTS the scores only need to preserve the within-group order (Eq. 2). For RL the paper adds a batch-stability condition (App. C.3). The paper studies two uses:

  1. Test-time scaling (TTS). The policy model generates its own rubrics, guided by an experience bank that GPT-5.6 Sol distills from verified historical groups (BM25 retrieval, leave-one-out). The scores drive a beam-style search: expansion every 20 steps, regression stopping, and tie-breaking rubrics. On SWE-bench Verified, CER@60 reaches 67.6% RM@8 on Nemotron 3 Ultra and 71.4% on Qwen 3.6 27B. The strongest reranker of 8 complete trajectories reaches 63.4% and 69.4%. On Nemotron, CER@20 already exceeds that reranker by 1.4 pp while using 15.3% of its tokens.
  2. RL. GPT-5.6 Sol builds task-specific rubrics offline from verified historical rollouts, and their weights are fitted to pairwise agreement with the verifier. Online rollouts of Qwen 3.5 9B are cut at 40 turns, and GPT-5.6 Luna scores them to give group-relative advantages in the TMax/DPPO objective. Over three best-effort repository-separated folds (250 train / 50 val / 200 test tasks), CER reaches a 51.6% test resolved rate, compared with 49.7% for TMax and 49.8% for TMax-40. At the matched 384-step checkpoint, CER uses 52.7% fewer online policy-and-judge tokens than TMax.

Further analyses cover experience transfer across models, rubric–verifier agreement (60.9%), a rubric taxonomy, component ablations on a 100-task subset, and a negative result for using CER as a process reward.

Critical strengths

S1. A relevant problem and a sensible, interpretable design with some controlled evidence. Long-horizon SWE agents are expensive to roll out. The paper's design scores a group of sibling prefixes under one shared rubric set and requires evidence for each criterion, which is a reasonable choice. The ablation supports it: removing sibling comparison costs 3.0–4.0 pp (Table 8). The design also sharply lowers the top-tie rate relative to the adaptive-rubric judges: Agentic Rubrics ties on 57.2% / 88.8% of groups and Adaptive Rubrics on 56.8% / 31.4%, against 11.8% / 3.0% for CER@20.

The closest-to-controlled TTS comparison is CER@20 against OpenHands Critic@20. CER@20 is a single round with no beam parents or regression stopping, though tie-breaking is still active. Like OpenHands Critic@20, it scores 20-step prefixes and selects one. CER@20 reaches 64.8% / 68.6% RM@8, against 57.8% / 64.2% (Table 1). This is the main reason I see value in the work.

S2. Candid reporting of negative evidence. The paper reports results that limit its own scope. These include the moderate online agreement with the verifier (Sec. 6.2), the fact that scores are not comparable across cutoffs and so cannot serve as a process reward (App. F.5), and the weaker results at short RL budgets (App. F.6). This makes it possible to judge what the method actually delivers.

Critical weaknesses

W1. The evidence does not establish the central claim that CER predicts the terminal reward early, and the main comparison lacks the most relevant baselines.

(a) Early prediction is not measured where the headline claims are made, and the direct measurements that exist are modest. The headline results are TTS results. For TTS, the paper never reports how well CER scores at a fixed cutoff rank the terminal outcomes (e.g. AUROC or within-group pairwise accuracy). It also never compares this against the baseline scorers on the same prefixes; only the downstream RM@k is reported.

Sec. 4.3 and App. F.1 give two reasons:

  • a prefix's value is latent, and a single completion is only one Bernoulli sample of it;
  • Acc_pair is not comparable, because CER's beam groups differ from the baselines' candidate sets.

Neither reason prevents scoring the same fixed prefixes with CER and with the baselines, for example the first t steps of the 8 shared rollouts. Label noise then pulls all scorers toward chance alike, so the comparison remains valid. On a subset, V(h_t) could also be estimated directly with several completions per prefix.

The direct measurements the paper does report concern the RL judge, and they are modest:

  • online pairwise agreement with the verifier is 60.9% (53.9–70.1% over training; ties count as incorrect; Sec. 6.2);
  • offline held-out accuracy after weight fitting is 72.88%. This is on a pair-level split in which held-out pairs share trajectories with the fitting pairs (App. D), so it is likely optimistic.

F.5 additionally shows that the scores are not comparable across cutoffs. In that single-rollout analysis with the Nemotron judge, the mean score rises from 0.431 to 0.852 and the BCE from 0.755 to 3.514 between steps 10 and 100. CER is therefore a within-cutoff ranker rather than a predictor of the terminal reward. This is narrower than the title and abstract suggest.

(b) The gain from being early is small, and the headline numbers are not early.

  • At @20, CER exceeds the strongest complete-trajectory reranker by 1.4 pp on Nemotron, at 15% of its tokens. On Qwen it falls 0.8 pp below that reranker (68.6 vs. 69.4 RM@8), at about 35% of its tokens.
  • The headline +4.2 / +2.0 pp come from @60. There, only 188/500 Nemotron tasks and 17/500 Qwen tasks still have all candidates unsubmitted (Table 6), so for most tasks the comparison involves finished trajectories. On the fully unsubmitted subsets, CER leads Adaptive Rubrics by about 2 pp at every Nemotron budget and on Qwen @40 (98 tasks), but trails it on Qwen @20 (63.7 vs. 64.8).
  • In RL, the 40-turn cutoff covers roughly two thirds of an average TMax trajectory (60.2 turns), and 27.1% of CER rollouts have already submitted within it. On fold 0, budgets of 10 and 20 turns fall below TMax (48.1 / 48.3% vs. 49.7%), and 30 turns only matches it (49.8%) (F.6, single runs). The RL benefit appears only once the prefix contains editing and validation (median entry steps 22 and 28, F.6), so it is not particularly early.
  • None of the TTS numbers has a confidence interval (App. F.1). Differences of 1–2 pp on 500 tasks from a single run are within sampling noise.

(c) Beyond @20, the gains are confounded with the search procedure, and the most relevant baselines are missing. From @40 on, CER runs inside its own multi-round beam search: beam parents, regression stopping and freshly sampled continuations. Tie-breaking rubrics are active from @20. No baseline uses search. OpenHands Critic@k scores the first k steps of the same 8 independently sampled rollouts, and the other baselines score the complete trajectories (App. E.1).

Table 8 indicates that both the scorer and the search contribute: removing sibling comparison costs 4.0 / 3.0 pp (Nemotron / Qwen), and removing beam parents costs 2.3 pp. However, no baseline scorer is ever placed inside the same search. Table 1 therefore cannot show how much of the @40–@80 gain comes from the early reward itself.

Several methods already prune or guide partial generations with a reward or value signal. They are neither compared against nor cited:

  • Speculative Rejection (Sun et al., Fast Best-of-N Decoding via Speculative Rejection, NeurIPS 2024). During Best-of-N decoding, it scores partial generations with a reward model and halts the lowest-scoring fraction, so compute goes to the promising candidates. This is the direct analogue of CER's TTS use. An agent-step version (score every 20 steps, halt the bottom fraction) with SWE-RM, OpenHands Critic or Adaptive Rubrics as the partial-trajectory scorer, run under the same token budget, would show whether existing scorers already gain as much as CER once they are given a pruning search. Together with Q1(b), it would separate CER's scorer from its search.
  • SWE-Search (Antoniades et al., SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement, ICLR 2025). It runs MCTS over a software agent, guided by a prompted LLM value agent. The value agent scores intermediate states with a numerical value and a written explanation, and a discriminator agent selects among the finished solutions. This is training-free LLM value estimation on unfinished SWE trajectories combined with search, which is close to CER's TTS setting. The differences are that SWE-Search scores single MCTS nodes on an absolute scale with fixed, state-specific value prompts, and has no adaptive rubrics, sibling comparison or experience bank. Its value prompt is also an obvious alternative scorer for Algorithm 1.
  • SWE-TRACE (Han et al., SWE-TRACE: Optimizing Long-Horizon SWE Agents Through Rubric Process Reward Models and Heuristic Test-Time Scaling, arXiv:2604.14820, 2026). It generates issue-specific weighted rubrics and scores provisional partial trajectories with a learned rubric evaluator, which it uses to prune candidate actions at test time on SWE-bench Verified. The same rubric PRM also shapes GRPO rewards. It is an arXiv preprint, so I do not expect a full reproduction. However, the overlap with both of CER's uses (rubric-scored prefixes for TTS and rubric rewards for RL on SWE-bench Verified) should at least be discussed, and ideally its prefix scorer should be compared.

Without these comparisons, it is unclear whether CER improves over existing prefix-level value estimation plus search. It may mainly improve over rerankers that are given no search at all.

W2. The RL improvement is not statistically convincing, and the token savings do not show that early, rubric-rewarded training is sufficient or what it costs relative to the full-rollout scheme.

(a) The improvement is small relative to the reported uncertainty, and that uncertainty is incomplete.

  • CER and TMax reach 51.6 ± 0.9 and 49.7 ± 1.1 (Table 2). On fold 1 the intervals overlap heavily (49.8 ± 2.1 vs. 49.1 ± 2.7), and the paper itself calls the gain "a small margin".
  • The intervals cover only evaluation sampling from one training run per method and fold (Sec. 4.3), not training variance. In long-horizon agent RL, training variance can easily be as large as the 1.9 pp gap.
  • The 3.5 pp gap at step 384 is measured on the 50-task validation split, which is also used for checkpoint selection.
  • The setting is narrow: 250 training tasks, on which TMax saturates within 128–256 steps (Sec. 5.2).
  • CER also draws on rubrics that GPT-5.6 Sol distills from verified rollouts of the training tasks. TMax gets no such teacher.

(b) The token saving is not established for the reported results, and it is unclear when it is free.

  • The 52.7% figure is not measured at the checkpoints behind Table 2. It is measured at the matched 384-step checkpoint (validation). The test numbers in Table 2 come from each method's best-validation checkpoint. TMax peaks by 128–256 steps, while CER is extended to 512 steps, which costs about as much as TMax's 256 steps (Sec. 5.2). At the checkpoints actually compared in Table 2, the saving could therefore be much smaller, or absent.
  • Only online tokens are counted. The offline stage consists of rolling out and verifying historical groups for every training task, then running GPT-5.6 Sol rubric synthesis with two refinement passes. It appears only as a formula (Eq. 5) and is never quantified. The RL rubrics are task-specific, so this cost is paid per training task and cannot be amortized across tasks as the Limitations section suggests.
  • Short prefixes are not shown to be sufficient. TMax-40 ≈ TMax shows that restricting gradients to the first 40 turns costs nothing when the reward is still computed on full rollouts. In CER, the reward itself is judged from the 40-turn prefix. The only evidence on how far this can be pushed is F.6, where performance drops at shorter cutoffs: 48.1 / 48.3 / 49.8% at 10 / 20 / 30 turns, against 52.7% at 40. F.6 reports no token counts, so the trade-off between tokens and performance across cutoffs is not characterized. There is also no breakdown for tasks that need more than 40 turns. Apart from the full-rollout length trend in Fig. 5(d), the paper does not analyze behavior after turn 40.
  • In 40.5% of the groups CER trains on, all eight full trajectories have the same binary resolution (F.4). In these groups binary resolution gives no signal, so the update rests entirely on the judge's within-group ordering. The paper does not report whether the passed-test rate (the RL verifier reward, App. E.1) still differs within these groups. Nor does it say whether such pairs enter the 60.9% agreement in Fig. 4. There is therefore no evidence that the rubric signal in these groups is more than noise. The paper's own batch-stability condition (App. C.3: no systematic directional bias across groups) is never checked for the RL judge (GPT-5.6 Luna). F.5 shows a length-dependent score drift for a different judge, which suggests that such biases can occur.
Questions for authors

Q1–Q5 could change my assessment. Q6 is a clarification.

Q1. Baselines and the search confound. (a) Can you compare against Speculative Rejection (agent-step version) and SWE-Search at matched token budgets, and discuss SWE-TRACE (ideally comparing its prefix scorer with CER's)? (b) Can you plug the existing scorers (SWE-RM, OpenHands Critic, Adaptive Rubrics) into Algorithm 1 in place of CER's rubric score? (c) CER@20 is effectively a single-round selection. For k ∈ {40, 60, 80}, can you report CER without search, scoring the first k steps of the same 8 shared rollouts and selecting one, as for OpenHands Critic@k? If CER's scorer still wins inside the same search, and also without search at larger k, W1(c) would be largely resolved.

Q2. Direct evidence for early prediction in TTS. (a) At fixed cutoffs t ∈ {20, 40, 60}, can you report AUROC and within-group pairwise accuracy against terminal outcomes, for CER and for the baselines, on the same prefixes (e.g. the first t steps of the 8 shared rollouts)? (b) On a subset, can you estimate V(h_t) with 4–8 completions per prefix and report the same metrics? (c) Table 6 gives the task-level counts. What fraction of the individual prefixes at each cutoff is already finished, and how do the metrics in (a) change when those prefixes are excluded? (d) The 72.88% held-out accuracy (App. D) uses a pair-level split with shared trajectories. What is the accuracy with a trajectory-level split?

Q3. TTS variance. How much does RM@8 vary across repeated TTS runs with different sampling seeds, for example on the 100-task ablation subset? Are the 1–2 pp differences in Table 1 and the 0.5–4 pp differences in Table 8 larger than this variation?

Q4. RL significance and the source of the gain. (a) Can you report several independent training seeds per fold, or at least for one fold, together with a paired test? (b) TMax-40 separates update truncation from the full-rollout baseline, but it still uses full-rollout verifier rewards. Can you add the missing cell of the design: the same GPT-5.6 rubric reward judged on full, untruncated trajectories? It would show whether the gain comes from the rubric reward or from judging early. It would also show how much performance the early cutoff costs relative to the full-rollout scheme.

Q5. Cost and sufficiency of early training. (a) Which checkpoint (training step) is behind each Table 2 entry, and what are its cumulative online tokens? What is the saving relative to TMax at those checkpoints? (b) What is the total cost including the offline stage (historical rollouts with verification, GPT-5.6 Sol rubric generation and refinement)? Please give it in tokens and, ideally, also in GPU-hours, API cost and wall-clock time. How does CER compare with TMax once this cost is included? (c) What are the token costs of the 10/20/30-turn runs in F.6? With them, resolved rate could be plotted against total tokens across cutoffs. (d) How do the resolved rates of CER and TMax compare when split by the number of turns a trajectory needs (≤ 40 vs. > 40)? (e) In the groups with identical binary resolution (40.5% of CER's training groups), does the passed-test rate still differ? What is the rubric–verifier agreement restricted to these groups?

Q6. Clarification. (a) In Table 6, does "every candidate trajectory" refer to the 8 shared baseline rollouts or to CER's own prefixes? (b) How is RM@4 defined for CER, given that CER returns one completed trajectory per budget?

Reviewer_sobbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Contextual Early Reward (CER) for evaluating long-horizon coding agents before their trajectories terminate. CER uses weighted, task- and stage-specific rubrics to score observable prefix behaviors through LLM judging. For test-time scaling, candidate comparisons and retrieved historical evaluation experiences guide rubric generation and trajectory search. For reinforcement learning, teacher-generated rubrics and weights are constructed offline and held fixed to provide rewards for truncated online rollouts. Experiments on SWE-bench Verified evaluate test-time search with Nemotron 3 Ultra and Qwen 3.6 27B, and three-fold RL training with Qwen 3.5 9B. CER reports RM@8 improvements of 4.2 and 2.0 percentage points over the strongest execution-free baselines, respectively. In RL, it improves mean test success from 49.7% to 51.6%, while reporting 52.7% fewer online policy-and-judge tokens. Additional analyses examine search components, cross-model experience transfer, and agreement with terminal rewards.

Critical strengths
  1. A practically useful approach to early trajectory evaluation: CER addresses the substantial cost of waiting for coding agents to finish before obtaining feedback. Its task- and stage-specific rubrics use observable behavioral evidence and historical judging experience to assess unfinished trajectories. This provides an interpretable mechanism for allocating rollout budgets and obtaining earlier training signals, making the problem formulation practically consequential.
  2. Empirical evidence across both search and RL settings: The paper evaluates test-time search on two policy models and RL on Qwen 3.5 9B, measuring actual task resolution rather than relying solely on LLM-assigned scores. The reported improvements over execution-free baselines, together with reduced online token usage, support the utility of early rubric rewards. The TMax-40 control is particularly useful for distinguishing the effects of prefix-only optimization from those of the reward design.
Critical weaknesses
  1. The weighting and aggregation choices need more justification: In the test-time search setting, the prompt in Appendix G asks the LLM to assign weights based on criterion importance, but does not explicitly explain how these weights will be used in Eq. (1). It is therefore unclear how the LLM chooses numerical weights that reflect appropriate tradeoffs between criteria. Eq. (1) uses a weighted average, but the paper does not clearly explain why this way of combining scores is appropriate. Explaining the weighting process and comparing it with uniform weighting and other aggregation rules would help show whether these choices lead to reliable trajectory rankings.
  2. The total cost advantage needs more evidence: Appendix A and A.1 discuss the upfront cost of collecting historical rollouts and building the experience and rubric banks, and explain how reuse could offset this cost. However, the comparison is mainly analytical. The paper does not report the actual construction cost or estimate how much reuse is needed for the online savings to outweigh it. Reporting these figures would clarify when CER offers a lower total cost in practice.
Questions for authors
  1. In the test-time search setting, does the LLM receive any instructions beyond Appendix G explaining how its weights will be combined in Eq. (1)? What motivates the use of a weighted average, and do the current experiments compare LLM-generated weights with uniform weighting or assess the stability of trajectory selections across repeated rubric generation?

  2. Appendix A.1 provides an analytical comparison of construction costs. Can the authors report the actual one-time cost of collecting and verifying historical rollouts and building the experience and rubric banks, including teacher-model calls? Based on these costs and the reported online savings, approximately how much reuse is needed for CER to become cheaper overall than the baselines?

Reviewer_99QLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a method for early terminal-reward prediction in long-horizon coding tasks, where verifiable rewards are usually available only after the full rollout is completed. Contextual Early Reward (CER) evaluates unfinished trajectory prefixes using task- and stage-specific rubrics, with applications to test-time scaling and reinforcement learning. For test-time scaling, CER retrieves judging experience from historical tasks and uses an LLM to generate shared adaptive rubrics for the candidate prefixes at each search round. These rubrics guide the selection of promising branches, while continuations scoring below their parent prefixes are pruned from further expansion. For reinforcement learning, a strong teacher model constructs task-specific rubrics offline using historical rollouts and their verifier rewards. During online training, a judge scores truncated rollouts using these rubrics, and the resulting rewards provide group-relative advantages for policy optimization without requiring complete rollouts. On SWE-bench Verified, CER improves RM@8 over the strongest execution-free baselines by 4.2 and 2.0 percentage points on Nemotron 3 Ultra and Qwen 3.6 27B, respectively. In RL experiments, CER achieves a 51.6% test resolved rate versus 49.7% for TMax, while reducing online policy-and-judge token usage by 52.7% at a matched training-step checkpoint.

Critical strengths

S1. The paper proposes a simple method to evaluate long-horizon coding agents before their rollouts are completed. The method uses explicit rubrics to provide interpretable early rewards and applies them to both test-time scaling and RL training. The experiments show that these early rewards can improve task performance while reducing online token costs. The authors also provide code to reproduce the experiments.

S2. In the test-time scaling experiments, CER outperforms trained reward models such as OpenHands Critic and SWE-RM on both evaluated models. It achieves these improvements without training a separate critic or reward model. Instead, it uses retrieved experience to generate task-specific rubrics and an LLM to score the prefixes. These rubrics adapt to different tasks and stages without requiring white-box access to the policy model. The criterion-level scores also make the evaluation more interpretable.

Critical weaknesses

W1. The paper’s central claim is reliable early terminal-reward prediction, but no TTS experiment reports a direct prediction-accuracy metric (only the downstream RM@k). Since CER’s scores are used to rank candidates, this could be measured directly via within-group ranking accuracy: once the candidates are completed, does CER rank a successful prefix above a failing one?

W2. The construction of the TTS experience bank is not clearly described. The paper describes the experience-generation procedure and the leave-one-out setup, but the scale and actual cost of constructing the TTS experience bank remain unclear. In particular, it does not explicitly report the total number of complete historical rollouts collected to build the TTS experience bank, the number of experiences retained after filtering, or the total tokens spent on bank construction.

W3. The RL improvement is small and may be an artifact of the small-data setup. CER beats the baselines by only about 2 % (Table 2), and the evidence is weak: there are only 3 folds, the per-fold numbers vary a lot, one fold is essentially a tie, and no multiple training seeds are reported (the CIs cover evaluation sampling only). More importantly, each fold trains on just 250 tasks, and the authors themselves say the baseline “quickly saturates” in this data-limited setting while CER benefits from “denser reward signals.” So CER’s advantage may come simply from denser rewards helping in a tiny-data regime, rather than from the rubric being a genuinely better signal.

Questions for authors
  1. In the RL setting, CER optimizes the teacher-assigned rubric weights to maximize pairwise agreement with verifier rewards, and reports an improvement from 66.41% to 72.88%. Does better pairwise agreement guarantee better downstream RL performance? Is there an experiment comparing RL performance with the original and optimized weights?

  2. In Table 1, you test different total search budgets of 20, 40, 60, and 80 turns, but the search interval is fixed at 20 turns per round, right? Why did you choose 20 turns? How sensitive is the method to this interval? Have you tried other intervals, such as 10 or 40 turns, while keeping the total search budget fixed?

  3. For the offline construction of task-specific rubrics in the RL setting (Section 3.3): the group size appears to be 8 (Table 5) and the number of refinement passes m = 3 (App. A.1). Is this correct? More importantly, how many rollout groups are collected per task to build the rubrics, and what is the total number of historical rollouts k? The construction cost (Eq. 5) is given only symbolically, so a concrete count would clarify the true offline cost.

  4. What is the actual cost of building the TTS experience bank, including collecting and verifying historical rollouts and generating and refining experiences with LLMs? How many labeled rollouts k and total tokens were used? Table 1 only reports the online token cost and excludes the cost of building this bank. If we include this cost and spread it across the tasks that reuse the bank, how many tasks must CER solve before its average token cost becomes lower than the baselines?

  5. The paper mentions that adaptive rubrics generated by a relatively weak model cannot match training with test-verifier rewards, but does not report the experimental results. How sensitive is CER to the teacher model used to generate rubrics? Have you tested using the same model as the rollout policy to generate rubrics for RL training?

  6. In Table 3, why does experience from the weaker model hurt the stronger model? Is the experience itself misleading, or does it cause the stronger model to change judgments that were already correct?

Reviewer_r3F7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Contextual Early Reward (CER), a rubric-based framework for evaluating unfinished trajectories of long-horizon coding agents. On SWE-bench Verified, CER improves test-time selection performance across two policies and achieves a mean test resolved rate of 51.6%, compared with 49.7% for TMax. Separately, a matched-checkpoint comparison reports 52.7% fewer online policy-and-judge tokens. The paper also analyzes component contributions, experience transfer, and limitations of prefix evaluation.

Critical strengths
  1. A useful approach to obtaining feedback before expensive coding trajectories terminate. CER addresses an important bottleneck in long-horizon agent evaluation and demonstrates applications in both test-time search and RL. The TTS improvements across two policies and the prefix-only training results provide meaningful evidence that contextual behavioral feedback can be useful before terminal verification. This practical contribution is the strongest consideration supporting acceptance.
  2. Informative analyses of contributing components and failure boundaries. The paper includes component ablations, cross-model experience transfer, cutoff-length experiments, and an explicit analysis of length-dependent scoring bias. These analyses identify both useful design choices and limitations, rather than presenting only favorable aggregate results. Their explanatory value strengthens the paper’s contribution even where the causal attribution of the RL gains remains incomplete.
Critical weaknesses
  1. End-to-end efficiency is not established. The reported 52.7% online token saving excludes historical rollout collection and teacher-based rubric construction for each training task. Without measured offline costs, the paper does not establish whether these savings offset the required preprocessing expense.
  2. Missing judge-matched rubric baselines for RL. The RL experiments compare CER only with test-verifier rewards, despite including several rubric-based baselines in TTS. Without a prefix-adapted rubric baseline using the same GPT-5.6 Luna judge and training setup, the contribution of CER’s task-specific rubrics remains unclear.
  3. Missing longer-prefix and full-trajectory rubric baselines. In Appendix F.6, RL performance improves throughout the tested 10–40-step range, with the best result at the longest cutoff. No longer-prefix or full-trajectory training with comparable rubric supervision is reported. The experiments therefore do not establish that early truncation provides a better performance–cost trade-off than longer-horizon rubric training.
Questions for authors

Clarification requests

  1. Offline trajectory provenance. Which model and checkpoint generated the historical rollouts used to construct the RL rubrics, and how many rollouts were collected per training task?
  2. End-to-end cost. What are the measured costs of historical rollout collection, terminal verification, and rubric generation/refinement? After including these costs, does CER remain cheaper than TMax within the reported training runs?
  3. Longer-horizon rubric training. With the same teacher and judge, how does CER perform when trained on longer prefixes or complete trajectories? Does the 40-step cutoff retain an advantage under matched token budgets? Suggestions for future work
  4. Adaptive truncation. Explore task-dependent stopping based on available diagnostic evidence or judging uncertainty, rather than a fixed cutoff. Such a mechanism should account for length-dependent scoring bias instead of directly comparing raw CER scores across steps.
  5. Judge-matched RL baseline. Under the same 40-step cutoff and RL configuration, how does CER compare with a fixed-rubric baseline, such as prefix-adapted OpenHands Rubrics, using the same GPT-5.6 Luna judge?
Reviewer_CSPoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether we can tell if a coding agent will succeed before its run is finished. The proposed method, CER, has an LLM judge score several unfinished runs of the same task side by side, using rubrics written for that task. The method is used in two ways:

  1. Test-time scaling. The policy model writes the rubrics, helped by "experiences" from past tasks that GPT-5.6 Sol summarized. The scores then guide a beam search.
  2. RL. GPT-5.6 Sol writes task-specific rubrics offline from past runs with known test results. During training, runs are cut at turn 40 and a cheaper judge scores them.

On SWE-bench Verified, CER improves RM@8 over the best baseline by 4.2 points (Nemotron 3 Ultra) and 2.0 points (Qwen 3.6 27B). In RL with Qwen 3.5 9B, it reaches 51.6% vs. 49.7% for TMax while using 52.7% fewer online token

Critical strengths
  1. The problem matters. Long agent runs are expensive, and many of them go wrong early. Using the same early score for both search and RL is a nice idea.

  2. Scoring sibling runs together under one shared rubric fits what both uses need: only the ranking inside a group matters. The ablation supports this. Removing the side-by-side comparison gives the largest drop (3.0 and 4.0 points, Table 8).

Critical weaknesses
  1. The baselines pick one of the same 8 finished runs, while CER runs a beam search that keeps sampling new runs. Part of the gain may come from the search, not from a better score. The paper does not test the same search with a baseline scorer.
  2. The experience bank is built from the other SWE-bench Verified tasks using their hidden test results, and the baselines get no such bank. The token savings also leave out the cost of building the bank and the RL rubrics.
  3. Table 1 has no error bars, and the ablations use only 100 tasks. The RL gain is 1.9 points from one run per fold, and the rubric scores agree with test results only about 61% of the time.
Questions for authors
  1. Can you run the same beam search (Algorithm 1) with a baseline scorer, such as SWE-RM, OpenHands Critic, or Adaptive Rubrics applied to unfinished runs, at the same step budget? This would show how much of the gain comes from the score and how much from the search.
  2. How do the test-time results change if the experience bank is built from data outside SWE-bench Verified? What happens if Adaptive Rubrics gets the same retrieved experiences? What exactly is in the "privileged diagnostic packet" used to write the experiences?
  3. For RL, which model produced the past runs used to build the rubrics, how many were there, and were they collected before or during training? What is the total cost once these runs and the GPT-5.6 Sol calls are included?
  4. Can you add significance tests for Table 1 and more RL seeds? A simple RL control, such as shuffling CER scores within each group, would also show whether the gain depends on the scores being correct.

Comments on the paper

1 comment

Thanks for the detailed rebuttal. W1: Thanks for running the extra experiments. The direct pairwise-accuracy numbers answer what I was asking, and CER does look better than the other value models, especially in the pooled column. W2/W3: Thanks for the explanations, these clear up my concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.