acceptodds
Withdrawn from ICLR 2027

When Unfinished Counts as Wrong: Generation Budgets Define the Target in Reasoning Monitor Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reasoning monitors estimate whether an ongoing reasoning trajectory will eventually yield a correct answer, enabling early stopping, verification, and runtime oversight. Many evaluations obtain the target label by matching answers from generations produced under a fixed token budget. Under a larger-budget prediction objective, however, this procedure can assign the same negative label to completed errors and trajectories that simply ran out of budget before producing their intended final answer. Our key observation is that the generation budget is therefore part of the prediction target, not merely an implementation detail. We audit this dependence across seven open models and 19 model–dataset configurations, comparing 256-token labels with paired 2048-token operational reference labels for 2,886 trajectories while holding monitor scores fixed. Heuristic truncation reaches 92.5%, label agreement falls to 28.5%, and point-estimate model rankings change. On DeepSeek-R1-7B/GSM8K, a last-step probe scores 0.892 AUC against short-budget labels but 0.568 against the reference labels. Budget sweeps and completion controls are consistent with completion-related confounding, whereas reference-label retraining yields limited, configuration-dependent recovery. These results expose a target-validity failure mode in reasoning-monitor evaluation and motivate paired label auditing before monitor performance is interpreted as longer-horizon correctness prediction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reasoning monitoring, reasoning language models, LLM evaluation, truncation bias, test-time reasoning, hidden-state probing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_fYxNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how generation budgets affect the reliability of evaluating reasoning monitors. The authors argue that existing evaluation protocols may conflate genuinely incorrect reasoning with unfinished generations when correctness labels are derived from outputs produced under a limited token budget. To study this issue, the authors introduce a paired reference-label audit that compares correctness labels obtained under a 256-token budget with those generated under a 2,048-token budget. The results show substantial label disagreements and changes in measured monitor performance.

Critical strengths
  • The paper identifies an important and meaningful issue in reasoning monitor evaluation.
  • The analysis is comprehensive and reveals substantial effects. The authors evaluate several models across 19 configurations. The experiments also demonstrate that changing the label construction procedure can significantly alter monitor AUC.
Critical weaknesses
  • The experiments compare a relatively restrictive 256-token budget against a 2,048-token reference. The short initial budget would make it easy to produce truncation for reasoning models. The authors include additional budget sweeps and experiments in Figure 2, but these analyses cover 50 examples with limited configuration. I suggest that the authors include a broader evaluation under representative practical protocols.
  • It is good that the authors acknowledge that 2,048-token outputs are not definitive ground truth. But this limitation raises concerns about the interpretation of some results. In Table 6, even with larger budgets, some trajectories remain unfinished. Therefore, the reported reference-label AUCs in some configurations may still depend on the chosen outcome horizon. A clearer separation between configurations with sufficiently stable references and those with unresolved outcomes would make the conclusions easier to interpret.
  • Given the limited and inconsistent improvements from reference-label retraining, what do the authors believe is the main bottleneck for longer-horizon correctness prediction? I would like to see stronger analysis.
Questions for authors

Please refer to weaknesses.

Reviewer_w63mOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper raises a problem that may appear when auditing and monitoring a reasoning trace. The main argument is that to build a classifier predicting the correctness of a rollout, some methods construct the classification label based on whether a reasoning rollout has the gold answer. But if the rollouts are short, most labels for unfinished rollouts are simply incorrect. So instead of learning correctness, the model learns a shortcut and predicts whether a rollout finishes or not. To validate this argument and through experiments over different probing predictors, the authors show that this problem appears in small budgets (for example, 256) and starts to disappear as the budget increases.

Critical strengths
  1. Although the main message is simple, it is good to have it documented. In particular, the paired audit with a fixed monitor is a clean, cheap check that others can use.
Critical weaknesses
  1. My main problem with this work is that it is limited in novelty, and the message is not significant for this venue. The paper argues that when constructing labels, if we count unfinished rollout as incorrect in our label construction phase, we may get an artificial boost in our probing model since the model may learn a shortcut by predicting whether the rollout is finished or not. But, first, this mostly happens for cases that rollouts are short (256 or 512 tokens) and disappears at 1024. Secondly, using this is not standard as only the Yuan et al. (2026) and David (2025) works use this kind of label, and both seems to be unreviewed preprints.

  2. The experiments are very limited in terms of datasets. In particular, the paper is missing results on harder datasets such as AIME.

Questions for authors
  1. Please refer to weaknesses.

  2. As partly mentioned by the authors and from my own experience, for monitoring purposes, many methods use a PRM and a probe is used mostly to predict the future correctness label, for example in pruning applications. In appendix D.2, the authors use ThinkPRM-1.5B to predict the labels for rollouts. However, for most models in Table 14, the performance of short and reference is close, unlike the probing models. So, what is the reason behind this? I think in general providing justifications on why PRM doesn't show the same behavior as probe can strengthen the position of the paper.

Reviewer_dD6uOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how generation budgets affect the evaluation of reasoning monitors. The paper demonstrates this point by evaluating the monitor trained on 256 token generation budget with longer generation budget while keeping the monitor score fixed. Across seven models and three datasets, many unfinished outputs become correct with more tokens, and monitor AUC often drops when evaluated against these longer-budget outcomes.

Critical strengths

The experiment settings mostly make sense. The experiments did not stop at the level of saying that the fixed monitor scores under a 256-token generation budget drop a lot when considering accuracy for 2048 tokens. The authors have tried to incorporate more in-depth analysis, though the results seem to be a bit weak.

Critical weaknesses
  1. The experimental setup is questionable. It is well-known that reasoning models often require thousands of tokens to solve these mathematical problems, so substantial truncation under a 256-token cap is unsurprising. It is unclear what insight an entire section quantifying this effect really provides. In addition, training monitors only on such short outputs also creates a restrictive setting.
  2. The settings are not presented clearly. For example, how the monitors were set up and trained is only partially documented. The authors also did not discuss whether these monitors are well-studied in the literature, or whether they simply came up with some ways to train these monitors.
Questions for authors

I am skeptical about the interpretation of the retraining results. The paper reports limited and inconsistent AUC improvements after retraining monitors to predict the reference labels. Why is this? Does it reflect limitations of the chosen monitoring methods or training data, or does a 256-token prefix contain insufficient information to predict success within 2,048 tokens? Have the authors evaluated established methods from the literature specifically designed to predict eventual correctness from partial reasoning trajectories?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.