- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how generation budgets affect the reliability of evaluating reasoning monitors. The authors argue that existing evaluation protocols may conflate genuinely incorrect reasoning with unfinished generations when correctness labels are derived from outputs produced under a limited token budget. To study this issue, the authors introduce a paired reference-label audit that compares correctness labels obtained under a 256-token budget with those generated under a 2,048-token budget. The results show substantial label disagreements and changes in measured monitor performance.
Critical strengths
- The paper identifies an important and meaningful issue in reasoning monitor evaluation.
- The analysis is comprehensive and reveals substantial effects. The authors evaluate several models across 19 configurations. The experiments also demonstrate that changing the label construction procedure can significantly alter monitor AUC.
Critical weaknesses
- The experiments compare a relatively restrictive 256-token budget against a 2,048-token reference. The short initial budget would make it easy to produce truncation for reasoning models. The authors include additional budget sweeps and experiments in Figure 2, but these analyses cover 50 examples with limited configuration. I suggest that the authors include a broader evaluation under representative practical protocols.
- It is good that the authors acknowledge that 2,048-token outputs are not definitive ground truth. But this limitation raises concerns about the interpretation of some results. In Table 6, even with larger budgets, some trajectories remain unfinished. Therefore, the reported reference-label AUCs in some configurations may still depend on the chosen outcome horizon. A clearer separation between configurations with sufficiently stable references and those with unresolved outcomes would make the conclusions easier to interpret.
- Given the limited and inconsistent improvements from reference-label retraining, what do the authors believe is the main bottleneck for longer-horizon correctness prediction? I would like to see stronger analysis.
Questions for authors
Please refer to weaknesses.