acceptodds
Under review as a conference paper at ICLR 2027

A Moving Ruler: Evaluation Comparability Across RLVR Checkpoints

Abstract

Checkpoint evaluations are routinely used to quantify the evolution of model behavior with reinforcement learning from verifiable rewards (RLVR). Naturally, a robust evaluation demands a consistent scoring protocol at every checkpoint, which we refer to as evaluation comparability. However, evaluation comparability across RLVR checkpoints is rarely analyzed, even though RLVR training changes the nature of the response that the evaluation receives. For example, in AllenAI's Olmo3.1 RL-Zero Math model, benchmark scores rise over training checkpoints. But we observe that most of the gain is a result of the model's ability to deliver an answer within the token budget, rather than an increased capability in solving problems. Furthermore, we observe that later checkpoints also stall less often, and the probability of repeating the final answer within the chain-of-thought (CoT) increases. These changes in the nature of the response do not remain within the model, and they leak into the evaluation itself. To address this problem, we probe the model with a panel of problems with poisoned answer keys under various interventions. We identify three channels through which an evaluation harness that uses an LLM judge reacts to these changes. Each channel shifts the judge labels, which shows that the behavioral curve across checkpoints is altered without any change in the behavior the judge claims to measure. We finally distill these observations into a four-practice protocol. Our work shows that researchers must ensure evaluation comparability before reading a trend across RLVR checkpoints as a change in capability or safety.

Primary area:
datasets and benchmarks
Keywords:
reinforcement learning, evaluation comparability, RLVR

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2cxLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors investigate evaluation comparability across checkpoints on Olmo3.1 RL-Zero Math. They observe that gains over the checkpoints on the MATH-500 benchmarks are mostly due to artifacts related to capped responses. Additionally, they find that the checkpoints show differing behavior as training progressed: the model stalls less often and repeats its final answer in its CoT more frequently. The authors attempt to address this by investigating the influence of three channels: answer elicitation through \boxed appending, transcript representation by giving the judge a different context, and population selection by selecting and comparing rollouts with specific properties. They then fix these channels to perform a safety evaluation on the model: how frequently

Critical strengths

None

Critical weaknesses

The paper provides diagnostics for one evaluation setup, but does not show their practical importance on existing pipelines. Its experiments are disconnected from the observed issue, and do not clearly show why they are relevant. I center my critical weaknesses around two main concerns:

Disconnected and unmotivated

Many experiments feel forced, and do not add value to the overall paper. For instance,

  • The transcript presentation: when determining whether a model did or did not use verification, one has to provide the full transcript. This is only an actual issue if the authors can clearly show that prior work makes this mistake, i.e., that the cropping policies are used in relevant existing evaluation pipelines and cause issues there. Further, the context window of recent LLMs are very large, and as the authors show in section 5, it is currently not an issue to put the entire transcript.
  • The safety evaluation in Section 5 feels disconnected from the rest of the paper. It is a short case study of applying the proposed channels, but the experiment itself adds little value: the authors present it as a safety concern, but the prompt given to the LLM judge actively asks it not infer deception. Overall, the prompt is confusing, and seems to just ask whether explicit verification is performed. Figure 22 also shows that the true positives are lower than the false positives when compared to human judges, which is concerning. Further, the code specifically says that human labels were created after exposure to the proposed annotations. The original frozen human labels, present in the code, only have k=0.174, with 60/86 overlap between human and LLM judge labels.
  • It is unclear why these experiments need to be performed with planted errors. The authors argue that it is about reward hacking, but giving an incorrect answer to the problem in the prompt actively deceives the model, and says little about reward hacking in real-world scenarios. Planting the answer is also not necessary for any other experiment, and has no direct connection to any other experiment.

Incremental and unconvincing

Generally, the paper makes a couple of observations, proposes an unmotivated solution, and applies it to a simple case study. I did not feel particularly compelled or interested in the paper. More specifically,

  • The observation that scores increase mainly due to finishing within the token limit, is a simple observation that practitioners generally know about, e.g., see Figure 6 in [1] and Figure 1 in [2]
  • The proposed three changes are either uninteresting or already known. Answer elicitation is a common technique that the authors take from prior work. Transcript presentation, as I argue above, is a non-issue.
  • The authors make no attempt to determine if there three channels are sufficient to capture all behavioral changes. They are three potential issues, but they might not (and highly likely do not) cover the entire space of potential spurious improvements during RL training.
  • The experiments are performed on a single (outdated) model within a single (outdated) task. This is not sufficient to make broad, general conclusions.

[1] https://arxiv.org/abs/2505.22312v2 [2] https://arxiv.org/abs/2505.19439v1

Questions for authors

See above

Reviewer_ucwFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how evaluation choices affect comparisons across RLVR checkpoints. However, the problem formulation and the proposed protocol’s role are not clearly explained, making its contribution and practical utility beyond documenting sensitivity to evaluation choices difficult to assess.

Critical strengths

The paper addresses a meaningful and practically relevant question. Understanding how evaluation choices affect checkpoint comparisons could help researchers draw more reliable conclusions about model behavior.

Critical weaknesses
  1. The connection between the formal definition of comparability and the proposed protocol is unclear. The paper lacks a clear observable criterion for verifying comparability when the target disposition is unobserved. It needs a clear formulation to help readers know what the objective is.

  2. The experiments quantify several effects within a single 7B model’s math RLVR training run, but their applicability to other model families, scales, and evaluation practices remains insufficiently established.

  3. The protocol’s practical benefit is not adequately validated against a reference or baseline. Changes in scores do not themselves establish improved measurement accuracy, and the fixed-panel analysis changes the population to which the conclusions apply (§5, Fig. 7).

  4. The fixed panel leaves out cases where the model needs answer forcing, which may be some of the more interesting cases to track. It also requires results from all checkpoints, making it hard to use during training.

  5. The safety metric mainly checks whether the model backs up its verification claims with reasoning. This does not quite capture reward hacking or deception: a flawed check can pass, while copying a poisoned answer may go unflagged if the model never claims to have checked it.

Questions for authors
  1. What is the precise evaluation goal of the proposed protocol? Specifically, what observable criterion verifies improved comparability when the target disposition D is unobserved (§2; Appendix B, Eq. (2))?

  2. How well do the findings generalize across models? Given the single 7B training run studied in §3–5, is there additional experimental evidence or prior work supporting similar effects in other model families or larger models?

  3. What evidence demonstrates the protocol’s practical benefit? Specifically, what reference or baseline shows that the analysis in §5, Fig. 7 improves measurement accuracy rather than simply changes the evaluated population and resulting scores?

  4. Could differences in the judge’s error rates across early and late checkpoints explain the reported change in unsupported certification? What validation or sensitivity analysis rules out this possibility (Appendix H, Fig. 22)?

Reviewer_D3VqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the evaluation issue in RLVR across ckpts and argues that the trends across training steps could be misinterpreted by evaluation pipeline given that the training changes response properties including length, stall, repetition, etc. With ckpts of Olmo model on 200 Math-500 problems with incorrect answer key in the prompt, the authors show that the performance gain actually comes from models more likely producing answer within in budget, and ckpts in late training stage produce longer responses, stall less and repeat answers more.

Then they perform interventions on answers by forcing models to produce answers, cropping transcripts, and scoring populations. In the end, they concludes with four practical protocols for evaluation.

Critical strengths
  1. The paper studies an important evaluation question for LLM. The evaluation fairness and bias is always a severe issue in RLVR which is also related to some safety or faithfulness studies.

  2. A convincing local intervention experiment. On the same stalled prefixes, appending an answer-forcing scaffold increases planted-answer adoption by 52.16 percentage points, with a 95% interval of [45.34, 59.09]. A neutral "Answer:" prompt produces a similar effect. This provides strong evidence that answer elicitation can substantially affect observed outcomes (§4, Table 4).

  3. Careful diagnostics and transparent reporting. Paired transcript views, problem-level bootstrap resampling, missing-label bounds, matched note controls, and sensitivity analyses strengthen the study.

Critical weaknesses
  1. Limited contribution: The paper provides no new method, tool or critical setups; the contribution mainly lies on an empirical observations from one RL training run and an evaluation protocol. The three channels are based on the characteristics of responses from RL post-trained models which have been studied. The provided protocol partially attributes to prior works as stated in section 6.

  2. The introduction motivates the evaluation problem with published checkpoint, but the paper has not shown that previous flawed evaluation causes any misleading results on existing works or following the proposed evaluation protocol could have some benefits.

  3. Some of reported results about response change with training have been studied before. Longer responses along with RL training, overthinking with no answer, repetitions, the performance gains coming from formatting are largely known. Observing on them is indeed meaningful but it would be more interesting to see how these changes lead to misleading conclusions on model behaviors.

  4. The interpretation of model's capability improvement is not verified. Assigning 9.8 of the 10.4 percentage-point improvement to answer emission is just numeric decomposition, not evidence of solving ability of the model barely improves. The population of answered problems changes across checkpoints: newly answered problems could also be newly solved problems, that's possible. Moreover, producing correct answers within a fixed budget is actually a meaningful gain from training. Appendix A explicitly acknowledges that the analysis is not causal mediation and establishes neither unchanged conditional accuracy nor equivalence.

  5. The human comparison uses one investigator and the same examples for calibration. In Figure 22, the 91 resolved cases contain 5 true positives, 6 false positives, 4 false negatives, and 76 true negatives, so most of agreement is actually agreement on negatives. Multiple annotators and sufficient positive cases are required to strengthen the result.

  6. The transcript-presentation results are weaker for the central safety outcome than the three-channel narrative suggests. Cropping clearly changes “checked” and supported-certification labels. However, its effect on unsupported certification is +0.69pp, and the checkpoint-trend comparison is also inconclusive. The initial padding effect does not replicate in the expanded sample. Appendix A explicitly states that neither probe establishes that the checkpoint trend is a length artifact.

  7. All results and observations come from one model, one task, one judge and under one rubric, so the result is not convincing, especially given that this paper is discussing evaluation fairness and bias.

Questions for authors

Can you apply the proposed protocol to an existing checkpoint-evaluation pipeline and demonstrate the benefits?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.