- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors investigate evaluation comparability across checkpoints on Olmo3.1 RL-Zero Math. They observe that gains over the checkpoints on the MATH-500 benchmarks are mostly due to artifacts related to capped responses. Additionally, they find that the checkpoints show differing behavior as training progressed: the model stalls less often and repeats its final answer in its CoT more frequently. The authors attempt to address this by investigating the influence of three channels: answer elicitation through \boxed appending, transcript representation by giving the judge a different context, and population selection by selecting and comparing rollouts with specific properties. They then fix these channels to perform a safety evaluation on the model: how frequently
Critical strengths
None
Critical weaknesses
The paper provides diagnostics for one evaluation setup, but does not show their practical importance on existing pipelines. Its experiments are disconnected from the observed issue, and do not clearly show why they are relevant. I center my critical weaknesses around two main concerns:
Disconnected and unmotivated
Many experiments feel forced, and do not add value to the overall paper. For instance,
- The transcript presentation: when determining whether a model did or did not use verification, one has to provide the full transcript. This is only an actual issue if the authors can clearly show that prior work makes this mistake, i.e., that the cropping policies are used in relevant existing evaluation pipelines and cause issues there. Further, the context window of recent LLMs are very large, and as the authors show in section 5, it is currently not an issue to put the entire transcript.
- The safety evaluation in Section 5 feels disconnected from the rest of the paper. It is a short case study of applying the proposed channels, but the experiment itself adds little value: the authors present it as a safety concern, but the prompt given to the LLM judge actively asks it not infer deception. Overall, the prompt is confusing, and seems to just ask whether explicit verification is performed. Figure 22 also shows that the true positives are lower than the false positives when compared to human judges, which is concerning. Further, the code specifically says that human labels were created after exposure to the proposed annotations. The original frozen human labels, present in the code, only have k=0.174, with 60/86 overlap between human and LLM judge labels.
- It is unclear why these experiments need to be performed with planted errors. The authors argue that it is about reward hacking, but giving an incorrect answer to the problem in the prompt actively deceives the model, and says little about reward hacking in real-world scenarios. Planting the answer is also not necessary for any other experiment, and has no direct connection to any other experiment.
Incremental and unconvincing
Generally, the paper makes a couple of observations, proposes an unmotivated solution, and applies it to a simple case study. I did not feel particularly compelled or interested in the paper. More specifically,
- The observation that scores increase mainly due to finishing within the token limit, is a simple observation that practitioners generally know about, e.g., see Figure 6 in [1] and Figure 1 in [2]
- The proposed three changes are either uninteresting or already known. Answer elicitation is a common technique that the authors take from prior work. Transcript presentation, as I argue above, is a non-issue.
- The authors make no attempt to determine if there three channels are sufficient to capture all behavioral changes. They are three potential issues, but they might not (and highly likely do not) cover the entire space of potential spurious improvements during RL training.
- The experiments are performed on a single (outdated) model within a single (outdated) task. This is not sufficient to make broad, general conclusions.
[1] https://arxiv.org/abs/2505.22312v2 [2] https://arxiv.org/abs/2505.19439v1
Questions for authors
See above