- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper evaluates OLMo-3-7B-Think checkpoints on InstrumentalEval, scored by GPT-4o as judge. It reports a non-monotonic trajectory: the instrumental rate rises from 25% (SFT) to 46% (step 200), then falls to 11.8% (final). Framing controls do not explain the trajectory. A 32B endpoint check shows the same reduction.
Critical strengths
Evaluating released intermediate checkpoints instead of only before/after endpoints is a good idea, and the observed non-monotonicity is a potentially useful caution for safety evaluations of post-training.
Critical weaknesses
- The measurement is not reliable, and the reported statistics contain errors that undermine the conclusions. -Only 76 items, one greedy sample per item, and one LLM judge with no human validation. Second-judge agreement is close to what one would get from a judge that labels almost everything non-convergent. In fact, IAR ≈ 100 − IR at every checkpoint, and the “Mean |Δscore|” column in Table 12 is nearly identical to the GPT-4o mean score in Table 11. This strongly suggests Gemini assigns ≈0 to nearly all responses, so the “higher agreement at the final checkpoint” is an artifact rather than evidence of robustness. -Numbers are inconsistent across the paper: the abstract reports p = 0.015 / 0.03 while the text reports 0.009 / 0.02; β̂₁,SFT is 0.0008 in Table 3 vs. 0.0088 in Table 10; Table 3 lists an “RLVR-0” checkpoint that does not exist; and the SFT→DPO and SFT→step-50 flip tables are identical. -The INLP projection leaves every checkpoint mean exactly unchanged, which suggests the analysis cannot have any effect by construction or contains a bug. -Model (2) includes item fixed effects α_i together with item-constant indicators H_i, S_i, O_i, which is not identified. -Because framing is fixed per item and every checkpoint is evaluated on the same items, framing cannot confound the paired checkpoint comparison in the first place, so the main “control” addresses a non-issue. -The breakpoint at step 200 is chosen at the observed peak, which makes the significance of β₂ circular.
- A major alternative explanation is not ruled out, and the claims are overstated.
- RLVR is known to lengthen reasoning traces, but generation is capped at 1024 tokens for a “Think” model. If later checkpoints spend more of the budget thinking, truncated or absent final answers would mechanically lower IC scores.
- Response lengths, truncation rates, refusal rates, and what text the judge actually sees are not reported.
- The interpretation assumes a purely verifier-based objective, but the OLMo-3 Think RL mixture reportedly also includes non-verifiable chat/IF rewards; please verify and discuss.
- The conclusion that RLVR has an “eventually positive effect on instrumental convergence,” and the title “Questioning Instrumental Convergence,” far exceed what a 76-item text benchmark on a 7B model can support.
Questions for authors
- What fraction of responses hit the 1024-token limit at each checkpoint, and does the trend survive when restricted to complete responses or with a larger budget?
- Please report the Gemini score distribution per checkpoint. Is the second judge nearly always outputting 0?
- Which p-values and coefficients are correct? Please reconcile the abstract, Section 4, and Tables 3 and 10.
- How was INLP applied so that all 16 means are exactly unchanged?
- Does the trend hold with multiple samples per item (non-greedy) and with human labels on a stratified subset?