acceptodds
Under review as a conference paper at ICLR 2027

Questioning Instrumental Convergence in RLVR: Checkpoint Evidence from OLMo-3

Abstract

Instrumental convergence is often invoked as a general reason to expect capable agents to acquire or express power-seeking subgoals: self-preservation, resource acquisition, deception, or resistance to oversight which poses an important threat to the society. For LLMs, reinforcement learning is the first stage of post-training that directly optimizes behavior against an objective; therefore, it is the stage where convergent instrumental behavior, on IC grounds, likely to appear. Does reinforcement learning with verifiable rewards (RLVR) make a model more likely to express instrumental-convergence behavior, or does this depend on the training regime, evaluation frame, and stage of post-training? We study this question as a checkpoint-conditioned behavioral trajectory rather than as a single before/after safety score. Using InstrumentalEval, we evaluate OLMo-3-7B-Think across SFT, DPO, 13 intermediate RLVR checkpoints, and the final RLVR model. The trajectory is nonmonotonic: instrumental rate rises from 25.0% at SFT to 46.1% near RLVR step 200, then falls to 11.8% at the final checkpoint. Paired item-level tests and step-wise regression support the final SFT-to-RLVR reduction ( by paired -test; by Wilcoxon signed-rank test; by step-wise regression). Prompt framing does not explain this pattern: a harmful-prompt indicator combined with other framing effects explains only 15% of within-checkpoint score variance on average, and Iterative Null-space Projection (INLP) removes a detectable framing direction without changing the SFT-to-final-RLVR gap. These results show that RLVR has significantly important and eventually positive effect on instrumental convergence under OLMo-7B-Think model structure and InstrumentalEval benchmark.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
RLVR, Instrumental Convergence, AI safety, LLM alignment, behavioral evaluation, OLMo-3

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CZFKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper evaluates OLMo-3-7B-Think checkpoints on InstrumentalEval, scored by GPT-4o as judge. It reports a non-monotonic trajectory: the instrumental rate rises from 25% (SFT) to 46% (step 200), then falls to 11.8% (final). Framing controls do not explain the trajectory. A 32B endpoint check shows the same reduction.

Critical strengths

Evaluating released intermediate checkpoints instead of only before/after endpoints is a good idea, and the observed non-monotonicity is a potentially useful caution for safety evaluations of post-training.

Critical weaknesses
  1. The measurement is not reliable, and the reported statistics contain errors that undermine the conclusions. -Only 76 items, one greedy sample per item, and one LLM judge with no human validation. Second-judge agreement is close to what one would get from a judge that labels almost everything non-convergent. In fact, IAR ≈ 100 − IR at every checkpoint, and the “Mean |Δscore|” column in Table 12 is nearly identical to the GPT-4o mean score in Table 11. This strongly suggests Gemini assigns ≈0 to nearly all responses, so the “higher agreement at the final checkpoint” is an artifact rather than evidence of robustness. -Numbers are inconsistent across the paper: the abstract reports p = 0.015 / 0.03 while the text reports 0.009 / 0.02; β̂₁,SFT is 0.0008 in Table 3 vs. 0.0088 in Table 10; Table 3 lists an “RLVR-0” checkpoint that does not exist; and the SFT→DPO and SFT→step-50 flip tables are identical. -The INLP projection leaves every checkpoint mean exactly unchanged, which suggests the analysis cannot have any effect by construction or contains a bug. -Model (2) includes item fixed effects α_i together with item-constant indicators H_i, S_i, O_i, which is not identified. -Because framing is fixed per item and every checkpoint is evaluated on the same items, framing cannot confound the paired checkpoint comparison in the first place, so the main “control” addresses a non-issue. -The breakpoint at step 200 is chosen at the observed peak, which makes the significance of β₂ circular.
  2. A major alternative explanation is not ruled out, and the claims are overstated.
  • RLVR is known to lengthen reasoning traces, but generation is capped at 1024 tokens for a “Think” model. If later checkpoints spend more of the budget thinking, truncated or absent final answers would mechanically lower IC scores.
  • Response lengths, truncation rates, refusal rates, and what text the judge actually sees are not reported.
  • The interpretation assumes a purely verifier-based objective, but the OLMo-3 Think RL mixture reportedly also includes non-verifiable chat/IF rewards; please verify and discuss.
  • The conclusion that RLVR has an “eventually positive effect on instrumental convergence,” and the title “Questioning Instrumental Convergence,” far exceed what a 76-item text benchmark on a 7B model can support.
Questions for authors
  1. What fraction of responses hit the 1024-token limit at each checkpoint, and does the trend survive when restricted to complete responses or with a larger budget?
  2. Please report the Gemini score distribution per checkpoint. Is the second judge nearly always outputting 0?
  3. Which p-values and coefficients are correct? Please reconcile the abstract, Section 4, and Tables 3 and 10.
  4. How was INLP applied so that all 16 means are exactly unchanged?
  5. Does the trend hold with multiple samples per item (non-greedy) and with human labels on a stratified subset?
Reviewer_eZinOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper is an empirical piece on the problem of instrumental convergence. Instrumental convergence is a well-known AI alignment concept whereby in the pursuit of any array of actual, terminal goals, agents learn a set of common “instrumental” goals such as power-seeking, resource acquisition, persistence, subterfuge etc because these are commonly required traits for the successful pursuit of other goals. There is much evidence that RL(VR) in particular may lead to the emergence of these characteristics (cf literature on reward hacking, reward seeking, science of scheming etc). The authors here, therefore, attempt to evaluate whether instrumental convergence arises during the RLVR training of a small open-weights model. To do this they apply an existing benchmark, InstrumentalEval, on a sequence of publicly available SFT, DPO and RL checkpoints of the small OLMo-3-7B-Think model. They find that instances of the behaviours in the benchmark (as judged by a stronger language agent) increase during the early post-training phases and then decrease in the second part of the RLVR.

Critical strengths
  • This is an extremely strong experimental problem; I am strongly of the belief that research in this particular area and on this particular problem, i.e. the arising of reward seeking and other instrumentally convergent behaviors in modern agent RL, is very high-impact and there needs to be much more work here (despite it being a recent field of growing interest).
  • The authors have a very clean approach to the problem: evaluate for instrumental goals throughout the various publicly-available RL checkpoints of an open-source model (OLMo).
  • There is some good attempt at discussion of the results.
  • The results are interesting, if unexplained and potentially noisy. I wouldn’t have quite expected this, so although I am not sure of the significance of the findings, I am glad to have learned of them.
Critical weaknesses
  • However, unfortunately I think the experimental setup was quite low-effort and not quite on point for this actual problem. I think using the OLMo checkpoints is a great idea but unfortunately the model is probably too small and the RLVR too basic to expect any real instrumental reasoning of the kinds evaluated in this benchmark. I am basically convinced that the true mechanisms by which instrumental convergence occurs will only happen at a much higher capability level and through much more extended optimisation pressure (and there is various evidence to support this). So I expect any results at this scale to be an artifact of the experimental setup rather than meaningful.
  • Indeed, with the above in mind, I am not surprised that the authors did not discover the emergence of instrumental behaviors during RLVR here. I am slightly surprised at the pattern they did find though, and do not fully understand it. I expect, as the authors allude to in the discussion, it is due to simple side-effects of the simple RL here.
  • I think the above exposes a flaw in the experimental methodology: the authors should also evaluate for a bunch of unrelated behaviors that are not predicted by instrumental convergence, and aim to study the gap between the prevalence of those behaviors and the instrumental ones (or the change in this gap, at least). This would be the experimental tool that would allow some stronger understanding of what happened here. That being said, I do not ultimately think there is a whole lot of value in running InstrumentalEval on OLMo models of this size.
  • Related to the above, I do not feel there is an adequate explanation of this work's place in the context of recent literature
  • Some parts of the paper are very weirdly written / poorly presented, e.g. the introduction and discussion being big blocks of text with no paragraphs or subsections.
Questions for authors
  1. Have you done any control experiments as described above, where you evaluate for unrelated behaviors?
  2. Are there any other follow-ups you propose that would uncover a better explanation of the pattern you saw?
  3. In the absence of being able to train your own model, which I understand and commend the adaptation to, have you considered using pre/post-RL checkpoints from some larger open-weights models and/or adapting the benchmark to test instrumental convergence in more controlled settings related to the training the model actually went through?
Reviewer_RnobOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how reinforcement learning with verifiable rewards (RLVR) affects instrumental convergence (IC)-related behaviors in large language models. The authors evaluate OLMo-3-7B-Think across 16 post-training checkpoints, including SFT, DPO, intermediate RLVR stages, and the final RLVR model, using 76 tasks from InstrumentalEval. Model responses are assessed by an LLM judge for behaviors such as shutdown avoidance, deception, and strategic alignment. The main finding is a non-monotonic trajectory: the instrumental rate increases from 25.0% at SFT to 46.1% around RLVR step 200, before declining to 11.8% at the final checkpoint. The authors use paired statistical tests to assess the endpoint reduction, examine potential prompt-framing effects through regression analysis and iterative null-space projection (INLP), and evaluate inter-judge agreement. They further report a similar endpoint reduction for OLMo-3-32B-Think, from 35.5% to 22.4%. Overall, the paper provides empirical evidence that IC-related behavioral expressions do not necessarily increase monotonically during RLVR training, highlighting the importance of examining intermediate training checkpoints rather than relying solely on endpoint comparisons.

Critical strengths
  1. Checkpoint-level analysis reveals non-monotonic behavioral changes during RLVR. Rather than relying on a conventional before-and-after comparison, the paper evaluates 16 checkpoints along the post-training trajectory while keeping the evaluation tasks and protocol fixed. The observed increase in IC-related behaviors during early RLVR, followed by a substantial decline at later stages, is an interesting empirical finding that would be missed by endpoint-only evaluations. This provides a useful perspective on how safety-related behaviors can evolve throughout RL training.

  2. The endpoint reduction is supported by additional comparisons. The authors complement the aggregate results with item-level paired comparisons, showing that more tasks switch from exhibiting IC-related behavior to not exhibiting it than in the opposite direction. A similar SFT-to-RLVR reduction is also observed in the larger OLMo-3-32B model. Although these results remain limited to one model family, they provide additional evidence that the endpoint reduction is not specific to the 7B model or merely an artifact of comparing aggregate scores.

Critical weaknesses
  1. Limited evidence connecting the observed behaviors to instrumental convergence in actual decision-making. The evaluation relies on model responses to hypothetical scenarios rather than actions taken by goal-directed agents in an interactive environment. While this is a reasonable behavioral proxy, it remains unclear whether the observed reduction reflects a change in the model's tendency to pursue instrumental subgoals or simply a change in its response distribution. Moreover, the paper does not directly investigate which aspects of RLVR training or reward optimization lead to the observed trajectory. This limits the extent to which the findings advance our understanding of instrumental convergence from an RL perspective.

  2. Insufficient robustness and clarity in the statistical validation. The main evaluation uses only 76 tasks, and the agreement between the two LLM judges is moderate (68.75% on average), dropping to 55.26% at step 200, where the highest instrumental rate is reported. It is therefore unclear how robust the non-monotonic trajectory is to the choice of judge. In addition, Equation (2) raises an identifiability concern by including item fixed effects alongside time-invariant framing indicators, while the reported coefficients in Equation (4) are difficult to reconcile with the stated response variable and checkpoint statistics. These issues weaken confidence in the statistical analyses and the effectiveness of the proposed framing controls.

Questions for authors
  1. Framing analysis (Equation (2) and INLP): Could the authors clarify the regression specification used in Equation (2)? As written, the inclusion of item fixed effects alongside time-invariant item-level framing indicators raises an identifiability issue, particularly given the description of fitting separate regressions for each checkpoint. It would also be helpful to clarify how INLP was applied to the score matrix and why the checkpoint means remain unchanged after projection in Table 11. These details are important for assessing whether the framing controls support the authors' conclusions.

  2. Statistical analysis of the training trajectory (Equation (4)): Could the authors clarify the response variable and scaling used in the piecewise regression? The reported coefficients are difficult to reconcile with the stated formulation and checkpoint statistics. Was step 200 selected as the breakpoint based on the observed peak, or specified independently? Providing the exact regression specification and corresponding statistical results would help establish the strength of the evidence for the reported trajectory.

  3. Robustness to the choice of LLM judge: The inter-judge agreement is relatively low around the reported peak at step 200 (55.26%). Do the authors observe a similar non-monotonic trajectory when using Gemini 2.0 Flash as the primary judge? Reporting the checkpoint-level instrumental rates under the second judge would help assess whether the main finding is robust to the evaluation procedure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.