acceptodds
Under review as a conference paper at ICLR 2027

Learning to Steer, Steering to See: Unveiling the Geometry of RLVR in Large Language Models via Trainable Vectors

Abstract

Reinforcement learning (RL) has emerged as a key paradigm for enhancing the reasoning capabilities of large language models. However, the high dimensionality of parameter updates makes RL training dynamics difficult to analyze, obscuring the mechanisms underlying these gains. In this work, we study reinforcement learning with verifiable rewards (RLVR) and use vector steering as an analytical tool to identify a low-dimensional effective manifold in activation space that is closely associated with RL-induced performance gains. We further uncover two key geometric properties of this manifold. (1) Effective Manifold Capacity: The capacity required to reproduce RL-induced gains can be very small, but it is not infinitely compressible. When the capacity is compressed to an extremely low level, the intervention dimensionality and the ability to express input-dependent corrections become key constraints, and this requirement further varies with injection depth. (2) Control Manifold Separation: Effective control directions lie predominantly in the low-variance complement of the activation principal subspace. Within the same task and base model, the learned geometry remains largely consistent across training configurations; across tasks, geometric alignment correlates with capability transfer. We conduct experiments on 5 LLMs and 6 tasks with verifiable rewards, and the results support the above properties. Based on these findings, we propose Alpha-Stabler, a plug-and-play training framework comprising two modules: a Predictor that monitors principal-subspace intrusion to provide early warnings of training collapse, and a Controller that removes the principal-subspace component of activation gradients during backpropagation while preserving their components in the orthogonal complement of the principal subspace. Experiments show that Alpha-Stabler can stabilize training for 2,000 steps and consistently enhance the gains brought by reinforcement learning. This work advances the understanding of RL from an activation-manifold perspective and offers practical insights for more robust post-training. Our code is available at: https://anonymous.4open.science/r/alpha-stabler.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Vector Analysis, Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tt3sOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper shifts the analytical perspective from parameter space to activation space to investigate the geometry of RL-induced behavioral changes and its relationship to capability transfer and training stability. The authors use trainable, layer-wise additive steering vectors as analytical probes and report two geometric properties. First, Effective Manifold Capacity: a small, input-invariant fixed-vector intervention can recover much of the gain induced by RL, but is limited by the teacher–student distribution gap, injection depth, and the need for input-dependent corrections. Second, Control Manifold Separation: sequentially learned orthogonal vectors can independently support task behavior, and effective directions predominantly occupy the low-variance complement of the leading activation principal-component subspace. The geometry is broadly consistent across configurations for the same task, and cross-topic directional alignment correlates with transfer gains.

Motivated by these findings, the paper introduces Alpha-Stabler, a plug-and-play RL stabilization framework. Its Predictor monitors principal-subspace intrusion (PSI) to flag potential training collapse, while its Controller removes the leading-principal-subspace component of activation gradients during backpropagation and preserves the orthogonal complement. Experiments cover DeepSeek-R1-Distill-Qwen-1.5B/7B (thinking mode) and Qwen3-4B/8B/14B (non-thinking mode), with mathematical reasoning, scientific reasoning, code generation, and instruction-following tasks, comparisons between on- and off-policy distillation and update parameterizations, and stabilization ablations. The theoretical appendix offers conditional local analysis rather than a general proof of practical RL dynamics.

Critical strengths
  1. A compact but informative mechanistic probe. A shared additive vector per controlled layer often recovers over 85% of RL-induced gains (Section 2.2, Figure 2). Comparisons across teacher–student gaps and token-dependent gating help separate expressivity limitations from parameter count (Section 2.3, Figures 3–4).

  2. A concrete link between geometric diagnosis and stabilization. Learned directions have only about 1–2% energy in the top-10% activation-PC subspace across four scientific topics (Section 3.2, Figure 7(a)). PSI increases before visible score deterioration in the illustrated collapse-prone trajectories (Figure 7(c)). Alpha-Stabler turns these observations into a Predictor and gradient-projection Controller (Section 4.1, Equations (6)–(8), Algorithm 1), and Figure 8 compares learning rates, activation times, and projected subspaces.

Critical weaknesses
  1. Limited independence and sample size of transfer evidence (Section 3.2, Figure 6(c)). The reported alignment–transfer fit reaches R² = 0.91, but is based on only six ordered source–target pairs drawn from four scientific topics. These related transfers do not constitute a large independent sample, and the figure alone does not establish cross-task predictive generality. Better to report per-pair uncertainty or evaluate additional topics and held-out transfers.
  2. The Predictor is not evaluated as a calibrated early-warning system (Section 3.2, Figure 7(c) and Section 4.2, Figure 8(a,b)). The trajectories show PSI rising before visible score loss in examples, but do not report a systematic false-alarm rate, missed-collapse rate, warning lead-time distribution, or detection accuracy over independent seeds. This is material because actionable prediction is a central stated contribution, not just post-hoc correlation.
  3. Stabilization statistics across independent runs are insufficiently characterized (Section 4.2, Figure 8(a–e)). The plotted scores and peak-PSI comparisons support the intervention in the evaluated settings, but the main figures do not consistently provide seed-aggregated collapse frequencies, confidence intervals, or dispersion of final scores. A few favorable curves cannot establish how reliably collapse is prevented across runs. You may report seed-level outcomes for every comparator.
Questions for authors
  1. Across independent seeds, tasks, and learning rates, what are the Predictor’s false-alarm and missed-collapse rates and warning lead-time distributions? How sensitive are these to Algorithm 1’s warm-up length, interval, EMA, and threshold multipliers?
  2. Please report seed-level/seed-aggregated final scores, collapse frequencies, and uncertainty for Alpha-Stabler, no control, gradient clipping, and alternative PC projections in Figure 8.
  3. Do off-principal energy concentration and PSI predictive utility hold when PCA bases are estimated on an independent fixed corpus, rather than warm-up actor rollouts, and when the retained-PC fraction differs from 10%?
  4. Can the authors quantify end-to-end GPU-hours, peak GPU memory, calibration/monitoring time and inference cost? Does Figure 12 show pre-collapse wall-clock trajectories only?
  5. Which additional empirical tests, if any, connect Appendix D’s surrogate-energy recovery ρ to the task-score recovery plotted in Figure 2?
Reviewer_6NHAOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks what reinforcement learning with verifiable rewards (RLVR) changes inside a language model. It freezes a base model and adds one trained vector to each of layers 5 to 20 to imitate its RL-tuned version. Analyses cover five models from 1.5B to 14B parameters on math, science, code and instruction following. Property 1 claims that these vectors recover most of the RL gain, within limits set by the teacher–student gap and depth. Property 2 claims that the learned directions avoid the principal subspace, the base model's top 10% of activation directions by variance, and that their alignment across topics tracks transfer. The proposed method, Alpha-Stabler, monitors how far RL moves activations into this subspace and removes that component from the activation gradients to prevent collapse. Appendix D adds a least-squares theory.

Critical strengths
  1. Asking how RL updates work in activation space is a good way to study RLVR.
  2. The activation-space monitor and correction appear to be new.
Critical weaknesses

W1. Most of the analysis is already published, and the theory adds almost nothing to it.

  • Earlier work reports Property 1 and its explanation. Sinii et al. (EMNLP 2025, arXiv:2505.18706) show that one RL-trained steering vector per layer on a frozen base model matches full RL fine-tuning. They also trace the offset's limit to its independence from the token and show that LoRA closes the gap, the argument of Sec. 2.3. Their follow-up (arXiv:2509.06608) sweeps single layers, finds that late layers fail, and trains orthogonal vectors as in Fig. 5. The paper cites neither and never says what distilling from an RL teacher adds over training the vectors with RL.

  • Property 2 is the activation-space version of a published result. That result is Zhu et al. (2025), "The Path Not Taken: RLVR Provably Learns Off the Principals", which the paper cites only for "update geometry" without discussing the link.

  • Appendix D explains neither property. Theorem D.1 is the standard split of a least-squares fit into a mean and a fluctuation. Theorem D.2 assumes the pattern of Property 2, with constants the paper never measures. As the appendix concedes, it "does not derive that pattern from the RLVR objective".

W2. The proposed stabilizer is not shown to help in practice, and the experiments lack proper comparisons.

  • The paper tests Alpha-Stabler only on a collapse it induces. All Alpha-Stabler runs use learning rates from 1e-5 to 1e-4 (Fig. 8c; Fig. 12), while the paper's own RL runs use 1e-6 (Table 2). So the claim that it can "consistently enhance the gains brought by reinforcement learning" is untested at standardized settings.
  • Alpha-Stabler is compared with only one outside method. Global gradient clipping scores 0.79, against 0.82 for Alpha-Stabler (Fig. 8e). The paper does not compare with a lower learning rate, a KL penalty, or Clip-Cov and KL-Cov from Cui et al. (2025b), which it cites.
  • The optimizer is part of the mechanism, and the paper neither studies nor names it. Lemma D.7 covers only a plain SGD step, and the appendix concedes that a concrete optimizer "requires its actual update decomposition" (l.2264). No result shows that the projection survives the per-coordinate step sizes of AdamW or the orthogonalized updates of Muon.

W3. Wrong template with the wrong page margins.

Questions for authors
  1. What does distilling from an RL teacher show that training the same vectors with RL (Sinii et al., 2025) does not?

  2. What share of their energy do vectors distilled from the non-RL teachers in Fig. 2(c) place in the principal subspace? Fig. 7(b) restricts steering to the top 30% of principal components, so please also restrict it to the top 10%.

  3. In Fig. 6(c), which direction of each topic pair enters the fit, and does it hold over all twelve ordered pairs of the four topics? The text reports six "ordered" pairs while the legend lists unordered ones, which is inconsistent, and six points make the R-squared of 0.91 weak evidence.

  4. At the learning rate of Table 2, does Alpha-Stabler change final held-out accuracy? A gain at standardized settings would help resolve most of this concern.

  5. Which optimizer trains the policy, and do the gradient projection and Properties 1 and 2 hold under AdamW and Muon? Muon flattens the singular values of each update, so a low-rank picture of RL updates need not carry over.

Reviewer_Q3VPOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper uses activation steering to study how much of an RLVR-trained model’s improvement can be recovered without updating the backbone.

Critical strengths

n/a

Critical weaknesses

The paper has a formatting discrepancy, and it seems to shrink the page margin.

Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.