acceptodds
Under review as a conference paper at ICLR 2027

Analyzing WEight-Space Orientation for Measuring Entropy and Accuracy in RLVR

10 views

Abstract

Reinforcement learning with verifiable rewards (RLVR) can improve accuracy while reducing predictive entropy.We show these two effects are independent; their concomitance depends on a particular direction of the RLVR update and the entropy decrease involves token specific changes. Using two model lineages OLMo-3 and Tulu-3/Llama and their checkpoints, we perturb their weights and output scores. We find the components of RLVR update define a low dimensional subspace. We compare the RLVR weight update with randomly rotated versions in that subspace with the same singular values, and update size. The learned direction sharply lowers entropy, while the random directions have little effect. This difference persists across the tested truncation ranks and on model-generated text. We then use RLVR to select 30 strongly changed weight matrices, but search for alternative changes to those matrices around the supervised fine-tuned (SFT) model. We show that a similar measured accuracy can occur without entropy collapse and that entropy loss occurs with accuracy decreases in other points of the subspace.

Primary area:
interpretability and explainable AI
Keywords:
Reinforcement Learning with Verifiable Rewards, LLM Post-Training, Entropy Collapse, Weight-Space Geometry, Model Weight Interventions

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nnWWOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why reinforcement learning with verifiable rewards lowers a model's entropy, by looking at the weight change it makes. The authors keep the size and structure of the RLVR update fixed but randomly change its direction. The learned direction lowers entropy, while random directions barely do, so direction is what matters. They then show that a random search around the SFT model can reach accuracy close to RLVR's without lowering entropy, suggesting that the accuracy gain does not require the entropy drop.

Critical strengths

The main control rotates the learned update while keeping its size, structure, and singular values fixed, so any difference in entropy can only come from direction. The result holds across two model families, several settings, and both fixed and model-generated text. This matters because it makes the central comparison easy to interpret and gives a reusable tool for studying what weight updates do.

Critical weaknesses

None

Questions for authors

Does the analysis also generalize to other RL algorithm like RLHF or DPO?

3 comments
AuthorsOpenReview ↗

Author Response

Thank you for your review, and to answer your question we used the Olmo-3-7B-DPO checkpoint to test for these effects. DPO shows a much weaker entropy drop, albeit hinting at a similar connection with orientation. We have also substantially rewritten and updated the paper to address the points raised in the reviews.

InterventionΔ from SFT (DPO)Δ from SFT (RLVR)
Full Checkpoint−0.0382-0.143
SPECTRUM−0.0002+0.002
WITHIN+0.0001−0.002
ORTHOGONAL−0.00020.000

Thank you for the detail rebuttal. I would keep my positive score.

AuthorsOpenReview ↗

reply to Reviewer nnWW's comment

Dear Reviewer, Many thanks for your support. We are glad that our rebuttal was appreciated!

Reviewer_A8h5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper compares the learned RLVR update with random rotations in its low-dimensional singular support that preserve its singular values and norm. The authors find that the learned orientation sharply lowers entropy, whereas the random orientations have little effect. A logit-level analysis shows that a fitted affine rescaling of SFT logits increases entropy, and that token-specific residual changes among high-probability tokens recover most of the entropy decrease.

Critical strengths
  1. The paper correctly recognizes that comparing SFT and RLVR checkpoints cannot by itself identify the source of entropy reduction.
  2. The distinction among TRUNCATED, WITHIN, ORTHOGONAL, and SPECTRUM controls is sensible.
Critical weaknesses
  1. The paper is hard to follow. The presentation makes it harder than necessary to identify what is being held fixed, what is being randomized, and which comparison is the decisive one. The terminology also shifts.
  2. The paper does not establish that entropy is a stable predictor of accuracy. I think the paper may propose a new method or insight to improve performance, at least the paper should clarify the practical implications of the analysis.
Questions for authors

Please see above.

1 comment
AuthorsOpenReview ↗

Author Response

Thank you for your review. Please see our response to Question 1 of Reviewer uUcg. Our main point is that entropy is not a predictor of accuracy. We show that configurations adjacent to the RLVR update can achieve the same accuracy without the entropy drop. This motivates future work on improving existing RLVR algorithms to find such configurations. We have also substantially rewritten and updated the paper to address the points raised in the reviews.

Reviewer_uUcgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates whether the entropy reduction associated with reinforcement learning with verifiable rewards depends on the direction of the learned weight update, and whether accuracy improvements necessarily reproduce that reduction. The authors analyze the weight differences between supervised fine-tuned (SFT) and RLVR checkpoints across two model lineages, OLMo-3 and Tulu-3/Llama. Their strongest control preserves the retained singular subspaces, singular values, and per-matrix F-norm while changing orientation. Learned updates reduce next-token entropy substantially more than these controls. This separation appears on shared teacher-forced text and short model generated trajectories. An alignment sweep further shows that entropy decreases as interventions approach the learned direction, whereas accuracy trends depend on the task. A separate Gaussian search around SFT perturbs 30 RLVR-selected matrices. The selected candidate achieves reported held-out accuracy of 76%, compared with 61% for SFT and 75% for RLVR, while retaining SFT-like entropy. Finally, a per-position affine decomposition of logits finds that the fitted vocabulary-wide rescaling increases entropy; token-specific residual changes among approximately the ten highest-probability SFT tokens recover most of the observed reduction. The central empirical contribution is evidence that update magnitude and spectrum alone do not explain RLVR’s entropy effect. The stronger mechanistic interpretation remains less established.

Critical strengths
  1. Interesting and relevant research question. The paper investigates an important aspect of RLVR by connecting weight-space update geometry with predictive entropy and reasoning performance, offering a useful perspective on the mechanisms of LLM post-training.

  2. Carefully designed controlled interventions. The spectrum-preserving rotations provide a relatively rigorous way to isolate the effect of update orientation from magnitude and spectral structure. The additional controls and rank-sensitivity experiments strengthen the main empirical findings.

  3. Consistent empirical observations. The results are supported by experiments on two model lineages and multiple reasoning benchmarks. The combination of fixed-text and free-running entropy evaluations provides complementary evidence.

  4. Complementary weight-space and logit-level analyses. The investigation of accuracy improvements without entropy reduction, together with the token-specific logit decomposition, provides additional insights beyond the primary orientation experiments.

Critical weaknesses
  1. Limited novelty and insufficient evidence of RLVR-specific mechanisms. The finding that learned update directions have stronger functional effects than random rotations is somewhat expected for optimized neural networks. Without comparisons against SFT or other post-training methods, it remains unclear whether the observed orientation sensitivity is specific to RLVR or a general property of learned weight updates. This limits the novelty and mechanistic significance of the central claim.

  2. Insufficient characterization of the critical weight-space directions. Although the experiments demonstrate that learned orientations matter, they do not identify which layers, matrices, or singular components are primarily responsible for entropy reduction. The alignment experiments modify all matrices simultaneously and use relatively few random draws, limiting their ability to reveal the underlying mechanism.

  3. Limited robustness of the accuracy–entropy dissociation result. The main evidence relies on a single Gaussian perturbation search and one held-out evaluation split. The paper lacks repeated searches, cross-task validation, and comparisons with randomly selected or parameter-matched matrix subsets. Consequently, the result is an interesting empirical example but does not establish a robust or general relationship between accuracy and entropy.

  4. Weak connection between weight-space orientation and token-level mechanisms. The weight-space interventions and logit-level analyses are conducted largely independently. The paper does not demonstrate that the learned orientations reduce entropy specifically through the identified high-probability token changes. Furthermore, the importance of high-probability tokens may partly reflect general mathematical properties of Shannon entropy rather than an RLVR-specific mechanism.

  5. Limited experimental generality and statistical validation. The study covers only two approximately 7B–8B model lineages, with relatively few random interventions in several experiments. Additional training seeds, model scales, evaluation tasks, and uncertainty estimates would strengthen the generality and reliability of the conclusions.

Questions for authors

Q1. Clarification of the central contribution. The experiments establish that the learned RLVR orientation is more effective at reducing entropy than random rotations. However, what evidence supports interpreting this as a mechanism specific to RLVR rather than a general property of task-optimized weight directions? Could the authors clarify the precise novelty relative to prior studies of RLVR weight geometry and orientation?

Q2. Effect of truncation and norm rescaling. In Section 3, the truncated learned updates sometimes produce larger entropy reductions than the full RLVR update. How much of this difference is attributable to removing singular components versus rescaling the retained components? Could the authors report results for unscaled truncated updates to disentangle these effects?

Q3. Interpretation of the alignment experiments. In Figure 3, entropy decreases consistently with increasing alignment, whereas accuracy behaves differently across tasks. Does this suggest that entropy reduction and accuracy improvements depend on different functional directions? How should this observation be reconciled with the discussion in Section 4 suggesting that the RLVR direction is essential for accuracy?

Q4. Reliability of the accuracy–entropy comparison. For the Top-I30 experiment in Figure 4, what are the exact candidate population, selection procedure, and evaluation sample size? Could the authors provide confidence intervals for the reported 76% versus 75% accuracy estimates and clarify whether the observed accuracy gain without entropy reduction remains stable across different evaluation samples?

Q5. Interpretation of the token-level mechanism. Section 5 reports that approximately ten high-probability tokens account for most of the entropy reduction. Is this concentration primarily due to their probability mass, or do these tokens exhibit particularly distinctive RLVR-induced logit changes? Could the authors clarify how their findings distinguish an RLVR-specific mechanism from the general sensitivity of Shannon entropy to high-probability tokens?

Q6. Consistency of entropy measurements. The paper uses different entropy evaluation protocols across experiments, including fixed-text versus free-running evaluation and full-vocabulary versus top-20 entropy. How sensitive are the conclusions to these choices? In particular, could the authors clarify whether the accuracy–entropy dissociation remains consistent when entropy is evaluated on identical fixed sequences?

2 comments
AuthorsOpenReview ↗

Author Response

Thank you for your thoughtful comments and review.


Re: Q1

The primary contribution of our work is that the entropy collapse and the performance gain observed with RLVR aren’t inexorably linked. There are readily accessible configurations in the neighborhood of the RLVR configuration which boost entropy while maintaining or improving accuracy, and they are characterized by the orientation of the update alone. Regarding random rotations, we would like to highlight that we did not simply look at random rotations (the orthogonal testing regime) but also rotations preserving RLVR’s spectra, etc.
Relative to prior studies of RLVR weight geometry, the work in GRRR (Qi et al., 2026) suggests that the important updates of RL happen off the diagonal of base model updates under some rotation. We connect the effect of the update further to a particular direction that RL chooses. The RLVR update is special; unlike for example SFT, it only affects very few components which are responsible for both the entropy and accuracy effects. We performed an additional experiment to check if another popular post-training technique (distillation) has the same effect.
Base → Distil studies the Δ between the Base and Distil checkpoints, and Base → RL does this for a Zero-RL (RLVR starting from Base model) setting. Thus each of these updates only have effects from that specific kind of post-training. Our results are unequivocal: we do not find the entropy drop characteristic to RLVR in the distilled checkpoint.

Update typeBase → DistilBase → RL
Base Model Entropy0.3321540.332154
Full update0.3338940.282960
TRUNCATED0.3335170.232118
SPECTRUM0.332554 ± 0.0004770.332702 ± 0.001513
WITHIN0.332649 ± 0.0005970.332745 ± 0.001499
ORTHOGONAL0.332465 ± 0.0003720.332197 ± 0.000069

Re: W1

Thank you for pointing this out, we have performed further experiments with another model (Qwen 3.5 9B) controlling for post-training technique, performed an experiment to delineate the effect of rescaling, and an additional Gaussian perturbation search (Tulu/Llama 3.1 on MATH500; σ values: 0.0005, 0.001, and 0.002) to further bolster empirical support for our claims.

MethodEntropyAccuracy
SFT0.40160.301 ± 0.02
Full RLVR0.35210.460 ± 0.03
Top-30 perturbation0.40490.445 ± 0.02

Re: Q2

Thank you for the very interesting question. Without the rescaling the size of the update is greatly reduced (and perhaps better corresponds to fewer gradient updates), thus motivating us to rescale and match norms for a fair head-to-head comparison. The effect of rescaling is as follows:

MethodEntropy
Base0.23394
Full update0.22164
Truncated (with rescaling)0.21344
Truncated (without rescaling)0.21788

Re: Q3

Your question is well founded. We were not clear in our statement about RLVR direction and accuracy (Line 321): direction is essential for accuracy improvement and entropy collapse seen with RLVR but NOT performance in general. We have rewritten that part in the updated paper to make this clear. Our apologies, and please see our response to Q1 for our main message.


Re: Q4

We have updated the paper (Section 4.1) to contain the exact candidate population, selection procedure, and evaluation sample sizes. The estimated standard error of the mean is 0.14% for the accuracy values.


Re: Q5

The main point with regard to this set of experiments is to complement previous studies by showing that while the major contributor to the entropy reduction happens because of the most probable tokens, RLVR induces a re-ranking of tokens across a wide range of the vocabulary. This re-ranking is definitely an effect of RLVR and not just Shannon entropy's sensitivity to high probability tokens because the re-ranking is happening on rather low probability tokens.


Re: Q6

We report complementary free-running and fixed-sequence entropy values for the core results. The top-20 truncation is a computation saving measure (vLLM is significantly faster when not generating the full distribution over the vocabulary), and we do not expect a significant change arising out of this.

Area_Chair_oySoOpenReview ↗

Reminder: Author discussions

Dear Reviewer,

This is a brief reminder that responding to rebuttal comments and engaging in discussions is a core part of a reviewer's responsibilities. Without your participation, it will be much more difficult to provide a well-informed recommendation for the submission.

I therefore kindly ask you to acknowledge the authors' rebuttal, indicate whether it has addressed your concerns and whether it affects your overall assessment. Specifically, please clarify which points have not been sufficiently addressed and respond to any questions raised by the authors.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.