- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how reward signals affect RLVR parameter updates, finding that dominant output-side directions are more reward-sensitive than input-side directions, and their alignment with true-reward directions correlates with model performance. Based on this observation, the authors propose a subspace regularizer that guides weak-reward training using true-reward update directions, recovering performance in several failing settings. They further show that guidance derived from only 64 labeled examples achieves performance comparable to full-data guidance.
Critical strengths
- The paper is generally well written and easy to follow.
- Analyzing spurious-reward training behavior from the perspective of parameter update geometry provides valuable insights.
Critical weaknesses
- Missing key few-ground-truth baselines and comparisons with semi-supervised methods. The paper does not compare the performance of models trained with limited ground-truth labels against the final two-stage models, making it difficult to assess the actual gains. Comparisons with existing semi-supervised RLVR methods are also missing.
- Lack of computational cost analysis. The two-stage training procedure may require nearly twice the computational cost of a single RLVR run, but the paper does not report actual training time, GPU-hours, or additional memory overhead.
- Insufficient analysis of labeled sample selection. The paper does not investigate how the selection of a small number of ground-truth examples affects the reference subspace and final performance.
- Insufficient explanation of the relationship between update geometry and model performance. The paper mainly relies on empirical correlations and directional regularization experiments, without adequately explaining why output-side directional similarity reflects final performance or whether this relationship is causal and consistent across different training settings.
Questions for authors
- How does the performance of the reference models trained with limited ground-truth labels compare with the proposed method? What happens if training continues directly from these reference models?
- How does the proposed method compare with existing semi-supervised RLVR methods, such as TraPO [1] and GeoMin [2]?
- What are the total GPU-hours, wall-clock time, and peak GPU memory usage of the two-stage training procedure? How much additional computational cost does it introduce compared to a single RLVR run?
- Are the 64 ground-truth-labeled examples randomly selected? How do different random subsets affect the reference subspace directions and final performance?
- Can the authors further explain the relationship between output-side directional similarity and final performance, potentially through deeper theoretical analysis or more direct empirical evidence?
- How does the proposed method compare with existing approaches based on low-rank parameter updates, such as AlphaRL [3] and RELEX [4]? Reference: [1] TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning [2] GeoMin: Data-Efficient Semi-Supervised RLVR via Geometric Distribution Modeling [3] On Predictability of Reinforcement Learning Dynamics for Large Language Models [4] You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories
6 comments
W4. Relationship between update geometry and performance
Our central finding is an empirical asymmetry between output and input directions. Leading output directions differ substantially across successful and failing rewards, while leading input directions remain comparatively similar. Table 3 of the original submission tests whether the choice of side matters for recovery. With Majority@16 on Llama-1B and the full-data True reference, output guidance reaches 8.69%, compared with 0.92% for input guidance. This demonstrates a difference in their effectiveness, while its mechanism remains open.
We added experiments to examine whether recovery requires the supervised direction . Specifically, we replace it with a Fisher-based direction computed from the base model's answer gradients. We use final-answer gradients because Majority@16 compares final answers, seeking a direction that permits useful adaptation while limiting drift elsewhere.
For each target matrix , is the gradient at the base model of the mean log probability of the answer tokens in generated response . We use tokens from the final boxed-answer marker to the response end and exclude responses without this marker. From retained responses, we form their output-side second moment and take its leading eigenvector
u_F^{(\ell)}=\operatorname{eigvec}_{\max}(\widehat F^{(\ell)}).$$ Using this unit leading eigenvector in Eq. 3 gives $$\mathcal L=\mathcal L_{\mathrm{Maj@16}}+ \frac{\lambda}{2}\sum_{\ell\in\mathcal M} \left\|(I-u_F^{(\ell)}u_F^{(\ell)\top}) (W^{(\ell)}-W_{\mathrm{base}}^{(\ell)})\right\|_F^2.$$ Here, $\mathcal M$ indexes the target matrices. The direction is fixed during training, while the weights remain trainable. The penalty constrains weight changes orthogonal to $u_F$ and leaves changes along it unpenalized. We include [INTUITOR](https://arxiv.org/abs/2505.19590) as another method that learns without verified answers. Its self-certainty reward uses token probability distributions along each response, whereas Majority@16 rewards agreement among 16 sampled final answers. We evaluate INTUITOR's reward under our training recipe. The table covers the three models where both unguided rewards fall below base accuracy under this recipe. | Method | Llama-1B | Llama-3B | Qwen3-4B | |---|---:|---:|---:| | Base | 3.36 | 15.76 | 32.85 | | INTUITOR | 0.03 | 0.00 | 5.78 | | Majority@16 | 0.55 | 0.79 | 4.80 | | Majority@16 + Fisher-based regularization (answer tokens) | 8.18 | 18.93 | 36.85 | Fisher regularization raises accuracy above that of the base model in all three settings without a supervised reference. Its reference directions have low weighted mean absolute cosine similarity with $u_{\mathrm{True}}$, at 0.053 on Llama-1B and 0.092 on Qwen2.5-Math. The Llama-1B result shows that recovery is possible despite little average reference overlap, motivating the controls below. #### Regularization controls We add **L2**, which penalizes the squared distance of each target matrix from its base value, and **random direction**, which penalizes weight changes orthogonal to one fixed, normalized Gaussian output vector per matrix. Both controls and Fisher regularization use the same target matrices and coefficient within each model. We also include $u_{\mathrm{True}}$ from training on all verified answers. | Method | Llama-1B | Qwen2.5-Math-1.5B | |---|---:|---:| | Base | 3.36 | 15.40 | | Majority@16 | 0.55 | 30.77 | | Majority@16 + L2 | 7.68 | 25.78 | | Majority@16 + random direction | 7.54 | 25.74 | | Majority@16 + Fisher-based regularization (answer tokens) | 8.18 | 26.74 | | Majority@16 + $u_{\mathrm{True}}$ (all labels) | 8.69 | 29.89 | On Llama-1B, L2 and random directions provide most of the recovery seen with True guidance without changing the reward or using verified answers. Fisher reaches 8.18% and True guidance reaches 8.69%, suggesting a further benefit from direction selection. True guidance also introduces labeled information, whose contribution is not isolated here. On Qwen2.5-Math, all regularized variants remain below unguided Majority@16. The evidence therefore supports regularization as a way to recover some failing reward settings, while leaving open why output guidance is more effective than input guidance.Questions
Q1. Reference accuracy and direct continuation
The first table in W1 reports the 64-label standalone accuracies. Guidance adds accuracy on Llama-3B and Qwen3-4B, while standalone True RL is stronger on the smaller models. We study recovery from the base model and leave direct continuation from the reference under matched total compute to future work.
Q2. Comparison with TraPO and GeoMin
The second table in W1 gives the requested comparison at 512 labels. Guidance leads both methods on Llama-1B and trails both on Qwen2.5-Math. This supports model-dependent effectiveness, rather than a claim of superiority over existing semi-supervised methods.
Q3. GPU-hours, wall-clock time, and peak memory
We have not collected controlled measurements of these quantities. The roughly twofold compute estimate in W2 counts the reference and guided runs and assumes comparable training budgets. The unsupervised variants remove the reference run, with an initial direction computation for Fisher.
Q4. Selection of the 64 labeled examples
The 64 verified examples were randomly selected for training the True RL reference.
Q5. Why does directional similarity relate to performance?
Figure 1b relates the alignment of final learned output directions with to accuracy. Our new controls address whether a True reference is required for recovery. W4 shows recovery without it, including with Fisher reference directions that have little average overlap with the True reference. This does not explain why the alignment of the final learned updates tracks accuracy, which remains an open question.
Q6. Comparison with low-rank update approaches
AlphaRL and RELEX extrapolate checkpoints from early training trajectories. Our soft penalty acts during training. Rank one specifies the unpenalized output direction, while higher-rank weight updates remain possible. We have not made a matched comparison with checkpoint extrapolation and do not claim an advantage over it.
Concluding remark
Our central finding is an asymmetry between output and input directions that informs regularization for RLVR with limited or no verified answers.
Response to Reviewer fpCv
We thank Reviewer fpCv for the detailed comments and positive assessment of the geometric analysis and presentation. We address the missing comparisons with standalone True RL, TraPO, and GeoMin, clarify the training stages and label selection, and add experiments without a supervised reference to examine the role of regularization.
All scores are mean accuracy (%) over six in-distribution math benchmarks. Base is the starting pretrained or instruction-tuned checkpoint before any RL training in our experiments. Llama-1B and Llama-3B denote Llama-3.2-Instruct models, and Qwen3-4B denotes Qwen3-4B-Base.
Weaknesses
W1. Missing few-label and semi-supervised baselines
Comparison with standalone True RL
We added the requested standalone comparison at the same 64-label budget, including Llama-3B and Qwen3-4B as additional settings where unguided Majority@16 fails. Here, uTrue(64) is extracted from True RL on 64 verified problems, and each guided run starts from the base weights.
Standalone True RL is stronger on both smaller models, confirming that a few labels can already be effective. On Qwen3-4B, guidance reaches 38.10%, compared with 37.17% for standalone True RL (+0.93) and 32.85% for the base model. On Llama-3B, guidance reaches 15.14%, compared with 12.69% for standalone True RL (+2.45), but remains below the 15.76% base. Guidance therefore adds accuracy in some settings at the cost of an additional run. Our experiments examine recovery from the base model. Comparison with direct continuation from the reference under matched total compute remains future work.
Semi-supervised comparison at 512 labels
We also added TraPO and GeoMin on DeepScaleR at the 512-label budget. These methods use labels to assess self-reward reliability. Our method uses them to select the output direction along which weight changes are unpenalized.
Guidance scores above TraPO and GeoMin on Llama-1B and below both on Qwen2.5-Math, while standalone True RL leads on both. GeoMin falls below the Llama-1B base, illustrating that some verified answers do not ensure improvement across models. Our aim is to understand and recover failing reward training. W4 examines this directly for Majority@16 without using verified answers.
W2. Computational cost
Guidance with uTrue(64) requires two RL runs, one to obtain the True reference and another for guided Majority@16 training. When their budgets are comparable, this is roughly twice the training compute of one RL run. The unsupervised L2, random direction, and Fisher variants in W4 each require one policy training run. Fisher also computes uF once beforehand. This removes the supervised reference stage, without implying equal runtime across methods. Q3 addresses the requested measurements.
W3. Selection of labeled examples
The 64 verified examples were randomly selected. Table 4 of the original submission studies guidance with references trained on 64, 256, or 512 labeled problems.