acceptodds
Under review as a conference paper at ICLR 2027

Recovering RLVR from Failing Spurious Rewards with Subspace Regularization

Abstract

Reinforcement learning with verifiable rewards (RLVR) has driven large gains in LLM reasoning, yet it remains unclear how the reward signal shapes the resulting parameter update. Recent work reports a striking model-dependent behavior, where weak or spurious rewards can improve Qwen models while severely degrading other families such as Llama models. We investigate this failure through the geometry of reward-induced parameter updates and empirically find that the dominant output-side direction is substantially more reward-dependent than the corresponding input-side direction. This finding suggests a simple intervention: rather than replacing a failing reward, we can guide its parameter updates toward the direction induced by the true reward. Based on this insight, we introduce a subspace regularizer that guides weak-reward training using the dominant output-side direction obtained under true rewards. Across reward settings that otherwise fail, this guidance substantially recovers performance across mathematical reasoning benchmarks. While obtaining this direction from fully labeled training data would require extensive ground-truth supervision, we further show that it can be reliably estimated from only a small number of ground-truth examples, retaining much of the benefit of full-ground-truth guidance. Together, our results show that the update direction induced by a reward provides an actionable signal for recovering RLVR when weak or spurious rewards fail.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning with Verifiable Rewards (RLVR), LLM Reasoning, Spurious Rewards, Parameter Update Geometry, Subspace Regularization, Ground-Truth Guidance

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fpCvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how reward signals affect RLVR parameter updates, finding that dominant output-side directions are more reward-sensitive than input-side directions, and their alignment with true-reward directions correlates with model performance. Based on this observation, the authors propose a subspace regularizer that guides weak-reward training using true-reward update directions, recovering performance in several failing settings. They further show that guidance derived from only 64 labeled examples achieves performance comparable to full-data guidance.

Critical strengths
  1. The paper is generally well written and easy to follow.
  2. Analyzing spurious-reward training behavior from the perspective of parameter update geometry provides valuable insights.
Critical weaknesses
  1. Missing key few-ground-truth baselines and comparisons with semi-supervised methods. The paper does not compare the performance of models trained with limited ground-truth labels against the final two-stage models, making it difficult to assess the actual gains. Comparisons with existing semi-supervised RLVR methods are also missing.
  2. Lack of computational cost analysis. The two-stage training procedure may require nearly twice the computational cost of a single RLVR run, but the paper does not report actual training time, GPU-hours, or additional memory overhead.
  3. Insufficient analysis of labeled sample selection. The paper does not investigate how the selection of a small number of ground-truth examples affects the reference subspace and final performance.
  4. Insufficient explanation of the relationship between update geometry and model performance. The paper mainly relies on empirical correlations and directional regularization experiments, without adequately explaining why output-side directional similarity reflects final performance or whether this relationship is causal and consistent across different training settings.
Questions for authors
  1. How does the performance of the reference models trained with limited ground-truth labels compare with the proposed method? What happens if training continues directly from these reference models?
  2. How does the proposed method compare with existing semi-supervised RLVR methods, such as TraPO [1] and GeoMin [2]?
  3. What are the total GPU-hours, wall-clock time, and peak GPU memory usage of the two-stage training procedure? How much additional computational cost does it introduce compared to a single RLVR run?
  4. Are the 64 ground-truth-labeled examples randomly selected? How do different random subsets affect the reference subspace directions and final performance?
  5. Can the authors further explain the relationship between output-side directional similarity and final performance, potentially through deeper theoretical analysis or more direct empirical evidence?
  6. How does the proposed method compare with existing approaches based on low-rank parameter updates, such as AlphaRL [3] and RELEX [4]? Reference: [1] TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning [2] GeoMin: Data-Efficient Semi-Supervised RLVR via Geometric Distribution Modeling [3] On Predictability of Reinforcement Learning Dynamics for Large Language Models [4] You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories
6 comments

Response to Reviewer fpCv

We thank Reviewer fpCv for the detailed comments and positive assessment of the geometric analysis and presentation. We address the missing comparisons with standalone True RL, TraPO, and GeoMin, clarify the training stages and label selection, and add experiments without a supervised reference to examine the role of regularization.

All scores are mean accuracy (%) over six in-distribution math benchmarks. Base is the starting pretrained or instruction-tuned checkpoint before any RL training in our experiments. Llama-1B and Llama-3B denote Llama-3.2-Instruct models, and Qwen3-4B denotes Qwen3-4B-Base.

Weaknesses

W1. Missing few-label and semi-supervised baselines

Comparison with standalone True RL

We added the requested standalone comparison at the same 64-label budget, including Llama-3B and Qwen3-4B as additional settings where unguided Majority@16 fails. Here, uTrue(64)u_{\mathrm{True}(64)} is extracted from True RL on 64 verified problems, and each guided run starts from the base weights.

MethodLlama-1BQwen2.5-Math-1.5BLlama-3BQwen3-4B
Base3.3615.4015.7632.85
Majority@16 (no labels)0.5530.770.794.80
Standalone True RL (64 labels)9.1331.6112.6937.17
Majority@16 + uTrue(64)u_{\mathrm{True}(64)}8.3828.5715.1438.10

Standalone True RL is stronger on both smaller models, confirming that a few labels can already be effective. On Qwen3-4B, guidance reaches 38.10%, compared with 37.17% for standalone True RL (+0.93) and 32.85% for the base model. On Llama-3B, guidance reaches 15.14%, compared with 12.69% for standalone True RL (+2.45), but remains below the 15.76% base. Guidance therefore adds accuracy in some settings at the cost of an additional run. Our experiments examine recovery from the base model. Comparison with direct continuation from the reference under matched total compute remains future work.

Semi-supervised comparison at 512 labels

We also added TraPO and GeoMin on DeepScaleR at the 512-label budget. These methods use labels to assess self-reward reliability. Our method uses them to select the output direction along which weight changes are unpenalized.

MethodVerified labelsLlama-1BQwen2.5-Math-1.5B
Base03.3615.40
Standalone True RL5129.6631.44
TraPO5127.9631.02
GeoMin5122.1830.85
Majority@16 + uTrue(512)u_{\mathrm{True}(512)}5128.5528.84

Guidance scores above TraPO and GeoMin on Llama-1B and below both on Qwen2.5-Math, while standalone True RL leads on both. GeoMin falls below the Llama-1B base, illustrating that some verified answers do not ensure improvement across models. Our aim is to understand and recover failing reward training. W4 examines this directly for Majority@16 without using verified answers.

W2. Computational cost

Guidance with uTrue(64)u_{\mathrm{True}(64)} requires two RL runs, one to obtain the True reference and another for guided Majority@16 training. When their budgets are comparable, this is roughly twice the training compute of one RL run. The unsupervised L2, random direction, and Fisher variants in W4 each require one policy training run. Fisher also computes uFu_F once beforehand. This removes the supervised reference stage, without implying equal runtime across methods. Q3 addresses the requested measurements.

W3. Selection of labeled examples

The 64 verified examples were randomly selected. Table 4 of the original submission studies guidance with references trained on 64, 256, or 512 labeled problems.

W4. Relationship between update geometry and performance

Our central finding is an empirical asymmetry between output and input directions. Leading output directions differ substantially across successful and failing rewards, while leading input directions remain comparatively similar. Table 3 of the original submission tests whether the choice of side matters for recovery. With Majority@16 on Llama-1B and the full-data True reference, output guidance reaches 8.69%, compared with 0.92% for input guidance. This demonstrates a difference in their effectiveness, while its mechanism remains open.

We added experiments to examine whether recovery requires the supervised direction uTrueu_{\mathrm{True}}. Specifically, we replace it with a Fisher-based direction uFu_F computed from the base model's answer gradients. We use final-answer gradients because Majority@16 compares final answers, seeking a direction that permits useful adaptation while limiting drift elsewhere.

For each target matrix ℓ\ell, Gi(ℓ)G_i^{(\ell)} is the gradient at the base model of the mean log probability of the answer tokens in generated response ii. We use tokens from the final boxed-answer marker to the response end and exclude responses without this marker. From NN retained responses, we form their output-side second moment and take its leading eigenvector

u_F^{(\ell)}=\operatorname{eigvec}_{\max}(\widehat F^{(\ell)}).$$ Using this unit leading eigenvector in Eq. 3 gives $$\mathcal L=\mathcal L_{\mathrm{Maj@16}}+ \frac{\lambda}{2}\sum_{\ell\in\mathcal M} \left\|(I-u_F^{(\ell)}u_F^{(\ell)\top}) (W^{(\ell)}-W_{\mathrm{base}}^{(\ell)})\right\|_F^2.$$ Here, $\mathcal M$ indexes the target matrices. The direction is fixed during training, while the weights remain trainable. The penalty constrains weight changes orthogonal to $u_F$ and leaves changes along it unpenalized. We include [INTUITOR](https://arxiv.org/abs/2505.19590) as another method that learns without verified answers. Its self-certainty reward uses token probability distributions along each response, whereas Majority@16 rewards agreement among 16 sampled final answers. We evaluate INTUITOR's reward under our training recipe. The table covers the three models where both unguided rewards fall below base accuracy under this recipe. | Method | Llama-1B | Llama-3B | Qwen3-4B | |---|---:|---:|---:| | Base | 3.36 | 15.76 | 32.85 | | INTUITOR | 0.03 | 0.00 | 5.78 | | Majority@16 | 0.55 | 0.79 | 4.80 | | Majority@16 + Fisher-based regularization (answer tokens) | 8.18 | 18.93 | 36.85 | Fisher regularization raises accuracy above that of the base model in all three settings without a supervised reference. Its reference directions have low weighted mean absolute cosine similarity with $u_{\mathrm{True}}$, at 0.053 on Llama-1B and 0.092 on Qwen2.5-Math. The Llama-1B result shows that recovery is possible despite little average reference overlap, motivating the controls below. #### Regularization controls We add **L2**, which penalizes the squared distance of each target matrix from its base value, and **random direction**, which penalizes weight changes orthogonal to one fixed, normalized Gaussian output vector per matrix. Both controls and Fisher regularization use the same target matrices and coefficient within each model. We also include $u_{\mathrm{True}}$ from training on all verified answers. | Method | Llama-1B | Qwen2.5-Math-1.5B | |---|---:|---:| | Base | 3.36 | 15.40 | | Majority@16 | 0.55 | 30.77 | | Majority@16 + L2 | 7.68 | 25.78 | | Majority@16 + random direction | 7.54 | 25.74 | | Majority@16 + Fisher-based regularization (answer tokens) | 8.18 | 26.74 | | Majority@16 + $u_{\mathrm{True}}$ (all labels) | 8.69 | 29.89 | On Llama-1B, L2 and random directions provide most of the recovery seen with True guidance without changing the reward or using verified answers. Fisher reaches 8.18% and True guidance reaches 8.69%, suggesting a further benefit from direction selection. True guidance also introduces labeled information, whose contribution is not isolated here. On Qwen2.5-Math, all regularized variants remain below unguided Majority@16. The evidence therefore supports regularization as a way to recover some failing reward settings, while leaving open why output guidance is more effective than input guidance.

Questions

Q1. Reference accuracy and direct continuation

The first table in W1 reports the 64-label standalone accuracies. Guidance adds accuracy on Llama-3B and Qwen3-4B, while standalone True RL is stronger on the smaller models. We study recovery from the base model and leave direct continuation from the reference under matched total compute to future work.

Q2. Comparison with TraPO and GeoMin

The second table in W1 gives the requested comparison at 512 labels. Guidance leads both methods on Llama-1B and trails both on Qwen2.5-Math. This supports model-dependent effectiveness, rather than a claim of superiority over existing semi-supervised methods.

Q3. GPU-hours, wall-clock time, and peak memory

We have not collected controlled measurements of these quantities. The roughly twofold compute estimate in W2 counts the reference and guided runs and assumes comparable training budgets. The unsupervised variants remove the reference run, with an initial direction computation for Fisher.

Q4. Selection of the 64 labeled examples

The 64 verified examples were randomly selected for training the True RL reference.

Q5. Why does directional similarity relate to performance?

Figure 1b relates the alignment of final learned output directions with uTrueu_{\mathrm{True}} to accuracy. Our new controls address whether a True reference is required for recovery. W4 shows recovery without it, including with Fisher reference directions that have little average overlap with the True reference. This does not explain why the alignment of the final learned updates tracks accuracy, which remains an open question.

Q6. Comparison with low-rank update approaches

AlphaRL and RELEX extrapolate checkpoints from early training trajectories. Our soft penalty acts during training. Rank one specifies the unpenalized output direction, while higher-rank weight updates remain possible. We have not made a matched comparison with checkpoint extrapolation and do not claim an advantage over it.

Concluding remark

Our central finding is an asymmetry between output and input directions that informs regularization for RLVR with limited or no verified answers.

Reviewer_JxhmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why weak or spurious rewards improve some LLMs but degrade other LLMs during the RL training. The authors analyze the weight changes through SVD and find that the leading output-side direction is more sensitive to the reward than the input-side direction. Based on this observation, they introduce a regularization term that guides weak-reward training using an output-side subspace obtained from ground-truth reward training.

Critical strengths

The question itself is interesting, and the comparison between output-side and input-side directions provides a useful empirical observation.

Critical weaknesses

I do not think the paper in its current form meets the ICLR bar.

First, the practical motivation is not clear to me. In the full setting, the method already requires training a reference model with ground-truth rewards. It is unclear why we would prefer the additional weak-reward training compared with directly using the reference model. I understand that the paper also considers constructing the reference from a small labeled subset, which provides a more practical motivation. However, the authors do not report the performance of the reference model trained on that same subset. Besides, when ground-truth answers are unavailable, an LLM-as-a-Judge could provide an alternative reward signal. The author should compare with these baselines in terms of both performance and total training cost.

Second, the current ablations do not show whether the recovery depends on the true-reward subspace. It would be great if the author could compare with a random output-side subspace of the same rank and ordinary L2 regularization toward the base model. Here, the orthogonal-complement ablation penalizes only the reference direction, whereas the proposed method penalizes all output directions outside it.

Questions for authors

see the weakness

7 comments

Response to Reviewer Jxhm

We thank Reviewer Jxhm for the interest in our research question and the comparison of output and input directions. We address the concerns with standalone True RL baselines, experiments without a supervised reference, and the requested L2 and random direction controls. All scores are mean accuracy (%) over six in-distribution math benchmarks. Base denotes the starting checkpoint before any RL training in our experiments. Llama-1B and Llama-3B are Llama-3.2-Instruct models, and Qwen3-4B is Qwen3-4B-Base.

Weaknesses

W1. Practical motivation, reference baselines, and training cost

We added the missing comparison with standalone True RL at the same 64-label budget. Llama-3B and Qwen3-4B provide two additional settings where unguided Majority@16 fails. Here, uTrue(64)u_{\mathrm{True}(64)} is an output direction extracted from True RL on 64 verified problems, and the guided run starts from the base weights.

MethodLlama-1BQwen2.5-Math-1.5BLlama-3BQwen3-4B
Base3.3615.4015.7632.85
Majority@16 (no labels)0.5530.770.794.80
Standalone True RL (64 labels)9.1331.6112.6937.17
Majority@16 + uTrue(64)u_{\mathrm{True}(64)}8.3828.5715.1438.10
Standalone True RL (all labels)10.8731.8523.1642.96

Standalone True RL is stronger on both smaller models, confirming that few labeled examples can already be effective. On Qwen3-4B, Majority@16 + uTrue(64)u_{\mathrm{True}(64)} reaches 38.10%, compared with 37.17% for its reference (+0.93), with both above the 32.85% base. On Llama-3B, guidance reaches 15.14%, compared with 12.69% for its reference (+2.45), but remains below its 15.76% base. Full-data True RL is stronger on all four models. Guidance also requires a reference run and a separate guided run, so these results do not establish an advantage under matched total compute.

To address practicality without this supervised reference stage, we added experiments using only the policy's own outputs. We keep Majority@16 as the reward and replace uTrueu_{\mathrm{True}} with a Fisher-based direction uFu_F computed from the base model's answer gradients. We use final-answer gradients because Majority@16 compares final answers, seeking a direction that permits useful changes to those answers while limiting drift elsewhere.

For each target matrix ℓ\ell, Gi(ℓ)G_i^{(\ell)} is the gradient at the base model of the mean log probability of the answer tokens in generated response ii. We use tokens from the final boxed-answer marker to the response end and exclude responses without this marker. From NN retained responses, we form their output-side second moment and take its leading eigenvector

u_F^{(\ell)}=\operatorname{eigvec}_{\max}(\widehat F^{(\ell)}).$$ Using this unit leading eigenvector in Eq. 3 gives $$\mathcal L=\mathcal L_{\mathrm{Maj@16}}+ \frac{\lambda}{2}\sum_{\ell\in\mathcal M} \left\|(I-u_F^{(\ell)}u_F^{(\ell)\top}) (W^{(\ell)}-W_{\mathrm{base}}^{(\ell)})\right\|_F^2.$$ Here, $\mathcal M$ indexes the target matrices. The direction remains fixed during training. The penalty constrains weight changes orthogonal to $u_F$ and leaves changes along it unpenalized. We include [INTUITOR](https://arxiv.org/abs/2505.19590) as another method that learns without verified answers. It rewards self-certainty computed from token probability distributions along each response, whereas Majority@16 rewards agreement among 16 sampled final answers. We evaluate INTUITOR's reward under our training recipe. The table covers the three models where both unguided rewards fall below base accuracy under this recipe. | Method | Llama-1B | Llama-3B | Qwen3-4B | |---|---:|---:|---:| | Base | 3.36 | 15.76 | 32.85 | | INTUITOR | 0.03 | 0.00 | 5.78 | | Majority@16 | 0.55 | 0.79 | 4.80 | | Majority@16 + Fisher-based regularization (answer tokens) | 8.18 | 18.93 | 36.85 | Fisher regularization raises accuracy above that of the base model in all three settings using only the policy's own outputs. It requires one regularized Majority@16 training run and an initial computation of $u_F$, with no supervised reference run. We distinguish this reduction in training stages from a measured runtime advantage. Controlled GPU-hour, wall-clock, and peak-memory comparisons remain unreported. The Qwen2.5-Math controls in W2 also show that regularization is not beneficial for every model. Regarding the suggested external judge, it is a useful alternative source of feedback. Our experiments study learning from the policy's own outputs, so a judge introduces supervision from another model beyond this setting. We have not made that comparison and do not claim an accuracy or cost advantage over judge-based training.

W2. Whether recovery depends on the True-reward subspace

We thank the reviewer for suggesting L2 and random subspace controls. The orthogonal-complement ablation in Table 3 of the original submission changes both the reference subspace and its dimension. In the main method, the fixed projector selects the direction along which weight changes are unpenalized. The complement ablation instead penalizes weight changes along uTrueu_{\mathrm{True}} and leaves orthogonal changes unpenalized. In both cases, the projector is fixed, not the model weights.

We therefore add L2, which penalizes the squared distance of each target matrix from its base value, and random direction, which uses one fixed, normalized Gaussian output vector per matrix. Random and Fisher regularization leave weight changes along one output direction per matrix unpenalized, matching the rank of the True reference. The controls and Fisher use the same target matrices and coefficient within each model, without verified answers. We also report guidance using uTrueu_{\mathrm{True}} from a True RL run on all labels.

MethodLlama-1BQwen2.5-Math-1.5B
Base3.3615.40
Majority@160.5530.77
Majority@16 + L27.6825.78
Majority@16 + random direction7.5425.74
Majority@16 + Fisher-based regularization (answer tokens)8.1826.74
Majority@16 + uTrueu_{\mathrm{True}} (all labels)8.6929.89

On Llama-1B, L2 and random directions provide most of the recovery seen with True guidance, reaching 7.68% and 7.54% against 0.55% unguided and 3.36% for the base model. Thus, substantial recovery does not require a True reference. Leaving weight changes along the Fisher direction unpenalized gives 8.18%, and using the True direction gives 8.69%. These averages suggest an additional benefit from direction selection, although the True direction also supplies labeled information. On Qwen2.5-Math, True guidance leads the regularized variants but remains below unguided Majority@16.

These controls show that regularization itself can make a failing unsupervised reward useful for training. They leave open how much of the additional gain with True guidance comes from supervision, and why output guidance is more effective than input guidance.

Concluding remark

Our central finding is an asymmetry between output and input directions that informs regularization for RLVR with limited or no verified answers.

Dear Reviewer Jxhm,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Reviewer_CurXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how the reward used in RLVR is reflected in the geometry of layerwise parameter updates. Training four LLMs with GRPO on DeepScaleR under six rewards (True, Majority, Format, Wrong Label, Inverted, Random), the authors find that the leading left (output-side) singular directions of the updates differ across rewards much more than the leading right (input-side) ones, and that alignment with the True-reward output direction tracks weak-reward performance. Based on this, they propose a regularizer that penalizes the part of the weak-reward update lying outside the span of the True-reward leading left singular vector. With references from full-data True-reward RLVR, this improves several failing weak/spurious-reward runs, and references built by running True-reward RLVR on only 64-512 labeled problems give similar results.

Critical strengths
  • S1. Interesting and original observation. The output-side singular directions of RLVR updates depend strongly on the reward, while the input-side directions are largely shared across rewards, and alignment with the True-reward output direction tracks how well a weak or spurious reward works. The pattern is consistent across four models and matches the known model-dependent success of spurious rewards (e.g., majority voting helps Qwen but not Llama). This is a useful lens on why spurious rewards work for some models and not others, and it is the main reason my rating is not lower. On its own, however, a single descriptive observation does not carry the paper.
  • S2. Thorough code release. The anonymous repository is extensive, which makes the work much easier to understand, verify and build on.
Critical weaknesses

W1. No demonstrated practical value.

  • The reference direction requires labeled data and a full True-reward RLVR run, so the method costs more than plain True-reward RLVR, yet no guided run reaches True-reward performance.
  • Rewards such as Inverted or Wrong Label are not used in practice, so recovering them has little value in itself.
  • In the only realistic setting, label-free majority-vote training, guidance does not help on the models where majority voting already works (Qwen), and where it fails (Llama) the guided runs still fall short of True-reward training.

W2. The gains most plausibly come from the few labeled examples rather than from the method.

  • In the few-label setting, the reference direction is extracted from a True-reward RLVR run on the n labeled problems. Prior work (Wang et al., 2025, cited in the paper) shows that RLVR on very few labeled examples can already yield large gains, so this reference run is itself likely a strong model, yet its accuracy is never reported.
  • Although the reference checkpoint is not used as the initialization, the guided run is explicitly steered toward its update direction. It is therefore unsurprising that the guided run beats the failing weak reward: in my view, the improvement is most plausibly carried over from the labeled run rather than produced by the weak reward or the regularizer.
  • The method spends extra compute to reach a result that is better than the failing weak reward but still below, and not on par with, standard True-reward training. Without the reference checkpoint's own accuracy, the paper cannot show that it adds anything beyond what the few labels already provide.
Questions for authors
  1. What is the accuracy of the few-label reference checkpoints themselves (with like n=64n = 64), and how do the guided weak-reward runs compare with them?
  2. Is there a realistic reward setting, i.e., one used in practice because labels are unavailable or expensive (e.g., majority-vote/self-consistency rewards, noisy verifiers or model-based judges), in which the guided run improves over both the unguided weak-reward run and the reference checkpoint trained on the same labels? If the authors can show even a modest gain in such a setting, I would consider the method practically valuable and would raise my rating.
5 comments

Response to Reviewer CurX

We thank Reviewer CurX for the detailed review and positive comments on the geometric observation and code release. To address the concerns about practical value and transfer from labeled training, we add experiments without a supervised reference, comparisons with standalone True RL, and regularization controls.

All scores are mean accuracy (%) over six in-distribution math benchmarks. Base is the starting pretrained or instruction-tuned checkpoint before any RL training in our experiments. Llama-1B and Llama-3B denote Llama-3.2-Instruct models, and Qwen3-4B denotes Qwen3-4B-Base.

Weaknesses

W1. Practical value

Following the reviewer's suggestion, we added experiments in the setting where verified training answers are unavailable. Specifically, we keep Majority@16 as the reward and replace the supervised reference direction uTrueu_{\mathrm{True}} with a direction computed from the base model's own answers. This removes the need for a True RL reference run.

We use a Fisher-based construction from final-answer gradients, since Majority@16 compares final answers. For each target matrix ℓ\ell, let Gi(ℓ)G_i^{(\ell)} be the gradient at the base model of the mean log probability of the answer tokens in generated response ii. We use tokens from the final boxed-answer marker to the response end and exclude responses without this marker. From NN retained responses, we form their output-side second moment and take its leading eigenvector

u_F^{(\ell)}=\operatorname{eigvec}_{\max}(\widehat F^{(\ell)}).$$ We use this unit leading eigenvector in place of $u_{\mathrm{True}}$ in Eq. 3, giving $$\mathcal L=\mathcal L_{\mathrm{Maj@16}}+ \frac{\lambda}{2}\sum_{\ell\in\mathcal M} \left\|(I-u_F^{(\ell)}u_F^{(\ell)\top}) (W^{(\ell)}-W_{\mathrm{base}}^{(\ell)})\right\|_F^2.$$ Here, $\mathcal M$ indexes the target matrices. The direction is fixed throughout training, while the weights remain trainable. The penalty constrains weight changes orthogonal to $u_F$ and leaves changes along it unpenalized. We also compare with [INTUITOR](https://arxiv.org/abs/2505.19590), another method that learns without verified answers. Its self-certainty reward uses token probability distributions along a generated response, whereas Majority@16 rewards agreement among 16 sampled final answers. We evaluate INTUITOR's reward under our training recipe. The table covers the three models where both unguided rewards fall below base accuracy under this recipe. | Method | Llama-1B | Llama-3B | Qwen3-4B | |---|---:|---:|---:| | Base | 3.36 | 15.76 | 32.85 | | INTUITOR | 0.03 | 0.00 | 5.78 | | Majority@16 | 0.55 | 0.79 | 4.80 | | Majority@16 + Fisher-based regularization (answer tokens) | 8.18 | 18.93 | 36.85 | Fisher regularization raises accuracy above that of the base model in all three settings without verified answers or a supervised reference. It requires one regularized Majority@16 training run, with $u_F$ computed once beforehand. This demonstrates a practical use for regularization when only the policy's own feedback is available. The benefit is model dependent, as the Qwen2.5-Math controls in W2 show, and does not establish an accuracy or compute advantage over full-data True RL.

W2. Recovery may come from the labeled reference

We thank the reviewer for pointing out that a few labeled problems can already produce a strong reference. Starting the guided run from the base does not remove the supervised information in uTrueu_{\mathrm{True}}. We address this by comparing against standalone True RL and then testing recovery without that information.

Comparison with standalone True RL

Table 4 of the original submission studies guidance using 64, 256, or 512 labels. We now report the missing standalone comparison at 64 labels, including Llama-3B and Qwen3-4B as additional settings where Majority@16 fails. Here, uTrue(64)u_{\mathrm{True}(64)} is extracted from a True RL checkpoint trained on 64 verified problems, and the guided run starts from the base weights.

MethodLlama-1BQwen2.5-Math-1.5BLlama-3BQwen3-4B
Base3.3615.4015.7632.85
Majority@16 (no labels)0.5530.770.794.80
Standalone True RL (64 labels)9.1331.6112.6937.17
Majority@16 + uTrue(64)u_{\mathrm{True}(64)}8.3828.5715.1438.10

Standalone True RL is stronger on both smaller models, confirming the value of the few labeled examples themselves. On Qwen3-4B, guidance reaches 38.10%, compared with 37.17% for its reference (+0.93) and 32.85% for the base model. On Llama-3B, guidance reaches 15.14%, compared with 12.69% for its reference (+2.45), but remains below its 15.76% base. These gains require an additional guided run and do not establish an advantage under matched total compute. Full-data True RL remains stronger than 64-label guidance on all four models.

Regularization controls

To test recovery without labeled information, we add two controls. 'Majority@16 + L2' penalizes the squared distance of each target matrix from its base value. 'Majority@16 + random direction' uses one fixed, normalized Gaussian output vector per matrix and penalizes weight changes orthogonal to it. 'Majority@16 + Fisher-based regularization (answer tokens)' uses uFu_F defined in W1. All three use the same target matrices and coefficient within each model.

MethodLlama-1BQwen2.5-Math-1.5B
Base3.3615.40
Majority@160.5530.77
Majority@16 + L27.6825.78
Majority@16 + random direction7.5425.74
Majority@16 + Fisher-based regularization (answer tokens)8.1826.74

On Llama-1B, L2, random direction, and Fisher regularization all recover failing Majority@16 training while keeping the reward unchanged. None uses a supervised reference, showing that regularization can enable recovery without transferring information from labeled training. Qwen2.5-Math provides a comparison where Majority@16 already improves over the base model and the regularized variants remain below unguided training.

Questions

Q1. What are the few-label reference accuracies?

With 64 labels, standalone True RL reaches 9.13% on Llama-1B, 31.61% on Qwen2.5-Math-1.5B, 12.69% on Llama-3B, and 37.17% on Qwen3-4B. The first table in W2 compares them with guidance.

Q2. Does a realistic reward improve over both unguided training and the same-label reference?

Yes, in the reported averages. On Qwen3-4B, guided Majority@16 reaches 38.10%, compared with 4.80% unguided and 37.17% for its 64-label reference. Llama-3B also exceeds both baselines, but remains below base accuracy. W2 gives the full comparison and training-cost qualification. Separately, W1 shows recovery without any labeled reference, addressing the setting where verified answers are unavailable.

Concluding remark

Our central finding is an asymmetry between output and input directions that informs regularization for RLVR with limited or no verified answers.

Dear Reviewer CurX,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Comments on the paper

1 comment

We thank the Area Chair and reviewers for the constructive discussion. We have uploaded a revised manuscript incorporating the additional experiments and clarifications presented during the rebuttal, with changes highlighted in red.

The revision includes:

  • Comparisons with standalone True RL and the semi-supervised baselines TraPO and GeoMin (Section 5, Tables 5–6).
  • The Fisher construction and unsupervised recovery results without verified training answers or a supervised reference run (Section 5, Table 7, and Appendix B).
  • L2 and random-direction controls showing that generic regularization accounts for much of the recovery on Llama-1B, together with a clearer distinction between the choice of guided side and output-direction selection (Tables 3 and 8). We have also clarified the training protocol and computational scope and revised the discussion of limitations and future work, including practical extensions for semi-supervised and unsupervised RLVR and the study of update geometry across domains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.