acceptodds
Under review as a conference paper at ICLR 2027

Label-Free Steering: Compressing Test-Time Reinforcement Learning into Bias-Only Subspaces

Abstract

Test-time reinforcement learning (TTRL) enables models to improve their reasoning without relying on labeled training data, but existing approaches typically optimize a large fraction of the model parameters. This raises a natural question: can effective test-time adaptation emerge when both the reward signal and the optimization space are severely restricted? We answer this question with label-free bias-only TTRL, which uses majority-vote pseudo-labels as rewards and optimizes only 100K bias parameters while keeping the pretrained backbone frozen. On MATH-500, our approach reaches 76.67% accuracy with Qwen2.5-7B, slightly exceeding our own labeled bias-steering reproduction while optimizing 76,000 fewer parameters than full-parameter TTRL. The same training procedure improves performance across vision-language and audio reasoning tasks, including MathVista, AI2D, LogicVista, and MMAU. We further show that the learned steering vectors transfer to 4,500 held-out MATH problems, indicating that the adaptation is not limited to the problems used during test-time optimization. Finally, we analyze why this highly restricted adaptation can work, showing that majority-vote reliability improves with rollout consensus and that bias subspaces with greater accessible gradient energy exhibit stronger downstream trainability. These results demonstrate that substantial test-time adaptation can emerge from optimizing a tiny bias-only subspace using entirely label-free rewards. Code is in https://anonymous.4open.science/r/label-free-bias-steering-840E

Primary area:
reinforcement learning
Keywords:
Label-Free Learning, Test-Time Reinforcement Learning, Bias-Only Adaptation, Activation Steering, Multimodal Reasoning, Parameter-Efficient Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fxWROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper combines majority-vote test-time RL with per-layer additive bias adaptation, training roughly 100K parameters while freezing the pretrained model. It evaluates text, visual and audio tasks, compares with voting, LoRA and full fine-tuning, and reports transfer from MATH-500 to 4,500 disjoint MATH problems. Conditional pseudo-label and projected-gradient analyses motivate the approach, with empirical studies of consensus and layer-wise gradient energy. The submission also audits task-specific sources of multimodal improvement.

Critical strengths

Practical evidence for extremely small label-free adaptation. Three-seed results and a label-source by parameterization comparison show gains with about 100K parameters. The frozen transfer experiment demonstrates that the result extends beyond the exact adaptation questions

Critical weaknesses

The conceptual novelty and breadth are limited. The method directly combines existing TTRL and bias-only RL. The main scientific contribution is therefore a useful empirical combination, with a narrower demonstrated scope.

Questions for authors

Were LoRA and ReFT allowed comparable configuration search budgets and suitable learning rates?

Reviewer_UQGDOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This submission studies constrained test-time reinforcement learning (TTRL). Specifically, they apply TTRL (Zuo et al., 2025) to learn bias vectors to adapt to various tasks. Experiments are conducted on reasoning-intensive benchmarks and Qwen2.5-7B family models (base, instruct, math, omni). The model performance is compared to self-consistency, different LoRA variants on q/v_proj layers, and full fine-tuning. On almost all model x benchmark configurations, the authors report that Bias-only TTRL approaches full fine-tuning performance and outperforms significantly more parameter-expensive LoRA runs.

Critical strengths
  • The proposal of Bias-only TTRL is very intriguing: just training 100K parameters with "label-free" rewards matches adaptations of all parameters.
  • Experiments are conducted on multiple (multimodal) domains (text, vision, audio) and across multiple reasoning-intensive tasks
Critical weaknesses
  • All experiments are limited to the Qwen2.5-7B family (base, instruct, math, omni). This raises questions about if the method works on other models and how it scales with parameter size.
  • Models are all evaluated with greedy decoding, which is provably suboptimal in combination with reasoning and underestimates the performance of the baselines, therefore overestimating the gains of this method. It is necessary to reevaluate the models under suitable configurations to quantify actual gains.
  • The comparisons change both the intervention type and where they act: LoRA is done only on q/v_proj, FullFT changes all weights, Bias-only TTRL only trains the MLP biases. The results are incomparable or at most support a much weaker claim than currently formulated.

Code:

  • There is a risk of test-label leakage: the three vision-language trainers track the best accuracy, reload that checkpoint, and evaluate it on the full test set. Further, it is unclear how the "best" checkpoint from the hyperparameter test was selected--this, too, may have been done on test.
  • Unclear if reasoning was used on AI2D and LogicVista, as the default in the code is direct answer, not CoT
  • There is a bug in the Mathvisa grader where negative signs are discarded as part of cleaning. I ran a short experiment with Qwen2.5-VL-7B-Instruct on MathVista and observed a 0.5% change in accuracy. This is modest, but worth investigating with TTRL models.

Minor:

  • A comparison to the full TTRL baseline would be desirable.
Questions for authors
  • Why is Bias-only TTRL with gold truth labels less effective than pseudo labels?
  • How can Bias-only TTRL outperform full fine-tuning (Qwen2.5-VL-7B-Instruct on MathVista)?
Reviewer_V6yEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces label-free test-time reinforcement learning by combining majority-vote pseudo-label rewards with GRPO which improves performance across various tasks.

Critical strengths
  1. The paper is well-written and easy to follow.
  2. Experimental results include comparisons against self-consistency, multiple LoRA configurations, and full fine-tuning, alongside a controlled ablation separating the effects of reward supervision and trainable parameterization.
Critical weaknesses
  1. The methodological novelty is limited: The proposed approach directly combines TTRL’s existing majority-vote reward, standard GRPO, and a previously proposed bias-only steering mechanism.
  2. Mechanistic explanation of the proposed method remains incomplete: The generic projected-gradient bound and the correlation across nine bias layers on a single 16-problem gradient-estimation batch do not directly explain the observed advantage over LoRA and ReFT.
  3. Authors’ own audit attributes roughly two-thirds of the MMAU gain to a phoneme-counting template, while the remaining subset shows only 1.4% improvement without convincing statistical support. The evidence for general audio reasoning improvement is limited.
Questions for authors

Refer to the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.