- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the training-inference mismatch problem in Hint-Augmented GRPO (HA-GRPO). The authors point out that GRPO often fails to learn from hard questions when all rollouts receive zero rewards. Existing HA-GRPO methods try to solve this by adding hints during training. However, these hints are not available at inference time, so the model may lose much of the benefit. To address this issue, the authors propose Activation Steering for Hinting (ASH). The main idea is to learn a global steering vector that captures the behavior induced by hints. During training, ASH generates responses with and without hints, and uses a DPO-style loss to update the steering vector. The model is then trained on both hinted and plain questions using GRPO. At inference time, the learned vector is injected into the model's activations, without requiring any hints. Experiments on four models show that ASH outperforms GRPO, QUESTA, SAGE, and HILL across six reasoning benchmarks. The authors also show that the learned steering vector can transfer to other tasks, such as LogiQA and LiveCodeBench.
Critical strengths
- I think the motivation of this paper is clear. The authors show that although hints can help GRPO learn from difficult questions, the model's performance drops quite a lot when the hints are removed. I find the idea of using activation steering to capture the behavior induced by hints quite interesting. It directly addresses the training-inference mismatch problem.
- I also like the idea of updating the steering vector during RL training, rather than learning a fixed vector offline. This allows the vector to follow the changes in the model during training. The DPO-style loss and the joint training on hinted and plain questions are also interesting designs. Another nice thing is that the model only needs a steering vector at inference time, without generating or retrieving any hints.
- The experimental results are promising. The authors test ASH on four models and show consistent improvements across six benchmarks. For example, compared with HILL, ASH improves the average pass@8 by around 2.60 and 2.21 points on Qwen3-1.7B-Base and Qwen3-4B-Base, respectively. I also appreciate the ablation in Appendix G, which shows the contribution of activation steering. The training cost analysis and the transfer results on LogiQA and LiveCodeBench are also helpful.
Critical weaknesses
I am curious about the choice of the steering layer. The authors simply use the middle layer following prior work, but there are no experiments to justify this choice. I think it might be possible to use some simple offline experiments to find a suitable layer without running the full RL training for every layer. For example, the authors could learn steering vectors at different layers with a frozen model and compare their performance on a small validation set. This could help identify which layer works better for activation steering, rather than directly choosing the middle layer.
Questions for authors
I noticed that the figures in the paper are inserted as raster images rather than vector graphics. Could the authors please check and use vector graphics instead? I think this would make the figures clearer, especially when zooming in.
3 comments
Response to author
Thank you for running the additional experiments and sharing the results. It’s helpful to see that several layers near the middle perform similarly on both models, which supports using the middle layer as a practical default.
My suggestion to train steering vectors at different layers while keeping the base model frozen was meant as a quick sanity check. It still requires training and evaluating a separate vector for each candidate layer, so it may not be an efficient way to choose a layer for a new model.
I’d be interested in your perspective on a follow-up question. Are there any inexpensive proxy metrics that you think could help identify promising steering layers for a given model without running a layer-wise sweep?
Follow-up Response
Thank you for your quick reply and for an interesting follow-up question.
From our perspective and expertise, we anticipate two directions for identifying a good steering layer for a new model without any training:
- Activation Perturbation (also called Activation Patching [1]) identifies which layers are important for a model’s behavior. We intervene on a specific layer's activation and test whether this intervention significantly alters the model’s output. If it does, we mark that layer as playing a key role in implementing the behavior. The effect of the intervention is quantified as the change in an output metric: , where is a metric measuring the difference between model outputs (e.g., KL divergence), is the input, and is the original activation.
- Information-Theoretic Quantities. A good steering layer is one where reasoning-relevant information is richest and most accessible. Two proxies can estimate this:
- mutual information between layer- activation and a reasoning-relevant variable (e.g., answer correctness), estimated with linear probes [2]
- intrinsic dimension of the activations, estimated with the maximum-likelihood estimator [3]. The intrinsic dimension typically varies strongly with depth, and a low-dimensional layer is one where a single vector can plausibly shift behavior.
Both directions are much cheaper than layer-wise training. They still require a layer-wise evaluation, which we believe is hard to avoid. In practice, we would shortlist layers with these proxies and train the steering vector only at those layers.
[1] Meng, Kevin, et al. "Locating and editing factual associations in GPT." Advances in neural information processing systems 35 (2022): 17359-17372.
[2] Alain, Guillaume, and Yoshua Bengio. "Understanding intermediate layers using linear classifier probes, 2018." URL https://arxiv.org/abs/1610.01644 1610 (2018).
[3] Ansuini, Alessio, et al. "Intrinsic dimension of data representations in deep neural networks." Advances in Neural Information Processing Systems 32 (2019).
We hope that we have addressed all your concerns and would be glad to discuss any questions further. We would greatly appreciate it if you would consider these clarifications in your final assessment.
Official Rebuttal
Dear Reviewer, thank you for thoroughly reviewing our paper and for appreciating our ideas and results.
Regarding the steering layer selection:
On both models, early layers hurt (down to -1.42 on 1.7B and -1.31 on 4B), and gains rise toward the middle and fade at the last layers (+0.26 and +0.41). Importantly, layers near the middle perform similarly (layers 14 and 19 on 1.7B, layers 18 and 23 on 4B). The method is therefore not sensitive to the exact middle layer, and a precise layer search would bring little additional benefit. This is consistent with prior work suggesting that early layers are too low-level to encode reasoning-relevant directions. We also hypothesize that late layers have too little remaining capacity to turn the steering signal into different outputs. The middle layer is thus a robust default that requires no per-model search, although we do not claim it is optimal for every model. We will add this ablation to the Appendix.
Thank you for pointing out the resolution issue with our figure; we will change it to a vector graphic in our revision.
We would be grateful if you would consider our responses and update your score during the discussion period, as we believe we have addressed your concerns. We would also be glad to engage further in discussion if you have any additional questions.