acceptodds
Under review as a conference paper at ICLR 2027

Get Hint-Augmented GRPO Done Right Through Activation Steering

Abstract

GRPO has become the dominant reinforcement learning method for eliciting reasoning capability in LLMs using verifiable rewards. However, it suffers from a fundamental failure mode: when a training question exceeds the base model's current capability, all generated responses fail and receive zero reward, yielding no gradient signal and stalling further capability expansion. An emerging class of GRPO extensions, termed Hint-Augmented GRPO (HA-GRPO), adds a “hint” to the context to reduce the difficulty of the question, thereby producing a usable gradient signal. Nevertheless, existing HA-GRPO methods share a key issue: no hint is available at inference time to append to the question, creating a training-inference context mismatch that leads to inconsistent performance gains. In this work, we reinterpret the function of hinting – not as reducing question difficulty, but as steering model behavior – and propose ASH (Activation Steering for Hinting), a novel HA-GRPO method built on this view. During training, the model is presented with both hinted and plain questions, while a trainable steering vector is optimized to widen the margins between the model's generation probabilities for responses to the hinted version relative to the plain version, thereby capturing the question-agnostic component of the behavioral shift induced by the hint. During inference, the trained steering vector is directly injected into the model's activation, reproducing the hint-induced behavior without requiring any hint in the context. Experiments with four LLMs (Qwen3-1.7B/4B-Base, Llama-3.2-1B/3B-Instruct) show that ASH consistently improves pass@ on competition-level mathematics benchmarks (AMC, OlympiadBench, AIME24, AIME25) and out-of-distribution benchmarks (Minerva, GPQA-Diamond). Notably, it improves average pass@8 of Qwen3-1.7B-Base and Qwen3-4B-Base by 2.60% and 2.21%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Reasoning, Reinforcement Learning with Verifiable Rewards, GRPO, Activation Steering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_v6pKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the training-inference mismatch problem in Hint-Augmented GRPO (HA-GRPO). The authors point out that GRPO often fails to learn from hard questions when all rollouts receive zero rewards. Existing HA-GRPO methods try to solve this by adding hints during training. However, these hints are not available at inference time, so the model may lose much of the benefit. To address this issue, the authors propose Activation Steering for Hinting (ASH). The main idea is to learn a global steering vector that captures the behavior induced by hints. During training, ASH generates responses with and without hints, and uses a DPO-style loss to update the steering vector. The model is then trained on both hinted and plain questions using GRPO. At inference time, the learned vector is injected into the model's activations, without requiring any hints. Experiments on four models show that ASH outperforms GRPO, QUESTA, SAGE, and HILL across six reasoning benchmarks. The authors also show that the learned steering vector can transfer to other tasks, such as LogiQA and LiveCodeBench.

Critical strengths
  • I think the motivation of this paper is clear. The authors show that although hints can help GRPO learn from difficult questions, the model's performance drops quite a lot when the hints are removed. I find the idea of using activation steering to capture the behavior induced by hints quite interesting. It directly addresses the training-inference mismatch problem.
  • I also like the idea of updating the steering vector during RL training, rather than learning a fixed vector offline. This allows the vector to follow the changes in the model during training. The DPO-style loss and the joint training on hinted and plain questions are also interesting designs. Another nice thing is that the model only needs a steering vector at inference time, without generating or retrieving any hints.
  • The experimental results are promising. The authors test ASH on four models and show consistent improvements across six benchmarks. For example, compared with HILL, ASH improves the average pass@8 by around 2.60 and 2.21 points on Qwen3-1.7B-Base and Qwen3-4B-Base, respectively. I also appreciate the ablation in Appendix G, which shows the contribution of activation steering. The training cost analysis and the transfer results on LogiQA and LiveCodeBench are also helpful.
Critical weaknesses

I am curious about the choice of the steering layer. The authors simply use the middle layer following prior work, but there are no experiments to justify this choice. I think it might be possible to use some simple offline experiments to find a suitable layer without running the full RL training for every layer. For example, the authors could learn steering vectors at different layers with a frozen model and compare their performance on a small validation set. This could help identify which layer works better for activation steering, rather than directly choosing the middle layer.

Questions for authors

I noticed that the figures in the paper are inserted as raster images rather than vector graphics. Could the authors please check and use vector graphics instead? I think this would make the figures clearer, especially when zooming in.

3 comments
AuthorsOpenReview ↗

Official Rebuttal

Dear Reviewer, thank you for thoroughly reviewing our paper and for appreciating our ideas and results.

Regarding the steering layer selection:

  • We chose the middle layer to avoid extensive layer-wise sweeps given our limited computational resources, and it keeps the method elegant by removing the need to search for a steering layer for each model.
  • We agree that an ablation study on this choice is necessary, and we thank you for suggesting a simplified experimental design. Following your suggestion, we conduct experiments with Qwen3-1.7B/4B-Base, where we train steering vectors at 7 different layers with frozen models. The table below reports pass@8 averaged across the competition-level mathematics benchmarks (AMC, OlympiadBench, AIME24, AIME25): | Qwen3-1.7B-Base | | Qwen3-4B-Base | | |---:|---:|---:|---:| | layer | average pass@8 | layer | average pass@8 | | 1 | 30.16 (-1.42) | 1 | 44.56 (-1.31) | | 5 | 30.76 (-0.82) | 7 | 45.33 (-0.54) | | 10 | 32.85 (+1.27) | 12 | 47.39 (+1.52) | | (middlemost) 14 | 33.72 (+2.14) | (middlemost) 18 | 49.18 (+3.31) | | 19 | 34.21 (+2.63) | 23 | 48.81 (+2.94) | | 23 | 32.30 (+0.72) | 29 | 46.75 (+0.88) | | 27 | 31.84 (+0.26) | 35 | 46.28 (+0.41) |

On both models, early layers hurt (down to -1.42 on 1.7B and -1.31 on 4B), and gains rise toward the middle and fade at the last layers (+0.26 and +0.41). Importantly, layers near the middle perform similarly (layers 14 and 19 on 1.7B, layers 18 and 23 on 4B). The method is therefore not sensitive to the exact middle layer, and a precise layer search would bring little additional benefit. This is consistent with prior work suggesting that early layers are too low-level to encode reasoning-relevant directions. We also hypothesize that late layers have too little remaining capacity to turn the steering signal into different outputs. The middle layer is thus a robust default that requires no per-model search, although we do not claim it is optimal for every model. We will add this ablation to the Appendix.

Thank you for pointing out the resolution issue with our figure; we will change it to a vector graphic in our revision.

We would be grateful if you would consider our responses and update your score during the discussion period, as we believe we have addressed your concerns. We would also be glad to engage further in discussion if you have any additional questions.

Reviewer_v6pKOpenReview ↗

Response to author

Thank you for running the additional experiments and sharing the results. It’s helpful to see that several layers near the middle perform similarly on both models, which supports using the middle layer as a practical default.

My suggestion to train steering vectors at different layers while keeping the base model frozen was meant as a quick sanity check. It still requires training and evaluating a separate vector for each candidate layer, so it may not be an efficient way to choose a layer for a new model.

I’d be interested in your perspective on a follow-up question. Are there any inexpensive proxy metrics that you think could help identify promising steering layers for a given model without running a layer-wise sweep?

AuthorsOpenReview ↗

Follow-up Response

Thank you for your quick reply and for an interesting follow-up question.

From our perspective and expertise, we anticipate two directions for identifying a good steering layer for a new model without any training:

  • Activation Perturbation (also called Activation Patching [1]) identifies which layers are important for a model’s behavior. We intervene on a specific layer's activation and test whether this intervention significantly alters the model’s output. If it does, we mark that layer as playing a key role in implementing the behavior. The effect of the intervention is quantified as the change in an output metric: ΔL=L(x∣interrupt(a))−L(x)\Delta L = L\big( x \mid \mathrm{interrupt}(\mathbf{a}) \big) - L(x), where LL is a metric measuring the difference between model outputs (e.g., KL divergence), xx is the input, and a\mathbf{a} is the original activation.
  • Information-Theoretic Quantities. A good steering layer is one where reasoning-relevant information is richest and most accessible. Two proxies can estimate this:
    • mutual information I(h(l);Y)I\big(\mathbf{h}^{(l)}; Y\big) between layer-ll activation and a reasoning-relevant variable YY (e.g., answer correctness), estimated with linear probes [2]
    • intrinsic dimension of the activations, estimated with the maximum-likelihood estimator [3]. The intrinsic dimension typically varies strongly with depth, and a low-dimensional layer is one where a single vector can plausibly shift behavior.

Both directions are much cheaper than layer-wise training. They still require a layer-wise evaluation, which we believe is hard to avoid. In practice, we would shortlist layers with these proxies and train the steering vector only at those layers.

[1] Meng, Kevin, et al. "Locating and editing factual associations in GPT." Advances in neural information processing systems 35 (2022): 17359-17372.

[2] Alain, Guillaume, and Yoshua Bengio. "Understanding intermediate layers using linear classifier probes, 2018." URL https://arxiv.org/abs/1610.01644 1610 (2018).

[3] Ansuini, Alessio, et al. "Intrinsic dimension of data representations in deep neural networks." Advances in Neural Information Processing Systems 32 (2019).

We hope that we have addressed all your concerns and would be glad to discuss any questions further. We would greatly appreciate it if you would consider these clarifications in your final assessment.

Reviewer_KKoNOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Activation Steering for Hinting (ASH) for improving language-model reasoning with hint-augmented GRPO. It addresses two challenges: difficult questions yield no useful relative reward signal when all sampled answers are incorrect, and models trained with hints may perform worse when those hints are unavailable at inference. ASH jointly trains the model on hinted and unhinted questions while learning a global activation-steering vector that favors behaviors elicited by hints. At inference, this vector is injected into the model’s internal activations without requiring a textual hint. Across four models and six benchmarks, ASH reports consistent pass@8 improvements over baselines.

Critical strengths
  • The proposed method is described clearly and easy to follow.
  • The ablations and transfers experiments are helpful to understand which factors contribute most to the performance.
Critical weaknesses
  • The paper misses a couple of baselines. They should be included in baseline comparison or the authors need to provide justification on why not including them.

    • “StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason”. The paper uses multiple levels of solution hints to guide exploration. It already describes hints as steering the policy towards better reasoning paths, making ASH’s conceptual “reframing” less original.
    • “Nudging the Boundaries of LLM Reasoning”. The paper already studies essentially the same starting problem: challenging prompts where all rollouts are incorrect. It provides insights into what constitutes a good hint that can unlock innovative reasoning paths.
    • “EXPO: Stable Reinforcement Learning with Expressive Policies”. The paper uses answer-conditioned self-explanations to obtain correct trajectories for difficult questions.
  • The paper only tested models of 1-4B parameters. As shown in prior work, the same algorithm that works at this scale may not work at larger scales. The authors should at least applies their method on 8-10B models.

  • The theorem bounds a reward gap using distributional divergence, but the paper does not establish that its actual steering objective necessarily reduces that divergence. More empirical studies would make the paper more sound.

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Official Rebuttal [1/2]

Dear Reviewer, thank you for your constructive feedback. Please find below our responses to your identified Weaknesses (W). We would be grateful if you would consider our responses and update your score during the discussion period, as we believe we have addressed your concerns.

W1. Missing baselines

Thank you for suggesting relevant papers. After reading the suggested papers, we would like to clarify the following:

  • Relation to existing baselines. EXPO, StepHint, and NuRL all fall into categories already covered in our Related Work, and we will add all three in the revised version. Specifically:
    • EXPO adds an auxiliary self-explanation term (ExP-SFT) to the GRPO objective, which still provides a useful gradient when all sampled rollouts are incorrect. However, unlike HA-GRPO methods, this term trains the model on answer-conditioned explanation, not answer-free problem solving, and the paper does not guarantee that this gradient improves the latter.
    • StepHint can be viewed as a generalized version of QUESTA, where it adaptively partitions the reference solution into 4 steps of hints rather than using the first 50% of sentences.
    • NuRL is a method using self-generated hints. It is similar to SAGE and HILL but simpler, and it predates both. We compared against SAGE and HILL because they are more recent and more representative of this category. These methods, like other HA-GRPO methods, keep the hint in the context. They therefore share the training-inference context mismatch that ASH is designed to remove.
  • New baseline: StepHint. As suggested, we added StepHint as a baseline. For a head-to-head comparison, we trained it with its released code on the same training data as ASH and evaluated it on the same benchmarks. The table reports pass@8. StepHint gives only modest gains over GRPO (+0.53/+0.81 on math and +0.84/+0.19 on OOD for 1.7B/4B). It beats QUESTA on math, which suggests that step-wise hints are better than a fixed prefix. However, its OOD gains are limited as it trails QUESTA by 2.84 points on 1.7B and is on par with it on 4B. It also falls below SAGE and HILL in all four settings. ASH outperforms StepHint by a large margin (+3.31/+4.48 on math and +4.95/+3.42 on OOD).
  • Originality. StepHint briefly describes hints as “steering the policy towards better reasoning paths”, but only as an intuition for why hints help. The method still appends hints in the context, so the mismatch remains. We directly adopt hinting-as-steering as our premise and build a method on it. ASH distills the question-agnostic part of the hint-induced behavioral shift into an activation steering vector that works at inference without any hint. We therefore believe StepHint does not diminish our work’s originality. | Method | Qwen3-1.7B-Base | | Qwen3-4B-Base | | |---|---:|---:|---:|---:| | | mathematics | OOD | mathematics | OOD | | Base | 31.58 | 39.45 | 45.87 | 52.68 | | GRPO | 35.66 | 55.05 | 50.54 | 66.86 | | QUESTA | 34.84 | 58.73 | 49.89 | 67.16 | | StepHint | 36.19 | 55.89 | 51.35 | 67.05 | | SAGE | 36.33 | 56.16 | 51.69 | 68.02 | | HILL | 37.53 | 57.00 | 53.58 | 68.34 | | ASH | 39.50 | 60.84 | 55.83 | 70.47 |

W2. Only tested models of 1-4B parameters

Thank you for this remark. Because of limited computational resources, our original experiments used models of 1-4B parameters, including Llama-3.2-1B/3B-Instruct and Qwen3-1.7B/4B-Base. Following your suggestion, we conducted additional experiments and report the results for Qwen3-8B-Base in the table below. ASH’s gains hold at 8B: it outperforms GRPO by +3.68 on competition-level math and +2.88 on out-of-distribution (OOD) benchmarks. These gains are consistent with those at 1-4B. QUESTA again shows no consistent gain over GRPO (-0.23 and +0.21), which matches the training-inference mismatch we describe in the paper. ASH outperforms QUESTA by +3.91 and +2.67. These results show that ASH’s benefit is not limited to small models, and we will add them to the final version.

MethodQwen3-8B-Base
mathematicsOOD
Base49.2252.12
GRPO53.8568.59
QUESTA53.6268.80
ASH57.5371.47
AuthorsOpenReview ↗

Official Rebuttal [2/2]

W3. The paper does not establish that its actual steering objective necessarily reduces that divergence

Thank you for this remark. Our theoretical analysis has two parts. We agree that Theorem 1 only bounds the performance gap in terms of distributional divergence, and it does not by itself show that our steering objective reduces that divergence. Besides the proof for Theorem 1, Appendix B connects the steering objective to the DPO framework (Rafailov et al., 2023). It does so by establishing preference between the hinted responses o+∼πθ(⋅∣q+)o^+ \sim \pi_\theta(\cdot \mid q^+) relative to the plain responses o∼πθ(⋅∣q)o \sim \pi_\theta(\cdot \mid q), evaluated under qq with the reference model πθ\pi_\theta. Within DPO, the implicit reward function parameterized by the model with activation steering πθv\pi_\theta^{\mathbf{v}} relative to the reference model πθ\pi_\theta is defined as:

rv(q,o)=βlog⁡πθv(o∣q)πθ(o∣q).r^{\mathbf{v}}(q, o) = \beta \log \frac{\pi_\theta^{\mathbf{v}}(o \mid q)}{\pi_\theta(o \mid q)}.

Importantly, minimizing L(v;q)\mathcal{L}(\mathbf{v}; q) performs maximum likelihood estimation of this implicit reward under the Bradley-Terry preference framework. As demonstrated by Rafailov et al. (2023), this estimation procedure aligns πθv(⋅∣q)∝πθ(⋅∣q)exp⁡(1βr∗(q,⋅))\pi_\theta^{\mathbf{v}}(\cdot \mid q) \propto \pi_\theta(\cdot \mid q) \exp\left( \frac{1}{\beta} r^{\ast}(q, \cdot) \right) with the target (optimal) π∗(⋅∣q)\pi^{\ast}(\cdot \mid q) – here, defined as πθ(⋅∣q+)\pi_\theta(\cdot \mid q^+) – that is, it pushes πθv(⋅∣q)\pi_\theta^{\mathbf{v}}(\cdot \mid q) to track πθ(⋅∣q+)\pi_\theta(\cdot \mid q^+). Therefore, minimizing L(v;q)\mathcal{L}(\mathbf{v}; q) is a surrogate for minimizing DKL(πθ(⋅∣q+)∥πθv(⋅∣q))D_{\text{KL}}\left(\pi_\theta(\cdot \mid q^+) \parallel \pi_\theta^{\mathbf{v}}(\cdot \mid q)\right).

AuthorsOpenReview ↗

A kind reminder

Dear Reviewer,

Thank you again for reviewing our paper. As the discussion period is coming to an end, we would be grateful if you could review our rebuttal and reconsider your final assessment, as we believe it addresses all of your concerns.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Official Rebuttal [1/3]

Dear Reviewer, thank you for your constructive feedback. Please find below our responses to your identified Weaknesses (W) and Questions (Q). We would be grateful if you would consider our responses and update your score during the discussion period, as we believe we have addressed your concerns.

W1. Insufficient support for the steering mechanism

Thank you for thoroughly reviewing our paper. Below are our point-by-point clarifications:

  • The vector-off-ASH trails HILL in evaluation. Under the vector-off condition, ASH reduces to a balance of GRPO and QUESTA because it jointly trains on both hinted and plain questions. Therefore, it is understandable that vector-off-ASH trails HILL in evaluation, which employs an advanced hint extraction mechanism and costs roughly 2 times the computational resources and training time. We agree that vector controls are important and therefore add the following two controls.
  • Compare with the simple equal-norm random vectors and difference-in-means. The table below shows the results of ASH-trained vectors, relative to equal-norm random vectors (where the vector is a random direction rescaled to the same L2 norm as the trained vector) and difference-in-means vectors (where the vector is the mean activation difference between hinted and plain questions, computed offline), transferred to a GRPO backbone. Steering with an equal-norm random vector leaves GRPO essentially unchanged (changes of -1.4 to +0.6 points, with inconsistent sign), so ASH’s gains do not come from perturbing activations at that norm. The difference-in-means vector, computed offline from hinted vs. plain questions, gives a small gain (+0.3 to +1.3 points), which shows that the hinting direction carries some useful signal. The ASH-trained vector is far larger: +2.2 and +3.2 points (Qwen3-1.7B, math and OOD) and +3.2 and +1.5 points (Qwen3-4B), about 2 to 5 times the difference-in-means gain. Offline contrastive vectors, as we discuss in Section 2, do not capture the behavior shift that hinting induces on complex reasoning tasks, whereas a vector trained jointly with the policy does. We will add these results to our revision. | Method | Qwen3-1.7B-Base | | Qwen3-4B-Base | | |---|---:|---:|---:|---:| | | mathematics | OOD | mathematics | OOD | | GRPO | 35.66 | 55.05 | 50.54 | 66.86 | | equal-norm random | 36.27 | 55.10 | 49.17 | 66.64 | | difference-in-means | 36.70 | 56.33 | 51.41 | 67.18 | | ASH-trained | 37.81 | 58.25 | 53.69 | 68.38 |
  • Gains could reflect response style or length. We agree that Equation 4 does not use correctness, so the steering vector may capture response style and length as well as content. We regard this as part of the hint’s effect, which every HA-GRPO method inherits, and ASH aims to keep that effect available at inference without the hint. All reported gains are pass@kk under verifiable rewards, so a shift in style or length improves our results only if it produces more correct answers. The same vector, trained only on math, also improves LogiQA and LiveCodeBench-v5 (Table 2) without retraining, where style and length conventions differ.
  • The vector's optimizer and final magnitude. We optimize the steering vector with Adam, as we do for the model. ASH uses no separate steering magnitude (equivalently, the magnitude is fixed at 1). Because v\mathbf{v} is trained, its norm is learned directly, and a scalar multiplier α\alpha would only reparameterize it: training α×v\alpha \times \mathbf{v} reaches exactly the same set of steering vectors as training v\mathbf{v}. The two choices differ only in the effective step size of the optimizer, which is already controlled by the learning rate.
  • The paper uses beta = 0.1. Thank you for this observation. We agree that in an idealized setting (unrestricted capacity, population loss), minimizing Equation 4 gives πθv(⋅∣q)∝πθ(⋅∣q)exp⁡(1βr∗(q,⋅))\pi_\theta^{\mathbf{v}}(\cdot | q) \propto \pi_\theta(\cdot | q) \exp\left( \frac{1}{\beta} r^*(q, \cdot) \right), which equals πθ(⋅∣q+)\pi_\theta(\cdot | q^+) only when β=1\beta = 1. We will revise the Appendix B wording.
    • Theorem 1 is unaffected. It relies only on bounded rewards and Pinsker’s inequality, so it holds for any β\beta. β\beta appears only in the heuristic that Equation 4 reduces the KL divergence.
    • That heuristic does not require reaching the idealized optimum. v\mathbf{v} is a single vector, updated once per training step, so it never gets near that optimum. What Equation 4 provides is the direction of the hint-induced shift; β\beta only controls when the sigmoid saturates. Because log-ratios are summed over long reasoning chains, β=0.1\beta = 0.1 keeps gradients active until the log-likelihood margin reaches about 1/β=101/\beta = 10, i.e., until the steered model favors the hinted response over the plain response by a factor of roughly e10e^{10}, whereas β=1\beta = 1 would saturate at a factor of only ee.
AuthorsOpenReview ↗

Official Rebuttal [2/3]

W2. Limited interpretability of evaluation gains

Thank you for thoroughly reviewing our paper. Below are our point-by-point clarifications:

  • pass@8 sits below the 4-option guessing ceiling. We agree that, when options are shown, the uniform-guess pass@8 on 4-option benchmarks like GPQA-Diamond is 90%. This value is an upper bound for a policy that always emits a valid option. It is not an expected score for a reasoning model, for 2 reasons. Answer consistency: a uniform guesser draws independent answers, so 8 samples cover all of the option space. A model’s samples are correlated, and it often repeats the same answer, right or wrong, across samples, which adds no coverage. Strict answer extraction: our protocol requires the model to produce a reasoning chain that ends in a final answer in the specified format (\boxed{}). A response that is truncated at the 4096-token limit, or whose final answer cannot be parsed, is scored as incorrect and is not replaced by a random guess. Measured pass@8 therefore reflects both reasoning quality and format and length compliance, and it can fall below the ceiling even when the model’s underlying accuracy is above chance.
  • How are multiple-choice questions presented and parsed? For 4-option benchmarks like GPQA-Diamond, the 4 options are written into the question as A, B, C, D lines. The model receives the same instruction as for math problems: "put your final answer within \boxed{}". Scoring uses the same pipeline as math benchmarks. We take the content of the last \boxed{…} in the response and compare it by exact match against the ground-truth letter. A response with no \boxed{}, including one truncated at the maximum length, is scored as incorrect, and we do not fall back to a random guess or to searching the text for an option.
  • Extraction failures and Answer diversity. We report extraction failures and number of distinct answers (for each question, how many different answers (e.g., A, B, C, D) appear among that question's extracted answers, which is 4.0 for a uniform-guess baseline) for Base, GRPO, and ASH with Qwen3-1.7B/4B-Base on GPQA-Diamond in the table below. The number of distinct answers is far below the 4.0 of a uniform guesser, so samples for the same question are correlated, and the 90% ceiling does not apply. ASH also has fewer distinct answers than GRPO (2.54 vs. 2.70; 2.25 vs. 2.37), so its higher pass@8 does not come from more diffuse answers. ASH’s GPQA-Diamond advantage over GRPO therefore likely reflects fewer extraction failures and especially better reasoning. | Method | Qwen3-1.7B-Base | | Qwen3-4B-Base | | |---|---:|---:|---:|---:| | | Extraction failure | No. distinct answers | Extraction failure | No. distinct answers | | Base | 71.97% | 1.78 | 58.14% | 1.72 | | GRPO | 28.40% | 2.70 | 15.90% | 2.37 | | ASH | 21.27% | 2.54 | 09.72% | 2.25 |
  • Missing sample count and seeds. For each test question, we draw n = 8 responses at temperature 0.6 and top-p 0.95, and compute pass@kk for k in {1, 2, 4, 8}. All training and evaluation use a seed of 0. We now state this in Appendix E.
  • No confidence intervals. Note that pass@kk is averaged over 8 samples per question, which removes most generation noise, and our conclusions do not rest on any single cell: ASH exceeds GRPO and HILL on all 24 model–benchmark combinations in Table 1, at every k (Figure 5). We now report the mean +/- std over 5 runs with 5 different seeds for each pass@8 of Base, GRPO, and ASH with Qwen3-1.7B/4B-Base on AIME24/25 below. Variability of pass@8 on AIME24/25 is modest for trained models (1.8 to 3.0 points) and lowest for ASH. ASH’s gains over GRPO (+2.4 to +4.9 points) exceed the seed standard deviation in all four cells. Together with the consistent gains across all benchmarks, models, and k, these results support ASH’s improvement over GRPO. | Method | Qwen3-1.7B-Base | | Qwen3-4B-Base | | |---|---:|---:|---:|---:| | | AIME24 | AIME25 | AIME24 | AIME25 | | Base | 08.91 +/- 4.963 | 14.44 +/- 4.789 | 29.39 +/- 4.578 | 27.17 +/- 4.498 | | GRPO | 21.48 +/- 2.789 | 14.87 +/- 2.981 | 33.88 +/- 2.357 | 32.12 +/- 2.651 | | ASH | 24.97 +/- 2.128 | 17.85 +/- 2.341 | 39.41 +/- 1.864 | 36.67 +/- 1.984 |
  • Explicit pass@1 comparisons. We indeed reported pass@1 (full pass@k) in Appendix F, and we explicitly report it on GPQA-Diamond below. ASH reaches 24.99% pass@1 on Qwen3-1.7B-Base and 38.92% on Qwen3-4B-Base. This is +6.18 and +4.98 points over GRPO (18.81 and 33.94), and +4.86 and +0.92 over HILL (20.13 and 38.00). For Qwen3-1.7B-Base, it is close to chance, and several baselines fall below it because truncated or unparseable responses are scored as incorrect. For Qwen3-4B-Base, the result is well above the 25% chance level. | Method | Qwen3-1.7B-Base | Qwen3-4B-Base | |---|---:|---:| | Base | 08.62 | 17.49 | | GRPO | 18.81 | 33.94 | | QUESTA | 22.87 | 36.19 | | SAGE | 19.37 | 37.50 | | HILL | 20.13 | 38.00 | | ASH | 24.99 | 38.92 |
AuthorsOpenReview ↗

Official Rebuttal [3/3]

Q1. Steering controls

Thank you for your question. As we described in W1 above, we optimize the steering vector with Adam, using the same learning rate as used for the model. ASH uses no separate steering magnitude (equivalently, the magnitude is fixed at 1). The final norms of the steering vector are 3.083 and 5.845 for Qwen3-1.7B-Base and Qwen3-4B-Base, respectively. As described above, we have also added two steering controls (equal-norm random vectors and difference-in-means) and clarified how the Appendix B argument applies at β=0.1\beta = 0.1.

Q2. Evaluation protocol

Thank you for your question. We have carefully responded to all of your concerns in the W2 above.

AuthorsOpenReview ↗

A kind reminder

Dear Reviewer,

Thank you again for reviewing our paper. As the discussion period is coming to an end, we would be grateful if you could review our rebuttal and reconsider your final assessment, as we believe it addresses all of your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.