acceptodds
Withdrawn from ICLR 2027

ZOSteer: Steering Language Models from Scalar Rewards Alone

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Activation steering controls a language model by adding a fixed vector to its residual stream, but existing methods derive this vector from contrastive prompt pairs or differentiable objectives, leaving behaviors specified only by a scalar, non-differentiable reward largely out of reach. We introduce ZOSteer, which estimates a steering vector as the zeroth-order gradient of the reward in activation space. We inject isotropic Gaussian perturbations at a single layer, score each perturbed model with the reward, and take the cross-covariance between perturbations and rewards. This estimates the gradient of the Gaussian-smoothed reward from forward passes alone, without contrastive data or backpropagation. Because the reward is an average over items, the same perturbations also yield a gradient for every training item at no extra cost. The residual spectrum of these item gradients reveals sub-populations whose gradients conflict with the majority, and reweighting them repairs the pooled direction without new forward passes. Across three 1B-scale models and four benchmarks, ZOSteer gives the largest gains on GSM8K for every model, and its reweighted variant closes most of the gap to the strongest baseline on bAbI QA20 and surpasses it on CUTE SubChar, typically with fewer perturbations than the hidden dimension. These results suggest that what a single vector can steer depends on how a task's items agree, not only on how the model represents the capability.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
activation steering, language models, linear representation hypothesis

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_DXwCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work presents a zeroth order steering approach that estimates the behavioral steering vector as the one that maximizes the expected reward over the local Gaussian perturbations in the residual stream at a certain layer. Using Stein's identity, they estimate the optimal gradient vector for this smoothed reward function by perturbing a set of training samples. Results on verifiable datasets (including an interesting coding dataset) and two small (1B) language models show that steering the middle layer with the proposed approach is more effective than the baselines.

Critical strengths
  • The proposed method is sound and is motivated by some theory.
  • The steering computation protocol with local perturbations in the residual stream is interesting.
  • They also analyze the residual activations from the items when the mean gradient vector is removed.
  • Considered datasets are probably interesting as they are not very common in the literature.
Critical weaknesses
  • Table 2 is missing.
  • Number of model forward passes is prohibitively large during steering vector derivation. While they mention it as a limitation, it is at the scale of the model dimension (millions), which is not justifiable and is no longer more efficient than backward passes (memory permitting). While they also mention batching as a solution, it does not scale as the model size increases. Also, perhaps a fair comparison of the baselines would be to consider the Gaussian-perturbed samples as additional samples for steering vector computation since their reward can still be calculated. For example, instead of just using 16 for DiffMean, it should be 16 * N to calculate the DiffMean vector (to also smooth the DiffMean vector in a way).
  • Evaluation is only conducted on small 1B models and generalizability/scalability to larger models remains open.
  • Benchmark selection is not clearly standard as other reward-based behavioral datasets such as refusal and toxicity were not considered, which are more relevant in the steering literatures [1].
  • It is not clear why smoothed reward is used. The current motivation is not really correct since the output of MδM_{\delta} should change everytime δ\delta changes and why is its gradient zero everywhere. The use of Gaussian-smoothed reward however instead of the actual reward however is interesting for generalizability of the average gradient direction without requiring more samples and just using the perturbations in the input.
  • Sample cost arguments in lines 172-181 are not completely clear and must be made more accessible.
  • Generation examples are not provided to substantiate the qualitative effectiveness for the proposed steering.
  • Stronger baselines can probably be added as well like angular steering, OdeSteer, ReFT.
  • Minor:
    • Eq.equation is used. Fix it to use either of them.

[1] Nguyen, Dung Viet, et al. "Activation steering with a feedback controller." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors
  • How is the extremely large number of forward passes during steering vector computation feasible or justifiable in real settings? Is there any way to circumvent it in budgeted settings, probably inspired by the zero-order black-box adversarial settings.
  • What if we consider other local perturbation mechanisms than Gaussian noise?
  • What happens if we use the perturbed vectors as additional samples for the baselines?
  • How do the methods perform on behavioral reward datasets and bigger llms?
  • How sensitive is the approach to α\alpha? Shouldn't baselines also tuned for this alpha parameter which is just a steering multiplier?
  • Why was a single middle layer used to steer instead of a layer sweep?
  • How sensitive is the vector with the number of training samples? I believe this could be effective when we have fairly few training samples but can still enummerate through local perturabtions.
Reviewer_zK7hOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for activation steering which estimates the steering vector based on the gradient of the reward in the activation space. The method uses Gaussian perturbations to the residual stream of the middle layer and estimates the gradient of the smooth reward perturbed by noise. The experiments are performed on two 1B and one 1.5B model. On GSM8K, the proposed method wins across all three models. Their reweighted version becomes competitive on bAbI QA20 and CUTE SubChar datasets and on MBPP no method is performing well.

Critical strengths

The paper strengths are the simplicity of the steering procedure and some promising empirical results. The weighting procedure is well motivated by the problem where different subgroups have very different estimated gradients.

Critical weaknesses

Limited Experiments.

The paper lacks some additional baselines from the existing literature.

The paper is missing compute cost analysis on the proposed method and the baselines, as well as the compute cost compared to the RL.

The biggest evaluated model size is small, 1.5B, so the performance and scalability due to compute cost on larger models is unknown. Also, for larger models the gradient dimension might be larger, and it’s unclear how feasible it is to estimate those gradients.

A big methodological weakness.

The subgroup analysis and hence reweighting approach requires human inspection, so the approach is not fully automated, questioning its reproducibility and generalizability to new datasets.

Questions for authors

See the weaknesses plus these additional questions:

  1. Can the authors add more baselines such as RePS “Improved Representation Steering for Language Models” Wu et al. (2025) (https://arxiv.org/pdf/2505.20809) since this method can be more competitive than BiPO and it’s in the same class of method using the preference data?

  2. How does the method compare to RL-trained steering vectors such as “Steering LLM Reasoning Through Bias-Only Adaptation” Sinii et al (2025) (https://aclanthology.org/2025.emnlp-main.467.pdf) in terms of performance and cost? Rl involves backpropagation and the current method is also not computationally cheap although it avoids backpropagation.

  3. Can the authors provide the reasons for picking the middle layer for steering and whether an alternative layer would provide more benefit?

  4. Can the authors provide some cost analysis in terms of FLOPs or wall clock training times for the proposed method, baselines, and also the cost of full RL tuning for reference? It would be helpful to know what the cost savings are of the proposed methods compared to even full RL.

  5. Can the authors include standard errors by varying the training set for example to include different subsets? For example, the training set for GSM8K used in the paper is only a subset of the official training data split for this data.

  6. How do results vary with the training data sample size?

  7. Have the authors considered doing reweighting on the subgroups whose gradients differ substantially although they might be positively aligned in terms of cosine similarity?

  8. Can the authors include some ablations for the reweighted procedure such as the the performance dependence on the subgroup weighting gamma parameter as well the number of subgroups?

  9. Important. Have the authors explored some automated approach to identify the subgroups leaving the human out of the loop? For example, could clustering based on the loadings be used to decide on the subgroups and automate the procedure?

Reviewer_GtjyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ZOSteer learns activation steering vectors for frozen language models using scalar rewards, without backpropagation or weight updates. It samples Gaussian activation perturbations and correlates them with rewards to estimate the gradient of a smoothed reward objective. The same evaluations yield per-example gradient estimates, which the authors use to identify conflicting subgroups and construct reweighted steering directions. Experiments across three models and four tasks demonstrate improvements on several benchmarks, though gains vary across tasks and the method requires substantial reward evaluation.

Critical strengths

The core method is mathematically coherent. ZOSteer perturbs activations, scores the resulting outputs, and uses the correlation between perturbations and rewards to estimate the gradient of a Gaussian-smoothed reward. Equation (3)’s identity is valid even for discontinuous correctness rewards.

Critical weaknesses
  1. The full runs use 4,096-16,384 perturbations, each evaluated on 128 items: approximately 0.52-2.1 million generations per model–task pair. The baselines collect 2,048 responses, although some additionally require backpropagation. The claim that fewer perturbations than the hidden dimension often suffice comes from Figure 2's analysis using test-selected steering strength. The authors disclose this, but it does not establish the same savings under deployable validation-based selection.

  2. The explanation of why reweighting helps needs stronger evidence. Subgroups are identified post hoc, sometimes from only four training examples. Reweighting improves four of six conflict cases, but no held-out subgroup-specific results show that it actually repairs the identified minority. Moreover, the exact pooled gradient already maximizes local improvement in the average smoothed reward. Reweighting’s benefit therefore needs an explanation involving estimation error, generalization, finite steps, or the difference between smoothed and deployed rewards, not simply “the minority was underweighted.”

  3. The strongest performance rankings lack uncertainty estimates. On GSM8K, ZOSteer improves over the unsteered models by 3.3- 6.44 percentage points, but its margins over the strongest baseline are only 0.38, 0.60, and 0.99 points. The paper provides no confidence intervals or repeated-run evidence establishing stable superiority.

I also found two concrete manuscript issues: Table 2's body is missing from the supplied PDF, and Appendix F says that increasing γ\gamma to 0.9 rotates in the opposite direction, although Equation (10)’s definition implies the same direction. These need clarification, but neither invalidates the basic estimator.

Questions for authors
  1. Figure 2 selects steering strength using test performance. Does the claim that fewer perturbations than the hidden dimension suffice also hold with validation-selected strength? What is the total generation budget, including pilot and validation evaluations?

  2. What evidence shows that reweighting improves the identified minority subgroup, beyond improving aggregate accuracy? Please provide the missing Table 2 values and clarify whether the detected conflicts are reproducible across independent perturbation samples.

  3. Are the reported improvements over the strongest baselines consistent across perturbation seeds and training-item samples? Given the small GSM8K margins, what uncertainty estimates support the reported ranking?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.