- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work presents a zeroth order steering approach that estimates the behavioral steering vector as the one that maximizes the expected reward over the local Gaussian perturbations in the residual stream at a certain layer. Using Stein's identity, they estimate the optimal gradient vector for this smoothed reward function by perturbing a set of training samples. Results on verifiable datasets (including an interesting coding dataset) and two small (1B) language models show that steering the middle layer with the proposed approach is more effective than the baselines.
Critical strengths
- The proposed method is sound and is motivated by some theory.
- The steering computation protocol with local perturbations in the residual stream is interesting.
- They also analyze the residual activations from the items when the mean gradient vector is removed.
- Considered datasets are probably interesting as they are not very common in the literature.
Critical weaknesses
- Table 2 is missing.
- Number of model forward passes is prohibitively large during steering vector derivation. While they mention it as a limitation, it is at the scale of the model dimension (millions), which is not justifiable and is no longer more efficient than backward passes (memory permitting). While they also mention batching as a solution, it does not scale as the model size increases. Also, perhaps a fair comparison of the baselines would be to consider the Gaussian-perturbed samples as additional samples for steering vector computation since their reward can still be calculated. For example, instead of just using 16 for DiffMean, it should be 16 * N to calculate the DiffMean vector (to also smooth the DiffMean vector in a way).
- Evaluation is only conducted on small 1B models and generalizability/scalability to larger models remains open.
- Benchmark selection is not clearly standard as other reward-based behavioral datasets such as refusal and toxicity were not considered, which are more relevant in the steering literatures [1].
- It is not clear why smoothed reward is used. The current motivation is not really correct since the output of should change everytime changes and why is its gradient zero everywhere. The use of Gaussian-smoothed reward however instead of the actual reward however is interesting for generalizability of the average gradient direction without requiring more samples and just using the perturbations in the input.
- Sample cost arguments in lines 172-181 are not completely clear and must be made more accessible.
- Generation examples are not provided to substantiate the qualitative effectiveness for the proposed steering.
- Stronger baselines can probably be added as well like angular steering, OdeSteer, ReFT.
- Minor:
- Eq.equation is used. Fix it to use either of them.
[1] Nguyen, Dung Viet, et al. "Activation steering with a feedback controller." International Conference on Learning Representations. Vol. 2026. 2026.
Questions for authors
- How is the extremely large number of forward passes during steering vector computation feasible or justifiable in real settings? Is there any way to circumvent it in budgeted settings, probably inspired by the zero-order black-box adversarial settings.
- What if we consider other local perturbation mechanisms than Gaussian noise?
- What happens if we use the perturbed vectors as additional samples for the baselines?
- How do the methods perform on behavioral reward datasets and bigger llms?
- How sensitive is the approach to ? Shouldn't baselines also tuned for this alpha parameter which is just a steering multiplier?
- Why was a single middle layer used to steer instead of a layer sweep?
- How sensitive is the vector with the number of training samples? I believe this could be effective when we have fairly few training samples but can still enummerate through local perturabtions.