- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work proposes StiefelSteer, a rotation-based activation steering approach that learns parameter-efficient rotations through Riemannian optimization. The key idea is to perform rotations within a learned low-dimensional subspace, with both the subspace and rotation operator jointly optimized. Experiments on three LLMs demonstrate improvements over existing steering methods while largely preserving general capabilities.
Critical strengths
- The geometric formulation is principled, and Riemannian optimization is a suitable approach for learning rotation operators under orthogonality constraints.
- The method achieves promising empirical results across different LLMs.
Critical weaknesses
- Limited novelty. There is already a line of research on rotation-based steering. This work mainly extends existing approaches by introducing a learnable low-dimensional subspace optimized through well-established Riemannian optimization techniques.
- Potentially unfair baseline comparisons. The proposed method steers up to 20 layers in some experiments, while certain baselines only steer one layer. Although the authors provide a layer-count ablation for RDO on one model, more comprehensive comparisons under matched intervention budgets would strengthen the results.
- Insufficient ablation studies. The contributions of learning the subspace and enforcing orthogonality are not sufficiently distinguished. Additional ablations are needed to isolate the effectiveness of these components.
- Clarity of presentation. The writing could be improved, particularly in the Introduction, where the motivation and key differences from existing rotation-based methods should be explained more clearly.
- Limited baseline coverage. Although the paper includes an optimization-based additive steering baseline (RDO), comparisons with other additive-based, neural network-based, and multi-step steering approaches are missing.
Questions for authors
See weakness