StiefelSteer: Invertible Refusal Steering via Low-Dimensional Rotations of the Residual Stream

Activation steering offers a cheap way to control refusal in language models at inference time. Recent work trains rotations of activations so that the intervention has a clear geometric meaning. These methods, however, define the rotation through external objects such as a refusal vector. We propose a self-contained procedure that learns parameter-efficient rotations directly using Riemannian optimization. Experiments confirm that the resulting scheme intervenes more efficiently than existing approaches. A broad ablation study shows which design choices matter for the method. These results support rotation-based steering as a reliable mechanism for controlling behavior of LLM.