Rotary Orientation Contraction in RoPE

Rotary Position Embeddings (RoPE) encode relative position by rotating pairs of query and key coordinates according to token position. However, these pairs already have content-dependent orientations before the positional rotation is applied, so content-dependent orientation and positional phase interact within the same angular term of the attention score. We introduce Rotary Orientation Contraction (ROC), a simple modification to RoPE that learns how strongly to contract these pre-existing orientations while preserving the standard RoPE frequencies and positional rotations. Across language-modeling experiments, ROC improves perplexity over matched RoPE baselines. These results identify pre-RoPE content geometry as a useful design dimension for rotary attention and show that it can be modified efficiently, with essentially no measured forward-pass overhead, while preserving the standard paired RoPE structure.