Rotary Healing Space Topology: Hardware-Fused Coordinate Re-mapping for Extreme KV Cache Eviction
Abstract
Memory constraints in autoregressive language models force the adoption of Key-Value (KV) cache physical truncation. While heuristic eviction methods successfully bound VRAM allocation, they introduce a deterministic generation collapse (e.g., infinite repetitive token loops or absolute syntactic disintegration) at extreme compression boundaries. We trace the root cause of this phase transition cliff to a topological fracture within Rotary Position Embeddings (RoPE). The mathematical failure does not originate from the physical memory voids. Instead, it arises because the underlying tensor framework forcefully re-indexes surviving tokens to contiguous physical positions to maintain memory continuity, thereby destroying the exact relative shift-invariance within the rotary subspace. To resolve this coordinate misalignment without incurring the latency penalty of physical tensor modifications, we propose Rotary Healing Space Topology (RHST). RHST operates as a hardware-level coordinate re-mapping operator with an macroscopic memory I/O footprint. Implemented as an extension to fused attention kernels, RHST bypasses memory bandwidth congestion by mapping physical indices back to their absolute causal coordinates directly at the hardware register level. Empirical evaluation across multi-hop retrieval and mathematical reasoning validates this algebraic intervention. Under extreme physical truncation, application-layer baselines suffer from absolute reasoning failure (0.00% accuracy on complex multi-step reasoning in MATH-500). RHST suppresses generation collapse and maintains deterministic reasoning integrity, effectively acting as a topological safety net that unlocks extreme heuristic compression safely.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.