acceptodds
Withdrawn from ICLR 2027

Rotary Healing Space Topology: Hardware-Fused Coordinate Re-mapping for Extreme KV Cache Eviction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Memory constraints in autoregressive language models force the adoption of Key-Value (KV) cache physical truncation. While heuristic eviction methods successfully bound VRAM allocation, they introduce a deterministic generation collapse (e.g., infinite repetitive token loops or absolute syntactic disintegration) at extreme compression boundaries. We trace the root cause of this phase transition cliff to a topological fracture within Rotary Position Embeddings (RoPE). The mathematical failure does not originate from the physical memory voids. Instead, it arises because the underlying tensor framework forcefully re-indexes surviving tokens to contiguous physical positions to maintain memory continuity, thereby destroying the exact relative shift-invariance within the rotary subspace. To resolve this coordinate misalignment without incurring the latency penalty of physical tensor modifications, we propose Rotary Healing Space Topology (RHST). RHST operates as a hardware-level coordinate re-mapping operator with an macroscopic memory I/O footprint. Implemented as an extension to fused attention kernels, RHST bypasses memory bandwidth congestion by mapping physical indices back to their absolute causal coordinates directly at the hardware register level. Empirical evaluation across multi-hop retrieval and mathematical reasoning validates this algebraic intervention. Under extreme physical truncation, application-layer baselines suffer from absolute reasoning failure (0.00% accuracy on complex multi-step reasoning in MATH-500). RHST suppresses generation collapse and maintains deterministic reasoning integrity, effectively acting as a topological safety net that unlocks extreme heuristic compression safely.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Models, KV Cache Compression, Efficient Inference, Rotary Position Embedding, Hardware-Aware Algorithms

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RnADOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates generation failures under KV-cache eviction, and attribute the reason of failure to the misalignment between physical cache indices and original RoPE positions. The authors then propose Rotary Healing Space Topology (RHST), which restores the original causal coordinates using a mapping Ioffset(k)=nI_{\mathrm{offset}}(k)=n. The intended relative phase is

=(m-I_{\mathrm{offset}}(k))\theta_j =(m-n)\theta_j.$$ The correction is integrated into a Triton fused-attention kernel to avoid expensive tensor re-rotation. Experiments on Llama-3-8B-Instruct report substantial improvements under 90\% KV eviction, including 47.40\% accuracy on 2WikiMultihopQA and 21.50\% on MATH-500-Concat, compared with 6.47\% and 0\% for SnapKV.
Critical strengths

The paper addresses an important issue in memory-constrained LLM inference: integrating coordinate correction into fused attention is a potentially useful systems optimization. Also, empirically, RHST reports substantial accuracy and latency improvements under extreme KV eviction. The matched-policy SnapKV ablation provides useful evidence supporting the proposed mechanism.

Critical weaknesses

However, from my perspective, physical KV eviction does not necessarily destroy RoPE positions. Correct implementations can preserve already-rotated keys and absolute query positions. Thus, RHST may primarily correct an avoidable implementation error rather than an inherent limitation of existing eviction methods.

Apart from that, there seems to be mathematical inconsistency. Appendix A uses an unrotated query and rotates keys by (m−n)θj(m-n)\theta_j. However, under the standard RoPE convention,

=q^\top R((n-m)\theta)k.$$ The displayed implementation therefore appears to use the opposite rotation direction. This raises concerns about the correctness of the proposed kernel and its claimed exact equivalence.
Questions for authors

Can the authors compare RHST against a correctly implemented KV-eviction baseline that preserves precomputed RoPE keys and absolute positional indices? Would RHST still provide substantial accuracy or latency improvements?

Reviewer_mTTTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies generation failures under aggressive KV-cache eviction and attributes them to a mismatch between compacted physical indices and original token positions. RHST stores pre-RoPE keys together with an absolute-position mapping and applies the corresponding rotations inside a fused decoding-attention kernel. The intended benefit is to preserve the positions of retained tokens while avoiding a separate cache-wide re-rotation pass. Experiments report accuracy and generation-collapse results on 2WikiMultihopQA and a constructed MATH-500-Concat task, primarily using Llama-3-8B under a 16 GB memory constraint. A SnapKV-policy experiment and a Qwen experiment examine policy independence and portability.

Critical strengths
  • A relevant implementation concern. Aggressive KV eviction requires consistent position handling, and avoiding a separate cache-wide transformation could be useful if such a transformation is otherwise necessary.

  • An attempt to control the retention policy. Section 4.2 applies RHST to the SnapKV retention policy and reports 45.80% accuracy on 2Wiki and 21.50% on MATH-500-Concat. Holding token selection fixed is a useful step toward isolating position handling.

Critical weaknesses
  • The central failure mechanism is not established, and the comparison does not isolate it. Section 3.2 attributes the failure to physical compaction replacing a token’s original RoPE coordinate with its new storage index. However, Section 3.3 acknowledges that standard frameworks store post-RoPE keys: moving an already-rotated key does not by itself apply a new rotation. The paper needs to identify the actual operation that changes the encoded position. Proposition 1 assumes a correct index mapping and substitutes it into the phase expression; this does not establish that compaction causes the asserted failure in the evaluated baselines. Moreover, Section 4.2 enforces contiguous compaction on all baselines, while RHST changes the key representation and Appendix B modifies input slicing, position tracking, and masking. The SnapKV-policy experiment holds token selection fixed, but does not show a position-correct baseline with equivalent host handling. The reported recovery therefore does not establish the claimed general failure of eviction methods or the necessity of the proposed fused repair. This is the most significant weakness.

  • The CHR definition and reported values are inconsistent or insufficiently specified. Section 4.1 defines CHR as factual divergence conditioned on valid text output. Yet Table 1 reports 0% MATH accuracy, 16.50% GCR, and 83.50% CHR for SnapKV. H2O and OjaKV likewise have 0% accuracy and CHR equal to 100% minus GCR. Under the stated distinction between collapsed and valid-but-incorrect outputs, the conditional error rate would be 100% in these cases. The entries instead appear to use an unconditional denominator. If CHR measures a different category of factual error, that category and its scoring rule need to be defined. This ambiguity directly affects the paper’s interpretation of collapse versus hallucination; raw counts and exact denominators are needed.

  • The zero-I/O and negligible decode-overhead claims are not supported by the measurements. Section 3.3 limits O(1) to each metadata lookup and explicitly acknowledges O(L_retained × d) rotation arithmetic. Section 3.4 nevertheless asserts zero macroscopic HBM bytes on the assumption that the mapping resides on chip. Listing 1 does not establish how this residency is maintained, and no measured memory-traffic breakdown is provided. Sections 4.4–4.5 primarily report TTFT, including 1,374.32 ms for RHST in Table 1. This does not quantify the recurring cost of rotating retained keys during decoding. Per-token decode latency, rotation overhead, and memory traffic against a position-correct baseline are necessary to support the central hardware-efficiency claim.

Questions for authors
  • Which exact baseline operation changes the original positional information? Please compare preserved post-RoPE keys, an unfused pre-RoPE implementation, and fused RHST using the same retention mask, absolute query positions, and host-side corrections. Are the Appendix B changes applied equally to all methods?

  • Could you clarify the rotation convention in Algorithm 1 and Listing 1? With the printed rotation matrix, separately rotating query and key gives qᵀR((n−m)θ)k, whereas the listing rotates the raw key by +(m−n)θ. Is another transformation omitted? Please provide a small numerical equivalence check against the reference attention and clarify whether the listing matches the executed implementation.

  • Please provide the raw counts and exact scoring rules for accuracy, GCR, and CHR. What is the CHR denominator, and how are valid-but-incorrect outputs distinguished from collapsed outputs?

  • How is the mapping allocated and kept on chip? Please report per-token decode latency, metadata and rotation costs, and measured memory traffic over several retained lengths, including the same position-correct baseline.

Reviewer_JgpPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work presents Rotary Healing Space Topology, a coordinate-remapping mechanism for KV cache eviction. The method stores pre-RoPE keys and maps retained cache slots to their original causal positions inside a Triton attention kernel, aiming to preserve positional consistency without application-layer tensor re-rotation. Experiments on multi-hop QA and concatenated mathematical reasoning tasks report reduced generation collapse under aggressive eviction in a 16 GB memory setting.

Critical strengths

This work provides kernel and host-integration code in Appendix A and B, making the proposed implementation more concrete.

Critical weaknesses
  1. My main concern is whether the motivating failure occurs in representative inference implementations. Figure 1(a) and Section 3.2 assume that compacting retained KV entries replaces their original RoPE positions with contiguous physical indices. However, the official SnapKV Llama implementation applies apply_rotary_pos_emb before cache compression and updating, and the vLLM Llama implementation similarly applies RoPE before handle KV cache. For already-rotated keys, moving a vector to another cache slot does not itself change its encoded phase, provided subsequent queries retain the correct absolute positions.
  2. The treatment of PagedAttention and related cache-management systems is insufficiently supported. Section 4.2 dismisses paging as structurally unviable under the 16 GB budget because of metadata overhead and poor locality, but provides no direct measurements establishing either as the bottleneck. Block metadata is typically small relative to the KV tensors. Moreover, Section 3.3 describes RHST’s own position metadata as negligible for a similar reason. The paper should quantify paged implementation under matched workloads.
  3. The deployment evaluation is narrow. The main results use INT8 8B model with a strict 16 GB ceiling, while the Qwen-2.5-7B extension is limited to one QA setting. This is a valid scenario, but does not establish that the reported failure or benefit is representative across inference workloads.
  4. The paper reports TTFT, but omits TPOT/inter-token latency, output-token throughput, and end-to-end generation latency. These measurements matter because RHST performs additional rotations over retained keys during decoding, with an arithmetic cost acknowledged in Section 3.3.
Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.