acceptodds
Under review as a conference paper at ICLR 2027

Swap-RoPE: Permutation-Invariant Block Encoding for LLM Judging

Abstract

Pairwise LLM judging repeatedly encodes the same candidates in different positions and contexts. We introduce \name, which combines shared block coordinates with causal attention restricted to a fixed prefix and each block's own history. We establish that a block's hidden states, keys, and values are independent of its competitors and preserved under token-aligned permutations. These properties permit cache reuse across different comparisons. In an all-pairs tournament over candidates, fresh candidate encodings decrease from to under fixed-content and cache-retention assumptions; comparison calls remain quadratic. We fine-tune Qwen3 models at 1.7B, 4B, and 8B and evaluate three mathematical benchmarks. Reported reusable key–value (KV) cache ratios increase from – under standard RoPE to –. Selection accuracy improves over vanilla-position fine-tuning in six of nine settings with four candidates and three of nine with eight candidates. The construction excludes cross-block attention during candidate encoding; verdict consistency and serving speedups require further evaluation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, Positional encoding

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_prLROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets repeated candidate prefill in all-pairs LLM-judge tournaments. Under standard RoPE and causal attention, a candidate's KV states depend on its placement and on the preceding competitor, so prefix caching reuses it only when it directly follows the shared prefix. SWAP-RoPE gives all candidate blocks the same positional origin and masks attention so that each block sees only the prefix and its own history, while the trailing readout sees all blocks. Proposition 1 shows that, in exact arithmetic, block hidden states, keys, and values depend only on the prefix and the block, so they survive swaps and competitor substitution; fresh candidate encodings in a tournament drop from (n2)+n−1\binom{n}{2}+n-1 to nn

Qwen3-1.7B/4B/8B are fine-tuned on the same self-generated judging traces with standard RoPE or SWAP-RoPE and evaluated on AIME 2024/2025 and HMMT 2025. The reported KV reuse ratio after a swap rises from 38.5–40.9% to 73.8–79.2%, and SWAP-RoPE beats vanilla RoPE in 6/9 settings at n=4n=4 and 3/9 at n=8n=8.

Critical strengths

[S1] The formalization is clean, and the reuse accounting is correct and honestly scoped. Proposition 1 holds under the stated assumptions, and the Table 2 counts are correct; reducing fresh candidate prefill from Θ(n2)\Theta(n^2) to Θ(n)\Theta(n) is practically relevant for best-of-nn selection over long reasoning traces. The paper consistently separates representation invariance from verdict consistency and encoding counts from serving speedup, and reports the n=8n=8 regressions openly.

Critical weaknesses

[W1] The construction and its invariance argument are closely anticipated by prior work that the paper neither cites nor compares against. Parallel Context Windows [1] reuses positional embeddings across windows, restricts attention within each window, and lets task tokens attend to all windows. Set-Based Prompting [2] gives parallel sub-sequences shared position ids, masks attention between them, and proves that outputs are order-independent; Proposition 1's per-layer locality is the natural intermediate step of that argument. Block-Attention [3] and APE [4] encode blocks independently for KV reuse, adapt the model through fine-tuning or calibration, and report measured prefill speedups, and PINE [5] targets position bias in LLM judges directly. What remains new is reuse framed around competitor substitution in all-pairs judging with a shared prefix, the Table 2 accounting, and SFT for judging; relative to [3], this amounts to the shared origin and the tournament setting. This is a real but incremental contribution, and it needs to be positioned against [1–5], ideally with an empirical comparison to at least one of them.

[W2] The empirical evidence does not yet support the claims.

  • [W2.1] By the paper's own analysis, a swap preserves the prefix, both blocks, and the readout, so the ideal ratio in Eq. 7 should be 100%, yet Table 3 reports 73.8–79.2%. The authors acknowledge that the shortfall cannot be attributed from the table (lines 386–391), and the paper does not say how rKVr_{\mathrm{KV}} was obtained. Only a single swap is measured, not competitor substitution across a tournament, and there are no wall-clock or memory figures.
  • [W2.2] Tables 4–5 report single numbers without variance or instance counts on 30-problem benchmarks. The mean New−Van. difference is +3.3 points at n=4n=4 and −1.5 points at n=8n=8. Neither the selection protocol nor "Orig." is defined, and majority voting, the natural baseline for math answer selection, is missing.
  • [W2.3] No code or implementation is linked or supplied. Proposition 1 and Table 2 can be checked on paper, but the Table 3 reuse ratios and the Tables 4–5 accuracies depend on the custom mask, position ids, and cache assembly, so they cannot be verified and the gap to the theory cannot be diagnosed. This does not imply the results are wrong, but it contributes to my score.

References

[1] Ratner et al. Parallel Context Windows for Large Language Models. ACL 2023. https://arxiv.org/abs/2212.10947

[2] McIlroy-Young et al. Order-Independence Without Fine Tuning (Set-Based Prompting). NeurIPS 2024. https://arxiv.org/abs/2406.06581

[3] Ma, Wang, Tian. Block-Attention for Efficient Prefilling. ICLR 2025. https://arxiv.org/abs/2409.15355

[4] Yang, Chen, Chen. APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding. ICLR 2025. https://arxiv.org/abs/2502.05431

[5] Wang et al. Eliminating Position Bias of Language Models: A Mechanistic Approach. ICLR 2025. https://arxiv.org/abs/2407.01100

Questions for authors

Clarifications:

  1. How is rKVr_{\mathrm{KV}} computed: by token counting, or by comparing cached tensors (with what tolerance, over how many prompts)? Which tokens fail the reuse test under SWAP-RoPE? A per-region breakdown with average lengths would help, including whether the ~40% RoPE ratio equals ∣P∣/N|P|/N.
  2. How is a candidate selected at n=4n=4 and n=8n=8: all-pairs judging with an aggregation rule (which rule?), or one prompt with K=nK=n blocks? How many candidate sets per problem are evaluated, and what are the confidence intervals? Since SFT used four solutions per problem, is n=8n=8 outside the training distribution?
  3. Is "Orig." the base model's single-sample accuracy? Its values are identical across Tables 4–5 and match the one-sample point in Figure 4.
  4. Were AIME/HMMT 2024–2025 problems excluded from the OpenMathReasoning subset, and were the self-generated judging traces filtered by correctness of the selected answer?

Suggestions that would change my assessment: 5. Comparisons with majority voting and with at least one of [1–3], plus a SWAP-RoPE result without SFT. 6. Measured prefill time and memory for a full n=8n=8 tournament against prefix caching, and DswapD_{\mathrm{swap}} (Eq. 6) under the actual bf16 implementation.

Format:

  • Seems like you used \name code in the abstract on this page and it didn't properly render. Perhaps you defined this macros in your main LaTex file and copied the abstract here.
Reviewer_ZiNpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Swap-RoPE to improve KV cache reuse in repeated pairwise LLM judging. It encodes each candidate independently, allowing its cache to be reused across comparisons and candidate permutations. The authors assign the same starting RoPE position to candidate blocks and use a block-wise attention mask to prevent candidates from attending to each other. The readout can then attend to all candidates. The paper analyzes permutation invariance and cache reusability, and evaluates the method on Qwen3 models of different sizes using mathematical reasoning benchmarks.

Critical strengths

The paper addresses a practical problem. Repeatedly comparing long reasoning trajectories involves redundant computation, so reusing candidate-level KV caches is relevant to test-time scaling and LLM-based evaluation.

The design is simple and easy to follow. Shared positional assignments and block-wise attention masking make candidate representations independent of other candidates. The theoretical analysis clarifies the conditions under which their caches can be reused.

Critical weaknesses

My main concern is the novelty of the method. As I understand it, one could independently prefill Prefix + Candidate A and Prefix + Candidate B, retain their local positions, and combine the KV caches for a shared readout. Under this construction, the shared positions and attention mask appear to formalize independent encoding. I would like to understand what additional algorithmic contribution Swap-RoPE provides. Some works also explore shared positional encodings and attention masking for permutation-invariant processing of candidate sets.

I am not convinced that the reported cache reuse translates into sufficient practical savings. The paper reports an estimated reuse ratio, but does not provide enough e2e serving measurements. Cache storage and memory transfer, together with readout computation and decoding, may reduce the actual speedup.

I also have concerns about judging quality. Preventing cross-candidate attention during encoding postpones all interactions to the readout stage. This may make fine-grained comparisons between long reasoning trajectories harder. The results are mixed, especially with eight candidates, where Swap-RoPE performs worse than the standard approach in several configurations. Without measured efficiency gains and a clearer explanation of these losses, it is difficult to judge whether the trade-off is worthwhile.

Questions for authors

What's your opinion of the position bias in LLM pairwise judging and how it relates to and influences your work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.