- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets repeated candidate prefill in all-pairs LLM-judge tournaments. Under standard RoPE and causal attention, a candidate's KV states depend on its placement and on the preceding competitor, so prefix caching reuses it only when it directly follows the shared prefix. SWAP-RoPE gives all candidate blocks the same positional origin and masks attention so that each block sees only the prefix and its own history, while the trailing readout sees all blocks. Proposition 1 shows that, in exact arithmetic, block hidden states, keys, and values depend only on the prefix and the block, so they survive swaps and competitor substitution; fresh candidate encodings in a tournament drop from to
Qwen3-1.7B/4B/8B are fine-tuned on the same self-generated judging traces with standard RoPE or SWAP-RoPE and evaluated on AIME 2024/2025 and HMMT 2025. The reported KV reuse ratio after a swap rises from 38.5–40.9% to 73.8–79.2%, and SWAP-RoPE beats vanilla RoPE in 6/9 settings at and 3/9 at .
Critical strengths
[S1] The formalization is clean, and the reuse accounting is correct and honestly scoped. Proposition 1 holds under the stated assumptions, and the Table 2 counts are correct; reducing fresh candidate prefill from to is practically relevant for best-of- selection over long reasoning traces. The paper consistently separates representation invariance from verdict consistency and encoding counts from serving speedup, and reports the regressions openly.
Critical weaknesses
[W1] The construction and its invariance argument are closely anticipated by prior work that the paper neither cites nor compares against. Parallel Context Windows [1] reuses positional embeddings across windows, restricts attention within each window, and lets task tokens attend to all windows. Set-Based Prompting [2] gives parallel sub-sequences shared position ids, masks attention between them, and proves that outputs are order-independent; Proposition 1's per-layer locality is the natural intermediate step of that argument. Block-Attention [3] and APE [4] encode blocks independently for KV reuse, adapt the model through fine-tuning or calibration, and report measured prefill speedups, and PINE [5] targets position bias in LLM judges directly. What remains new is reuse framed around competitor substitution in all-pairs judging with a shared prefix, the Table 2 accounting, and SFT for judging; relative to [3], this amounts to the shared origin and the tournament setting. This is a real but incremental contribution, and it needs to be positioned against [1–5], ideally with an empirical comparison to at least one of them.
[W2] The empirical evidence does not yet support the claims.
- [W2.1] By the paper's own analysis, a swap preserves the prefix, both blocks, and the readout, so the ideal ratio in Eq. 7 should be 100%, yet Table 3 reports 73.8–79.2%. The authors acknowledge that the shortfall cannot be attributed from the table (lines 386–391), and the paper does not say how was obtained. Only a single swap is measured, not competitor substitution across a tournament, and there are no wall-clock or memory figures.
- [W2.2] Tables 4–5 report single numbers without variance or instance counts on 30-problem benchmarks. The mean New−Van. difference is +3.3 points at and −1.5 points at . Neither the selection protocol nor "Orig." is defined, and majority voting, the natural baseline for math answer selection, is missing.
- [W2.3] No code or implementation is linked or supplied. Proposition 1 and Table 2 can be checked on paper, but the Table 3 reuse ratios and the Tables 4–5 accuracies depend on the custom mask, position ids, and cache assembly, so they cannot be verified and the gap to the theory cannot be diagnosed. This does not imply the results are wrong, but it contributes to my score.
References
[1] Ratner et al. Parallel Context Windows for Large Language Models. ACL 2023. https://arxiv.org/abs/2212.10947
[2] McIlroy-Young et al. Order-Independence Without Fine Tuning (Set-Based Prompting). NeurIPS 2024. https://arxiv.org/abs/2406.06581
[3] Ma, Wang, Tian. Block-Attention for Efficient Prefilling. ICLR 2025. https://arxiv.org/abs/2409.15355
[4] Yang, Chen, Chen. APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding. ICLR 2025. https://arxiv.org/abs/2502.05431
[5] Wang et al. Eliminating Position Bias of Language Models: A Mechanistic Approach. ICLR 2025. https://arxiv.org/abs/2407.01100
Questions for authors
Clarifications:
- How is computed: by token counting, or by comparing cached tensors (with what tolerance, over how many prompts)? Which tokens fail the reuse test under SWAP-RoPE? A per-region breakdown with average lengths would help, including whether the ~40% RoPE ratio equals .
- How is a candidate selected at and : all-pairs judging with an aggregation rule (which rule?), or one prompt with blocks? How many candidate sets per problem are evaluated, and what are the confidence intervals? Since SFT used four solutions per problem, is outside the training distribution?
- Is "Orig." the base model's single-sample accuracy? Its values are identical across Tables 4–5 and match the one-sample point in Figure 4.
- Were AIME/HMMT 2024–2025 problems excluded from the OpenMathReasoning subset, and were the self-generated judging traces filtered by correctness of the selected answer?
Suggestions that would change my assessment: 5. Comparisons with majority voting and with at least one of [1–3], plus a SWAP-RoPE result without SFT. 6. Measured prefill time and memory for a full tournament against prefix caching, and (Eq. 6) under the actual bf16 implementation.
Format:
- Seems like you used
\namecode in the abstract on this page and it didn't properly render. Perhaps you defined this macros in your main LaTex file and copied the abstract here.