- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates frequency allocation in positional interpolation methods for context window extension. YaRN, a well-known positional interpolation method, modifies the RoPE frequencies by changing some frequencies while keeping others unchanged during fine-tuning for context extension. The main contribution of this paper is essentially limited to replacing part of YaRN's frequency allocation scheme with a new mathematical formulation. Furthermore, the authors do not perform fine-tuning in their context extension experiments, despite fine-tuning typically being employed for this purpose. In addition, some baseline results are missing. For example, Table 2 does not report the LongBench v2 results for Llama-3-8B with YaRN, even though YaRN is one of the main baselines. Overall, I find the experimental setup questionable, and the omission of certain baseline results further undermines the validity of the evaluation. Therefore, I recommend rejecting this paper.
Critical strengths
The authors are addressing the important problem of frequency allocation for context expansion.
Critical weaknesses
Novelty
The main contribution of the proposed method is essentially limited to replacing part of YaRN's frequency allocation scheme with a new mathematical formulation.
Experimental Setup
Typically, studies on context extension using YaRN report results after fine-tuning. However, the authors do not perform any fine-tuning in their main experiments (Tables 1 and 2). While the performance obtained without fine-tuning, i.e., under length extrapolation, may be interesting, I believe such results should be presented as supplementary evidence rather than as the primary evaluation. For a paper focusing on context extension, I find this experimental setup highly questionable. Furthermore, the reported RULER scores are considerably lower than those reported in some previous studies, making it difficult for me to assess whether the experimental setup and results are reliable. For example, a previous study [1] reported a RULER score of 68.69% for YaRN after fine-tuning Llama-3-8B with a context length of up to 32K. Although the experimental conditions are not identical, this highlights the importance of evaluating the proposed method under standard fine-tuning-based context extension settings. In addition, several baseline results are missing. For example, Table 2 does not report the LongBench v2 results for Llama-3-8B with YaRN, despite YaRN being one of the main baselines. Presentation The authors appear to have conducted a thorough literature review. However, the table captions are unnecessarily lengthy while omitting important information. It is difficult to identify which benchmark is being evaluated at a glance. In particular, the caption of the table reporting RULER results does not clearly identify the benchmark as RULER. Instead, the authors use the term "Full-13," which appears to be their own terminology and is not used in the original RULER paper. This makes the results unnecessarily difficult to interpret.
Overall, I find the experimental setup inappropriate for adequately evaluating the proposed method as a context extension technique. In particular, the lack of fine-tuning-based comparisons and the omission of several important baseline results substantially weaken the experimental evidence. Combined with the limited methodological novelty, these concerns lead me to recommend rejection of this paper.
[1] A Controlled Study on Long Context Extension and Generalization in LLMs. COLM 2025.
Questions for authors
See the Weaknesses.