- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The work explores the idea that RoPE context length extension via position interpolation (PI) or YaRN fine-tuning should be done in two phases: first, train on longer context without change, then change the RoPE frequencies and temperature for the remainder of the extension training, typically 10%. This appears to be a novel idea with the potential to improve outcomes at no additional cost. The experiments provided show significantly improved validation-set perplexity within the region 2-4x beyond a 4x finetuned length, but mildly worse perplexity at 1x the finetuned length.
Critical strengths
The strongest part of the paper is the experimental evidence of reduced perplexity at 2-4x beyond a 4x finetuned length. This is the fundamental result, and supports the idea that a period of training at longer context with the model held unchanged is important prior to applying other extension methods such as PI or YaRN. The multiple seeds and ranges shown are helpful demonstrating confidence within the limitations of the experimental regime.
Critical weaknesses
The weakest parts of the paper are the quality of the writing, lack of comparative evidence about what happens within the original model context length through the finetuned length, the small scale and training dataset of the experiments, and the lack of retrieval and context specific benchmarks.
The paper is dense with numerical information, and is exceptionally difficult to read, idiomatic of common LLM output. The abstract uses terminology (e.g. "release") only somewhat defined or implicitly stated later in the paper itself; one should not be required to read the paper to understand the abstract. The 53 pages of appendices are quite challenging to approach. Such a large volume of information needs exceptional description, organization and summarization, which are not present.
While the evidence of ppl improvement at 8x and 16x context is intriguing, the ppl is mildly worse at the end of the 4k finetuned context length. This only emphasizes the need for evaluation at shorter lengths to see whether or not the method damages the model more at those lengths than YaRN or PI would.
As admitted in the limitations section and Appendix S, the scale of the experiments are extremely small, and are trained on very little data. Much smaller, indeed, than the experiments shown in YaRN paper upon which the method is based. Furthermore, the 137M model is pretrained at 1k context on PG19, which I view as an inappropriate corpus for this context length. If you examine this corpus, the beginning of many of the documents are wildly different than the expected contents later on, and we have found it is an inconsistent testing/training ground at best for important retrieval and context tests. I recommend, at a minimum, testing the method on the same models as used in the YaRN paper with similar context lengths. If you are looking for a different long context dataset, ProLong may be of use. There may also be some smaller modern well trained models available you could use for more extensive evaluations if compute is a major hurdle.
Finally, as admitted in Appendix S, the current lack of downstream benchmarks unfortunately dramatically limits the scope of utility for this research, making it indicative that there may be something of interest related to the method, but stopping short of indicating that it could be used effectively in practice for real world model inference.
Questions for authors
Further explanation of the described geometric motivation in the introduction and section 3 would be useful. I am unclear on exactly what is meant about the hold training "on the angle range that 16K inference will use, whereas the static arm must extrapolate fourfold in angle". There seems to be an implicit conflation of the 4x finetuned context extension to 4k, and the further 4x extrapolation to 16k. If these two expansions need to be done in equal ratio, I do not see that requirement listed in the paper or ablated. Is the goal to always have the finetuned context expansion ratio match the eventual ratio further extrapolated beyond the finetuned context? If so, this is an interesting point that should be made more clearly with ablations demonstrating its importance.
It would be helpful if you could explain the rationale and evidence for the custom YaRN-family target. Although the paper provides results for both classic YaRN and the custom YaRN target, the focus on the custom target muddies the conclusion significantly. Further, it is not clear if the custom target is specifically useful due to some special properties of the paper's pretraining regime, or is generally applicable to other models.
I would be open to increasing my score if the experimental evidence was dramatically enhanced in all three ways outlined in the Critical Weaknesses above. I strongly recommend significant writing improvements, but strong enough experiments would be a very dramatic improvement even in the face of only moderately revised writing.