- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Skip-RoPE supplies genomic source coordinates to RoPE when packing noncontiguous fragments sampled from the human reference genome. This separates the number of input tokens from the genomic sampling span. The paper studies sampling configurations, trains a 100M-parameter genomic language model on 30B tokens, and evaluates long-input language modeling, genomic prediction tasks, and enhancer–gene prediction with frozen representations. Its main reported outcomes are a favorable fixed-suffix perplexity–training-compute trade-off and useful predictive performance when parts of the intervening DNA are omitted at inference.
Critical strengths
-
The authors show that a simple training scheme can provide a favorable trade-off between prediction quality and training cost in the studied dense-attention setting. Figure 1b compares the schemes in terms of training FLOPs and fixed-suffix perplexity with 1 Mb of preceding context. The sampling experiments also show which configurations work well and when excessive sparsity or fragmentation degrades performance. These results provide useful guidance for choosing genomic pretraining configurations.
-
The paper's downstream results support the practical value of the method and show the performance trade-off when fewer input tokens are used. On rE2G, linear probes on frozen embeddings achieve 0.536 AUPRC versus 0.412 for the reported distance baseline, with an advantage in each reported distance group. Sparse inputs achieve 0.517 at the stated 2.1× token reduction. Table 3's DNALongBench results are also encouraging relative to the listed sequence models. Figure 5 also shows that adding infill improves prediction relative to the anchor-only extreme in the evaluated pipeline.
Critical weaknesses
The paper leaves open how much Skip-RoPE improves E2G prediction over closely related position-skip training schemes. PoSE [1], which the paper cites, already tested aligning position indices with the source text. Table 2 provides relevant internal evidence—Skip-RoPE improves over Sequential-RoPE on seven of eight tasks, but the gains over Random-RoPE are mixed. The strongest E2G evaluations do not include these internal controls. The work therefore establishes a viable genomic training scheme, while leaving its additional benefit for E2G prediction over a close training alternative uncertain. This limits the strength of the incremental contribution and is my main reservation about acceptance. The language-modeling training savings and downstream predictive results also do not establish reduced training compute at matched E2G quality.
[1] Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training. In The Twelfth International Conference on Learning Representations (ICLR), 2024. arXiv:2309.10400.
Questions for authors
-
If a comparable Random-RoPE checkpoint is available among the existing models, how does it perform under the same rE2G protocol, and what was its training compute? This would most directly clarify the value of the proposed training choice for downstream prediction.
-
Please confirm whether Figure 1b uses the same 16 validation loci as the breakpoint analysis, and specify the final sampling/RoPE configuration and FLOP-counting convention for the reported training scheme.
W1. The paper leaves open how much Skip-RoPE improves E2G prediction over closely related position-skip training schemes. PoSE [1], which the paper cites, already tested aligning position indices with the source text. Table 2 provides relevant internal evidence—Skip-RoPE improves over Sequential-RoPE on seven of eight tasks, but the gains over Random-RoPE are mixed. The strongest E2G evaluations do not include these internal controls. The work therefore establishes a viable genomic training scheme, while leaving its additional benefit for E2G prediction over a close training alternative uncertain. This limits the strength of the incremental contribution and is my main reservation about acceptance. The language-modeling training savings and downstream predictive results also do not establish reduced training compute at matched E2G quality.
A1. We agree that the initial manuscript lacked a clear narrative on the incremental nature of our contribution over existing methods and thank the reviewer for raising the point that PoSE tested aligning position indices with the source text. We note that PoSE was applied as a post-training context-extension method and that the authors concluded that aligned positions have “relatively little impact on the outcomes of fine-tuning”.
In our latest revision, we have expanded the manuscript with a needle-in-a-haystack (NIAH) eval showcasing distal-retrieval to measure long context utility. As seen in Section 4.2 and Figure 5, this task shows clear and conclusive benefits in long-range modeling performance that is attributed to the use of faithful coordinates.
As such, we have modified our Contribution 3 to focus on the compute efficiency gain during pretraining for long-context robustness, and added a new Contribution 4 highlighting the importance of using faithful genomic co-ordinates that became directly evident through our new evaluation. The latter especially is a new finding over previous literature that asserts that whether coordinates are matched to their underlying tokens is of little consequence.
In order to improve the general clarity of our paper, we choose to focus on the NIAH eval as the primary illustration of our new Contribution 4 as it directly probes the attention mechanism, and have moved previous benchmark model comparisons into the Appendix.
Q1. If a comparable Random-RoPE checkpoint is available among the existing models, how does it perform under the same rE2G protocol, and what was its training compute? This would most directly clarify the value of the proposed training choice for downstream prediction.
A2. We thank the reviewer for this question, but as discussed above we have reframed our contributions. To summarize, since the original submission, we have expanded our paper with a new needle-in-a-haystack eval which directly clarifies the value of Skip-RoPE over Random-RoPE in leveraging long-context utility under both matched data and compute.
Q2. Please confirm whether Figure 1b uses the same 16 validation loci as the breakpoint analysis, and specify the final sampling/RoPE configuration and FLOP-counting convention for the reported training scheme.
A3. This is correct. Figure 1b uses the same 16 validation loci as the breakpoint analysis, scoring the same 512 bp suffixes with 1 Mb of preceding context.
The selected sampling configuration was T=65,536, W=2,097,152, K=2: two 32,768 bp fragments sampled independently with replacement, retained in draw order, with independent reverse-complement probability 0.5 and faithful genomic coordinates. The original RoPE base was 5e7, but our NIAH eval revealed that this value was undertuned, so we retrained on the selected sampling configuration with theta 2e9.
Training FLOPs count linear-layer and causal-attention matrix multiplications, with two operations per multiply-add and backward estimated as twice forward. They exclude non-matmul operations, optimizer updates, communication, evaluation, and activation recomputation.