acceptodds
Under review as a conference paper at ICLR 2027

Skip-RoPE: understanding long-range genomic interactions using skip-wise training

Abstract

Short-read sequencing generates genomic data at scale through parallel processing of millions of DNA fragments, typically spanning hundreds of base pairs, whereas underlying regulatory interactions can span millions. Genomic language models (gLMs) often bridge the gap by training on long contiguous inputs assembled from short reads, but this incurs significant data generation and computational cost. We propose a modification of Rotary Position Embeddings (RoPE), Skip-RoPE, that encodes genomic distances between packed fragments by using their aligned genomic coordinates to skip missing bases. This biologically-informed decoupling of context window size and genomic distance lets a 100M-parameter model trained to an effective context length of Mb using 60% fewer FLOPs than full-length training. Skip-RoPE also supports evaluation on discontinuous sequence inputs. We evaluate a 1 Mb model by predicting enhancer-gene interactions in a CRISPRi dataset through trained classifiers on frozen embeddings. Using Skip-RoPE, we achieve comparable results to full-context evaluation while requiring up to fewer input tokens. Apart from training and inference efficiency gains, we believe that this will enable a new class of gLMs that directly target raw short-read data.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
genomic language models, positional embeddings, long-context modeling, efficient training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8B6eOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Skip-RoPE supplies genomic source coordinates to RoPE when packing noncontiguous fragments sampled from the human reference genome. This separates the number of input tokens from the genomic sampling span. The paper studies sampling configurations, trains a 100M-parameter genomic language model on 30B tokens, and evaluates long-input language modeling, genomic prediction tasks, and enhancer–gene prediction with frozen representations. Its main reported outcomes are a favorable fixed-suffix perplexity–training-compute trade-off and useful predictive performance when parts of the intervening DNA are omitted at inference.

Critical strengths
  • The authors show that a simple training scheme can provide a favorable trade-off between prediction quality and training cost in the studied dense-attention setting. Figure 1b compares the schemes in terms of training FLOPs and fixed-suffix perplexity with 1 Mb of preceding context. The sampling experiments also show which configurations work well and when excessive sparsity or fragmentation degrades performance. These results provide useful guidance for choosing genomic pretraining configurations.

  • The paper's downstream results support the practical value of the method and show the performance trade-off when fewer input tokens are used. On rE2G, linear probes on frozen embeddings achieve 0.536 AUPRC versus 0.412 for the reported distance baseline, with an advantage in each reported distance group. Sparse inputs achieve 0.517 at the stated 2.1× token reduction. Table 3's DNALongBench results are also encouraging relative to the listed sequence models. Figure 5 also shows that adding infill improves prediction relative to the anchor-only extreme in the evaluated pipeline.

Critical weaknesses

The paper leaves open how much Skip-RoPE improves E2G prediction over closely related position-skip training schemes. PoSE [1], which the paper cites, already tested aligning position indices with the source text. Table 2 provides relevant internal evidence—Skip-RoPE improves over Sequential-RoPE on seven of eight tasks, but the gains over Random-RoPE are mixed. The strongest E2G evaluations do not include these internal controls. The work therefore establishes a viable genomic training scheme, while leaving its additional benefit for E2G prediction over a close training alternative uncertain. This limits the strength of the incremental contribution and is my main reservation about acceptance. The language-modeling training savings and downstream predictive results also do not establish reduced training compute at matched E2G quality.

[1] Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training. In The Twelfth International Conference on Learning Representations (ICLR), 2024. arXiv:2309.10400.

Questions for authors
  • If a comparable Random-RoPE checkpoint is available among the existing models, how does it perform under the same rE2G protocol, and what was its training compute? This would most directly clarify the value of the proposed training choice for downstream prediction.

  • Please confirm whether Figure 1b uses the same 16 validation loci as the breakpoint analysis, and specify the final sampling/RoPE configuration and FLOP-counting convention for the reported training scheme.

1 comment

W1. The paper leaves open how much Skip-RoPE improves E2G prediction over closely related position-skip training schemes. PoSE [1], which the paper cites, already tested aligning position indices with the source text. Table 2 provides relevant internal evidence—Skip-RoPE improves over Sequential-RoPE on seven of eight tasks, but the gains over Random-RoPE are mixed. The strongest E2G evaluations do not include these internal controls. The work therefore establishes a viable genomic training scheme, while leaving its additional benefit for E2G prediction over a close training alternative uncertain. This limits the strength of the incremental contribution and is my main reservation about acceptance. The language-modeling training savings and downstream predictive results also do not establish reduced training compute at matched E2G quality.

A1. We agree that the initial manuscript lacked a clear narrative on the incremental nature of our contribution over existing methods and thank the reviewer for raising the point that PoSE tested aligning position indices with the source text. We note that PoSE was applied as a post-training context-extension method and that the authors concluded that aligned positions have “relatively little impact on the outcomes of fine-tuning”.

In our latest revision, we have expanded the manuscript with a needle-in-a-haystack (NIAH) eval showcasing distal-retrieval to measure long context utility. As seen in Section 4.2 and Figure 5, this task shows clear and conclusive benefits in long-range modeling performance that is attributed to the use of faithful coordinates.

As such, we have modified our Contribution 3 to focus on the compute efficiency gain during pretraining for long-context robustness, and added a new Contribution 4 highlighting the importance of using faithful genomic co-ordinates that became directly evident through our new evaluation. The latter especially is a new finding over previous literature that asserts that whether coordinates are matched to their underlying tokens is of little consequence.

In order to improve the general clarity of our paper, we choose to focus on the NIAH eval as the primary illustration of our new Contribution 4 as it directly probes the attention mechanism, and have moved previous benchmark model comparisons into the Appendix.

Q1. If a comparable Random-RoPE checkpoint is available among the existing models, how does it perform under the same rE2G protocol, and what was its training compute? This would most directly clarify the value of the proposed training choice for downstream prediction.

A2. We thank the reviewer for this question, but as discussed above we have reframed our contributions. To summarize, since the original submission, we have expanded our paper with a new needle-in-a-haystack eval which directly clarifies the value of Skip-RoPE over Random-RoPE in leveraging long-context utility under both matched data and compute.

Q2. Please confirm whether Figure 1b uses the same 16 validation loci as the breakpoint analysis, and specify the final sampling/RoPE configuration and FLOP-counting convention for the reported training scheme.

A3. This is correct. Figure 1b uses the same 16 validation loci as the breakpoint analysis, scoring the same 512 bp suffixes with 1 Mb of preceding context.

The selected sampling configuration was T=65,536, W=2,097,152, K=2T=65,536,\ W=2,097,152,\ K=2: two 32,768 bp fragments sampled independently with replacement, retained in draw order, with independent reverse-complement probability 0.5 and faithful genomic coordinates. The original RoPE base was 5e7, but our NIAH eval revealed that this value was undertuned, so we retrained on the selected sampling configuration with theta 2e9.

Training FLOPs count linear-layer and causal-attention matrix multiplications, with two operations per multiply-add and backward estimated as twice forward. They exclude non-matmul operations, optimizer updates, communication, evaluation, and activation recomputation.

Reviewer_JNWJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present skip-RoPE, which levearges actual genomic coordinates instead of conventionally using positional indices for RoPE to train their genomic language models on discontinuous DNA fragments. The paper demnostrates that this skipping mechanism can extend effective context length, while reducing training compute and maintaining competitive performance.

Critical strengths
  • The paper is straigthforward and simple to follow . The motivation is also well described, which makes the proposed method suit well to their targeted setting.
  • The experiments provide useful insights about how discontinuously fragmented sample sequence affect long-context generalization.
  • the compute efficiency seems promising, and the evalutaiton covers several relvant genomic benchmarks.
Critical weaknesses
  • Limited novelty. The method is essentially RoPE with modifications to the fragmented setting. Similar positional skipping techniques have been already discussed in previous work, and the core diffrence is adopting this to actual genomic distances.
  • Limited applicability. Simlar to the previous point, this is tuned for this particular genomics setting, which is quite difficult to be unviersally adopted.
  • Limited evidence of long-range biological understanding. The claim is backed up with perplexity based measurements, which does tell that the model is more performant than the baselines, but hard to see if the model is actually understanding genomic regions.

Overall, I find this paper interesting and the efficienc ygain seems promising, but the technical novelty is limited and the experiments do not fully backup the claimed benefits for improved long-rage genomic interactions.

Questions for authors
  • Other than this paper tackling genomic coordinates, what are the major differences and improvemnets over existing positional skipping methods like PoSE?
  • Can the authors provide stronger evidence that show effectiveness of this method for extending effective context length?
  • Could this method be validated in other settings that are not genomics?
2 comments

W1. Limited novelty. The method is essentially RoPE with modifications to the fragmented setting. Similar positional skipping techniques have been already discussed in previous work, and the core diffrence is adopting this to actual genomic distances.

A1. We agree with the reviewer that our method is a modification of RoPE, but would still like to highlight our contributions over it.

Our contribution has three parts. First, unlike prior work in PoSE, we find that the the usage of faithful co-ordinates leads to a benefits within genomic data. This is evident both in our new Table 1, measuring long-context robustness, and Figure 5, measuring distal retrieval.

Second, we show that that this method can be leveraged throughout pre-training to reduce the compute cost of long-context models over traditional methods.

Finally, we want to point out that most modern sequenced biological data is fragmented (short-read data), and this method could enable pre-training directly on short sequencing reads.

W2. Limited applicability. Simlar to the previous point, this is tuned for this particular genomics setting, which is quite difficult to be unviersally adopted.

A2. We appreciate this question and acknowledge that it has not been properly discussed. We have added the following to the Future Work section: “Finally, we believe Skip-RoPE is applicable to other settings with data that can be aligned to a shared axis; specifically, we believe it has great promise for time-series data, which can also be irregularly sampled similar to short-read biological data.” Our rationale is that Skip-RoPE mainly needs tokens arranged along a shared, objective axis. Time series have this structure too. Clinical and physiological records, such as ICU vital signs and wearable sensor streams, have uneven sampling data aligned to a common time axis. We therefore expect our approach to extend naturally to these settings.

Second, we believe genomics is itself a broad domain. Genomic foundation models such as AlphaGenome (from DeepMind) have drawn wide attention, yet the field is still young, and better ways to train these models and use their data remain an open, high-impact problem. A method tailored to genomic data therefore serves a large and growing community, not a niche one.

Finally, data is the real bottleneck. Much of today’s clinical sequencing data is short-read, for example sequencing for personalized cancer vaccines and cell-free DNA sequencing for multi-cancer early detection. Our method is designed to make better use of exactly this kind of fragmented short-read data. It therefore applies directly to these use cases and, more broadly, to other short-read sequencing assays.

W3. Limited evidence of long-range biological understanding. The claim is backed up with perplexity based measurements, which does tell that the model is more performant than the baselines, but hard to see if the model is actually understanding genomic regions.

A3. We appreciate the reviewer in raising this point. The main contribution of our paper, as highlighted above, is the ability to train long-context models efficiently without a significant loss in performance, and demonstrate this in two ways:

  • We use perplexity based metrics to directly probe the attention mechanism of the model, both using a fixed-suffix validation and a needle-in-a-haystack eval.
  • We do not seek to design new biological evals, but use common genomic benchmarks that probe various aspects of biological understanding.

The Nucleotide Transformer benchmark (Table 2) tests recognition of promoters, enhancers, splice sites and histone-mark regions, which probe transcriptional regulation, RNA processing and chromatin state. Our model outperforms the baselines on these tasks, showing that it captures local sequence grammar.

For long-range understanding, we use three widely adopted long-context benchmarks. The Long-Range Benchmark (Table 3) has inputs of around 100 kb, with tasks such as distal variant effect prediction. The other two are enhancer–gene benchmarks, which ask whether the model can link regulatory regions that are far apart. On DNALongBench (Table 4), Skip-RoPE gets an AUPRC of 0.552, against 0.431 for distance alone, so the model is picking up more than how close the two regions are.

W1. Limited novelty. The method is essentially RoPE with modifications to the fragmented setting. Similar positional skipping techniques have been already discussed in previous work, and the core diffrence is adopting this to actual genomic distances.

A1. We agree with the reviewer that our method is a modification of RoPE, but would still like to highlight our contributions over it.

Our contribution has three parts. First, unlike prior work in PoSE, we find that the the usage of faithful co-ordinates leads to a benefits within genomic data. This is evident both in our new Table 1, measuring long-context robustness, and Figure 5, measuring distal retrieval.

Second, we show that that this method can be leveraged throughout pre-training to reduce the compute cost of long-context models over traditional methods.

Finally, we want to point out that most modern sequenced biological data is fragmented (short-read data), and this method could enable pre-training directly on short sequencing reads.

Q1. Other than this paper tackling genomic coordinates, what are the major differences and improvemnets over existing positional skipping methods like PoSE?

A4. We believe that the major differences and improvements over PoSE beyond genomic coordinates are:

  • Fragmented pretraining: PoSE is used as an efficient tool for context-length extension of pretrained models through fine-tuning (Zhu et al., 2024), whereas Skip-RoPE is used throughout the entirety of pretraining. We pretrain directly on potentially overlapping, unordered fragments and study how fragment length, genomic span, and token budget affect performance.
  • Sparse inference: We also use discontinuous inputs during evaluation, retaining comparable enhancer–gene prediction performance with up to 2.1× fewer tokens.
  • Evidence that alignment matters: PoSE reports little sensitivity to matching positions with source content. We demonstrate that faithful coordinates matter for both long-context robustness and distal retrieval using targeted ablations.

Q2. Can the authors provide stronger evidence that show effectiveness of this method for extending effective context length?

A5. In response to this request, we have introduced an additional distal retrieval task in Section 3.4 that specifically targets the usage of distal tokens over long contexts. Not only does this provide stronger evidence for Skip-RoPE extending effective context length, but we further demonstrate in Section 4.2 and Figure 5 that this benefit is strongest in Skip-RoPE, compared to Random offsets, which aligns closely with the core mechanism championed in Zhu et al., 2024, and other positional variants. We thank the reviewer for this suggestion and believe that this additional evidence strengthens the narrative around the effectiveness of our proposed method over alternatives.

Q3. Could this method be validated in other settings that are not genomics?

A6. Our work primarily focuses on genomic applications, motivated by the natural fit between skip-wise training and the fragmented, but distally aligned, nature of high-throughput short-read sequencing data. Within the field of genomics, we are excited about the broad range of applications described above.

Beyond genomics, we welcome further extensions of the method to fields of study where faithful alignment of disjoint data along a shared axis is intuitively important. One broad class of data that satisfies this requirement is time-series data, which has had a great deal of interest with recent foundation model releases such as Timestral, and similarly to genomic data can also be irregularly sampled; for instance, clinical and physiological data, such as ICU vital signs, EHR laboratory measurements, and wearable sensor streams, are characterized by substantial missingness and irregular sampling intervals. The demonstrated strength of Skip-RoPE in generative tasks in Figure 5 could be extended into imputation of missing time-series data.

Reviewer_5cfKOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Gene-regulatory interactions can span up to megabases, but sequencing reads are short and full attention over Mb-length sequences is expensive. The paper proposes Skip-RoPE: pack K discontinuous fragments into a T-token context and give RoPE their aligned genomic coordinates instead of sequential indices. This decouples compute (T) from genomic reach (W). Using semi-synthetic fragments sampled from hg38, a (T,W,K) sweep with a break-point metric B suggests a 64k-token model reaches 1 Mb effective context with about 2.5× fewer FLOPs. The released 100M model is competitive on NT and LRB and beats the distance baseline on enhancer–gene prediction.

Critical strengths
  1. Simple, cheap idea with practical guidance. It needs almost no architectural change, and the sweep yields concrete corpus-design rules: S = W/T ≈ 32 and read length ≥ 512 bp (Fig. 4).
  2. Promising enhancer–gene results. With frozen linear probes only, it beats distance in every distance bin (Table 4) and beats other sequence-only models on DNALongBench (Table 3).
Critical weaknesses
  1. The metric used to establish 1 Mb “effective context” does not measure use of long-range context. The onset B is defined as the point where the penalty CE(C) − CE(1k) exceeds 0.02 nats. It therefore measures the absence of degradation as context grows. A model that ignores tokens beyond a few kb would have B → ∞. Likewise, fixed-suffix perplexity at 1 Mb (Figure 1b) does not show whether 1 Mb of context helps relative to shorter context. B is also estimated from only 16 suffix sequences with a single seed; the 95% CI for the T = 4k model is already 19–56 kb. The power-law fit and the 2.5× FLOPs saving both rest on this estimate. This is consequential because compute-efficient long-context training is the paper’s main contribution.
  2. The comparisons needed to substantiate the method are missing or inconclusive.
    • Contribution 3 claims benchmark results comparable to full-length training. However, no compute- or context-matched contiguous model is evaluated on NT, LRB, ENCODE-rE2G, or DNALongBench; the FLOPs comparison is made only through B.
    • What distinguishes Skip-RoPE from PoSE (Zhu et al., 2024) is the use of true genomic coordinates, but the supporting evidence is weak. In Table 2, Skip-RoPE, Random RoPE, and Sequential RoPE differ by at most ~0.02 on any task. Skip-RoPE is the worst of the three on Causal eQTL and below Random RoPE on CAGE. No seeds or confidence intervals are reported.
    • On LRB, the baselines are fine-tuned while Skip-RoPE uses frozen probes. The protocol and provenance of the Table 3 baseline numbers are not stated.
    • As a result, neither the efficiency claim nor the specific value of coordinate alignment is established.
Questions for authors

Clarifications

  1. For Skip-RoPE models, does suffix CE decrease as C grows beyond T, i.e., is CE(C) < CE(T) for C ≫ T? A controlled distal-retrieval test would show directly whether long-range context is used, for example planting a copy of the suffix at distance d (generalizing the repeat case in Figure 2b).
  2. What is B for the Random and Sequential RoPE variants trained on the same fragmented data? If they also reach large B, that would support the concern that B measures robustness rather than long-range use.
  3. Can you report downstream results (LRB, ENCODE-rE2G, DNALongBench) for a contiguous model trained with matched compute, and for one trained at the longest feasible context? This is necessary to support Contribution 3.
  4. On ENCODE-rE2G, a mean-pooled embedding of the full span may implicitly encode span length, i.e., distance. How does logistic regression on distance plus the embedding compare with distance alone?

Suggestions for future work

  1. Training on real aligned reads, or on sampling that mimics assay coverage, sequencing errors, and indels, to test the motivating claim about raw short-read data.
  2. Ablating fragment overlap (allowed by Algorithm 1). Overlapping fragments share both coordinates and content, which may create a copy shortcut.
3 comments

We thank the reviewer for their incisive comments, which have helped us to significantly improve the manuscript. We particularly agree that a distal-retrieval task is well-suited for establishing long-context utility, and have introduced a new genomic needle-in-a-haystack (NIAH) task in Section 3.4 which demonstrates clear separation in behaviour between Skip-RoPE, Random, and Sequential, as seen in Section 4.2 and Fig. 5.

We amend our contributions to clearly distinguish training efficiency gains from our new claim that using faithful genomic coordinates improves both long-context robustness and distal retrieval, which is a new finding over previous literature that asserts that whether coordinates are matched to their underlying tokens is of little consequence.

W1a. The metric used to establish 1 Mb “effective context” does not measure use of long-range context. The onset B is defined as the point where the penalty CE(C) - CE(1k) exceeds 0.02 nats. It therefore measure the absence of degradation as context grows. A model that ignores tokens beyond a few kb would have B → ∞.

A1a. We agree with the reviewer on the limitations of this metric, and that a purely local model would be able to tolerate infinite lengths. As such, we have amended the manuscript to reframe the onset B as a measure of robustness, not utility, across long contexts.

In order to measure the use of long context, we have introduced a new genomic needle-in-a-haystack (NIAH) task in Section 3.4 which measures the ability for our trained models to copy an in-context ‘needle’ that is spliced onto the end of a long sequence. We differentiate between long-context usage and local decoding by measuring the benefit versus a baseline where the needle tokens undergo a dinucleotide-preserving shuffle. Using this new eval, we find that Skip-RoPE improves distal retrieval over both Random and Contiguous models, measured by per-base accuracy across a range of depths and context lengths.

W1b. Likewise, fixed-suffix penalty at 1 Mb (Figure 1b) does not show whether 1 Mb of context helps relative to shorter context.

A1b. While we agree with the reviewer on the limitations of a fixed-suffix penalty, we maintain that long-context robustness is a practical prerequisite that, in combination with the NIAH results, demonstrate the efficacy of Skip-RoPE as a compute-efficient method for training long-context models. The utility of such long-context genomic language models has been demonstrated by existing sequence and sequence-to-function models (Nguyen et. al, 2025, Brixi et al. 2026, Avsec et al. 2026).

W1c. B is also estimated from only 16 suffix sequences with a single seed; the 95% CI for the T = 4k model is already 19-56 kb. The power-law fit and the 2.5x FLOPs saving both rest on this estimate. This is consequential because compute-efficient long-context training is the paper’s main contribution.

A1c. We acknowledge that the wide CI on the 4k model could cast doubt on the statistical strength of subsequent results. In particular, the 2.5x FLOPs saving is indeed one of the paper’s core contributions. Due to the size of the sweep grid, length of the validation sequences (up to 4M), and computational/time constraints, we were unable to increase the number of suffix sequences or training seeds. However, we did re-run the subset of configurations used for the compute-efficient claim across 3 seeds and show in Figure 7 that this core result is robust across training runs.

W2a. The comparisons needed to substantiate the method are missing or inconclusive. Contribution 3 claims benchmark results comparable to full-length training. However, no compute- or context-matched contiguous model is evaluated on NT, LRB, ENCODE-rE2G, or DNALongBench; the FLOPs comparison is made only through B. What distinguishes Skip-RoPE from PoSE (Zhu et al., 2024) is the use of true genomic coordinates, but the supporting evidence is weak. In Table 2, Skip-RoPE, Random RoPE, and Sequential RoPE differ by at most ~0.02 on any task. Skip-RoPE is the worst of the three on Causal eQTL and below Random RoPE on CAGE. No seeds or confidence intervals are reported.

A2a. We agree with the reviewer on the primary distinction of Skip-RoPE versus PoSE and note that additionally Skip-RoPE is used throughout the entirety of pretraining, rather than as an efficient tool for context-length extension as in Zhu et al., 2024.

Following the expansion of our paper with the reviewer’s suggested distal-retrieval eval, we have modified our Contribution 3 to focus on the compute efficiency gain during pretraining for long-context robustness, and added a new Contribution 4 highlighting the importance of using faithful genomic co-ordinates that became directly evident through the reviewer’s suggested evaluation.

In order to improve the clarity of our paper, we choose to focus on the NIAH eval as the primary illustration of our new Contribution 4, and have moved previous benchmark model comparisons into the Appendix. In all such cases, we have augmented our results with a standard deviation measured over 3 training seeds.

W2b. On LRB, the baselines are fine-tuned while Skip-RoPE uses frozen probes. The protocol and provenance of the Table 3 baseline numbers are not stated.

A2b. We acknowledge that ideally the LRB protocol should be consistent across models and had originally stated in Footnote 2 that we chose to probe frozen embeddings due to computational constraints.

We have since carried out parameter-efficient fine-tuning runs, and have updated Table 2 with the latest results. Further, we have included a new appendix entry that describes our fine-tuning protocol, and cites the sources from which the Table 3 baseline numbers were obtained. We note that, since our original submission, Zhao et al have published a v4 GeneZip manuscript, and have amended Table 4 with their latest results.

W2c. As a result, neither the efficiency claim nor the specific value of coordinate alignment is established.

A2c. We thank the reviewer for their constructive comments. Having responded to them by reframing our existing results as a prerequisite for long-context utility, and showing improved performance versus Random and Sequential variants on our new NIAH eval, we believe the quality of our results and clarity of our manuscript have greatly been improved.

Q1. For Skip-RoPE models, does suffix CE decrease as C grows beyond T, i.e., is CE(C) < CE(T) for C >> T? A controlled distal-retrieval test would show directly whether long-range context is used, for example planting a copy of the suffix at distance d (generalizing the repeat case in Figure 2b).

A3. We agree with the reviewer on this suggestion, and have included a new eval as discussed above to address this concern. Addressing the specific question the reviewer has, we generally see that CE(C) ~ CE(T) for T << C < B, which reflects the low entropy of DNA sequences.

Q2. What is B for the Random and Sequential RoPE variants trained on the same fragmented data? If they also reach large B, that would support the concern that B measures robustness rather than long-range use.

A4. We measured B across Skip, Random, and Sequential RoPE variants across matched training data and recorded values of 1,682, 620, and 281 kb, respectively (Table 1). This correlates with respective performances on the NIAH task (Fig. 5) and we dedicate Section 4.2 in the revised manuscript to the comparative analysis of these positional encoding schemes.

Q3. Can you report downstream results (LRB, ENCODE-rE2G, DNALongBench) for a contiguous model trained with matched compute, and for one trained at the longest feasible context? This is necessary to support Contribution 3.

A5. We appreciate the prior necessity of this, but after changes to the manuscript arising from the reviewer’s other comments on distal retrieval, we have reframed our paper to instead focus on the pretraining cost reduction, long-context robustness and distal retrieval, and the importance of using faithful genomic co-ordinates. We believe this is a more accurate framing of our contributions, and thank the reviewer for helping us improve the quality of our manuscript by bringing attention to them.

Q4. On ENCODE-rE2G, a mean-pooled embedding of the full span may implicitly encode span length, i.e., distance. How does logistic regression on distance plus the embedding compare with distance alone?

A6. We agree that mean-pooling does not remove sequence-length information, and that full-span embeddings may therefore encode enhancer–promoter distance. As a result, this evaluation may not isolate distance encoding through faithful coordinates, and thus we do not establish ENCODE-rE2G performance as direct evidence of that capability. However, mean pooling is a standard readout for genomic foundation models such as Nucleotide Transformer and DNABERT-2, and we adopted it to stay consistent with this convention. Alternative readouts, such as last-token, max-pooling, or fixed-anchor mean-pooling, could reduce this potential leakage, and we consider them an important direction for future work.

For the requested ablation, we compared logistic regression on log-transformed distance alone, the frozen mean-pooled embedding alone, and their concatenation, using the same cross-validation and regularization-selection protocol. Results for three Skip-RoPE training seeds at θ=2×109\theta=2\times10^9 are:

\text{Seed} & \text{Distance} & \text{Embedding} & \text{Both} & \Delta\text{AP vs. distance}\\;[95\\%\text{ CI}] \\\\ \hline 71 & 0.398 & 0.544 & 0.519 & +0.121\\;[0.055,\\,0.179] \\\\ 73 & 0.398 & 0.485 & 0.489 & +0.091\\;[0.023,\\,0.150] \\\\ 74 & 0.398 & 0.496 & 0.483 & +0.086\\;[0.028,\\,0.140] \end{array}$$ **S1.** Training on real aligned reads, or on sampling that mimics assay coverage, sequencing errors, and indels, to test the motivating claim about raw short-read data. **A7.** We thank for the reviewer for this valuable suggestion, and do agree that our method is valid on data of similar sequence length as true short-read data. We are happy to share that such experiments are part of ongoing work on real datasets with ~100bp reads, including assays such as RNA-seq, ATAC-seq, and cell-free DNA methylation profiling. Their coverage patterns also reflect biologically meaningful signals that support downstream applications, such as predicting the effects of DNA variants and predicting patients disease status, etc. **S2.** Ablating fragment overlap (allowed by Algorithm 1). Overlapping fragments share both coordinates and content, which may create a copy shortcut. **A8.** We agree with the reviewer that overlapping fragments may encourage a copy shortcut, and this would be an interesting followup. We believe it would be an especially valuable investigation in conjunction with training on real short-read data, and whether de-duplication or removal of overlapping reads may be required. While still preliminary, some initial internal results show that models trained on overlapping and non-overlapping sequenced data have broadly similar downstream performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.