acceptodds
Under review as a conference paper at ICLR 2027

SpanDNA: Resolution-Aware Spectral Supervision for Genomic Pretraining

Abstract

Masked language modeling (MLM) does not explicitly specify which sequence-wide statistics should be readily accessible from a pooled genomic representation. We study a training-only auxiliary objective based on the Resolution-aware Nucleotide Spectrum (RNS), a compact Fourier representation of nucleotide indicator signals that combines composition with phase-sensitive low-frequency coordinates. predicts a 36-dimensional RNS-4 target jointly with MLM and discards the auxiliary head after pretraining. Using a 117M Transformer with the same pretraining data and update budget, three pretraining seeds, and three finetuning seeds per checkpoint, mean MCC across 18 tasks increases from 0.6421 to 0.6500. The mean gain is positive for each of the three reported pretraining seeds, and 15 of 18 task-level means improve. Fixed-descriptor experiments show that RNS contains predictive signal beyond composition under linear classification. In a single-pair representation analysis, frozen linear probes recover the 32 non-DC coordinates with after RNS supervision, compared with under MLM alone. These results show that RNS is an effective auxiliary target in the evaluated setting and that it changes the linear accessibility of supervised sequence-level statistics without modifying the deployed encoder.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
DNA language models, DNA foundation models, Masked language modeling, Spectral representation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_NStxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces the Resolution-aware Nucleotide Spectrum (RNS), a truncated discrete Fourier transform (DFT) of four nucleotide indicator signals, as an auxiliary pretraining target. RNS reduces to nucleotide composition at R=0R=0 and uniquely determines the sequence at full resolution. The authors establish completeness, a Parseval–Hamming identity, and a reverse-complement rule. SpanDNA, a 117M-parameter BPE Transformer, jointly optimizes masked language modeling (MLM) and regression of a 36-dimensional RNS-4 target from a span-weighted pooled representation, discarding the auxiliary head after pretraining. Across three pretraining and three fine-tuning seeds on 18 NT benchmark tasks, mean MCC improves from 0.6421 to 0.6500, with gains across all seeds and on 15 of 18 tasks. Additional analyses include a fixed-descriptor sweep over RR, frozen-representation probes, and a single-seed comparison against alternative targets.

Critical strengths
  1. Method adds zero inference cost, since the head is discarded after pretraining and the encoder, parameter count and latency are unchanged (Fig. 7)
  2. Rigorous design: three pretraining seeds, three finetuning seeds each, with a clearly specified statistical protocol.
  3. The component analysis makes a real case for retaining phase, showing that phase-retaining coordinates outperform magnitude alone as fixed descriptors (Fig. 2b).
  4. The writing is clear and the scoping is honest, with the authors openly acknowledging that they cannot attribute the gain to the spectral form.
  5. The core experiments are reproducible, with code, tokenizer, configs and hyperparameters all provided.
Critical weaknesses
  1. SpanDNA does not outperform a comparable existing model. DNABERT-2, rerun locally at the same size, wins 14 of 18 tasks. The overall tie depends on splice donors, where DNABERT-2’s reported score duplicates its splice-all score (Table 4) and should be verified.
  2. The improvement over MLM is small and uneven. It averages under one MCC point and comes mostly from splice and enhancer tasks, while the histone and promoter confidence intervals include zero (Table 7). The significance tests also treat overlapping tasks (e.g. splice all vs. acceptors and donors) as independent.
  3. Key ablations are missing. The paper is framed around resolution, yet every pretraining run uses R = 4. Without an RNS-only or randomly initialized baseline, it is also unclear how much each objective contributes.
  4. The evaluation does not test the paper’s motivation. RNS is meant to make sequence statistics easy to read out linearly from pooled representations, but all downstream tasks fully finetune the encoder. The probing analysis only shows that the supervised coordinates become recoverable, which is expected. It uses one task and one seed pair, and is never linked to downstream gains.
  5. The theory does not apply to the target actually used. Completeness and the Hamming identity require the full spectrum (~500 frequencies for 1,000-bp windows), but training uses only four. RNS-4 is essentially base composition plus coarse trends at 250-bp scales and above. The paper should state what this captures and what it misses.
Questions for authors
  1. Why do H3K9me3 and promoter TATA decline under RNS pretraining (Table 8) when RNS-4 features improve all 18 tasks as fixed descriptors (§4.2)?
  2. Do the splice-site gains survive randomly shifting the finetuning inputs, given that a site-centered window makes the exon/intron composition contrast exactly the low-frequency signal RNS-4 encodes?
  3. Does RNS still help under k-mer or single-nucleotide tokenization, given that DNABERT’s fixed k-mers lead on all three splice tasks?
  4. Does a magnitude-only target reproduce the gain, since Fig. 2b shows phase matters for fixed descriptors but not for pretraining?
  5. How were λ and its warm-up chosen, how sensitive are results to λ, and was λ re-tuned for the composition and Multiscale targets?
  6. Why does MLM-only trail DNABERT-2 on most non-splice tasks at similar size and corpus, and would RNS help as continued pretraining of DNABERT-2?
  7. Does the gain hold under frozen-encoder probes on the downstream tasks, which is what the paper’s motivation predicts?
  8. Does the MLM-only non-DC R² stay negative with cross-validated α?
2 comments

Thank you for the specific checks on downstream prediction, splice alignment, and the external baseline.

1.The fixed-descriptor gains show predictive information beyond nucleotide composition for both tasks. The fine-tuning comparison measures the incremental effect of adding RNS to an MLM-pretrained encoder. Information useful to the fixed linear classifier need not improve this stronger baseline. The cause of the two task-specific declines remains open.

2.Yes. We fine-tuned on centered and randomly shifted 400-bp crops from the same 600-bp examples, retaining the focal site with offsets up to 64 bp. Mean gain fell from +0.0075 to +0.0039 MCC. Shifted gains were +0.0041, +0.0033, and +0.0042 across the three pairs. Both models' absolute MCC fell after shifting.

3.Overlapping k-mers in the original DNABERT share bases and encode local context. RNS complements this with a low-frequency Fourier target for composition and coarse positional structure. For a given DNA span, it is defined independently of tokenization and can be used with k-mer or single-nucleotide inputs. Our reported gains are for BPE. Transfer requires a matched MLM/RNS comparison for each tokenizer.

4.Full RNS-4 achieved 0.3997 mean MCC as a fixed descriptor versus 0.3618 for composition plus magnitude, a gain of +0.0379. Phase retains information about where coarse patterns occur relative to the window origin, which motivated retaining the real and imaginary components. Whether a magnitude-only training target reproduces the downstream gains remains open.

5.In the original pretraining runs, we fixed the maximum auxiliary weight at λ=50\lambda=50 and ramped to it linearly over the first 5,000 updates to introduce the auxiliary signal gradually. Global composition and multiscale RNS used the same weight and ramp. Sensitivity to λ\lambda remains open.

In a separate 5,000-update continuation from a shared MLM checkpoint, RNS-4 led same-width nine-bin composition by +0.0041 MCC on 18 frozen tasks and +0.0038 on four fine-tuned tasks. Excluding splice, the leads were +0.0030 and +0.0020. This compares targets under a fixed budget. Weight sensitivity would require varying λ\lambda within a target.

6.Architecture, tokenizer, and training recipe differ, so the gap cannot be attributed to one factor. Our continuation shows a benefit from adding RNS to our MLM checkpoint. Transfer requires a paired continuation from DNABERT-2. The unrounded DNABERT-2 scores are 0.847996 for splice-all and 0.848170 for splice-donors. Both round to 0.848 in Table 4. We will report more precision.

7.Yes. We froze the original encoders and fitted L2-regularized logistic regression to the final span-weighted representation. We selected regularization by validation MCC on paired splits and evaluated on the official test sets.

Checkpoint pairMLMMLM+RNSGainGain excluding splice
10.53540.5483+0.0129+0.0058
20.53520.5479+0.0127+0.0057
30.53550.5484+0.0129+0.0058
Mean0.53540.5482+0.0128+0.0058

8.With the same fixed α=1\alpha=1, final-layer non-DC R2R^2 was −0.1760 for MLM-only and 0.9089 for MLM+RNS. These values establish the gap at the reported setting. The sign under cross-validated α\alpha remains open. In Q7, the downstream probe selects its regularization on validation MCC and shows a positive test-MCC gain in all three checkpoint pairs.

Related concerns. Task overlap and theory.

Excluding splice leaves +0.0054 MCC on 15 fully fine-tuned tasks. Given overlap among splice tasks, we will describe the task-level intervals and tests as descriptive. The main comparison estimates the effect of adding RNS to MLM. RNS-only and random-initialization baselines would answer separate questions. For 1,000-bp inputs, RNS-4 captures composition and wavelengths from 250 to 1,000 bp. Exact reconstruction and the Parseval–Hamming identity require the full spectrum.

I thank the authors for their response and additional experiments. The frozen-encoder probes are a useful addition, and the clarification regarding rounding resolves the Table 4 discrepancy.

However, my main concerns remain unresolved. SpanDNA still underperforms a parameter-matched DNABERT-2 on 14 of 18 tasks. Beyond splice prediction, the gains are marginal (approximately +0.005 MCC), and the distribution-shift experiment reduces the splice gain by half. The spectral target also shows only a narrow advantage over composition (+0.002–0.004 MCC) in a single short run, with no results across multiple seeds. Further, the ablations on R and λ, as well as a magnitude-only target, remain missing. Several questions (Q1, Q3, Q4, Q5, and Q8) also remain insufficiently addressed.

Given these unresolved concerns, I do not see sufficient evidence to justify raising my score.

Reviewer_nhJTOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

During DNA language model training, predicting masked nucleotides is supplemented with predicting low-frequency Fourier features of the entire DNA sequence to capture global sequence organization. This auxiliary task is used only during pretraining, with no additional modules at inference time.

Critical strengths
  1. Downstream tasks use whole-sequence representations, but MLM does not directly constrain which global statistics these representations should preserve. RNS provides a concrete, computable auxiliary objective that incorporates overall composition and positional structure into supervision.
  2. Requires no additional annotations and no extra inference cost.
Critical weaknesses
  1. The performance improvement is not obvious.
  2. The impact of resolution selection in pretraining has not been sufficiently evaluated.
Questions for authors
  1. Why was R=4 selected? How sensitive are downstream results to the frequency order used during pretraining?
  2. What additional computational and memory costs does RNS supervision introduce during pretraining?
2 comments

Thank you for asking us to clarify the resolution choice and the cost of pretraining.

1.We fixed R=4R=4 before the descriptor sweep to capture coarse positional structure with 36 coordinates. The fixed-descriptor comparison yielded 0.3499 MCC for composition, 0.3997 for RNS-4, and 0.4372 for the 260-dimensional RNS-32. RNS-4 improved over composition on all 18 tasks. Higher orders can add predictive information to the descriptor. The effect of changing frequency order during pretraining remains unmeasured.

2.MLM+RNS uses the same encoder as MLM-only and adds a small prediction head during pretraining. Online target computation and the auxiliary head add training overhead in the current implementation. The throughput-based compute estimate is reported in the Efficiency Measurements appendix, and peak training memory remains unmeasured. We remove the head for inference, where measured latency and peak allocated GPU memory were comparable to MLM-only.

Thanks for the clarification. I will maintain my original score.

Reviewer_45C5OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes SpanDNA, which adds a training-only auxiliary objective to the MLM pretraining of a DNA language model: the model predicts the low-frequency Fourier spectrum of the sequence from its pooled representation, and the auxiliary head is removed after pretraining. Across 18 downstream tasks, the average MCC improves from 0.6421 to 0.6500.

Critical strengths
  • The method adds sequence-level supervision without external labels or extra inference-time parameters. Descriptor and representation analyses show what the auxiliary objective makes accessible, and the paper is careful to separate the full-spectrum theory from the truncated target it actually evaluates.
Critical weaknesses
  • The distinct contribution of spectral supervision isn’t established. The full-spectrum propositions follow from DFT invertibility and Parseval’s identity but don’t explain why the compact RNS-4 training target works. There’s no comparison with a similarly sized non-spectral position-aware target (e.g., composition in contiguous bins).

  • The frozen-probe result only confirms recovery of explicitly supervised features. So the evidence doesn’t separate the spectral design from a generic sequence-level auxiliary objective. The demonstrated benefit is too small to carry practical significance: +0.0079 mean MCC over matched MLM, and essentially a tie with locally rerun DNABERT-2. With the spectral contribution unresolved, this limits the case for acceptance.

Questions for authors
  • Could the authors clarify how the hyperparameters were selected, specifically how R = 4 was determined given that the sweep is stated not to be used for selecting the pretraining optimum, and how λ = 50 and the SmoothL1 β = 1 were chosen and how sensitive the results are to them

  • Since the evidence that phase information is useful comes only from Figure 2b, where RNS is fed directly as fixed features to a logistic regression , have the authors run a pretraining ablation that uses magnitude-only prediction as the auxiliary target, and if not, what supports the design choice of preserving phase during pretraining?

3 comments

Thank you for asking what supports the choice of target and its hyperparameters.

1.We set R=4R=4 before the descriptor sweep as a compact 36-coordinate operating point that preserves four coarse frequency modes alongside composition. The sweep measures the information available at different descriptor widths. We used the standard Smooth L1 default β=1\beta=1 with mean reduction. The original training recipe fixes the maximum auxiliary weight at λ=50\lambda=50 and ramps it linearly over the first 5,000 optimizer steps to introduce the auxiliary signal gradually. Controlled pretraining sweeps over RR, λ\lambda, and β\beta remain open.

2.We have not run a magnitude-only pretraining ablation. Phase retains information about where coarse patterns occur relative to the window origin. This motivated retaining the real and imaginary components in the training target. As fixed descriptors, full RNS-4 achieved 0.3997 mean MCC versus 0.3618 for composition plus magnitudes. The descriptors have 36 and 20 coordinates, respectively. The downstream effect of phase as a pretraining target remains unmeasured.

Related concern. Does the truncated target offer an advantage over a simpler position-aware target?

In matched 5,000-update continuations from a shared MLM-only checkpoint, RNS-4 led a 36-coordinate nine-bin composition target by +0.0041 MCC on 18 frozen tasks and +0.0038 on four fine-tuned tasks. The leads without splice were +0.0030 and +0.0020. A comparison over the original training budget from random initialization remains open. With the original models, the new frozen-encoder evaluation of actual downstream labels gave +0.0128 MCC across three checkpoint pairs and +0.0058 without splice. The submitted full fine-tuning gain is modest, and mean MCC is close to the local DNABERT-2 rerun.

At R=4R=4, the target contains composition and wavelengths N,N/2,N/3,N/4N, N/2, N/3, N/4 for a window of length NN. It omits finer structure. Exact reconstruction and the Parseval–Hamming identity apply to the complete spectrum. The downstream controls above provide the empirical evidence for the truncated training target.

Area_Chair_wnXgOpenReview ↗

Reminder to engage in reviewer-author discussion

Hi 45C5,

As there are less than two days left in the author-reviewer discussion period, I would like to remind you to kindly respond to the authors' rebuttal. After November 18, reviewer responses are not be allowed and you will no longer be able to ask any follow-up questions. If you have no follow-ups, a short and polite response acknowledging you have read and understood the rebuttal and have no further questions is sufficient.

Thanks, AC

Thanks authors for addressing some of my concerns, i will raise the score.

Reviewer_Y8YSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

SpanDNA introduces a training-only auxiliary objective for genomic language-model pretraining. The added objective is a compact, phase-preserving Fourier representation of nucleotide indicator signals to supervise span-weighted pooled representations. The authors show through controlled experiments with a 117M-parameter encoder, three pretraining seeds, and 18 downstream tasks, that the method improves mean MCC from 0.6421 to 0.6500. The results support a modest benefit from sequence-level auxiliary supervision but in current form do not establish that it is the Fourier-specific structure that drives the gains or that the method improves on existing genomic language models.

Critical strengths

(1) MLM and MLM+RNS conditions use the same encoder, data, training budget, and fine-tuning protocol. This provides a meaningful comparison of the auxiliary objective. (2) All three pretraining seeds show positive aggregate gains, with improvement in 15/18 downstream tasks. This increases confidence that the result is not confined to a single run. (3) Auxiliary prediction head is removed after pretraining, thus adding no inference-time parameters or computation to deployed encoder. (4) The spectral representation retains phase information and has a resolution parameter that connects nucleotide composition to increasingly-detailed sequence descriptors. Accompanying theoretical analysis gives this construction a clear mathematical interpretation. (5) Authors report negative task-level results and describe the limits of uncertainty estimates.

Critical weaknesses

(1) Mean MCC increases by 0.0079 (from 0.6421 to 0.6500). SpanDNA about matches the locally-rerun DNABERT-2 result of 0.6496. So, the experiments demonstrate improvement over matched MLM baseline but not a clear advance over existing models. As a result, comparisons with published results under different evaluation budgets should be interpreted cautiously. (2) Ablations do not compare RNS against non-spectral position-aware targets, such as binned nucleotide composition, or k-mer counts. Similar results for global and multi-scale RNS also leave unresolved the value of the specific spectral construction. I find this to be the key threat to the novelty claim. (3) Auxiliary-target ablations use one pre-training seed, which also has the largest reported RNS gain. The composition-only performance decrease is difficult to interpret without sensitivity analyses for the loss weight lambda and training schedule. This is a robustness issue. (4) The increase in linear recoverability shows that the model learns to encode the supervised target (expected from the training objective). However, the single-task single-seed-pair probing analysis does not establish that this change explains downstream gains. (5) There is no ablation of span-weighted pooling against mean or CLS pooling, isolation of contribution of RoPE at base-pair centers, or adequate justification of choice of R=4. (6) The evaluation is limited to one model scale, sequence windows of at most 1,000 bp, and the 18 nucl transformer classification tasks. Generalization to broader downstream settings remains untested. Potential dependence of phase reference on window alignment also warrants discussion.

Questions for authors

(1) can you provide results across multiple seeds for the auxiliary-target ablations, given that the current comparison uses only seed 42, which also shows the largest RNS gain?

(2) How does RNS compare with non-spectral, position-aware targets, such as binned nucleotide composition or k-mer counts, to establish that the gains are specific to the fourier supervision?

(3) What motivates the choice of lambda=50 and R=4. How sensitive may downstream results be to these choices?

(4) Can you isolate the contribution of span-weighted pooling by comparing it with mean pooling or CLS pooling under the same training setup?

(5) How robust are reported gains across larger model scales, or longer sequence windows, or broader downstream benchmarks?

2 comments

Thank you for identifying the comparison with a simpler position-aware auxiliary objective.

1.The composition and multiscale ablations each use one pretraining run. The nine-bin control also uses one continuation run per objective from a common source checkpoint. We agree that seed 42 gives the largest gain in the original MLM/RNS comparison. The three original checkpoint pairs address consistency for MLM versus MLM+RNS. The ranking of auxiliary objectives across pretraining seeds remains open. We used the same nominal weight and schedule for the original composition and multiscale controls, and observed the composition-only decline under that recipe. Its sensitivity to the loss weight remains open.

2.RNS-4 achieved higher mean MCC than a same-width nine-bin composition target in both evaluations at 5,000 continuation updates. Nine contiguous bins with four nucleotide proportions each give 36 coordinates. All branches start from the same MLM-only checkpoint and use the same continuation budget.

Continuation objective18 frozen tasks, mean MCC4 fine-tuned tasks, mean MCC
MLM only0.53200.6470
MLM + nine-bin composition0.53790.6512
MLM + RNS-40.54200.6550

The four fine-tuned tasks are H3K4me1, enhancer prediction, promoter-all, and splice-sites-all. RNS-4 leads nine-bin by +0.0041 MCC under frozen evaluation and +0.0038 under full fine-tuning. Excluding splice, the leads are +0.0030 over 15 frozen tasks and +0.0020 over three fine-tuned tasks. Both auxiliary objectives improve over continued MLM on the full task sets. This measures the benefit over the tested positional target under the 5,000-update protocol. A comparison from random initialization at the original budget and a k-mer auxiliary objective remain open.

3.We fixed R=4R=4 before the descriptor sweep to keep the target compact at 36 coordinates while covering composition and four coarse spatial frequencies. The fixed descriptor improved from 0.3499 MCC at R=0R=0 to 0.3997 at R=4R=4, and higher orders retained additional signal. In the original runs, λ=50\lambda=50 was a fixed recipe setting with a 5,000-step linear ramp to introduce the auxiliary signal gradually. We have no controlled pretraining sweep over RR or λ\lambda. Global and multiscale RNS achieved similar scores in the single-run comparison at 36 and 252 coordinates, respectively.

4.A matched mean- or CLS-pooling ablation is still open, as is a separate ablation of base-pair-centered RoPE. Span weighting gives each BPE token a contribution proportional to the number of bases it covers. Both main conditions use the same pooling and RoPE, so the comparison measures the objective's effect with those choices held fixed.

5.The evidence covers the 117M encoder, windows up to 1,000 bp, and the 18 NT tasks. Cross-scale, long-context, and cross-benchmark robustness remain unmeasured. In the window-offset test, the mean gain fell from +0.0075 to +0.0039 MCC and stayed positive in all three shifted checkpoint pairs.

Related concerns. Frozen downstream utility and DNABERT-2.

With the original frozen checkpoints, linear classifiers predicting actual downstream labels gained +0.0128 MCC over MLM on 18 tasks and +0.0058 without splice. This directly evaluates linear downstream prediction. The local DNABERT-2 rerun has similar mean full fine-tuning MCC. We base our conclusion on gains over the matched internal MLM baseline.

Reviewer_Y8YSOpenReview ↗

Thanking the authors but keeping the score

I thank the authors for the additional experiments (in particular, the 9-bin composition control and the window-offset test). I am however keeping my overall rating at 2 and my soundness score at 3 for the following reasons. While the authors have added experiments that now clearly show the limitations I raised in my original review, my original concerns remain. For example:

(1) The 9-bin comparison gives RNS-4 an advantage, though lower without splice. However, the results come from a single 5,000-update continuation with one seed. I am not yet convinced that the gain is specific to the Fourier construction. This could still be a more general benefit of just using a position-aware target. This is a key issue for me that remains.

(2) The results in the window-offset experiment suggest that a substantial part of the signal is tied to window position. This is particularly relevant for splice.

(3) Several important comparisons additionally remain unmeasured. See original revew on sweeps, magnitude-only target, pooling ablation, multi-seed comparison of auxiliary targets... (4) The paper also made resolution key to the motivation for the work. But every pretraining run still has R = 4.

Reviewer_MdhmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces SpanDNA, which adds an auxiliary objective to MLM pretraining for genomic sequences. They predict low-frequency Fourier coefficients of the nucleotide indicator signals (RNS) from a pooled sequence representation. Across 18 downstream tasks, the authors report a performance gain, supported by theoretical properties of RNS and linear-probing analyses.

Critical strengths
  1. The auxiliary training objective is simple and adds no inference cost, making it easy to adopt.
Critical weaknesses
  1. The evaluation does not test the paper’s stated motivation. All downstream results use full fine-tuning, where linear accessibility of frozen features is not the limiting factor. In NLP, frozen pooled BERT embeddings are considered weak but improve when fine-tuned (Reimers & Gurevych, 2019; Ethayarajh, 2019). A model trained to predict RNS has RNS linearly recoverable, which is expected by construction.
  2. The performance gain is too small. All runs do not account for pretraining or fine-tuning variance.
  3. The source of improvement is not identified. The RNS phase is referenced to the start of the window, while the encoder uses RoPE, which provides no explicit absolute position. The gain may therefore come from teaching the model where nucleotides lie within the window. This would particularly help on feature-aligned tasks such as splice sites. Without controls against simpler position-aware targets, it remains unclear whether the improvement is specific to the spectral formulation.
  4. Most of the hyperparameters are chosen arbitrarily, such as R=4R=4 and the RNS loss weight (λ=50\lambda = 50).
Questions for authors
  1. How do MLM-only and MLM+RNS compare when the encoder is frozen and only a linear classifier/regressor is trained on each downstream task?
  2. Does the performance gain persist when downstream windows are randomly shifted so that features (splice sites) are not at a fixed position?
  3. Were the fine-tuning hyperparameters tuned for either model?
  4. Evaluation at larger model scales, longer contexts, and on other benchmarks (e.g., BEND, variant-effect prediction) would show whether the method generalises well.
2 comments

Thank you for proposing tests of linear accessibility and window alignment.

1.MLM+RNS improved mean MCC in all three checkpoint pairs on the 18 NT tasks. The gains were +0.0129, +0.0127, and +0.0129, with a mean of +0.0128. Excluding all three splice tasks, the mean gain was +0.0058. We froze the entire encoder and fitted L2-regularized logistic regression to its final span-weighted representation. Both methods used the same three stratified splits and regularization grid. We selected regularization on validation MCC, refitted on the official training split, and evaluated on the official test split. This experiment predicts the downstream labels directly. The original ridge probe measures recovery of the spectral target.

2.Yes, under the tested offsets. On the three-class splice task, we compared centered and randomly shifted 400-bp crops from the same original 600-bp sequences. The shifts were at most 64 bp and kept the focal site within the crop. We fine-tuned each checkpoint separately under each condition using the same samples, offsets, and split seeds for both methods. The mean gain fell from +0.0075 MCC with centered crops to +0.0039 with shifted crops. The shifted gain was positive in each checkpoint pair, and both methods had lower absolute MCC after shifting. Alignment affects the size of the gain.

3.Neither model received a separate hyperparameter sweep. Both used AdamW, a learning rate of 2×10−52\times10^{-5}, batch size 8, and three epochs. We used validation MCC to select an epoch and evaluated once on the official test split. The submitted experiment evaluated three checkpoint pairs and three fine-tuning seeds per checkpoint. The pairwise gains were +0.0104, +0.0068, and +0.0064 MCC.

4.These settings remain untested. Our evidence covers the 117M encoder, input windows up to 1,000 bp, and the 18 NT classification tasks.

5.In a matched 5,000-update continuation from one MLM-only checkpoint, RNS-4 led a 36-dimensional nine-bin composition target by +0.0041 MCC on 18 frozen tasks and +0.0038 on four fine-tuned tasks. The gains excluding splice were +0.0030 and +0.0020. This comparison tests the choice of target during continued pretraining. Its ranking over the full training budget from random initialization remains open. In the original recipe, R=4R=4 gives a compact target and λ=50\lambda=50 is a fixed setting with a 5,000-step ramp. We have no pretraining sensitivity sweep for either.

Reviewer_MdhmOpenReview ↗

Response to Authors

I thank the authors for the clarifications. The results suggest that most of its predictive signal comes from window-anchored positional information rather than from the spectral formulation. Given that the performance improvement is about 0.01 MCC with untuned fine-tuning settings, I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.