- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes STEP, a unified encoder pretraining framework for heterogeneous scientific time series. It compresses signals of varying lengths using a learnable patch mechanism, recovers scale information lost during normalization via statistical compensation, and uses cross-domain feature distillation to transfer representation knowledge from SPEAR, TimeMoE, and BrainOmni into a shared encoder. Evaluated on seven scientific time-series classification tasks, the paper demonstrates the transferability of domain foundation models and STEP's downstream performance.
Critical strengths
- The research problem is practical. The paper addresses significant variations in length, frequency, and numerical scale across scientific time series, and explores leveraging existing foundation models to mitigate data scarcity in scientific pretraining, holding clear practical value.
- The experiments cover multiple scientific domains and include solid methodological validation. Spanning seven scientific classification tasks, the paper provides component ablations, single-teacher versus multi-teacher distillation comparisons, representation alignment, and low-data experiments, offering preliminary evidence for the effectiveness of cross-domain knowledge transfer.
Critical weaknesses
- The reporting of core multi-teacher distillation results is incomplete. Figure 4 only shows a radar chart for the seven tasks without a complete numerical table. Without precise numbers, the performance gaps between multi-teacher, single-teacher, random initialization, and direct fine-tuning are hard to verify.
- Key definitions and implementations of adaptive patching remain problematic. In Equation (1), the patch stride depends only on sequence length rather than input content, and the paper fails to fully define Gaussian Windows. These issues obscure the actual learning process of the patching mechanism.
- The independent contribution of multi-teacher knowledge complementarity lacks sufficient validation. Because single-teacher and multi-teacher experiments use different pretraining corpora and schedules without matched budgets, it remains unclear whether gains stem from teacher complementarity or simply more training investment.
Questions for authors
- Can the authors provide complete numerical values and statistical variations for random initialization, each single-teacher, and multi-teacher STEP across all seven tasks (supplementing Figure 4), and clearly compare them with the direct fine-tuning results in Table 2?
- Since patch strides depend only on sequence length, how does STEP adapt to scientific signals of identical length but different dynamic features? Please clarify the formula for Gaussian Windows, the gradient computation for continuous strides, and the actual optimization process of the first-stage patch learning.
- Can the authors compare single-teacher and multi-teacher distillation under matched pretraining data scales and compute budgets to verify the independent benefits of teacher complementarity? Also, why was BrainOmni chosen instead of CBraMod (which participated in direct transfer)?
- Can the authors provide architecture comparisons under similar model capacities and input context conditions to clarify how much of STEP's performance advantage stems from new designs like adaptive patching?
3 comments
Response to Reviewer znHo(2/3)
Q2(a). Adaptation to Signals with Different Dynamics:
The stride is learned by an MLP that takes sequence length as input and is optimized jointly with the encoder, allowing the mapping to be learned from data characteristics rather than specified by a fixed rule. We also tested including the input signal in the MLP’s inputs, but observed less stable training. We therefore use the following length-conditioned mapping:
\qquad S=\min\left(S_{\mathrm{pred}},T/3\right), \qquad W=\lceil2S\rceil.$$ Here $T$ is the valid input length, $S_{\mathrm{pred}}$ is the predicted stride, and $S$ is the stride used to extract patches. Different task-trained models can therefore learn different patching strategies for similar input lengths. As shown in Table R3-3, we examine 100 examples each from GWOSC and MarmAudio using the models trained on the respective tasks. Despite similar mean input lengths (64,000 vs. 64,800), the mean learned strides differ substantially (80.18 vs. 121.86), resulting in different patch sizes and encoder token counts. Table R3-3: Results of learned patching strategies on GWOSC and MarmAudio. | Dataset | Mean input length | Learned stride | Patch size | Encoder tokens | |---|---|---|---|---| | GWOSC | 64,000 | 80.18 | 161 | 399 | | MarmAudio | 64,800 | 121.86 | 244.5 | 265 | **Q2(b). Gaussian Windows and Stride Optimization** We clarify below three implementation details of adaptive patching: the Gaussian-window formulation, gradient propagation through continuous strides, and the first-stage optimization procedure. **Gaussian-window formulation** Given the stride $S$ defined in Q2(a), we set the window size to $$W=\lceil2S\rceil.$$ Each patch gathers $W$ consecutive feature vectors from the convolutional output $\mathbf{X}'$ and applies Gaussian weights before within-patch temporal and channel attention. For patch $n$, the sampled positions and continuous window center are $$t_{n,j}=\lfloor nS\rfloor+j, \qquad \mu_n=nS+\frac{W}{2},$$ where $j\in\{0,\ldots,W-1\}$ and $n\in\{0,\ldots,P-1\}$. For $T>W$, the number of patches before subsampling is $$P=\left\lceil\frac{T-W}{S}\right\rceil+1.$$ To accommodate the final patch, the input and accompanying statistics are padded with zeros on the right by $$p=\lceil(P-1)S\rceil+W-T$$ positions before convolution and patch gathering. The Gaussian weights and resulting windowed features are $$G_{n,j}=\exp\left[-\frac{(\mu_n-t_{n,j})^2}{\alpha W^2}\right], \qquad \alpha=0.5,$$ $$\widetilde{\mathbf{X}}(n,j)=G(n,j)\mathbf{X}'(t(n,j)).$$ The weighted features are then processed by within-patch temporal and channel attention. **Gradient propagation through continuous strides** Although patch gathering uses discrete indices, the Gaussian center depends continuously on $S$. With the discrete indices and window size locally fixed, $$\frac{\partial G_{n,j}}{\partial S} =-\frac{2n(\mu_n-t_{n,j})}{\alpha W^2}G_{n,j}.$$ Indices and window sizes are recomputed in each forward pass, but gradients do not propagate through indexing or rounding. Since the applied stride is $$S=\min\left(S_{\mathrm{pred}},\frac{T}{3}\right),$$ Gaussian-weight gradients reach the stride predictor when clipping is inactive; this path has zero derivative with respect to $S_{\mathrm{pred}}$ when clipping is active. An additional gradient path is provided by the differentiable output-length surrogate, computed from the unrounded, unclipped predicted stride: $$N'=\frac{1}{4}\left(\frac{T-2S_{\mathrm{pred}}}{S_{\mathrm{pred}}}+1\right).$$ The factor $1/4$ accounts for two successive reductions by a factor of two. This surrogate is distinct from the integer patch count and actual subsampled length. When $N'\le0$, the implementation applies a Huber penalty to $\max(0,5-N')$ without evaluating $\log N'$; the logarithmic length penalty is evaluated only for $N'>0$. Thus, the length loss can update the stride predictor even when stride clipping is active. **First-stage optimization** During steps 0–1,000, we optimize the stride predictor using only the stride and output-length penalties, without classification supervision. These penalties encourage $$S_{\mathrm{pred}}\le400, \qquad 5\le N'\le200,$$ establishing a suitable patch resolution before task-specific learning. These are soft penalties rather than hard bounds, and gradients reach the stride predictor through $S_{\mathrm{pred}}$ and the differentiable length surrogate $N'$. Following this initialization, classification supervision is introduced with the Transformer backbone frozen during steps 1,000–2,000, and the full model is optimized jointly after step 2,000. *(The rest of your comments will be addressed in Response 3).*Response to Reviewer znHo(3/3)
Q3. Data Volume and Multi-Teacher Gains:
The TimeMoE-only distillation uses 48.21M samples from Time-300B. Multi-teacher distillation uses the same Time-300B samples, supplemented with 0.28M LibriSpeech samples and 0.75M BrainOmni samples, totaling 49.24M unique samples—an increase of only 2.14%. Thus, TimeMoE-only and multi-teacher distillation can be viewed as a comparison under approximately equal data budgets. Despite this small increase in data volume, Multi-teacher STEP outperforms TimeMoE-only STEP on six of seven tasks, as shown in Figure 4 of the main paper. This suggests that the gains are not simply a consequence of a large increase in pretraining data volume and is consistent with the benefit of complementary teacher supervision.
Q3. Choice of BrainOmni as the Neural-Signal Teacher:
We selected BrainOmni because it jointly models EEG and MEG, with pretraining on approximately 2,000 hours of EEG and 656 hours of MEG. In contrast, CBraMod is pretrained on EEG from TUEG. BrainOmni therefore offers broader neural-signal coverage as a distillation teacher. As noted in the paper, its two-stage tokenizer–foundation-model pipeline prevents straightforward end-to-end fine-tuning under our direct-transfer protocol, so it is excluded from Table 2. This does not prevent its use as a frozen teacher.
Q4. Architecture Comparisons and Model Capacity:
For baseline comparisons, we retain each model’s native input-processing pipeline to evaluate it in its intended configuration. Moirai provides a particularly relevant comparison: it also supports long sequences through patching, but uses predefined patch sizes rather than STEP’s learnable patching mechanism. Moirai has approximately 311M parameters, compared with STEP’s 125.47M. Thus, STEP’s advantage cannot be attributed to greater model capacity or simply to the ability to process long inputs.
In the from-scratch architecture comparison in Table 3, STEP outperforms Moirai on five of seven tasks, supporting the architectural advantages of STEP independently of pretraining.
We additionally compare multi-teacher STEP with existing foundation models by freezing all backbones and training only task-specific classification heads. As shown in Table R3-4, STEP achieves the best performance on three tasks and the second-best on three others, demonstrating the transferability of its pretrained representations without encoder adaptation.
Table R3-4: Macro-F1 of foundation models with frozen backbones on downstream scientific time-series tasks.
| GWOSC | LEAVES | STEAD | MarmAudio | SleepEDF | TUAB | RadarComm | Avg_Rank | |
|---|---|---|---|---|---|---|---|---|
| Whisper-large-v3 | 67.06 | 34.68 | 97.9 | 69.42 | 49.6 | 61.03 | 8.61 | 4.00 |
| Spear-xlarge-speech-audio | 91.21 | 41.87 | 92.33 | 77.84 | 64.37 | 73.41 | 55.12 | 3.00 |
| TimeMoE | 59.87 | 80.53 | 78.65 | 41.34 | 65.65 | 73.92 | 57.71 | 3.14 |
| Moirai-1.1-R-large | 56.25 | 64.76 | 94.53 | 41.9 | 48.43 | 69.79 | 42.79 | 4.29 |
| CBraMod | 46.57 | 14.89 | 87.96 | 16.65 | 63.56 | 74.3 | 13.55 | 4.86 |
| Multi-Teacher STEP | 62.50 | 70.79 | 97.22 | 92.80 | 66.90 | 76.95 | 55.98 | 1.71 |
Fine-tuned results are reported in Table 2 for the foundation models and Figure 4 for STEP. Rather than repeat the per-task scores, Table R3-2 above reports the mean macro-F1 and mean per-task rank across all seven tasks. STEP achieves the highest mean macro-F1 and the best mean rank, demonstrating its overall advantage after task-specific fine-tuning.
STEP achieves the strongest overall performance across all three evaluation settings: training from scratch, frozen-encoder evaluation, and task-specific fine-tuning—highlighting its effective architecture, transferable pretrained representations, and adaptability across scientific tasks.
Response to Reviewer znHo(1/3)
We thank the reviewer for the valuable comments and for acknowledging the practical relevance and extensive experimental validation of our work. Our responses are as follows:
Q1a. Complete numerical values of Figure 4:
Table R3-1 below provides the numerical results underlying Figure 4, including randomly initialized STEP, each single-teacher variant, and multi-teacher STEP across all seven tasks.
Table R3-1: Complete numerical results underlying Figure 4 with full encoder fine-tuning.
Q1b. Direct fine-tuning results:
Direct fine-tuning results for the foundation models are reported in Table 2 of the main paper. To facilitate comparison, Table R3-2 below summarizes the mean macro-F1 and mean per-task rank of the fine-tuned models. STEP achieves the highest mean macro-F1 and the best mean rank, demonstrating its overall advantage after task-specific fine-tuning.
Table R3-2: Overall performance after task-specific fine-tuning.
(The rest of your comments will be addressed in Response 2&3).