acceptodds
Under review as a conference paper at ICLR 2027

STEP: Scientific Time-Series Encoder Pretraining via Cross-Domain Distillation

Abstract

Scientific time series are central to scientific AI but are typically sparse, highly heterogeneous, and limited in scale, making unified representation learning particularly challenging. Meanwhile, foundation models pretrained on relevant time series domains such as audio, general time series, and brain signals contain rich knowledge, but their applicability to scientific signals remains underexplored. In this paper, we investigate the transferability and complementarity of foundation models from relevant time series domains, and study how to effectively leverage them to build a unified encoder for scientific time series. We first systematically evaluate relevant foundation models, showing the effectiveness of knowledge transfer to scientific tasks and their complementary strengths. Based on this observation, we propose STEP, a Scientific Time Series Encoder Pretraining framework via cross-domain distillation. STEP introduces adaptive patching to handle extreme-length sequences and a statistics compensation scheme to accommodate diverse numerical scales. It further leverages cross-domain distillation to integrate knowledge from multiple foundation models into a unified encoder. By combining complementary representations across different domains, STEP learns general-purpose and transferable features tailored for scientific signals. Experiments on seven scientific time series tasks demonstrate that STEP provides both an effective structure and an effective pretraining paradigm, taking a STEP toward scientific time series representation learning.

Primary area:
learning on time series and dynamical systems
Keywords:
Scientific Time Series, Foundation Model, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_znHoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes STEP, a unified encoder pretraining framework for heterogeneous scientific time series. It compresses signals of varying lengths using a learnable patch mechanism, recovers scale information lost during normalization via statistical compensation, and uses cross-domain feature distillation to transfer representation knowledge from SPEAR, TimeMoE, and BrainOmni into a shared encoder. Evaluated on seven scientific time-series classification tasks, the paper demonstrates the transferability of domain foundation models and STEP's downstream performance.

Critical strengths
  • The research problem is practical. The paper addresses significant variations in length, frequency, and numerical scale across scientific time series, and explores leveraging existing foundation models to mitigate data scarcity in scientific pretraining, holding clear practical value.
  • The experiments cover multiple scientific domains and include solid methodological validation. Spanning seven scientific classification tasks, the paper provides component ablations, single-teacher versus multi-teacher distillation comparisons, representation alignment, and low-data experiments, offering preliminary evidence for the effectiveness of cross-domain knowledge transfer.
Critical weaknesses
  • The reporting of core multi-teacher distillation results is incomplete. Figure 4 only shows a radar chart for the seven tasks without a complete numerical table. Without precise numbers, the performance gaps between multi-teacher, single-teacher, random initialization, and direct fine-tuning are hard to verify.
  • Key definitions and implementations of adaptive patching remain problematic. In Equation (1), the patch stride depends only on sequence length rather than input content, and the paper fails to fully define Gaussian Windows. These issues obscure the actual learning process of the patching mechanism.
  • The independent contribution of multi-teacher knowledge complementarity lacks sufficient validation. Because single-teacher and multi-teacher experiments use different pretraining corpora and schedules without matched budgets, it remains unclear whether gains stem from teacher complementarity or simply more training investment.
Questions for authors
  • Can the authors provide complete numerical values and statistical variations for random initialization, each single-teacher, and multi-teacher STEP across all seven tasks (supplementing Figure 4), and clearly compare them with the direct fine-tuning results in Table 2?
  • Since patch strides depend only on sequence length, how does STEP adapt to scientific signals of identical length but different dynamic features? Please clarify the formula for Gaussian Windows, the gradient computation for continuous strides, and the actual optimization process of the first-stage patch learning.
  • Can the authors compare single-teacher and multi-teacher distillation under matched pretraining data scales and compute budgets to verify the independent benefits of teacher complementarity? Also, why was BrainOmni chosen instead of CBraMod (which participated in direct transfer)?
  • Can the authors provide architecture comparisons under similar model capacities and input context conditions to clarify how much of STEP's performance advantage stems from new designs like adaptive patching?
3 comments
AuthorsOpenReview ↗

Response to Reviewer znHo(1/3)

We thank the reviewer for the valuable comments and for acknowledging the practical relevance and extensive experimental validation of our work. Our responses are as follows:

Q1aa. Complete numerical values of Figure 4:

Table R3-1 below provides the numerical results underlying Figure 4, including randomly initialized STEP, each single-teacher variant, and multi-teacher STEP across all seven tasks.

Table R3-1: Complete numerical results underlying Figure 4 with full encoder fine-tuning.

InitializationGWOSCLEAVESSTEADMarmAudioSleepEDFTUABRadarCommAvg_Rank
STEP-Random95.6983.4196.4493.6872.2177.8160.844.57
STEP-Spear97.3684.2398.7898.4574.8478.5760.442.57
STEP-TimeMoE96.1985.9599.5195.1373.3778.2160.852.71
STEP-BrainOmni94.7284.6596.8793.1972.5878.7761.163.57
STEP-Multi-Teacher96.4886.2498.5897.9074.8679.7261.461.57

Q1bb. Direct fine-tuning results:

Direct fine-tuning results for the foundation models are reported in Table 2 of the main paper. To facilitate comparison, Table R3-2 below summarizes the mean macro-F1 and mean per-task rank of the fine-tuned models. STEP achieves the highest mean macro-F1 and the best mean rank, demonstrating its overall advantage after task-specific fine-tuning.

Table R3-2: Overall performance after task-specific fine-tuning.

Whisper-large-v3Spear-xlarge-speech-audioTimeMoE-200MMoirai-1.1-R-largeCBraModSTEP(Multi)
Avg-F1 ↑79.4578.2279.2482.0575.8585.03
Avg-Rank ↓3.1443.143.574.292.86

(The rest of your comments will be addressed in Response 2&3).

AuthorsOpenReview ↗

Response to Reviewer znHo(2/3)

Q2(a). Adaptation to Signals with Different Dynamics:

The stride is learned by an MLP that takes sequence length as input and is optimized jointly with the encoder, allowing the mapping to be learned from data characteristics rather than specified by a fixed rule. We also tested including the input signal XX in the MLP’s inputs, but observed less stable training. We therefore use the following length-conditioned mapping:

\qquad S=\min\left(S_{\mathrm{pred}},T/3\right), \qquad W=\lceil2S\rceil.$$ Here $T$ is the valid input length, $S_{\mathrm{pred}}$ is the predicted stride, and $S$ is the stride used to extract patches. Different task-trained models can therefore learn different patching strategies for similar input lengths. As shown in Table R3-3, we examine 100 examples each from GWOSC and MarmAudio using the models trained on the respective tasks. Despite similar mean input lengths (64,000 vs. 64,800), the mean learned strides differ substantially (80.18 vs. 121.86), resulting in different patch sizes and encoder token counts. Table R3-3: Results of learned patching strategies on GWOSC and MarmAudio. | Dataset | Mean input length | Learned stride | Patch size | Encoder tokens | |---|---|---|---|---| | GWOSC | 64,000 | 80.18 | 161 | 399 | | MarmAudio | 64,800 | 121.86 | 244.5 | 265 | **Q2(b). Gaussian Windows and Stride Optimization** We clarify below three implementation details of adaptive patching: the Gaussian-window formulation, gradient propagation through continuous strides, and the first-stage optimization procedure. **Gaussian-window formulation** Given the stride $S$ defined in Q2(a), we set the window size to $$W=\lceil2S\rceil.$$ Each patch gathers $W$ consecutive feature vectors from the convolutional output $\mathbf{X}'$ and applies Gaussian weights before within-patch temporal and channel attention. For patch $n$, the sampled positions and continuous window center are $$t_{n,j}=\lfloor nS\rfloor+j, \qquad \mu_n=nS+\frac{W}{2},$$ where $j\in\{0,\ldots,W-1\}$ and $n\in\{0,\ldots,P-1\}$. For $T>W$, the number of patches before subsampling is $$P=\left\lceil\frac{T-W}{S}\right\rceil+1.$$ To accommodate the final patch, the input and accompanying statistics are padded with zeros on the right by $$p=\lceil(P-1)S\rceil+W-T$$ positions before convolution and patch gathering. The Gaussian weights and resulting windowed features are $$G_{n,j}=\exp\left[-\frac{(\mu_n-t_{n,j})^2}{\alpha W^2}\right], \qquad \alpha=0.5,$$ $$\widetilde{\mathbf{X}}(n,j)=G(n,j)\mathbf{X}'(t(n,j)).$$ The weighted features are then processed by within-patch temporal and channel attention. **Gradient propagation through continuous strides** Although patch gathering uses discrete indices, the Gaussian center depends continuously on $S$. With the discrete indices and window size locally fixed, $$\frac{\partial G_{n,j}}{\partial S} =-\frac{2n(\mu_n-t_{n,j})}{\alpha W^2}G_{n,j}.$$ Indices and window sizes are recomputed in each forward pass, but gradients do not propagate through indexing or rounding. Since the applied stride is $$S=\min\left(S_{\mathrm{pred}},\frac{T}{3}\right),$$ Gaussian-weight gradients reach the stride predictor when clipping is inactive; this path has zero derivative with respect to $S_{\mathrm{pred}}$ when clipping is active. An additional gradient path is provided by the differentiable output-length surrogate, computed from the unrounded, unclipped predicted stride: $$N'=\frac{1}{4}\left(\frac{T-2S_{\mathrm{pred}}}{S_{\mathrm{pred}}}+1\right).$$ The factor $1/4$ accounts for two successive reductions by a factor of two. This surrogate is distinct from the integer patch count and actual subsampled length. When $N'\le0$, the implementation applies a Huber penalty to $\max(0,5-N')$ without evaluating $\log N'$; the logarithmic length penalty is evaluated only for $N'>0$. Thus, the length loss can update the stride predictor even when stride clipping is active. **First-stage optimization** During steps 0–1,000, we optimize the stride predictor using only the stride and output-length penalties, without classification supervision. These penalties encourage $$S_{\mathrm{pred}}\le400, \qquad 5\le N'\le200,$$ establishing a suitable patch resolution before task-specific learning. These are soft penalties rather than hard bounds, and gradients reach the stride predictor through $S_{\mathrm{pred}}$ and the differentiable length surrogate $N'$. Following this initialization, classification supervision is introduced with the Transformer backbone frozen during steps 1,000–2,000, and the full model is optimized jointly after step 2,000. *(The rest of your comments will be addressed in Response 3).*
AuthorsOpenReview ↗

Response to Reviewer znHo(3/3)

Q3aa. Data Volume and Multi-Teacher Gains:

The TimeMoE-only distillation uses 48.21M samples from Time-300B. Multi-teacher distillation uses the same Time-300B samples, supplemented with 0.28M LibriSpeech samples and 0.75M BrainOmni samples, totaling 49.24M unique samples—an increase of only 2.14%. Thus, TimeMoE-only and multi-teacher distillation can be viewed as a comparison under approximately equal data budgets. Despite this small increase in data volume, Multi-teacher STEP outperforms TimeMoE-only STEP on six of seven tasks, as shown in Figure 4 of the main paper. This suggests that the gains are not simply a consequence of a large increase in pretraining data volume and is consistent with the benefit of complementary teacher supervision.

Q3bb. Choice of BrainOmni as the Neural-Signal Teacher:

We selected BrainOmni because it jointly models EEG and MEG, with pretraining on approximately 2,000 hours of EEG and 656 hours of MEG. In contrast, CBraMod is pretrained on EEG from TUEG. BrainOmni therefore offers broader neural-signal coverage as a distillation teacher. As noted in the paper, its two-stage tokenizer–foundation-model pipeline prevents straightforward end-to-end fine-tuning under our direct-transfer protocol, so it is excluded from Table 2. This does not prevent its use as a frozen teacher.

Q4. Architecture Comparisons and Model Capacity:

For baseline comparisons, we retain each model’s native input-processing pipeline to evaluate it in its intended configuration. Moirai provides a particularly relevant comparison: it also supports long sequences through patching, but uses predefined patch sizes rather than STEP’s learnable patching mechanism. Moirai has approximately 311M parameters, compared with STEP’s 125.47M. Thus, STEP’s advantage cannot be attributed to greater model capacity or simply to the ability to process long inputs.

In the from-scratch architecture comparison in Table 3, STEP outperforms Moirai on five of seven tasks, supporting the architectural advantages of STEP independently of pretraining.

We additionally compare multi-teacher STEP with existing foundation models by freezing all backbones and training only task-specific classification heads. As shown in Table R3-4, STEP achieves the best performance on three tasks and the second-best on three others, demonstrating the transferability of its pretrained representations without encoder adaptation.

Table R3-4: Macro-F1 % of foundation models with frozen backbones on downstream scientific time-series tasks.

GWOSCLEAVESSTEADMarmAudioSleepEDFTUABRadarCommAvg_Rank
Whisper-large-v367.0634.6897.969.4249.661.038.614.00
Spear-xlarge-speech-audio91.2141.8792.3377.8464.3773.4155.123.00
TimeMoE59.8780.5378.6541.3465.6573.9257.713.14
Moirai-1.1-R-large56.2564.7694.5341.948.4369.7942.794.29
CBraMod46.5714.8987.9616.6563.5674.313.554.86
Multi-Teacher STEP62.5070.7997.2292.8066.9076.9555.981.71

Fine-tuned results are reported in Table 2 for the foundation models and Figure 4 for STEP. Rather than repeat the per-task scores, Table R3-2 above reports the mean macro-F1 and mean per-task rank across all seven tasks. STEP achieves the highest mean macro-F1 and the best mean rank, demonstrating its overall advantage after task-specific fine-tuning.

STEP achieves the strongest overall performance across all three evaluation settings: training from scratch, frozen-encoder evaluation, and task-specific fine-tuning—highlighting its effective architecture, transferable pretrained representations, and adaptability across scientific tasks.

Reviewer_PrekOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents STEP, a Scientific Time Series Encoder Pretraining framework via cross domain distillation. It investigates the transferability of foundation models trained on various time-series signals and identifies their complementary strengths on scientific tasks. With the observations, STEP proposes learnable adaptive patching to handle varying sequence lengths and channel dimensions, a statistics compensation scheme with per-sample and per-channel normalization strategy to preserve signal-level information during normalization. The framework pretrains the encoder with multi-teacher distillation to integrate representations from different domains. Experiments on scientific time-series tasks demonstrate that STEP improves performance across different domains.

Critical strengths
  • This paper provides a comprehensive analysis of how foundation models pretrained on related time-series domains transfer to scientific tasks with different strengths. These observations motivated the design of STEP by combining knowledge from different domains with the multi-teacher cross-domain distillation approach.
  • The proposed framework achieves competitive results across domains on from-scratch architecture comparison. Multi-teacher distillation further shows improvement over random initialization. These results show that cross-domain distillation can be useful for scientific time series.
Critical weaknesses
  • The methodological novelty is limited as the components build on established techniques. The adaptive patching only depends on only the sequence length, which might limit its ability to adapt to the heterogeneous time series signals.
  • The comparisons with other architectures use considerably different model capacities and input processing. It makes it difficult to isolate the benefit of the proposed approach. In addition, the multi-teacher STEP is not compare against the fine-tuned foundation models shown in Table 2. The advantage of STEP over existing pre-trained models remains unclear.
Questions for authors
  1. Multi-teacher training involves larger training data volume and more optimization steps. How much of the gain can be attributed to the additional data and supervision?
  2. Are other tasks in addition to classification evaluated?
  3. Does STEP account for time-series with irregularly sampled time stamps?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Prek(1/2)

We thank the reviewer for the valuable comments and for acknowledging the insights and empirical results of our work. Our responses are as follows:

W1. Learned Adaptive Patching:

Although the stride predictor takes only sequence length as input, its MLP is optimized jointly with the encoder through data-dependent gradients, allowing it to learn the length-to-stride mapping from the data characteristics rather than follow a fixed rule. We also experimented with including the input signal in the MLP’s inputs, but found this less stable than the current design.

To illustrate this, we examine 100 examples each from GWOSC gravitational−wavedetectiongravitational-wave detection and MarmAudio (marmoset vocalization classification) using the models trained on the respective tasks. Despite similar mean input lengths, the mean learned strides differ, yielding different patch sizes and encoder token counts, as shown in Table R2-1 below. These results show that the patching strategies reflect task-dependent learning, rather than input length alone determining a predefined rule.

Table R2-1: Results of learned patching strategies on GWOSC and MarmAudio.

DatasetMean input lengthLearned stridePatch sizeEncoder tokens
GWOSC64,00080.18161399
MarmAudio64,800121.86244.5265

W2. Comparisons with Foundation Models:

For the baseline comparisons, we retain each model’s native input-processing pipeline to evaluate it in its intended configuration. Moirai provides a particularly relevant comparison: it also supports long sequences through patching and has approximately 311M parameters, compared with STEP’s 125.47M. Thus, STEP’s advantage over Moirai cannot be attributed to greater model capacity or simply to the ability to process long inputs.

In the from-scratch architecture comparison in Table 3, STEP outperforms Moirai on five of seven tasks, supporting the architectural advantages of STEP independently of pretraining.

We additionally compare multi-teacher STEP with existing foundation models by freezing all backbones and training only task-specific classification heads. As shown in Table R2-2, STEP achieves the best performance on three tasks and the second-best on three others, demonstrating the transferability of its pretrained representations without encoder adaptation.

Table R2-2: Macro-F1 % of foundation models with frozen backbones on downstream scientific time-series tasks.

GWOSCLEAVESSTEADMarmAudioSleepEDFTUABRadarCommAvg_Rank
Whisper-large-v367.0634.6897.969.4249.661.038.614.00
Spear-xlarge-speech-audio91.2141.8792.3377.8464.3773.4155.123.00
TimeMoE59.8780.5378.6541.3465.6573.9257.713.14
Moirai-1.1-R-large56.2564.7694.5341.948.4369.7942.794.29
CBraMod46.5714.8987.9616.6563.5674.313.554.86
Multi-Teacher STEP62.5070.7997.2292.8066.9076.9555.981.71

Fine-tuned results are reported in Table 2 for the foundation models and Figure 4 for STEP. Rather than repeat the per-task scores, Table R2-3 below summarizes the mean macro-F1 and mean per-task rank across all seven tasks. STEP achieves the highest mean macro-F1 and the best mean rank, demonstrating its overall advantage after task-specific fine-tuning.

Table R2-3: Overall performance after task-specific fine-tuning.

Whisper-large-v3Spear-xlarge-speech-audioTimeMoE-200MMoirai-1.1-R-largeCBraModSTEP(Multi)
Avg-F1 ↑79.4578.2279.2482.0575.8585.03
Avg-Rank ↓3.1443.143.574.292.86

STEP achieves the strongest overall performance across all three evaluation settings: training from scratch, frozen-encoder evaluation, and task-specific fine-tuning—highlighting its effective architecture, transferable pretrained representations, and adaptability across scientific tasks.

(The rest of your comments will be addressed in Response 2/2).

AuthorsOpenReview ↗

Response to Reviewer Prek(2/2)

Q1. Data Volume and Multi-Teacher Gains:

The TimeMoE-only distillation uses 48.21M samples from Time-300B. Multi-teacher distillation uses the same Time-300B samples, supplemented with 0.28M LibriSpeech samples and 0.75M BrainOmni samples, totaling 49.24M unique samples—an increase of only 2.14%. Thus, TimeMoE-only and multi-teacher distillation can be viewed as a comparison under approximately equal data budgets. Despite this small increase in data volume, Multi-teacher STEP outperforms TimeMoE-only STEP on six of seven tasks, as shown in Figure 4 of the main paper. This suggests that the gains are not simply a consequence of a large increase in pretraining data volume and is consistent with the benefit of complementary teacher supervision.

Q2 & Q3. Task Scope and Irregular Sampling:

STEP is designed for scientific time-series classification and evaluated on seven classification tasks. We prioritize classification because many scientific phenomena are difficult to forecast. Forecasting is therefore outside the scope of our evaluation. We will clarify this scope in the revised manuscript and explore other task types in future work.

We thank the reviewer for highlighting irregular sampling. STEP learns temporal patterns directly from the observed sequence, without explicitly modeling sampling intervals or using timestamps as inputs. We will clarify this design in the revised paper.

Thank you for the additional experiments and clarifications. The frozen-encoder and fine-tuning results strengthen the evidence for STEP’s usefulness, and the Moirai comparison partly addresses my concern about model capacity. The scope and limitations are now clearer. My main concerns remain: patching depends only on length within a fixed model, and differences in capacity, input processing, and optimization make it difficult to isolate the architectural benefits. The small increase in unique samples also does not rule out gains from additional training or greater data diversity rather than complementary teachers.

The response strengthens the empirical case. However, given the limited methodological novelty and remaining experimental concerns, I retain my original scores.

AuthorsOpenReview ↗

Response to Reviewer Prek's remaining concerns

Thank you for acknowledging that our rebuttal has addressed some of your concerns. We would like to clarify the following points:

  1. Our stride predictor uses a learnable MLP to learn the length-to-stride mapping from the characteristics of the training data rather than follow a fixed rule. We also experimented with including the input signal in the MLP’s inputs, but found this less stable than the current design.
  2. For baseline comparisons, we retain each model’s native capacity, input processing, and optimization settings. We highlight Moirai as a relevant comparison because it has greater capacity and a more similar input-processing mechanism. To isolate the architectural benefits of individual components, we also provide ablation experiments in Section 6.2.
  3. Both TimeMoE-only and multi-teacher distillation train the same student architecture with the same number of parameters under comparable amount of training data, thus offering setting of comparable training budgets. This suggests that the gains come from complementary teachers, not from additional training or greater data diversity.

We appreciate your suggestions and hope our response fully resolves your concern.

Reviewer_HkCkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

STEP combines learnable patching, per-channel normalization with statistics compensation, and distillation from audio, general time-series, and neural-signal foundation models. The paper first compares the transfer performance of existing models, then evaluates the STEP architecture and its distilled initializations on seven scientific classification tasks. Multi-teacher distillation improves over random initialization across the task suite, although individual teachers remain better on some tasks.

Critical strengths
  1. Scientific time series are sparse and highly heterogeneous, which makes unified representation learning difficult. Studying the transferability and complementarity of foundation models from related domains is a reasonable way to approach this problem.

  2. The seven tasks cover substantial differences in sequence length, sampling frequency, and channel count, providing a diverse evaluation of the proposed framework.

  3. Separating the transfer study, architecture comparison, and distillation experiments makes the paper easy to follow, and the component ablations help explain the observed gains.

Critical weaknesses
  1. The gains after task-specific fine-tuning are encouraging, but it is unclear how much comes from reusable pretrained features and how much depends on adapting the encoder to each task. This distinction matters for the claim of a unified scientific representation.

  2. The teacher comparison also changes the data: each teacher supervises its own domain, so adding teachers broadens the student's pretraining data. The current comparisons do not establish whether the gains require complementary teacher knowledge or whether broader data exposure would be enough.

Questions for authors
  1. How much of STEP's advantage remains when the encoder is frozen, especially on a domain excluded from distillation?
  2. Does multi-teacher distillation still help when the student sees the same pretraining data and receives a comparable training budget?
3 comments
AuthorsOpenReview ↗

Response to Reviewer HkCk

We thank the reviewer for the valuable comments and for acknowledging the motivation and experimental rigor of our work. Our responses are as follows:

W1&Q1. Transferability of Frozen STEP:

We additionally evaluate all foundation models with their backbones frozen, training only task-specific classification heads. As shown in Table R1-1, STEP ranks first on three tasks and second on three others. The evaluation includes astronomy (GWOSC and LEAVES), seismology (STEAD), and radar (RadarComm), domains absent from distillation, providing evidence that STEP’s pretrained representations transfer to unseen domains.

Table R1-1: Macro-F1 % of foundation models with frozen backbones on downstream scientific time-series tasks.

GWOSCLEAVESSTEADMarmAudioSleepEDFTUABRadarCommAvg_Rank
Whisper-large-v367.0634.6897.969.4249.661.038.614.00
Spear-xlarge-speech-audio91.2141.8792.3377.8464.3773.4155.123.00
TimeMoE59.8780.5378.6541.3465.6573.9257.713.14
Moirai-1.1-R-large56.2564.7694.5341.948.4369.7942.794.29
CBraMod46.5714.8987.9616.6563.5674.313.554.86
Multi-Teacher STEP62.5070.7997.2292.8066.9076.9555.981.71

W2&Q2. Data Volume and Multi-Teacher Gains:

The TimeMoE-only distillation uses 48.21M samples from Time-300B. Multi-teacher distillation uses the same Time-300B samples, supplemented with 0.28M LibriSpeech samples and 0.75M BrainOmni samples, totaling 49.24M unique samples—an increase of only 2.14%. Thus, TimeMoE-only and multi-teacher distillation can be viewed as a comparison under approximately equal data budgets. Despite this small increase in data volume, Multi-teacher STEP outperforms TimeMoE-only STEP on six of seven tasks, as shown in Figure 4 of the main paper. This suggests that the gains are not simply a consequence of a large increase in pretraining data volume and is consistent with the benefit of complementary teacher supervision.

Thank you for your responses and additional experiments, which address some of my concerns. However, the response compares only sample counts and does not address my concern about whether the multi-teacher and single-teacher settings used a comparable training budget. I will keep my score.

AuthorsOpenReview ↗

Response to Reviewer HkCk's remaining concerns

Thank you for acknowledging that our rebuttal has addressed some of your concerns. We would like to clarify that the amount of training samples is directly related to the training budget. Both TimeMoE-only and multi-teacher distillation train the same student architecture with the same number of parameters under comparable amount of training data, thus offering setting of comparable training budgets. We appreciate your suggestions and hope our response fully resolves your concern.

Comments on the paper

1 comment
Area_Chair_kgQ2OpenReview ↗

Reminder: Please Participate Actively in the Discussion

Dear Reviewers,

Please actively participate in the discussion by responding to the authors’ replies and engaging with your fellow reviewers. This discussion is essential for clarifying concerns, resolving differences, and reaching a well-supported consensus on the final recommendation.

Thank you for your time and valuable contributions.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.