acceptodds
Under review as a conference paper at ICLR 2027

Byte-Pair Patching: Controllable Dynamic Tokenization for Time-Series Foundation Models

Abstract

Compression of input data, usually handled at the tokenization stage, is crucial for scaling time-series foundation models without incurring prohibitive computational costs. While static patch-based compression matches the performance of expensive point-wise tokenization, recent works have shown the scope for adaptive techniques that vary the patch size based on statistical properties of input data like entropy or frequency. To this end, inspired by Byte-Pair Encoding (BPE) for language, authors in Götz et al. (2025) expand an initial vocabulary of discretized numerical values by fusing them into motifs based on the frequency of occurrence in the training data. This method presents promising improvements in performance but only offers modest improvements to compression rates. In this paper, we introduce Byte-Pair Patching (BPP), a novel tokenizer that overcomes this limitation by applying BPE on fixed-size patches rather than point-wise observations, thus guaranteeing higher compression rates by design. Further, BPP also enables test-time control over compression, allowing a single pre-trained checkpoint to operate at different levels of input compression. Across GIFT-Eval, and 20 commonly used datasets, BPP provides up to increased compression versus point-wise tokenization and remains competitive relative to vanilla BPE and fixed-size patching baselines while improving inference latency by up to compared to the latter.

Primary area:
learning on time series and dynamical systems
Keywords:
Time-Series Forecasting, Time-Series Foundation Models, Dynamic Tokenization, Byte-Pair Encoding, Time-Series Motif Discovery, Sequence Compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_147QOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Byte-Pair Patching (BPP), a controllable dynamic tokenizer for time-series foundation models that combines fixed-size patching with Byte-Pair Encoding (BPE). In the discovery stage, normalized time series are divided into non-overlapping patches, assigned discrete identifiers using K-means, and merged into variable-length motifs through BPE. In the representation stage, these identifiers determine only motif boundaries, while fixed-dimensional token embeddings are constructed from the corresponding continuous observations using patch projection, intra-motif self-attention, pooling, and cross-attention. An occurrence-count threshold controls which merge rules are applied at inference, allowing a single trained checkpoint to operate at different compression levels without retraining. Experiments on GIFT-Eval and fixed-window forecasting across 20 datasets show competitive forecasting accuracy relative to BPE and fixed-patching baselines, with compression reaching 18.54× relative to point-wise tokenization on GIFT-Eval and model-side inference latency reduced by up to 25% compared with fixed patching of size 16, excluding tokenization time.

Critical strengths
  1. Addressing the token count in time series holds clear practical significance. Since this length directly impacts computational costs, applying necessary compression is highly justified.
  2. By supporting inference-time compression control without retraining, this method significantly reduces the training cost for practical deployment.
Critical weaknesses
  1. The absence of comparisons with stronger TSFMs, such as Sundial [1] and Chronos-2 [2], restricts a comprehensive assessment of the method's competitive performance.
  2. The existing ablations examine representation stage design choices, continuous versus quantized inputs, and compression thresholds, but do not isolate the contribution of BPE-derived motif boundaries under an otherwise identical representation stage.

[1] Sundial: A Family of Highly Capable Time Series Foundation Models

[2] Chronos-2: From Univariate to Universal Forecasting

Questions for authors
  1. Would it be possible to compare the proposed method with stronger TSFMs like Sundial and Chronos-2, or alternatively report the MASE metric to facilitate a unified evaluation on GIFT-Eval?
  2. Could the authors compare BPE-derived motif boundaries with fixed-length grouping of the same base patches while keeping the representation stage identical?
  3. Is it feasible to transfer BPP to other TSFM architectures and evaluate the performance by either full retraining on the identical pre-training corpus or applying a direct fine-tuning approach?
  4. Could the authors report the inference time comparison with the tokenization process included?
5 comments
AuthorsOpenReview ↗

Addressing reviewer 147Q concerns

We thank the reviewer for reviewing our manuscript. We are grateful that the reviewer finds our main task practically significant and that BPP significantly reduces the training cost of practical deployment. Here are our responses to the reviewer’s concerns:

Weaknesses

  1. Our original experiments use a shared backbone and Chronos-based training pipeline to isolate the effect of tokenization. Comparisons across different architectures can provide useful context, but they are very difficult to use to draw meaningful conclusions about the tokenizer’s contribution, since they use different pretraining corpora, different data preprocessing/normalization pipelines, and none release the exact pretraining code and data behind their public checkpoints.

    Nevertheless, we do acknowledge that evaluation with another TSFM would strengthen our results; therefore, we integrated BPP into Sundial-base and fine-tuned both the modified model and the original model under the same protocol. BPP reduces the average input token count from 28.2128.21 to 20.9720.97, a reduction of more than 2525%, while MASE changes from 0.85370.8537 to 0.86490.8649 and MSE from 0.65080.6508 to 0.66750.6675. We kindly refer the reviewer to Section 2 of our main comment for further discussion and subsection 4.6 of the revised manuscript for the complete results. We hope this additional evaluation addresses the reviewer’s concern.

  2. To emphasize BPE’s contribution, we added BPP-nobpe, an ablation that removes BPE while keeping the representation module and training setup unchanged. BPP-n achieves lower relative MSE than BPP-nobpe at both scales, improving from 0.70700.7070 to 0.69530.6953 at the tiny scale and from 0.67830.6783 to 0.67250.6725 at the small scale, while increasing compression from 8×8\times to 15.10×15.10\times. These results show that BPE merging improves the accuracy and compression trade-off. Beyond these gains, BPE also enables test-time compression control in combination with our representation module, a capability lost when BPE is removed. The results, added to Table 2 under BPP-noBPE, are highlighted in blue and summarized in the table below:

TokenizerAvg. Compression (×)Tiny Params. (M)Tiny Rel. MSETiny Rel. MAESmall Params. (M)Small Rel. MSESmall Rel. MAE
BPP-nobpe8.003.680.70700.915120.250.67830.9071
BPP-n15.103.680.69530.911620.250.67250.8965
BPP-e18.543.680.70960.929620.250.69590.9195

Questions

  1. We kindly ask the reviewer to refer to our response to Weakness 1.
  2. We kindly refer the reviewer to our response to Weakness 2, where we discuss this question in detail.
  3. Thank you for raising this question. Yes, BPP can be transferred to another TSFM architecture through either training from scratch or fine-tuning. Our main experiments demonstrate training from scratch using the Chronos pipeline. To test transfer to an existing pretrained model, we additionally replaced Sundial-base’s tokenizer with BPP and fine-tuned it on the Chronos corpus. BPP reduces its average input token count by more than 2525%, with only a small increase in forecasting error, supporting the feasibility of this approach. We kindly refer the reviewer to Section 2 of our main comment for the setup and results.
  4. We originally reported model-side latency because our BPE implementation incurred Python overhead from processing each series separately. We have since optimized the implementation to reduce this overhead without changing the tokenization procedure. As requested, we now measure latency with tokenization included under the same setting as the original experiment. Tokenization takes 0.0120.012 ms per sample for BPP, compared with 0.0060.006 ms for fixed patching. Including both tokenization and model computation, BPP-n and BPP-e remain 1010% and 1717% faster than P16, respectively. We will include these end-to-end measurements in the final version of the paper.

We hope this discussion and the additional experiments address the reviewer’s concerns about our work.

Thank you for the thoughtful responses and the effort put into the additional experiments. These help clarify the method's benefits. I have three follow-up questions:

  1. How is MASE computed and aggregated in Table 3? Does it follow the official GIFT-Eval leaderboard protocol, using the geometric mean of MASE normalized by the seasonal-naive baseline?
  2. What is the end-to-end inference speedup on Sundial-base with BPP, including tokenization?
  3. On Sundial, the performance degradation from adding BPP seems noticeably larger than the gains reported in the BPE ablations. Although these are different settings, could you clarify whether the actual inference savings justify this trade-off?
AuthorsOpenReview ↗

Responding to reviewer 147Q follow-up questions

We thank the reviewer for their response and are pleased that our clarifications and experiments helped clarify the benefits of our method. Below, we answer the reviewer’s follow-up questions.

  1. Yes, we follow exactly the official GIFT-Eval leaderboard protocol. For all configurations, we compute MASE using the official splits, prediction horizons, and seasonalities, then divide it by the corresponding MASE of the Seasonal Naive baseline from the GIFT-Eval results repository. We report the geometric mean of these ratios.

  2. We measured Sundial’s end-to-end inference latency with a batch size of 1024, following the Sundial evaluation notebook in the GIFT-Eval repository. Compared with Sundial’s original P16 tokenizer, BPP with base patch size 16 reduces total latency by 11.5% and 17.0% at kmink_{min} values of 10,00010,000 and 5,0005,000, respectively. Sundial’s flow-matching head accounts for a substantial portion of inference time and takes similar time across all three models. If we exclude this head’s timing, the latency gains are approximately 21% and 31%, respectively.

  3. Thank you for raising this point. In the BPE ablation, both models are trained under identical conditions, with only the BPE module removed. BPP-n achieves superior accuracy while only using about half as many tokens. Although the accuracy improvement is modest, it comes alongside substantially greater compression and test-time compression control. These findings support our claim that BPE is necessary for the combination of accuracy, adaptive compression, and test-time flexibility provided by our design.

    In the Sundial experiment, BPP is at a disadvantage. Specifically, the pretrained Sundial backbone uses the TimeBench dataset for training with a 16-point fixed-size patching tokenizer at the input layer. We replace this patcher with our BPP tokenizer and fine-tune using the Chronos corpus to align the tokenizer and the backbone. The “Original” baseline in Table 3 of the most recent draft simply tunes the existing backbone and static patching tokenizer on the Chronos corpus. We hypothesize that this mismatch in training corpora and the limited fine-tuning budget result in the accuracy gap.

    We resort to this setup only because the Sundial training corpus is not publicly available: Indeed, training a BPP-fused Sundial backbone from scratch with this dataset would be the fairest baseline to compare against the numbers reported by Sundial. Despite this constraint, the adapted BPP-Sundial variant with kmin=10,000k_{min}=10,000 uses 21% fewer tokens with a modest relative MASE increase of about 0.9%, from 0.854 to 0.862. We will sharpen the text around Table 3 to clarify this.

Reviewer_BZsxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a dynamic tokenization scheme inspired from Götz et. al's paper on BPE for time series critiquing that it doesnt offer efficiency boosts. They propose simple boundary discovery and token ID learning scheme for the time series byte-pair patching. They evaluate the approach using T5-style encoders on 10,000 random samples from Chronos corpus. They want to show competitve performance and efficiency (but the results are not supportive of this hypothesis based on my reading).

Critical strengths
  • Motivation is strong. Efficiency in TSFMs using long horizon context is do suffer from latency issues.
  • Paper is clearly written.
Critical weaknesses
  • Limited Evaluation -- baselines and compression techniques are not covered comprehensively. Several classic data compression technques form data miniing community like SAX, VQ-VAE are missing.

  • The results are not supporting of the claims. Overall the pareto-optimality of BPP in inference latency and MSE is not demostrated convincingly.

Questions for authors
  • Will you have to retrain the TSFMs to integrate BPP?
  • Is there a Training dataset bias? How sensitive are the learned codebook to training corpus? Table 4 in Appendix shows signifiant variance. Please dicuss.
  • For Table 1 whats the base model? BPP performance does not seem statistically not significant. Please provide paired-tests or other measures to demonstrate its performance superiority as claimed in the paper.
  • Please compare and show compression efficiency with classic data compression like SAX, TVQ-VAE, others (referred below).

SAX: Jessica Lin, Eamonn Keogh, Stefano Lonardi, and Bill Chiu. “A Symbolic Representation of Time Series, with Implications for Streaming Algorithms.” ACM SIGMOD Workshop on Research Issues in Data Mining and Knowledge Discovery, 2003. Classical dimensionality reduction and symbolic representation of time series

TimeVQVAE: Daesoo Lee, Sara Malacarne, and Erlend Aune. “Vector Quantized Time Series Generation with a Bidirectional Prior Model.” AISTATS, PMLR 206:7665–7693, 2023.

Original VQ-VAE: Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. “Neural Discrete Representation Learning.” NeurIPS, 2017. Foundational reference for learned discrete representations using vector quantization.

4 comments
AuthorsOpenReview ↗

Addressing reviewer BZsx concerns

We would like to thank the reviewer for reading our manuscript, and we’re delighted that the reviewer finds our paper clearly written with strong motivation. Here is our response to the reviewer’s concerns:

Weaknesses

  1. We thank the reviewer for this suggestion. We have added TOTEM, a VQ-VAE-based time-series tokenizer, as a baseline, with results reported in the revised Table 1. BPP-n achieves lower MSE and MAE than both TOTEM variants at both model scales. At the small scale, its relative MSE is 0.67250.6725, compared with 0.69630.6963 for TOTEM-P4 and 0.71850.7185 for TOTEM-P16, while providing 15.10×15.10\times compression versus 4×4\times and 16×16\times, respectively. BPP-e achieves 18.54×18.54\times compression, offering similar accuracy to TOTEM-P4 at over 4.6×4.6\times its compression and outperforming TOTEM-P16 with approximately 1616% greater compression. We have also expanded our evaluation with additional experiments described in Section 2 of our main comment. Regarding SAX, we appreciate its relevance as an established symbolic representation method. Based on our understanding, SAX averages observations within fixed segments and discretizes the resulting means, discarding within-segment variations. Our fixed-patching baselines instead learn embeddings directly from all observations within each patch. Since our objective is efficient and adaptive tokenization with competitive forecasting accuracy and not compression alone, we consider learned fixed patching, adaptive tokenizers, and VQ-VAE-based tokenization more directly aligned comparisons for this study.

  2. We understand the reviewer’s concern that the forecasting accuracy differences relative to the strongest fixed-patching baseline are modest. However, our contribution is an adaptive tokenizer with test-time compression control, rather than significant forecasting gains. BPP-e provides 18.54×18.54\times compression compared with 16×16\times for P16, while maintaining competitive accuracy. In our model-side latency experiment, this translates into a reduction of up to 2525% relative to P16. BPP also allows a single checkpoint to operate at different compression levels without retraining, a capability not offered by the baselines in the evaluations. These compression gains and test-time flexibility, rather than accuracy improvements alone, are the main benefits of our method. For further details on this, please kindly check Section 1 of our main comment.

Questions

  1. Replacing an existing model’s tokenizer with BPP requires fine-tuning or retraining to adapt the model to the new input representations. In our main experiments, we train the models from scratch, while in the newly added Sundial experiments (Section 2 of the main comments), we fine-tune the pretrained model. Once adapted to BPP, the model can operate at different compression levels without further training.

  2. To clarify, we train the tokenizer on 100,000100,000 randomly sampled series from the Chronos pretraining corpus, which covers a wide range of time-series domains. This diversity provides a broad set of recurring patterns for learning the BPE vocabulary. Using the original continuous values for the final representations further reduces reliance on the patterns captured by the learned codebook, since discrete IDs determine only motif segmentations rather than replacing the signal values. Regarding the reviewer’s concern about variability, Table 5 reports 0.27680.2768 ± 0.00210.0021 for BPP-n and 0.27780.2778 ± 0.00250.0025 for BPP-e, compared with 0.27740.2774 ± 0.00200.0020 for P16. These standard deviations are below 11% of the respective means, indicating small aggregate variability comparable to fixed patching. Although variability differs across individual datasets, it is not consistently higher for BPP. These experiments vary the forecasting-model training seed while keeping the tokenizer fixed, so they reflect model-training variability rather than codebook sensitivity.

  3. In Table 1, all methods use a T5 encoder followed by an MLP regression head, with tiny, small, and base model scales. At each scale, the backbone architecture is shared across methods; only the tokenizer and its corresponding input layer differ. Regarding the concern about forecasting accuracy, we would like to clarify that we do not claim significant forecasting advantage over the baselines. Our claim is competitive accuracy alongside greater compression and efficiency, and test-time compression control. We kindly ask the reviewer to check the response to Weakness 2 for a more detailed discussion.

  4. We kindly ask the reviewer to refer to our response to Weakness 1.

We hope these clarifications and additional experiments address the reviewer’s concerns and clarify the main benefits of our method.

Thank you for your response and for running TOTEM baseline. But I am not convinced that simple data compression methods are unsuitable tokenizer. If its possible to use some non-learnable simple method since your goal is efficiency, it would position the technical contribution better.

I am inclined to maintain my score.

AuthorsOpenReview ↗

Addressing reviewer BZsx concerns

We thank the reviewer for their response and for clarifying their concern. We should clarify that our evaluations already include non-learned compression methods. Besides fixed patching at several patch sizes, TimeSqueeze uses a simple, non-learned, signal-dependent rule to determine variable-length patch boundaries. Its compression decisions are based on local signal differences and power rather than a learned vocabulary. Our results demonstrate BPP’s competitive accuracy, compression benefits, and test-time flexibility compared with these baselines. We hope these comparisons address the reviewer’s concern regarding simple compression techniques. However, if the reviewer has another method in mind, we would be happy to conduct additional experiments.

Dear Reviewer,

Thank you again for your time and constructive feedback. We hope you have had a chance to review our clarification to your response , and would appreciate it if you could let us know of any remaining concerns while there is still time for us to respond before the discussion period closes.

Reviewer_yjELOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes a tokenization strategy for time-series (TS) that combines fixed-size patching with adaptive tokenization based on motif discovery. The TS is first split into fixed-size patches. Similar patches are then clustered, and recurring sequences of cluster tokens are merged into variable-length motifs, in the style of BPE. The authors compare this tokenizer with several alternatives by training the same model on each: a T5 encoder trained from scratch with an MLP regression head for forecasting. Evaluation uses the GIFT-Eval benchmark. The paper reports better compression rates and lower inference latency than the other tokenizers, with comparable forecasting accuracy.

Critical strengths
  • The proposed tokenization strategy clearly improves compression rate and inference latency without hurting forecasting accuracy, which is a very practical benefit.
Critical weaknesses
  • The technical novelty with respect to Götz et al. (2025) is limited. The main difference is applying first a standard fixed patching and a clustering step before BPE. Furthermore, this brings back the constraint of fixed patch, that BPE is supposed to remove with variable-length motifs. In the proposed tokenizer, P must also be fixed in advance, and we again need to assume that L is divisible by P. The paper also never explicitly states which value of P is used for the proposed approach.
  • The scope of the experiments is very narrow. The different tokenization methods are compared only on the T5 encoders trained from scratch with an MLP forecasting head. The tokenizer is never tested inside an existing time-series foundation model (TSFM). The only task is forecasting, and only univariate time-series are used. As a result, the experiments do not show whether the efficiency gains hold for the settings that motivate the work (TSFMs).
Questions for authors
  1. What value of P is used for the proposed tokenizer, and how sensitive are the results to it?
  2. Can the tokenizer be plugged into an existing TSFM, or used to pretrain one, to show that the gains hold at scale?
  3. Could you extend the evaluation beyond univariate forecasting, for example to multivariate series, classification or anomaly detection?
3 comments
AuthorsOpenReview ↗

Addressing reviewer yjEL concerns

We thank the reviewer for reviewing our manuscript and finding our work practically beneficial. Your concerns are addressed below:

Weaknesses

  1. Thank you for raising this point. We acknowledge that the distinction from prior work could have been explained more clearly. However, we argue that BPP is not merely a naïve combination of patching and BPE. We use K-means IDs to enable BPE-based boundary discovery, but preserve the continuous values when constructing tokens, and use a variable-length motif encoder to represent motifs. Even if we ablate the patching component, it is still systematically different from ordinary BPE (Gotz et al. 2025) in the sense that ours do not entail loss of granularity from discretization. Our ablations demonstrate the importance of each of these design components. Together, these components also enable test-time compression control, which, to the best of our knowledge, is not provided by previous BPE-based tokenizers. Section 1 of our main comment discusses these distinctions and the supporting experiments in detail. Our contribution is therefore the complete tokenizer pipeline of BPP and the capabilities it enables, rather than the individual components alone. We also appreciate the reviewer’s concern about retaining limitations of fixed patching. Fixed patching requires choosing between large patches that may compress information too aggressively and smaller patches that incur higher computational cost. While we agree that BPP retains a minimum granularity determined by P, BPP starts from a moderate base patch size and adaptively merges patches according to patterns in each input. Although P remains fixed after training, the final token lengths and compression are input-dependent, and compression can be adjusted at inference. Thus, P sets the minimum token length rather than fixing the final segmentation.

  2. We would like to clarify that our original experiments already follow a time-series foundation model pipeline. All models are trained on the full Chronos pretraining corpus using the Chronos training pipeline and evaluated zero-shot on GIFT-Eval without dataset-specific fine-tuning. For horizons beyond 64 steps, predictions are generated autoregressively in 64-step blocks. Using a shared architecture and training protocol adapted from Chronos allows us to isolate the effects of tokenization. Nevertheless, we agree that testing BPP within another established TSFM would strengthen the evaluation. Following the reviewers’ suggestions, we integrated BPP into Sundial-base through tokenizer replacement and fine-tuning. Our results show that BPP reduces its average input token count by more than 2525%, with only a small increase in forecasting error. Section 2 of our main comment provides the setup and results. In addition, based on the reviewer’s suggestion, we have extended the evaluation to multivariate time-series classification on seven UEA datasets, with results reported in Appendix B.4. We fine-tune the pretrained T5-tiny models using the same protocol and average results over five seeds. Each channel is tokenized and encoded independently, and the resulting representations are combined for classification. In this experiment, BPP-n achieves 79.379.3% average accuracy, compared with 76.776.7% for P16, while using fewer tokens per channel, 37.737.7 versus 41.241.2. Compared with P8, it achieves slightly better average accuracy, 79.379.3% versus 78.978.9%, using less than half the tokens, 37.737.7 versus 82.082.0. BPP-e further reduces the token count to 31.531.5 while retaining 78.478.4% accuracy. These results provide evidence that BPP’s accuracy–compression benefits extend beyond the univariate forecasting setting of the main evaluations.

Questions

  1. We thank the reviewer for bringing this up. We use P=8 in our experiments. In the additional Sundial experiments reported in Section 2 of our main comment, the main observed effect of changing P from 8 to 16 is on compression. The larger patch size substantially reduces token counts, while forecasting accuracy remains similar, suggesting that accuracy is not particularly sensitive to this choice within the tested setting.
  2. We kindly ask the reviewer to refer to our response to Weakness 2 and Section 2 of our main comment.
  3. We kindly ask the reviewer to check our discussion in the Second Weakness regarding new experiments.

We hope the new experiments and clarifications help to address the reviewer’s concern and provide stronger evidence for BPP’s contribution.

I thank the authors for their detailed response and the additional experiments.

  • As I understand it, the main advantage over BPE-based tokenizers (Götz et al., 2025) is test-time compression control. However, it seems that the gains obtained by adjusting the compression ratio at inference appear marginal (cf. the results on Sundial in table 3).
  • Integrating BPP into Sundial is a valuable addition. However, without comparisons to other baselines, such as alternative tokenizers or simply increasing Sundial's fixed patch size, it is difficult to assess the actual improvement. I believe that going further in this direction, i.e., demonstrating strong transfer to existing TSFMs, would make the paper much stronger.

I am increasing my score to 2, as I see the potential of the approach transferred to TSFMs, but I remain unconvinced about the technical novelty compared to BPE.

AuthorsOpenReview ↗

Responding to reviewer yjEL follow-up concerns

We appreciate the reviewer's reconsideration and increased score. We prioritized the experiments with TOTEM and Sundial tokenizers due to the tight turnaround of the rebuttal period, and agree that other baselines would help strengthen our story.

We would like to add some more detail on the first point in the reviewer's response:

Test-time compression control is not BPP's only advantage over the method of Götz et al. By applying BPE over patches rather than individual observations, BPP strictly improves the compression efficiency offered by vanilla BPE, as we demonstrate in Table1: 15.10× for BPP-n and 18.54× for BPP-e, compared with 4.08× for point-wise BPE. Further, models trained from scratch using BPP also achieve better forecasting accuracy. Our ablations suggest that preserving continuous values contributes to this improvement: BPP constructs token representations from the original observations, whereas point-wise BPE represents motifs using discrete token IDs.

Additionally, we note that in the Sundial experiments, BPP (based on P=16) improves the inference time in comparison to the Original Sundial tokenizer by 11.5% even without the test-time increased vocab setting (k_min=10,000). With the test-time vocab expansion, this improvement increases to 17.0% (k_min=5000). These indicate non-trivial improvements that are not marginal in effect. We detail the potential reasons for the modest MSE underperformance of BPP in our response to Reviewer 147Q. We hypothesize that this occurs because BPP is at a disadvantage: the Sundial Backbone is pretrained using its fixed-size patching tokenizer. We replace this with BPP and fine-tune rather than training from scratch.

We would appreciate it if the reviewer could improve the score if these points address their concern.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Addressing the shared concerns of the reviewers

We thank all reviewers for their service and invaluable feedback. In this comment, we address the shared concerns across the reviews; their individual concerns are addressed in the replies to each of the reviews. We have also revised the manuscript, with changes highlighted in blue.

  1. Novelty and design of BPP. We would like to clarify that BPP is not simply a naïve combination of fixed patching and BPE. Its contribution lies in the complete pipeline: BPP first uses (K)-means IDs to apply BPE over patches, then constructs the final tokens from the raw continuous values, and finally uses a variable-length motif encoder to embed variable-length motifs.

    In fact, our ablations highlight the importance of our design choices.

    • In the revised Table 1, the use of quantized values instead of raw continuous inputs degrades relative MSE. For BPP-n, we record a decrease from 0.72560.7256 (BPP-n-q) to 0.67220.6722 (BPP-n), and from 0.75140.7514 (BPP-e-q) to 0.68390.6839 (BPP-e) for BPP-e.
    • Table 2 supports our choice of a dedicated variable-length motif encoder. Compared to natural alternatives, we observe better performance and robustness to compression using our design under both normal and high compression regimes. Under high compression, our encoder achieves a relative MSE of 0.68390.6839, compared with 0.70490.7049 for the padded MLP and 0.72910.7291 for max pooling.

    Additionally, our design also enables BPP’s test-time compression control. Owing to the fact that BPP constructs token representations directly from the continuous values rather than using a lookup embedding for each BPE token, additional merge rules can be activated after training without introducing new parameters or retraining the model. To the best of our knowledge, this capability has not been provided by previous time-series tokenizers.

  2. Evaluation and application to TSFMs. We would like to clarify that our original experiments already follow a TSFM setting. To isolate the effect of tokenization, we adapted the publicly available Chronos training pipeline, replacing its decoder with an MLP regression head while keeping its pretraining corpus and training setup. The forecasting models are trained on the full Chronos pretraining corpus and evaluated zero-shot on GIFT-Eval and the other datasets. For datasets with horizons longer than our 64-step output, we do auto regressive forecasting in 64-step blocks. Therefore, these experiments train and evaluate time-series foundation models rather than dataset-specific forecasting models. We have clarified this setup in the revised experiments section.

    However, we agree that evaluation with another established TSFM would strengthen our results. Following reviewers yjEL and 147Q’s suggestions, we integrated BPP into Sundial-base (128M parameters). For this, we replace the original tokenizer with our BPP module and fine-tune this BPP-integrated model on the Chronos corpus. As a control, we also fine-tuned Sundial with its original tokenizer on the same dataset. The results are reported in a newly added subsection 4.7. Even though Sundial was not originally trained with BPP or the Chronos dataset, our fine-tuning, which is for 50,00050,000 steps, enables adaptation of the model to the new tokenizer. In this experiment, BPP improves the average input token count of Sundial by more than 2525%, while suffering a small increase in forecasting error. This emphasizes the applicability of BPP as a technique and highlights the few-shot transferability of BPP to different TSFMs.

    In addition, following Reviewer BZsx’s suggestion, we have also added the VQ-VAE-based TOTEM tokenizer as a new baseline, with results highlighted in blue in Table 1. BPP-n achieves lower MSE and MAE than both TOTEM variants at both model scales. For example, at the small scale, its relative MSE is 0.67250.6725, compared with 0.69630.6963 for TOTEM-P4 and 0.71850.7185 for TOTEM-P16, while providing 15.10×15.10\times compression versus 4×4\times and 16×16\times, respectively. BPP-e further increases compression to 18.54×18.54\times, achieving similar accuracy to TOTEM-P4 at over 4.6×4.6\times its compression and outperforming TOTEM-P16 with approximately 1616% more compression.

We hope these clarifications and additional experiments address the reviewers’ shared concerns and provide stronger evidence for the benefits of our method.

The author has responded to everyone's comments. Please reply when it is convenient for you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.