acceptodds
Under review as a conference paper at ICLR 2027

ZEST: Zero-Shot End-to-End Time Series State Detection

Abstract

Time series state detection (TSSD) identifies state boundaries and recurring states in multivariate sequences. Deep self-supervised TSSD methods typically combine target-specific representation learning with clustering, hindering direct transfer to unseen sequences. Our analyses identify persistent conflicts in temporal-proxy supervision and limited transfer gains from enlarging the pretraining corpus under the tested fixed-budget settings. We introduce ZEST, a zero-shot, end-to-end framework that recasts TSSD as hierarchical structured generation, adapting a pretrained vision-language model to predict state structure from global sequence images. To build scalable supervision without manual annotation, we use forecast consistency to guide state-prototype construction from large-scale unlabeled real-world time series. A controllable scheduler then recombines these prototypes into multivariate sequences with exact state labels. To capture state relations across granularities, we construct hierarchical supervision for joint generation of leaf segmentations and state-merge trees. The adapted model provides a default leaf partition and multiple coarser candidates on unseen sequences without target labels, parameter updates, hyperparameter tuning, or clustering. Experiments on six benchmarks demonstrate strong zero-shot leaf performance, while a separate targetlabel-assisted study shows the utility of selecting among the predicted granularities.

Primary area:
learning on time series and dynamical systems
Keywords:
time series analysis, state detection, vision-language model

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_UNrhOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a framework named ZEST to deal with the time series state detection task. ZEST identifies state changes and recurring states in unseen time series by reading their plots with a fine-tuned vision-language model. It builds training examples from unlabeled real signals and predicts both a segmentation and a hierarchy for grouping similar states.

Critical strengths

This paper develop a framework that combines real waveform fragments with controlled state scheduling to generate labeled multivariate sequences without manual annotation. The proposed framework can predict boundaries and recurring state identities on unseen sequences.

Critical weaknesses
  1. The main concern is the paper has limited model novelty. The state detector uses an existing VLM, and a standard autoregressive loss. The main contribution lies on task related supervision construction and output formulation.

  2. The motivation is not clear. State detection is closely related to time-point classification. This has been studied by many previous methods, like TNC[1] and PLanTS[2]. And the paper mainly discusses limitations of contrastive methods that construct training pairs using temporal proximity, but ignores other contrastive methods use signal similarity or address false negatives, like SoftCLT[3] and PLanTS[2].

  3. For the experiments, Zest is trained on a large corpus with generated state labels, where baselines use different training setups. Comparing against other detectors trained on the same corpus or against representation learning methods adapted for time point classification may help the results more convincing

[1]Unsupervised representation learning for time series with temporal neighborhood coding.

[2]PLanTS: Periodicity-aware Latent-state Representation Learning for Multivariate Time Series.

[3]Soft contrastive learning for time series.

Questions for authors
  1. Beyond the supervision construction and output format, what is technically new in the state detector itself?

  2. What is the fundamental difference between the state detection task studied in this paper and time-point classification? Does ZEST handle any setting or capability that cannot be addressed by existing time-point classification methods?

  3. The motivation focuses on temporal-proximity-based contrastive learning, but how does it apply to methods such as SoftCLT and PLanTS that use signal similarity or address false negatives?

  4. Since ZEST is trained on generated sequences with task-specific state labels, while the baselines use different training setups, how to determine whether its gains come from the generated supervision or from the VLM-based structured prediction approach?

4 comments
AuthorsOpenReview ↗

[1/3] Response to W1 and Q1: Novelty and Technical Contributions

We thank the reviewer for the comments and questions on the novelty, motivation, and experimental evaluation of our work. Below, we clarify ZEST’s task setting and technical contributions, explain the rationale behind its design, and discuss the experimental evidence supporting our approach.


For Weakness 1 and Question 1, we recognize that the original manuscript may not have sufficiently explained our contributions, potentially giving the impression that ZEST simply fine-tunes an existing VLM.

In the following, we would like to clarify our novelty and technical contributions.

1. Novelty: data construction and task adaptation for TSSD

Our contributions are twofold: scalable supervision construction from unlabeled real-world time series and hierarchical structured generation for direct detection on unseen sequences. Using a VLM backbone and an autoregressive loss is not our main contribution.

We focus on constructing high-quality TSSD supervision at scale and adapting pretrained VLMs for state detection. This goal has a clear motivation. On the one hand, VLMs’ visual priors show promise for end-to-end state detection, but performance is limited without task adaptation (Table 1). On the other hand, the two-stage methods in Section 3 show limited transfer gains after pretraining and still require target-specific clustering. We therefore investigate how data construction and task adaptation can jointly enable zero-shot, end-to-end detection.

2. Technical contributions: addressing supervision acquisition and state structure modeling

The first challenge is to combine real-world dynamics with precise state supervision. Real time series often lack dense state labels, while purely synthetic signals may miss real-world diversity. Thus, we mine and expand reusable waveform prototypes from unlabeled real signals using prediction consistency (Section 4.1.1), then schedule multivariate states to control their differences, durations, and recurrence (Section 4.1.2). This preserves real-world patterns while determining state identities and boundaries, enabling TSSD supervision at scale (Section 4.1).

The second challenge is to adapt general visual capabilities for state structure prediction. TSSD requires accurate state identification while accommodating different state granularities. Thus, we develop hierarchical structured generation with joint supervision of segment boundaries, state identities, and a state merge tree. One prediction yields a default leaf-level segmentation and consistent coarser alternatives, without target-specific training or clustering (Section 4.2).

Together, these designs enable zero-shot, end-to-end TSSD with a deep learning model. Data construction ablations validate the supervision strategy, while comparisons using the same backbone and results on six benchmarks support the full adaptation framework (Sections 5.2 and 5.4). We have clarified our contributions and motivations in the Abstract, Introduction and Section 3.3, and explained in Sections 4.1–4.2 how our designs address these challenges.

AuthorsOpenReview ↗

[2/3] Response to W2 and Q2–Q3: Task Formulation and Motivation

Thank you for these comments. We address three points: (1) the relationship between time series representation learning, state detection, and time-point classification (W2 and Q2 Sentence 1); (2) ZEST’s objective and novelty (Q2 Sentence 2); and (3) how false-negative handling relates to our motivation (Q3).

1. Representation learning, state detection, and time-point classification

Time series representation learning maps time points, windows, or entire instances into representations for downstream tasks. TNC [8], PLanTS [9], and SoftCLT [10] are self-supervised contrastive representation learning methods. Downstream tasks still require a predictor or decoder.

Time series state detection (TSSD) is an established task [1–4] that identifies changes in the system states reflected in time series. Its results can be expressed as pointwise state labels. Representative deep learning methods, including Time2State [1] and E2USD [2], follow a two-stage pipeline: learning window representations through contrastive learning, then clustering them to obtain state labels. Contrastive representation learning is therefore an important technical foundation for these TSSD methods. We discuss these methods in Sections 1, 3, and 6.

Time-point classification describes an output format—assigning a class label to each time point—but does not by itself define a specific task. For example, anomaly detection [6–7] and state detection [1–4] can both produce pointwise labels but differ in semantics and objectives. The main classification experiments in PLanTS and SoftCLT are instance-level; PLanTS also analyzes state transitions through representation trajectories. TNC evaluates window-based hidden-state classification and provides a representation learning foundation for TSSD. Time2State analyzes and compares TNC and develops an objective tailored to state detection.

In summary, contrastive learning underpins these two-stage TSSD methods, and state detection is a specific task whose results can be expressed as pointwise state labels.

2. ZEST’s objective and novelty

Objective: ZEST aims to enable zero-shot, end-to-end TSSD: once trained at scale, it directly predicts state boundaries and identities for unseen sequences, without target labels, parameter updates, hyperparameter tuning, or target-side clustering.

Novelty: Existing TSSD methods [1–2] follow a “representation learning + clustering” pipeline. Their original pipelines do not support the zero-shot, end-to-end detection targeted by ZEST. Section 3 highlights three issues.

  • First, many false negatives persist even after filtering, impairing representation learning (Figure 2(a–b)).
  • Second, even with encoders pretrained using ground-truth state relations and frozen at inference, these methods do not consistently outperform their target-trained counterparts under the tested training budget (Figure 2(c–d)).
  • Finally, improving the encoder alone does not eliminate target-side clustering. State labeling still requires this separate step, so the pipeline remains two-stage rather than end-to-end.

3. How false-negative handling relates to our motivation

First, false-negative handling cannot eliminate the need for target-side fitting in two-stage TSSD, although it can improve representations. SoftCLT uses signal and temporal distances to assign soft pair weights, while PLanTS incorporates periodicity and signal similarity.

Second, our analysis already includes improvements to representation learning and false-negative handling. Specifically, Time2State analyzes limitations of TNC and introduces LSE-Loss to model intra- and inter-state relationships. E2USD further develops false negative cancellation contrastive learning to filter negatives based on similarity.

Third, Section 3 shows that false negatives persist after filtering (Figure 2(a–b)). Even pretraining encoders with ground-truth state relations from our generated corpus yields no consistent detection gains over target-specific training (Figure 2(c–d)). These results show that improving pair supervision alone does not ensure consistent detection transfer.

These findings motivate ZEST’s joint design of state supervision construction and direct state structure prediction, enabling detection without target-specific fitting.

We acknowledge the contributions of TNC, SoftCLT, and PLanTS and have added their citations to the manuscript.

AuthorsOpenReview ↗

[3/3] Response to W3 and Q4: Experimental Comparisons and Sources of Improvement

Thank you for these suggestions. We first discuss our contributions and experimental design (W3), then discuss the sources of performance improvement (Q4).

1. Contributions and experimental validation (W3)

[W3 Sentence 1] As discussed above, our goal is zero-shot, end-to-end TSSD without target-specific fitting. Our technical contributions are scalable supervision construction (data strategy) and adaptation for state detection (adaptation strategy). Our experiments therefore evaluate the two strategies. We adapt Qwen 3.5 2B to structured state prediction using our generated supervision. In Table 1, the resulting ZEST model outperforms both the original backbone and most baselines on unseen sequences. Together, these results validate the effectiveness of our technical contributions.

[W3 Sentence 2] Section 3 already examines your suggestion by pretraining the encoders of existing two-stage methods on our generated corpus. We construct positive and negative pairs directly from state labels, avoiding false negatives introduced by temporal-proxy pairing. Under the tested training budget, these methods do not consistently outperform their target-trained counterparts (Figure 2(c–d)).

In summary, the main experiments evaluate ZEST’s detection capability, while Section 3 examines how existing methods perform with the generated data and state supervision.

2. Sources of performance improvement (Q4)

Both the generated supervision and the structured prediction contribute to the final performance. The manuscript evaluates their roles through corresponding experiments, as detailed below.

For supervision construction, ZEST-leaf uses leaf-level segmentation without granularity selection, yet ranks first in 11 of 12 comparisons and outperforms the original Qwen 3.5 2B across all benchmarks (Section 5.2). This demonstrates the overall benefit of task adaptation with generated supervision. Section 5.4 further compares different data construction methods. Our prototype mining and expansion strategy outperforms random cropping, validating the supervision construction strategy itself.

For structured generation, ZEST-oracle’s gains over ZEST-leaf show that the generated merge trees contain better segmentations than the default leaf-level output (Section 5.2). Oracle selects granularity using ground-truth labels, showing the best performance attainable among these candidates. Section 5.3 further shows that selecting granularity with a few target labels improves detection at time points not used for selection, demonstrating the practical value of hierarchical outputs. Furthermore, Appendix E.5.1 compares different input representations. Image input achieves better detection performance or efficiency than the numerical encoder and text tokens, supporting the value of visual input in our task adaptation strategy.

Together, these experiments show that both supervision construction and task adaptation contribute to ZEST’s detection performance.


[1] Chengyu Wang, Kui Wu, Tongqing Zhou, and Zhiping Cai. 2023. Time2State: An Unsupervised Framework for Inferring the Latent States in Time Series Data. Proceedings of the ACM on Management of Data, 1(1), Article 17, 1–18.

[2] Zhichen Lai, Huan Li, Dalin Zhang, Yan Zhao, Weizhu Qian, and Christian S. Jensen. 2024. E2Usd: Efficient-yet-effective Unsupervised State Detection for Multivariate Time Series. Proceedings of the ACM Web Conference 2024, 3010–3021.

[3] Yasuko Matsubara, Yasushi Sakurai, and Christos Faloutsos. 2014. AutoPlait: Automatic Mining of Co-evolving Time Sequences. Proceedings of the ACM SIGMOD International Conference on Management of Data, 193–204.

[4] Arik Ermshaus, Patrick Schäfer, and Ulf Leser. 2025. CLaP – State Detection from Time Series. Proceedings of the VLDB Endowment, 19(1), 70–83.

[5] Arik Ermshaus, Patrick Schäfer, and Ulf Leser. 2023. ClaSP: Parameter-free Time Series Segmentation. Data Mining and Knowledge Discovery, 37, 1262–1300.

[6] Shreshth Tuli, Giuliano Casale, and Nicholas R. Jennings. 2022. TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data. Proceedings of the VLDB Endowment, 15(6), 1201–1214.

[7] Zahra Zamanzadeh Darban, Geoffrey I. Webb, Shirui Pan, Charu C. Aggarwal, and Mahsa Salehi. 2024. Deep Learning for Time Series Anomaly Detection: A Survey. ACM Computing Surveys, 57(1), Article 15, 1–42.

[8] Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. 2021. Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding. International Conference on Learning Representations (ICLR).

[9] Jia Wang, Xiao Wang, and Chi Zhang. 2025. PLanTS: Periodicity-aware Latent-state Representation Learning for Multivariate Time Series. arXiv preprint arXiv:2509.05478.

[10] Seunghan Lee, Taeyoung Park, and Kibok Lee. 2024. Soft Contrastive Learning for Time Series. International Conference on Learning Representations (ICLR).

AuthorsOpenReview ↗

Follow-up on Our Responses

We sincerely appreciate the time and effort you have devoted to reviewing our work. Your questions have helped us clarify the task setting and present our contributions and experimental evidence more clearly.

As the discussion period draws to a close, we would be grateful to hear whether our responses and revisions address your concerns, and would be happy to provide any further clarification.

Thank you again for your time and consideration!

Reviewer_RyXFOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

PC verification needed regarding desk rejection

There is no one-line spacing between any Figure/Table and its caption in this paper, and the paper thus does not comply with the ICLR format; verification by the PC appears necessary. Separately from this issue, however, I provide my comments and evaluation of the research content below.


This study proposes ZEST, a pipeline that allows TSSD to be learned in an end-to-end manner instead of following the conventional two-stage approach, along with a synthetic dataset generation method for this purpose. A prototype set is constructed from normalized waveforms, and a scheduler combines these prototypes to generate synthetic sequences and their ground truth together. A VLM is then trained to predict state segments and a merge tree, jointly addressing state discrimination and hierarchical structure in time series.

I think this study connects the problem analysis, model design, data construction, and experiments in a highly convincing way. In particular, I found it impressive that the authors do not stop at constructing a large synthetic dataset, but first examine the effects and limitations of expanding the data pool for two existing models and then connect these findings to the motivation for directly predicting state structure. Furthermore, I find the design of jointly predicting a merge tree interesting. My impression was that the necessary components are well arranged within a single end-to-end pipeline without unnecessary additions. Accordingly, I recommend strong acceptance.

Critical strengths

[S1] Model architecture and data generation grounded in a clear problem analysis

  • The Introduction explains the reasoning behind this study very clearly. It analyzes the limitations of existing approaches in two stages (¶2), explains the challenges faced in addressing them (¶3), and then connects these to how the challenges were overcome (¶4). The writing and structure are clear enough for readers who are even unfamiliar with TSSD to understand. Among the papers I have read recently, this Introduction stood out for its readability, and I personally really enjoyed reading it.
  • I also found it impressive that this reasoning leads to the use of a VLM, and that the authors construct and learn not only state segments but also a merge tree. I consider this an appealing approach because it goes beyond simply replacing the encoder of an existing pipeline with a large model and also designs what the model should predict.
  • I also consider the construction of the synthetic dataset for model training sound. It is interesting that the authors select some segments from real time series based on the consistency of Chronos predictions to construct a prototype set, and then use a scheduler to generate synthetic sequences and state ground truth together. They also use MOMENT to obtain similarities between prototypes or ground truth for the merge hierarchy. I find it appealing that existing TSFMs and waveform generation techniques are connected to their respective roles to construct a learning task for state discrimination and hierarchy prediction from unlabeled real data.
  • In other words, it is easy to understand why each component of the model and data generation pipeline is needed, and I consider the overall design to have few unnecessary elements.

[S2] Clear experimental design and interpretation of results

  • I found the experiments in §4.2 particularly impressive. The authors apply the synthetic dataset generated using the method in §5.1 to Time2State and E2USD, showing that the mismatch between temporal proxies and actual state relations persists even as the data pool grows. They also find that supervision using actual state relations does not consistently lead to improved downstream transfer. These results naturally lead to the argument that, under the evaluated conditions, supervision and the final prediction approach should be considered alongside expansion of the data pool.

  • This motivation leads to the choice of a VLM, and I wondered whether approaches such as multimodal TSFMs/TSLMs might also have been considered as options if research on them had already been active.

  • Typically, the authors could have first presented the performance of a model combining a synthetic dataset and a VLM and then limited their analysis to verifying the necessity of each component through ablations. However, this study first applies the data to existing models to analyze their limitations and connects the results to the motivation for directly predicting state structure. I think this ordering of experiments conveys the need for a VLM-based approach quite convincingly.

  • I also found it impressive that the comparisons are not limited to conventional TSSD methods but also include recent VLM/LLM approaches. Of course, as discussed below, the scope of interpretation may differ depending on whether target-specific training is performed. Nevertheless, I appreciate the breadth of the experiments in examining different approaches together and showing the extent of the performance improvement provided by the proposed pipeline.

Critical weaknesses

[W1] Additional information on training and evaluation costs

  • ZEST is designed to train for only 1 epoch on the training split of a corpus consisting of one million generated sequences. It is impressive that this training enables application to new time series without additional training, but reporting the total time and resources required for corpus generation and model training would help assess the practical costs more clearly.

  • Table 17 and Appendix E.6 provide latency on HAS, peak GPU memory during inference, and the stages included in or excluded from the measurements for each method. However, in addition to the current measurements, reporting the total processing time including rendering, corpus generation cost, training time, and peak memory during training would help evaluate practical utility.

  • Also, the measurements for Time2State and E2USD include target-specific training and clustering, whereas ZEST's pretraining is excluded. Distinguishing one-time preparation costs from costs incurred for each new time series would make the trade-offs of each approach easier to understand.

[W2] Further support for some comparison conditions and evaluation evidence

  • Including recent VLMs/LLMs as comparison methods is a strength. However, caution seems necessary when interpreting the results because ZEST is adapted to the state detection task using synthetic data, whereas the comparison models are evaluated without corresponding additional adaptation. The authors also explain that this comparison demonstrates the effect of the full task-adaptation recipe and does not isolate the contributions of individual components.

  • In addition, evaluation metrics other than ARI and further statistical validation would strengthen the evidence for performance. For example, it would be helpful to also provide results using WARI/SMS, proposed in "Toward Interpretable Evaluation Measures for Time Series Segmentation" (ICLR'25). Given the large reported gaps over the baselines, I expect the overall trends to hold even if the metrics are changed, but this should be confirmed through actual results.

  • Paired statistical significance analysis of the performance differences from the baselines would also provide clearer support for the superiority of this method.

Questions for authors

The following questions are intended to clarify the concerns above and better understand the model's design and practical costs.

[Q1] Could you provide additional information on the computational costs of data generation, model training, and the full inference process?

  • I have checked the training hardware in Table 14 and the HAS latency and peak inference memory in Appendix E.6. In addition, it would be helpful to provide the time and resources required to generate the synthetic corpus, the actual time needed for 1 epoch of training, and peak GPU memory during training. Furthermore, separately reporting raw-sequence-to-output latency including offline rendering and initial costs such as model loading would help clarify the costs of practical deployment.
  • Distinguishing costs incurred only once from those incurred whenever the model is applied to a new dataset or time series would make this even easier to understand.

[Q2] I would like to know why Chronos-2, MOMENT-1-large, and Qwen3.5-2B were chosen for their respective stages.

  • As I understand it, Chronos-2 is used to select segments for prototypes, MOMENT-1-large to represent similarities between waveforms, and Qwen3.5-2B to predict the final state segments and hierarchical structure.
  • What was the rationale for choosing each model for its role? I would like to know whether the choices were based on each model's characteristics, preliminary comparative experiments or computational cost, or other reasons.

[Q3] Could you provide additional WARI/SMS results and statistical validation?

  • I would like to know whether the performance improvements over existing baselines hold when evaluated using WARI/SMS, proposed in "Toward Interpretable Evaluation Measures for Time Series Segmentation" (ICLR'25).
  • Furthermore, in addition to bootstrap intervals, appropriate paired statistical analysis would help determine whether the performance differences are significant.
  • Given the current performance gaps, I do not expect the overall conclusions to change substantially, but confirming this through additional validation would make the findings more convincing.
5 comments
AuthorsOpenReview ↗

[1/3] Response to W1 and Q1: Computational costs

We are sincerely grateful to the reviewer for recognizing the strengths of our work and for the constructive feedback. Below, we address the reviewer’s concerns and questions point by point.


For Weakness 1 and Question 1, we report the costs of offline corpus preparation, one-time model adaptation, target-sequence processing, and startup.

1. Offline corpus preparation. We reran the complete construction pipeline, starting from real, unlabeled source time series and ending with training-ready sequences, labels, and images. The measurements cover Chronos-2-based segment scoring and selection, IAAFT augmentation, MOMENT-1-large embedding extraction, state scheduling, sequence generation, and image rendering. We used two AMD EPYC 9354 processors with 32 cores each and seven RTX 4090 GPUs, generating one million labeled sequences.

StageWall-clock time (hours)Peak host memory (GB)Peak GPU memory (GB)GPU-hours
Chronos-2 segment scoring and selection99.3020.7312.41653.80
IAAFT prototype augmentation0.4118.3400
MOMENT embedding extraction and prototype library construction3.1415.2015.2521.98
State scheduling and labeled sequence generation0.1914.6100
Image rendering and corpus assembly2.1767.100
Complete offline preparation pipeline105.2167.115.25675.78

Peak host memory measures the maximum combined memory usage of the relevant processes and their child processes. Peak GPU memory measures the maximum simultaneous sum of memory usage across all GPUs in use.

The complete pipeline took 105.21 hours, from initial source-data processing to completion of the training corpus. Most of this time was spent scoring and selecting segments from the large-scale BLAST corpus. Corpus preparation is a one-time cost.

2. One-time model adaptation. One epoch of LoRA fine-tuning for ZEST-2B took 13.50 hours on seven RTX 4090 GPUs, corresponding to 94.52 GPU-hours. The maximum peak memory usage on any single GPU was 13.32 GB, and the peak simultaneous total across all seven GPUs was 93.24 GB.

3. Deployment and target-sequence processing. We measured routine sequence processing and startup separately. Across all 404 evaluation sequences, ZEST-2B took an average of 4.110 seconds per sequence on a single RTX 4090:

StageAverage time
Plot rendering and input construction0.222 s/sequence
Preprocessing and data transfer0.024 s/sequence
Generation, output validation, and corrective regeneration when needed3.865 s/sequence
Total4.110 s/sequence

Timing begins with the raw sequence already loaded into memory and includes image rendering and the complete detection process. It excludes process startup and disk reads. Peak GPU memory usage was 4.297 GB.

To measure startup costs separately, we used one fixed sequence in three independently launched processes. Startup, including model loading, took an average of 8.85 seconds. The first request after startup, including file reading and sequence processing, took 9.68 seconds on average. The directly measured time from process launch to the first output averaged 18.52 seconds. These startup measurements are reported separately from the average processing time over the 404 sequences.

In our evaluation, Time2State and E2USD require no source-side model adaptation, but require fitting and clustering for each target sequence. ZEST incurs the one-time corpus preparation and adaptation costs reported above, after which the adapted model can be applied to new sequences.

AuthorsOpenReview ↗

[2/3] Response to W2 and Q3: Comparison Conditions and Additional Evaluation Evidence

Thank you for these constructive suggestions. We address the model comparison conditions and additional evaluation evidence below.

1. Clarifying the interpretation of the results. We agree that the results should be interpreted in light of each model’s task adaptation. First, the same-backbone comparison between ZEST and Qwen3.5-2B demonstrates the effectiveness of our complete task-adaptation recipe. Second, under our evaluation settings, Qwen3.5-2B without TSSD adaptation outperforms the time-series VLM/LLM models included in the comparison. This suggests that adaptation to other time-series analysis tasks does not necessarily yield better state-detection performance. We have clarified the adaptation settings and refined the corresponding analysis in Sections 5.1 and 5.2.

2. Additional evaluation with WARI and SMS. Following your suggestion, we conducted additional evaluations using WARI and SMS, proposed by Chavelli et al. [1]. We compare ZEST with Time2State and E2USD, two representative deep learning methods. The evaluation covers all six datasets: we first average sequence-level scores within each dataset and then take the equally weighted mean across datasets. For each method, all four metrics are computed from the same predictions on the original time axis. ZEST uses its default leaf-level output.

MethodARI ↑NMI ↑WARI ↑SMS ↑
ZEST-2B64.4070.6772.3873.77
Time2State51.9459.9259.4660.61
E2USD53.2257.6157.5765.46

All scores are multiplied by 100.

ZEST achieves the highest dataset-macro WARI and SMS among the three methods, showing that its performance advantage over Time2State and E2USD is retained under the additional metrics. We have added the reference and experimental results to the revised manuscript (Section 5.1 and Appendix E.7).

3. Paired statistical analysis. Following your suggestion, we conducted paired statistical analyses of ZEST and each baseline using their scores on the same test sequences. The test statistic is the difference in macro-average scores, with equal weighting across the six datasets.

To account for dependencies among related sequences, we grouped sequences by subject or acquisition setting and performed a two-sided paired cluster permutation test with 100,000 permutations, jointly flipping the signs of all paired differences within each cluster. We also performed 10,000 paired cluster bootstrap resamples within datasets to estimate 95% confidence intervals for the macro-average differences. Both procedures preserve the aggregation scheme: averaging sequence scores within each dataset, followed by equal weighting across the six datasets. We applied the Holm correction jointly across the eight comparisons formed by two baselines and four metrics: ARI, NMI, WARI, and SMS.

The table below reports the four WARI and SMS comparisons. All reported p-values are adjusted jointly across the eight comparisons.

ComparisonMetricMacro-average difference95% confidence intervalHolm-adjusted p-value
ZEST − Time2StateWARI+12.92[8.80, 16.84]0.000080
ZEST − Time2StateSMS+13.16[9.96, 16.15]0.000080
ZEST − E2USDWARI+14.81[12.00, 16.90]0.001070
ZEST − E2USDSMS+8.31[4.77, 11.39]0.000080

Macro-average differences and confidence intervals are reported on the score × 100 scale.

Across the six evaluated benchmarks, ZEST achieves significantly higher macro-average WARI and SMS than Time2State and E2USD, with Holm-adjusted p-values below 0.01 for all four comparisons. This analysis uses fixed predictions from each method.


[1] Félix Chavelli, Paul Boniol, and Michaël Thomazo. Toward Interpretable Evaluation Measures for Time Series Segmentation. NeurIPS, 2025.

AuthorsOpenReview ↗

[3/3]: Response to Q2: Choice of Chronos-2, MOMENT-1-large, and Qwen3.5-2B

Thank you for asking us to clarify these choices. Our model selection was guided primarily by the pretrained capabilities required at each stage and our computational budget.

Chronos-2 was selected for its pretrained zero-shot forecasting capability. This allows us to obtain forecasts for prototype mining without requiring state annotations or training a separate forecasting model for each source series. Specifically, we use the consistency of prediction errors across overlapping contexts to identify candidate stable fragments.

MOMENT-1-large was selected for its zero-shot representation capability. This allows us to compare prototypes in a shared representation space without training an additional representation model. We aggregate window embeddings within each prototype and use the resulting representations to control prototype similarity and help construct hierarchical supervision.

Qwen3.5-2B was selected for its strong multimodal capabilities, open weights, and compact size. Its vision-language interface fits our formulation of predicting state segments and merge trees from waveform plots, while its 2B parameter scale makes task adaptation and deployment practical within our computational budget.

We chose these widely used models for their respective roles, although other models with suitable capabilities could also be used.

Reviewer_7gjiOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes a novel approach to zero-shot time-series state detection based on pre-trained Vision-Language Models (VLMs). It argues that the traditional two-stage approach—representation learning followed by state inference—does not generalize well to zero-shot settings. This is because the representation learning stage relies on temporal proximity, while the inference stage depends on fitting clustering models to new target sequences, failing to provide an end-to-end solution. These limitations are further quantified by Proposition 1 in a Markov setting and by diagnostic experiment results in Section 4.2. To address these drawbacks, the authors introduce ZEST. This method uses forecast errors as a signal to identify a set of reusable prototypes from raw time series within a large corpus, subsequently creating embeddings for them. These prototypes are extended and concatenated to generate sequences with known boundaries and state IDs. The sequences are then converted into signal plots and used as image inputs for a VLM, which is fine-tuned to predict the segments and their corresponding IDs for zero-shot state detection. Experimental results on both synthetic data and standard benchmarks demonstrate significantly superior performance over baseline approaches on metrics including ARI and NMI, which can be further boosted by label-assisted granularity selection. Finally, an ablation study on data sources and prototype creation methods confirms the effectiveness of the proposed approach.

Critical strengths
  1. Using a vision-language model for state detection is a novel approach. To the best of my knowledge, this work is the first to make use of visual signals for global structure recognition in time-series data. Such a choice is intuitive and a good fit for modern VLMs.
  2. The presentation is clear and easy to follow. Fig. 1 compares the two-stage approach with ZEST and illustrates the corpus construction in a single figure. Sec. 4 further highlights the flaws of the two-stage pipeline before Sec. 5 introduces the new design. Table 1 uses colour coding to emphasise ZEST's strong performance.
  3. The experimental results are strong. In Table 1, ZEST-leaf is the best approach in 11 of 12 comparisons across six benchmarks and eleven baselines.
Critical weaknesses
  1. The choice of using an image input over the raw signal is not motivated or examined by any experimental results. Section 5 briefly mentions that a global image shows changes and recurrences in a single input. To strengthen this motivation, the paper should test an obvious alternative: using the same VLM backbone, prototype construction, and training data, but with the time-series signal provided as language tokens or through a numeric encoder.

  2. In the domain of time series, fine-tuning a 2B VLM—even with LoRA—incurs a heavy computational cost. Can the authors justify this approach by showing either that the same method works with smaller VLM models, or showing the cost-accuracy trade with baseline models' required similar training setups (e.g., GPU resources) and costs?

  3. I am also concerned about potential data leakage into the VLM from its backbone’s pretraining. Qwen3.5-2B’s pretraining data is not public, and the benchmarks are well-known: prior papers and repositories contain plots of these exact sequences with colored state annotations. If the VLM encountered such figures during pretraining, the zero-shot claim is weakened. In this work, the unadapted Qwen3.5-2B already shows competitive performance on some benchmarks (HAS and SKAB in Tab. 1). Could these be signs of information leakage from pretraining, or does the VLM simply already possess the relevant capability?

  4. The performance of ZEST on the target dataset ultimately depends on the data source and the recipe for prototype creation. The ablation study already shows that changes in either of these can lead to degraded performance.

  5. Although the paper is easy to follow, I would suggest reorganizing it. The method is not introduced until Section 5 (the end of page 4), while Section 4 uses more than one page to diagnose existing two-stage methods. Section 4 strengthens the arguments that temporal-proximity supervision is flawed and that pretrained encoders transfer poorly, but it does not motivate the specific design choices of ZEST.

Questions for authors

Apart from the questions in the critical weakness section, I have an additional question for the author: Qwen3.5 supports reasoning through a built-in thinking mode. Does ZEST make use of the reasoning capability of the model? If not, why?

7 comments
AuthorsOpenReview ↗

[1/5] Response to Weakness 1: Choice of Input Representation

We sincerely thank the reviewer for carefully reading our manuscript and providing thoughtful, constructive feedback!

Below, we address each concern and question point by point and explain the corresponding revisions to the manuscript.


For Weakness 1, we agree that comparing input representations would strengthen the motivation.

We aimed to compare three input representations: image tokens from time-series plots, text tokens from numerical values, and continuous patch embeddings from a numeric encoder [1]. However, text serialization produced input sequences too long for fine-tuning and would run out of memory on seven RTX 4090 GPUs.

Therefore, we compared the remaining two representations: image tokens and numeric patch embeddings. For both variants, we fine-tuned Qwen3.5-2B with LoRA using the same generated data and prediction targets. In addition, the numeric encoder was randomly initialized and trained from scratch. The table below reports zero-shot leaf ARI and NMI, the average number of input tokens per sequence, and end-to-end inference time on a single RTX 4090.

Input TypeLeaf ARI ↑Leaf NMI ↑Avg. Input Tokens ↓Inference Time (s/seq) ↓
Image Tokens64.4070.6712944.11
Numeric Encoder*12.4425.8219779.21

* The numeric encoder uses a patch length of 32 time steps and a stride of 16 time steps.

The numeric encoder’s lower accuracy may partly reflect difficulties in aligning time-series and language representations [2]. Unlike the pretrained visual pathway, the numeric encoder was randomly initialized and trained using only autoregressive supervision on the segmentation and merge-tree targets, without a representation-alignment objective.

By comparison, image tokens achieve better accuracy and faster inference than numeric patch embeddings. This supports our design intuition that images offer an effective and efficient way to visually identify state changes. Furthermore, with fixed rendering settings, image token counts stay bounded as sequence length increases. Numeric patching produces more tokens for longer sequences, increasing inference cost.

These results support image input as an accurate and efficient choice under the tested resource budget. We have added this comparison and the text-input feasibility discussion to Section 5.4 and Appendix E.5.1. Thanks for the reviewer's constructive suggestion again.


[1] Nie, Y., Nguyen, N. H., Sinthong, P., & Kalagnanam, J. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. In Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023.

[2] Yashwante, P., & Yu, R. Time Series, Vision, and Language: Exploring the Limits of Alignment in Contrastive Representation Spaces. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.

AuthorsOpenReview ↗

[2/5] Response to Weakness 2: Model Size and Computational Costs

Thank you for raising this concern. We additionally fine-tuned Qwen3.5-0.8B using the same settings as the 2B model. We compared both versions with Time2State and E2USD, two established deep learning methods for TSSD, in terms of accuracy, training cost, memory usage, and processing time.

MethodsLeaf ARI ↑Leaf NMI ↑Source TrainingSource Peak Memory (GB, all GPUs) ↓Target Time (s/seq) ↓Target Peak Memory (GB, all GPUs) ↓
ZEST-2B64.4070.6794.52 GPU-h93.244.1104.297
ZEST-0.8B61.8168.8658.39 GPU-h55.023.7211.750
Time2State51.9459.92N/AN/A5.3291.009
E2USD53.2257.61N/AN/A9.2780.095

Source training refers to ZEST’s one-time LoRA fine-tuning, after which the model is reused for new sequences. Target time covers the complete detection procedure for a new sequence, including image rendering for ZEST and per-sequence training and clustering for Time2State and E2USD. Target time and peak memory are measured on a single RTX 4090.

In terms of computational efficiency, ZEST-2B achieves lower average latency than Time2State and E2USD under the evaluated settings, benefiting from its zero-shot capability. In addition, ZEST-0.8B requires fewer GPU-hours for fine-tuning and further reduces inference latency, offering a lower-cost alternative with a modest decrease in average accuracy. These results highlight a trade-off between computational efficiency and detection accuracy.

In terms of memory usage, both ZEST variants require more GPU memory than Time2State and E2USD. However, ZEST-0.8B reduces peak inference memory to 1.75 GB, making deployment on memory-constrained devices more practical. Future work could explore distilling ZEST into a smaller task-specific detector to further reduce memory requirements while preserving detection accuracy.

We have added this comparison and discussion to Section 5.4 and Appendix E.5.2.

AuthorsOpenReview ↗

[3/5] Response to Weakness 3: Potential Pretraining Leakage

Thank you for raising this concern. We address it in two parts.

1. Clarifying what we can verify.

We can confirm that no evaluation benchmark data leaked into ZEST’s fine-tuning process. However, without access to Qwen’s complete pretraining data, we cannot rule out leakage of the annotated plots into the backbone’s pretraining process. To address this concern, we conducted an additional experiment to assess generalization to unseen samples.

2. Evaluating generalization on new samples.

For this additional experiment, we used TSAGen [1] to generate new test sequences and their annotations. These samples were used exclusively for evaluation and could not have been memorized during backbone pretraining.

TSAGen was used by Time2State [2] to construct the original "Synthetic" benchmark, which was subsequently adopted by E2USD [3] and our work.

We compared the unadapted Qwen3.5-2B, ZEST, Time2State, and E2USD on both the original and newly generated test sets. Time2State and E2USD provide reference points for assessing differences in difficulty between the two test sets. ZEST uses its default leaf-level output, and ARI and NMI are reported on a 0–100 scale.

MethodOriginal ARI ↑New ARI ↑Δ\Delta ARIOriginal NMI ↑New NMI ↑Δ\Delta NMI
Qwen3.5-2B19.2719.41+0.1436.9237.36+0.44
ZEST96.8496.68-0.1694.2294.24+0.02
Gain (ZEST − Qwen)77.5777.27-0.357.3056.88-0.42
Time2State81.7681.67-0.0984.0783.71-0.36
E2USD88.4388.01-0.4280.2580.04-0.21

Original and New denote the original and newly generated Synthetic test sets, respectively. Δ = New − Original.

ZEST achieves similar gains over Qwen on both test sets, indicating that the benefits of task adaptation extend to new samples. Qwen also maintains comparable performance on the new samples, supporting its detection capability on unseen sequences.

Together, these results suggest that ZEST’s advantage reflects generalization beyond memorized benchmark examples. We have added this experiment to the revised manuscript (Appendix E.6, Table 19).


[1] Chengyu Wang, Kui Wu, Tongqing Zhou, Guang Yu, and Zhiping Cai. TSAGen: Synthetic Time Series Generation for KPI Anomaly Detection. IEEE Transactions on Network and Service Management, 19(1): 130–145, 2022.

[2] Chengyu Wang, Kui Wu, Tongqing Zhou, and Zhiping Cai. Time2State: An Unsupervised Framework for Inferring the Latent States in Time Series Data. Proceedings of the ACM on Management of Data, 1(1): 17:1–17:18, 2023.

[3] Zhichen Lai, Huan Li, Dalin Zhang, Yan Zhao, Weizhu Qian, and Christian S. Jensen. E2Usd: Efficient-yet-effective Unsupervised State Detection for Multivariate Time Series. In Proceedings of the ACM on Web Conference 2024 (WWW ’24), pp. 3010–3021, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.