acceptodds
Under review as a conference paper at ICLR 2027

VQ-PACT: Learning Compositional Temporal Grammars for Text-Controlled Time Series Generation

Abstract

Text-controlled time series generation aims to synthesize sequences that follow natural-language descriptions while remaining faithful to real data distributions. Existing approaches often couple semantic understanding and sequence modeling within a unified generative representation, binding reusable temporal regularities to observed joint patterns. This limits the faithful realization of textual semantics and makes accurate local-event generation difficult to reconcile with the preservation of other temporal attributes. To address this challenge, we propose VQ-PACT, a text-controlled generation framework that combines disentangled representations of temporal factors with joint generation under semantic constraints. Its temporal grammar supports factor reuse and composition while preserving multiple numerical realizations of the same semantics, improving semantic consistency and the accuracy of local-event realization. Specifically, VQ-PACT first learns temporal primitives for morphological skeletons, rhythms, and local events, and strengthens their representational stability through recombination-based training to form a reusable temporal grammar. It then aligns textual descriptions with the dynamic shapes of these primitives and composes them according to the specified temporal semantics, constructing a complete temporal representation. Finally, selective semantic constraints are integrated with temporal decoding and scale restoration to realize textual requirements in the corresponding factors and their specified scopes. Experiments on twelve real-world datasets, Synth-U, and real report text from Time-MMD demonstrate improved overall distributional fidelity and fine-grained text control, together with generalization to unseen factor combinations under compositional holdout evaluation.

Primary area:
learning on time series and dynamical systems
Keywords:
Text-controlled time series generation, temporal grammar, disentangled representation, fine-grained semantic control, compositional generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Vx6yOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes VQ-PACT, a text-conditioned time-series generation framework combining factor-specific vector-quantized codebooks, phrase-level semantic grounding, an autoregressive compositional prior, and selective constraint realization. The framework is evaluated on twelve real-world datasets, Synth-U, and Time-MMD, with additional analyses of fidelity, controllability, generalization, and computational efficiency.

Critical strengths
  1. The paper addresses a meaningful problem and presents a comprehensive generation pipeline. The factorized representation is intuitively motivated, and the evaluation covers diverse datasets and multiple complementary metrics.
  2. The paper's multifaceted evaluation provides a more comprehensive perspective for assessing time series models.
Critical weaknesses
  1. Lack of clarity in methodological presentation and training pipeline. The paper is difficult to follow due to unclear methodological descriptions and insufficient implementation details. First, Figure 2 contains four subfigures (a–d), yet its caption provides no explanation of their respective roles. Moreover, Appendix F "Full Objective" section introduces eight loss terms for Stage 1 representation learning, making a complete training pipeline or pseudocode essential for understanding how these objectives are integrated and optimized. Also, several key components and notations are introduced without clear definitions. For example, the auxiliary decoders D_f and the joint decoder Dec (Lines 174–178) are not explicitly defined. Similarly, Appendix F.2 introduces regional supervision as part of the training procedure, but its implementation and execution are not clearly explained. Additionally, the meaning and role of the descriptor vocabulary remain ambiguous. The statement in Line 227, "Code cross-entropy and attribute regression supervise complete programs," also raises an important question: at which training stage are these supervision objectives applied? Overall, the methodological presentation lacks clarity and coherence, making the proposed framework unnecessarily difficult to understand and reproduce.

  2. Insufficient attribution of the main performance gains. Appendix Table 14 shows that Event Joint increases from approximately 10% for learned proposals to 93% after deterministic realization. This raises the question of whether the reported controllability primarily arises from the proposed compositional representations or from the rule-based realization mechanism.

  3. Insufficient explanation of Factor-OOD evaluation and computational efficiency. Section 4.5 introduces the Factor-OOD evaluation without clearly defining it in the main manuscript. Consequently, it is difficult to understand the experimental setup and, more importantly, how this evaluation directly supports the claimed contributions of VQ-PACT. The authors should clarify the definition, motivation, and relevance of Factor-OOD to the proposed method. Furthermore, the analysis of Computational Cost is insufficient given the complexity of the proposed framework. In particular, Stage 1 involves eight loss terms, suggesting a potentially substantial training overhead. A convincing efficiency evaluation should report training memory consumption, the execution time of individual modules, and overall training time or FLOPs. Without these measurements, it is difficult to assess the actual computational cost of VQ-PACT or substantiate its efficiency claims, especially considering the complexity of its architecture and training objectives.

Questions for authors

Please refer to Critical Weaknesses

Reviewer_WTCVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing text-controlled time series generators couple text understanding and sequence generation, so local changes in the description can disturb global properties in time sequences. To address this, the paper proposes VQ-PACT. It decomposes local information into skeleton, rhythm, and event views, quantizes them into factor-specific codebooks with a recombination invariance loss, and grounds textual phrases in the decoded behavior of codes through a deterministic verbalizer. Experiments on multiple real-world datasets, Synth-U, and others. Additional analyses also cover held-out feature combinations, inference ablations, and computational cost to demonstrate the effectiveness of the proposed model.

Critical strengths
  1. The motivation of the paper is clear and the proposed method has interpretable designs.
  2. The proposed alignment text conditioned priors and constraint realization mechanisms provide an explicit way to connect natural language descriptions with temporal features and support control during generation.
  3. The paper presents extensive experiments covering multiple real-world datasets. It also includes several additional analysis such as factorization compared with unified codebooks, computation costs and ablation study.
Critical weaknesses
  1. The training objectives and the ablation studies are only in the appendix, so the reader must often move between the main text and the appendix. The paper has more than 30 pages, but it does not give important design details. It does not specify the architectures of EE, DD, and DecDec, the model hyperparameters, or the meaning of several symbols. Without these details, it is difficult to examine the method and to reproduce the results. I recommend that the authors move the loss equations into Section 3 and add a notation table and detailed architecture explanations.
  2. The paper assumes time series data could be decomposed to skeleton, rhythm, and event, but real-world scenarios could be more complicated than this and often these signals could be related to each other. The paper should provide enough evidence to demonstrate these three signals are good enough even for complicated temporal patterns.
  3. Following weakness above, although the paper provides an illustrative example in Figure 1 about the motivation of the paper, it is still not clear how this is reflected in real-world data. Besides, it is not clear how the disentanglement of these signals perform compared to existing models and how each representation controls the final time series patterns. Some experiment ideas could be manually creating some patterns of each signal and using its representations in decoding to see if the final generation actually aligns with the pattern. This should help indicate the contribution of separation of signals.
  4. The discussion of objective function is not comprehensive. There exists multiple loss functions as mentioned in the appendix, but it is not clear how each loss is calculated and there is no ablation study on how each loss contributes to the model performance. More details should be provided.
  5. It seems the experiment on real-world datasets does not evaluate whether the generated sequence actually follows the text reports. If the model trained on these datasets ignore the texts, they could have the same performance as what are presented in table1 and table2. It would be better if the authors could also demonstrate good alignment between sequence and text on real-world data as well.
Questions for authors
  1. Which components of VQ-PACT are learned versus fixed by design, and how does the proposed decomposition differ from classical trend/seasonality/residual decomposition?
  2. How does the method handle texts with multiple events / changes such as “spike in the beginning and dip at the end”?
Reviewer_uXyZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

VQ-PACT learns differnet codebooks for trends, rhythms, and local events, and aligns text phrases with corresponding codes. A text conditioned AR model predicts code and a joint decoder reconstructs the sequence shape, with a mixture density network (MDN) sampling the overall mean and scale to restore the sequence’s numerical values.

Critical strengths

the idea of structured decomposition into trend, event and rhythm is classic and connects naturally to controlled generation

Critical weaknesses
  1. It seems like that phrase to code alignment is a main contribution, but is it used in the Synth-U evaluation? What evidence shows that this mechanism improves text control?

  2. I am not sure how much of the local control on Synth-U comes from the learned representation, since the pipeline also uses a separate background completion network and predefined event waveforms. The codebook comparison shows some benefit, but can the authors clarify the separate contributions of factorization and the invariance loss when the same post-processing is used?

Questions for authors
  1. Clarification: how are decoded shapes assigned text descriptions, and how are input phrases matched to these descriptions?
  2. Please also see weaknesses

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on the Event Joint Gain and the Contributions of the Eight Training Losses

Thank you for your further feedback. To directly address your concerns, we conducted additional controlled experiments to identify the sources of the Event Joint improvement and evaluate the individual and combined contributions of all eight training losses. We are committed to transparency and will release our training and evaluation code, ablation configurations, and experimental logs to support independent reproduction and verification. The results are presented below.

Q2: What accounts for the increase in Event Joint from approximately 10% to 93% after realization?

R2: To directly identify the source of this improvement, we further decomposed local event realization into two operations: background completion and text-specified event template insertion. We conducted an additional controlled experiment to quantify the contributions of these two operations.

Specifically, we enabled or disabled each operation independently while keeping the same batch of generated proposals fixed:

Background completionText-specified event templateEvent Joint
OffOff14.85%
OnOff5.73%
OffOn60.30%
OnOn93.10%

The results provide three findings:

  1. Event template insertion is the primary source of improvement. Adding the text-specified event template increases Event Joint from 14.85% to 60.30%, a gain of 45.45 percentage points.
  2. Background completion further improves event accuracy. When combined with event insertion, it raises Event Joint from 60.30% to 93.10%, an additional 32.80 percentage points.
  3. Background completion alone cannot satisfy the requested events. Without event insertion, Event Joint is only 5.73%, below the original 14.85%. Its benefit therefore comes from working together with event insertion rather than generating the requested events independently.

These results clarify the performance attribution: the substantial increase in Event Joint is mainly driven by deterministic local event realization, rather than the learned representation alone. The learned generator provides the initial sequence, while event templates and background completion enforce the requested local events. We will explicitly distinguish the performance before and after realization and incorporate these controlled comparisons into the revised manuscript.

Q3: What does each of the eight training losses contribute, and are they all necessary?

R3: We conducted individual and combined loss ablations, retraining the representation model and its corresponding generator for each setting while keeping local event processing fixed. The complete model achieves the highest post-realization Event Joint, and full-sequence reconstruction makes the largest contribution.

Stage 1 training settingEvent Joint beforeEvent Joint after
Full model (all eight losses)14.85%93.10%
Remove full-sequence reconstruction2.88%42.68%
Remove local-shape reconstruction14.38%85.88%
Remove vector quantization loss14.49%85.75%
Remove factor-view reconstruction14.83%85.17%
Remove factor-sum reconstruction14.82%85.39%
Remove leakage and sparsity regularization14.91%84.76%
Remove recombination invariance15.64%84.96%
Remove NULL auxiliary classification14.95%86.15%
Remove local-shape, factor-view, and factor-sum reconstruction13.98%86.10%
Remove leakage/sparsity regularization and recombination invariance15.69%85.27%
Keep only full-sequence reconstruction and VQ loss16.52%85.07%
Keep only full-sequence reconstruction17.45%85.21%

“Keep only” refers to Stage 1 training losses. All variants retain discrete codebooks, hard quantization, and subsequent generator training.

Removing full-sequence reconstruction reduces Event Joint from 14.85% to 2.88% before realization and from 93.10% to 42.68% afterward. This demonstrates that the same event-processing procedure cannot fully compensate for poor initial proposals.

Removing any one of the other seven losses yields 84.76%–86.15% post-realization Event Joint, below the full model's 93.10%. However, simpler objectives can sometimes achieve higher accuracy before realization. For example, retaining only full-sequence reconstruction gives 17.45% before realization but 85.21% afterward. These results show that the full objective performs best on post-realization Event Joint among the tested settings, although the eight losses have different effects and are not necessarily indispensable for every evaluation metric.

We sincerely thank the reviewer for these constructive comments and careful consideration. We hope our additional experiments and clarifications address your remaining concerns and strengthen your confidence in our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.