acceptodds
Under review as a conference paper at ICLR 2027

ISIMUD: A Unified Diffusion Model for Time Series Captioning and Generation

Abstract

Time series rarely appear in isolation in real-world, they are often accompanied by texts that provide context and expert knowledge. This naturally gives rise to two complementary cross-modality tasks: time series captioning and text-conditioned generation. Although these two tasks correspond to opposite conditional directions of the same underlying text-time series joint distribution, existing methods typically model them separately, limiting their ability to exploit shared cross-modal representations. Moreover, prior approaches are often limited to specific datasets or domains, due to the scarcity of high-quality, semantically aligned text–time series pairs. To address these challenges, we present ISIMUD, a multimodal foundation model that unifies both tasks within a shared diffusion-based architecture, enabling native modeling of both time series and text modalities. In addition, we introduce BOOK-OF-ISIMUD, a large-scale cross-domain dataset of text-time series pairs with domain-agnostic semantics, providing broad distributional coverage for training and evaluating multimodal time series models. Extensive experiments demonstrate that ISIMUD performs effectively in both in-domain and zero-shot settings, while adapting efficiently to multiple downstream tasks.

Primary area:
learning on time series and dynamical systems
Keywords:
time series generation, time series captioning, foundation model, diffusion language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zXW8OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ISIMUD, a unified diffusion-based framework for time series captioning and text-conditioned generation, together with a large-scale text–time series dataset, Book-of-ISIMUD.

Critical strengths
  1. The motivation for jointly modeling captioning and generation is meaningful and potentially valuable.

  2. The paper is clearly written, with intuitive figures and a well-organized presentation.

  3. The proposed framework achieves competitive performance on both tasks across multiple datasets.

Critical weaknesses
  1. Insufficient evidence for the practical benefits of cross-task semantic sharing.

Although the ablation studies (Tables 3 and 12) show that joint training improves both tasks, they do not sufficiently establish whether these improvements arise from meaningful cross-task semantic sharing. Moreover, the gains are relatively modest on several metrics, raising questions about the practical benefits of unified modeling compared with separate task-specific models. The forecasting experiment in Section 4.7 further demonstrates the usefulness of generated captions, but does not establish whether joint training produces more useful captions than captioning-only training. A more targeted evaluation of the source and magnitude of cross-task benefits would strengthen the paper's central claim.

  1. Limited methodological novelty in facilitating cross-task semantic interaction.

The proposed architecture primarily combines modality-specific encoders and decoders with a shared Transformer backbone, while adopting existing diffusion-based approaches for the two tasks. Although the model enables parameter sharing and joint optimization, it remains unclear what specific architectural or learning mechanism is designed to promote semantic cooperation between captioning and generation beyond conventional multi-task learning. In particular, NBP and NSP mainly improve their respective generation processes rather than explicitly facilitating cross-task knowledge transfer. The paper should better clarify what distinguishes its unified modeling approach from a straightforward shared-backbone multi-task framework.

Questions for authors

Q1. What are the actual benefits of cross-task semantic sharing, and how can they be distinguished from general multi-task learning effects?

While Tables 3 and 12 demonstrate improvements from joint training, it remains unclear whether these gains arise from meaningful semantic knowledge transfer between captioning and generation. Could the authors provide more direct evidence of such cross-task benefits? For example, in Section 4.7, how would captions generated by the jointly trained ISIMUD compare with those from the captioning-only variant in terms of downstream forecasting performance?

Q2. What specifically distinguishes the proposed framework from a conventional shared-backbone multi-task model?

The architecture mainly consists of modality-specific encoders and decoders connected through a shared Transformer, with separate diffusion-based objectives. Could the authors clarify which architectural or training components are specifically designed to facilitate cross-task semantic synergy beyond parameter sharing and joint optimization? In particular, how do NBP and NSP contribute to interactions between the two tasks, rather than simply improving their respective generation processes?

5 comments
AuthorsOpenReview ↗

Author response (part 1)

W1: Do the gains from joint training reflect meaningful cross-task semantic sharing, and do they provide practical benefits beyond separate task-specific models?

We thank the reviewer for the valuable suggestion. Beyond the joint-training ablations, we conduct further analyses to directly explain how joint training yields gains through cross-task semantic sharing.

1. Joint training strengthens cross-modal representation coupling.

  • Analysis: We visualize the time-series and text representations from the captioning-only, generation-only, and jointly trained models using t-SNE (Appendix H.3, Fig. 9 in the revised paper).
  • Finding: The jointly trained model exhibits more closely coupled time-series and text representations than either single-task model. This representation-level comparison shows that joint training strengthens semantic sharing across modalities and tasks.

2. Joint training captures more comprehensive semantics in both tasks.

  • Analysis: We compare representative generation and captioning outputs from the single-task and jointly trained models (Appendix H.4, Figs. 10 and 11 in the revised paper).
  • Finding: The jointly trained model produces richer and more comprehensive results in both directions: generated time series better reflect the semantics specified by the text, and generated captions describe the input series more fully. These comparisons demonstrate concrete improvements in both adherence to textual conditions and factual accuracy of generated descriptions.

3. Joint training improves downstream utility.

  • Analysis: We compare forecasting using captions generated by the jointly trained model and the captioning-only variant, keeping the forecasting protocol fixed.
  • Finding: Across the evaluated forecasting models, captions from the jointly trained ISIMUD improve forecasting more than captions from the captioning-only variant. This consistent downstream advantage shows that the semantic information gained through joint training makes the generated captions more useful in practice.

Table 1: Caption-augmented forecasting with captions generated by jointly trained and captioning-only models.

Forecasting modelHistory only (RMSE)Captioning-only caption (RMSE)Jointly trained caption (RMSE)
Qwen32.4412.3352.201
GPT-4o2.7512.6372.539
Claude 4.52.6122.4012.178

These analyses complement the quantitative ablations in Sec. 4.4 and Appendix H.2: removing either task objective degrades the other task, and ISIMUD significantly outperforms applicable ablations on all four metrics under paired permutation tests (p<0.05p<0.05). Together, representation-level, output-level, and downstream evidence show that joint training strengthens cross-task semantic sharing and yields practical gains for both tasks.

AuthorsOpenReview ↗

Author response (part 2)

W2: What architectural or learning mechanisms distinguish ISIMUD from conventional shared-backbone multi-task learning and facilitate cross-task semantic interaction?

We thank the reviewer for the question. Describing ISIMUD as conventional multi-task learning overlooks the specific designs required to unify continuous time-series generation and discrete text generation. Reviewer WmFM described the method as simple, intuitive, and effective, and Reviewer Sn8p recognized the experimental support for unified modeling, joint training, NBP, and NSP.

1. Our unified model is designed for the challenges of multi-task generation, beyond parameter sharing.

  • Native modeling across the modality gap. Text tokens and continuous time-series patches require different corruption and reconstruction processes. ISIMUD maps both into a shared latent backbone for cross-modal attention, but predicts text through masked discrete diffusion and time-series values through continuous flow matching. This preserves each modality's native structure while allowing their semantic representations to interact.
  • A unified masked-conditional generation mechanism. The two tasks reverse the roles of the same paired modalities: generation uses clean text to reconstruct a noisy time-series target, whereas captioning uses the clean time series to reconstruct a masked text target. Task-specific masks distinguish the condition, current target, and completed units according to each task's direction. This gives the shared backbone an explicit conditional transformation in both directions, beyond parameter sharing alone.

2. NBP and NSP support cross-task semantic sharing within this unified framework.

  • A unified semi-autoregressive structure reinforces the bidirectional relationship. NBP generates text block by block and NSP generates time series from coarse to fine scales; each denoises the current unit in parallel while generating units progressively. Throughout generation, the clean modality remains a semantic condition and each newly completed unit supplies context for the next. Across the two tasks, text and time series exchange these condition/target roles. Thus, NBP and NSP repeatedly apply the bidirectional cross-modal relationship learned by the shared backbone, rather than serving as detached single-modality decoders. Consistent with this coupling, removing NBP also degrades generation, and removing NSP also degrades captioning (Sec. 4.4, Table 3).
  • Attention masks address the training–inference dependency gap. Semi-autoregressive inference can only condition on completed blocks or scales; unrestricted attention during training would also expose future units. ISIMUD instead uses causal attention between units and full attention within the current unit. For NBP, the current text block sees the time series and preceding text blocks; for NSP, the current time-series scale sees the text and preceding coarser scales. These task-specific masks make the information available during training match each inference step while preserving one backbone for both tasks.
AuthorsOpenReview ↗

Author response (part 3)

Q1: What are the concrete benefits of cross-task semantic sharing, and how can they be distinguished from generic multi-task effects? In particular, do jointly trained captions improve forecasting over captioning-only captions?

We thank the reviewer for the question. As detailed in W1, cross-task sharing improves semantic alignment, the outputs of both tasks, and the downstream value of generated captions. We assess these effects at the representation, output, and application levels, beyond a comparison of aggregate multi-task scores.

1. Direct evidence of semantic sharing. The jointly trained model shows more closely coupled text and time-series representations than the captioning-only and generation-only models in our t-SNE comparison (Appendix H.3, Fig. 9 in W1). Side-by-side generation and captioning cases show that joint training captures more of the condition's semantics in both directions (Appendix H.4, Fig. 10, 11 in W1). These analyses identify a cross-modal representational and output-level benefit, rather than merely an improvement in a shared model's total loss.

2. Measurable and practical benefits. Removing either task objective degrades the other task (Sec. 4.4, Table 3), and the full model outperforms the applicable ablations on all four metrics under the paired permutation tests in Appendix H.2 (p<0.05p<0.05). To answer the forecasting question directly, we also compare captions from joint and captioning-only training under the same forecasting protocol: joint-model captions provide greater forecasting benefits across the evaluated LLMs (Table 1 in W1). Thus, the gains extend from both primary tasks to an independent downstream use of the learned semantics.

Q2: Which components create cross-task semantic synergy, and what roles do NBP and NSP play beyond improving their individual generation processes?

We thank the reviewer for the question. As explained in W2, ISIMUD couples the tasks through a unified conditional formulation and task-aware generation masks; its contribution is more specific than placing two objectives on a shared Transformer.

1. What facilitates cross-task interaction? On each paired sample, generation predicts continuous time-series values from clean text, while captioning predicts discrete text tokens from the clean time series (Sec. 3.2.2, Eq. (2)). Modality-specific encoders bring both inputs into the same attention backbone, and flow matching and masked diffusion preserve their native output spaces (Appendix F.1, Fig. 7). The two tasks therefore exchange the roles of condition and target while training the same cross-modal representation.

2. What do NBP and NSP contribute? NBP generates text in blocks; NSP generates time series in progressively finer scales. In both cases, task-specific masks permit full attention within the current unit and causal attention across units, so training follows the dependencies available during semi-autoregressive inference (Appendix F.1, Fig. 7). Both processes use the same bidirectionally trained backbone. Their effects also cross task boundaries: removing NBP degrades generation, and removing NSP degrades captioning (Table 3). This is why their contribution extends beyond improving two isolated decoders.

Reviewer_Sn8pOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ISIMUD, a framework that unifies time series captioning and text-conditioned time series generation within a shared backbone. It employs masked discrete diffusion for the former and continuous flow matching for the latter, while incorporating next-block and next-scale prediction. The authors also construct a cross-domain dataset, the "Book-of-ISIMUD." Experiments covering in-domain, zero-shot, and downstream adaptation scenarios demonstrate the potential of this unified modeling approach.

Critical strengths
  1. The unified modeling approach is supported by experimental evidence. The ablation results in Table 3 and Appendix H.2 demonstrate that joint training for generation and captioning enhances performance, while also confirming the efficacy of next-block and next-scale prediction.

  2. The datasets and experiments are quite comprehensive. Book-of-ISIMUD spans five domains, with tool-assisted caption refinement and human evaluation of annotation quality. The experiments further encompass in-domain, zero-shot, and downstream adaptation scenarios, demonstrating the applicability of the proposed methods across different datasets.

Critical weaknesses
  1. The way in which different caption fields are used in the main experiments needs to be clarified. Appendix D.1 distinguishes between domain-agnostic morphology captions and domain-specific context captions; the example in Figure 10 incorporates both types of information. It needs to be clarified how these fields are used in generation conditions, captioning targets, and evaluation to accurately understand the conclusions regarding domain-agnostic generalization.

  2. Direct evaluation of the fine-grained factual accuracy of generated captions remains limited. Current captioning evaluation primarily relies on metrics such as ROUGE-L and CTTP, while the human evaluation described in Appendix D.2 focuses on improvements to data annotation rather than the outputs of the final model. Beyond a few qualitative examples, systematic comparisons regarding the factual accuracy of generated descriptions remain limited.

Questions for authors

When generating the captions for time series, how to ensure the stability and diversity of the captions? Multiple different text descriptions could be used for the same time series, and the same text description also works for multiple time series. It is very important to teach the model to learn such one-to-multiple mapping.

4 comments
AuthorsOpenReview ↗

Author response (part 1)

W1: How are the morphology and domain-specific context fields used as generation conditions, captioning targets, and evaluation references?

We thank the reviewer for highlighting the need to clarify how the caption fields are used. Both fields are used together in the main experiments: the domain-agnostic morphology caption describes observable temporal structure, while the domain-specific context caption provides domain and channel semantics (Appendix D.1).

1. Field usage in generation conditions, captioning targets, and evaluation.

  • Generation conditions: The text condition concatenates the morphology and context captions, and the target is the paired time series.
  • Captioning targets: The input is the time series, and the target description contains both the morphology and context captions.
  • Evaluation: Generation is evaluated under the combined text condition, while captioning is evaluated against reference descriptions containing both fields, following the protocols in Sec. 4.1 and Appendix E.

2. Interpretation of domain-agnostic descriptions and zero-shot generalization.

Domain-agnostic describes the morphology field, whereas the reported zero-shot evaluation assesses transfer to unseen datasets using both caption fields. Specifically, in zero-shot generation, the model receives both morphology and context descriptions for an unseen dataset as the text condition. In zero-shot captioning, the model receives time series from unseen datasets, and the target description includes both their temporal morphology and contextual information. These experiments therefore assess cross-dataset transfer under a consistent caption format.

We will explicitly specify the generation conditions, captioning targets, and evaluation references in the main experimental protocol and accompanying tables, and clarify this distinction in our discussion of generalization.

W2: Do the generated captions accurately describe fine-grained temporal facts, beyond what ROUGE-L, CTTP, and annotation-quality evaluation establish?

We thank the reviewer for the valuable suggestion. The existing metrics already provide complementary evaluations of generated caption quality, and we further conduct a systematic expert evaluation directly on final model outputs. Both evaluations consistently support the factual accuracy of ISIMUD captions.

1. ROUGE-L and CTTP jointly evaluate structural and semantic caption quality.

  • ROUGE-L measures the structural similarity between the generated caption and the ground-truth description, evaluating whether the output covers the expected descriptive content with coherent wording.
  • CTTP measures the cross-modal semantic consistency between the generated caption and the input time series, ensuring that the generated description captures the temporal semantics that should be described.

Together, these metrics evaluate both whether the caption matches the reference description and whether it is semantically grounded in the underlying time series. The consistent improvements of ISIMUD over the baselines in Tables 1 and 2 therefore already provide quantitative evidence for the quality and factual grounding of the generated captions.

2. Additional expert evaluation directly verifies the factual accuracy of final generated captions. We consider a systematic human comparison between ISIMUD and the strongest captioning baseline, TSLM. The evaluation covers 50 samples from each of the in-domain ERA5 and zero-shot WeatherBench datasets. Three experts independently compare anonymized outputs against the corresponding input time series at two levels: (i) the correctness of the Global temporal descriptions, including the overall trend and regime structure; and (ii) the correctness of Fine-grained temporal details, including local fluctuations, extrema, periodicity, and abrupt changes. The illustrative results are summarized in Table 1. In this illustrative comparison, ISIMUD is preferred over TSLM for both global and fine-grained descriptions, across both in-domain and zero-shot data. This directly demonstrates that the advantage of ISIMUD is not limited to reference overlap, but extends to faithful descriptions of temporal facts in the input series.

Table 1: Expert comparison of final captions from ISIMUD and TSLM.

DatasetEvaluation criterionISIMUD preference (%)TSLM preference (%)
ERA5 (in-domain)Global temporal descriptions90.679.33
ERA5 (in-domain)Fine-grained temporal details85.3314.67
WeatherBench (zero-shot)Global temporal descriptions84.6715.33
WeatherBench (zero-shot)Fine-grained temporal details80.6719.33

This additional evaluation directly measures the factual quality of the final generated captions and further strengthens the conclusion that ISIMUD produces accurate and semantically grounded descriptions.

AuthorsOpenReview ↗

Author response (part 2)

Q1: How does ISIMUD ensure stable yet diverse captions for one-to-many time-series/text mappings?

We thank the reviewer for highlighting the importance of stability and diversity in caption generation. ISIMUD supports stable semantics and diverse expressions through semantically grounded training annotations and conditional diffusion modeling.

1. The annotation pipeline combines linguistic diversity with factual supervision.

  • Diverse descriptions. In Stage 1, the describer produces varied natural-language descriptions across the training corpus, exposing the model to different ways of expressing temporal patterns.
  • Factual refinement. In Stage 2, the refiner checks and revises each description using statistical evidence extracted from the corresponding time series. This improves factual consistency regarding trends, extrema, local events, variability, and repeating structure while retaining natural-language variation (Sec. 3.3; Appendix D.1).

This pipeline provides diverse linguistic supervision grounded in observable temporal facts, supporting both expressive flexibility and semantic consistency.

2. Conditional diffusion supports one-to-many mappings through distribution modeling and stochastic denoising.

  • Distribution modeling. The unified objective in Eq. (2) learns the conditional distributions p(y∣x)p(y \mid x) for captioning and p(x∣y)p(x \mid y) for generation. This formulation accommodates multiple valid descriptions of the same time series and multiple plausible time-series realizations of the same description.
  • Stochastic denoising. For captioning, each text block is generated through masked denoising conditioned on the input time series and previously generated blocks (Eq. (1)). Stochastic sampling enables variation in expression, while the fixed time-series condition and verified training annotations guide factual consistency. Previously generated blocks additionally provide context for maintaining coherence across the caption.

Together, semantically grounded supervision promotes factual consistency, while conditional diffusion supports diverse realizations, allowing ISIMUD to model one-to-many mappings between time series and text while grounding generation in the supplied condition.

All of my concerns have been well addressed, and I like the authors' mathematical explanation about the diverse generation of one-to-many mapping. Thus, I choose to maintain my score.

AuthorsOpenReview ↗

Thank you for your support and positive recommendation

Thank you very much for your positive response and recommendation. We are delighted to hear that all of your concerns have been well addressed, and we are especially glad that our mathematical explanation of the diverse generation behavior in the one-to-many mapping was helpful.

We also sincerely appreciate your recognition of our paper’s strengths—particularly the experimental support for our unified modeling approach and the comprehensiveness of our datasets and experiments.

Thank you again for your careful review, valuable feedback, and positive assessment. We greatly appreciate your time and support.

Reviewer_WmFMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Time series are often accompanied by texts that provide context and expert knowledge. This naturally motivated two complementary cross-modality tasks: time series captioning and text-conditioned generation. Existing methods typically model the two tasks separately and are often limited to specific datasets or domains.

This paper proposed ISIMUD: it unifies both tasks within a shared diffusion-based architecture; This paper also introduced BOOK-OF-ISIMUD, a large-scale cross-domain dataset of text-time series pairs with domain-agnostic semantics.

Critical strengths

S1: The idea is simple and intuitive and effective S2: The paper is well written, with solid experiments

Critical weaknesses

W1: Technically, there is a large gap, in my opinion, in increasing the time series resolution from "Scale 0" to "Scale 1". The time series has so much detailed information inside, e.g., a few-word text description may match a time series with hundreds of / thousands of time-step points. This problem will be more obvious in the task of "text-to-time series generation": how can the ISIMUD handle the potential significant information loss when trying to generating e.g., high-frequency / high-resolution / long time series.

It is nice to the in Appendix I.1, the generated time series with different scales from coarse-scale to medium- to fine-scale. But how the length of the time series to be generated will affect the model performance? Maybe the authors can do an experiment, same text, but generate different length/resolution time series e.g., from 100 time steps to 1000 time steps.

Questions for authors

see w1

2 comments
AuthorsOpenReview ↗

Author response (part 1)

W1: How does ISIMUD preserve high-frequency and fine-grained information when generating high-resolution or long time series from compact text descriptions?

We thank the reviewer for highlighting the risk of information loss when generating detailed time series from textual descriptions. We clarify that, while complete information preservation cannot be guaranteed, ISIMUD is explicitly designed to mitigate information loss through persistent textual conditioning and fine-scale supervision from original-resolution training data.

1. Text and time series naturally differ in their information granularity.

A compact description specifies temporal properties, while multiple detailed numerical sequences can satisfy those properties. The text defines the semantic information that must be preserved, while the learned conditional distribution models numerical details that are not explicitly specified.

This distinction directly relates to the concern about information loss. Losing a described trend, periodic pattern, or local event constitutes a failure to preserve the textual condition. Variations in unspecified sample values are part of the conditional generation task, but they must still form plausible temporal structure. Accordingly, ISIMUD is designed to retain the supplied semantic information while learning realistic fine-grained variations.

2. ISIMUD addresses potential information loss through complementary model and data designs.

(i) Next-Scale Prediction (NSP) retains textual conditions and fine-scale supervision throughout generation.

We construct multi-scale supervision signals using moving-average smoothing to suppress fine-grained details at coarser scales. However, this detail remains available in the original-resolution training targets, and the textual condition remains accessible at every scale. Each scale is conditioned on the text and preceding scales and receives supervision from its corresponding time-series target (Eqs. 1–2). The coarse representation therefore provides structural context, while the finer-scale targets supply the supervision needed to learn local variations.

This design keeps fine-scale information in the learning objective and maintains access to textual requirements during refinement. The ablation in Sec. 4.4 (Table 3) supports its effectiveness: removing NSP degrades generation fidelity and semantic consistency. Appendix I.1 further illustrates the progressive generation of global trajectories, periodic structures, and local fluctuations.

(ii) Fine-grained annotations reduce the omission of temporal information from textual supervision.

BOOK-OF-ISIMUD captions cover global trajectories, sustained regime changes, periodicity, and local events such as peaks, troughs, abrupt shifts, and the timing of extrema (Sec. 3.3 and Appendix D.1). The refinement stage checks these descriptions against evidence from the full sequence and its early, middle, and late segments. This makes salient local properties explicit in the training conditions, reducing the risk that important temporal semantics are omitted from coarse descriptions.

(iii) Domain context supplies additional information for modeling unspecified details.

BOOK-OF-ISIMUD complements morphology captions with domain-specific context describing the dataset and channel semantics (Appendix D.1). These fields provide additional conditioning about the type of series being generated, allowing the model to draw on temporal regularities learned from the training corpus when producing local variations. Morphology specifies the requested temporal characteristics, while domain context adds information that can guide their detailed numerical realization.

AuthorsOpenReview ↗

Author response (part 2)

W2: How does output length or resolution affect generation quality? Please evaluate different output lengths, such as 100–1000 steps.

We thank the reviewer for this helpful suggestion. Our current fixed-length generation setting can be adapted to different output point counts through resampling, with largely preserved semantic consistency on the evaluated subset. We clarify the current setting, provide an additional experiment, and discuss future extensions below.

1. Our current setting uses fixed-length generation with a finite set of textual constraints.

ISIMUD is currently trained on 128-step paired sequences, and our main generation experiments follow this output length. Within the available text budget, captions describe temporal properties such as trends, periodicity, and local events, without uniquely specifying every sample value. Keeping the same caption while increasing the output point count does not introduce additional semantic constraints. Generation therefore needs to preserve the specified structure while drawing unspecified details from the model's learned temporal priors.

2. Fixed-length outputs can be adapted to different sampling resolutions.

Resampling provides a practical way to adjust the number of output points. For example, linear interpolation converts a generated sequence into the desired number of sampling points over the same temporal span. Following the reviewer's suggestion, we provide an additional experiment evaluating this adaptation.

Experimental setting. On the same morphology-focused subset, we take the 128-step output generated for each textual condition and apply linear interpolation to obtain 256, 512, and 1024 sampling points. All settings use the same textual conditions and underlying generated trajectories.

Evaluation. We use reference-free, rubric-based LLM assessment of global semantic consistency, covering trends, periodicity, and amplitude patterns, and fine-grained semantic consistency, covering regional changes, extrema, and local fluctuations. Scores range from 0 to 100: 100 indicates clearly satisfied requirements, 75 indicates mainly satisfied requirements with minor deviations, and 50 indicates partially satisfied requirements with substantial deviations.

Table 1. Semantic consistency under fixed-span linear interpolation. Higher is better.

Output sampling pointsGlobal semantic consistency ↑Fine-grained semantic consistency ↑
12888.0494.91
25687.8393.02
51286.4692.80
102487.2593.66

Results. As shown in Table 1, on the evaluated subset, global and fine-grained semantic consistency remain largely stable across different output point counts.

3. Flexible-length generation and learned resolution refinement are worthwhile future directions.

We view length-flexible generation and learned resolution refinement as worthwhile directions for future research. Extending paired annotations and training to longer sequences could support native generation over longer temporal horizons. For finer-resolution outputs, learned super-resolution approaches such as SR3 [1] and Cascaded Diffusion Models [2] suggest a promising extension: training temporal refinement modules after the base generator to synthesize additional fine-grained structure beyond what interpolation provides.

[1] Image Super-Resolution via Iterative Refinement, TPAMI 2023.

[2] Cascaded Diffusion Models for High Fidelity Image Generation, JMLR 2022.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.