- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ISIMUD, a unified diffusion-based framework for time series captioning and text-conditioned generation, together with a large-scale text–time series dataset, Book-of-ISIMUD.
Critical strengths
-
The motivation for jointly modeling captioning and generation is meaningful and potentially valuable.
-
The paper is clearly written, with intuitive figures and a well-organized presentation.
-
The proposed framework achieves competitive performance on both tasks across multiple datasets.
Critical weaknesses
- Insufficient evidence for the practical benefits of cross-task semantic sharing.
Although the ablation studies (Tables 3 and 12) show that joint training improves both tasks, they do not sufficiently establish whether these improvements arise from meaningful cross-task semantic sharing. Moreover, the gains are relatively modest on several metrics, raising questions about the practical benefits of unified modeling compared with separate task-specific models. The forecasting experiment in Section 4.7 further demonstrates the usefulness of generated captions, but does not establish whether joint training produces more useful captions than captioning-only training. A more targeted evaluation of the source and magnitude of cross-task benefits would strengthen the paper's central claim.
- Limited methodological novelty in facilitating cross-task semantic interaction.
The proposed architecture primarily combines modality-specific encoders and decoders with a shared Transformer backbone, while adopting existing diffusion-based approaches for the two tasks. Although the model enables parameter sharing and joint optimization, it remains unclear what specific architectural or learning mechanism is designed to promote semantic cooperation between captioning and generation beyond conventional multi-task learning. In particular, NBP and NSP mainly improve their respective generation processes rather than explicitly facilitating cross-task knowledge transfer. The paper should better clarify what distinguishes its unified modeling approach from a straightforward shared-backbone multi-task framework.
Questions for authors
Q1. What are the actual benefits of cross-task semantic sharing, and how can they be distinguished from general multi-task learning effects?
While Tables 3 and 12 demonstrate improvements from joint training, it remains unclear whether these gains arise from meaningful semantic knowledge transfer between captioning and generation. Could the authors provide more direct evidence of such cross-task benefits? For example, in Section 4.7, how would captions generated by the jointly trained ISIMUD compare with those from the captioning-only variant in terms of downstream forecasting performance?
Q2. What specifically distinguishes the proposed framework from a conventional shared-backbone multi-task model?
The architecture mainly consists of modality-specific encoders and decoders connected through a shared Transformer, with separate diffusion-based objectives. Could the authors clarify which architectural or training components are specifically designed to facilitate cross-task semantic synergy beyond parameter sharing and joint optimization? In particular, how do NBP and NSP contribute to interactions between the two tasks, rather than simply improving their respective generation processes?
5 comments
Author response (part 2)
W2: What architectural or learning mechanisms distinguish ISIMUD from conventional shared-backbone multi-task learning and facilitate cross-task semantic interaction?
We thank the reviewer for the question. Describing ISIMUD as conventional multi-task learning overlooks the specific designs required to unify continuous time-series generation and discrete text generation. Reviewer WmFM described the method as simple, intuitive, and effective, and Reviewer Sn8p recognized the experimental support for unified modeling, joint training, NBP, and NSP.
1. Our unified model is designed for the challenges of multi-task generation, beyond parameter sharing.
- Native modeling across the modality gap. Text tokens and continuous time-series patches require different corruption and reconstruction processes. ISIMUD maps both into a shared latent backbone for cross-modal attention, but predicts text through masked discrete diffusion and time-series values through continuous flow matching. This preserves each modality's native structure while allowing their semantic representations to interact.
- A unified masked-conditional generation mechanism. The two tasks reverse the roles of the same paired modalities: generation uses clean text to reconstruct a noisy time-series target, whereas captioning uses the clean time series to reconstruct a masked text target. Task-specific masks distinguish the condition, current target, and completed units according to each task's direction. This gives the shared backbone an explicit conditional transformation in both directions, beyond parameter sharing alone.
2. NBP and NSP support cross-task semantic sharing within this unified framework.
- A unified semi-autoregressive structure reinforces the bidirectional relationship. NBP generates text block by block and NSP generates time series from coarse to fine scales; each denoises the current unit in parallel while generating units progressively. Throughout generation, the clean modality remains a semantic condition and each newly completed unit supplies context for the next. Across the two tasks, text and time series exchange these condition/target roles. Thus, NBP and NSP repeatedly apply the bidirectional cross-modal relationship learned by the shared backbone, rather than serving as detached single-modality decoders. Consistent with this coupling, removing NBP also degrades generation, and removing NSP also degrades captioning (Sec. 4.4, Table 3).
- Attention masks address the training–inference dependency gap. Semi-autoregressive inference can only condition on completed blocks or scales; unrestricted attention during training would also expose future units. ISIMUD instead uses causal attention between units and full attention within the current unit. For NBP, the current text block sees the time series and preceding text blocks; for NSP, the current time-series scale sees the text and preceding coarser scales. These task-specific masks make the information available during training match each inference step while preserving one backbone for both tasks.
Author response (part 3)
Q1: What are the concrete benefits of cross-task semantic sharing, and how can they be distinguished from generic multi-task effects? In particular, do jointly trained captions improve forecasting over captioning-only captions?
We thank the reviewer for the question. As detailed in W1, cross-task sharing improves semantic alignment, the outputs of both tasks, and the downstream value of generated captions. We assess these effects at the representation, output, and application levels, beyond a comparison of aggregate multi-task scores.
1. Direct evidence of semantic sharing. The jointly trained model shows more closely coupled text and time-series representations than the captioning-only and generation-only models in our t-SNE comparison (Appendix H.3, Fig. 9 in W1). Side-by-side generation and captioning cases show that joint training captures more of the condition's semantics in both directions (Appendix H.4, Fig. 10, 11 in W1). These analyses identify a cross-modal representational and output-level benefit, rather than merely an improvement in a shared model's total loss.
2. Measurable and practical benefits. Removing either task objective degrades the other task (Sec. 4.4, Table 3), and the full model outperforms the applicable ablations on all four metrics under the paired permutation tests in Appendix H.2 (). To answer the forecasting question directly, we also compare captions from joint and captioning-only training under the same forecasting protocol: joint-model captions provide greater forecasting benefits across the evaluated LLMs (Table 1 in W1). Thus, the gains extend from both primary tasks to an independent downstream use of the learned semantics.
Q2: Which components create cross-task semantic synergy, and what roles do NBP and NSP play beyond improving their individual generation processes?
We thank the reviewer for the question. As explained in W2, ISIMUD couples the tasks through a unified conditional formulation and task-aware generation masks; its contribution is more specific than placing two objectives on a shared Transformer.
1. What facilitates cross-task interaction? On each paired sample, generation predicts continuous time-series values from clean text, while captioning predicts discrete text tokens from the clean time series (Sec. 3.2.2, Eq. (2)). Modality-specific encoders bring both inputs into the same attention backbone, and flow matching and masked diffusion preserve their native output spaces (Appendix F.1, Fig. 7). The two tasks therefore exchange the roles of condition and target while training the same cross-modal representation.
2. What do NBP and NSP contribute? NBP generates text in blocks; NSP generates time series in progressively finer scales. In both cases, task-specific masks permit full attention within the current unit and causal attention across units, so training follows the dependencies available during semi-autoregressive inference (Appendix F.1, Fig. 7). Both processes use the same bidirectionally trained backbone. Their effects also cross task boundaries: removing NBP degrades generation, and removing NSP degrades captioning (Table 3). This is why their contribution extends beyond improving two isolated decoders.
Author response (part 1)
We thank the reviewer for the valuable suggestion. Beyond the joint-training ablations, we conduct further analyses to directly explain how joint training yields gains through cross-task semantic sharing.
1. Joint training strengthens cross-modal representation coupling.
2. Joint training captures more comprehensive semantics in both tasks.
3. Joint training improves downstream utility.
Table 1: Caption-augmented forecasting with captions generated by jointly trained and captioning-only models.
These analyses complement the quantitative ablations in Sec. 4.4 and Appendix H.2: removing either task objective degrades the other task, and ISIMUD significantly outperforms applicable ablations on all four metrics under paired permutation tests (p<0.05). Together, representation-level, output-level, and downstream evidence show that joint training strengthens cross-task semantic sharing and yields practical gains for both tasks.