- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes VT-Diff, which renders a multivariate series with missing values as line charts, extracts global and local visual features with a pretrained ViT, converts them via a cross-attention bottleneck with learnable temporal anchors into per-timestep conditions injected into a CSDI-style denoising network, thereby "perceiving structure in the visual domain while generating values in the temporal domain," and compares against 17 baselines on 7 datasets under 3 missingness patterns.
Critical strengths
- A novel cross-modal idea that avoids the trap of "generating in pixel space." The authors clearly distinguish what visual representations are good at (geometric shape, overall trend) from what numerical imputation requires (precise values), and choose to let vision serve only as a condition while keeping generation in the temporal domain. This is a sensible design choice. Using learnable temporal anchors as queries to cross-attend to visual patches, aligning 2-D spatial tokens into 1-D per-step conditions, is a clean interface that could transfer to other conditional diffusion frameworks.
- The motivation targets a real pain point. Context degradation of conditional diffusion models under high missing rates and long contiguous gaps is a recognized challenge. Approaching it from the angle of "supplementing structural priors" complements frequency-domain priors (FGTI, LSCD) and graph priors (PriSTI) naturally.
Critical weaknesses
- The core motivation does not hold at the information level, or at least is not argued: the rendered image contains no new information, and the "visually discernible shape" comes precisely from the interpolation it is supposed to overcome.
- The rendered image is a deterministic function of plus linear interpolation. Inside a long gap, the "macroscopic shape" shown in the image is simply a straight line; under high missing rates, the so-called "still discernible shape" also comes from linearly connecting sparse observations. Therefore, the core claim that "the shape remains visually discernible even when numerical observations are incomplete" amounts to "the contour after linear interpolation," which provides no information in the missing regions that a temporal encoder cannot obtain.
- If the gains are real, the only remaining possible sources are: inductive biases or prior knowledge from the pretrained ViT (why ImageNet natural-image priors apply to line charts needs justification); a larger receptive field and a different encoding scheme; or additional parameter capacity. The paper does not disentangle these explanations, nor does it provide any theoretical or mechanistic analysis, e.g., in what sense is visual encoding better than directly feeding the interpolated series and mask into a temporal encoder?
- Rendering itself also introduces loss and bias, which the paper does not discuss:
- Values are quantized to pixel resolution, and the more variables there are, the lower the per-cell resolution;
- Global coordinate bounds are taken from the entire training set, making them sensitive to outliers and distribution shift;
- The grid layout introduces artificial spatial adjacency: variables in neighboring cells have no semantic relation, yet ViT's spatial attention treats them as neighbors.
- The technical contribution is mainly a combination of mature components, without matching depth of analysis.
- The denoising network follows the CSDI-style temporal/feature Transformer; "learnable queries cross-attending to external tokens" is standard practice in Perceiver, DETR, and Q-Former; the auxiliary reconstruction loss on the global token is merely a conventional regularizer, and its output is not used at inference.
- The methodological novelty lies in "plugging ViT features into CSDI," yet the integration is not carefully examined: why does fine-tuning only the last two layers avoid "attention collapse"? How do the temporal anchor queries correspond to actual time positions? Is this purely learned, or is there a positional prior? What justifies the division of labor between the global token and the patch tokens?
- The positioning relative to related work is also insufficient. The paper does not discuss works that convert time series into images and use visual diffusion models for generation and imputation (e.g., ImagenTime, Naiman et al., NeurIPS 2024), nor works that use pretrained vision models for time series tasks (e.g., VisionTS). The feature comparison in Table 1 is overly simplified and does not establish the distinctiveness of this work.
Questions for authors
Clarifications (answers may affect my rating):
- Since the rendered image is a deterministic function of the input, please argue from an information or representation perspective: what structure can visual encoding provide that a temporal encoder would in principle find hard to learn? Why do ImageNet-pretrained priors apply to line charts?
- Inside long gaps, the shape in the image is just linear interpolation. How exactly does the visual condition help recover non-linear variations inside the gap? Please provide a mechanistic explanation or targeted analysis.
- The grid layout places different variables in adjacent cells. Does ViT's spatial attention introduce meaningless cross-variable associations? How does this compare with "rendering each variable separately with a shared encoder"?
- Please clarify the inference procedure: is rendering done with mask or ? Does the merge in line 8 of Algorithm 2 overwrite evaluation positions? How should the inconsistency between line 11 of Algorithm 1 and Eq. 7 be understood?
- A brief experimental question: could you add controls with "a non-visual encoder of equal capacity" or "a randomly initialized ViT" to separate the effect of the visual modality from model capacity? How should some results (e.g., a CRPS of 0.003 on Stock, and errors that do not vary monotonically with gap length) be explained?
Suggestions:
- Compare with other imaging schemes (GAF, RP, delay embedding) to justify the choice of line charts.
- Reduce rhetorical expressions and add quantitative analysis of how the conditioning signal works, e.g., the mutual information between the condition and the true missing values, or probing analyses.
7 comments
Responses to Q4, Q5 and Suggestions
Q4: Mask Usage and Inference Procedure.
A4: We sincerely thank the reviewer for identifying these inconsistencies.
In our formulation, denotes the original observation mask, whereas denotes the conditional mask obtained by artificially hiding a subset of originally observed values. During training and masked evaluation, rendering is based on the conditional observations , rather than the complete observations indicated by . We acknowledge that Algorithm 2 does not explicitly distinguish these masks. In particular, using in Line 8 would overwrite artificially masked evaluation positions. The merging operation should instead use the conditional mask :
We have corrected Algorithm 2 accordingly. Regarding Algorithm 1, Line 11 omits the mask weighting and normalization specified in Eq. (7). We have revised the pseudocode to match the original observation-weighted reconstruction objective.
Q5: Visual Contribution and Performance Analysis.
A5: We sincerely thank the reviewer for this constructive comment. To disentangle visual representation, model capacity, and pretraining effects, we compared a numerical encoder, a randomly initialized ViT, and a pretrained ViT under identical experimental settings. The results under 50% point missingness (RMSE/MAE) are presented as follows:
| Conditioning Variant | ETTh1 | ILI | PEMS-Traffic |
|---|---|---|---|
| Numerical Conditioning | 0.354/0.256 | 0.600/0.324 | 0.354/0.136 |
| Randomly Initialized Visual Conditioning | 0.362/0.257 | 0.441/0.223 | 0.423/0.221 |
| Pretrained Visual Conditioning (Ours) | 0.302/0.174 | 0.397/0.196 | 0.337/0.126 |
The results show that pretrained visual conditioning consistently achieves lower RMSE and MAE across all three datasets, demonstrating the importance of visual pretraining and its effectiveness beyond model capacity and auxiliary supervision.
Regarding the performance on Stock, the low CRPS may be partly attributed to the complementary information among its six financial variables. Open, High, Low, Close, and Adjusted Close capture closely related aspects of stock price movements, while Volume provides additional information about trading activity. These inter-variable relationships may provide useful contextual information for reconstructing missing values, contributing to the model's strong imputation performance. We have added the controlled comparison to Section 4.2 and the discussion of Stock performance to Appendix E.1, with the revisions highlighted in blue.
Q6: Comparison of Line Charts with Alternative Time-Series Representations.
A6: We sincerely thank the reviewer for this valuable suggestion. Following this suggestion, we compared line charts with three alternative imaging schemes, namely GAF, RP, and Delay Embedding, under identical experimental settings. The results on ETTh1, ILI, and PEMS-Traffic under 50% point missingness (RMSE/MAE) are presented as follows:
| Visual Representation | ETTh1 | ILI | PEMS-Traffic |
|---|---|---|---|
| GAF | 0.346/0.216 | 0.459/0.236 | 0.586/0.291 |
| RP | 0.343/0.205 | 0.807/0.491 | 0.496/0.219 |
| Delay Embedding | 0.319/0.179 | 0.409/0.247 | 0.393/0.149 |
| Visual Conditioning (Ours) | 0.302/0.174 | 0.397/0.196 | 0.337/0.126 |
As shown in the table, line-chart-based visual conditioning achieves the lowest RMSE and MAE across all three datasets. This advantage may be attributed to its direct preservation of temporal order, value trajectories, and local geometric patterns. These results support the use of line charts as an effective representation for our visual conditioning framework. We have added this comparison and the corresponding discussion to the Appendix, with the revisions highlighted in blue.
Q7: Quantitative Analysis of the Information Encoded in the Conditioning Signal.
A7: We sincerely thank the reviewer for this valuable suggestion. In VT-Diff, the conditioning signal c is generated through cross-attention between learnable temporal queries and visual patch tokens. Each query corresponds to a specific time position and retrieves relevant visual features, providing temporally aligned structural information to guide the diffusion denoising process.
Our existing comparison between visual and numerical conditioning provides indirect quantitative evidence of the effectiveness of this mechanism. As shown in the controlled comparison reported in Q5, visual conditioning achieves lower RMSE and MAE across ETTh1, ILI, and PEMS-Traffic under identical experimental settings. However, we acknowledge that these results do not directly quantify the information encoded in c, as suggested by the reviewer. We have acknowledge the lack of direct information-level analysis as a potential limitation.
Responses to W1, W2
Q8: Rendering-Induced Information Loss and Bias.
A8: We sincerely thank the reviewer for highlighting these important limitations. We acknowledge that pixel quantization may lose fine-grained numerical details, training-set coordinate bounds may be sensitive to outliers and distribution shifts, and grid layouts may introduce artificial spatial proximity between unrelated variables. In our framework, visual rendering provides auxiliary structural guidance, while numerical reconstruction remains in the original time-series domain. Nevertheless, these rendering-related effects may influence visual feature extraction. We have clarified these trade-offs and discussed them as potential limitations in the Appendix, with the revisions highlighted in blue.
Q9: Technical Novelty and Component Contributions.
A9: We sincerely thank the reviewer for highlighting these important limitations. We acknowledge that pixel quantization may lose fine-grained numerical details, training-set coordinate bounds may be sensitive to outliers and distribution shifts, and grid layouts may introduce artificial spatial proximity between unrelated variables. Although visual rendering serves only as an auxiliary source of structural guidance and numerical reconstruction remains in the original time-series domain, these rendering-related effects may still influence visual feature extraction. We have therefore acknowledged these issues as potential limitations and discussed them in the Appendix, with the revisions highlighted in blue.
Q10: Rationale for Visual-Temporal Integration.
A10: We sincerely thank the reviewer for this insightful comment. Regarding the ViT fine-tuning strategy, we freeze most pretrained parameters and fine-tune the patch embeddings and the final two Transformer layers with a smaller learning rate. This strategy aims to preserve pretrained visual representations while allowing task-specific adaptation. It is intended to reduce the risk of disrupting pretrained features during cross-modal alignment, rather than guarantee the avoidance of attention collapse.
For temporal alignment, the L learnable queries are assigned to the L temporal positions in sequential order. Their representations are learned during training, without an additional handcrafted positional prior. Each query generates a corresponding time-step-wise condition through cross-attention with visual patch tokens.
The global token and local patch tokens serve complementary roles. The global token captures macroscopic trajectory information and is used for auxiliary trend reconstruction during training, whereas local patch tokens preserve fine-grained variations and generate the conditioning signal for diffusion denoising. Although the global token is not directly used during inference, its reconstruction objective regularizes the shared visual encoder.
We have revised the manuscript to provide a clearer explanation of the ViT fine-tuning strategy, the temporal alignment mechanism, and the complementary roles of global and local visual representations.
Q11: Comparison with Related Visual Time-Series Methods.
A11: We sincerely thank the reviewer for pointing out these relevant studies.
ImagenTime (Naiman et al., NeurIPS 2024) employs invertible transformations to convert time series into image representations and utilizes visual diffusion models for time-series generation and imputation. VisionTS explores the use of pretrained vision models for time-series forecasting. These approaches differ from VT-Diff in their task objectives and the role of visual representations. Specifically, VT-Diff focuses on probabilistic time-series imputation. It uses line-graph representations of incomplete observations to extract visual structural features, which are transformed into time-aligned conditioning signals for numerical-domain diffusion. Unlike approaches that perform generation or reconstruction primarily in the image domain, VT-Diff retains numerical-domain diffusion to ensure precise value reconstruction.
We have expanded the Related Work section to discuss ImagenTime and VisionTS, and revised Table 1 to provide more detailed comparisons of their tasks, visual representations, and roles of visual models.
Responses to Q1, Q2 and Q3
We sincerely appreciate the reviewer's careful evaluation and constructive feedback. Thank you for highlighting these important issues. We address each comment below.
Q1: Visual Encoding Advantages and Pretraining Transferability.
A1: We sincerely thank the reviewer for this insightful question. We agree that visual rendering introduces no additional information beyond the masked observations and interpolated values, but provides a different structural inductive bias. By arranging variable-wise trajectories into a 2D grid, VT-Diff enables the ViT to capture geometric patterns, including local fluctuations, turning points, and global trends. Although temporal encoders can theoretically learn similar patterns, it is difficult for them to infer these geometric relationships, particularly when missing observations disrupt temporal continuity. To validate its effectiveness, we compare VT-Diff with a capacity-matched numerical encoder under 50% point missingness. The results (RMSE/MAE) are presented as follows:
As shown in the table, visual conditioning consistently outperforms numerical conditioning, supporting the effectiveness of the proposed visual representation. Regarding ImageNet pretraining, we leverage transferable low-level visual features, such as edges, orientations, and contours, rather than natural-image semantics. The attention maps in Fig. 6 provide qualitative evidence that the ViT attends to observation-dependent visual regions in time-series line charts, supporting its ability to capture relevant visual structures.
We have further clarified the motivation for visual encoding and the role of ImageNet pretraining in the revised manuscript.
Q2: Nonlinear Reconstruction in Long Missing Gaps.
A2: We sincerely thank the reviewer for this important question. We acknowledge that linear interpolation does not provide the true nonlinear structure within missing intervals. In VT-Diff, the interpolated segments serve as a visual representation rather than the final imputation results.
Specifically, the ViT extracts structural features from available observations, while the Cross-Attention Bottleneck aligns local visual features with temporal positions. These features condition the diffusion model, which learns temporal dependencies from training data and generates plausible nonlinear variations within missing regions. Thus, nonlinear reconstruction arises from the learned conditional generative process rather than the interpolated lines themselves. We have further clarified this mechanism in Section 3.2 (Visual Rendering of Time Series), with the revision highlighted in blue.
Q3: Potential Spurious Cross-Variable Associations in Grid Layouts and the Choice of Independent Variable Encoding.
A3: We sincerely thank the reviewer for this thoughtful question. The grid layout provides a consistent spatial organization rather than assuming that adjacent variables are correlated. Each variable is rendered in a separate cell with a distinct color, while ViT self-attention enables interactions across all visual patches, allowing the model to learn potentially useful cross-variable relationships.
Compared with encoding each variable separately using a shared encoder, our approach enables joint visual modeling of both individual trajectories and cross-variable context. Separate encoding can avoid direct spatial interactions between unrelated variables, but also limits cross-variable feature learning within the visual encoder.
Nevertheless, the grid layout has a limitation: spatial adjacency may introduce spurious cross-variable associations that do not reflect actual variable relationships. We have added a discussion of this limitation to appendix, with the revision highlighted in blue.