acceptodds
Under review as a conference paper at ICLR 2027

C4TS: Co-Encoding and Co-Decoding via Temporal-Visual Cross-Modal Collaboration for Time Series Forecasting

Abstract

Visual representations of time series have been widely explored to improve forecasting. In temporal–visual feature-fusion approaches, interaction between parallel temporal and visual encoders feeds a single temporal decoding pathway, yet does not inherently reconcile the structural differences between temporal token sequences and spatially organized visual features. In temporal–visual prediction-fusion approaches, parallel decoding preserves both forecasting pathways, but globally shared weights cannot adapt their combination to individual input instances and future positions. We propose Co-Encoding and Co-Decoding via Temporal–Visual Cross-Modal Collaboration for Time Series Forecasting (C4TS), a framework that coordinates two forecasting pathways through both collaborative encoding (co-encoding) and collaborative decoding (co-decoding). For co-encoding, Spatial-Aware Visual Adaptation bridges the structural gap between temporal and visual representations by adapting spatially organized visual features into a cross-attention memory, enabling visual features to enhance temporal representations. For co-decoding, Instance × Future-Patch Adaptive Fusion assigns distinct fusion weights to each input instance and future patch, adaptively combining temporal predictions with visual reconstruction forecasts. These weights are generated by learnable future-patch queries that retrieve historical evidence separately from the temporal and visual representations. Experiments on public time series benchmarks demonstrate that C4TS achieves state-of-the-art overall forecasting performance.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Forecasting, Multimodal Learning, Cross-Modal Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WB7mOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

C4TS combines a PatchTST-style temporal pathway with a VisionTS-style visual pathway: a pretrained MAE reconstructs a masked, period-structured image, with only LayerNorm fine-tuned. The two pathways interact at two points. (i) Co-encoding: Spatial-Aware Visual Adaptation (Sec. 3.3, Eqs. 3–8) restores visible MAE tokens to their 2D grid, applies two dilated depthwise-conv residual blocks plus row/column embeddings, and uses the result as cross-attention memory for the temporal tokens (visual→temporal only). (ii) Co-decoding: Instance × Future-Patch Adaptive Fusion (Sec. 3.4, Eqs. 9–13) uses K learnable future-patch queries. These read stop-gradient temporal and visual memories and produce a sigmoid weight α_k per instance and future patch, broadcast across variables, to mix Ŷ_T and Ŷ_V. Training supervises both pathways and the fused output (Eq. 14). With L = 512 on 7 TSLib benchmarks, C4TS has the best average rank (2.071) and the lowest overall average MSE (0.298) among 9 baselines (Table 1). ETT ablations are in Table 2, and attention and gate visualizations in Fig. 3.

Critical strengths
  • [Importance of the research problem] How to combine numerical forecasting with MAE-based visual reconstruction (VisionTS) is a timely question, and Fig. 1 positions it clearly against feature-level fusion (Time-VLM) and global prediction fusion (DMMV).
  • [Novelty of the approach] Two well-motivated modules: Spatial-Aware Visual Adaptation, which turns the 2D MAE token grid into a cross-attention memory (Sec. 3.3), and Instance × Future-Patch Adaptive Fusion with learnable future-patch queries (Sec. 3.4, Eqs. 9–13).
  • [Soundness and feasibility] The engineering is careful. The gate is zero-initialized so α starts at 0.5, gate inputs use stop-gradient, only visible historical tokens enter the adapter, and only the MAE LayerNorm parameters are fine-tuned.
  • [Experiments and practical impact] All methods use a common L = 512, with reruns of DMMV, VisionTS, VPNet and FreqCycle at that length (Sec. 4.1). C4TS has the best average rank (2.071) and the lowest overall average MSE (0.298, Table 1).
  • [Experiments and practical impact] The ablations are informative about direction: adding temporal→visual attention hurts (+2.747%, Table 2).
  • [Theoretical and application-level implications] The results indicate that temporal and visual-reconstruction forecasters have complementary inductive biases. This is a useful insight for ensemble design.
  • [Theoretical and application-level implications] Per-patch weights could serve as a horizon-level monitoring signal.
Critical weaknesses
  • [Importance of the research problem] It is a fairly narrow architectural question within a fast-moving niche rather than a broadly important problem.
  • [Novelty of the approach] Each module is an incremental variant of known ideas (cross-modal attention; input-dependent gating). The main step beyond DMMV is replacing a global scalar weight with an input- and horizon-dependent gate.
  • [Soundness and feasibility] The fusion weight α_k is averaged over variables (Eq. 12) and broadcast across channels, even though channels in Weather, Electricity and Traffic differ greatly in periodicity.
  • [Soundness and feasibility] There are no parameter counts, runtime or memory figures, despite running a ViT-scale MAE per variable.
  • [Soundness and feasibility] Code is not available at submission.
  • [Experiments and practical impact] Margins are small (0.298 vs. 0.302 for APRNet and 0.304 for VisionTS and PatchTST), with no seeds or standard deviations.
  • [Experiments and practical impact] The module ablations are each under 1% (no spatial adaptation +0.603%, no feature interaction +0.302%, uniform averaging +0.748%), and they cover ETT only.
  • [Experiments and practical impact] Most of the gain comes from averaging the two pathways (0.3481 → 0.3346 with uniform averaging).
  • [Experiments and practical impact] Some baseline reproductions look anomalous (DMMV ETTh2 MSE 0.574, average rank 9.429).
  • [Theoretical and application-level implications] There is no account of when the visual pathway should dominate. Fig. 3 shows that the weights vary, not that they track relative error.
  • [Theoretical and application-level implications] Deployment cost per variable is high relative to an overall MSE gain of about 0.004, and the image construction assumes a known period.
Questions for authors
  1. Can you provide mean ± std over multiple seeds for Table 1 and Table 2, and paired tests for the co-decoding ablations?
  2. What is the MSE of an ensemble of a separately trained PatchTST (L = 512) and VisionTS, averaged uniformly or with a global weight?
  3. Why does DMMV degrade so much at L = 512 (e.g., ETTh2)? Did you try its native look-back?
  4. Did you try per-variable α_{c,k}? How do results change on high-dimensional datasets (ECL, Traffic)?
  5. Is there a correlation between the gate's α_k and the relative per-patch errors of the two pathways on held-out data?
  6. What are the parameter count and wall-clock cost compared with PatchTST and VisionTS?
8 comments
AuthorsOpenReview ↗

Responses to Critical Weaknesses for Reviewer WB7m - Critical Weakness 1

Dear Reviewer WB7m,

Thank you for your detailed assessment and constructive suggestions. We have expanded the evaluation with four-seed ablations, additional fusion-granularity controls, independent-pathway ensembles, a random-initialization control for the MAE, and parameter and inference-cost measurements. We also provide DMMV reproduction logs and a minimal C4TS evaluation package in the supplementary material.

Below, we respond point by point to your comments. W1–W5 correspond to the five thematic categories under “Critical Weaknesses,” in their original order. Within each category, we address its individual comments separately. Q1–Q6 correspond to the six items under “Questions For Authors,” in their original order. Section and table numbers refer to the revised manuscript.

W1. Importance of the research problem

Time series forecasting supports decision-making in a wide range of applications, including energy management, transportation, and weather monitoring. Improving the ability to predict future behavior from historical observations therefore has practical value across multiple domains.

Architectural advances remain an important way to improve forecasting. Different architectures encode different assumptions about temporal structure and determine which patterns can be extracted and used for prediction. In particular, numerical forecasting and visual reconstruction provide complementary ways to process the same observations: one directly models numerical sequences, while the other exposes temporal patterns through a structured image representation. Understanding how these forecasting mechanisms can work together is therefore a meaningful architectural question.

C4TS investigates this question beyond simply adding visual features to a temporal predictor or averaging two forecasts. Its central objective is to enable collaboration between numerical forecasting and visual reconstruction while preserving both as complete forecasting pathways. This raises two concrete design questions:

• Co-encoding: How can historical visual representations be adapted to enhance temporal forecasting through feature interaction while preserving the MAE’s visual reconstruction mechanism?

• Co-decoding: How can the two completed forecasts be combined adaptively across input instances and future prediction positions?

Sections 3.2–3.4 address these questions through spatially adapted co-encoding and instance × future-patch co-decoding. The revised experiments further distinguish the contributions of individual pathways, independent ensembles, feature interaction, and adaptive prediction fusion. In particular, the independent uniform and validation-weighted ensembles provide direct controls for determining whether the collaborative framework offers benefits beyond combining separately trained forecasters.

The value of this study therefore lies in both the resulting forecasting model and the experimental examination of how two different forecasting mechanisms can collaborate at the representation and prediction levels. The seven-dataset evaluation assesses forecasting performance, while the expanded controlled ablations examine the architectural choices underlying that collaboration.

AuthorsOpenReview ↗

Responses to Critical Weaknesses for Reviewer WB7m - Critical Weakness 2

W2. Novelty of the approach and differences from DMMV

The architectural contribution of C4TS is to coordinate numerical forecasting and visual reconstruction at both the representation and prediction levels while preserving two complete forecasting pathways. This is realized through two complementary designs:

• Co-encoding: Spatial-Aware Visual Adaptation restores and processes the two-dimensional grid of historical MAE tokens, constructing a visual memory that enhances temporal representations through cross-attention. The original MAE encoding remains available to the visual decoder, preserving its reconstruction-based forecasting mechanism. • Co-decoding: Learnable future-patch queries retrieve information from the temporal and visual historical representations and generate fusion weights for each input instance and future patch. These weights directly combine the two completed forecasts over the prediction horizon. • Cross-attention and gating are the underlying operations. The contribution concerns how historical visual information is adapted for interaction, how both forecasting mechanisms are retained, and how their outputs are coordinated across future prediction positions. Sections 3.2–3.4 describe these architectural choices.

Compared with DMMV, the differences concern both co-encoding and co-decoding.

First, DMMV does not include the co-encoding mechanism used in C4TS. DMMV connects its forecasting pathways through seasonal–trend decomposition or a visual-backcast residual formulation and combines their outputs through prediction-level fusion. It does not construct a spatially adapted visual memory that temporal tokens query through cross-attention.

C4TS explicitly introduces this feature-interaction pathway. Spatial residual blocks and row–column positional embeddings adapt the historical visual token grid before temporal tokens read it. Meanwhile, the original MAE encoding continues to support visual reconstruction. This allows visual information to enhance temporal forecasting without replacing the visual forecasting pathway.

Second, the fusion mechanisms differ in the dimensions along which their weights can vary. To make this distinction precise, we use the notation [I, F, V, B]. The first three dimensions indicate whether weights vary across input instances, forecast positions or patches, and variables, respectively. A value of 1 indicates sharing along that axis. B describes the branch-weight parameterization: one coefficient determining complementary weights, or two separately learned coefficients.

The comparison below also includes an additional instance × variable × future-patch three-dimensional gate, which we evaluate as an ablation and discuss in detail in W3.

Fusion mechanismRepresentationWeight sharing
Global single-parameter fusion[1, 1, 1, 1]Shared across instances, positions, and variables
Released DMMV-S[1, 1, V, 1]Variable-specific; shared across instances and forecast positions
Released DMMV-A[1, 1, V, 2]Two coefficients per variable; shared across instances and forecast positions
C4TS[I, F, 1, 1]Instance- and future-patch-dependent; shared across variables
Additional three-dimensional gate[I, F, V, 1]Instance-, future-patch-, and variable-dependent

Throughout our manuscript, DMMV refers to DMMV-A. DMMV’s fusion weights are learned during training and shared across input instances and forecast positions at inference.

The distinction is that C4TS generates fusion weights conditioned on the current input and future patch, allowing the contributions of the two pathways to vary across instances and future prediction positions.

Specifically, the future-patch queries in Section 3.4 read the temporal and visual historical representations to generate a visual weight for each instance–future-patch pair. That weight directly combines the numerical forecast and visual reconstruction for the corresponding future patch.

Thus, the advance beyond DMMV is not limited to replacing its fusion coefficients. C4TS introduces spatially adapted feature interaction together with future-patch-conditioned prediction fusion, enabling collaboration during encoding while retaining and adaptively combining both complete forecasting pathways during decoding.

AuthorsOpenReview ↗

Responses to Critical Weaknesses for Reviewer WB7m - Critical Weakness 3

W3. Soundness and feasibility

a) Shared fusion weights across variables

Within each dataset, all variables are recorded at the same sampling interval. Consequently, common calendar cycles, such as daily cycles, correspond to the same number of time steps across channels. This provides a consistent temporal basis for constructing period-structured images and defining future patches.

The shared gate in C4TS does not require identical temporal characteristics across variables. Each variable retains its own temporal forecast and visual-reconstruction forecast; what is shared is the relative weighting of these two forecasting mechanisms within an instance–future-patch pair. The gate aggregates information across variables to determine this weighting, and its output changes with the input instance and future patch.

To directly evaluate whether variable-specific weights improve this design, we added two fusion controls in Section 4.3 and Appendix C, Table 5. Variable-wise fusion learns a separate coefficient for each variable, shared across instances and future patches. The instance × variable × future-patch three-dimensional gate, also listed in the comparison in W2, generates a separate coefficient for every variable within each instance–future-patch pair.

Across the four ETT datasets, four forecasting horizons, and four random seeds, variable-wise fusion achieves an average MSE of 0.3339, and the three-dimensional gate achieves 0.3328, compared with 0.3324 for the adopted instance × future-patch gate. Thus, explicitly allowing variable-specific fusion does not improve the overall ETT result. These controlled comparisons support sharing the fusion weights across variables while adapting them to the current input and future prediction position.

Given the substantially larger scale of Weather, Electricity, and Traffic, repeating the full ablation suite across multiple random seeds on these datasets would incur prohibitively high computational costs. We therefore conduct the systematic fusion-granularity comparisons on ETT and evaluate overall forecasting performance on all seven datasets.

b) Parameter counts, inference latency, and memory consumption

We have added parameter counts and measured inference costs in Appendix D, Table 7. The comparison includes C4TS, its temporal and visual branches, VisionTS, DMMV, and Time-VLM. For example, the ETTh1 results are:

ModelTotal parametersTrainable parametersMedian latencyPeak memory
C4TS118.56M6.71M26.41 ms510.14 MiB
C4TS temporal branch6.17M6.17M3.12 ms35.13 MiB
C4TS visual branch111.91M0.06M19.00 ms485.39 MiB
VisionTS111.91M0.06M23.47 ms485.49 MiB
DMMV114.49M2.59M43.28 ms511.09 MiB
Time-VLM131.18M19.59M21.82 ms532.50 MiB

Each entry reports the maximum of that metric across the four forecasting horizons. Measurements use the same GPU, an input length of 512, and a test batch size of 1. For each horizon, latency is the median of 50 forward passes following 10 warm-up passes. Appendix D documents the execution settings.

The C4TS temporal branch serves as the temporal-only reference. It is a separate implementation inspired by PatchTST-style modeling, rather than the official PatchTST implementation. Holding this temporal architecture fixed directly quantifies the additional cost of introducing the visual pathway and collaboration modules.

The MAE parameters are shared across variables, rather than replicated into a separate parameter set for each channel. Processing more variables nevertheless increases computation and activation memory. Most MAE parameters remain frozen, which explains the difference between total and trainable parameter counts. The measurements make these computational costs explicit alongside the forecasting results.

c) Code availability and reproducibility

The supplementary c4ts_minimum package provides a minimal C4TS evaluation setup for verification during the review process. It includes model inference code, data-loading utilities, environment specifications, and instructions for obtaining the checkpoint and dataset. The package supports evaluation on ETTh1 with a forecast horizon of 96 and provides reference metrics and checksum verification.

We also provide the dmmv_logs directory, containing all run settings, execution logs, and recorded results from our DMMV reproduction experiments.

As stated in the manuscript’s reproducibility statement, we will release the complete code, training and evaluation scripts, and experimental configurations upon acceptance.

Reviewer_mgNuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes C4TS, a long-term time series forecasting framework that combines temporal modeling and visual reconstruction. The model adopts a dual-path architecture, utilizing a Transformer and a pretrained Masked Autoencoder (MAE) for numerical forecasting and visual reconstruction, respectively. The authors introduce a Co-Encoding module that incorporates visual features into temporal representations through Spatial-Aware Visual Adaptation and Cross-Attention. Additionally, a Co-Decoding module employs a Future-Patch Query Gate to generate adaptive fusion weights for different input instances and future patches, thereby combining the predictions from both pathways. Experiments on widely used benchmarks demonstrate competitive overall forecasting performance.

Critical strengths

Strength 1

The paper combines a numerical Temporal Branch with a pretrained MAE-based Visual Branch, generating predictions in the numerical and image spaces, respectively. Unlike methods that perform only feature-level fusion, C4TS preserves two complete forecasting pathways and enables collaboration at both the feature and prediction levels through Co-Encoding and Co-Decoding.

Strength 2

Figure 1 effectively illustrates the differences between C4TS and existing feature-level and prediction-level fusion approaches. Figure 2 also provides a clear overview of the proposed architecture. However, Figure 2(b), labeled Co-encoding, appears to omit the temporal pathway and only illustrates the Spatial-Aware Visual Adaptation module, which may cause some confusion.

Critical weaknesses

Weakness 1

Limited cross-variate dependency modeling. The Temporal Branch, Visual Branch, and Cross-Attention in Co-Encoding all operate in a channel-independent manner. Interactions across variables mainly occur through Variable Pooling in the Gate. Consequently, cross-variate information can only indirectly adjust the fusion weights of two independently generated forecasts, rather than directly enhancing the predictive representations of individual variables. This may limit the model's ability to capture complex cross-variate dependencies. It could also partly explain the relatively weak performance of C4TS on the Traffic dataset, which contains more than 800 variables, where it is even outperformed by some purely temporal forecasting models.

Weakness 2

Marginal improvements from the key components. In Table 2, the Co-Encoding and Co-Decoding ablation studies (the two core components of this paper) show MSE differences on the order of 10^{-3}. In practice, simply changing the random seed can often lead to MSE variations of a similar magnitude. Therefore, based on the ablation results alone, the improvements brought by these two core components appear rather marginal, raising concerns about their necessity and potential redundancy.

Questions for authors

Question 1

The paper compares C4TS with various temporal forecasting baselines. However, incorporating a pretrained visual backbone may introduce substantial parameter and computational overhead. Could the authors provide a comparison of parameter counts and inference latency between C4TS and the temporal baselines to better quantify the additional cost introduced by the visual pathway?

Question 2

Have the authors considered prior work that similarly incorporates visual information into temporal representations through Cross-Attention, such as AURORA: Towards Universal Generative Multimodal Time Series Forecasting (ICLR 2026)? Although AURORA primarily focuses on zero-shot and few-shot time series forecasting, whereas C4TS adopts a full-shot supervised setting, the two methods share similarities in their cross-modal feature interaction mechanisms. It would be valuable to discuss this related work and further clarify the architectural differences and novel contributions of the proposed Co-Encoding module.

4 comments
AuthorsOpenReview ↗

Responses to Critical Weaknesses for Reviewer mgNu - Critical Weakness 1

Dear Reviewer mgNu,

Thank you for your careful assessment and constructive suggestions. We have expanded the evaluation with four-seed ablations, additional variable-specific fusion controls, and parameter and inference-cost measurements. We have also added a discussion of AURORA.

Below, we respond point by point to your comments: W1–W2 correspond to the items under “Critical Weaknesses,” and Q1–Q2 correspond to the items under “Questions For Authors,” in their original order. Section and table numbers refer to the revised manuscript.

W1. Cross-variate dependency modeling and performance on Traffic

C4TS adopts channel-independent forecasting pathways to study collaboration between numerical forecasting and visual reconstruction. Each variable is processed using shared model parameters, while the gate aggregates information across variables to determine the relative contribution of the two pathways for each input instance and future patch. This design focuses on collaboration between forecasting mechanisms rather than explicitly constructing cross-variate predictive representations.

To evaluate whether sharing fusion weights across variables restricts performance, we have added two controls in Section 4.3 and Appendix C, Table 5:

• Variable-wise fusion: learns a separate fusion coefficient for each variable, shared across instances and future patches.

• Instance × variable × future-patch fusion: generates a separate fusion weight for each variable within every instance–future-patch pair.

Across four ETT datasets, four forecasting horizons, and four random seeds, the variable-wise gate achieves an average MSE of 0.3339, and the instance × variable × future-patch three-dimensional gate achieves 0.3328, compared with 0.3324 for the adopted instance × future-patch gate. Adding variable-specific weights therefore does not improve the overall ETT result. These controls provide an empirical basis for sharing weights across variables while adapting them to the current instance and future patch.

These experiments specifically test whether each variable benefits from its own mixture of the two forecasts. Modeling interactions between variables inside the forecasting encoders would address a different question. The gate comparisons support the chosen fusion granularity without implying that cross-variate dependencies cannot be useful.

The variable-specific gate experiments currently cover ETT. Given the substantially larger scale of Weather, Electricity, and Traffic, repeating the full ablation suite across multiple random seeds on these datasets would incur prohibitively high computational costs. We therefore use the four ETT datasets for systematic multi-seed architectural comparisons and the seven-dataset main table for broader forecasting evaluation.

However, a large number of variables does not necessarily mean that all pairwise relationships are equally informative for forecasting. For example, Electricity contains individual customers’ electricity-consumption series: one customer’s consumption need not directly determine another’s. Similarly, Traffic contains measurements from different road locations, and the presence of 862 variables does not imply that every pair of locations has a strong, stable predictive relationship. Channel-independent modeling, as exemplified by PatchTST, provides an established alternative: it processes each channel separately with shared parameters, without requiring explicit interactions between all variables.

Empirically, on Traffic, which contains 862 variables, C4TS ranks second in average MSE and third in average MAE among the compared methods. The top three methods under each metric in the main table are:

RankMethod by MSEAverage MSEMethod by MAEAverage MAE
1VPNet0.379FreqCycle0.243
2C4TS0.386VisionTS0.255
3VisionTS / APRNet0.388C4TS0.257

VisionTS and APRNet tie at the third-lowest reported MSE. Although C4TS does not achieve the best result on Traffic, its second-place MSE and third-place MAE demonstrate competitive performance on this high-dimensional dataset.

AuthorsOpenReview ↗

Responses to Critical Weaknesses for Reviewer mgNu - Critical Weakness 2

W2. Magnitude and reliability of the improvements from co-encoding and co-decoding

We have repeated the full model and ablation configurations with four random seeds across the four ETT datasets and four forecasting horizons. Section 4.3 and Table 2 report the aggregate results and percentage changes, while Appendix C, Tables 4–6, provide the detailed mean and standard deviation for each dataset–horizon setting.

The magnitude of an improvement should be considered in the context of strong forecasting baselines. For context, we calculated a combined error score by averaging the mean MSE and mean MAE over the seven benchmark datasets. We then measured each method’s relative improvement over the strongest baseline under this score in its own paper’s comparison table. Under this common aggregation rule, VisionTS and DMMV-A achieve improvements of approximately 1.77% and 1.63%, compared with 1.16% for C4TS. These contextual comparisons use the respective papers’ evaluation protocols and illustrate that improvements over strong baselines in this setting are often on this scale.

Moreover, a component ablation measures an incremental contribution within an otherwise unchanged model, rather than the difference between two complete forecasting systems. Within this controlled setting, the percentage improvements averaged over four random seeds provide empirical support for the effectiveness of the proposed modules, even when their magnitude is below 1%. Their value should therefore be assessed through controlled comparisons and multi-seed results, rather than an absolute improvement threshold alone.

Specifically, relative to full C4TS, removing spatial mixing increases average MSE by 0.594%, while removing spatial adaptation and feature interaction increases it by 0.233%. Replacing the proposed gate with uniform averaging or a jointly learned global scalar increases average MSE by 0.637% and 0.624%, respectively. These percentages are calculated from the unrounded four-seed aggregate results. Together with the per-setting means and standard deviations reported in Appendix C, these controlled comparisons provide empirical support for the contributions of spatial adaptation, feature interaction, and instance × future-patch fusion. Further ablation results are provided in Section 4.3, Table 2, and Appendix C of the revised manuscript.

Given the substantially larger scale of Weather, Electricity, and Traffic compared with ETT, repeating the full ablation suite across multiple random seeds on these datasets would incur prohibitively high computational costs; we therefore conduct the multi-seed evaluation and ablation studies on the four ETT datasets.

AuthorsOpenReview ↗

Responses to Questions for Authors for Reviewer mgNu - Question 1

Q1. Parameter counts and inference latency relative to temporal forecasting models

We have added parameter counts and measured inference costs in Appendix D, Table 7. The comparison includes C4TS, its temporal and visual branches, VisionTS, DMMV, and Time-VLM.

We use the C4TS temporal branch as the temporal-only reference. It is a separate implementation inspired by PatchTST-style modeling, rather than the official PatchTST implementation. This reference holds the temporal architecture fixed and directly measures the additional cost of introducing the visual pathway and collaboration modules.

For example, the ETTh1 entries are:

ModelTotal parametersTrainable parametersMedian latencyPeak memory
C4TS118.56M6.71M26.41 ms510.14 MiB
C4TS temporal branch6.17M6.17M3.12 ms35.13 MiB
C4TS visual branch111.91M0.06M19.00 ms485.39 MiB
VisionTS111.91M0.06M23.47 ms485.49 MiB
DMMV114.49M2.59M43.28 ms511.09 MiB
Time-VLM131.18M19.59M21.82 ms532.50 MiB

Each entry reports the maximum of that metric across the four forecasting horizons, as specified in Table 7. Measurements use the same GPU, an input length of 512, and a test batch size of 1. For each horizon, latency is the median of 50 forward passes following 10 warm-up passes. Appendix D records the execution settings.

These measurements quantify two complementary comparisons. Relative to the temporal-only reference, the visual pathway introduces additional parameters and computation. Relative to VisionTS, the full collaborative framework has a smaller cost increment. Most MAE parameters remain frozen, explaining the difference between C4TS’s total and trainable parameter counts.

The MAE parameters are shared across variables rather than replicated into separate parameter sets. Processing additional variables nevertheless increases computation and activation memory. Table 7 provides measurements across all four ETT datasets, making the accuracy–cost tradeoff explicit under the reported settings.

AuthorsOpenReview ↗

Responses to Questions for Authors for Reviewer mgNu - Question 2

Q2. Architectural differences from AURORA

Thank you for highlighting AURORA[1]. We have added its citation and expanded the discussion in Section 2.3. Both methods use cross-attention to incorporate visual information into temporal representations. The distinction concerns how the visual memory is constructed and how this interaction participates in the overall forecasting architecture.

First, the visual representations are organized differently before interaction. AURORA distills visual and textual tokens using learnable queries, constructs modality-guided temporal self-attention, and subsequently performs cross-attention fusion. C4TS explicitly restores the two-dimensional grid of visible historical MAE tokens and adapts this grid before temporal tokens query it.

As described in Section 3.3, the C4TS adapter uses spatial residual blocks with local and dilated depthwise convolutions, followed by row and column embeddings. These operations retain the historical grid resolution while incorporating spatial neighborhood and positional information into the visual memory. The distinction is therefore the explicit spatial adaptation of the historical token grid, rather than the use of cross-attention itself.

Second, C4TS preserves the MAE reconstruction mechanism alongside temporal forecasting. Spatial adaptation is applied to the visual-to-temporal interaction path, while the original MAE encoding remains available to the visual reconstruction decoder. The same historical visual encoding therefore supports temporal feature enhancement and a complete visual-reconstruction forecast, as described in Sections 3.2–3.3.

Third, the decoding mechanisms differ. AURORA generates forecasts through prototype-guided flow matching, using multimodal representations to construct future conditions and prototypes. C4TS retains a temporal predictor and a visual reconstruction pathway, each producing a complete numerical forecast. The two pathways are jointly trained, and their outputs are combined using history-conditioned instance × future-patch weights, as described in Sections 3.4–3.5.

Thus, the architectural contribution is the combination of spatially adapted historical visual memory with two complete forecasting pathways and future-patch prediction fusion. The revised related-work discussion makes these differences explicit, beyond the distinction between full-shot and zero-/few-shot evaluation settings.

Additional reproducibility material

The supplementary c4ts_minimum package provides a minimal C4TS evaluation setup for verification during the review process. It includes model inference code, data-loading utilities, environment specifications, and instructions for obtaining the checkpoint and dataset. The package supports evaluation on ETTh1 with a forecast horizon of 96 and provides reference metrics and checksum verification. As stated in the manuscript’s reproducibility statement, we will release the complete code, training and evaluation scripts, and experimental configurations upon acceptance.

[1] Xingjian Wu, Jianxin Jin, Wanghui Qiu, Peng Chen, Yang Shu, Bin Yang, and Chenjuan Guo. AURORA: Towards Universal Generative Multimodal Time Series Forecasting. International Conference on Learning Representations (ICLR), 2026.

Reviewer_6YxbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

C4TS keeps two complete forecasting pathways: a PatchTST-style temporal pathway and a VisionTS-style MAE reconstruction pathway. They interact in two places: (i) Co-encoding: spatial-aware visual adaptation turns visible MAE tokens into a memory that temporal tokens cross-attend to. (ii)Co-decoding: a future-patch query gate produces a visual weight for each (instance, future-patch) pair, using stop-gradient inputs from both encoders. With L = 512, C4TS reports the best average rank among 10 methods on 7 benchmarks.

Critical strengths
  1. The positioning is clear and the design is careful. Both forecasting mechanisms are preserved (unlike Time-VLM), and the gate is zero-initialized (α=0.5\alpha = 0.5) with stop-gradient inputs. This avoids the gate shortcutting encoder learning.

  2. The protocol is fairer than typical for this line of work. Baselines whose original lookback differs from 512 (DMMV, VisionTS, VPNet, FreqCycle) are re-run at L = 512. The ranking scopes (14 vs. 70 comparisons) are stated explicitly.

Critical weaknesses
  1. Instance×future-patch fusion beats a global scalar by only 0.68% (0.3321 vs. 0.3344) and uniform averaging by 0.75%. Removing co-encoding entirely costs just 0.30% (0.3331). Removing only the spatial adapter (0.3341) is worse than removing interaction altogether, so naive interaction hurts and the adapter mostly repairs it. No seeds or standard deviations are reported, and ablations cover only the 4 ETT datasets. On the evidence given, the two named contributions are not shown to matter.

  2. Overall MSE is 0.298 vs. 0.302 for APRNet and 0.304 for VisionTS/PatchTST, with MAE 0.322 vs. 0.323 for VisionTS. Some baselines are copied from papers, while others are re-run with "all other hyperparameters unchanged" at a new lookback, which can disadvantage them. PatchTST still wins ETTh2.

  3. Learned gated prediction fusion of numerical and visual forecasters already exists (DMMV's fusion gate; TriTS's time-step-level gating), as does feature-level visual–temporal fusion (Time-VLM). The contribution reduces to a finer gating granularity plus a conv adapter.

  4. C4TS runs MAE-Base per variable (862 channels on Traffic), a PatchTST encoder, an adapter, and a query gate. No parameter, memory, or latency comparison is given vs. VisionTS or PatchTST, although cost is the first question for a dual-pathway model.

Questions for authors
  1. The paper describes DMMV's fusion as a global learned scalar, but DMMV's released code describes an adaptive fusion gate. Which exact DMMV variant (S or A) and which gate implementation were re-run, and why does DMMV degrade so much at L = 512 (e.g., ETTm2 0.429)?

  2. Do learned α\alpha values correlate with measurable properties, such as per-instance periodicity strength or VisionTS's period-alignment error? Or is the heatmap variation (Fig. 3c) unexplained by any input statistic?

  3. Have you tried ablating the pretrained MAE (random-init ViT) to separate the benefit of pretraining from the benefit of the visual pathway structure?

9 comments
AuthorsOpenReview ↗

Responses to Critical Weaknesses for Reviewer 6Yxb - Critical Weakness 1

Dear Reviewer 6Yxb,

Thank you for your careful assessment and constructive questions. We have expanded the experimental evaluation with four-seed ablations, additional fusion controls, a random-initialization control for the MAE, and parameter and inference-cost measurements. We also provide DMMV reproduction logs and a minimal C4TS evaluation package in the supplementary material. Below, we respond point by point to all four weaknesses and three questions: W1–W4 correspond to the items under “Critical Weaknesses,” and Q1–Q3 correspond to the items under “Questions For Authors,” in their original order. Section and table numbers refer to the revised manuscript.

W1. Contribution of co-encoding and co-decoding, variability across seeds, and ablation coverage

We have repeated the full model and ablation configurations with four random seeds across the four ETT datasets and four forecasting horizons. Section 4.3 and Table 2 report the aggregate results and percentage changes, while Appendix C, Tables 4–6, provide the detailed mean and standard deviation for each dataset–horizon setting.

The magnitude of an improvement should be considered in the context of strong forecasting baselines. For context, we calculated a combined error score by averaging the mean MSE and mean MAE over the seven benchmark datasets. We then measured each method’s relative improvement over the strongest baseline under this score in its own paper’s comparison table. Under this common aggregation rule, VisionTS and DMMV-A achieve improvements of approximately 1.77% and 1.63%, compared with 1.16% for C4TS. These contextual comparisons use the respective papers’ evaluation protocols; they illustrate that improvements over strong baselines in this setting are often on this scale.

Moreover, a component ablation measures an incremental contribution within an otherwise unchanged model, rather than the difference between two complete forecasting systems. Within this controlled setting, the percentage improvements averaged over four random seeds provide empirical support for the effectiveness of the proposed modules, even when their magnitude is below 1%. Their value should therefore be assessed through the controlled comparisons and multi-seed results, rather than an absolute improvement threshold alone.

Specifically, relative to full C4TS, removing spatial mixing increases average MSE by 0.594%, and removing spatial adaptation and feature interaction increases it by 0.233%. Replacing the proposed fusion with uniform averaging or a global scalar increases average MSE by 0.637% and 0.624%, respectively. These percentages are calculated from the unrounded four-seed aggregate results. The uniform-fusion control retains the joint training objective, including supervision of the fused prediction, so that the comparison isolates the fusion mechanism rather than removing an additional training signal.

Regarding the ordering of the co-encoding ablations, the results show why the interaction mechanism matters: introducing visual information without appropriate spatial processing does not automatically improve temporal forecasting. The spatial adapter is designed to make the two representations compatible by retaining and processing the MAE tokens’ two-dimensional organization. The full design outperforms both the no-spatial-mixing control and the no-interaction control. Adding temporal-to-visual cross-attention also increases average MSE by 3.329%, further supporting the specific interaction direction used in C4TS.

The ETT experiments provide a systematic evaluation across different sampling frequencies, forecasting horizons, and random initializations. The seven-dataset main comparison evaluates broader forecasting performance, while the expanded ETT ablations isolate the architectural choices. Given the substantially larger scale of Weather, Electricity, and Traffic compared with ETT, repeating the full ablation suite across multiple random seeds on these datasets would incur prohibitively high computational costs; we therefore conduct the multi-seed evaluation and ablation studies on the four ETT datasets.

AuthorsOpenReview ↗

Responses to Critical Weaknesses for Reviewer 6Yxb - Critical Weakness 2

W2. Main-table margins, baseline reproduction protocol, and performance on ETTh2

Our baseline-selection rule is based on the input length: we use published results when the reported lookback is already 512, and reproduce methods whose original lookback differs from 512. This gives all methods the same amount of historical input. Changing the lookback does not necessarily disadvantage a method. Retaining the released hyperparameters provides a reproducible, release-based comparison without introducing an additional author-selected hyperparameter search. Comprehensive hyperparameter re-optimization of every external baseline is not a routine requirement of such reproduction; it would constitute a separate experimental protocol.

In the revised main table, C4TS achieves an overall average MSE of 0.299, an overall average MAE of 0.322 and the best average rank of 2.143. Its improvement should be evaluated against the scale of progress over strong baselines in closely related work, together with its performance across datasets, as discussed in W1.

For ETTh2 specifically, PatchTST achieves an average MSE of 0.331, compared with 0.344 for C4TS. Our main-table conclusion concerns aggregate performance across benchmarks, rather than a claim of winning on every dataset. The temporal branch in C4TS is a separate implementation inspired by PatchTST-style modeling, not the official PatchTST implementation; the published PatchTST result therefore should not be interpreted as the performance of C4TS’s internal temporal branch.

Indeed, PatchTST outperforms every other method in our comparison on ETTh2 in both horizon-averaged MSE and MAE, regardless of whether those results are taken from the original papers or obtained through reproduction. Within our controlled ablation, the independently trained temporal branch has an average ETTh2 MSE of 0.3491, compared with 0.3441 for full C4TS. Thus, although the official PatchTST remains the strongest method on this particular dataset, the complete C4TS framework improves over its own temporal-only counterpart.

The comparison protocol and updated results are described in Sections 4.1–4.2 and Appendix B; the temporal-only ablation appears in Appendix C, Table 6.

AuthorsOpenReview ↗

Responses to Critical Weaknesses for Reviewer 6Yxb - Critical Weakness 3

W3. Differences from existing gated and feature-level fusion methods

The contribution of C4TS is the specific coordination of two complete forecasting mechanisms: numerical forecasting and MAE-based visual reconstruction. Co-encoding exchanges representations between these pathways, while co-decoding combines their completed forecasts using weights explicitly indexed by future patches.

The distinctions from the cited approaches are as follows:

• DMMV: Its released fusion coefficients vary across variables but are shared across input instances and forecast positions. C4TS instead generates fusion weights from the current input representations for each future patch. We detail this distinction, including the released DMMV implementation, in Q1 below.

• Time-VLM: Feature-level visual–temporal fusion does not by itself preserve a complete MAE reconstruction pathway that produces its own forecast. C4TS retains this forecasting mechanism alongside the numerical pathway and coordinates them at both the representation and prediction levels.

• TriTS: TriTS focuses on combining time-, frequency-, and vision-domain representations through time-step-level modality weighting over the historical window. C4TS instead explicitly associates its fusion weights with future forecast patches: learnable future-patch queries read the encoded historical information and determine how to combine the numerical forecast and visual reconstruction for each future patch. The distinction therefore concerns both the temporal role of the weights—historical representation weighting versus future-patch prediction fusion—and the forecasting architecture. C4TS preserves two complete forecasting pathways and coordinates them through both visual-to-temporal feature interaction and adaptive fusion of their final predictions.

Accordingly, our distinction is not simply “finer temporal granularity.” It concerns where the weights act, how future prediction positions are represented, and which forecasting mechanisms are preserved.

Sections 3.3–3.4 specify these operations: spatial adaptation restores and processes the historical MAE token grid before visual-to-temporal interaction, and future-patch queries generate the output-fusion weights. The expanded ablations in Section 4.3 separately examine spatial processing, interaction direction, and fusion granularity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.