- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MixBench-TS studies whether commonly used MTS benchmarks contain exploitable cross-channel structure. It evaluates several statistical coupling measures, defines a model-based channel-dependence gain using TSMixer versus a channel-independent TSMixer variant, and selects datasets with positive gain to form a new benchmark.
Critical strengths
The central question is important. Dataset properties can strongly bias conclusions about whether cross-channel modeling is useful, and a benchmark explicitly designed to examine this issue could be valuable.
Critical weaknesses
- The proposed lagged-MI diagnostic is not equivalent to predictive cross-channel information. MIlag measures whether pairwise lagged mutual information exceeds contemporaneous mutual information, but it does not condition on the target channel's own history. The paper explicitly defines it this way. Therefore, a high MIlag value does not directly measure the additional information that another channel contributes to forecasting after the target history is already known.
- The paper does not sufficiently position MIlag relative to existing spatial/temporal heterogeneity analyses. Prior MTS benchmarking work BasicTS has already shown that temporal and spatial heterogeneity strongly affects model rankings and the value of different spatial modeling strategies. MIlag captures a different statistic, but the paper does not establish whether it contributes explanatory power beyond these existing dataset-characterization measures.
- CD gain is architecture-dependent and is defined using only TSMixer versus TSMixer-CI. The manuscript itself acknowledges that CD gain reflects whether TSMixer can exploit the dependency. A negative TSMixer gain therefore cannot justify the stronger conclusion that a dataset lacks exploitable cross-channel structure. It may simply mean that this particular mixing mechanism is inappropriate.
- The benchmark construction is circular with respect to the main claim. Candidate datasets are explicitly retained only when TSMixer achieves positive CD gain. The resulting benchmark is then used to show that channel-dependent models become more competitive. This introduces a strong selection effect. An independent family of CD/CI models or a held-out selection/evaluation procedure is necessary.
- The CI/CD dichotomy is overly restrictive. Modern models need not be purely channel-independent or fully channel-dependent. CPiRi, for example, explicitly combines content-driven cross-channel relational modeling with channel permutation invariance, aiming to retain CI-style structural flexibility while exploiting cross-channel information. Similar selective/grouped interaction models further blur a binary CI/CD taxonomy. The benchmark should therefore test whether adaptive or selective channel interaction benefits from the proposed coupling criteria rather than treating CI and CD as mutually exclusive strategies.
- The traffic evaluation is insufficient for the paper's general conclusion. Traffic is included among the standard datasets, and METR-LA/PEMS-BAY appear only in the appendix, but the paper does not systematically test strong spatial-temporal models or large-scale datasets. A central claim about the usefulness of cross-channel modeling would be much more convincing on benchmarks such as METR-LA, PEMS-BAY, PeMS, and LargeST, where explicit spatial dependencies are known to matter.
Questions for authors
- Does CD gain remain predictive when computed using several fundamentally different CD/CI model pairs rather than TSMixer alone?
- How much additional explanatory power does MIlag provide beyond existing spatial/temporal heterogeneity measures?
- If dataset selection is performed on one model family, do the conclusions persist on completely held-out model families and test periods?
- How do hybrid or selective interaction methods such as CPiRi behave as a function of MIlag and CD gain?
- Do the conclusions hold on METR-LA, PEMS-BAY, PeMS, and LargeST with strong spatiotemporal models?
3 comments
I appreciate the additional experiments, but I remain unconvinced by the rebuttal. The authors have not adequately addressed my central concerns, and the arguments provided do not sufficiently resolve these methodological issues. On this basis, I will keep my score unchanged.
We would like to ask the reviewer to refer to the newly add comment Finalized ModernTCN-based CD gain Experiments above, which we hope now addresses in more detail W3 and Q1
We would appreciate it a lot if the reviewer refer to the specific central methodological issues that are not addressed yet by our rebuttal, so we can clarify it more in a further comment or in the final version of the paper.
We would like to thank the reviewer for the insightful and detailed feedback.
W1: We agree with the reviewer on the description of the method, but the main intuition, as motivated in the introduction, is that lagged coupling between variates matters for forecasting. Since forecasting concerns future timesteps, datasets with more prominent lagged coupling benefit more from multivariate modeling. Figure 1 shows the correlation between MI lag and additional exploitable cross-channel information (validated by the CDg metric).
W2 and Q2: We thank the reviewer and from our understanding the reviewer is refering to BasicTS+ benchmark [1]. Although r1/r2 and ϕlag are both model-free, pairwise, fraction-of-pairs statistics computed on training data, they answer different questions:
Figure 1 supports this with the clear correlation between CD gain and MI lag.
W3 and Q1: We are running experiments with ModernTCN [6], a state-of-the-art CNN-based CD model; we form its CI variant by removing the ConvFFN2 layer, with full tuning as for TSMixer. Preliminary results on 16 of the 20 datasets (the 4 remaining high-dimensional datasets are still running) show a degree of consistency: the CD gains from ModernTCN and TSMixer agree in sign on 10/16 datasets. We acknowledge some sensitivity to model choice in gain magnitude on a small subset: in 4/16 datasets, the CD gain differs by more than 5% in test MSE. Please see Table 14 in the revised appendix; we will update it with the full suite once the experiments finish.
W4, Q3 and Q4: We agree regarding possible selection bias from using TSMixer in both dataset selection and model evaluation. We therefore added four models (two CI, two CD) to the MixBench-TS evaluation: PatchTST [2], SRSNet [3], TimeFilter [4], and DUET [5], and removed the TSMixer-CI/CD variants from the comparison. CD models still show a clear edge on MixBench-TS, with lower test MSE on 6/10 and lower MAE on 8/10 datasets. Full results of the 8-model comparison on both metrics are in the updated Table 19 in the appendix.
W5: We agree that adding more models with selective channel mixing strengthens the findings (as addressed above). However, we disagree that the CI/CD dichotomy is overly restrictive: in our view, models that selectively apply cross-channel exploitation are CD models. We follow [7], which defines CD models as “a model that forecasts each channel’s future values using the history of all channels” and CI models as “a model that forecasts each channel’s future values using on the history of this specific channel”. This also generalizes to the CI loss definition: “The CI loss is the mean of the losses of all channels, with each channel’s loss being minimized independently”.
W6 and Q5: We thank the reviewer for suggesting LargeST [8]. As the reviewer noted, we already include METR-LA/PEMS-BAY, as well as Traffic (862 channels), one of the highest-dimensional datasets in the forecasting literature. Scaling to LargeST (716 to 3,834 variates) is non-trivial in a study of this size due to compute and memory costs. Moreover, our aim was to cover datasets with diverse characteristics rather than one class. On spatio-temporal data, we already observe the same MIlag–CDg correlation: Traffic (27% MIlag) has a CDg of -9.2 and METR-LA (38%) -1.5, whereas PEMS-BAY, with more lagged pairs (69%), has a higher CDg of +6.4. Please see Table 11 for a clearer picture of this correlation.
[1] Shao et al. "Exploring progress in multivariate time series forecasting: Comprehensive benchmarking and heterogeneity analysis." IEEE TKDE, 2024.
[2] Nie et al. "A time series is worth 64 words: Long-term forecasting with transformers." ICLR, 2023.
[3] Wu et al. "Enhancing time series forecasting through selective representation spaces: A patch perspective." NeurIPS, 2025.
[4] Hu et al. "TimeFilter: Patch-specific spatial-temporal graph filtration for time series forecasting." arXiv:2501.13041, 2025.
[5] Qiu et al. "DUET: Dual clustering enhanced multivariate time series forecasting." KDD, 2025.
[6] Luo and Wang. "ModernTCN: A modern pure convolution structure for general time series analysis." ICLR, 2024.
[7] Han et al. "The capacity and robustness trade-off: Revisiting the channel independent strategy for multivariate time series forecasting." IEEE TKDE, 2024.
[8] Liu et al. "LargeST: A benchmark dataset for large-scale traffic forecasting." NeurIPS, 2023.