- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper examines pseudo-multimodality in time series forecasting: numerical histories and prompts derived from them provide different representations of the same observations. It argues that the language pathway should be assessed by its conditional predictive utility, rather than representational agreement alone. CIRF constructs optional residual corrections to a complete numerical forecast; CIRF-R addresses additional sensitivity under shared perturbations. Benchmark, routing, and representation-replacement experiments investigate correction utility and distinguish it from benefits attributable to language pretraining or matched prompts.
Critical strengths
- The pseudo-multimodality perspective is important for multimodal time series research. The paper distinguishes acquiring another observation from deriving another representation, while recognizing that the latter can still help a particular predictor. Assessing this help through conditional utility provides a useful basis for deciding when to incorporate a language-derived correction.
- The attribution experiments are thoughtfully designed. LLM replacement/randomization, prompt masking/mismatching, numerical-only models, and Stats-MLP distinguish the roles of pretraining, prompt content, and encoding. Fixed-candidate routing and oracle comparisons separately examine correction potential and selection ability. These controls provide useful guidance for interpreting fusion results, including negative outcomes.
Critical weaknesses
- The relationship between agreement and correction utility remains empirically underexplored. The routing experiments establish heterogeneous utility, but do not jointly examine representation agreement, conditional predictability, and actual loss reduction. A focused diagnostic using existing checkpoints would help identify when agreement is informative or misleading and explain which cases benefit from utility-based selection.
- The shared-perturbation mechanism is not separately diagnosed. Table 4 supports the combined robustness procedure, but always regenerates prompts from the same corrupted history. It therefore leaves open how coupling the disturbances across views affects sensitivity beyond that of a correction path generally. A matched shared-versus-independent perturbation experiment would distinguish these effects and clarify the conditions in which the proposed common-source sensitivity control matters.
Questions for authors
- On existing ETT and Weather tasks, could you relate a justified agreement measure and the conditional predictor's normalized residual to candidate loss reduction, gate decisions, and oracle utility? Keeping candidates fixed, when does agreement identify beneficial corrections, and which cases require utility-based selection?
- In one existing corruption setting, could you compare perturbing the numerical input and the history used to generate prompts with shared versus independent disturbances? Holding checkpoints, perturbation distributions, and per-view intensity fixed, how do excess sensitivity and routing behavior change for CIRF and CIRF-R?
2 comments
Thank you for running both experiments. The Q1 results strengthen the paper’s central insight that representation agreement is a poor proxy for correction utility. The Q2 results clarify that sharing the noise had little effect in the tested setting. The current gate still leaves room for improvement in selecting useful corrections, so I’ll keep my score at 3 and make the case for acceptance to the other reviewers and the AC during discussion.
Thank you for these focused diagnostic suggestions.
Q1. Agreement, residuals and correction utility
Agreement describes shared structure; correction acceptance depends on whether the decoded increment reduces numerical forecast error. Useful candidates also occur at low agreement, supporting CIRF's separation of residual construction and utility-based acceptance.
We associated validation-fitted Procrustes cosine between zX,zP and normalized conditional residual r=∥zP−Cϕ(zX)∥2/(∥zP∥2+10−8) with candidate MSE gain, gate probabilities/decisions and Oracle utility on the same window–variable pairs. Forecasts, candidates and Gate thresholds were fixed; proxy rules were selected on validation and frozen at test.
Three-seed mean test MSE from supplementary checkpoints, separate from Table 2 runs. Oracle selects using future targets.
Agreement and benefit. Agreement–utility Spearman correlations are −0.003 to 0.054. On Weather, the Agreement rule accepts 99.07% and operates near Always-on, whereas Gate accepts 75.36% and achieves lower mean MSE than both proxies. These compare policies at their validation-selected acceptance rates. In validation-defined 3×3 agreement–residual strata, Weather's high-residual groups have positive mean utility across all agreement levels and seeds, including low agreement. Thus low agreement alone is not a reason to discard a candidate; the decoded correction's effect is the relevant acceptance criterion.
Utility-based acceptance. ETTm2 illustrates why group averages and individual decisions need to be distinguished. Its group-mean candidate utilities are negative, yet Oracle can still select beneficial individuals. Applying every correction increases error; rejecting every candidate leaves useful corrections unexploited. On Weather, residual–utility correlations are positive (0.083–0.100), but on ETTm2 they are negative (−0.112 to −0.095), while residual–gate-probability correlations remain positive (0.418–0.462 and 0.101–0.166). Residual size therefore needs to be interpreted together with the forecasting context, rather than used as an acceptance score by itself.
For decoded increment d=Y^+−Y^X and numerical error u=Y−Y^X, U=(2u⊤d−∥d∥22)/H. Positive utility depends on the increment's direction and magnitude relative to the forecast error. CIRF consequently uses the conditional residual to construct a candidate and learns acceptance from loss reduction (Huber training labels; MSE in this diagnosis).
On ETTm2, Gate accepts 0.23% and avoids most of the error increase from mandatory correction. Its MSE remains 0.341×10−3 above Numerical and above both proxies; Oracle's lower error marks the remaining selection opportunity. These results support optional correction and identify where its acceptance rule can improve.
Q2. Shared versus independent perturbations
Changing whether the two views share noise has little effect on either model's EA or routing in this setting. CIRF-R* retains its lower observed EA under both noise conditions, so this behavior is not restricted to identical perturbations in the two histories.
We fixed each checkpoint and validation threshold on ETTh1-96, using statistics-based prompts and Gaussian scale 0.20 in train-standardized units. Numerical noisy histories were identical across conditions; only prompt-history noise changed from shared to independently drawn at the same strength and distribution. EA uses each checkpoint's own numerical branch as reference.
Means over three training seeds using one fixed paired noise set across checkpoints. Accept/Flip are percentages; Flip is relative to clean. CIRF-R* is a supplementary robust-training batch with a different numerical backbone and additional noisy-forecast supervision, reported separately from Table 4.
Shared-minus-independent EA is (−8.01±6.95)×10−6 for CIRF and (0.65±2.09)×10−6 for CIRF-R* (mean ± sample SD of paired training-seed differences). Switching noise dependence changes 0.0701% and 0.0342% of gate decisions, respectively. These changes are small relative to each model's EA. For these checkpoints and paired draws, the measured sensitivity and routing behavior is therefore largely preserved when the views receive independent perturbations.