acceptodds
Withdrawn from ICLR 2027

Beyond Plug-and-Play: Do Forecasting Plugins Pay Off Across Models and Settings?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Forecasting plugins are increasingly proposed as plug-and-play or model-agnostic enhancements for time-series forecasting. However, architectural compatibility alone does not establish their effectiveness or practical deployment value across datasets, prediction horizons, and backbone models. We introduce PlugBench-TS, a controlled benchmark of ten forecasting enhancements spanning jointly fitted modules, training objectives, normalization, retrieval, output adaptation, and online forecast revision. It measures paired forecasting gain, checkpoint-pairing dependence where replacement is valid, and deployment memory and latency. The primary matrix covers 960 plugin–context combinations across four backbones, six datasets, and four prediction horizons, each evaluated in three paired runs. The results show why compatibility and a single accuracy number are not enough: the same enhancement can help in one context and hurt in another; a gain with its fitted backbone may disappear after a checkpoint change; and a small parameter count can conceal substantial retrieval memory and inference time. Building on these observations, we study whether the state used by online, static, and retrieval plugins can be reduced while preserving useful forecasting gains. The case studies reveal both successful reductions and accuracy–cost trade-offs. Overall, PlugBench-TS provides a reproducible basis for evaluating plugin value and for guiding future plugin design and deployment.

Primary area:
learning on time series and dynamical systems
Keywords:
time series forecasting, forecasting plugins, benchmarking, checkpoint dependence, deployment cost

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aUFqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PlugBench-TS, a benchmark for comparing, under controlled conditions, plugins that are combined with a backbone model to improve time-series forecasting performance (e.g., add-on modules, training objectives, normalization, retrieval, output correction, and online adaptation). PlugBench-TS covers 10 plugins, 4 backbones, 6 datasets, and 4 forecasting horizons (with lookback L=96), and includes three measurements (paired payoff, checkpoint-pairing dependence, and deployment burden). Paired payoff quantifies the performance improvement by comparing the MSE of the plugin-equipped system with that of its matched bare backbone on identical targets. Checkpoint-pairing dependence measures whether the improvement persists when the fitted plugin state is attached to a different, separately trained backbone checkpoint of the same architecture. Deployment burden measures each plugin's additional parameters, persistent state such as retrieval memory, and latency. In addition, the authors propose the contract-matched plugin transformation (CMPT) procedure, which reduces the state held by a plugin while preserving its original execution rules, and examine how much of the original plugin's performance gain is retained after the reduction.

Critical strengths
  • S1: Authors specify each plugin's information access, integration mode, fitting, state, and emission timing as an execution contract, and on this basis define three measurements (paired payoff, checkpoint-pairing dependence, and deployment burden). This provides a criterion for evaluating plugin value and cost from multiple perspectives that a single accuracy number cannot capture.
  • S2: The supplementary material provides code and seed-level result CSVs, allowing the main reported numbers to be recomputed and verified from the underlying data.
  • S3: This paper connects the benchmark's state and cost measurements to CMPT, and quantitatively reports, including failure cases, the resource savings and the retained performance gain when plugin state is reduced within the bounds of the execution rules.
Critical weaknesses
  • W1: Four of the six datasets in the main experiments belong to the ETT family, and the data come from only three sources, which limits how far the results can be generalized. Moreover, many of the measurements, such as paired gain, parameter count, and latency, are already widely used in prior work, so the novelty of the measurements themselves is limited relative to existing benchmarks (e.g., TFB, BasicTS+, TSPP, TimeRecipe, and CombinationTS).
  • W2: The 10 evaluated plugins were proposed for different purposes. For example, COSA, TAFAS, and FAN were proposed to address non-stationarity and distribution shift, whereas GTR and KUP-BI were proposed to improve accuracy. Although the paper separates online-revision methods into their own cohort, plugins designed for distribution shift are still evaluated only by average MSE on a standard chronological split. The plugins should be categorized by their intended purpose, and evaluation conditions suited to each purpose (e.g., analysis stratified by shift severity) should be provided.
  • W3: Although most symbols are defined individually where they first appear, there is no notation table. This makes some sections difficult to follow.
  • W4: The checkpoint-replacement experiment describes the replacement target as an "independently trained checkpoint." However, according to the released code, for the joint plugins (GLAFF, GTR, FAN, KUP-BI) the bare checkpoint is trained from the same initial weights as the joint backbone, and for PIR the bare checkpoint is the starting point of PIR's refinement itself. The results in Table 3 are therefore difficult to interpret as replacement between independent checkpoints.
  • Minor: According to the rank settings in Appendix C.4 and the released code, CPLR is a low-rank adapter whose source is the COSA-F setting (fixed batch size B=48) from the original COSA paper. However, the main text of Section 5.2 only states that CPLR "factorizes" this reference, without describing the low-rank structure, naming the source (COSA-F), or expanding the acronym.
Questions for authors
  • Q1: As noted in W3, could you add a table summarizing the notation used in the paper?
  • Q2: Beyond Traffic, which was evaluated as a separate extension, have you considered applying the benchmark to additional datasets? Relatedly to W2, since COSA, TAFAS, FAN, and others were proposed to address distribution shift, have you considered synthetic perturbation data with controlled shift types and intensities, such as TTFBench (Grover & Etemad, 2025), or an analysis of real data stratified by shift severity? Could you also report the results of the input-history noise stress test (ROBUSTNESS.json) implemented in the released code?
  • Q3: For each CMPT variant (CPLR, low-rank Ada-Y, CPLM, and the KUP-BI index layout), including CPLR as noted in the Minor comment, could you describe (1) the source setting (e.g., whether CPLR's source is COSA-F rather than the main-matrix COSA run with PAAS), (2) the structure and rank of the modified operator, (3) whether the variant is obtained by decomposing the source or is newly initialized and trained, and (4) the rank-selection procedure? Please also explain why the Ada-Y source MSE in Table 17 does not match the Ada-Y results in the main matrix (Tables 26 and 27).
  • Q4: Regarding W4, could you specify how arms A and D are initialized in the checkpoint-replacement experiment? In particular, for PIR, the bare checkpoint used in arm D appears to be the starting point of PIR's refinement itself. Could you also report replacement results using independent checkpoints initialized with different seeds?
Reviewer_WczbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors investigate the use of "plugins" for time series forecasting - specifically their robustness, forecasting performance gain, cost etc. They propose PlugBench-TS, a benchmark of 10 plugins that span modules/training objectives/normaliZation/retrieval etc. They do a total of 960 experiments across these plugins and base model architectures/datasets/prediction horizons. The experiments are done with the plugin, without the plugin etc. to investigate the usefulness of the plugin. The authors document their results that show that gains are not consistent across datasets / the cost factor - retrieval memory/inference time may be added by plugins.

Critical strengths
  • The motivation of the paper is very good and justified - there have been several such add-ons proposed for forecasting, and the performance is always different in different datasets. A benchmark that brings these add-on plugins together and unifies them, would be very useful.
  • The authors seem to do a good job in surveying the literature for such plugins - across categories such as training objectives, architecture changes etc - so the scope is pretty wide. I appreciate that.

Unfortunately, beyond the motivation, the paper is incomprehensible due to the heavy usage of AI in writing/ potentailly even in experimentation (lots of undefined jargon, clearly not human understandable) -- which I expand on below.

Due to that I don't consider these strengths sufficient to support acceptance.

Critical weaknesses
  • First off, the paper uses the same datasets used as the original papers - and several works such as [1] have shown that these are not useful / are limited. Quoting from [1]:

    We demonstrate that these "standard" datasets often exhibit dominant autocorrelation patterns and seasonal cycles that can be effectively captured by simpler linear or statistical models, rendering complex deep learning architectures frequently no more performant than their classical counterparts for these specific data characteristics

    [1] Saqur, Raeid, et al. "Seeking SOTA: Time-Series Forecasting Must Adopt Taxonomy-Specific Evaluation to Dispel Illusory Gains." arXiv preprint arXiv:2603.15506 (2026).

    There have been others proposed (Gift-eval etc) that the community has adapted since. So, the experimental setup itself is unconvincing to me - the results/conclusions made on these datasets don't really indicate that they transfer to any other datasets/distributions.

  • After revisiing the abstract/conclusion after reading the paper, I realize High-level summary unclear -- should be in the intro/conclusions:

    • Which plugins help across datasets?
    • What factors of the dataset matter when choosing the plugin?
    • How does this paper help a practitioner choose plugins? Of course, the gain is expected to be different in each setting by there shuld be extrapolation/advice offered based on the benchmarked datasets
  • The plugins that have been evaluated are not described in detail - there is no background section that introduces these plugins in fair detail, allowing contextualization of the experiments/results that follow. The only introduction of the plugins is in Sec 2 - and it heavily uses non-standard jargon that I do not understand and offers too little insight into what each plugin does. For example I paste one paragraph below, this literally provides one line of explanation for each plugin - either provide background or expand on each of them; since I don't understand this, I have zero context to understand the rest of the paper.

    Backbones and forecasting plugins. Our controlled suite spans four forecasting architectures: linear decomposition with DLinear, temporal patching with PatchTST, variate-token attention with iTransformer, and multiscale mixing with TimeMixer (Zeng et al., 2023; Nie et al., 2023; Liu et al., 2024; Wang et al., 2024b). The plugins we study act at different stages. FAN modifies frequency normalization (Ye et al., 2024), LatentTSF forecasts in a learned latent space (Yang et al., 2026), and FreDF modifies the training objective (Wang et al., 2025). GLAFF combines global calendar derived and local information (Wang et al., 2024a), while GTR introduces learned global temporal references (Cao et al., 2026).

  • As said above, the paper is very much incomprehensible right page 1 due to the heavy usage of AI in writing/ potentailly even in experimentation (lots of undefined jargon, clearly not human understandable). It is clearly not reviewed by humans and verified that it is even human-understandable. I attempted to read the paper 3 times slowly to comprehend it but still couldn't. I highlight as many examples below as I can (there are too many sentences in the paper to highlight them all):

    • Our contract-matched plugin transformation procedure (CMPT) uses the recorded execution rules and resource costs to test smaller online adapters, static output maps, and retrieval indices. It"

      Unclear what CMPT means - not defined well anywhere

    • PlugBench-TS records what a plugin may read

      What does this mean? Isn't this implicit based on where it is placed / how it is used? What is it going to "Read" outside what's available (historical data / activations at a layer erc).

    • Fig 2 - CMPT - unclear what it means. What is "state reduction"? What does "reduce plugin-owned state" mean? A lot of the text in the figure is incomprehensible.

    • Table 1 - The main information - non-revised ... and online-revision cohort - unclear --- background must be given to even contextualize what they originally claim + why its important etc

    • The iTransformer and TimeMixer model classes come from a pinned TSLib snapshot; DLinear and PatchTST use pinned author-source implementations. Within each dataset–horizon–backbone context, the bare implementation and training recipe are

    What does "pinned TS Lib snapshot" mean? What is the pinned author-source implementation?

    • The matrix includes source-reported pairings and extensions to unreported datasets or backbones, always with the pinned backbone recipe. This distinguishes performance under a common comparison from results obtained after tuning a backbone for a particular plugin. Integration details can matter: our controlled GTR–PatchTST pairing retains PatchTST’s RevIN, whereas the released GTR configuration disables it; FAN is added to some backbones that already normalize their inputs.

    None of the above text is complete (" our controlled GTR–PatchTST pairing retains PatchTST’s RevIN", " FAN is added to some backbones that already normalize their inputs" - what does any of this exactly mean?)

    • Line 226:

      Execution validity: is the declared method actually being evaluated? A run is admitted only when its realized trace obeys ΓP = (read, integrate, fit, state, emit): variables are available when read; insertion, replacement, and disabled components match the registered integration; fitting and frozen/trainable flags match; state follows its schedule; and scored predictions follow the emission rule. Paired comparisons additionally hold c fixed, including targets, origins, and scaling. Sourcepaper reproduction further requires a matching author recipe and provenance.

    I'm not sure what this is. It looks like an LLM did the experimentst and this should have been an internal check that the authors might have used (to ensure the LLM used the right code etc). But it appears in the paper.

    • The identity characterizes output correction; it does not decompose a jointly trained system’s gain. We measure paired gains; fitted probes do not identify the population optimum m. Context profiles stratify these gains by dataset, backbone, horizon, and lookback, but do not create independent replications.

    Incomprehensible - "population optimum"? What does the last sentence mean

    • Other examples from the same page: "operational interaction constrast" "Not a causal allocation" - what does any of this mean? Heavy use of undefined, incomprehensible jargon makes it hard to read and understand anything

    • I was unable to understand the paper after that; it's more incomprehensible than ever.

  • Other questions:

    • What is a "checkpoint" - is it the same model trained with a different seed? Is it an earlier checkpoint? -- this does not make sense as an ablation axis. Same with "We report context-level paired gains, same architecture checkpoint replacements" (Line 54-55)
    • Fig 1: Caption says " Each cell gives paired MSE gain after" but figure says it is MSE gain percentage (not absolute gain) - which is the right one?
Questions for authors

See above.

Reviewer_DQ3AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper benchmarks ten published "plug-and-play" forecasting plugins on four backbones, six datasets and four horizons. The plugins cover normalization, losses, retrieval, output correction and online adaptation. Each one is compared against the same bare backbone trained with a fixed recipe. Besides accuracy, the paper checks whether a trained plugin still works after the backbone is retrained, and what the plugin costs to run. The main findings are: gains depend heavily on the setting and often vanish outside the original papers' setups; plugin state often breaks on a new checkpoint; and parameter count hides real memory and latency costs. A final section tries to shrink plugin state while keeping the gain.

Critical strengths
  1. The question is worth asking, and to my best knowledge, was not benchmarked yet. Existing benchmarks compare whole models or architecture components. This one asks whether a published add-on still helps on a backbone it wasn't tuned for.

  2. The experiments are well controlled. Bare and plugin runs share splits, scaling, seeds and backbone recipe. Source-reported and new pairings are reported separately, and the original authors' settings are reproduced as a sanity check.

Critical weaknesses
  1. The scope is too big, and "plugin" is a moving target. There are far more model-agnostic methods than ten; normalization alone has RevIN, Dish-TS and SAN besides FAN, it is hard to benchmark every model-agnostic approaches. Also, a plugin stops being a plugin once people agree it works. For example, RevIN was proposed initially as a model-agnostic add-on and PatchTST's original code uses it as a hyperparamter, while later it becomes a more default approach as its effectiveness is agreed broadly.

  2. While the work have done many experiments, it lacks of variety. Four of the six datasets are ETT, and the main grid uses a single lookback. Everything is multivariate long-horizon forecasting, with no univariate or short-term (e.g., M4) setting. The main claim is that gains depend on the setting, but the settings covered in this benchmark are still limited.

  3. No clear takeaway. After reading the paper, it feels like none of these plugins is safe to use. Yet they are published methods that do work in their own settings, as the paper's own reproductions show. The more useful question is when each one helps, and the results already hint at an answer. FAN helps DLinear, the only backbone here without built-in normalization, in 15/24 cells, but TimeMixer in only 2/24 (Table 6). TFB already describes datasets by trend, seasonality, stationarity and similar properties, and TimeRecipe links design choices to such properties. Doing the same for plugins would make this benchmark much more useful.

Questions for authors

Please refer to the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.