- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PlugBench-TS, a benchmark for comparing, under controlled conditions, plugins that are combined with a backbone model to improve time-series forecasting performance (e.g., add-on modules, training objectives, normalization, retrieval, output correction, and online adaptation). PlugBench-TS covers 10 plugins, 4 backbones, 6 datasets, and 4 forecasting horizons (with lookback L=96), and includes three measurements (paired payoff, checkpoint-pairing dependence, and deployment burden). Paired payoff quantifies the performance improvement by comparing the MSE of the plugin-equipped system with that of its matched bare backbone on identical targets. Checkpoint-pairing dependence measures whether the improvement persists when the fitted plugin state is attached to a different, separately trained backbone checkpoint of the same architecture. Deployment burden measures each plugin's additional parameters, persistent state such as retrieval memory, and latency. In addition, the authors propose the contract-matched plugin transformation (CMPT) procedure, which reduces the state held by a plugin while preserving its original execution rules, and examine how much of the original plugin's performance gain is retained after the reduction.
Critical strengths
- S1: Authors specify each plugin's information access, integration mode, fitting, state, and emission timing as an execution contract, and on this basis define three measurements (paired payoff, checkpoint-pairing dependence, and deployment burden). This provides a criterion for evaluating plugin value and cost from multiple perspectives that a single accuracy number cannot capture.
- S2: The supplementary material provides code and seed-level result CSVs, allowing the main reported numbers to be recomputed and verified from the underlying data.
- S3: This paper connects the benchmark's state and cost measurements to CMPT, and quantitatively reports, including failure cases, the resource savings and the retained performance gain when plugin state is reduced within the bounds of the execution rules.
Critical weaknesses
- W1: Four of the six datasets in the main experiments belong to the ETT family, and the data come from only three sources, which limits how far the results can be generalized. Moreover, many of the measurements, such as paired gain, parameter count, and latency, are already widely used in prior work, so the novelty of the measurements themselves is limited relative to existing benchmarks (e.g., TFB, BasicTS+, TSPP, TimeRecipe, and CombinationTS).
- W2: The 10 evaluated plugins were proposed for different purposes. For example, COSA, TAFAS, and FAN were proposed to address non-stationarity and distribution shift, whereas GTR and KUP-BI were proposed to improve accuracy. Although the paper separates online-revision methods into their own cohort, plugins designed for distribution shift are still evaluated only by average MSE on a standard chronological split. The plugins should be categorized by their intended purpose, and evaluation conditions suited to each purpose (e.g., analysis stratified by shift severity) should be provided.
- W3: Although most symbols are defined individually where they first appear, there is no notation table. This makes some sections difficult to follow.
- W4: The checkpoint-replacement experiment describes the replacement target as an "independently trained checkpoint." However, according to the released code, for the joint plugins (GLAFF, GTR, FAN, KUP-BI) the bare checkpoint is trained from the same initial weights as the joint backbone, and for PIR the bare checkpoint is the starting point of PIR's refinement itself. The results in Table 3 are therefore difficult to interpret as replacement between independent checkpoints.
- Minor: According to the rank settings in Appendix C.4 and the released code, CPLR is a low-rank adapter whose source is the COSA-F setting (fixed batch size B=48) from the original COSA paper. However, the main text of Section 5.2 only states that CPLR "factorizes" this reference, without describing the low-rank structure, naming the source (COSA-F), or expanding the acronym.
Questions for authors
- Q1: As noted in W3, could you add a table summarizing the notation used in the paper?
- Q2: Beyond Traffic, which was evaluated as a separate extension, have you considered applying the benchmark to additional datasets? Relatedly to W2, since COSA, TAFAS, FAN, and others were proposed to address distribution shift, have you considered synthetic perturbation data with controlled shift types and intensities, such as TTFBench (Grover & Etemad, 2025), or an analysis of real data stratified by shift severity? Could you also report the results of the input-history noise stress test (ROBUSTNESS.json) implemented in the released code?
- Q3: For each CMPT variant (CPLR, low-rank Ada-Y, CPLM, and the KUP-BI index layout), including CPLR as noted in the Minor comment, could you describe (1) the source setting (e.g., whether CPLR's source is COSA-F rather than the main-matrix COSA run with PAAS), (2) the structure and rank of the modified operator, (3) whether the variant is obtained by decomposing the source or is newly initialized and trained, and (4) the rank-selection procedure? Please also explain why the Ada-Y source MSE in Table 17 does not match the Ada-Y results in the main matrix (Tables 26 and 27).
- Q4: Regarding W4, could you specify how arms A and D are initialized in the checkpoint-replacement experiment? In particular, for PIR, the bare checkpoint used in arm D appears to be the starting point of PIR's refinement itself. Could you also report replacement results using independent checkpoints initialized with different seeds?