acceptodds
Under review as a conference paper at ICLR 2027

CopyTestBench: Cross-Channel Retrieval in Transformer Forecasting

Abstract

Real-world forecasting benchmarks can hide important differences between models: similar errors may result from different computations, and failures are hard to interpret when the optimal predictor and irreducible error are unknown. We introduce CopyTestBench (CTBench), a diagnostic, capability-level benchmark with known rules and zero Bayes error, where the correct target always appears in the input window. Rather than only ranking forecasters by aggregate error, CTBench tests whether they retrieve, route, and preserve delayed cross-channel dependencies under pure-copy, daily switching, and rotating switching, with varying distractors and autocorrelation. Ridge regression nearly recovers the exact solution on pure-copy tasks and achieves the lowest overall error, showing that the dependency is observable and linearly solvable. Under the same training protocol, none of the transformer-based forecasters matches this accuracy. Several instead rely on the target level or switching pattern. Some models beat gate-aware persistence under daily switching, but none do so under rotating switching. Controlled experiments identify failures in source routing and normalization. Vanilla Transformer solves one pure-copy dataset under a constant learning rate, while iTransformer remains near the window-mean baseline. Oracle routing, combined with changes to per-channel instance normalization and encoder layer normalization, raises from to , with consistent results across seeds and on an exact-copy task built from real measurements. CTBench therefore connects controlled capability evaluation to mechanistic diagnosis, providing a way to uncover weaknesses hidden by real-world benchmarks and guide the development of stronger Transformer-based forecasters.

Primary area:
datasets and benchmarks
Keywords:
Transformer-based Time-Series Forecasting, Transformers, Benchmark, Datasets

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vCNUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces CopyTestBench (CTBench), a diagnostic benchmark with known rules and zero Bayes error, designed to evaluate a model's ability to retrieve, route, and preserve information across channels. The authors assess twelve forecasting models, finding that ridge regression can perform pure-copy tasks with near-perfect accuracy, whereas most Transformer-based models perform poorly. Further analysis attributes the failure of iTransformer to issues with source routing and normalization; targeted interventions substantially improved its performance, with consistent results across different random seeds and on a copy task constructed from real-world measurements.

Critical strengths
  1. The overall structure of the manuscript is clear and logically organized.
  2. The experiments are comprehensive and support the main conclusions.
Critical weaknesses
  1. CTBench’s controlled copy tasks are useful for diagnosis but do not capture the noise and complex dependencies of real-world forecasting, which may limit the generalizability of the findings.
  2. While CTBench enables the diagnosis of retrieval failures under controlled conditions, the paper does not evaluate whether the proposed improvements enhance predictive performance on the original real-world forecasting tasks.
Questions for authors
  1. Why is R2=0.95R^2= 0.95 chosen as the threshold for considering a task successfully solved?
2 comments

Thank you for your review. We address your concerns below and describe the additional experiments and planned revisions. The reviewer notes that the controlled copy tasks do not capture the noise and the complex dependencies of real-world forecasting.

We agree that CTBench is not a realistic forecasting task. It is not designed to be one. The absence of noise is what makes it a diagnostic. With zero Bayes error, all remaining error belongs to the model. Real data cannot offer this. We present CTBench as a unit test of one capability (Section 1), not as a predictor of rankings on real benchmarks. Passing is therefore necessary for this capability, but not sufficient for better accuracy in general. We nevertheless tested whether the findings survive each kind of added complexity.

Observation noise. We added Gaussian noise to the target with variance equal to 10% and 50% of the noise-free target variance. This lowers the best achievable R2R^2 to 0.91 and 0.67. Ridge reaches at least 96% of this ceiling, and iTransformer only 65 to 69%. The ranking of models does not change.

More complex dependency. With lookback 192, we built a target that combines two sources at two different delays, Yt=Xt−96(1)+Xt−144(3)Y_t = X_{t-96}^{(1)} + X_{t-144}^{(3)}. Ridge recovers it exactly. iTransformer reaches shape R2R^2 is 0.12.

More and stronger distractors. With five distractors at autocorrelation 0.9, ridge is still exact. iTransformer's shape R2R^2 falls from 0.17 to 0.09.

Real signals. On ETTh1 measurements with a constructed copy target (Appendix J), the unmodified iTransformer reaches R2R^2 0.543. The intervention reaches 0.978. It can be applied here only because we built the target and therefore know which channel it copies.

In the revised paper we will add the noisy-target, two-source and distractor variants as a new appendix section. We will also update item 4 of the limitations (Appendix L), which currently says that the target has no observation noise.

The reviewer notes that the paper does not evaluate whether the proposed improvements enhance predictive performance on the original real-world forecasting tasks.

We first want to clarify what the intervention is. It supplies the correct attention route by hand. This is possible only because the benchmark tells us which channel holds the answer. A real forecasting dataset has no known route, so the oracle intervention cannot be applied to it. For this reason the paper presents it as a mechanistic test and not as a proposed method (Section 1, Section 5 and Appendix L). The part that could be carried to real forecasting is the change to normalization. We have one result in this direction. With ten independent training series and ordinary training, iTransformer with shared-statistics normalization and the encoder LayerNorm removed recovers most of the copy (shape R2R^2 0.83), against 0.29 for iTransformer with its original normalization under the same training. The two changes were applied together, so we cannot yet say which one is responsible. The details are in our reply to Reviewer 3a2U. We have not evaluated this change on standard forecasting benchmarks, and we make no claim about it. The paper states this as open question Q3. In the revised paper we will state this limit in the abstract and in the conclusion, so that the paper claims a diagnostic of one capability and nothing about accuracy on real forecasting tasks.

Why is R2≥0.95R^2 \geq 0.95 chosen as the threshold?

It is an operational convention, not a theoretical boundary. R2≥0.95R^2 \geq 0.95 means that the remaining error is at most 5% of the variance of the evaluated target. The exact solution has R2=1R^2 = 1, and ridge reaches at least 0.999 on every pure-copy dataset. The threshold therefore marks high-accuracy recovery, not numerical exactness. The conclusions do not depend on its exact value. On the pure-copy datasets the best model other than ridge reaches 0.776 and iTransformer at most 0.103 (Section 4), so any threshold between 0.78 and 0.999 gives the same verdicts. Because raw R2R^2 still credits a model for predicting each window's level, we also report a shape R2R^2 that removes each window's mean (Section 3).

Reviewer_vCNUOpenReview ↗

Response to authors

Thank you for the detailed response and additional analyses, which address my concerns. I will therefore raise my rating from 2 to 3.

Reviewer_3a2UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CopyTestBench, a diagnostic benchmark for evaluating cross-channel retrieval in Transformer-based time-series forecasting. The benchmark constructs delayed-copy tasks with known optimal solutions and zero Bayes error, covering pure-copy, daily switching, and rotating switching settings with varying distractors. Experiments on twelve forecasting models show that Ridge regression nearly solves the pure-copy tasks, while several Transformer models struggle to recover the required dependencies. The authors further investigate iTransformer through attention analysis, input-output Jacobians, and normalization interventions. Oracle routing combined with normalization changes improves its R² from 0.101 to 0.987, suggesting potential bottlenecks in source routing and information preservation.

Critical strengths
  1. Clear motivation and interpretable benchmark design. The zero-Bayes-error construction provides a useful setting for distinguishing failures to learn cross-channel dependencies from irreducible forecasting errors. Including simple reference baselines and an exact linear solution makes the evaluation more informative than aggregate forecasting metrics alone.

  2. Useful mechanistic analysis. The paper goes beyond reporting model errors by examining attention weights, input-output Jacobians, and intermediate representations. The controlled interventions on routing and normalization provide meaningful evidence about where information may be lost.

Critical weaknesses
  1. The benchmark covers a rather narrow range of retrieval tasks. All 27 main configurations use the same pair of random-walk source trajectories, with lookback, horizon, and delay fixed at 96. Although additional source draws and alternative processes are examined, they cover only selected settings. It remains unclear whether the reported model failures generalize to different retrieval distances, source distributions, and more complex cross-channel dependencies.

  2. The model comparisons are sensitive to training choices. Most neural models are evaluated under a shared learning rate and training budget, despite substantial differences in their original training configurations. Appendix E.3 shows that Vanilla Transformer reaches R² = 0.967 on one pure-copy dataset after changing the learning-rate schedule. This raises concerns about whether some observed failures reflect architectural limitations or insufficient optimization.

  3. The mechanistic interventions rely heavily on oracle information. The strongest result in Table 2 (R² = 0.987) requires explicitly routing attention to the correct source, together with changes to instance normalization and encoder layer normalization. While this demonstrates that the modified architecture can express the solution, it does not show that the model can learn the required routing under ordinary training. The intervention also changes the original input representation, making its implications for standard forecasting architectures less clear.

  4. The connection to real-world forecasting remains limited. The additional ETTh1 experiment uses real measurements but still constructs an artificial delayed-copy target. It therefore does not establish that the identified bottlenecks are important for ordinary forecasting tasks.

Questions for authors

As above.

3 comments

Thank you for your review. We address each concern below.

Generalization across distances, distributions and dependencies. We extended the pure-copy task with the variants below. All retain zero Bayes error. Results are means over three independent benchmark instances, except the reference and AR(1) settings, which use five. Each instance has independent training, validation and test series. Training uses ten epochs and validation-selected settings.

\hline \textbf{Variant} & \textbf{Ridge shape } R^2 & \textbf{iTransformer shape } R^2 \\\\ \hline \text{Reference: lookback 96, delay 96} & 1.00 & 0.17 \\\\ \text{Lookback 192, delay 96} & 1.00 & 0.43 \\\\ \text{Lookback 192, delay 144} & 1.00 & 0.13 \\\\ \text{Lookback 192, delay 192} & 1.00 & 0.04 \\\\ \text{Stationary AR(1) sources} & 1.00 & 0.21 \\\\ \text{Five distractors with autocorrelation 0.9} & 1.00 & 0.09 \\\\ \text{Two sources at two delays: } Y(t)=X_1(t-96)+X_3(t-144) & 1.00 & 0.12 \\\\ \hline \end{array}$$ Ridge remains essentially exact, while iTransformer's weak recovery persists. On AR(1) sources, Vanilla Transformer reaches $R^2=0.983$ and shape $R^2=0.982$, showing that recovery depends on the architecture and source process. CARD and PatchTST also fail to solve any of 15 datasets spanning five source draws and three distractor autocorrelations. CARD's shape $R^2$ never exceeds 0.72, and PatchTST's is negative throughout. We also evaluated switching on five independent instances, each containing one training, two validation and four test series. Each model uses one training seed per instance and ten epochs. Settings selected on the first instance's validation data are fixed across all five. Under rotating switching, no evaluated trained model beats gate-aware persistence (GP) on any instance with default or selected settings. Mean MSE under selected settings is 0.075 for GP, 0.738 for iTransformer and 4.094 for Vanilla Transformer. Daily-switching values are 0.077, 0.277 and 2.657, respectively. **Optimization**. Tuning rescues Vanilla Transformer. At constant learning rate $10^{-4}$ and 100 epochs, three-seed mean $R^2$ is $0.969 \pm 0.005$ and shape $R^2$ is $0.960 \pm 0.004$ (mean and SD). Every seed exceeds 0.95 on both metrics. With the same rate and budget, iTransformer's shape $R^2$ values are -0.090, -0.103 and 0.154. More independent training series improve its recovery but leave it below the threshold. We retuned CARD, PatchTST and the SAMformer family using four learning rates, each with the model's own schedule and a constant schedule, for 20 epochs. Vanilla Transformer already reaches $R^2=0.967$ at 20 epochs under its successful constant-rate setting. We also evaluated the other models' published settings, including budgets up to 100 epochs. None of the runs reached $R^2=0.95$. Repeating each model's best setting with two additional seeds gave a highest three-seed mean of 0.72, for Simple Transformer with RevIN. Thus, the tested tuning alternatives rescue Vanilla Transformer but do not bring the other evaluated models to the threshold. **Oracle routing**. We agree that the oracle result alone does not establish learned routing. However, Its purpose is to test whether supplying the correct route permits recovery and to identify which normalization choices limit that recovery. We also tested learned attention in iTransformer. With ten independent training series per instance, shared normalization and encoder LayerNorm removed, mean shape $R^2$ reaches 0.832 versus 0.288 for iTransformer. Both use ten epochs at constant $10^{-4}$, across three independent instances with one training seed each. This supports partial learned recovery without prescribed routing. Shared normalization uses the observed target window's statistics and changes the input representation. To test the gate/distractor scale cues described in Appendix L, we removed them in the target-row-only oracle control. Recovery remains high: $R^2=0.987$ without these cues versus 0.982 with them. The learned-attention experiment still requires this control. We will add these results and clarify their scope in Section 5 and the intervention appendix. **Relevance to ordinary forecasting**. CopyTestBench is a diagnostic benchmark. It tests whether a model can perform one specific operation, retrieving a delayed segment from another channel, under conditions where the correct answer is known. The ETTh1 experiment shows that the same failure and the same repair appear when the source has real measurement statistics. It does not show that this failure explains errors on ordinary forecasting benchmarks, where the relevant cross-channel dependencies are unknown. We will state this scope explicitly in the introduction and in the limitations in Appendix L.

Thank you for your response to the review comments. I still have the following two questions:

  1. Regarding Learned Attention, the results reported by the authors (Shape R² = 0.832) are based on only three independent benchmark instances, with each instance using just a single training seed. Could the authors clarify whether the model maintains this performance improvement when potential scale-related shortcuts are excluded?

  2. Furthermore, the normalization modifications proposed by the authors have not yet been evaluated on standard, real-world time-series forecasting benchmarks. Beyond the artificially constructed delayed-copy task, do the authors have any experimental evidence demonstrating that these modifications improve performance on conventional, real-world time-series forecasting tasks?

Thank you for the follow-up.

Learned attention without scale shortcuts, with more seeds. Under shared statistics every channel is normalized with the target's window mean and standard deviation. We therefore added a version, "shared, sources only", in which only the two source channels and the target are normalized with the target's statistics. The gate and the distractor are normalized with their own statistics, as in the stock model, so they no longer contain any information about the target's scale.

We also increased the evidence from three runs to nine per version (three generated benchmark instances, three training seeds each). All other settings are unchanged (ten training series, learned attention, same training budget).

\hline \textbf{Normalization} & \textbf{Encoder LayerNorm} & \textbf{Shape }R^2\textbf{, mean of 9 runs} & \textbf{Range over runs} \\\\ \hline \text{Per-channel (stock)} & \text{on} & 0.28 & \text{0.20 to 0.34} \\\\ \hline \text{Per-channel} & \text{off} & 0.28 & \text{0.24 to 0.33} \\\\ \hline \text{Shared} & \text{on} & 0.86 & \text{0.75 to 0.90} \\\\ \hline \text{Shared} & \text{off} & 0.82 & \text{0.71 to 0.88} \\\\ \hline \text{Shared, sources only} & \text{on} & 0.86 & \text{0.80 to 0.91} \\\\ \hline \text{Shared, sources only} & \text{off} & 0.84 & \text{0.71 to 0.91} \\\\ \hline \end{array}$$ The gain does not depend on the scale cue. With the target's statistics given only to the sources, shape $R^2$ stays at 0.86 (LayerNorm on) and 0.84 (LayerNorm off), against 0.28 for the stock model. The result is also stable across seeds. The weakest of the 18 sources-only runs scores 0.71, and the best of the 18 per-channel runs scores 0.34, so the two groups do not overlap on any instance or seed. **Standard real-world benchmarks**. The modifications in layer normalisation and scaling each channel are diagnostic interventions, designed to locate where cross-channel copy information is lost inside iTransformer. They are *not proposed as a forecasting method*. It is about cross-channel retrieval, and *it does not claim better accuracy on standard forecasting*. However, We trained iTransformer on ETTh1, Weather and Exchange at horizons 96, 192, 336 and 720, using the original authors' per-dataset hyperparameters and three seeds per setting. Shared statistics need one designated target, so all runs forecast the OT channel from all channels. We compare four versions: stock (Original model), LayerNorm removed, shared statistics, and both changes together. Test MSE, mean of three seeds (change vs stock in brackets): $$\begin{array}{|l|c|c|c|c|c|} \hline \textbf{Dataset} & \textbf{Horizon} & \textbf{Stock} & \textbf{LayerNorm off} & \textbf{Shared stats} & \textbf{Both} \\\\ \hline \text{ETTh1} & 96 & 0.058 & 0.057\\ (-1\\%) & 0.061\\ (+5\\%) & 0.094\\ (+63\\%) \\\\ \hline & 192 & 0.074 & 0.073\\ (-1\\%) & 0.081\\ (+10\\%) & 0.115\\ (+56\\%) \\\\ \hline & 336 & 0.084 & 0.083\\ (-1\\%) & 0.095\\ (+14\\%) & 0.195\\ (+134\\%) \\\\ \hline & 720 & 0.085 & 0.100\\ (+17\\%) & 0.163\\ (+90\\%) & 0.369\\ (+332\\%) \\\\ \hline \text{Weather} & 96 & 0.00133 & 0.00126\\ (-6\\%) & 0.00130\\ (-2\\%) & 0.00862\\ (+547\\%) \\\\ \hline & 192 & 0.00159 & 0.00154\\ (-3\\%) & 0.00158\\ (-1\\%) & 0.00991\\ (+523\\%) \\\\ \hline & 336 & 0.00172 & 0.00170\\ (-2\\%) & 0.00160\\ (-7\\%) & 0.00907\\ (+426\\%) \\\\ \hline & 720 & 0.00219 & 0.00219\\ (0\\%) & 0.00200\\ (-9\\%) & 0.00890\\ (+307\\%) \\\\ \hline \text{Exchange} & 96 & 0.105 & 0.099\\ (-6\\%) & 0.102\\ (-3\\%) & 0.126\\ (+20\\%) \\\\ \hline & 192 & 0.206 & 0.209\\ (+1\\%) & 0.199\\ (-4\\%) & 0.246\\ (+19\\%) \\\\ \hline & 336 & 0.400 & 0.408\\ (+2\\%) & 0.366\\ (-8\\%) & 0.540\\ (+35\\%) \\\\ \hline & 720 & 1.083 & 1.120\\ (+3\\%) & 0.808\\ (-25\\%) & 0.721\\ (-33\\%) \\\\ \hline \end{array}$$ The modifications do not give a **general improvement** on standard benchmarks. - Removing LayerNorm is neutral. It stays within a few percent of stock everywhere except ETTh1 at horizon 720. - Shared statistics depend on the dataset. They are worse on ETTh1 at every horizon, neutral to slightly better on Weather, and better on Exchange at long horizons. The Exchange gains come where the stock error is already very large, so we do not read them as evidence of a better forecaster. - Combining both changes is clearly harmful on ETTh1 and Weather. This matches the role of the modifications in the paper. Our results show that the same choice also removes the cross-channel level and scale information that a copy task needs. The benchmark results mark the limit of our claim. We will add both tables to the appendix and make this scope explicit in the main text.
Reviewer_XfmpOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CopyTestBench, a diagnostic benchmark for cross-channel retrieval in time-series forecasting. Its exact-copy tasks have zero Bayes error and vary source switching and distractor properties. Beyond model comparisons, the paper uses Jacobians, representation probes, and controlled interventions to investigate routing and normalization failures in iTransformer.

Critical strengths
  1. The benchmark has a known exact solution and includes simple reference baselines, making model errors easier to interpret. Reporting both standard and shape R² helps distinguish prediction of the overall window level from recovery of within-window variation.

  2. The paper combines Jacobian analysis, layer-wise probes, and controlled routing and normalization interventions to investigate iTransformer’s copying failure. The source-process controls and learning-rate sweep also clarify that the observed failures depend on the architecture, data, and training settings.

The controlled benchmark and intervention-based diagnosis provide a useful contribution. Although the scope is narrow and some controls remain incomplete, the paper generally presents its findings with appropriate qualifications.

Critical weaknesses
  1. Robustness is less established for the switching tasks. The main grid shares one source-path pair, while the independent-source experiments and learning-rate sweep cover only pure-copy settings. Since adjusting the learning rate and schedule enables Vanilla Transformer to solve the reference copy task, it remains unclear whether the switching failures persist across new source realizations and training settings.

  2. Although the best intervention is replicated across three seeds, most other intervention configurations use only one seed. Repeating the main ablations across seeds would strengthen confidence in the effects attributed to routing and normalization.

Questions for authors
  1. Have you evaluated rotating switching across independent source draws or alternative learning-rate schedules? If so, does the failure to outperform gate-aware persistence persist?

  2. Beyond replicating the best intervention, have you repeated the main routing and normalization ablations across seeds? If so, are the relative effects consistent?

2 comments

Thank you for your review. We address your concerns below and describe the additional experiments and planned revisions. The question was whether the rotating-switching failure persists on independent source draws and with other learning-rate schedules. Yes, we ran both checks, and the failure persists. Under rotating switching, no model outperforms Gate-aware persistence (GP), which predicts each future step by repeating the most recent observed target value with the same gate state, on any independent source draw, with any tested learning rate or schedule.

Table 1 reports the mean of three training seeds. We also checked every run separately. There are 270 rotating-switching runs: ten attention-based models, nine configurations and three seeds. None of them has a lower MSE than GP. Ridge regression and XGBoost are also above GP on all nine configurations. We additionally generated five independent benchmark instances for each switching mode. Each instance contains one training series, two validation series, and four test series, generated independently. Each model was retrained on each instance with one training seed and a ten-epoch budget.

Learning-rate search. We tested three types of schedule.

  • Halving: the default of the implementation, which halves the learning rate after each epoch.
  • Constant: the learning rate stays fixed during training.
  • Cosine: the learning rate decreases smoothly from its starting value to zero, following a cosine curve.

For iTransformer we tested seven configurations: the constant and cosine schedules at starting rates 10−410^{-4}, 3×10−43 \times 10^{-4} and 10−310^{-3}, plus the default halving schedule. For Vanilla Transformer we tested four configurations. One of them is the constant rate 10−410^{-4} that solves the pure-copy task in Appendix E.3 (Table 8). We selected the configuration on the validation data of the first instance only. We then kept it fixed for the other four instances.

Results. Neither model outperforms GP on any of the five instances. This holds in both switching modes, with the selected configuration and with the default one. Across five benchmark instances, rotating switching yielded mean test MSE of 0.075 for gate-aware persistence, compared with 0.738 for iTransformer and 4.094 for Vanilla Transformer. For daily switching, the corresponding MSE values were 0.077, 0.277, and 2.657, respectively. The failure to outperform GP under rotating switching is not specific to the source paths of the main grid. It is also not specific to the shared learning rate and schedule. The daily-switching advantage reported in Table 1 is less robust and depends on the source realization. We will add the independent-draw switching experiment and the seed-by-seed check as a new appendix section.

You asked whether we repeated the main routing and normalization ablations across seeds, and whether the relative effects are consistent. Yes. We repeated all seven configurations of Table 2 with three training seeds. The three main effects have the same direction and a similar size in every seed.

Table 2 with three seeds. Same reference dataset and same ten-epoch protocol as Table 2. Values are means and standard deviations across seeds.

\hline \textbf{Attention} & \textbf{Instance norm} & \textbf{Encoder LN} & R^2 \\\\ \hline \text{learned} & \text{per-channel} & \text{on (stock)} & 0.090 \pm 0.033 \\\\ \hline \text{learned} & \text{shared} & \text{off} & -0.043 \pm 0.107 \\\\ \hline \text{oracle} & \text{none} & \text{on} & 0.696 \pm 0.110 \\\\ \hline \text{oracle} & \text{per-channel} & \text{on} & 0.795 \pm 0.008 \\\\ \hline \text{oracle} & \text{per-channel} & \text{off} & 0.817 \pm 0.016 \\\\ \hline \text{oracle} & \text{shared} & \text{on} & 0.905 \pm 0.013 \\\\ \hline \text{oracle} & \text{shared} & \text{off} & 0.987 \pm 0.001 \\\\ \hline \end{array}$$ We also crossed all the factors: learned or oracle attention, three instance-normalization settings (per-channel, shared, none), and encoder LayerNorm on or off. We used three new seeds and two schedules, halving and constant. The effects repeat under both schedules. The last row shows that high recovery does not require fixing every attention row. It is enough to fix the attention of the row of the target token and leave the others learned. $$\begin{array}{|l|c|c|} \hline \textbf{Configuration} & R^2\\ \textbf{halving} & R^2\\ \textbf{constant} \\\\ \hline \text{Stock} & 0.058 & 0.155 \\\\ \hline \text{Oracle routing, original normalization} & 0.806 & 0.761 \\\\ \hline \text{Oracle routing, shared statistics, LN on} & 0.904 & 0.873 \\\\ \hline \text{Oracle routing, per-channel statistics, LN off} & 0.818 & 0.791 \\\\ \hline \text{Oracle routing, shared statistics, LN off} & 0.988 & 0.980 \\\\ \hline \text{Only the target row clamped, shared statistics, LN off} & 0.982 & 0.975 \\\\ \hline \end{array}$$

Thank you for the rebuttal. I do not have further questions and have decided to maintain my positive rating.

Comments on the paper

1 comment

We sincerely thank reviewers Xfmp and vCNU for their positive evaluations of the paper.

We would also like to reiterate the intended contribution of this work. CopyTestBench is not proposed as a new forecasting architecture intended to improve aggregate MSE on standard real-world benchmarks. Its purpose is diagnostic: to isolate a specific, fully observable forecasting capability, reveal methodological limitations that aggregate benchmarks can conceal, and provide mechanistic evidence about where current forecasters fail. The normalization and routing interventions are therefore diagnostic probes, not a deployable forecasting method. Rather than pursuing another incremental improvement on increasingly saturated benchmarks, CTBench raises concrete questions about what current state-of-the-art models can and cannot compute, increases awareness of these limitations, and provides a controlled basis for developing stronger architectures in future work. Designing such an improved forecasting method is an important follow-up direction, but it is deliberately outside the scope of this benchmark paper.

We hope our responses clarified this point enough. If any concerns remain for reviewer v3a2U, we would be very happy to address them during the remaining short discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.