- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces CopyTestBench (CTBench), a diagnostic benchmark with known rules and zero Bayes error, designed to evaluate a model's ability to retrieve, route, and preserve information across channels. The authors assess twelve forecasting models, finding that ridge regression can perform pure-copy tasks with near-perfect accuracy, whereas most Transformer-based models perform poorly. Further analysis attributes the failure of iTransformer to issues with source routing and normalization; targeted interventions substantially improved its performance, with consistent results across different random seeds and on a copy task constructed from real-world measurements.
Critical strengths
- The overall structure of the manuscript is clear and logically organized.
- The experiments are comprehensive and support the main conclusions.
Critical weaknesses
- CTBench’s controlled copy tasks are useful for diagnosis but do not capture the noise and complex dependencies of real-world forecasting, which may limit the generalizability of the findings.
- While CTBench enables the diagnosis of retrieval failures under controlled conditions, the paper does not evaluate whether the proposed improvements enhance predictive performance on the original real-world forecasting tasks.
Questions for authors
- Why is chosen as the threshold for considering a task successfully solved?
2 comments
Response to authors
Thank you for the detailed response and additional analyses, which address my concerns. I will therefore raise my rating from 2 to 3.
Thank you for your review. We address your concerns below and describe the additional experiments and planned revisions. The reviewer notes that the controlled copy tasks do not capture the noise and the complex dependencies of real-world forecasting.
We agree that CTBench is not a realistic forecasting task. It is not designed to be one. The absence of noise is what makes it a diagnostic. With zero Bayes error, all remaining error belongs to the model. Real data cannot offer this. We present CTBench as a unit test of one capability (Section 1), not as a predictor of rankings on real benchmarks. Passing is therefore necessary for this capability, but not sufficient for better accuracy in general. We nevertheless tested whether the findings survive each kind of added complexity.
Observation noise. We added Gaussian noise to the target with variance equal to 10% and 50% of the noise-free target variance. This lowers the best achievable R2 to 0.91 and 0.67. Ridge reaches at least 96% of this ceiling, and iTransformer only 65 to 69%. The ranking of models does not change.
More complex dependency. With lookback 192, we built a target that combines two sources at two different delays, Yt=Xt−96(1)+Xt−144(3). Ridge recovers it exactly. iTransformer reaches shape R2 is 0.12.
More and stronger distractors. With five distractors at autocorrelation 0.9, ridge is still exact. iTransformer's shape R2 falls from 0.17 to 0.09.
Real signals. On ETTh1 measurements with a constructed copy target (Appendix J), the unmodified iTransformer reaches R2 0.543. The intervention reaches 0.978. It can be applied here only because we built the target and therefore know which channel it copies.
In the revised paper we will add the noisy-target, two-source and distractor variants as a new appendix section. We will also update item 4 of the limitations (Appendix L), which currently says that the target has no observation noise.
The reviewer notes that the paper does not evaluate whether the proposed improvements enhance predictive performance on the original real-world forecasting tasks.
We first want to clarify what the intervention is. It supplies the correct attention route by hand. This is possible only because the benchmark tells us which channel holds the answer. A real forecasting dataset has no known route, so the oracle intervention cannot be applied to it. For this reason the paper presents it as a mechanistic test and not as a proposed method (Section 1, Section 5 and Appendix L). The part that could be carried to real forecasting is the change to normalization. We have one result in this direction. With ten independent training series and ordinary training, iTransformer with shared-statistics normalization and the encoder LayerNorm removed recovers most of the copy (shape R2 0.83), against 0.29 for iTransformer with its original normalization under the same training. The two changes were applied together, so we cannot yet say which one is responsible. The details are in our reply to Reviewer 3a2U. We have not evaluated this change on standard forecasting benchmarks, and we make no claim about it. The paper states this as open question Q3. In the revised paper we will state this limit in the abstract and in the conclusion, so that the paper claims a diagnostic of one capability and nothing about accuracy on real forecasting tasks.
Why is R2≥0.95 chosen as the threshold?
It is an operational convention, not a theoretical boundary. R2≥0.95 means that the remaining error is at most 5% of the variance of the evaluated target. The exact solution has R2=1, and ridge reaches at least 0.999 on every pure-copy dataset. The threshold therefore marks high-accuracy recovery, not numerical exactness. The conclusions do not depend on its exact value. On the pure-copy datasets the best model other than ridge reaches 0.776 and iTransformer at most 0.103 (Section 4), so any threshold between 0.78 and 0.999 gives the same verdicts. Because raw R2 still credits a model for predicting each window's level, we also report a shape R2 that removes each window's mean (Section 3).