- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes to apply a learnable time-warping layer prior to feeding a time series into a Neural Operator. To be specific, this layer:
- Depends on the input: the input observations and intervals influence the warping;
- Is stochastic: the parametrisation is defined by a Gaussian normal distribution;
- Preserves chronological order.
To accomodate for that, the authors also introduce a large complex multistep architecture, the "Hierarchical Multi-Scale Temporal Module".
Critical strengths
To the best of my knowledge, this specific combination of parts hasn't been explored by prior work (although I am not an expert in the area). I like the main idea that your paper seems to rest on: that time series has varying informativeness. Indeed: can learnable time warping could improve the quality of the methods; what are the properties of the time series that necessitate such warping, and what cases can serve as counter examples -- these are all very interesting research questions, alas, not tested clearly by this paper. To be clear: this is conjecture on my part, I was not able to find explicit motivation for your work in the paper's text.
Critical weaknesses
As already mentioned in the strengths, I could not find a clean and explicit motivation for your approach in your paper. That is a crucial component for all research papers, arguably more important than empirical results and novelty. Based solely on the main text, I cannot answer with certainty: what question/problem the authors are addressing here.
To be specific, usually in the Introduction, papers describe, what specific weaknesses prior work has, what problems are established for older approaches, and then the sets out to solve, or at least analyze these problems, with the proposed novel approach.
Your paper is built as a detailed technical report for the proposed method. The Introduction section recounts, roughly, what prior work did, and what they do, concluding that prior work didn't do the proposed temporal warping for Neural Operators. Without motivating, why one might be interested in doing such temporal warping, what benefits it may bring, what problems it solves that plagued prior approaches. The other sections also do not explain, what problem is solved, why one might want to solve this problem with this time-warping approach (possibly, what other approaches are to solving this problem, if any).
Consequently, your work establishes no portable findings, that one could then use in another paper, other than that this specific proposed architecture works good on some datasets but not others; without actually specifying, what is required for it to work.
The lack of motivation is not simply a wording issue. Your experiments should be testing that motivation, should be testing your hypotheses and the properties that your method supposedly has. It's beneficial to even lose in some places, given that you show what properties of the benchmark made your method lose -- that's also very valuable.
With the above in mind, the massive 2-4 times wins in the empirical results do not have a clean explanation. Therefore I had to come up with some possible explanations myself, all of them concerning the fairness of the evaluation:
- Your method has more parameters than the competitors. I cannot tell that based solely on the paper's text, I did not find the competitors' parameter counts. But, I checked against KRNO on Shallow Water -- and your model is ~30x bigger than KRNO. Therefore this isn't really a fair comparison. That is also mirrored by the ablations: Table 7 shows that most of the gains come from the switch from KRNO to your larger multistage hierarchical architecture, not from the main contribution, the temporal warping.
- The evaluation is not aligned between all methods. The numbers for other methods are copied from their papers, and your code might not be faithfully reproducing the evaluation routine that competitiors use. Indeed, I did find some mismatches, more on them below.
- The seeds are cherry-picked: 2021, 2024, 2025, 2027 -- these do look suspicious. However, these years do line up with the methods cited, and I don't believe that seed-picking on its own could have produced such a drastic change.
Eval mismatches. These are things to check in your provided code. I may very well be wrong, I did not have the time to reproduce the code or walk over it in detail, the list is from brief targeted reads:
- You always eval on all channels. KRNO uses last-channel eval for Crypto, for each of the 8 features.
- Your data preparation, prepare_temporal.py, does not seem to write weights, and therefore the eval may not load weights, scoring RMSE instead of WRMSE on Crypto.
- On MuJoCo, irregular with missingness, you interpolate the raw inputs prior to passing them into the model. KRNO does not, they handle missing observations via masks (I don't see masks passed into the forecasting model in your code). That could explain your constant score that's seemingly independent of missingness on MuJoCo.
- You use a 5-step-ahead forecasting routine for Shallow Water in your code. The competitors use a 0.6s recursive rollout, totalling 60 steps (recursively/autoregressively, 5 steps at a time).
Questions for authors
These questions are based on the weaknesses written above, refer there for more detail.
- How do you motivate your method? I would like to see both prior literature grounding in terms of the problem that your method could solve, some empirical validation for the existance of said problem in real data, for your competitors, and some (ideally also empirically-backed, or theory-backed) explanation as to why your method solves that problem.
- Are the eval mismatches I highlighted in my review true? Please, check these places, fix where necessary. I would like to see a fair comparison to the competitors, at matched model sizes, horizons, at random seeds, etc. That could go the other way, in which case you will need to be more specific with the choice of benchmarks, to make narrow and verifiable claims, aligned with the motivation.