- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work studies the effect of local (site-wise) standardisation with finite sample estimates changes translation equivariance compared to global standardisation when forecasting periodic systems (Lorenz-96). This effect can bias ablations experiments that compare the effect of positional encodings with equivariant controls. The results studying generalisation of the empirical findings to other systems (Allen–Cahn and Gray-Scott) are mixed.
However, arriving at this summary required an immense effort in reverse-engineering the very dense manuscript. It reads more like a compressed experimental log than a human-readable manuscript.
Critical strengths
The paper studies a practical problem in preprocessing data for equivariant forecasting. The study tries to carefully control parameter counts, random seeds, and validation sets. That means that the findings for the Lorenz-96 system are solid I think.
Critical weaknesses
Clarity and Presentation:
The manuscript is full of undefined acronyms and experimental jargon in the first 5 pages and it is immensely hard to follow and obfuscates the actual question and findings. (For example: POS, SHAM, P, G, and Mc are central to understanding Abstract and Introduction that also uses jargon like "locking selection", "metric certificates", and "five-block centering-only study". But what POS and SHAM are really only explained on Page 6.) The manuscript reads like a shorthand log without a clear scientific narrative (especially the last part of the introduction from line 048). Also the Results sections (particularly 5.1. - 5.4) are hard to parse and consist of numerical readouts and defensive statements about statistical caveats. The manuscript never cleanly explains the experimental setup, motivation, or intuition. In its current form, the paper does not communicate the assumptions, methods and findings in an understandable or verifiable way and requires a major rewrite of the first 6 pages at least.
Generalization of the findings:
In my understanding the central mathematical observation (that site-wise scaling does not commute with shifts if the statistics are not spatially constant) seems elementary. The more interesting fixed-backbone theory in Section 3.3 does not predict the supervision effect in the actual neural network experiments (Section 5.2) if I understood this correctly. This would leave a very narrow conclusion for the paper.
Questions for authors
- I would ask how the authors plan to rewrite the first half of the paper to make it accessible so that a reader with a standard Machine Learning background can follow the motivation, definitions and experimental setting and interpret the reported results.
- Why do you think the jointly trained CNN in Section 5.2 goes in the opposite direction of Proposition 2?
- Why does the frozen-backbone adapter in Section 5.1 fail to reproduce the repair effect?
The authors did not submit a rebuttal, so the concerns raised in the review remain unaddressed.