- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates how structure, dimensionality, and distribution shape influence synthetic pretraining transfer
Critical strengths
The paper investigates an interesting research question regarding which properties of synthetic pretraining data influence downstream transfer. The experiments across different models and input preprocessing pipelines provide useful empirical observations about the sensitivity of transfer performance to data characteristics.
Critical weaknesses
The central claim of the paper, the Transfer-Invariance Principle, lacks sufficient theoretical and empirical justification. The authors argue that the model's input transformation, rather than the data modality, determines which statistical properties of pretraining data govern downstream transfer. However, no formal theoretical justification or conditions under which this principle holds are provided. In particular, per-window z-score normalization preserves higher-order moments such as skewness and kurtosis, so distribution shape remains accessible to the model. Furthermore, the normalization experiment in Section 4.5 does not isolate distribution shape from window scale, as the authors themselves acknowledge. The experiments also fail to independently control structure, dimensionality, and distribution shape, making it difficult to attribute the observed transfer improvements to specific properties. While the results demonstrate that input preprocessing can influence transfer performance, they do not sufficiently establish the broader Transfer-Invariance Principle or its generalizability across different models and modalities.
The authors make broad claims about the properties that govern the transferability of pretraining data across different modalities. However, the experimental evaluation is largely restricted to time-series classification and calcium imaging, with no experiments involving text pretraining or language models. Although the authors include language in their analysis in Table 1, they do not demonstrate whether the proposed properties of structure, dimensionality, and distribution shape meaningfully characterize transferable pretraining data for language models.
Questions for authors
Addressing all the weaknesses raised in the review would help resolve my concerns regarding the paper.
2 comments
Thank you for the review. Your point that z-scoring preserves higher moments is exactly the case that Proposition 1 now separates out, and your request to isolate shape from scale identified the mechanism.
No formal justification or conditions for the principle. In short: new Proposition 1 (Section 3.2) gives the condition under which a property provably cannot matter, T∘g = T. The confirming sweep covers the remaining, empirical part. Statement and proof sketch in our response to R2 Q1.
Z-scoring preserves skewness and kurtosis, so the shape stays accessible. In short: we agree, and this is what makes the null under normalization informative rather than a tautology.
- The model receives post-z-score excess kurtosis from 7 to 49, and transfer does not move (spread 0.011).
| Corpus | Excess kurtosis (after per-window z-score) | Per-window z-score | No normalization | Window SD matched |
|---|---|---|---|---|
| Gaussian anchor | 7.1 | 0.840 ± 0.006 | 0.839 ± 0.007 | 0.839 ± 0.007 |
| Warp p=1.5 | 16.7 | 0.838 ± 0.008 | 0.831 ± 0.014 | 0.841 ± 0.009 |
| Warp p=2.0 | 27.6 | 0.837 ± 0.008 | 0.792 ± 0.033 | 0.732 ± 0.085 |
| Warp p=3.0 | 49.3 | 0.829 ± 0.009 | 0.776 ± 0.050 | 0.690 ± 0.068 |
| No pretraining | — | 0.801 ± 0.009 | 0.807 ± 0.007 | 0.807 ± 0.007 |
UCR accuracy (Table 8; 22 datasets, 20 seeds).
- Removing normalization activates the same warp (5.8×). Matching each window's mean and SD exactly to the Gaussian anchor removes the effect again (0.25×). The input transform therefore decides whether a visible property affects transfer: a property that T removes cannot matter, and a property that T keeps can matter but need not.
Section 4.5 does not isolate shape from window scale. In short: it now does, and the result sharpens the mechanism (full table in our response to R1 W4).
- Matching only the distribution of window SDs, with window means free, raises the spread to 13.7×.
- Matching each window's mean and SD exactly removes the effect (0.25×; heaviest tail vs. anchor p = 0.50).
- The axis remains the distribution of each unit's values. A heavy-tailed unit reaches a windowed model as windows with a different mean and scale (top 1% of windows: 97% of input energy vs. 4% for the Gaussian corpus). The observed behavior across the three conditions is consistent with this channel: z-scoring closes it and the warp is inert, removing normalization opens it and the warp is active, and the exact window match closes it again. Proposition 1 guarantees only the first step, that changes the transform removes cannot matter; the rest is measured.
Structure, dimensionality and shape are not controlled independently. In short: each is now varied with the others held fixed.
- Shape at fixed structure. The monotone warp keeps every rank correlation; Spearman coupling is 0.393 at every warp level.
- Structure at fixed shape. The n_roots × warp factorial under four pipelines (10 seeds per cell; η² table in our response to R1 W4). Structure explains 78% of the run-level variance under per-window normalization and 51% under the exact window match, the warp 33% without normalization.
- Dimensionality at fixed structure and shape, on calcium. Effective rank moves 2.4 → 17.7, nearly the full real range (2.1–16.8), while coupling (0.109), skewness (4.07), excess kurtosis (54.7) and SNR (6.89) stay identical:
| Corpus effective rank | 2.4 | 6.4 | 12.1 | 17.7 |
|---|---|---|---|---|
| Mean ± SD (3 seeds) | 0.338 ± 0.006 | 0.210 ± 0.107 | 0.260 ± 0.125 | 0.315 ± 0.043 |
| Median | 0.341 | 0.177 | 0.301 | 0.319 |
POYO+, drifting gratings; chance 0.125. No monotone trend.
- Shape components on Chronos. Sweeping tail shape and SNR separately, the tail carries the effect, and SNR matters only near the noise floor.
- What calcium can and cannot isolate. In calcium, matching the heavy tail lowers coupling (0.89 → 0.18), and matching SNR lowers it to 0.05, so the calcium shape and calcium structure cannot be set independently. The calcium-shaped corpora also differ from the Gaussian one in unit scale (about 4× the mean and 1.5× the SD; Section 3.5), and so do the shape-only and SNR-only corpora of Figure 3c (SNR-only about 14× the SD). On calcium we therefore credit the transform as a whole. The causal effect of the shape is isolated on PatchTST and Chronos (Section 4.5; Limitations, Appendix I).
Generalizability beyond time series, e.g. language. In short: the claims now cover time series and neural data, tested on three architectures with three input transforms. The abstract speaks of time-series corpora, and the language and vision rows of Table 1 are labeled descriptive only. Language models (e.g. byte-level vs. tokenized inputs) are a natural next test.
General response to all reviewers
We thank the reviewers for careful and constructive reviews.
Each row is detailed, with its tables, in the response to the reviewer who asked.