acceptodds
Under review as a conference paper at ICLR 2027

Structure, Dimensionality, and Distribution Shape: What Makes Pretraining Data Transferable, and Why the Input Pipeline Decides Which

Abstract

What makes a pretraining corpus transferable? We describe a corpus by three properties: its structure, its dimensionality, and the distribution shape of each unit’s values. We propose the Transfer-Invariance Principle: the same property can be irrelevant to transfer under one input transform and decisive under another, so the model’s input transform, not the data modality, decides which properties can matter. To test it we build TransferForge, a generator that places a synthetic population at any prescribed structure, dimensionality, and distribution shape. KernelSynth, the synthetic prior of Chronos, cannot, because it draws independent series. When the model z-scores each input window (PatchTST on UCR), the distribution shape is inert and the structural setting governs. Synthetic pretraining then matches a real corpus of the same size and exceeds it once scaled. When the model encodes calcium ∆F/F unnormalized (POYO+), the shape becomes a governing constraint, and a shape-matched synthetic corpus replaces real pretraining with zero real neural activity. Flipping only the input normalization, with the model and data fixed, changes the effect of a shape manipulation 5.3×, so the pipeline, not the modality, sets the switch. The principle yields a recipe that chooses what to synthesize before any training. Pretraining data then becomes something to manufacture for the model, most valuably where real data is scarce, expensive, or private.

Primary area:
learning on time series and dynamical systems
Keywords:
synthetic pretraining, transfer invariance, time-series foundation models, pretraining data design, distribution shift, neural decoding, foundation models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2vagOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how structure, dimensionality, and distribution shape influence synthetic pretraining transfer

Critical strengths

The paper investigates an interesting research question regarding which properties of synthetic pretraining data influence downstream transfer. The experiments across different models and input preprocessing pipelines provide useful empirical observations about the sensitivity of transfer performance to data characteristics.

Critical weaknesses

The central claim of the paper, the Transfer-Invariance Principle, lacks sufficient theoretical and empirical justification. The authors argue that the model's input transformation, rather than the data modality, determines which statistical properties of pretraining data govern downstream transfer. However, no formal theoretical justification or conditions under which this principle holds are provided. In particular, per-window z-score normalization preserves higher-order moments such as skewness and kurtosis, so distribution shape remains accessible to the model. Furthermore, the normalization experiment in Section 4.5 does not isolate distribution shape from window scale, as the authors themselves acknowledge. The experiments also fail to independently control structure, dimensionality, and distribution shape, making it difficult to attribute the observed transfer improvements to specific properties. While the results demonstrate that input preprocessing can influence transfer performance, they do not sufficiently establish the broader Transfer-Invariance Principle or its generalizability across different models and modalities.

The authors make broad claims about the properties that govern the transferability of pretraining data across different modalities. However, the experimental evaluation is largely restricted to time-series classification and calcium imaging, with no experiments involving text pretraining or language models. Although the authors include language in their analysis in Table 1, they do not demonstrate whether the proposed properties of structure, dimensionality, and distribution shape meaningfully characterize transferable pretraining data for language models.

Questions for authors

Addressing all the weaknesses raised in the review would help resolve my concerns regarding the paper.

2 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for careful and constructive reviews.

  • What the paper contributes. We identify which properties of a synthetic pretraining corpus can govern transfer, show that the model's input transform constrains which of them matter (the Transfer-Invariance Principle), and operationalize this in TransferForge, a generator that makes these properties explicit settings. The paper therefore does not stop at what matters: it turns what matters into something one can manufacture. Proposition 1 rules out properties the transform removes, and one confirming sweep checks the rest; the three properties do not determine all of transfer.
  • What the reviews asked. All three reviews ask for the same two controls. First, separate the distribution shape from the window scale in the normalization flip (R1, R2, R3). Second, vary structure, dimensionality and shape independently (R1, R3). R2 and R3 also ask for a formal statement of the principle. The revision adds each, and the controls identify the channel through which the shape reaches a windowed model.
  • Scope. For a target model, the generator produces data at the setting that sweep selects. Under per-window z-scoring (PatchTST on UCR), the low-dimensional setting transfers 4% better than the high-dimensional one and, scaled up, 4.2% above a real corpus. Under unnormalized calcium input (POYO+), the default setting needs no calcium-specific step. The axes are the properties we set, not a complete description of a corpus. The claims cover time series and neural data.
What the controls showEvidence
A property that the input transform removes cannot change transferNew Proposition 1 (Section 3.2). Z-scoring keeps kurtosis (7 → 49), yet the shape warp is inert, so surviving the transform is necessary, not sufficient
Without normalization, the shape acts through each window's mean and scaleMatching each window's mean and SD exactly to a Gaussian anchor removes the shape effect (heaviest tail 0.840 vs. anchor 0.843, p = 0.50)
The pipeline decides whether structure or shape governsStructure explains 78% of run-level variance under per-window z-scoring and the shape 33% without normalization (10 seeds per cell)
On calcium, TransferForge data needs no calcium-specific stepWith no real data it is not distinguishable from full-real pretraining at 5 seeds. Added to 19 real sessions, every finetuning run leaves chance (10/10 vs. 3/5); the gain over the 19 sessions alone is significant for the Gaussian mixture (p = 0.028)

Each row is detailed, with its tables, in the response to the reviewer who asked.

  • Corrections. While running these controls we found two errors in the submission and corrected both. (1) The Gaussian + scarce-real corpus (0.220) drew 43 of its 50 "synthetic" sessions from real recordings. Rerun with 50 synthetic sessions, it achieves 0.368 ± 0.058 (5 seeds). (2) The calcium-shaped units were rescaled by the real unit's mean and SD from an unstandardized transform output, so they sit at about 4× the real mean and 1.5× the real SD. Section 3.5 and Appendix F now state this.
  • What we withdraw. (1) The replacement headline for calcium, that synthetic-only pretraining outperforms real pretraining (R2). (2) The 56% advantage of the calcium-shaped corpus over the Gaussian one in the scarce-real mixture, which came from the error above. (3) The interior optimum in n_roots. The UCR data show a drop at the high-dimensional end instead. The language and vision rows of Table 1 are now labeled descriptive only (R3).

Thank you for the review. Your point that z-scoring preserves higher moments is exactly the case that Proposition 1 now separates out, and your request to isolate shape from scale identified the mechanism.

No formal justification or conditions for the principle. In short: new Proposition 1 (Section 3.2) gives the condition under which a property provably cannot matter, T∘g = T. The confirming sweep covers the remaining, empirical part. Statement and proof sketch in our response to R2 Q1.

Z-scoring preserves skewness and kurtosis, so the shape stays accessible. In short: we agree, and this is what makes the null under normalization informative rather than a tautology.

  • The model receives post-z-score excess kurtosis from 7 to 49, and transfer does not move (spread 0.011).
CorpusExcess kurtosis (after per-window z-score)Per-window z-scoreNo normalizationWindow SD matched
Gaussian anchor7.10.840 ± 0.0060.839 ± 0.0070.839 ± 0.007
Warp p=1.516.70.838 ± 0.0080.831 ± 0.0140.841 ± 0.009
Warp p=2.027.60.837 ± 0.0080.792 ± 0.0330.732 ± 0.085
Warp p=3.049.30.829 ± 0.0090.776 ± 0.0500.690 ± 0.068
No pretraining—0.801 ± 0.0090.807 ± 0.0070.807 ± 0.007

UCR accuracy (Table 8; 22 datasets, 20 seeds).

  • Removing normalization activates the same warp (5.8×). Matching each window's mean and SD exactly to the Gaussian anchor removes the effect again (0.25×). The input transform therefore decides whether a visible property affects transfer: a property that T removes cannot matter, and a property that T keeps can matter but need not.

Section 4.5 does not isolate shape from window scale. In short: it now does, and the result sharpens the mechanism (full table in our response to R1 W4).

  • Matching only the distribution of window SDs, with window means free, raises the spread to 13.7×.
  • Matching each window's mean and SD exactly removes the effect (0.25×; heaviest tail vs. anchor p = 0.50).
  • The axis remains the distribution of each unit's values. A heavy-tailed unit reaches a windowed model as windows with a different mean and scale (top 1% of windows: 97% of input energy vs. 4% for the Gaussian corpus). The observed behavior across the three conditions is consistent with this channel: z-scoring closes it and the warp is inert, removing normalization opens it and the warp is active, and the exact window match closes it again. Proposition 1 guarantees only the first step, that changes the transform removes cannot matter; the rest is measured.

Structure, dimensionality and shape are not controlled independently. In short: each is now varied with the others held fixed.

  • Shape at fixed structure. The monotone warp keeps every rank correlation; Spearman coupling is 0.393 at every warp level.
  • Structure at fixed shape. The n_roots × warp factorial under four pipelines (10 seeds per cell; η² table in our response to R1 W4). Structure explains 78% of the run-level variance under per-window normalization and 51% under the exact window match, the warp 33% without normalization.
  • Dimensionality at fixed structure and shape, on calcium. Effective rank moves 2.4 → 17.7, nearly the full real range (2.1–16.8), while coupling (0.109), skewness (4.07), excess kurtosis (54.7) and SNR (6.89) stay identical:
Corpus effective rank2.46.412.117.7
Mean ± SD (3 seeds)0.338 ± 0.0060.210 ± 0.1070.260 ± 0.1250.315 ± 0.043
Median0.3410.1770.3010.319

POYO+, drifting gratings; chance 0.125. No monotone trend.

  • Shape components on Chronos. Sweeping tail shape and SNR separately, the tail carries the effect, and SNR matters only near the noise floor.
  • What calcium can and cannot isolate. In calcium, matching the heavy tail lowers coupling (0.89 → 0.18), and matching SNR lowers it to 0.05, so the calcium shape and calcium structure cannot be set independently. The calcium-shaped corpora also differ from the Gaussian one in unit scale (about 4× the mean and 1.5× the SD; Section 3.5), and so do the shape-only and SNR-only corpora of Figure 3c (SNR-only about 14× the SD). On calcium we therefore credit the transform as a whole. The causal effect of the shape is isolated on PatchTST and Chronos (Section 4.5; Limitations, Appendix I).

Generalizability beyond time series, e.g. language. In short: the claims now cover time series and neural data, tested on three architectures with three input transforms. The abstract speaks of time-series corpora, and the language and vision rows of Table 1 are labeled descriptive only. Language models (e.g. byte-level vs. tokenized inputs) are a natural next test.

Reviewer_3wyeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Many models are pre-trained using synthetic data, with the implicit assumption that this pre-training will then transfer effectively to the real tasks. The paper studies whether this transfer is actually guaranteed, and, more specifically, which properties of the synthetic pre-training data determines the transferability. Three dimensions are proposed: structure, dimensionality and distribution shape. The central hypothesis, called the Transfer-Invariance Principle, is that the relevance of these properties depends mostly on the model's input pre-processing pipeline rather than on the data modality itself. This hypothesis is challenged by a synthetic data generator instroduced in the paper, called TransferForge, and designed to control these properties. This generator is used to evaluate synthetic pre-training accross several architectures (time series analysis, neural decoding and forecasting).

Critical strengths

The research problem addressed by the problem is generally interesting. I had never thought of it myself, since I rarely work with synthetic pre-training, but it makes complete sense and I enjoyed reading about it and following the authors in their study. I think that it is probably a relevant problem for many in the ML community, and therefore that the paper could be largely interesting. What is particularly interesting in the approach taken by the authors is that they do not simply demonstrate the effectiveness of synthetic pre-training, but they try to identify which properties make the transfer successful. Even though the paper is not theoretically grounded, this study is interesting and important. in particular, the empirical results, performed in diverse experimental settings, are promising and suggest that carefully designed synthetic corpora may substantially reduce the need for large real pre-training data. These results are complemented bz the introduction of a synthetic data generation framework, that provides a concrete way to generate datasets with different structural and statistical characteristics.

Critical weaknesses

My principal frustration with this paper is that it is not providing any theoretical perspective on its claims. As much as I appreciated the transfer-invariance principle and its empirical demonstration, I think it would benefit from getting a strong(er) theoretical backup. This is particularly the case since I feel that the principle lacks sufficient evidence. The paper makes strong claims about it, but the experimental evidence is more limited. For instance, the normalization experiment does not fully disentangle distribution shape from scale effect.

Globally, the evaluation covers relatively few architectures and downstream tasks. As a consequence, it is difficult to assess whether the conclusions extend beyond the considered settings. Several experiments rely on very few seeds (max 3), and some configurations have unequal training budgets. This limits the statistical significance of the conclusions. Following this direction, some of the strongest claims (e.g. synthetic pretraining outperforming real pretraining) are not supported by statistically significant differences.

Finally, I have concerns about the practical value of the proposed principle. The authors present a procedure to identify which data properties matter before training, but Algorithm 1 ultimately requires evaluating several pretrained configurations. It is therefore closer to a guided hyperparameter search than a genuinely predictive method. Furthermore, constructing the synthetic datasets still requires real data statistics and calibration, somewhat qualifying the claims about replacing real pretraining data.

Questions for authors
  • Is there any strong theoretical evidence supporting the transfer-invariance principle?
  • Can you answer my doubts about Algorithm 1 requiring evaluating several pretrained configurations?
3 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for careful and constructive reviews.

  • What the paper contributes. We identify which properties of a synthetic pretraining corpus can govern transfer, show that the model's input transform constrains which of them matter (the Transfer-Invariance Principle), and operationalize this in TransferForge, a generator that makes these properties explicit settings. The paper therefore does not stop at what matters: it turns what matters into something one can manufacture. Proposition 1 rules out properties the transform removes, and one confirming sweep checks the rest; the three properties do not determine all of transfer.
  • What the reviews asked. All three reviews ask for the same two controls. First, separate the distribution shape from the window scale in the normalization flip (R1, R2, R3). Second, vary structure, dimensionality and shape independently (R1, R3). R2 and R3 also ask for a formal statement of the principle. The revision adds each, and the controls identify the channel through which the shape reaches a windowed model.
  • Scope. For a target model, the generator produces data at the setting that sweep selects. Under per-window z-scoring (PatchTST on UCR), the low-dimensional setting transfers 4% better than the high-dimensional one and, scaled up, 4.2% above a real corpus. Under unnormalized calcium input (POYO+), the default setting needs no calcium-specific step. The axes are the properties we set, not a complete description of a corpus. The claims cover time series and neural data.
What the controls showEvidence
A property that the input transform removes cannot change transferNew Proposition 1 (Section 3.2). Z-scoring keeps kurtosis (7 → 49), yet the shape warp is inert, so surviving the transform is necessary, not sufficient
Without normalization, the shape acts through each window's mean and scaleMatching each window's mean and SD exactly to a Gaussian anchor removes the shape effect (heaviest tail 0.840 vs. anchor 0.843, p = 0.50)
The pipeline decides whether structure or shape governsStructure explains 78% of run-level variance under per-window z-scoring and the shape 33% without normalization (10 seeds per cell)
On calcium, TransferForge data needs no calcium-specific stepWith no real data it is not distinguishable from full-real pretraining at 5 seeds. Added to 19 real sessions, every finetuning run leaves chance (10/10 vs. 3/5); the gain over the 19 sessions alone is significant for the Gaussian mixture (p = 0.028)

Each row is detailed, with its tables, in the response to the reviewer who asked.

  • Corrections. While running these controls we found two errors in the submission and corrected both. (1) The Gaussian + scarce-real corpus (0.220) drew 43 of its 50 "synthetic" sessions from real recordings. Rerun with 50 synthetic sessions, it achieves 0.368 ± 0.058 (5 seeds). (2) The calcium-shaped units were rescaled by the real unit's mean and SD from an unstandardized transform output, so they sit at about 4× the real mean and 1.5× the real SD. Section 3.5 and Appendix F now state this.
  • What we withdraw. (1) The replacement headline for calcium, that synthetic-only pretraining outperforms real pretraining (R2). (2) The 56% advantage of the calcium-shaped corpus over the Gaussian one in the scarce-real mixture, which came from the error above. (3) The interior optimum in n_roots. The UCR data show a drop at the high-dimensional end instead. The language and vision rows of Table 1 are now labeled descriptive only (R3).
AuthorsOpenReview ↗

Q1 (1/2) and related weakenesses

Thank you for the positive and thoughtful review. Your question about theory led to Proposition 1, which states exactly which part of the principle is provable.

Q1. Is there theoretical support for the Transfer-Invariance Principle? In short: yes, for one direction. A property that the input transform removes provably cannot change transfer (new Proposition 1, Section 3.2; proof in Appendix J).

  • Statement. Let T be the input transform, P the corpus distribution, and g a manipulation of one corpus property. If T∘g = T on the support of P, then T#(g#P) = T#P. Any training procedure that sees the data only through T then yields the same distribution of weights, and so the same transfer.
  • Consequence. Per-window z-scoring removes every per-window affine change, such as a window's mean or scale, so those changes cannot affect PatchTST. The controls in our response to R1 W4 confirm this: matching each window's mean and SD removes the shape effect even without normalization.
  • The converse does not hold, and the data show it. Z-scoring keeps skewness and kurtosis (post-z-score excess kurtosis 7 → 49), so the shape is visible to PatchTST, yet transfer stays within seed noise (spread 0.011). Surviving the transform is therefore necessary, not sufficient. This is the precise scope of the principle ("the transform decides which properties can matter"), and it is why the recipe ends with one confirming sweep.

The normalization experiment does not disentangle shape from scale. In short: it now does. Matching each window's mean and SD to the Gaussian anchor removes the shape effect (0.25× the normalized spread; p = 0.50), so without normalization the shape acts through these two window statistics. Full table in our response to R1 W4.

Few seeds and unequal budgets. In short: every UCR row now has 10 seeds and every calcium row 5 seeds, and corpora compared directly share the same budget.

  • UCR. Table 2 and the scaling curve (Figure 2) are rerun at 10 seeds. All UCR conclusions hold: ours vs. KernelSynth +1.3% (8/10 seeds, p = 0.015), ours vs. real at equal size p = 0.56, and the scaled corpus 4.2% above real at 16,384 windows.
Windows128512102420484096819216384
Ours (n_roots=4)0.801 ± 0.0050.804 ± 0.0090.803 ± 0.0080.812 ± 0.0110.819 ± 0.0070.838 ± 0.0060.843 ± 0.008
KernelSynth0.803 ± 0.0080.802 ± 0.0110.806 ± 0.0050.811 ± 0.0080.824 ± 0.0100.837 ± 0.0060.841 ± 0.009

UCR accuracy against corpus size (mean ± SD over 10 seeds; real corpus at 1776 windows 0.808 ± 0.011; no pretraining 0.803 ± 0.007).

  • Synthetic data can be generated in any quantity, and that pays. A real corpus is fixed at 1776 windows. TransferForge data keeps improving with size and exceeds the real corpus by 4.2% at 16,384 windows. KernelSynth also gains from scale and ends at the same accuracy, so scale is an advantage of synthetic data in general. TransferForge adds the choice of what to scale: at a fixed size, its setting alone changes UCR accuracy by 4% (Section 4.2).

  • Calcium. Every row of Table 3 now has 5 finetuning seeds and reports the median and the number of runs that leave chance:

Pretraining corpusMean ± SDMedianRuns leaving chance
Calcium-shaped synthetic (0 real traces)0.344 ± 0.1270.3974/5
Gaussian synthetic (0 real traces)0.338 ± 0.0970.4005/5
Calcium-shaped synthetic + scarce real (19 sessions)0.343 ± 0.0520.3415/5
Gaussian synthetic + scarce real (19 sessions)0.368 ± 0.0580.3935/5
Full real (77 sessions; reference)0.387 ± 0.0140.3825/5
Scarce real (19 sessions)0.239 ± 0.1140.2913/5
No pretraining0.316 ± 0.0330.3155/5

POYO+, drifting gratings, 15-epoch full finetune on the same 139 sessions; held-out test accuracy at the best-validation checkpoint; chance 0.125; a run leaves chance above 0.15.

  • Budgets. Corpora compared directly share the same pretraining budget or step count (Section 4.1). The caption of Figure 3 lists the two exceptions: the single-seed, 5-epoch n_roots sweep of Figure 3a, which is not compared with the other panels, and runs rerun at 12 instead of 15 finetuning epochs after hitting the cluster time limit.
AuthorsOpenReview ↗

Q1 (2/2) & Q2

Synthetic outperforming real is not statistically supported. In short: we agree and withdrew that claim. We now report two calcium findings, each with its sample-size limit.

  • TransferForge needs no calcium-specific step on calcium. With no real data, its Gaussian output and the calcium-shaped corpus are not distinguishable from full-real pretraining (Mann–Whitney p ≥ 0.55); with 5 seeds and SDs up to 0.13 this test has limited power, so we claim neither equivalence nor superiority. 7 of their 10 runs exceed the best no-pretraining run (0.365), and their medians are 26–27% above it. Finetuning runs that stay at chance widen the spread: two finetuning seeds of the same checkpoint give 0.383 and 0.118, so the stall comes from finetuning, not from the corpus.
  • Synthetic data supplements scarce real data. The submission reported this as its second calcium result. 19 real sessions alone fall 24% below no pretraining, with 2 of 5 runs at chance. Adding 50 synthetic sessions raises them by 44% (calcium-shaped) and 54% (Gaussian), and every run leaves chance (10/10). Against the 19 sessions alone the gain is significant for the Gaussian mixture (one-sided Mann–Whitney p = 0.028) and a direction for the calcium-shaped one (p = 0.075). Against no pretraining, the mixtures are 9% and 17% higher, which is not significant (p ≥ 0.075).
  • Correction. The submitted Gaussian mixture (0.220) drew 43 of its 50 "synthetic" sessions from real recordings. Rerun with 50 synthetic sessions (seeds 42–46: 0.393, 0.267, 0.407, 0.376, 0.399), it achieves 0.368 ± 0.058, so the 56% shape contrast in the mixture is withdrawn.

Few architectures and tasks. In short: the principle is tested on three architectures with three different input transforms, and the flip replicates on three corpora.

  • PatchTST (per-window z-score), Chronos (mean scaling and quantization), and POYO+ (unnormalized), across classification, forecasting and neural decoding.
  • The normalization flip holds on our corpus, on real FRED-MD/exchange series (2.9× wider spread without normalization), and on KernelSynth (2.0×). The warp is inert under normalization on every base.

Q2. Algorithm 1 still evaluates several pretrained configurations. In short: yes, its last step trains, and the saving comes from pruning before that step.

  • Cost. The three axes have four measures (structure, dimensionality, and tail and SNR for the shape). Testing each at three levels takes 12 pretraining runs one measure at a time and 81 for the full grid. The recipe trains three runs only for each measure that survives the transform.
  • Calcium. Step 1 marks structure dead with no training: after the calcium transform, coupling varies < 2× across n_roots, within the spread of real sessions. The confirming n_roots sweep is indeed flat.
  • UCR. Step 1 leaves the shape live, and the sweep finds it inert. This is the case Proposition 1 leaves open, and it is why the confirming sweep stays. Section 4.6 now states this cost.

Real data statistics are still needed. In short: the revision lists exactly what each corpus takes from real data (Appendix F).

  • The UCR corpus uses no real data.
  • The calcium corpus uses no real activity trace. It takes three summary statistics of one calibration session (skewness, excess kurtosis, SNR), each template unit's mean and SD, and the file format and stimulus metadata of the templates. Its units sit at about 4× the real mean and 1.5× the real SD, because the transform output is rescaled without being standardized first (Section 3.5, Appendix F).
Reviewer_gzsAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work claims that three properties of data, structure, dimension, and the shape of the distribution, determine the transferability of synthetic pretraining data. Structure is measured by mean absolute pairwise correlation, dimensionality by effective rank, and distribution shape by SNR. The authors then propose a generator, TransferForge, and conduct experiments on multiple datasets to demonstrate that the results support their claim.

Critical strengths
  • The idea of designing synthetic pretraining data according to the model’s input processing is interesting
  • The experiments cover multiple benchmarks and include classification, forecasting, and neural decoding.
Critical weaknesses
  • The paper lacks sufficient preliminaries to help readers understand the relevant concepts, for example, CauKer.
  • The "structure" uses Pearson correlation, which measures the linear relationship. It may not capture nonlinear or lagged dependencies.
  • As an extension of CauKer, this paper lacks a direct comparison with its predecessor. It's difficult to evaluate how much gain the proposed pipeline contributes.
  • The experiments do not sufficiently isolate the factors claimed to govern transfer, for example, the experiment (Figure 4(a) ) changes both distribution shape and window scale
Questions for authors

None.

3 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for careful and constructive reviews.

  • What the paper contributes. We identify which properties of a synthetic pretraining corpus can govern transfer, show that the model's input transform constrains which of them matter (the Transfer-Invariance Principle), and operationalize this in TransferForge, a generator that makes these properties explicit settings. The paper therefore does not stop at what matters: it turns what matters into something one can manufacture. Proposition 1 rules out properties the transform removes, and one confirming sweep checks the rest; the three properties do not determine all of transfer.
  • What the reviews asked. All three reviews ask for the same two controls. First, separate the distribution shape from the window scale in the normalization flip (R1, R2, R3). Second, vary structure, dimensionality and shape independently (R1, R3). R2 and R3 also ask for a formal statement of the principle. The revision adds each, and the controls identify the channel through which the shape reaches a windowed model.
  • Scope. For a target model, the generator produces data at the setting that sweep selects. Under per-window z-scoring (PatchTST on UCR), the low-dimensional setting transfers 4% better than the high-dimensional one and, scaled up, 4.2% above a real corpus. Under unnormalized calcium input (POYO+), the default setting needs no calcium-specific step. The axes are the properties we set, not a complete description of a corpus. The claims cover time series and neural data.
What the controls showEvidence
A property that the input transform removes cannot change transferNew Proposition 1 (Section 3.2). Z-scoring keeps kurtosis (7 → 49), yet the shape warp is inert, so surviving the transform is necessary, not sufficient
Without normalization, the shape acts through each window's mean and scaleMatching each window's mean and SD exactly to a Gaussian anchor removes the shape effect (heaviest tail 0.840 vs. anchor 0.843, p = 0.50)
The pipeline decides whether structure or shape governsStructure explains 78% of run-level variance under per-window z-scoring and the shape 33% without normalization (10 seeds per cell)
On calcium, TransferForge data needs no calcium-specific stepWith no real data it is not distinguishable from full-real pretraining at 5 seeds. Added to 19 real sessions, every finetuning run leaves chance (10/10 vs. 3/5); the gain over the 19 sessions alone is significant for the Gaussian mixture (p = 0.028)

Each row is detailed, with its tables, in the response to the reviewer who asked.

  • Corrections. While running these controls we found two errors in the submission and corrected both. (1) The Gaussian + scarce-real corpus (0.220) drew 43 of its 50 "synthetic" sessions from real recordings. Rerun with 50 synthetic sessions, it achieves 0.368 ± 0.058 (5 seeds). (2) The calcium-shaped units were rescaled by the real unit's mean and SD from an unstandardized transform output, so they sit at about 4× the real mean and 1.5× the real SD. Section 3.5 and Appendix F now state this.
  • What we withdraw. (1) The replacement headline for calcium, that synthetic-only pretraining outperforms real pretraining (R2). (2) The 56% advantage of the calcium-shaped corpus over the Gaussian one in the scarce-real mixture, which came from the error above. (3) The interior optimum in n_roots. The UCR data show a drop at the high-dimensional end instead. The language and vision rows of Table 1 are now labeled descriptive only (R3).
AuthorsOpenReview ↗

W1 & W2 & W3

Thank you for the detailed review. Your requests for a direct CauKer comparison and for separating shape from window scale led to the two controls that now carry the paper.

W1. Missing preliminaries, e.g. CauKer. In short: the revision now introduces both generators and each model's input transform where they are first used.

  • How KernelSynth and CauKer generate data (Related work, "Synthetic pretraining data"). KernelSynth draws independent series from Gaussian processes with randomly composed kernels. CauKer places a few GP root series in a random structural causal model and samples nodes as univariate series.
  • Which parameters each generator has (same paragraph). Neither has a parameter that sets structure, dimensionality or distribution shape.
  • The input transform T of each model (opening of Section 3.2). PatchTST z-scores each 256-sample window. POYO+ embeds each ΔF/F sample linearly without normalization. Chronos divides by the mean absolute value and quantizes.

W2. Pearson correlation misses nonlinear and lagged dependence. In short: four nonlinear and lagged measures order the populations as Pearson does.

  • We recomputed structure on the same channel pairs of every TransferForge population (28 grid settings × 3 warp levels, plus the calcium corpora) with Spearman correlation, distance correlation, kNN mutual information, and the largest |cross-correlation| over lags ±10 (new Appendix C).
PopulationsnSpearmandistance corr.mutual informationlagged xcorr
Gaussian280.9950.9970.9620.997
Warped, p=2280.9980.9980.9681.000
Warped, p=3280.9940.9970.9530.996
Calcium-transformed50.9000.9000.6000.800
Range, Gaussian(0.05–0.73)(0.07–0.73)(0.01–4.11)(0.09–0.66)
Range, calcium(0.03–0.05)(0.05–0.08)(0.008–0.014)(0.06–0.09)

Rank agreement (Spearman ρ across populations) of each measure with Pearson coupling; range of each measure in parentheses.

  • The calcium transform lowers structure under every measure. Across n_roots 1–64 the calcium corpora vary by at most 2.1× under any measure, while the Gaussian populations vary 7–14× (Pearson 13×, Spearman 14×, distance correlation 10×, lagged cross-correlation 7×; mutual information about 300×). The structure collapse of Section 4.4 is therefore not an artifact of linear correlation.
  • The shape manipulation leaves rank structure exactly unchanged. The warp of Figure 4a is a monotone pointwise map, so Spearman coupling is 0.393 at every warp level.

W3. No direct comparison with CauKer. In short: CauKer is now in Table 2 at both corpus sizes. At about 10k windows TransferForge outperforms CauKer and KernelSynth. Its practical advantage is that the setting is exposed and can be chosen for the pipeline. At the 1776 windows of the real corpus, the synthetic priors and the real corpus tie.

  • We ran the official CauKer generator at its default settings (18 nodes, ≤ 6 parents, 4 series of length 512 per graph) through the same PatchTST, budget and 22-dataset probe, with 10 seeds.
Pretraining corpus~10k windows1776 windows
Ours (n_roots=4)0.842 ± 0.0060.811 ± 0.013
CauKer (official)0.836 ± 0.0080.811 ± 0.006
KernelSynth (official)0.832 ± 0.0080.813 ± 0.006
Real (FRED-MD + exchange)—0.808 ± 0.011
No pretraining0.803 ± 0.007
  • At about 10k windows our corpus is 0.8% above CauKer (8 of 10 seeds, paired t-test p = 0.045; 15 of 22 datasets, Wilcoxon p = 0.07) and 1.3% above KernelSynth (8/10, p = 0.015). At 1776 windows the three synthetic corpora and the real corpus tie (p ≥ 0.27).
  • Why setting the property matters. CauKer couples series through a random graph, but no CauKer parameter sets the coupling strength or the number of independent modes, so a CauKer corpus lands wherever its random graphs place it. TransferForge sets both. For PatchTST on UCR, the principle points to the low-dimensional setting (n_roots = 1–4), which transfers 4% better than the high-dimensional one (n_roots = 64) at the same size. A practitioner who needs a different pipeline sets a different property with the same generator.

W4. Figure 4a changes both distribution shape and window scale. In short: we agree, and the controls you asked for show that the shape acts through each window's mean and scale. The input transform still decides whether it acts.

  • Why the window statistics carry the shape. A model sees each unit only through finite windows, so a heavy-tailed unit reaches it as windows with a different mean and scale. Without normalization, the top 1% of windows carry 97% of the input energy at the heaviest tail, against 4% for the Gaussian corpus.
  • The controls (normalization off in both; 22 UCR datasets; new Tables 8 and 9 in Appendix H):
Pipeline (PatchTST, 22 UCR datasets)Spread across warp levelsvs. normalizedSeeds
Per-window z-score0.0111×20
No normalization0.0635.8×20
No normalization, distribution of window SD matched to the anchor (window means free)0.15113.7×20
No normalization, window mean and SD matched exactly to the anchor window at the same position0.0030.25×10
  • The last row is the clean test. The corpora then differ only in within-window shape, and the warp no longer matters: the heaviest-tailed corpus achieves 0.840 against 0.843 for the anchor (Welch p = 0.50). Per-window z-scoring removes exactly these two statistics, which is why the warp is inert with normalization on. Section 4.5 is rewritten accordingly.
  • Structure under the same controls. Crossing n_roots {1, 4, 64} with the warp shows that the pipeline decides which factor governs (main-text Table 4):
Per-window z-scoreNo normalizationWindow SD matchedWindow mean and SD matched
Structure (n_roots), η²0.780.010.030.51
Warp (distribution shape), η²0.020.330.390.01

Every cell (UCR accuracy, mean ± SD over 10 seeds; Table 9):

n_rootsWarpPer-window z-scoreNo normalizationWindow SD matchedWindow mean and SD matched
1Gaussian0.840 ± 0.0080.837 ± 0.0100.836 ± 0.0090.836 ± 0.009
1p=20.838 ± 0.0070.808 ± 0.0200.796 ± 0.0780.840 ± 0.011
1p=30.837 ± 0.0090.789 ± 0.0230.697 ± 0.0690.842 ± 0.011
4Gaussian0.842 ± 0.0060.842 ± 0.0080.843 ± 0.0080.843 ± 0.008
4p=20.839 ± 0.0050.783 ± 0.0450.738 ± 0.0810.841 ± 0.008
4p=30.831 ± 0.0110.780 ± 0.0590.691 ± 0.0620.840 ± 0.011
64Gaussian0.806 ± 0.0100.824 ± 0.0070.824 ± 0.0060.824 ± 0.006
64p=20.800 ± 0.0090.808 ± 0.0210.797 ± 0.0260.821 ± 0.009
64p=30.801 ± 0.0100.776 ± 0.0450.757 ± 0.0980.811 ± 0.014
—No pretraining0.803 ± 0.0070.807 ± 0.0080.807 ± 0.0080.807 ± 0.008
  • Under the exact match the warp is inert at every n_roots (largest change 1.6%), and the structure effect returns: n_roots = 4 outperforms n_roots = 64 by 2–4% at every warp level (Welch p < 1e-4). n_roots = 1 and 4 do not differ (p ≥ 0.10).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.