acceptodds
Withdrawn from ICLR 2027

Pre-Pre-Training Before Pixels and Genes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

16 views

Abstract

Foundation models trained on distinct modalities are hypothesized to converge toward similar representations. We turn this observation around and ask whether some universal prior can be learned from synthetic data before exposure to natural data and how the learned priors transfer across different domains. We investigate this question through synthetic pre-pre-training, in which transformers are first trained on data generated from structured processes without any target-domain semantics followed by domain-specific pre-training. We treat synthetic pre-pre-training as a design space, varying the synthetic substrate (neural cellular automata or Dyck formal language), training objective (masked reconstruction or autoregressive prediction), and target domain (vision and genomics). Across three vision benchmarks and ten genomics tasks, we observe that synthetic pre-pre-training generally improves downstream performance, but that its effectiveness depends on our studied three design factors. In vision, masked reconstruction consistently outperforms autoregressive prediction for both pre-pre-training sources. Masked 2D NCA pre-pre-training achieves the highest accuracy on all three benchmarks, improving upon the randomly initialized and trained baseline by on average. In genomics, both synthetic sources transfer effectively, with autoregressive pre-pre-training on Dyck sequences outperforming the randomly initialized baseline by approximately 6 points on NTv3 mean score and 2.6% on the Genomics Classification tasks. These results suggest that synthetic pre-pre-training is a promising way to instill general computational priors before domain-specific pre-training.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
pre-pre-training, pretraining, synthetic data, genomics, vision, representation learning, neural cellular automata

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TSo8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work explores pre-pre-training, an additional pre-training phase on synthetic data. The authors vary three design factors: synthetic data source (neural cellular automata and Dyck formal languages), training objective (autoregressive prediction and masked reconstruction), and downstream domain (vision and genomics). Experiments show that pre-pre-training consistently improves downstream performance in both domains (vs. training from scratch).

Critical strengths

This paper investigates synthetic pre-pre-training and demonstrates that representations learned from synthetic data can transfer across both vision and genomics domains.

Critical weaknesses
  1. The idea of pre-pre-training lacks novelty. Prior work has demonstrated that a two-stage pre-training (for example, MAE pre-training followed by weakly supervised pre-training) can significantly improve downstream performance for billion-scale models[1]. [1] The effectiveness of MAE pre-pretraining for billion-scale pretraining, ICCV 2023.

  2. The motivation for using synthetic data for the first pre-training stage is insufficiently justified. The authors note that ''unlabeled genomic sequences are abundant, but generating high-quality functional annotations is challenging'' (Line 77). However, this observation only supports the value of self-supervised learning on unlabeled data; it does not explain why synthetic data is needed or advantageous as the first pre-training stage.

  3. The experimental design is not logically sound. To properly demonstrate the advantage of pre-pre-training (i.e., two-stage pre-training), comparisons should be made against standard single-stage pre-training. Instead, the paper only benchmarks pre-pre-training against training from scratch, mimetic initialization, and FractalDB (Sec. 3.3, Fig. 3–4). This leaves the incremental benefit of adding a synthetic pre-pre-training stage unclear. Additionally, experiments are conducted exclusively on the small-scale parameters (eg, ViT-Tiny), which limits the generalizability of the findings to larger foundation model scales.

Questions for authors
  1. Please clarify the motivation for using synthetic data for the first pre-training stage. Specifically, what unique advantages does synthetic pre-pre-training offer relative to self-supervised pre-training performed directly on natural unlabeled data from the target domain?

  2. Please add baseline results for standard single-stage pre-training to the paper, so that the incremental performance gain from the additional synthetic pre-pre-training stage can be properly quantified.

Reviewer_a11MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether training transformers on structured synthetic data before natural-domain training provides transferable initialization. It compares neural cellular automata and Dyck sequences under autoregressive and masked objectives, transferring backbone parameters to vision and genomics. The experiments report benefits from masked NCA in vision and Dyck variants in genomics, including after substantial natural-sequence pretraining. The intended contribution is an empirical study of how synthetic source, training objective, and target domain influence transfer.

Critical strengths
  1. Transferring the backbone tests a meaningful form of reusable computation. Replacing the domain-specific input/output interfaces while retaining transformer blocks asks whether internal processing learned from synthetic structure is useful beyond its original vocabulary. Evaluating both images and genomic sequences makes this question more informative than a single-domain warm-start result.
  2. The study exposes task-dependent compatibility rather than presenting one universal recipe. The contrast between masked NCA in vision and Dyck variants in genomics suggests that the usefulness of synthetic structure depends on how it is learned and subsequently used. The mimetic initialization and FractalDB comparisons provide useful initial alternatives, and the persistence of genomic gains after natural pretraining makes the observed effect worth understanding.
Critical weaknesses
  1. The experiment does not isolate the objective as a causal design factor. Table 4 changes the vision Dyck context length from 1,024 to 196, batch size from 32 to 256, and transfer checkpoint from 5,000 to 15,000 updates between AR and masked training, alongside vocabulary, learning rate, weight decay, and embedding treatment. These choices alter the learned computation and optimization independently of the objective. Thus the result supports a better complete recipe, but not the stronger conclusion that masking itself causes better transfer. Matching these factors, or narrowing the claim to recipe comparisons, would resolve this mismatch between the question and experiment.
  2. The transferred benefit is not yet identified as a general computational prior. Removing semantic content does not remove all domain-relevant structure: NCA locality and geometry can supply inductive biases, and additional optimization can provide a favorable initialization without learning the proposed algorithmic skills. The fixed downstream budget does not distinguish these explanations. In addition, the domain-specific backbone and positional configurations mean that success in both domains does not demonstrate one shared learned initialization transferring unchanged to both. Structure-destroying controls and a matched-total-compute baseline would clarify what is being transferred and what practical advantage the synthetic stage provides.
Questions for authors
  1. Clarification: Which Table 4 differences are necessary for the AR/masked comparison, and which arose from recipe tuning? Does an objective-only comparison with matched generator, vocabulary, context, embedding treatment, and compute support the same conclusion? If not available, would you restrict the conclusion to the evaluated complete recipes?
  2. Evidence affecting the recommendation: Does the benefit survive comparison with equally costly additional natural-domain training and with synthetic data whose temporal or nesting structure has been disrupted while retaining basic token statistics? These controls would distinguish useful structure from a generic optimization warm start.
  3. Clarification: Are any identical pretrained backbone weights transferred to both domains, or are all pre-pretraining models domain-configured separately? Please distinguish generality of the procedure from universality of one learned prior. Also clarify the NCA geometry and temperature actually used, since Section 3.1 and Table 3 report different values; these determine the structure whose transfer is being interpreted.
Reviewer_fHM3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether pre-pre-training transformers on structured synthetic data can provide useful inductive biases for downstream tasks. The authors study two synthetic data sources, neural cellular automata (NCA) and Dyck formal languages, combined with autoregressive and masked reconstruction objectives. They evaluate these approaches on three image classification benchmarks and ten genomics tasks, comparing against random and mimetic initialization baselines. The results show that synthetic pre-pre-training generally improves downstream performance, although the best configuration depends on the target domain. Masked NCA performs best for vision, while Dyck-based pre-pre-training achieves the strongest average results in genomics. Notably, improvements in genomics persist even after extensive pre-training on natural DNA sequences.

Critical strengths
  1. Interesting and potentially useful idea. The idea of learning general computational priors from structured synthetic data before training on real data is compelling. If effective, this could provide a flexible way to improve model initialization without relying on additional domain-specific data. The approach is particularly interesting for domains where high-quality annotated data is limited.

  2. Broad empirical evaluation with promising results. The paper evaluates the approach across two distinct domains, vision and genomics, showing that different synthetic data sources and objectives can improve downstream performance. Particularly encouraging is that the improvements in genomics persist even after extensive pre-training on natural sequences, suggesting that synthetic pre-pre-training can provide benefits beyond early optimization.

Critical weaknesses
  1. Training objectives are not compared under the same conditions. The autoregressive and masked Dyck configurations differ substantially in learning rate, context length, batch size, and training steps (Appendix A, Table 4). This makes it difficult to attribute the performance differences in Figure 3 to the training objective.

  2. Lack of compute-matched baselines. The paper does not compare synthetic pre-pre-training against other forms of additional training with similar computational budgets. This makes it unclear whether the improvements come from the structure of the synthetic data or simply from additional training.

  3. [Minor] Motivation does not fully align with the experiments. The paper motivates its approach through the idea of universal representations across modalities (Introduction). However, the experiments use separate models and training configurations for vision and genomics, without investigating whether the learned representations are shared or aligned. The results support the usefulness of synthetic pre-pre-training across domains, but not necessarily the existence of universal representations.

Questions for authors
  1. Can the authors provide compute-matched comparisons against simpler pre-training alternatives, such as random or shuffled synthetic sequences? This would help establish whether the improvements come from learning the structure of the synthetic data rather than simply performing additional training.

  2. What motivated the different training configurations for autoregressive and masked Dyck pre-pre-training (Appendix A, Table 4)? Have the authors evaluated whether the performance differences persist when the training objectives are compared under matched conditions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.