- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work explores pre-pre-training, an additional pre-training phase on synthetic data. The authors vary three design factors: synthetic data source (neural cellular automata and Dyck formal languages), training objective (autoregressive prediction and masked reconstruction), and downstream domain (vision and genomics). Experiments show that pre-pre-training consistently improves downstream performance in both domains (vs. training from scratch).
Critical strengths
This paper investigates synthetic pre-pre-training and demonstrates that representations learned from synthetic data can transfer across both vision and genomics domains.
Critical weaknesses
-
The idea of pre-pre-training lacks novelty. Prior work has demonstrated that a two-stage pre-training (for example, MAE pre-training followed by weakly supervised pre-training) can significantly improve downstream performance for billion-scale models[1]. [1] The effectiveness of MAE pre-pretraining for billion-scale pretraining, ICCV 2023.
-
The motivation for using synthetic data for the first pre-training stage is insufficiently justified. The authors note that ''unlabeled genomic sequences are abundant, but generating high-quality functional annotations is challenging'' (Line 77). However, this observation only supports the value of self-supervised learning on unlabeled data; it does not explain why synthetic data is needed or advantageous as the first pre-training stage.
-
The experimental design is not logically sound. To properly demonstrate the advantage of pre-pre-training (i.e., two-stage pre-training), comparisons should be made against standard single-stage pre-training. Instead, the paper only benchmarks pre-pre-training against training from scratch, mimetic initialization, and FractalDB (Sec. 3.3, Fig. 3–4). This leaves the incremental benefit of adding a synthetic pre-pre-training stage unclear. Additionally, experiments are conducted exclusively on the small-scale parameters (eg, ViT-Tiny), which limits the generalizability of the findings to larger foundation model scales.
Questions for authors
-
Please clarify the motivation for using synthetic data for the first pre-training stage. Specifically, what unique advantages does synthetic pre-pre-training offer relative to self-supervised pre-training performed directly on natural unlabeled data from the target domain?
-
Please add baseline results for standard single-stage pre-training to the paper, so that the incremental performance gain from the additional synthetic pre-pre-training stage can be properly quantified.