- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Procedural Core, a reusable initialization for Transformers based on procedural pretraining. Building on prior work on procedural warm-up, the authors train a small recurrent ViT, then expand its weights across depth and width to initialize larger models. The method improves performance over random initialization on ImageNet-1K and CIFAR-100. Additionally, the authors demonstrate gains in self-supervised learning and language modeling.
Critical strengths
- The idea of replacing repeated procedural warm-up with a compact, reusable initialization is interesting and practically motivated.
- The method is simple and can be applied to different model sizes without modifying the downstream architecture.
- The paper provides extensive experiments across multiple tasks and domains, along with useful ablations and analysis.
Critical weaknesses
W1. Insufficient comparison with procedural warm-up. Although Procedural Core slightly outperforms direct procedural warm-up in Figure 2 (left), neither Section 4.1 nor the appendix clearly specifies the exact pretraining configurations and computational costs of the warm-up baseline. Figure 2 also omits its convergence curve, making it difficult to assess whether Procedural Core provides a meaningful advantage beyond the final accuracy.
W2. Limited evidence in self-supervised learning. Section 4.2 only compares against random initialization, without a procedural warm-up baseline. Moreover, the DINO improvement decreases from +0.6 to +0.3 pp, while linear probing improves by only +0.06 pp. It remains unclear whether these marginal gains are significant or whether the recurrent core provides any advantage over conventional procedural pretraining.
W3. Limited justification for the recurrent architecture. The method shares a single recurrent block across all intermediate layers. Figure 5 includes an ablation against a parameter-matched non-recurrent model; however, this comparison also changes the effective computational depth.
W4. Limited exploration of weight expansion strategies. The method directly replicates weights across depth and tiles matrices across width. While Figure 5 provides some ablations, the comparisons are limited.
W5. Incomplete computational efficiency analysis. Since the primary motivation is to avoid repeated procedural pretraining, the paper lacks a systematic comparison of total training compute against direct procedural warm-up.
Questions for authors
Q1 (W1): Could the authors provide the exact procedural pretraining configurations (e.g., training steps, pretraining time) for both methods, and include the procedural warm-up convergence curve in Figure 2?
Q2 (W2): Could the authors include procedural warm-up as a baseline for the 4.2 experiments to show whether the improvements are meaningful?
Q3 (W3): Have the authors considered using multiple recurrent middle blocks (e.g., A–B–B–C–C–D, or non-recurrent) instead of a single shared block? A comparison with such architectures, ideally controlling for computational depth and parameter count, would help justify the current design.
Q4 (W4): Could the authors compare the proposed width tiling with alternative expansion strategies, such as interpolation-based expansion or randomized expansion?
Q5 (W5): Could the authors report the GPU-hours or FLOPs for Procedural Core and procedural warm-up, including auxiliary pretraining costs? An amortized cost analysis across multiple target models would help substantiate the claimed efficiency advantage.
9 comments
W2/Q2: Self-supervised learning. We agree the DINO gains are small (+0.3 k-NN at 300 epochs).
Procedural warm-up baseline. As requested, we added procedural warm-up, and also Mimetic initialization, as baselines in the same DINO setup. k-NN accuracy at epoch 100:
| Initialization | k-NN at epoch 100 (%) |
|---|---|
| Default | 66.8 |
| Mimetic | 67.0 |
| Procedural warm-up | 66.0 |
| Procedural Core | 67.4 |
Mimetic initialization gives a smaller gain than the core (+0.2 vs. +0.6), and procedural warm-up is below default at this point in training (−0.8), so the core outperforms warm-up under DINO by 1.4 points at epoch 100. These runs are ongoing; we will post the full 300-epoch results in this thread within a few days and add them to the paper. To our knowledge, this is the first evaluation of procedural warm-up or structured initialization for self-supervised vision.
Why the gains are small under DINO. Our analysis attributes much of the core's benefit to suppressing high-norm tokens (Section 5.3). We repeated this analysis at 16 checkpoints of both DINO runs: neither model develops such tokens at any point in training, consistent with Darcet et al. (2024), so the core's main mechanism has nothing to act on.
We also use the standard DINO recipe, which was developed for random initialization, without retuning it for the core (Appendix C). The narrowing of the k-NN gap from +0.6 at epoch 100 to +0.3 at epoch 300 suggests that this recipe gradually overrides the structure introduced at initialization.
W3/Q3: Justification for a single recurrent block. Fig. 5a already contains the comparison suggested in the review at fixed computational depth. All auxiliary models have 12 layers, and we vary only how they are tied. U=4 is exactly the A–B–B–C–C–D pattern (A, B×5, C×5, D); U=2 uses two shared halves without separate input and output blocks (A×6, B×6); and U=12 is fully non-recurrent. Here is the CIFAR-100 accuracy after expanding each core into a ViT-T (Fig. 5a; separate runs from Table 3):
| Auxiliary model (12 layers) | Unique blocks | CIFAR-100 top-1 (%) |
|---|---|---|
| Default initialization (no core) | — | 72.1 |
| A×6, B×6 | 2 | 74.1 |
| A, B×10, C (ours) | 3 | 74.6 |
| A, B×5, C×5, D | 4 | 74.1 |
| Non-recurrent | 12 | 73.5 |
Transfer peaks with a single shared middle block between separate input and output blocks. Fig. 5b complements this along the other axis: at matched parameter count, and therefore lower depth, a non-recurrent core gives a +1.7% relative gain over default vs. +4.7% for ours.
Author Response to Reviewer JLxN
W1/Q1: Comparison with procedural warm-up. We added the full configuration of both procedural stages to Appendix A1 and the warm-up training curve to Fig. 2 (right). Both stages use the same procedural data, objective and per-run budget, and differ in the model trained:
Overall, the core needs about 4.5× less procedural compute than a ViT-B warm-up and is trained only once for all targets (see W5). The training curves show that its advantage is not limited to final accuracy (ViT-B, ImageNet-1K top-1, mean over 3 seeds):
Both procedural initializations start slower than default, but the core catches up by epoch 100 and then leads both baselines. It stays ahead of warm-up at every logged epoch. Procedural Core also exceeds warm-up on CIFAR-100 (74.6 ± 0.5 vs. 73.9 ± 0.1) and, by large margins, on frozen-backbone dense tasks: ImageNet-S mAP 42.9 vs. 36.6, VOC07 CorLoc 18.4 vs. 14.1, and NYUv2 RMSE 0.998 vs. 1.027, with ADE20K on par (Fig. 11).