acceptodds
Under review as a conference paper at ICLR 2027

Procedural Core: A Compact Recurrent Initialization for Vision Transformers

Abstract

Transformers are typically trained from random initialization, requiring all their capabilities to emerge from large-scale optimization. Recent work showed that a small amount of abstract procedurally generated data can help acquire generic inductive structure at low cost. However, this adds a pretraining stage that must be repeated for every target model. We propose Procedural Core, an initialization strategy that captures this generic structure into a compact set of weights that can be reused across models. We train a minimal recurrent transformer on procedural data, then expand its weights to initialize transformers of arbitrary width and depth. The resulting initialization improves performance on image classification, self-supervised visual learning (DINO), and modeling natural language (FineWeb-Edu) and code (CodeParrot). For image classification, expanding a 1M-parameter core to initialize an 85M-parameter ViT-Base improves ImageNet top-1 accuracy by 2.2pp over standard random initialization. Our analysis identifies recurrence as essential for learning compact weights that transfer across models. In ViTs, we localize a key benefit in the suppression of high-norm tokens that produces substantial improvements in zero-shot segmentation (ImageNet-S mAP 32.3 → 42.9), object localization (VOC07 CorLoc 9.9 → 18.4), and depth estimation (NYUv2 RMSE 1.104 → 0.998). This demonstrates that transformers need not start from a blank slate, and can be initialized with generic capabilities at low cost with no domain- or task-specific data.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
transformer initialization, transfer learning, inductive bias, procedural pretraining

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JLxNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Procedural Core, a reusable initialization for Transformers based on procedural pretraining. Building on prior work on procedural warm-up, the authors train a small recurrent ViT, then expand its weights across depth and width to initialize larger models. The method improves performance over random initialization on ImageNet-1K and CIFAR-100. Additionally, the authors demonstrate gains in self-supervised learning and language modeling.

Critical strengths
  1. The idea of replacing repeated procedural warm-up with a compact, reusable initialization is interesting and practically motivated.
  2. The method is simple and can be applied to different model sizes without modifying the downstream architecture.
  3. The paper provides extensive experiments across multiple tasks and domains, along with useful ablations and analysis.
Critical weaknesses

W1. Insufficient comparison with procedural warm-up. Although Procedural Core slightly outperforms direct procedural warm-up in Figure 2 (left), neither Section 4.1 nor the appendix clearly specifies the exact pretraining configurations and computational costs of the warm-up baseline. Figure 2 also omits its convergence curve, making it difficult to assess whether Procedural Core provides a meaningful advantage beyond the final accuracy.

W2. Limited evidence in self-supervised learning. Section 4.2 only compares against random initialization, without a procedural warm-up baseline. Moreover, the DINO improvement decreases from +0.6 to +0.3 pp, while linear probing improves by only +0.06 pp. It remains unclear whether these marginal gains are significant or whether the recurrent core provides any advantage over conventional procedural pretraining.

W3. Limited justification for the recurrent architecture. The method shares a single recurrent block across all intermediate layers. Figure 5 includes an ablation against a parameter-matched non-recurrent model; however, this comparison also changes the effective computational depth.

W4. Limited exploration of weight expansion strategies. The method directly replicates weights across depth and tiles matrices across width. While Figure 5 provides some ablations, the comparisons are limited.

W5. Incomplete computational efficiency analysis. Since the primary motivation is to avoid repeated procedural pretraining, the paper lacks a systematic comparison of total training compute against direct procedural warm-up.

Questions for authors

Q1 (W1): Could the authors provide the exact procedural pretraining configurations (e.g., training steps, pretraining time) for both methods, and include the procedural warm-up convergence curve in Figure 2?

Q2 (W2): Could the authors include procedural warm-up as a baseline for the 4.2 experiments to show whether the improvements are meaningful?

Q3 (W3): Have the authors considered using multiple recurrent middle blocks (e.g., A–B–B–C–C–D, or non-recurrent) instead of a single shared block? A comparison with such architectures, ideally controlling for computational depth and parameter count, would help justify the current design.

Q4 (W4): Could the authors compare the proposed width tiling with alternative expansion strategies, such as interpolation-based expansion or randomized expansion?

Q5 (W5): Could the authors report the GPU-hours or FLOPs for Procedural Core and procedural warm-up, including auxiliary pretraining costs? An amortized cost analysis across multiple target models would help substantiate the claimed efficiency advantage.

9 comments
AuthorsOpenReview ↗

Author Response to Reviewer JLxN

W1/Q1: Comparison with procedural warm-up. We added the full configuration of both procedural stages to Appendix A1 and the warm-up training curve to Fig. 2 (right). Both stages use the same procedural data, objective and per-run budget, and differ in the model trained:

Procedural warm-upProcedural Core
Model trainedViT-B (the target)ViT-T with tied middle blocks, 3 weight-sharing instances
Data and objectivek-Dyck (k = 64, length 196), masked closing tokens (ratio 0.5)Same
Steps × batch15,000 × 256 (~3.8M examples, 1% of the ImageNet budget)Same, per instance
OptimizerAdamW, lr 2 × 10⁻³, cosine scheduleSame
Compute per sample17.6 GFLOPs1.3 GFLOPs per instance
Repeated for each target?YesNo

Overall, the core needs about 4.5× less procedural compute than a ViT-B warm-up and is trained only once for all targets (see W5). The training curves show that its advantage is not limited to final accuracy (ViT-B, ImageNet-1K top-1, mean over 3 seeds):

EpochDefaultProcedural w/uProcedural Core
5069.6764.3767.46
10074.4073.4074.93
15075.5375.7677.35
20075.8577.8478.70
25076.5278.8479.45
30077.7379.3979.74

Both procedural initializations start slower than default, but the core catches up by epoch 100 and then leads both baselines. It stays ahead of warm-up at every logged epoch. Procedural Core also exceeds warm-up on CIFAR-100 (74.6 ± 0.5 vs. 73.9 ± 0.1) and, by large margins, on frozen-backbone dense tasks: ImageNet-S mAP 42.9 vs. 36.6, VOC07 CorLoc 18.4 vs. 14.1, and NYUv2 RMSE 0.998 vs. 1.027, with ADE20K on par (Fig. 11).

W2/Q2: Self-supervised learning. We agree the DINO gains are small (+0.3 k-NN at 300 epochs).

Procedural warm-up baseline. As requested, we added procedural warm-up, and also Mimetic initialization, as baselines in the same DINO setup. k-NN accuracy at epoch 100:

Initializationk-NN at epoch 100 (%)
Default66.8
Mimetic67.0
Procedural warm-up66.0
Procedural Core67.4

Mimetic initialization gives a smaller gain than the core (+0.2 vs. +0.6), and procedural warm-up is below default at this point in training (−0.8), so the core outperforms warm-up under DINO by 1.4 points at epoch 100. These runs are ongoing; we will post the full 300-epoch results in this thread within a few days and add them to the paper. To our knowledge, this is the first evaluation of procedural warm-up or structured initialization for self-supervised vision.

Why the gains are small under DINO. Our analysis attributes much of the core's benefit to suppressing high-norm tokens (Section 5.3). We repeated this analysis at 16 checkpoints of both DINO runs: neither model develops such tokens at any point in training, consistent with Darcet et al. (2024), so the core's main mechanism has nothing to act on.

We also use the standard DINO recipe, which was developed for random initialization, without retuning it for the core (Appendix C). The narrowing of the k-NN gap from +0.6 at epoch 100 to +0.3 at epoch 300 suggests that this recipe gradually overrides the structure introduced at initialization.

W3/Q3: Justification for a single recurrent block. Fig. 5a already contains the comparison suggested in the review at fixed computational depth. All auxiliary models have 12 layers, and we vary only how they are tied. U=4 is exactly the A–B–B–C–C–D pattern (A, B×5, C×5, D); U=2 uses two shared halves without separate input and output blocks (A×6, B×6); and U=12 is fully non-recurrent. Here is the CIFAR-100 accuracy after expanding each core into a ViT-T (Fig. 5a; separate runs from Table 3):

Auxiliary model (12 layers)Unique blocksCIFAR-100 top-1 (%)
Default initialization (no core)—72.1
A×6, B×6274.1
A, B×10, C (ours)374.6
A, B×5, C×5, D474.1
Non-recurrent1273.5

Transfer peaks with a single shared middle block between separate input and output blocks. Fig. 5b complements this along the other axis: at matched parameter count, and therefore lower depth, a non-recurrent core gives a +1.7% relative gain over default vs. +4.7% for ours.

Reviewer_XfMfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Procedural Core, a reusable initialization strategy for Transformers that aims to transfer generic computational structure learned from procedurally generated data into target models of different sizes. The method trains a compact auxiliary Transformer with three unique blocks, including a recurrently shared middle block, on abstract nested-symbol sequences. The learned weights are then expanded in depth and width to initialize larger target models, which are subsequently trained on their downstream tasks as usual.

Critical strengths

The paper addresses an important limitation of prior procedural warm-up methods by attempting to amortize the cost of learning generic structure into a compact set of reusable weights.

The proposed approach does not require task-specific data or a specialized target architecture during core training. Its weight expansion procedure offers a straightforward way to initialize models with different depths and widths.

The experiments cover supervised and self-supervised vision, dense visual prediction, natural language modeling, and code modeling. The improvements in frozen-backbone dense prediction are particularly interesting because they suggest that the initialization may affect more than classification-specific representations.

Critical weaknesses
  1. The novelty relative to prior procedural warm-up is not fully isolated. The paper explicitly builds on prior work that trains Transformers on procedurally generated data. Its main contribution is compressing and transferring the learned structure through recurrent parameter sharing and weight expansion. Although the parameter-matched ablation supports the role of recurrence, it remains unclear how much of the improvement comes from the recurrent architecture itself, the resulting weight statistics, or other differences in the initialization procedure.

  2. The benefits are sensitive to the downstream training recipe. The reported ImageNet classification improvement disappears under the stronger DeiT-III recipe. This raises questions about the robustness of the method and whether its gains primarily reflect improved optimization under particular training settings rather than a broadly useful initialization.

  3. The efficiency claim is not sufficiently quantified. A central motivation is to avoid repeating procedural warm-up for every target model. However, the paper does not provide a comprehensive comparison of total compute, wall-clock training time, or the break-even point after accounting for the cost of training the core. The practical savings therefore remain insufficiently established.

  4. The proposed mechanism is suggestive but not fully established. The analysis links transfer to the suppression of high-norm tokens and the attention value/output pathway. While the component-shuffling experiments provide useful evidence, they do not establish that this mechanism fully explains the gains or that it generalizes across model architectures and tasks.

  5. Some reported gains are modest or uneven across settings. For example, the reported linear evaluation of DINO features improves only slightly, and ImageNet classification does not improve under the stronger training recipe. A clearer characterization of when Procedural Core helps, when it does not, and how its benefits scale would strengthen the paper.

  6. The transfer to substantially different architectures and modalities remains insufficiently characterized. The experiments cover several domains, but it is unclear how robustly the method transfers to architectures with different block designs or to modern video and generative Transformers. The generality of the initialization should therefore be interpreted within the scope of the evaluated configurations.

Questions for authors
  1. Contribution beyond prior procedural warm-up. Could the authors further isolate the contribution of recurrent parameter sharing from the effects of procedural pretraining itself? In particular, how does Procedural Core compare with parameter-matched non-recurrent cores and alternative weight-expansion strategies under identical training budgets?

  2. Robustness to downstream training recipes. The ImageNet classification gain disappears under the DeiT-III recipe. Could the authors analyze which aspects of the training recipe determine whether Procedural Core helps, and whether the benefit persists under stronger data augmentation, regularization, and longer training?

  3. Total computational cost and practical savings. Could the authors report the core-training cost, target-model training time, and total compute relative to standard initialization and direct procedural warm-up? For how many target models must a core be reused before its training cost is amortized?

  4. Evidence for the proposed mechanism. How strongly does the suppression of high-norm tokens explain the downstream improvements? Could the authors provide interventions that directly manipulate token norms or the value/output pathway to distinguish causal effects from correlations?

  5. Transfer across architectures and scales. How well does the method transfer to target Transformers with substantially different block designs, normalization schemes, or attention implementations? Are the gains maintained as the target model scale increases?

  6. Implications for video and generative Transformers. Has the method been evaluated on video Transformers or diffusion Transformers, where temporal processing and iterative denoising introduce different computational requirements? If not, what challenges do the authors anticipate when transferring the proposed initialization to these settings?

9 comments
AuthorsOpenReview ↗

Author Response to Reviewer XfMf

We thank the reviewer for the detailed and constructive review. We address each point below.

W1/Q1: Isolating the contribution beyond procedural warm-up. We agree that our data and training objective come from prior work, as Section 3 states. Our contribution is to compress the structure that procedural warm-up induces into a compact recurrent core, and to expand it deterministically into targets of any width and depth. Shinnick et al. (2026) identify such a compact instantiation of procedurally warmed-up weights as important for future work. Procedural core, trained once on a ViT-T, initializes five target architectures, and matches or exceeds direct warm-up. Below, we separate the factors named in the review. All comparisons use the same procedural data, number of procedural examples, and downstream recipe.

(i) How much comes from procedural warm-up, and how much from the core. We decompose each gain into the part that procedural warm-up of the target provides on its own and the part the core adds on top:

MetricDefaultProcedural warm-upProcedural CoreShare of gain added by the core
ImageNet top-1, ViT-B (%)77.679.479.818%
CIFAR-100 top-1, ViT-T (%)72.173.974.628%
ImageNet-S mAP, frozen ViT-B (%)32.336.642.959%
VOC07 CorLoc, frozen ViT-B (%)9.914.118.451%
NYUv2 RMSE ↓, frozen ViT-B (m)1.1041.0270.99827%
ADE20K mIoU, frozen ViT-B (%)26.629.028.8−9%

For classification, procedural warm-up provides most of the gain, and the core matches it at a fraction of the cost by turning it from a separate training stage into an initialization (see W3). For frozen-backbone dense tasks, the core adds about half or more of the total gain on ImageNet-S and VOC07, and about a quarter on NYUv2. On ADE20K, warm-up is slightly ahead.

(ii) The recurrent architecture itself. For the ViT-T target, warm-up and core use the same number of training examples and differ in weight tying. The CIFAR-100 row above therefore measures recurrence at matched compute: +0.7 points over warm-up. Recurrence also changes what is learned. Recurrent weights have a broader singular spectrum than untied weights trained on the same data (Fig. 6), and transfer depends on it: truncating to the top singular directions hurts the core much more than the warm-up weights (Fig. 7). A non-recurrent core with the same number of unique parameters, trained on the same examples, gives a +1.7% relative gain over default vs. +4.7% for the recurrent core (Fig. 5b), and transfer peaks at three unique blocks (Fig. 5a).

(iii) Weight statistics. Shuffling all core weights before training preserves their marginal statistics but destroys their structure. On ImageNet with ViT-B, this gives 77.66%: only 0.20 points above default (77.46%) and 1.77 points below the core (79.43%) in the same set of single-seed runs (Fig. 10b). The shuffled weights therefore keep almost none of the 1.97-point gain, which comes from learned structure rather than weight statistics.

(iv) DINO. We have also compared Procedural Core with procedural warmup (and mimetic initialization) for self-supervised DINO models. Whilst the gains are modest, Procedural Core provides the largest benefits out of these initializations. These runs are ongoing; we will post the full 300-epoch results in this thread within a few days and add them to the paper.

Initializationk-NN at epoch 100 (%)
Default66.8
Mimetic67.0
Procedural warm-up66.0
Procedural Core67.4

In summary, the procedural warm-up supplies most of the classification gain, and recurrence makes this structure compact, turning it from a separate training stage into a simple initialization. Procedural core also outperforms procedural warm-up substantially on dense tasks and provides more benefits for DINO.

W2/Q2: Robustness to the training recipe. We report on this in Section 6 and Appendix C. Our new analysis identifies which aspect of the recipe determines whether the core improves classification.

Which aspect of the recipe matters. Our analysis points to one factor, which is whether the default model develops the high-norm tokens that the core suppresses (Section 5.3). We repeated this analysis under DeiT-III and DINO. Under DeiT-III, the default model barely develops these tokens (2.2% of the attention mass vs. 50.4% under our main recipe). This leaves little for the core to fix:

RecipeAttention mass on high-norm tokens (%), default → coreMax/median token norm, default → coreClassification gain
Main recipe (Xu et al., 2023), ViT-B50.4 → 17.79.75 → 7.39+2.2 top-1
DeiT-III, ViT-B2.2 → 0.03.96 → 2.21−0.1 top-1
DINO, ViT-S0.0 → 0.01.59 → 1.51+0.3 k-NN

Across these three recipes, the classification gain is large only where the default model develops many high-norm tokens.

Optimization or a useful initialization? Two observations show that the core does more than ease optimization. First, under our main recipe, its advantage grows during training rather than fading like a head start: the gap over default rises from +0.5 at epoch 100 to about +2.9 at epochs 200–250, and remains about +2 at the end of training (Fig. 2, right). Second, under DeiT-III, top-1 is unchanged (82.4 vs. 82.5), but the frozen-backbone gains persist on ImageNet-S (+2.07 mAP) and VOC07 (+7.70 CorLoc), so the final representation remains measurably different. The benefit also extends beyond one recipe: to fine-tuning on CIFAR-100 (89.7 vs. 88.4; Table 4), to language and code modeling (about 4% lower perplexity; Fig. 4), and to DINO.

Augmentation, regularization and training length. Our main recipe already uses RandAugment, Mixup, CutMix and label smoothing over 300 epochs. DeiT-III changes optimization, augmentation and regularization at once; isolating each change would need a separate ImageNet run, which is beyond our rebuttal budget. DeiT-III is also heavily optimized for randomly initialized ViTs, and we did not retune it for Procedural Core. As discussed in Appendix C, the best protocol for a structured initialization need not match one developed for random initialization, so DeiT-III should not be read as an upper bound on the method's benefit.

W3/Q3: Total computational cost and practical savings. We agree and thus added a cost analysis in Appendix A6.

Core-training cost and total compute. Direct procedural warm-up trains each target separately on procedural data. We instead train one core and expand it into each target. Let R denote one procedural-training run of a ViT-T: 15,000 steps with batch size 256 on k-Dyck (3.84M examples), or R ≈ 1.5 × 10¹⁶ FLOPs (3.84M examples × 1.3 GFLOPs per forward pass, × 3 to include the backward pass). The core is trained as three weight-sharing ViT-T instances, each with this budget (Appendix A1), so it costs 3R, about 0.2% of the FLOPs of a 300-epoch ViT-B training run on ImageNet-1K. With the data budget held fixed, warm-up costs scale with each target's FLOPs per example:

Method / targetProcedural-training compute
Default initialization0
Direct warm-up: ViT-T, 6 / 12 / 24 layers0.5 R / 1 R / 2 R
Direct warm-up: ViT-S3.5 R
Direct warm-up: ViT-B13.5 R
Direct warm-up: all five targets≈ 20.5 R
Procedural Core: shared across all five targets3 R

Relative to standard initialization, the core adds about 0.2% of one ViT-B training run once. Relative to direct warm-up, it reduces procedural-training compute by about 4.5× for a single ViT-B target and about 7× across all five targets.

Break-even. The core's cost is recovered with the first ViT-S or ViT-B target. Each further target adds no procedural cost, and expanding the core takes under a second.

Target-model training time. This is unchanged: the target is trained with the same recipe and number of epochs as with default initialization, and only its starting weights differ. The core also reaches the default model's final ImageNet accuracy earlier, between epochs 150 and 200 of the same 300-epoch schedule (Fig. 2, right), which suggests that a shorter schedule could reach equal accuracy.

Reviewer_sQEcOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Procedural Core, a compact initialization learned by training a small recurrent Transformer on procedural data. Its weights are expanded to initialize different target models, avoiding repeated procedural warm-up. The paper evaluates several training settings and analyzes the learned weights and token behavior.

Critical strengths
  1. Interesting motivation. Learning a compact, reusable initialization from abstract data is an interesting idea that addresses a practical limitation of procedural warm-up.

  2. Extensive experiments and analysis. The paper covers several learning settings and downstream tasks. Architectural ablations, spectral analysis, and component-shuffling experiments provide a detailed examination of the method.

Critical weaknesses
  1. Limited scaling evidence. Most architectural ablations are conducted on CIFAR-100 with small ViTs. Although larger-model results are included, the paper does not systematically evaluate how one core transfers across larger model sizes.

  2. Unclear computational savings. The paper does not directly quantify the cost advantage over procedural warm-up. Parameter sharing reduces storage, but repeated block applications still require computation. Core-training costs and savings from reuse across multiple targets should be reported.

Questions for authors
  1. Cost comparison. Could the authors report FLOPs or GPU-hours for core training and direct procedural warm-up, including the total cost across multiple targets? How do recurrent and non-recurrent auxiliary models compare under matched computational budgets?

  2. Small gains under DINO. Why are the improvements much smaller under DINO than under supervised training? A comparison using the same backbone size and common feature evaluations would help separate these factors. Do other initialization methods show similarly small gains under DINO?

  3. Mechanisms under DINO. Do the spectral properties, value/output pathway dependence, and suppression of high-norm tokens also hold under DINO? Tracking these properties during training could help explain why the performance advantage is small and gradually decreases.

7 comments
AuthorsOpenReview ↗

Author Response to Reviewer sQEc

We thank the reviewer for the positive assessment and the precise questions. We address each point below.

W1: Scaling across target sizes. We agree the evidence was spread across several figures, and we now gather it in one table which we will add to the final paper. The same ~1M-parameter core initializes every vision model in the paper, from ~3M to 85M parameters:

TargetParamsDatasetDefaultProcedural CoreGain
ViT-T, 6 layers~3MCIFAR-10069.371.3+2.0
ViT-T, 12 layers~6MCIFAR-10072.174.6+2.5
ViT-T, 24 layers~11MCIFAR-10072.475.2+2.8
ViT-S~22MCIFAR-10072.075.3+3.3
ViT-S, DINO (k-NN)~22MImageNet-1K72.072.3+0.3
ViT-B~85MImageNet-1K77.679.8+2.2

On CIFAR-100, the gain grows with depth (+2.0 at 6 layers to +2.8 at 24) and with width (+3.3 for ViT-S vs. +2.5 for ViT-T). On ImageNet-1K, the core still gives +2.2 when expanded ~85× into ViT-B. The gain does not shrink as the target grows.

The ablations use CIFAR-100, but our main analyses do not. The token-norm analysis, component shuffling and all dense evaluations (Figs. 8–11) use the ImageNet-trained ViT-B models. ViT-B is also the largest ViT in the procedural warm-up work we build on (Shinnick et al., 2026) and in related initialization and synthetic-pretraining studies (Zheng et al., 2025; Nakamura et al., 2024). Trockman & Kolter (2023) and Xu et al. (2023) evaluate up to ViT-S and ViT-T respectively. A ViT-L run on ImageNet-1K is beyond our academic compute budget; we note this in the limitations.

W2/Q1: Computational cost. We agree that the computational savings should be quantified, and we have added a cost analysis in the appendix. As the reviewer notes, parameter sharing does not reduce the computation required for repeated block applications. Our compute savings come from training a small auxiliary model and reusing it across targets.

Direct procedural warm-up trains each target separately on procedural data. We instead train one core and expand it into each target. Let R denote one procedural-training run of a ViT-T: 15,000 steps with batch size 256 on k-Dyck (3.84M examples), or R ≈ 1.5 × 10¹⁶ FLOPs (3.84M examples × 1.3 GFLOPs per forward pass, × 3 to include the backward pass). The core is trained as three weight-sharing ViT-T instances, each with this budget (Appendix A1), so it costs 3R. With the per-run data budget held fixed, the costs of direct warm-up scale with each target's FLOPs per example:

Method / targetEstimated procedural-training compute
Direct warm-up: ViT-T, 6 / 12 / 24 layers0.5 R / 1 R / 2 R
Direct warm-up: ViT-S3.5 R
Direct warm-up: ViT-B13.5 R
Direct warm-up: all five targets≈ 20.5 R
Procedural Core: shared across all five targets3 R

For scale, the core costs approximately 0.2% of the FLOPs of a 300-epoch ViT-B training run on ImageNet-1K. Procedural Core therefore reduces procedural-training compute by a factor of approximately 4.5 for a single ViT-B target and approximately 7 across all five targets. Each additional target requires no further procedural training; expanding the core takes under a second.

Recurrent vs. non-recurrent auxiliary models. For the ViT-T target, direct procedural warm-up provides the non-recurrent comparison: an untied ViT-T trained with the same width, total number of block applications, sequence length, batch size and training steps as the recurrent core. The recurrent core reaches 74.6% CIFAR-100 accuracy, compared with 73.9% for direct warm-up, a 0.7-point improvement (Table 3). At matched parameter count, the recurrent core gives a 4.7% relative gain over the default baseline, compared with 1.7% for a non-recurrent core (Fig. 5b).

Q2: Smaller gains under DINO. We agree the DINO gains are small, and we have now investigated why.

Firstly, the reviewer is right that the two experiments differ in more than the training objective: the supervised experiment uses ViT-B evaluated by top-1 accuracy, and the DINO experiment uses ViT-S evaluated by k-NN and linear probing. Each follows the standard protocol for its setting: ViT-B with top-1 accuracy for supervised ImageNet training, using the recipe of Xu et al. (2023), and ViT-S/16 with k-NN and linear evaluation for DINO (Caron et al., 2021). The absolute gains are therefore not directly comparable, and we clarified this in Section 4.2. Within each setting, however, the comparison to the default initialization is controlled, and the DINO gain is small on its own terms.

We see two reasons for the smaller gain under DINO:

  1. Nothing for the main mechanism to act on. Our analysis attributes much of the core's benefit to suppressing high-norm tokens (Section 5.3). Neither DINO model develops such tokens: no token in any image exceeds 2.2× its image's median norm (see Q3). This is consistent with Darcet et al. (2024), who report that the original DINO is free of these artifacts.
  2. The DINO recipe was not retuned for our initialization. We use the standard DINO configuration, which was developed for random initialization, without adjusting it for models initialized with Procedural Core. As we discuss for DeiT-III in Appendix C, the best training protocol for a structured initialization need not match one developed for random initialization, and other optimization, augmentation or regularization choices may better preserve the structure introduced at initialization. The narrowing of the gap from +0.6 at 100 epochs to +0.3 at 300 suggests that the standard recipe gradually overrides that structure.

Do other initializations show similarly small gains under DINO? To our knowledge, no structured initialization has been evaluated under DINO before, so we ran two baselines in the same DINO setup: Mimetic initialization (Trockman & Kolter, 2023) and direct procedural warm-up (Shinnick et al., 2026). k-NN accuracy at epoch 100:

Initializationk-NN at epoch 100 (%)
Default66.8
Mimetic67.0
Procedural warm-up66.0
Procedural Core67.4

Mimetic initialization gives a smaller gain than the core (+0.2 vs. +0.6), and procedural warm-up is below default at this point in training (−0.8). Small gains under DINO are therefore common to structured initializations rather than specific to our method, and the core gives the largest gain of the three. These runs are ongoing; we will post the full 300-epoch results in this thread within a few days and add them to the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Upload

We thank the reviewers for their valuable feedback, which has helped us improve the manuscript. We uploaded a revision with new results and clarifications (in blue in the PDF):

  • (Pages 4, 16, 17) New computational cost analysis comparing Procedural Core with direct procedural warm-up, including the cost amortized across target models and the break-even point (Appendix A.6, Table 3).
  • (Pages 5, 15) Full configuration of the procedural warm-up baseline (Appendix A.1, Table 1) and its training curve (Figure 2), showing that Procedural Core stays ahead of the warm-up throughout training.
  • (Pages 5, 6) New DINO baselines with Mimetic initialization and procedural warm-up, complete to 300 epochs: Procedural Core gives the largest gain at every evaluated epoch (Figure 3).
  • (Pages 5, 22) New analysis of the smaller gains under DINO and DeiT-III: tracking high-norm tokens across 16 DINO checkpoints and under DeiT-III shows that these recipes produce few or no high-norm tokens for the core to suppress (Appendix B.10, Tables 9 and 10).
  • (Pages 7, 21, 22) New comparison of width expansion strategies (random padding, block-diagonal, interpolation, and a shuffled control), showing that the gain comes from the core's learned structure rather than the expansion scheme (Appendix B.9, Table 8).
  • (Page 21) Consolidated results across target sizes, from ViT-T to ViT-B (Appendix B.8, Table 7).
  • (Pages 6, 15) Tying patterns used in the recurrence ablation of Figure 5a, including the A–B–B–C–C–D configuration (Appendix A.1).
  • (Pages 1, 9) Clarified the role of high-norm token suppression as a key contributor to the gains, and refined the comparison with direct warm-up on dense tasks (Section 5.4).

We remain available in case of any further request.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.