acceptodds
Under review as a conference paper at ICLR 2027

WINR: Neural Implicit Representations for Weight Space

Abstract

Training large neural networks from scratch for every architectural variant remains a major computational bottleneck. Model growth methods reduce this cost by warm-starting larger target networks from smaller pretrained checkpoints, but they typically rely on hand-crafted rules such as weight tiling or layer duplication. These discrete heuristics restrict expansion to rigid scaling patterns and initialize large portions of the target network as direct copies of the source. In this paper we ask whether model growth can instead be formulated as a continuous field, and propose WINR, a method that replaces discrete tensor expansion with a learned continuous weight field. Rather than treating pretrained parameters as arrays to be copied, WINR views them as samples of an underlying function that maps structured geometric and semantic coordinates to weights. A single field, fitted to one or more pretrained networks, can be queried to synthesize weights for target architectures with new shapes, depths, or layer compositions. This continuous formulation naturally supports fractional expansion and cross-architecture synthesis beyond standard copy-based growth rules. Experiments on transformer model zoos show that WINR improves convergence over random initialization and achieves competitive performance against existing model growth methods.

Primary area:
generative models
Keywords:
WINR, implicite neural network, model expansion, weight generation, neural field

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pR2AOpenReview ↗
Rating
3(−1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose using implicit neural representations of weights (WINR) for creating warm-start initializations for training large models. The main conceptual idea is to view weights as a continuous field on an abstract space that encodes relative position of the weights within the architecture as well as further metadata.

The evaluation focuses on generating warm-start weights for language models MobileLLM-R1, Gemma, and BERT with up to ca. 400 M parameters. WINR shows promising performance for encoding weights at fixed architectural parameters and creating initializations for expanding width within the same model family (MobileLLM-R1) but struggles to generalize across model families (MobileLLM-R1 to Gemma). The authors show empirically that diverse source weights aide performance and conduct several ablations. Overall, the submission’s contributions are novel as well as interesting both on a conceptual and technical level with relevance to several communities, including researchers working on weight space learning, initialization, and optimization. I will list critical strengths, critical weaknesses, and questions in the dedicated fields below.

As a minor comment, I want to make the authors aware of a broken citation to LEMON on line 302 of their submission.

Critical strengths

Applying INRs to weights is conceptually interesting and, to the best of my knowledge, novel. The approach could thus open a new line of research within weight space learning.

The evaluation is well designed to support the main claims of the papers. The ablation studies are illuminating, and the comparison against tiling plus noise in Section 4.4 establishes an honest baseline that any method like the author’s needs to beat. All these experiments are thoroughly documented. These points underscore the general scientific rigor of the submission.

Critical weaknesses

WINR shows weak performance when generalizing to different model families. While this is a clear weakness of the method, I would not consider it a reason for rejection, as the task is challenging and existing methods do (to my knowledge) not consider this task at all.

Questions for authors
  1. In section 4.1, a maximum of 1B tokens was used. How was this endpoint chosen? Justifying this choice carefully is important as the ranking among methods appears to depend on the number of tokens in Fig. 3.

2, In the ablation study the method that performs best is inr_blend. This method uses a rather weak learned component (β∈[0.01,0.1]\beta \in [0.01, 0.1]). I would be curious whether the authors have some intuition for the reasons behind this.

3 comments

We thank the reviewer for the positive and thoughtful assessment. We have fixed the broken LEMON citation on line 302. Below, we address the weakness and questions directly with new experiments.

Common setting (MobileLLM). MobileLLM-R1-140M (hidden 576, MLP 2048, 15 layers) is grown to MobileLLM-R1-360M (hidden 1024, MLP 4096) and pre-trained on SlimPajama (SP): AdamW, β=(0.9,0.95)\beta=(0.9,0.95), weight decay 0.1, 2% warm-up followed by cosine decay over the full budget, sequence length 1,024, 32,768 tokens/step, bf16. We report validation PPL on 2M held-out tokens.


W1. Generalization across model families

"WINR shows weak performance when generalizing to different model families."

We agree that this is the main limitation of the submission.

Likely explanation. A coordinate-to-weight field can transfer effectively only when source and target coordinates have compatible meaning. Across model families, this correspondence is weak:

  • Arbitrary neuron order. Pretrained weights have little structure over neuron index (adjacent-entry correlation 0.005 vs. 0.001 for a random shuffle), and function-preserving permutations can change the coordinate assignment without changing the model function.
  • No shared basis. Gemma-3-270M shares neither its residual basis nor its 262k-token vocabulary with the donors (MobileLLM-R1-140M and SmolLM2-360M), so the same coordinate need not represent the same feature across models.
  • Partial generation. Only the 18 gate_proj tensors were decoded from the field; the remaining tensors, including the embedding matrix (about 62% of the parameters), were randomly initialized.

We therefore investigated whether explicitly aligning the source and target representation spaces improves transfer.

Cross-family alignment.

  • Aligned embeddings: each Gemma token is converted to text, re-tokenized by each donor, and represented by the average donor embedding of its pieces, projected to Gemma's width by PCA.
  • Aligned MLPs: least-squares maps between donor hidden spaces and the aligned space rotate the donor MLP tensors into Gemma's basis.
  • Fields: fields are fitted to the rotated tensors (reconstruction cosine 0.994) and decoded into Gemma using the donor layer at the nearest relative depth.

Gemma-3-270M was trained for 200M FineWeb tokens with the same recipe for every initialization (AdamW, lr 5e-4, cosine schedule, 65,536 tokens/step, bf16), using 3 seeds.

Gemma-3-270M initializationWikiText-2 test PPL ↓FineWeb val loss ↓
Random73.1 ± 0.93.994 ± 0.004
Aligned embeddings, random body60.6 ± 0.93.836 ± 0.003
Aligned embeddings + field-decoded aligned MLPs54.8 ± 0.83.756 ± 0.009
Aligned embeddings + copied aligned MLPs (no field)55.6 ± 2.53.756 ± 0.009

Alignment substantially improves cross-family transfer. Field-decoded aligned MLPs reduce perplexity by about 25% relative to random initialization, improve over aligned embeddings alone, and perform comparably to directly copying the aligned MLP weights. Without alignment, the same donor fields provide no gain.

These results indicate that source-target correspondence is an important factor in cross-family transfer. We will add this experiment to Sec. 4.2 and state its current limitations: the embeddings are explicitly aligned rather than generated by the field, and attention weights remain randomly initialized.


Q1. How was the 1B-token endpoint chosen?

"…the ranking among methods appears to depend on the number of tokens in Fig. 3."

The 1B-token endpoint was a practical compute budget rather than a convergence criterion. It allowed us to evaluate all initializations and ablations under the same budget. The Gemma experiment uses approximately the same budget: 15K steps ×\times 65,536 tokens ≈\approx 0.98B tokens.

We agree that a single endpoint can obscure changes in ranking, so we evaluated the Figure 3 setting throughout the full 1B-token run.

Setting: SP, 1B tokens (30,518 steps), seed 42. Tiling and inr_blend use lr 3e-4. Exact-coordinate decoding queries the same per-tensor fields at the exact source coordinates for existing neurons and at intermediate coordinates for new neurons; its lr (1e-5) was selected by a sweep for this method only.

Initializationlr5M25M100M200M300M500M750M1B
Tiling3e-470.347.041.839.938.335.131.529.8
inr_blend (β=0.05\beta=0.05)3e-497.951.643.139.837.533.229.127.4
Field, exact source coordinates1e-531.729.327.827.427.227.026.726.6

The ranking of inr_blend and tiling is budget-dependent: inr_blend is worse through 100M tokens, becomes comparable around 200M, and finishes 2.4 PPL ahead at 1B. Exact-coordinate decoding remains ahead at every evaluated checkpoint from 5M to 1B tokens.

We will therefore replace the single-endpoint comparison in Fig. 3 with full training curves and state explicitly that 1B tokens is a fixed compute budget rather than a convergence point.


Q2. Why does inr_blend, with a small learned component, perform best?

Short answer. At small β\beta, the gain of inr_blend over tiling comes from the corrections and from introducing a small perturbation, rather than from the specific values predicted by the field.

(a) Matched-noise control. We keep tiling and the corrections (fan-in rescaling and spectral restoration) fixed and replace the field with Gaussian noise matching its per-tensor mean and standard deviation.

InitializationSP val PPL @1B ↓
Tiling only29.8
Tiling + corrections, no perturbation28.0
Tiling + matched noise + corrections27.4
inr_blend (tiling + field + corrections)27.4

Of the 2.4-PPL gain over tiling, the corrections account for 1.8 PPL, while the remaining 0.6 PPL is reproduced by matched noise. Thus, for inr_blend at β=0.05\beta=0.05, the precise field-generated values do not provide an advantage over a matched perturbation.

(b) Training dynamics. The training curves are consistent with a small perturbation helping optimization move away from the highly duplicated tiled initialization. inr_blend initially performs worse than tiling (97.9 vs. 70.3 at 5M tokens), becomes comparable around 200M tokens, and then improves faster:

Initialization5M200M500M1BΔ\Delta 200M→1B
Tiling70.339.935.129.8-10.1
inr_blend (β=0.05\beta=0.05)97.939.833.227.4-12.4

We view this as a plausible interpretation of the optimization behavior rather than a mechanism established by the current experiments.

(c) Effect of the mixing coefficient. We also compare β=0.5\beta=0.5 and β=0.05\beta=0.05 under FineWeb pre-training for 45M tokens, evaluated on WikiText-2 test PPL over 3 seeds.

Mixing modeβ=0.5\beta=0.5β=0.05\beta=0.05
inr_tile_merge300.3 ± 13.2223.3 ± 2.4
inr_merge299.7 ± 5.9228.1 ± 3.2
inr_blend246.7 ± 6.0238.5 ± 7.6

Reducing β\beta substantially improves the merge variants, indicating that preserving most of the tiled initialization is important.

(d) Where the field contributes beyond a perturbation. When queried at the exact source coordinates, the same fields generate every linear weight without tiling and outperform tiling at every evaluated checkpoint in the 1B-token run. A control that fills the new neurons randomly instead of using field-generated values gives essentially the same result (27.62 vs. 27.72 PPL at 300M tokens, lr 1e-4). Thus, in this setting, the field's contribution comes primarily from accurately reconstructing the existing pretrained neurons (cosine 0.997), rather than from the values assigned to new neurons.

We will revise Sec. 4.4 accordingly

I thank the authors for answering my questions and conducting additional experiments to elucidate the role of the learned component in inr_blend.

I found the additional cross-architecture experiments helpful for understanding why this task might be hard. Additionally, the exact source coordinate version of WINR introduced during the discussion shows improved performance. I encourage the authors to make it a more prominent part of their work, while carefully examining the origin of the improved performance.

However, the discussion also surfaced an additional weakness of WINR in the inr_blend version. The method performs on par with adding noise of comparable magnitude to tiled weights. It is very surprising that this effect was not observed in the ablation presented in Section 4.4 “Tile-plus-noise control” of the initial submission, which appeared to be designed to test for it. Moreover, weighing the learned component in inr_blend more strongly decreases performance substantially, further drawing into question that useful information is learned in the inr_blend version.

In light of the discussion, I have to lower my score: Overall, the presented method is interesting and the paper insightful, but the discussion has raised serious questions about the performance of WINR compared to rather weak baselines.

Reviewer_2iccOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper fits an implicit neural representation with a Fourier-feature encoded inputs over the coordinates of a pretrained network’s weights (layer, row, column, plus layer type and architecture), then queries this field on a larger grid to initialize a larger network. The authors argue that weights vary smoothly enough over these coordinates that the field gives a better initialization than copy-based growth such as HyperCloning, and that, because it is a field, it can be sampled at any size. A graph-conditioned variant fits one field across several source architectures and decodes it into new shapes or model families. They reconstruct MobileLLM-R1-140M, growing MobileLLM-R1 from 140M to 360M, initializing Gemma-3-270M from fields fit to other families, and growing BERT from 1–5 sources compared with LEMON.

Critical strengths

Their formulation is tempting as representing the values in a network's weights as a continuous field would allow for outputting models of any size. In the limit the field itself might be able to learn a generalized function over all possible neural networks to avoid training networks from scratch whenever the architecture changes. This is the main reason I think the idea is worth pursuing.

The multi-source BERT field is competitive with LEMON, and adding more source networks clearly helps over a single source.

Critical weaknesses

The authors claim that the field is what led to their performance, but they did not ablate the field itself for MobileLLM while keeping the tiling and corrections. It would have been nice to see ablations on the performance with tiling only doesnt reach the same performance. Without this its unclear if the neural field itself lead to the performance or if it could have just been attributable to the tiling. The pure field mode in table 4 is the worst setting. Figure 6 shows the tile plus noise control but only for bert, not at the paper settings for mobileLLM.

Also the premise of the paper rests on that network weights vary smoothly over their row and column positions, which is the entire argument for using a neural field. This was never shown, and permutation symmetry suggests it may not hold.

Questions for authors

You show 1 billion tokens for MobileLLM-360M and 15k steps for Gemma. How many tokens is 15K steps? The batch size and sequence length aren’t given. Its currently unclear why those 2 settings were chosen for those models.

Why does Figure 3 use inr_rescale when inr_blend is best in Table 4? How does inr_blend compare with tiling in the figure 3 setting?

What is the compute cost for fitting a MobileLLM?

5 comments

We thank the reviewer for the detailed comments. We ran the requested controls; the results clarify where the gains come from, and we revise our claims accordingly.In all experiment we refit the field from scratch.

Common setting (MobileLLM)

MobileLLM-R1-140M (hidden 576, MLP 2048, 15 layers) is grown to the MobileLLM-R1-360M shape (hidden 1024, MLP 4096) and then pre-trained on large corpora without data repetition.

  • SlimPajama (SP): AdamW, β=(0.9,0.95)\beta=(0.9,0.95), weight decay 0.1, 2% warm-up followed by cosine decay; sequence length 1,024; 32,768 tokens/step; bf16; validation PPL on 2M held-out tokens; seed 42.
  • FineWeb (Table 4 setting): FineWeb sample-10BT, 45M training tokens, sequence length 512, 16,384 tokens/step, cosine schedule; WikiText-2 test PPL; 3 seeds (42, 1234, 7777).

We will replace the WikiText-2-trained results in the paper with these pre-training results: 45M tokens on WikiText-2 correspond to approximately 19 epochs and mainly measure overfitting. we also refitted the fields from scratch and performed lr search.


Weaknesses

W1. Ablating the field while keeping tiling and the corrections

"...they did not ablate the field itself for MobileLLM while keeping the tiling and corrections ... Figure 6 shows the tile plus noise control but only for BERT, not at the paper settings for MobileLLM."

We ran this control at the MobileLLM settings. inr_blend constructs each weight as

corr((1−β) tile(Wsrc)+βF)\mathrm{corr}((1-\beta)\,\mathrm{tile}(W_{\mathrm{src}})+\beta F),

where FF is the field and corr denotes fan-in rescaling and spectral restoration. We keep tiling and the corrections fixed and replace only FF with either matched Gaussian noise (same per-tensor mean and standard deviation as FF) or nothing.

(a) SlimPajama, 300M tokens, β=0.05\beta=0.05, lr=3e-4

InitializationSP val PPL @300M ↓
Tiling only31.3
Tiling + corrections, no field31.2
Tiling + matched noise + corrections31.0
inr_blend (tiling + field + corrections)31.0
inr_merge (tiling + field, no corrections)31.1
inr_rescale (field + corrections, no tiling)48.7
Random initialization49.4

At 300M tokens, all tiling-based variants are within 0.3 PPL.

(b) Figure 3 setting: SlimPajama, 1B tokens, lr=3e-4

Initialization100M300M500M1B
Tiling only41.838.335.129.8
Tiling + corrections, no field43.137.633.628.0
Tiling + matched noise + corrections43.237.633.427.4
inr_blend (tiling + field + corrections)43.137.533.227.4

The reviewer's concern is justified for the blend mode. At 1B tokens, inr_blend improves over tiling by 2.4 PPL, but matched noise reaches the same 27.4 PPL. Tiling + corrections already improves by 1.8 PPL, while the remaining 0.6 PPL is reproduced by the matched perturbation.

Thus, the blend result does not show an advantage specific to the learned field. We will report this ablation in Sec. 4.1 and revise the attribution accordingly.

(c) Learning-rate sensitivity

We swept the peak learning rate on SlimPajama for 50M tokens:

SP val PPL @50M1e-43e-41e-33e-3
Tiling33.233.942.9107.8
inr_blend33.735.050.1103.2
inr_rescale223.5151.089.399.4
inr245.2161.1100.9100.3
Random initialization309.4174.4122.4149.7
in most cases the proposed method outperform random init

Tiling and inr_blend prefer lower learning rates, while random and pure-field initializations prefer higher rates. At each method's best rate, the ranking is unchanged: inr_blend remains at the level of tiling, while the pure-field modes outperform random initialization (89.3 and 100.3 vs. 122.4).

(d) Where the field itself matters

The submitted method decodes the larger target on a uniform grid, so the target queries do not preserve the exact coordinates of existing source neurons and the source function is lost before training (>10^7 PPL at step 0).

We instead query the same fields at the exact source coordinates for existing neurons and at intermediate coordinates for new neurons, with a function-preserving correction. No tiling is used. This preserves the source function much more closely (34.7 PPL at step 0).

SlimPajama, 300M tokens, seed 42

InitializationlrSP val PPL @300M ↓
Field, uniform grid (inr_rescale)3e-448.7
Field, exact source coordinates1e-526.97
Tiling only1e-429.87

SlimPajama, 1B tokens, seed 42

InitializationlrSP val PPL @1B ↓
Field, exact source coordinates1e-526.6
inr_blend3e-427.4
Tiling only3e-429.8

FineWeb pre-training, WikiText-2 test, 45M tokens, 3 seeds

InitializationlrWT2 test PPL @45M ↓
Field, uniform grid (inr_rescale)3e-4434.0 ± 1.6
Field, exact source coordinates1e-538.7 ± 0.1
Tiling only1e-458.6 ± 0.3

The uniform-grid and exact-coordinate variants use the same fields at the same capacity; only the query coordinates differ.

Learning-rate sensitivity of exact-coordinate decoding

lr1e-53e-51e-43e-4
Field, exact source coordinates26.9727.0427.7228.65

Even at its worst tested rate (28.65), exact-coordinate decoding outperforms tiling (29.87 at 1e-4; 31.27 at 3e-4).

Filling the new neurons randomly instead of using field-generated values gives 27.62 PPL versus 27.72 at lr=1e-4. Thus, the field's contribution in this setting comes primarily from accurately decoding the existing pretrained neurons (reconstruction cosine 0.997), rather than from the values assigned to new neurons.


W2. Weight smoothness and permutation symmetry

"...the premise of the paper rests on that network weights vary smoothly over their row and column positions ... This was never shown, and permutation symmetry suggests it may not hold."

We agree that this distinction was not stated clearly. WINR treats weights as samples of a continuous function over normalized coordinates; it does not assume that neighboring entries of a pretrained weight matrix have similar values. The high-frequency Fourier band in Sec. 3.2 is specifically used to represent fine-grained weight variation. We will clarify this point in Sec. 3.

We evaluate all 105 linear weight matrices of MobileLLM-R1-140M using adjacent correlation, the TV ratio between adjacent and random pairs (1 = no spatial structure), and low-frequency 2D Fourier energy.

Function-preserving permutations consistently reorder residual channels, MLP units, KV groups, and query heads across all affected tensors. The resulting models remain functionally equivalent, with maximum logit differences of 5×10−55\times10^{-5} on 4×256 random tokens in fp32.

WeightsAdjacent corr. (rows / cols)TV ratioLow-freq. energy
Pretrained checkpoint0.005 / 0.0020.9831.00%
Random function-preserving permutation0.005 / 0.0020.9840.99%
Smoothing function-preserving permutation0.035 / 0.0290.9641.20%
Random shuffle of all entries0.001 / 0.0011.0000.98%
Field decoded at source size (576)0.016 / 0.0100.9791.01%
Field decoded at target size (1024)0.355 / 0.3970.7582.99%

Pretrained weights show little structure over neuron index, consistent with the reviewer's concern. The field reproduces this fine-grained structure at the source size and becomes smoother only when evaluated at new coordinates.

The field fit itself is insensitive to ordering. Identical fields fitted to the original tensor, a random function-preserving permutation, and a full entry shuffle achieve:

  • layer-0 q: cosine 0.9983 / 0.9981 / 0.9983
  • layer-0 o: cosine 0.9984 / 0.9984 / 0.9982

Thus, reconstruction does not depend on smoothness over neuron index.

Permutation symmetry. We additionally fitted fields independently to three function-equivalent versions of MobileLLM-R1-140M and decoded them at the exact source coordinates into MobileLLM-R1-360M.

Source neuron orderSP val PPL at step 0 ↓SP val PPL @300M ↓
Original35.9226.97
Random function-preserving permutation35.1926.98
Smoothing function-preserving permutation34.7426.97

All three orderings reach the same final result within 0.01 PPL, and their training curves coincide from 5M tokens onward. For reference, tiling starts at 2×10^8 PPL and reaches 29.87 under the same benchmark (lr=1e-4).

Thus, permutation itself does not affect downstream performance when source-coordinate correspondence is preserved.

We also define a canonical neuron ordering based only on weight values. Applied to four differently permuted copies of the source, it returns bit-identical tensors for all 138/138 tensors, making the representation permutation-invariant by construction.

We will add this analysis and state explicitly that the field relies on continuity of its representation, not on smoothness of pretrained weights over neuron index.

Q1. Tokens in 15K Gemma steps and consistency of the two settings

Gemma-3-270M settingValue
Training dataFineWeb-Edu, sample-10BT
EvaluationWikiText-2 test perplexity
Sequence length4,096 tokens
Batch4 sequences/device × 4 gradient-accumulation steps = 16 sequences/step
Tokens per step65,536
OptimizerAdamW, lr=5e-4, cosine schedule, 2,000 warm-up steps
Precision / seedbf16 / 42

15,000×65,536≈0.98B15{,}000 \times 65{,}536 \approx \mathbf{0.98B} tokens.

Thus, the Gemma experiment and the 1B-token MobileLLM experiment use approximately the same training budget, expressed once in steps and once in tokens. We will report both in tokens and add the full Gemma batch configuration to App. D.


Q2. Why Figure 3 uses inr_rescale, and inr_blend vs. tiling

Figure 3 was designed to isolate pure field-based initialization and therefore used inr_rescale; the mixing modes were evaluated separately in Table 4.

We have now evaluated inr_blend under the same Figure 3 setting.

SlimPajama, 1B tokens, lr=3e-4, seed 42

SP val PPL ↓5M25M100M200M300M500M750M1B
Tiling70.347.041.839.938.335.131.529.8
inr_blend (β=0.05\beta=0.05)97.951.643.139.837.533.229.127.4

inr_blend starts behind tiling, becomes comparable at approximately 200M tokens, and finishes 2.4 PPL ahead at 1B tokens. As shown in W1, matched noise reproduces the same final gain.

Updated Table 4: FineWeb pre-training

We re-ran Table 4 using FineWeb pre-training and WikiText-2 test evaluation. Each field-based mode uses a learning rate selected by a 5M-token sweep on WikiText-2 validation PPL; baselines were not tuned.

Learning-rate sweep, β=0.5\beta=0.5

Initialization1e-53e-51e-43e-4
inr_blend2775.61863.61393.41255.2
inr_merge2971.32059.51444.61227.8
inr_tile_merge2960.22024.91440.21234.1
inr_src2884.62064.41538.01267.3
inr_extend2882.72068.11534.51272.8
inr_rescale3086.82241.31604.11357.6
inr3153.82336.51691.91348.4

All field-based modes select lr=3e-4.

45M-token FineWeb results, 3 seeds

InitializationWT2 test @5M ↓@25M ↓@45M ↓
inr_blend1473 ± 39423 ± 24246.7 ± 6.0
inr_merge1506 ± 27463 ± 25299.7 ± 5.9
inr_tile_merge1507 ± 71465 ± 29300.3 ± 13.2
inr_src1447 ± 53585 ± 16431.8 ± 11.1
inr_rescale1670 ± 63620 ± 18434.0 ± 1.6
inr_extend1449 ± 26592 ± 20438.4 ± 8.8
inr1627 ± 58633 ± 8488.8 ± 14.0
Tiling90.9 ± 1.965.6 ± 2.258.6 ± 0.3
Random initialization2300 ± 78985 ± 43756.6 ± 42.0

Field-based modes use lr=3e-4; tiling and random initialization use lr=1e-4 and were not tuned.

Mixing modes with β=0.05\beta=0.05

Initialization1e-53e-51e-43e-4WT2 test @45M ↓
inr_tile_merge4254256017001295223.3 ± 2.4
inr_merge4213251716931285228.1 ± 3.2
inr_blend4403239816771376238.5 ± 7.6

At β=0.5\beta=0.5, the ranking among the field-based modes matches the original Table 4: inr_blend performs best, followed by the merge variants, while inr performs worst. Reducing β\beta to 0.05 improves the mixing modes substantially and brings them within approximately 15 PPL of one another.

At the 45M-token budget, all field-based modes outperform random initialization but remain behind tiling. At 1B tokens, inr_blend overtakes tiling, although W1 shows that matched noise reproduces the same final gain.

We will replace Table 4 with these FineWeb results and update Figure 3 to show the full trajectories for inr_rescale, inr_blend, tiling, and the noise controls.


Q3. Compute cost of fitting WINR

Fitting all 105 fields of MobileLLM-R1-140M takes about 4 GPU-hours on one H200 (130–165 s per tensor), or about 2 hours when two fits share a GPU; the fits are independent and parallelise across GPUs. Decoding the 360M model from the fields then takes under a minute. Generating the 360M model from the fitted fields takes about 50 seconds on one H200. Fitting all 105 fields takes about 4 GPU-hours, or about 2 hours to 1hours on one GPU with two fits running at once . The table below no longer has the shared-GPU row.
pprox 50\text{ s}$ (48 s and 50 s across 2 runs) |

Per-tensor fit times (Layer 0, sequential on 1 H200 GPU):

TensorShapeFit Time
q, k, v, o576×576576 \times 576 or 192×576192 \times 576131–137 s
gate, up, down2048×5762048 \times 576 or 576×2048576 \times 2048145–163 s
Reviewer_MRkYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

he paper proposes to study the model growth problem as querying new architectural coordinates in a continuous weight space. The proposed method - WINR - treats the pre-trained tensors as evaluations of an underlying continuous weight field over a structured coordinate space. WINR is applied to scaling pre-trained models on single-architecture and multi-architecture settings. Evaluations show promising expansion results on single-architecture setting while limited on cross-family transfer for decoder only LM. For BERT style LM, it shows competitive performance over LEMON.

Critical strengths
  1. The problem formulation through the lense of querying coordinates in weight space is general and flexible. The coordinate based generation provides a convenient interface for synthesizing
  2. The experiments cover distinct settings, including decoder-only LM, BERT style-LM, intra and inter family transfers. Interstingly the error in predicted weights doesn't impact downstream tasks much on mobile llm.
Critical weaknesses
  1. Missing important related work. Prior work NeRN [1] already proposes to learn coordinate-to-weight mappings for pre-trained neural networks. The difference between WINR and NeRN should be clearly discussed.
  2. No explanation on why cross-architecture synthesis doesn't perform. Unlike image pixels, neighbouring neurons or embedding rows of LM do no automatically have related meanings. Neuron permutations can peeserve a model's function while substantially changing the coordinate-to-weight mapping which may explain why cross-architecture synthesis performs poorly.

[1] M. Ashkenazi et. al, NeRN - Learning Neural Representations for Neural Networks

Questions for authors
  1. What are the key technical differences between WINR and NeRN?
  2. What explains the limited improvements in the Gemma cross-family experiment?
5 comments

We thank the reviewer for the careful reading and the precise questions. We address the the weaknesses and the questions below. ---

W1. Missing important related work. Prior work NeRN [1] already proposes to learn coordinate-to-weight mappings for pre-trained neural networks. The difference between WINR and NeRN should be clearly discussed. Q1. What are the key technical differences between WINR and NeRN?

We thank the reviewer for pointing us to NeRN. We will add it, together with NeuMeta (Yang & Wang, ECCV'24), to the related work in Sec. 2. Both NeRN and WINR learn a coordinate-to-weight mapping, but they solve different problems, and the technical choices follow from that.

NeRN (Ashkenazi et al., ICLR'23)WINR (ours)
Goalcompress and reconstruct one fixed networkinitialise a different target: a larger model, or another architecture
Coordinatesinteger indices (layer, filter, channel), defined only for the source shapecontinuous normalised coordinates (depth, row, col) ∈ [0,1]³, which can be queried on any target grid
Outputa whole k×k convolution kernel per coordinateone scalar weight per coordinate (arcsinh-normalised), so any matrix-shaped tensor can be represented (attention, MLP, embeddings)
ArchitecturesCNNstransformers and CNNs
Training objectivereconstruction + output and feature distillation (needs data)reconstruction only (no data needed)
Smoothnessimposed through a kernel permutation and an optional regularisernot imposed
Sharing across modelsnosemantic / graph conditioning shares one field across several architectures (Sec. 3.3)
New shapesno: the integer coordinates have no meaning outside the source shapeyes: any target shape, several sources, other architectures

In short, NeRN is a compression method for a fixed network; WINR is an initialisation method that decodes a learned weight representation into new shapes and architectures.

To make the comparison concrete, we also ran NeRN head-to-head, both on its own task and on ours.

(a) NeRN's own task: reconstructing ResNet-20 on CIFAR-10.

  • Setting.
  • Checkpoint: ResNet-20 (option-B shortcuts), 92.59% test accuracy.
  • Represented weights: all convolution kernels, 270,256 weights, as in NeRN.
  • NeRN: run with its official code and its best Table-1 configuration (120 of 350 epochs, with distillation). We added gradient clipping 0.05 after two NaN divergences with the repository settings.
  • WINR: one field shared by the whole network, with coordinates (depth, out-channel, in-channel, kernel position) and a layer-type embedding; dual-band Fourier features, arcsinh targets, log-cosh loss, 100k steps; no distillation.
  • Metric: CIFAR-10 test accuracy of the reconstructed network, without fine-tuning.
  • Timing: wall-clock on a shared H200.
methodpredictor parameters (× represented weights)neuron permutationdistillationtest accuracyfit time
original ResNet-20–––92.59%–
NeRN170.6K (0.63×)yesyes92.20%4,175 s
WINR170.2K (0.63×)nono84.40%773 s
WINR170.2K (0.63×)yesno91.07%789 s
WINR269.0K (1.0×)nono92.56%589 s
WINR538.0K (2.0×)nono92.60%1,152 s
  • At NeRN's compressed size (0.63×), NeRN's permutation and distillation pay off: NeRN reaches 92.20%, against 91.07% for WINR with the same permutation and no distillation.
  • From 1× capacity, WINR reconstructs the network essentially losslessly (92.56% vs 92.59%) without any permutation or distillation, and fits about 7× faster.

WINR is not designed for compression. Its goal is a faithful, data-free representation that can be decoded into other shapes.

(b) Our task: growing an LLM.

  • Setting.
  • Growth: MobileLLM-R1-140M → 360M shape.
  • NeRN port: NeRN was adapted to the 105 linear tensors of the 140M model: one 5-block MLP (hidden 3224) with NeRN's positional encoding, its greedy permutation, 30k reconstruction steps and 60 distillation steps (64 min). The predictor budget (≈32M parameters) matches WINR's. To grow, the normalised denser grid is queried.
  • Shared parts: embeddings and norms are built identically for all methods.
  • Training: SlimPajama for 300M tokens (validation perplexity, seed 42), and WikiText-2 for 45M tokens (best test perplexity, 3 seeds), with the same training pipeline for every arm.
initialisationreconstruction cosineSlimPajama val PPL @300MWikiText-2 best PPL
random–49.4169.7 ± 0.5
NeRN, ported0.73142.4183.0 ± 0.6
tiling (lr 3e-4 / 1e-4)–31.3 / 29.8742.2 ± 0.9
WINR, structured decoding0.99726.9723.47 ± 0.09

NeRN's integer coordinates define no position for neurons that do not exist in the source. Its decoded 360M model is therefore close to a random initialisation (42.4 vs 49.4), while WINR's structured decoding beats tiling (26.97 vs 29.87).

The NeRN port received about one GPU-hour of tuning and is an adaptation, not NeRN at its best. WINR's structured decoding uses lr 1e-5 (SlimPajama) and 3e-5 (WikiText-2), tuned for our method only; tiling, random and NeRN were not tuned.


Weakness 2 and Q2: smoothness, permutation symmetry and the cross-family result

Setting common to the analyses below.

  • Model. All analyses use MobileLLM-R1-140M (hidden size 576, MLP width 2048, 15 layers): its 105 linear weight tensors (q, k, v, o, gate, up, down in each layer) and its token-embedding matrix (128,256 × 576).
  • Fields. One field per tensor, with the paper's architecture and the same number of parameters as the tensor; 30k fitting steps.
  • Function-preserving permutations. We permute the residual channels (one order shared by all layers and the embedding), the MLP units of each layer, the KV groups, and the query heads within each group. Every tensor that reads or writes a permuted axis is permuted consistently. We checked that the permuted model computes the same function: on 4 × 256 random tokens in fp32 its logits differ from the original's by at most 5×10⁻⁵.
  • A random permutation draws these orders at random.
  • A smoothing permutation orders neurons along a greedy shortest path through their weight vectors. This is the most favourable order for smoothness that preserves the function.

W2 (part 1). Unlike image pixels, neighbouring neurons or embedding rows of an LM do not automatically have related meanings.

We agree, and we measured this. For each weight matrix we computed three quantities and averaged them over the 105 tensors:

  • the correlation between adjacent entries (neuron i vs neuron i+1), along rows and along columns;
  • the TV ratio: mean |difference| between adjacent entries divided by that between random pairs (1 = adjacency carries no information);
  • the share of 2D Fourier energy in the lowest 10% × 10% of frequencies (≈1% for white noise).
MobileLLM-R1-140M weightsadjacent correlation (rows / cols)TV ratiolow-frequency energy
pretrained checkpoint0.005 / 0.0020.9831.00%
random function-preserving permutation0.005 / 0.0020.9840.99%
smoothing function-preserving permutation0.035 / 0.0290.9641.20%
random shuffle of all entries (null)0.001 / 0.0011.0000.98%

Weights. Pretrained weights are as unstructured over neuron index as a random shuffle of their own entries. Even the most favourable function-preserving order only reaches a correlation of 0.03.

Embedding rows. On the mean-centred embedding matrix, the cosine similarity between consecutive token ids is 0.019. Random token pairs give 0.0006, and a token's nearest neighbour gives 0.57 (2,000 sampled tokens). Row order is essentially unrelated to meaning.

What the field does instead. The field does not rely on smoothness. Identical fields fitted to a tensor, to a random function-preserving permutation of it, and to a full shuffle of its entries reach the same accuracy (layer-0 q: cosine 0.9983 / 0.9981 / 0.9983). The field memorises the values at the source coordinates.

The difficulty arises only when the field is queried between those coordinates:

field decoded atadjacent correlation (rows / cols)TV ratiolow-frequency energy
the source size (576)0.016 / 0.0100.9791.01%
a larger target size (1024)0.355 / 0.3970.7582.99%

At a new size the generated weights are smooth interpolations between neighbouring source neurons. Since those neurons are unrelated, an interpolated weight corresponds to no real neuron. We will remove any statement stating that pretrained weights vary smoothly from the paper's motivation.

W2 (part 2). Neuron permutations can preserve a model's function while substantially changing the coordinate-to-weight mapping, which may explain why cross-architecture synthesis performs poorly.

We agree that permutation sensitivity is an important concern. To test it directly, we constructed three functionally equivalent versions of MobileLLM-R1-140M: the original model, a random function-preserving permutation, and a smoothing permutation. We refitted WINR independently on each version and decoded them into MobileLLM-R1-360M while preserving the exact coordinates of existing source neurons. All models were trained under identical settings.

Source neuron orderSlimPajama val. PPL @300M ↓WikiText-2 test PPL ↓
Original26.9723.62
Random function-preserving permutation26.9823.67
Smoothing permutation26.9723.57
Tiling baseline29.8742.20

The three functionally equivalent orderings produce essentially identical downstream performance (within 0.01 PPL on SlimPajama and 0.1 PPL on WikiText-2). This shows that permutation itself does not degrade generation when source-coordinate correspondence is preserved. The difficulty arises when resizing requires interpolation between coordinates whose neuron ordering has no functional meaning. We will clarify this distinction in the revision.


W2 (part 3). No explanation on why cross-architecture synthesis doesn't perform.
Q2. What explains the limited improvements in the Gemma cross-family experiment?

We agree that the original submission did not sufficiently discuss this result. It is challenging to isolate the exact cause because Gemma differs substantially from the source models in architecture, representation space, and tokenizer. Together with the permutation analysis above, this suggests that the lack of a meaningful correspondence between source and target neurons may limit direct cross-architecture resampling.

To further investigate this hypothesis, we explicitly aligned the source and target representation spaces before fitting and decoding the fields:

Gemma-3-270M initializationWikiText-2 test PPL ↓FineWeb val. loss ↓
Random73.1 ± 0.93.994 ± 0.004
Aligned embeddings, random body60.6 ± 0.93.836 ± 0.003
Aligned embeddings + field-decoded aligned MLPs54.8 ± 0.83.756 ± 0.009
Aligned embeddings + aligned MLP weights copied (no field)55.6 ± 2.53.756 ± 0.009

The improvement after alignment suggests that source-target correspondence is an important factor in cross-family transfer. We will add this analysis to Sec. 4.2 while making clear that it does not establish a unique cause for the limited Gemma gains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.