- Rating
- 3(−1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose using implicit neural representations of weights (WINR) for creating warm-start initializations for training large models. The main conceptual idea is to view weights as a continuous field on an abstract space that encodes relative position of the weights within the architecture as well as further metadata.
The evaluation focuses on generating warm-start weights for language models MobileLLM-R1, Gemma, and BERT with up to ca. 400 M parameters. WINR shows promising performance for encoding weights at fixed architectural parameters and creating initializations for expanding width within the same model family (MobileLLM-R1) but struggles to generalize across model families (MobileLLM-R1 to Gemma). The authors show empirically that diverse source weights aide performance and conduct several ablations. Overall, the submission’s contributions are novel as well as interesting both on a conceptual and technical level with relevance to several communities, including researchers working on weight space learning, initialization, and optimization. I will list critical strengths, critical weaknesses, and questions in the dedicated fields below.
As a minor comment, I want to make the authors aware of a broken citation to LEMON on line 302 of their submission.
Critical strengths
Applying INRs to weights is conceptually interesting and, to the best of my knowledge, novel. The approach could thus open a new line of research within weight space learning.
The evaluation is well designed to support the main claims of the papers. The ablation studies are illuminating, and the comparison against tiling plus noise in Section 4.4 establishes an honest baseline that any method like the author’s needs to beat. All these experiments are thoroughly documented. These points underscore the general scientific rigor of the submission.
Critical weaknesses
WINR shows weak performance when generalizing to different model families. While this is a clear weakness of the method, I would not consider it a reason for rejection, as the task is challenging and existing methods do (to my knowledge) not consider this task at all.
Questions for authors
- In section 4.1, a maximum of 1B tokens was used. How was this endpoint chosen? Justifying this choice carefully is important as the ranking among methods appears to depend on the number of tokens in Fig. 3.
2, In the ablation study the method that performs best is inr_blend. This method uses a rather weak learned component (). I would be curious whether the authors have some intuition for the reasons behind this.
3 comments
The ranking of inr_blend and tiling is budget-dependent: inr_blend is worse through 100M tokens, becomes comparable around 200M, and finishes 2.4 PPL ahead at 1B. Exact-coordinate decoding remains ahead at every evaluated checkpoint from 5M to 1B tokens.
We will therefore replace the single-endpoint comparison in Fig. 3 with full training curves and state explicitly that 1B tokens is a fixed compute budget rather than a convergence point.
Q2. Why does inr_blend, with a small learned component, perform best?
Short answer. At small , the gain of inr_blend over tiling comes from the corrections and from introducing a small perturbation, rather than from the specific values predicted by the field.
(a) Matched-noise control. We keep tiling and the corrections (fan-in rescaling and spectral restoration) fixed and replace the field with Gaussian noise matching its per-tensor mean and standard deviation.
| Initialization | SP val PPL @1B ↓ |
|---|---|
| Tiling only | 29.8 |
| Tiling + corrections, no perturbation | 28.0 |
| Tiling + matched noise + corrections | 27.4 |
inr_blend (tiling + field + corrections) | 27.4 |
Of the 2.4-PPL gain over tiling, the corrections account for 1.8 PPL, while the remaining 0.6 PPL is reproduced by matched noise. Thus, for inr_blend at , the precise field-generated values do not provide an advantage over a matched perturbation.
(b) Training dynamics. The training curves are consistent with a small perturbation helping optimization move away from the highly duplicated tiled initialization. inr_blend initially performs worse than tiling (97.9 vs. 70.3 at 5M tokens), becomes comparable around 200M tokens, and then improves faster:
| Initialization | 5M | 200M | 500M | 1B | 200M→1B |
|---|---|---|---|---|---|
| Tiling | 70.3 | 39.9 | 35.1 | 29.8 | -10.1 |
inr_blend () | 97.9 | 39.8 | 33.2 | 27.4 | -12.4 |
We view this as a plausible interpretation of the optimization behavior rather than a mechanism established by the current experiments.
(c) Effect of the mixing coefficient. We also compare and under FineWeb pre-training for 45M tokens, evaluated on WikiText-2 test PPL over 3 seeds.
| Mixing mode | ||
|---|---|---|
inr_tile_merge | 300.3 ± 13.2 | 223.3 ± 2.4 |
inr_merge | 299.7 ± 5.9 | 228.1 ± 3.2 |
inr_blend | 246.7 ± 6.0 | 238.5 ± 7.6 |
Reducing substantially improves the merge variants, indicating that preserving most of the tiled initialization is important.
(d) Where the field contributes beyond a perturbation. When queried at the exact source coordinates, the same fields generate every linear weight without tiling and outperform tiling at every evaluated checkpoint in the 1B-token run. A control that fills the new neurons randomly instead of using field-generated values gives essentially the same result (27.62 vs. 27.72 PPL at 300M tokens, lr 1e-4). Thus, in this setting, the field's contribution comes primarily from accurately reconstructing the existing pretrained neurons (cosine 0.997), rather than from the values assigned to new neurons.
We will revise Sec. 4.4 accordingly
I thank the authors for answering my questions and conducting additional experiments to elucidate the role of the learned component in inr_blend.
I found the additional cross-architecture experiments helpful for understanding why this task might be hard. Additionally, the exact source coordinate version of WINR introduced during the discussion shows improved performance. I encourage the authors to make it a more prominent part of their work, while carefully examining the origin of the improved performance.
However, the discussion also surfaced an additional weakness of WINR in the inr_blend version. The method performs on par with adding noise of comparable magnitude to tiled weights. It is very surprising that this effect was not observed in the ablation presented in Section 4.4 “Tile-plus-noise control” of the initial submission, which appeared to be designed to test for it. Moreover, weighing the learned component in inr_blend more strongly decreases performance substantially, further drawing into question that useful information is learned in the inr_blend version.
In light of the discussion, I have to lower my score: Overall, the presented method is interesting and the paper insightful, but the discussion has raised serious questions about the performance of WINR compared to rather weak baselines.
We thank the reviewer for the positive and thoughtful assessment. We have fixed the broken LEMON citation on line 302. Below, we address the weakness and questions directly with new experiments.
Common setting (MobileLLM). MobileLLM-R1-140M (hidden 576, MLP 2048, 15 layers) is grown to MobileLLM-R1-360M (hidden 1024, MLP 4096) and pre-trained on SlimPajama (SP): AdamW, β=(0.9,0.95), weight decay 0.1, 2% warm-up followed by cosine decay over the full budget, sequence length 1,024, 32,768 tokens/step, bf16. We report validation PPL on 2M held-out tokens.
W1. Generalization across model families
We agree that this is the main limitation of the submission.
Likely explanation. A coordinate-to-weight field can transfer effectively only when source and target coordinates have compatible meaning. Across model families, this correspondence is weak:
gate_projtensors were decoded from the field; the remaining tensors, including the embedding matrix (about 62% of the parameters), were randomly initialized.We therefore investigated whether explicitly aligning the source and target representation spaces improves transfer.
Cross-family alignment.
Gemma-3-270M was trained for 200M FineWeb tokens with the same recipe for every initialization (AdamW, lr 5e-4, cosine schedule, 65,536 tokens/step, bf16), using 3 seeds.
Alignment substantially improves cross-family transfer. Field-decoded aligned MLPs reduce perplexity by about 25% relative to random initialization, improve over aligned embeddings alone, and perform comparably to directly copying the aligned MLP weights. Without alignment, the same donor fields provide no gain.
These results indicate that source-target correspondence is an important factor in cross-family transfer. We will add this experiment to Sec. 4.2 and state its current limitations: the embeddings are explicitly aligned rather than generated by the field, and attention weights remain randomly initialized.
Q1. How was the 1B-token endpoint chosen?
The 1B-token endpoint was a practical compute budget rather than a convergence criterion. It allowed us to evaluate all initializations and ablations under the same budget. The Gemma experiment uses approximately the same budget: 15K steps × 65,536 tokens ≈ 0.98B tokens.
We agree that a single endpoint can obscure changes in ranking, so we evaluated the Figure 3 setting throughout the full 1B-token run.
Setting: SP, 1B tokens (30,518 steps), seed 42. Tiling and
inr_blenduse lr 3e-4. Exact-coordinate decoding queries the same per-tensor fields at the exact source coordinates for existing neurons and at intermediate coordinates for new neurons; its lr (1e-5) was selected by a sweep for this method only.inr_blend(β=0.05)