- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Config2Weights, a framework for generating initial model weights directly from architectural configurations. The authors investigate two conditioning approaches, flat conditioning and graph conditioning, and two generation methods, direct mapping and conditional latent flow. They evaluate the framework on both LoRA adapter initialization and full-model pretraining, using training loss, perplexity, and downstream task accuracy.
Critical strengths
Interesting research direction. Generating useful initial weights directly from architectural configurations is an interesting and potentially valuable idea. If successful, it could reduce the cost of exploring new architectures and provide an alternative to random initialization or handcrafted model-expansion methods. Extending weight generation from LoRA adapters to full transformer checkpoints is also an ambitious direction.
Critical weaknesses
-
Mixed empirical results. Although the proposed approach demonstrates improvements in some settings, the results are inconsistent, and the evidence for a substantial practical advantage remains limited.
- Diminishing gains with model size (Figure 3). For LoRA fine-tuning, the improvement in final HellaSwag accuracy over random initialization decreases from 12.35 percentage points on Llama-1B to 4.05 points on Llama-3B and 1.68 points on Llama-8B. This raises questions about whether the approach will remain beneficial at larger model scales.
- Limited advantage over existing methods (Figure 4). C2W achieves nearly identical final training losses to Gstack-direct and HyperCloning. Although its ability to handle different architectures without handcrafted transformations is useful, the experiments do not establish a substantial performance advantage over these established initialization approaches.
- Modest gains after continued pretraining (Figure 5). On MobileLLM-R1-140M, the generated initialization improves test perplexity by 3.29% and test loss by 2.39% after approximately 100M tokens. These improvements are positive but relatively modest, leaving open the question of whether they persist or become negligible with longer pretraining.
- Inconsistent downstream improvements (Figure 6). The improvement is considerably larger on ARC-Easy than on ARC-Challenge or MMLU. Additional-seed evaluations show identical mean ARC-Challenge accuracy for generated and random initializations (16.81%), while MMLU performance remains close to chance. Thus, it remains unclear whether the proposed method improves downstream capabilities beyond accelerating early optimization.
-
Limited evidence of generalization to genuinely novel architectures. Although MobileLLM-R1-140M is excluded from generator training, the training zoo still contains MobileLLM-R1-360M from the same family. This is not necessarily data leakage, but it makes the held-out experiment less convincing as evidence of generalization across architectural families. Similarly, the model-expansion experiments involve target architectures structurally related to models in the training zoo. While the paper includes stronger held-out experiments, a more systematic evaluation of generalization as a function of architectural similarity would strengthen the central claim.
-
Insufficient justification and controlled ablations. The paper introduces several design choices, including flat versus graph conditioning, direct mapping versus latent flow, and latent-alignment objectives. However, the experiments do not sufficiently isolate the contribution of each choice. In particular, the 3B experiment compares pipelines that differ not only in conditioning but also in chunk size, decoder, and training objective. Consequently, it is difficult to determine which component is responsible for the improvement. A more systematic ablation study would help explain when and why the proposed approach works.
Questions for authors
1. How persistent are the benefits of generated initialization?
The gains over random initialization appear to diminish with longer training in some settings. For example, Figure 5 shows modest improvements after 100M tokens, while Figure 8 shows that the advantage nearly disappears for GPT-small after 10B tokens but remains substantial for the unseen U4 architecture after 5B tokens.
What determines whether generated initialization provides a persistent advantage or merely accelerates early optimization? Could the authors evaluate the training compute or number of tokens required to achieve matched validation loss and downstream accuracy? Given the substantial cost of constructing the training zoo and training the weight generator, an amortized compute-efficiency analysis would also help establish the practical value of the approach.
2. Why is DCLM score used in Figure 8?
Earlier experiments evaluate the generated initializations using training loss, perplexity, and downstream task accuracy, whereas Figure 8 uses DCLM score to evaluate longer-horizon pretraining. Could the authors explain this choice and report the corresponding validation losses and downstream benchmark results? In particular, would the conclusion about the persistence of initialization benefits remain consistent across these metrics?
6 comments
W4: Modest gains after continued pretraining and persistence over longer horizons (Figure 5).
Modest gains after continued pretraining (Figure 5). On MobileLLM-R1-140M, the generated initialization improves test perplexity by 3.29% and test loss by 2.39% after approximately 100M tokens. These improvements are positive but relatively modest, leaving open the question of whether they persist or become negligible with longer pretraining.
We agree that the gains at tokens are modest, and our longer-horizon runs show that persistence depends on the target architecture:
- MobileLLM Family Holdout ( tokens, 3 seeds): C2W Flow maintains a statistically consistent advantage over random initialization in both final test cross-entropy ( vs. ) and 8-task downstream accuracy ( vs. ; see Reviewer P83q, Q4).
- In-Range vs. Extrapolated Targets ( tokens): On in-range
GPT-small, the gap largely closes by tokens, whereas on the extrapolated, unseenU4architecture, the advantage persists through tokens (Q1–Q2 below).
W5: Inconsistent downstream improvements across benchmarks (Figure 6).
Inconsistent downstream improvements (Figure 6). The improvement is considerably larger on ARC-Easy than on ARC-Challenge or MMLU. Additional-seed evaluations show identical mean ARC-Challenge accuracy for generated and random initializations (16.81%), while MMLU performance remains close to chance. Thus, it remains unclear whether the proposed method improves downstream capabilities beyond accelerating early optimization.
We agree. On MobileLLM-R1-140M after tokens, only ARC-Easy shows a separation in mean accuracy across three seeds ( vs. ), and we do not claim a reliable gain on ARC-Challenge or MMLU, where a model remains near random chance.
The per-benchmark breakdown for Figure 8 (Q2) reflects a similar pattern and gains are not uniform across all sub-tasks:
- GPT-small (Flow vs. Random across 22 DCLM components): Improves on , ties on , and trails on .
- Unseen U4 (Flow vs. Random across 22 DCLM components): Improves on , ties on , and trails on .
We will frame these findings accurately as an aggregate capability gain strongest on extrapolated targets such as U4 rather than a uniform improvement across every individual benchmark.
W6: Limited evidence of cross-family generalization and lack of a graded architectural-distance analysis.
Limited evidence of generalization to genuinely novel architectures. Although MobileLLM-R1-140M is excluded from generator training, the training zoo still contains MobileLLM-R1-360M from the same family. This is not necessarily data leakage, but it makes the held-out experiment less convincing as evidence of generalization across architectural families. Similarly, the model-expansion experiments involve target architectures structurally related to models in the training zoo. While the paper includes stronger held-out experiments, a more systematic evaluation of generalization as a function of architectural similarity would strengthen the central claim.
We agree that the original MobileLLM setup evaluated a held-out configuration rather than a held-out model family. To address this, we reran the experiment with all MobileLLM checkpoints strictly excluded from the training zoo (using SmolLM2-135M, SmolLM2-360M, and Qwen3-0.6B-Base as sources) under a fixed generation pipeline. Across three seeds on MobileLLM-R1-140M-base:
- C2W Flow outperforms random initialization ( vs. final test CE; vs. 8-task accuracy).
- Unmodified C2W Direct does not ( CE; accuracy).
- Closest-source weight copy remains stronger ( CE; accuracy).
(Full results are detailed in our response to Reviewer P83q, Q4.) We acknowledge that we evaluate a single excluded family rather than a graded continuum of architectural distances, and we will explicitly state the absence of a systematic architectural-distance sweep as a limitation.
W7: Insufficient justification and controlled ablations across design choices.
Insufficient justification and controlled ablations. The paper introduces several design choices, including flat versus graph conditioning, direct mapping versus latent flow, and latent-alignment objectives. However, the experiments do not sufficiently isolate the contribution of each choice. In particular, the 3B experiment compares pipelines that differ not only in conditioning but also in chunk size, decoder, and training objective. Consequently, it is difficult to determine which component is responsible for the improvement. A more systematic ablation study would help explain when and why the proposed approach works.
We agree that the 3B comparison (Sec. 4.2.4) does not isolate conditioning alone, as the two pipelines differ across multiple components (App. F.5). To provide clean component-level comparisons, our in our additional experiments isolate design choices under strictly fixed pipelines across three settings:
- Direct Mapping vs. Latent Flow (MobileLLM Family Holdout): Compares direct and flow generation while holding the generator backbone, graph conditioning, canonical weight representation, and decoder identical (Reviewer P83q, Q4).
- Factorized Grid (NNiT Zoo, G3): Systematically evaluates weight representations generator variants training regimes.
- Mapper vs. Flow under Fixed Zoo & Decoder (G1, G2): Isolates deterministic mapping versus flow matching on identical GPT-2 and Llama zoos. he latent-alignment objective is ablated separately in Sec. 4.3.
Finally, our empirical sweep on wiki-text across zoo sizes indicates that cross-architecture extrapolation remains sensitive to the interplay between zoo coverage and the pretraining corpus we observed substantially different transfer gaps using the exact same model zoo when changing only the pretraining dataset. Pinpointing the exact mechanism behind this dataset sensitivity and establishing principled remedies remain open questions that we will discuss candidly in the revision.
Questions
Q1: Persistence of initialization gains, tokens/compute to matched loss, and amortized efficiency analysis.
How persistent are the benefits of generated initialization? What determines persistence, and can the authors report tokens or compute to matched validation loss and an amortized efficiency analysis?
Persistence Across Targets & Regimes: Persistence depends strongly on target scale relative to the source zoo and the pretraining corpus:
- In-Range Target (
GPT-small, tokens):Flowmaintains a lower training loss than random initialization across all ten checkpoints, though the gap narrows to by tokens (trajectories in Q2). - Extrapolated Unseen Target (
U4, tokens): OnU4an unseen architecture wider and larger than every source model,Flowmaintains a more durable advantage at tokens, remaining below random initialization and below HyperCloning. - WikiText / LoGAH Size-Transfer Protocol (
G1): In the reassessed GPT-2 protocol, gains over random initialization are substantially larger even though the source zoo models ( parameters) are far smaller than the targets (the smallest target,GPT-2 Smallat , is larger than the largest source model).
Tokens to Matched Performance (Proxy Metrics): Validation loss was not logged during these pretraining runs, so exact tokens-to-matched-validation-loss cannot be reported. Using the logged training loss and DCLM Core curves as proxies:
- On
U4( budget):Flowsurpasses random initialization's final DCLM Core score () at tokens ( of the budget), and its smoothed training loss drops below random's final loss () by tokens. - On
GPT-small( budget): Both generated initializations reach random initialization's final DCLM Core score () by tokens ( of the budget).
Amortized Efficiency Analysis: We agree that constructing a model zoo from scratch is expensive zoo creation and fine-tuning accounted for the majority of our experimental compute. However, the intended practical workflow for C2W is to leverage existing publicly pretrained checkpoints (on the Hugging Face Hub) rather than training a zoo from scratch; we trained custom zoos solely to conduct controlled studies on failure modes across wide architectural variations.
Excluding source-zoo construction, generator training is a one-time amortized cost of hours for the Mapper and hours for Flow on TPU v6e (Reviewer HWtn, W5).
Q2: Rationale for using DCLM Core in Figure 8, corresponding loss trajectories, and downstream benchmark breakdown.
Why is DCLM score used in Figure 8? Please report the corresponding validation losses and downstream results. Would the conclusion about persistence remain consistent across metrics?
We thank the reviewer for the balanced assessment. We recovered the training-loss trajectories and per-benchmark results behind Figure 8 (Q1–Q2) and added comparisons with learned weight generators (G1–G3 in the general response).
Weaknesses
W1: Mixed empirical results and limited evidence of a substantial practical advantage.
We agree that our empirical results vary across regimes, and we will revise our claims to explicitly delineate where C2W yields substantial gains versus where its advantages are modest or absent.
At the same time, our expanded evaluations demonstrate clear practical advantages over existing learned weight-generation methods, with the strongest gains occurring on compact targets and zoos trained on consistent corpora:
Our additional experiments indicate that initialization quality depends strongly on the underlying pretraining corpus and source-zoo composition; formally characterizing this relationship remains an important open direction.
W2: Diminishing gains with increasing model scale (Figure 3).
We agree that the final-step accuracy margin narrows as backbone scale increases, as noted in App. B. Larger backbones leave substantially less headroom; for example, random initialization already reaches 91.76% on
Llama-8B. However, C2W provides substantial early-training acceleration: at step 56, C2W outperforms random initialization by +30.10 points onLlama-3Band +6.86 points onLlama-8B, reaching the final accuracy of random initialization at step 169 (1B), step 226 (3B), and step 339 (8B, Flow) out of 623 total steps (Table 8).Additionally, our new experiments confirm that scaling behavior depends on the task and training regime:
140M,1B, and3BLlama targets.Together, these findings indicate that while early-stage convergence speedups hold across scales, absolute final-performance gains are most pronounced for smaller target models.
W3: Limited advantage over existing model-expansion methods (Figure 4).
We agree that copy-based model morphisms remain highly competitive when expanding a parent architecture, and that the margin over them is narrow (indeed, in our new MobileLLM family holdout, closest-source weight copying outperforms C2W; Reviewer P83q, Q4). The primary advantage of C2W over heuristic morphisms is structural flexibility: a single trained generator synthesizes initializations for arbitrary target configurations including architectures that are not direct layer- or width-expansions of a specific parent model.
To benchmark C2W directly against learned weight-generation methods with the same objective, we conducted an additional comparison against LoGAH following its exact protocol (G1): pretraining 81 GPT-2-style source models (7.7M–28.4M parameters) on
WikiText-103and evaluating fine-tuned test perplexity onWikiText-2:The LoGAH was reproduced in jax and retrained on TPU.