acceptodds
Under review as a conference paper at ICLR 2027

Config2Weights: Generating LLM Initializations from Architecture Configurations

Abstract

Training large language models (LLMs) from scratch remains prohibitively expensive, motivating growing interest in weight-space generative modeling as a route to direct parameter synthesis. Existing generative approaches, however, are largely confined to small vision models or low-rank adapters, while heuristic model expansion methods rely on rigid, architecture-specific transformations. We propose **Config2Weights**, a generative initialization framework that learns from a collection of pretrained LLM weights to initialize new architectures conditioned solely on their architectural configuration. Config2Weights comprises two complementary variants: a direct configuration-to-weight mapping and a conditional latent flow operating in the latent space of a pretrained conditional variational autoencoder over model weights. We evaluate Config2Weights on both LoRA adapters and full-model weights across in-distribution and out-of-distribution settings, including unseen architectures and configurations. Across these settings, Config2Weights achieves competitive performance against existing weight-expansion methods.

Primary area:
generative models
Keywords:
config2weights, c2w, weight generation, LLMs weights generation, Generative initialization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YVKWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Config2Weights, a framework for generating initial model weights directly from architectural configurations. The authors investigate two conditioning approaches, flat conditioning and graph conditioning, and two generation methods, direct mapping and conditional latent flow. They evaluate the framework on both LoRA adapter initialization and full-model pretraining, using training loss, perplexity, and downstream task accuracy.

Critical strengths

Interesting research direction. Generating useful initial weights directly from architectural configurations is an interesting and potentially valuable idea. If successful, it could reduce the cost of exploring new architectures and provide an alternative to random initialization or handcrafted model-expansion methods. Extending weight generation from LoRA adapters to full transformer checkpoints is also an ambitious direction.

Critical weaknesses
  1. Mixed empirical results. Although the proposed approach demonstrates improvements in some settings, the results are inconsistent, and the evidence for a substantial practical advantage remains limited.

    • Diminishing gains with model size (Figure 3). For LoRA fine-tuning, the improvement in final HellaSwag accuracy over random initialization decreases from 12.35 percentage points on Llama-1B to 4.05 points on Llama-3B and 1.68 points on Llama-8B. This raises questions about whether the approach will remain beneficial at larger model scales.
    • Limited advantage over existing methods (Figure 4). C2W achieves nearly identical final training losses to Gstack-direct and HyperCloning. Although its ability to handle different architectures without handcrafted transformations is useful, the experiments do not establish a substantial performance advantage over these established initialization approaches.
    • Modest gains after continued pretraining (Figure 5). On MobileLLM-R1-140M, the generated initialization improves test perplexity by 3.29% and test loss by 2.39% after approximately 100M tokens. These improvements are positive but relatively modest, leaving open the question of whether they persist or become negligible with longer pretraining.
    • Inconsistent downstream improvements (Figure 6). The improvement is considerably larger on ARC-Easy than on ARC-Challenge or MMLU. Additional-seed evaluations show identical mean ARC-Challenge accuracy for generated and random initializations (16.81%), while MMLU performance remains close to chance. Thus, it remains unclear whether the proposed method improves downstream capabilities beyond accelerating early optimization.
  2. Limited evidence of generalization to genuinely novel architectures. Although MobileLLM-R1-140M is excluded from generator training, the training zoo still contains MobileLLM-R1-360M from the same family. This is not necessarily data leakage, but it makes the held-out experiment less convincing as evidence of generalization across architectural families. Similarly, the model-expansion experiments involve target architectures structurally related to models in the training zoo. While the paper includes stronger held-out experiments, a more systematic evaluation of generalization as a function of architectural similarity would strengthen the central claim.

  3. Insufficient justification and controlled ablations. The paper introduces several design choices, including flat versus graph conditioning, direct mapping versus latent flow, and latent-alignment objectives. However, the experiments do not sufficiently isolate the contribution of each choice. In particular, the 3B experiment compares pipelines that differ not only in conditioning but also in chunk size, decoder, and training objective. Consequently, it is difficult to determine which component is responsible for the improvement. A more systematic ablation study would help explain when and why the proposed approach works.

Questions for authors

1. How persistent are the benefits of generated initialization?

The gains over random initialization appear to diminish with longer training in some settings. For example, Figure 5 shows modest improvements after 100M tokens, while Figure 8 shows that the advantage nearly disappears for GPT-small after 10B tokens but remains substantial for the unseen U4 architecture after 5B tokens.

What determines whether generated initialization provides a persistent advantage or merely accelerates early optimization? Could the authors evaluate the training compute or number of tokens required to achieve matched validation loss and downstream accuracy? Given the substantial cost of constructing the training zoo and training the weight generator, an amortized compute-efficiency analysis would also help establish the practical value of the approach.

2. Why is DCLM score used in Figure 8?

Earlier experiments evaluate the generated initializations using training loss, perplexity, and downstream task accuracy, whereas Figure 8 uses DCLM score to evaluate longer-horizon pretraining. Could the authors explain this choice and report the corresponding validation losses and downstream benchmark results? In particular, would the conclusion about the persistence of initialization benefits remain consistent across these metrics?

6 comments

We thank the reviewer for the balanced assessment. We recovered the training-loss trajectories and per-benchmark results behind Figure 8 (Q1–Q2) and added comparisons with learned weight generators (G1–G3 in the general response).

Weaknesses

W1: Mixed empirical results and limited evidence of a substantial practical advantage.

Mixed empirical results. Although the proposed approach demonstrates improvements in some settings, the results are inconsistent, and the evidence for a substantial practical advantage remains limited.

We agree that our empirical results vary across regimes, and we will revise our claims to explicitly delineate where C2W yields substantial gains versus where its advantages are modest or absent.

At the same time, our expanded evaluations demonstrate clear practical advantages over existing learned weight-generation methods, with the strongest gains occurring on compact targets and zoos trained on consistent corpora:

  • Where gains are largest: GPT-2 and Llama size transfer (G1, G2) and zero-shot robotic policy generation across unseen architectures (G3).
  • Where gains are modest: Extended pretraining on in-range targets (e.g., GPT-small at 10B10\text{B} tokens, Q2) and certain downstream benchmarks (W5). Furthermore, when a structurally close source checkpoint already exists, direct weight copying remains stronger (Reviewer P83q, Q4).

Our additional experiments indicate that initialization quality depends strongly on the underlying pretraining corpus and source-zoo composition; formally characterizing this relationship remains an important open direction.


W2: Diminishing gains with increasing model scale (Figure 3).

Diminishing gains with model size (Figure 3): the HellaSwag gain over random initialization falls from 12.35 points on Llama-1B to 4.05 on 3B and 1.68 on 8B.

We agree that the final-step accuracy margin narrows as backbone scale increases, as noted in App. B. Larger backbones leave substantially less headroom; for example, random initialization already reaches 91.76%91.76\% on Llama-8B. However, C2W provides substantial early-training acceleration: at step 5656, C2W outperforms random initialization by +30.10+30.10 points on Llama-3B and +6.86+6.86 points on Llama-8B, reaching the final accuracy of random initialization at step 169169 (1B), step 226226 (3B), and step 339339 (8B, Flow) out of 623623 total steps (Table 8).

Additionally, our new experiments confirm that scaling behavior depends on the task and training regime:

  • On WikiText size transfer (G2), C2W outperforms random initialization across 140M, 1B, and 3B Llama targets.
  • On robotic control policies (G3), C2W generates complete task policies for unseen architectures that succeed zero-shot without fine-tuning.

Together, these findings indicate that while early-stage convergence speedups hold across scales, absolute final-performance gains are most pronounced for smaller target models.


W3: Limited advantage over existing model-expansion methods (Figure 4).

Limited advantage over existing methods (Figure 4): C2W reaches nearly identical final training losses to Gstack-direct and HyperCloning.

We agree that copy-based model morphisms remain highly competitive when expanding a parent architecture, and that the margin over them is narrow (indeed, in our new MobileLLM family holdout, closest-source weight copying outperforms C2W; Reviewer P83q, Q4). The primary advantage of C2W over heuristic morphisms is structural flexibility: a single trained generator synthesizes initializations for arbitrary target configurations including architectures that are not direct layer- or width-expansions of a specific parent model.

To benchmark C2W directly against learned weight-generation methods with the same objective, we conducted an additional comparison against LoGAH following its exact protocol (G1): pretraining 8181 GPT-2-style source models (7.7M–28.4M7.7\text{M}–28.4\text{M} parameters) on WikiText-103 and evaluating fine-tuned test perplexity on WikiText-2:

Target ModelRandom InitLocal LoGAH-T / LoGAH-SC2W (Ours)
GPT-2 Small644.2552.2 / 513.3131.0
GPT-2 Medium779.0471.2 / 479.2120.8
GPT-2 Large808.7454.3 / 466.2115.8

The LoGAH was reproduced in jax and retrained on TPU.


W4: Modest gains after continued pretraining and persistence over longer horizons (Figure 5).

Modest gains after continued pretraining (Figure 5). On MobileLLM-R1-140M, the generated initialization improves test perplexity by 3.29% and test loss by 2.39% after approximately 100M tokens. These improvements are positive but relatively modest, leaving open the question of whether they persist or become negligible with longer pretraining.

We agree that the gains at 100M100\text{M} tokens are modest, and our longer-horizon runs show that persistence depends on the target architecture:

  • MobileLLM Family Holdout (1B1\text{B} tokens, 3 seeds): C2W Flow maintains a statistically consistent advantage over random initialization in both final test cross-entropy (3.680±0.0043.680 \pm 0.004 vs. 3.717±0.0043.717 \pm 0.004) and 8-task downstream accuracy (42.28%42.28\% vs. 40.48%40.48\%; see Reviewer P83q, Q4).
  • In-Range vs. Extrapolated Targets (5B–10B5\text{B}–10\text{B} tokens): On in-range GPT-small, the gap largely closes by 10B10\text{B} tokens, whereas on the extrapolated, unseen U4 architecture, the advantage persists through 5B5\text{B} tokens (Q1–Q2 below).

W5: Inconsistent downstream improvements across benchmarks (Figure 6).

Inconsistent downstream improvements (Figure 6). The improvement is considerably larger on ARC-Easy than on ARC-Challenge or MMLU. Additional-seed evaluations show identical mean ARC-Challenge accuracy for generated and random initializations (16.81%), while MMLU performance remains close to chance. Thus, it remains unclear whether the proposed method improves downstream capabilities beyond accelerating early optimization.

We agree. On MobileLLM-R1-140M after 1B1\text{B} tokens, only ARC-Easy shows a separation in mean accuracy across three seeds (35.41±1.3035.41 \pm 1.30 vs. 33.61±0.5033.61 \pm 0.50), and we do not claim a reliable gain on ARC-Challenge or MMLU, where a 140M140\text{M} model remains near random chance.

The per-benchmark breakdown for Figure 8 (Q2) reflects a similar pattern and gains are not uniform across all sub-tasks:

  • GPT-small (Flow vs. Random across 22 DCLM components): Improves on 1111, ties on 11, and trails on 1010.
  • Unseen U4 (Flow vs. Random across 22 DCLM components): Improves on 1818, ties on 11, and trails on 33.

We will frame these findings accurately as an aggregate capability gain strongest on extrapolated targets such as U4 rather than a uniform improvement across every individual benchmark.


W6: Limited evidence of cross-family generalization and lack of a graded architectural-distance analysis.

Limited evidence of generalization to genuinely novel architectures. Although MobileLLM-R1-140M is excluded from generator training, the training zoo still contains MobileLLM-R1-360M from the same family. This is not necessarily data leakage, but it makes the held-out experiment less convincing as evidence of generalization across architectural families. Similarly, the model-expansion experiments involve target architectures structurally related to models in the training zoo. While the paper includes stronger held-out experiments, a more systematic evaluation of generalization as a function of architectural similarity would strengthen the central claim.

We agree that the original MobileLLM setup evaluated a held-out configuration rather than a held-out model family. To address this, we reran the experiment with all MobileLLM checkpoints strictly excluded from the training zoo (using SmolLM2-135M, SmolLM2-360M, and Qwen3-0.6B-Base as sources) under a fixed generation pipeline. Across three seeds on MobileLLM-R1-140M-base:

  • C2W Flow outperforms random initialization (3.6803.680 vs. 3.7173.717 final test CE; 42.28%42.28\% vs. 40.48%40.48\% 8-task accuracy).
  • Unmodified C2W Direct does not (3.8893.889 CE; 39.28%39.28\% accuracy).
  • Closest-source weight copy remains stronger (3.4903.490 CE; 45.21%45.21\% accuracy).

(Full results are detailed in our response to Reviewer P83q, Q4.) We acknowledge that we evaluate a single excluded family rather than a graded continuum of architectural distances, and we will explicitly state the absence of a systematic architectural-distance sweep as a limitation.


W7: Insufficient justification and controlled ablations across design choices.

Insufficient justification and controlled ablations. The paper introduces several design choices, including flat versus graph conditioning, direct mapping versus latent flow, and latent-alignment objectives. However, the experiments do not sufficiently isolate the contribution of each choice. In particular, the 3B experiment compares pipelines that differ not only in conditioning but also in chunk size, decoder, and training objective. Consequently, it is difficult to determine which component is responsible for the improvement. A more systematic ablation study would help explain when and why the proposed approach works.

We agree that the 3B comparison (Sec. 4.2.4) does not isolate conditioning alone, as the two pipelines differ across multiple components (App. F.5). To provide clean component-level comparisons, our in our additional experiments isolate design choices under strictly fixed pipelines across three settings:

  1. Direct Mapping vs. Latent Flow (MobileLLM Family Holdout): Compares direct and flow generation while holding the generator backbone, graph conditioning, canonical weight representation, and decoder identical (Reviewer P83q, Q4).
  2. Factorized 4×2×44 \times 2 \times 4 Grid (NNiT Zoo, G3): Systematically evaluates 44 weight representations ×\times 22 generator variants ×\times 44 training regimes.
  3. Mapper vs. Flow under Fixed Zoo & Decoder (G1, G2): Isolates deterministic mapping versus flow matching on identical GPT-2 and Llama zoos. he latent-alignment objective is ablated separately in Sec. 4.3.

Finally, our empirical sweep on wiki-text across zoo sizes indicates that cross-architecture extrapolation remains sensitive to the interplay between zoo coverage and the pretraining corpus we observed substantially different transfer gaps using the exact same model zoo when changing only the pretraining dataset. Pinpointing the exact mechanism behind this dataset sensitivity and establishing principled remedies remain open questions that we will discuss candidly in the revision.

Questions

Q1: Persistence of initialization gains, tokens/compute to matched loss, and amortized efficiency analysis.

How persistent are the benefits of generated initialization? What determines persistence, and can the authors report tokens or compute to matched validation loss and an amortized efficiency analysis?

Persistence Across Targets & Regimes: Persistence depends strongly on target scale relative to the source zoo and the pretraining corpus:

  • In-Range Target (GPT-small, 10B10\text{B} tokens): Flow maintains a lower training loss than random initialization across all ten checkpoints, though the gap narrows to 0.01000.0100 by 10B10\text{B} tokens (trajectories in Q2).
  • Extrapolated Unseen Target (U4, 5B5\text{B} tokens): On U4 an unseen architecture wider and larger than every source model, Flow maintains a more durable advantage at 5B5\text{B} tokens, remaining 0.06330.0633 below random initialization and 0.01040.0104 below HyperCloning.
  • WikiText / LoGAH Size-Transfer Protocol (G1): In the reassessed GPT-2 protocol, gains over random initialization are substantially larger even though the source zoo models (7.7M–28.4M7.7\text{M}–28.4\text{M} parameters) are far smaller than the targets (the smallest target, GPT-2 Small at 124M124\text{M}, is ∼4.4×\sim 4.4\times larger than the largest source model).

Tokens to Matched Performance (Proxy Metrics): Validation loss was not logged during these pretraining runs, so exact tokens-to-matched-validation-loss cannot be reported. Using the logged training loss and DCLM Core curves as proxies:

  • On U4 (5B5\text{B} budget): Flow surpasses random initialization's final DCLM Core score (0.11080.1108) at 2.0B2.0\text{B} tokens (40%40\% of the budget), and its smoothed training loss drops below random's final loss (3.02983.0298) by 4.5B4.5\text{B} tokens.
  • On GPT-small (10B10\text{B} budget): Both generated initializations reach random initialization's final DCLM Core score (0.08790.0879) by 7.0B7.0\text{B} tokens (70%70\% of the budget).

Amortized Efficiency Analysis: We agree that constructing a model zoo from scratch is expensive zoo creation and fine-tuning accounted for the majority of our experimental compute. However, the intended practical workflow for C2W is to leverage existing publicly pretrained checkpoints (on the Hugging Face Hub) rather than training a zoo from scratch; we trained custom zoos solely to conduct controlled studies on failure modes across wide architectural variations.

Excluding source-zoo construction, generator training is a one-time amortized cost of ∼2–4\sim 2–4 hours for the Mapper and ∼4–6\sim 4–6 hours for Flow on TPU v6e (Reviewer HWtn, W5).


Q2: Rationale for using DCLM Core in Figure 8, corresponding loss trajectories, and downstream benchmark breakdown.

Why is DCLM score used in Figure 8? Please report the corresponding validation losses and downstream results. Would the conclusion about persistence remain consistent across metrics?

Reviewer_HWtnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Config2Weights (C2W) learns architecture-conditioned initializations from collections of pretrained LLM weights or LoRA adapters. It represents heterogeneous tensors as fixed-size chunks with architectural, tensor-role, shape, and positional metadata. Weights are predicted through either a Mapper or flow matching. Experiments cover reconstruction, LoRA transfer, and full-model initialization. The results provide evidences for useful initialization, but the advantages over simpler initialization and model-reuse strategies remain insufficiently explained.

Critical strengths
  • The shared, architecture-conditioned initialization setting is good. Learning from heterogeneous LLM checkpoints to initialize unseen configurations is an good extension beyond architecture-specific parameter generation, with potential relevance to architecture exploration.
  • LoRA transfer provides a positive result. The Llama family is excluded from generator training. Flow matching improves final HellaSwag accuracy by 12.35, 4.05, and 1.68 percentage points on the 1B, 3B, and 8B targets, respectively, with similar gains from Mapper.
Critical weaknesses
  • Unclear novelty. p-diff [1], RPG [2] also support sophisticated weight generation. C2W should demonstrate advantage in cross-architecture transfer, rather than weight generation itself.

  • Insufficient baselines. Statistics-matched random initialization, disrupted cross-tensor correspondence, and stronger model-reuse baselines are missing. Besides, C2W should compare with p-diff and RPG.

  • Cross-architecture evidence is limited. Most gains come from LoRA transfer to pretrained models on a seen task. In full-model experiments, Flow even underperforms random initialization.

  • Experiment is unclear. The 3B experiment changes multiple design choices. U4 shows genuine extrapolation, but architectural differences are unclear and its gain over HyperCloning is only 0.0041.

  • Unclear efficiency. Generator-training costs are missing, so net computational savings cannot be assessed.

[1] Neural Network Diffusion, https://arxiv.org/abs/2402.13144

[2] Scaling Up Parameter Generation: A Recurrent Diffusion Approach, https://arxiv.org/abs/2501.11587

Questions for authors

See weaknesses.

5 comments

We thank the reviewer for suggesting p-diff and RPG. We completed a local RPG comparison (W2) and added two learned-generator baselines whose setting is closer to ours, LoGAH and NNiT (G1 and G3 in the general response).

Weaknesses

W1: Unclear novelty relative to prior weight-generation methods (p-diff [1], RPG [2]) and need to demonstrate cross-architecture transfer.

Unclear novelty: p-diff [1] and RPG [2] also support sophisticated weight generation. C2W should demonstrate an advantage in cross-architecture transfer.

We agree that our primary contribution lies in cross-architecture transfer, rather than weight generation in isolation. While p-diff and RPG are sophisticated methods, they focus on generating weights in isolation for a fixed architecture and fixed task. Accordingly, throughout the paper, we focus our evaluation on the more challenging cross-architecture transfer setting.

  • p-diff: Unconditional generation trained on multiple random-seed checkpoints of a single architecture on the same task, sampling from that specific model's latent weight manifold to improve performance on that same task.

  • RPG: Introduces conditioning (, class-conditioned sampling or checkpoint index), but the target architecture and dataset remain strictly fixed and the conditioning signal is unrelated to architectural structure; a separate generator is trained per dataset and per architecture.

In contrast, C2W conditions directly on the architectural configuration (alongside task embeddings if needed) to synthesize weights for novel architectures or task/datasets absent from the training zoo. Our cross-architecture transfer evidence spans five distinct evaluations:

  1. Held-out Llama family for LoRA generation (Sec. 4.1).
  2. Unseen U4 architecture, which is wider and larger than any source model (Sec. 4.2.5).
  3. GPT-2 and Llama size transfer across parameter scales (G1, G2).
  4. Eight unseen policy architectures on the NNiT robotic control zoo (G3).
  5. MobileLLM family holdout under a fixed pipeline (reported in our response to Reviewer P83q, Q4).

We will reframe the related work and positioning around learned cross-architecture weight generators to make this distinction explicit.


W2: Insufficient baselines — missing statistics-matched initializations, coordinate-disrupted controls, stronger model-reuse baselines, and comparisons with p-diff and RPG.

Insufficient baselines: statistics-matched random initialization, disrupted cross-tensor correspondence, stronger model-reuse baselines, and comparisons with p-diff and RPG are missing.

We have expanded our baseline suite across all requested axes:

1. Statistics-Matched and Shuffle Controls

In the LoRA setting, we added three controls built from ground-truth adapters: (i) a source-statistics-matched Gaussian, (ii) a within-tensor entry shuffle, and (iii) a norm- and spectrum-preserving shuffle. Across all four model–task settings, all three controls remain at base-model accuracy, whereas C2W Flow outperforms them by +2.48+2.48 to +45.76+45.76 percentage points, with 95%95\% question-paired bootstrap intervals excluding zero (full tables in our response to Reviewer P83q, Q3).

2. Stronger Model-Reuse Baselines

In the MobileLLM family holdout (Reviewer P83q, Q4), we include a closest-source weight-copy reference. While C2W improves substantially over random initialization, direct weight copying from the closest source checkpoint remains stronger (3.4903.490 vs. 3.6803.680 final test cross-entropy for C2W Flow).

3. Local Comparison with RPG (LLaMA-7B Rank-4 DoRA)

We conducted a local comparison in the rank-44 LLaMA-7B DoRA setting from RPG's Table 3, using the exact same 5050 source checkpoints from a single Commonsense-170K fine-tuning trajectory:

  • Architectural & Algorithmic Differences: RPG generates normalized parameter chunks via a diffusion denoiser (1,0001,000 DDPM steps) using recurrent Mamba features over tensor/chunk positions and checkpoint identity (950.81M950.81\text{M} parameters). C2W encodes complete adapters via a full-width CVAE (4.55M4.55\text{M} parameters, prioritizing reconstruction over compression), models graph connections across factors, modules, and layers, and generates structured latents via rectified flow (24.06M24.06\text{M} parameters; 6464 Heun steps), decoding separate rank-44 A/BA/B factors and DoRA magnitudes (plus training-derived normalization arrays).
  • Conditioning Variants: Because all 5050 source checkpoints share a single architecture and training corpus, a static task label cannot distinguish checkpoints. We therefore evaluate three conditioning modes (fixing checkpoint index i=2i = 2 prior to evaluation to match RPG's sampled checkpoint):
    • Structure: Architectural metadata alone.
    • Checkpoint ID: Structural metadata + an RPG-style checkpoint-index embedding.
    • Progress: Structural metadata + normalized trajectory position p=i/49p = i/49 encoded via sine/cosine features.
  • Zero-Shot Benchmark Results (21,152 total examples across 7 benchmarks): (Accuracy in %; evaluated without downstream fine-tuning using identical prompts, decoding, and scoring. Bold indicates our best generator.)
MethodBoolQPIQASIQAHellaSwagARC-EARC-COBQAMean
Original adapter68.8481.1878.6184.5180.9864.3378.0076.63
RPG + BN calibration68.7881.3978.0585.0680.7263.9178.0076.56
Source-mean control68.7581.3478.4084.8880.7264.0878.4076.65
CVAE reconstruction68.8781.3478.5684.3280.9864.1678.6076.69
C2W: structure flow68.6581.2378.4585.0280.8163.9178.0076.58
C2W: checkpoint-ID flow68.4481.1278.2084.8880.7764.0877.6076.44
C2W: progress flow68.6281.2878.5184.9780.8163.9178.4076.64
C2W: deterministic mapper68.4781.1878.3084.9180.8964.0878.2076.58
  • Compute Comparison (TPU chip-hours, excluding source-zoo creation):

Against RPG on the 4 rank LoRA it took approximately 22hours to pretrained the RPG generators on 2 chips v6-e TPU sever. while it took our method approximately 4 to5 hours to pretrained the CVAE and the a generative model o top. | The ∼4.4×\sim 4.4\times lower compute reflects different training schedules—20,00020{,}000 flow updates vs. 80,00180{,}001 RPG updatesrather than a matched-budget speedup.*

  • Interpretation: Local RPG measurements required post-hoc BatchNorm recalibration on source weights. C2W flow` achieves parity with both RPG (76.64%76.64\% vs. 76.56%76.56\%) and the ground-truth adapter (76.63%76.63\%) using ∼33×\sim 33\times fewer generator parameters (28.61M28.61\text{M} vs. 950.81M950.81\text{M}). However, in this single-trajectory setting, neither C2W nor RPG outperforms the simple source-mean control (76.65%76.65\%), and single-seed differences do not establish statistical superiority—confirming that single-architecture trajectory benchmarks do not test cross-architecture or cross-task transfer.

4. Why LoGAH and NNiT Are More Direct Baselines (and Note on p-diff)

Because p-diff resamples single-model checkpoints and RPG conditions on checkpoint states for a fixed task and architecture, neither is designed for unseen architectures or tasks. In contrast, LoGAH (graph hypernetworks) and NNiT (ICML 2026) directly target cross-architecture weight synthesis:

  • Against LoGAH (GPT-2 size transfer, G1): C2W achieves 115.8–131.0115.8–131.0 test perplexity versus 454.3–552.2454.3–552.2 for local LoGAH implementations (on TPU).
  • On the NNiT zoo (G3): C2W synthesizes policies across eight unseen architectures, achieving 99.55%/99.40%/79.90%99.55\% / 99.40\% / 79.90\% average success rates.

We did not run p-diff because reproducing the RPG baseline alone required ∼1\sim 1 day per task, and we prioritized baselines aligned with cross-architecture transfer additionally RPG is improved version of p-diff and ours method already outperformed RPG.

W3: Limited cross-architecture evidence , most gains come from LoRA transfer on seen tasks, and Flow underperforms random initialization in full-model experiments.

Cross-architecture evidence is limited: most gains come from LoRA transfer to pretrained models on a seen task. In full-model experiments, Flow even underperforms random initialization.

We agree with this nuanced assessment. For in-architecture resampling, generation works reliably and yields weights that perform on par with or better than ground-truth adapters. For out-of-distribution (OOD) architectures, performance remains uneven, for example, in the 3B experiment, the flat-conditioned Flow does not match random initialization, and generalizing to unseen LoRA target architectures proves just as challenging as generalizing to unseen full models.

However, our expanded experiments demonstrate that cross-architecture transfer succeeds across multiple settings, governed largely by the target training corpus and source-zoo size and composition:

  • Zero-Shot Unseen Policy Architectures (G3): On compact robotic policy networks, generated weights for unseen architectures function immediately without any downstream training.
  • GPT-2 and Llama Size Transfer (G1, G2): C2W consistently provides stronger initializations than both random initialization and our local LoGAH implementations.
  • MobileLLM Family Holdout (Reviewer P83q, Q4): Across three seeds under a fixed pipeline, C2W Flow reliably outperforms random initialization (3.680±0.0043.680 \pm 0.004 vs. 3.717±0.0043.717 \pm 0.004 test CE; 42.28%42.28\% vs. 40.48%40.48\% downstream accuracy), whereas unmodified direct generation does not.

We will revise the discussion to clearly delineate where cross-architecture transfer succeeds versus where OOD scaling remains an open challenge.


W4: Confounded 3B comparison and small margin / unclear architectural specifications on U4.

The 3B experiment changes multiple design choices. U4 shows genuine extrapolation, but its architectural differences are unclear and its gain over HyperCloning is only 0.0041.

  • 3B Experiment Positioning (Sec. 4.2.4): We agree that the 3B comparison is confounded—the two pipelines differ in conditioning, chunk size, decoder, and objective (as noted in App. F.5), and the flat-conditioned Flow struggles at that scale. We will explicitly reframe this result as a scalability/feasibility demonstration for the graph-conditioned pipeline rather than a controlled ablation.

  • U4 Architectural Extrapolation & Margin: U4 comprises 1212 layers, hidden width 15361536, 2424 attention heads, and 418.7M418.7\text{M} parameters, whereas the eight source models span 7–147–14 layers, widths 704–1024704–1024, and 89.7–203.7M89.7–203.7\text{M} parameters (Table 11) making U4 wider and more than 2×2\times larger than any source checkpoint. We agree that the DCLM benchmark margin over HyperCloning (0.13700.1370 vs. 0.13290.1329) is modest and single-seed. The recovered training loss at 5B5\text{B} tokens exhibits the same ordering Flow (2.96662.9666) << HyperCloning (2.97702.9770) << Random (3.02983.0298) (full trajectories in our response to Reviewer YVKW, Q2). Accordingly, we claim parity with HyperCloning on U4 rather than superiority, noting that C2W achieves this without handcrafted source-to-target weight transformations.

  • Role of Zoo Composition & Pretraining Data: Our LoGAH-protocol experiments (G1, G2) indicate that initialization gains depend strongly on the pretraining text distribution and source-zoo composition. While the WikiText experiments show that useful initializations can be synthesized from substantially smaller source models, our broader evaluations show this does not hold uniformly across all datasets; principled zoo design and sizing remain open research directions.


W5: Missing generator-training costs and unclear net computational efficiency.

Unclear efficiency: generator-training costs are missing, so net computational savings cannot be assessed.

We agree that reporting generator-training overhead is essential for assessing net efficiency, and we will add a dedicated cost breakdown table to the revised paper: First we will provided the training cost of the additional experiment against the baselines. in General the training cost is due by the bootleneck in the CVAE. in the additional experiment to demonstrate we can speed up the training we use bottleneck free VAE against RPG where the training set contains 50 pretrained model LoRAs.

  • Generator Training Time: Training the deterministic mapping generator requires ∼2–3\sim 2–3 hours, and training the Flow model requires ∼4–6\sim 4–6 hours on TPU v6e (with comparable or faster runtimes on H100 GPUs depending on system load). This is a one-time amortized cost, as a single trained generator can initialize multiple target architectures.
  • Inference / Weight Generation Time: Synthesizing a full checkpoint takes ∼3–5\sim 3–5 minutes with Flow and under 6060 seconds for Llama-3.2-3B using the graph pipeline (App. A).
  • End-to-End RPG Comparison:
    | Method | CVAE training | Generator training | Total training | |---|---:|---:|---:| | Rank-4 RPG | — | ≈22 h 31 min | ≈22 h 31 min | | CVAE + mapper | 7 min 38 s | 3 h 31 min | 3 h 38 min | | CVAE + flow | 7 min 38 s | 3 h 8 min | 3 h 15 min |

References

  • [1] Wang et al., Neural Network Diffusion, arXiv:2402.13144 (2024).
  • [2] Wang et al., Scaling Up Parameter Generation: A Recurrent Diffusion Approach, arXiv:2501.11587 (2025).
Reviewer_P83qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Config2Weight, a framework for generating network initialization for LLM from the given architectural configurations. The authors develop a unified representation of heterogeneous weights that places positional chunks in weight spaces. They also conduct cross-architecture evaluation from adapters to full models, demonstrating improvements in early pre-training against random initialization.

Critical strengths

The paper explores an interesting problem of architecture-conditioned weight generation, which can potentially benefit LLM pre-training costs. The proposed chunk-based representation and metadata architectural conditioning enable weight generation across heterogeneous model architectures. The paper evaluates both LoRA adapters and full-model weights, showing improvement on training efficiency over random initialization in several settings.

Critical weaknesses

The most critical concern is the assumption of the setup. This paper assumes that LLM weights can be merely determined on architectural schema. This is highly questionable, since model weights are also strongly affected by training data, optimization schedule, and initialization, besides their architecture. In Sec 4.1, the model is trained to generate LoRA adapters across various families and datasets, but how can task-specific weights be generated without task conditioning, given that metadata u_i only consists of architecture information? The limited number of independent checkpoints also raises concerns about the generator's generation capability. Table 5 shows that the generator are trained with chunks derived from only 22 checkpoints. This means the pretrained weights are highly entangled with the architecture. It is not clear whether the generator really learns generalizable arch-conditioned weight distributions or merely memorizes some checkpoint-specific patterns. An analysis should be provided to support the main claim. The full-model experiments (Sec. 4.2) primarily compare C2W against random initialization and two model morphisms. While these experiments demonstrate that generated initialization can improve early training, they do not establish whether gains come from learning transferable knowledge from pretrained weights. In Sec 4.2.2, the generator shows comparable performance with Gstack and HyperCloning, but the target models are expansions of architectures already represented in the source zoo. This cannot demonstrate generalization to fundamentally unseen architectures (from totally different families). In Figure 6, the held-out MobileLLM experiment shows improvements on ARC-Easy and MMLU, but worse or near performance on ARC-Challenge. Also, it is worth noting that models with random init exhibit almost no improvement during training on all three downstream tasks. This raises concerns about whether those baselines are well-tuned.

Questions for authors
  1. Task conditioning and the learning objective. Section 3.2 defines the conditioning metadata uiu_i using architectural information, while Section 4.1 trains on adapters from 15 datasets. How does the generator distinguish adapters trained for different tasks on the same architecture, and what specifically makes a generated adapter a “HellaSwag adapter”? Is task identity encoded implicitly, are separate generators trained for different tasks, or is another selection mechanism used? If no task information is provided, should the output instead be interpreted as a task-agnostic initialization sampled from a mixture of task-specific weight distributions?

  2. Generalization versus checkpoint memorization. Although the LoRA zoo contains 330 adapters, these are derived from only 22 base-model checkpoints, and chunks from the same checkpoint are strongly correlated. What evidence distinguishes learning a transferable architecture-conditioned weight distribution from memorizing checkpoint-specific patterns? Could the authors provide controlled experiments with multiple independently trained checkpoints of the same architecture, varying training data or random seeds, and evaluate on held-out checkpoints? An ablation varying the number of source checkpoints, rather than the number of chunks, would also help assess this concern.

  3. Source of the initialization gains. The comparisons in Section 4.2 show that C2W can provide a useful initialization, but do not isolate whether it transfers learned knowledge or primarily reproduces favorable weight statistics. Could the authors compare against initializations that match source weights’ per-layer statistics, or controls that shuffle weights within tensors while preserving their marginal distributions? Would C2W retain its advantage over these controls under the same training budget?

  4. Scope of architectural generalization. The targets in Section 4.2.2 are expansions of source models, and the held-out MobileLLM configuration in Section 4.2.3 has a same-family checkpoint in the training zoo. Although Section 4.2.4 includes a held-out Llama-3.2-3B target, its comparison also changes the conditioning mechanism, chunk size, decoder, and objective. Could the authors provide a controlled full-model evaluation holding out entire model families while keeping the generation pipeline fixed? How does performance change as the target becomes structurally more distant from the source architectures?

  5. Baseline tuning and robustness of downstream gains. Table 13 reports identical mean ARC-Challenge accuracy for C2W and random initialization, while both remain close to chance on MMLU. Appendix F.4 uses identical optimization settings for both initializations, but were learning rates and warmup schedules tuned separately with comparable search budgets? Do the downstream advantages persist against a tuned random-initialization baseline and with longer training, and which gains are statistically reliable across seeds?

7 comments

We thank the reviewer for the detailed review. We ran three experiments in direct response: task-identification and task-swap checks (W2, Q1–Q2), source-statistics and shuffle controls (Q3), and a fixed-pipeline MobileLLM family holdout (Q4). G1–G3 refer to the general response

Weaknesses

W1: Assuming weights are uniquely determined by architecture.

The setup assumes LLM weights are determined by the architectural schema, although they also depend on training data, optimization schedule, and initialization.

We agree that weights are a function of the training data, random seed, optimizer, and compute stack, and we do not claim that architecture alone determines them. In our setting, we use one checkpoint per model, so the configuration serves as a unique identifier for that checkpoint. The question we study is whether a distributional relationship can be learned between configuration embeddings and their pretrained weights—regardless of dataset and optimization setup—so that generated weights for a novel configuration converge faster or require no training. This is why the paper pairs a deterministic Mapper with a Flow model: the Mapper provides a point estimate, while the Flow models the variability that the configuration does not explain (Sec. 3.4). We will revise the problem statement to make this explicit.


W2: Task conditioning and sample size in LoRA generation.

How can task-specific LoRA weights be generated without task conditioning, given that uiu_i only contains architecture information? The limited number of independent checkpoints also raises concerns.

We apologize for the missing details in the text. The LoRA generators are task-conditioned: in addition to architectural metadata, they receive a learned 512-dimensional task-ID embedding and a text embedding of the task description. No separate generator is trained per task. Furthermore, the "22 checkpoints" count is per task: the model zoo contains 22 backbones×15 tasks=330 adapters22 \text{ backbones} \times 15 \text{ tasks} = 330 \text{ adapters}, meaning each task is represented by 22 adapters. We will add the task-conditioning details to Sec. 3.2 and Sec. 4.1 and correct the statement claiming conditioning is purely architectural. Evidence demonstrating that the task condition is actively utilized is provided in our responses to Q1–Q2.


W3: Potential memorization due to limited checkpoint count (Table 5).

Table 5: chunks come from only 22 checkpoints, so it is unclear whether the generator learns generalizable architecture-conditioned distributions or memorizes checkpoint-specific patterns.

Please see Q2 for our task-identification and task-swap analyses, and Q3 for controls that separate coordinate structure from weight statistics. We agree that the in-distribution LoRA results alone cannot rule out memorization. Rather, evidence for generalization comes from our held-out targets: the Llama family in Sec. 4.1, the MobileLLM family holdout in Q4, and the unseen architectures in G1–G3.


W4: Attribution of gains to transferable knowledge vs. general initialization effects (Sec. 4.2).

The full-model experiments (Sec. 4.2) primarily compare C2W against random initialization and two model morphisms. While these experiments demonstrate that generated initialization can improve early training, they do not establish whether gains come from learning transferable knowledge from pretrained weights.

We agree that Sec. 4.2 alone does not definitively isolate knowledge transfer. The strongest evidence we currently have includes:

  1. LoRA controls (Q3): Generated adapters outperform statistics-matched baselines.
  2. MobileLLM family holdout (Q4): C2W-Flow improves over random initialization, though ablations indicate that much of the benefit stems from the generated embeddings.
  3. Zero-shot robotic policies (G3): On unseen robotic-policy architectures, generated weights are immediately functional without any training.
  4. Additional WikiText evaluation: In our new WikiText experiments, the proposed method achieves the best performance on unseen architectures.

We have not yet run statistics-matched controls on the full-model experiments and will explicitly state this as a limitation in the revision.


W5: Generalization to fundamentally unseen architectures (Sec. 4.2.2).

In Sec 4.2.2, the generator shows comparable performance with Gstack and HyperCloning, but the target models are expansions of architectures already represented in the source zoo. This cannot demonstrate generalization to fundamentally unseen architectures (from totally different families).

We agree, and Sec. 4.2.2 notes that generated weights may lie close to those of structurally neighboring sources. That experiment is specifically intended to show that a single generator can handle both expansion directions without handcrafted transformations—not to serve as evidence for family-level generalization. For cross-family generalization, please refer to:

  • The new MobileLLM family holdout with a fixed pipeline (Q4),
  • U4 in Sec. 4.2.5, which belongs to an unseen family and is wider and larger than every source model, and
  • The size-transfer results in G1–G2.

W6: Non-uniform downstream gains and baseline tuning (Figure 6).

In Figure 6, the held-out MobileLLM experiment shows improvements on ARC-Easy and MMLU, but worse or near performance on ARC-Challenge. Also, it is worth noting that models with random init exhibit almost no improvement during training on all three downstream tasks. This raises concerns about whether those baselines are well-tuned.

We agree that the downstream gains in Figure 6 are not uniform across benchmarks. Our three-seed replication (Table 13) confirms this pattern:

  • ARC-Easy: Higher for C2W in mean (35.41±1.3035.41 \pm 1.30 vs. 33.61±0.5033.61 \pm 0.50).
  • ARC-Challenge: Identical in mean (16.8116.81).
  • MMLU: Remains near random chance for both (25.6425.64 vs. 25.2825.28).

Both initializations use identical hyperparameters and data ordering (App. F.4). Because a 140M-parameter model trained on 1B tokens remains near chance on MMLU and ARC-Challenge, there is limited headroom for any initialization method to show measurable separation on those tasks. Please see Q5 for further details on baseline tuning.

Questions

Q1: Task conditioning and the learning objective : how does the generator distinguish adapters for different tasks on the same architecture?

Task identity is explicitly provided via a learned 512-dimensional task-ID embedding alongside a text embedding of the task description (see W2). Consequently, the generator outputs a task-specific adapter rather than sampling from a task-agnostic mixture.

The task-swap analysis below confirms that this conditioning is actively utilized: using the same generator, conditioning on HellaSwag achieves the highest HellaSwag accuracy, conditioning on an unrelated task degrades performance, and shuffling the generated weights destroys the learned benefit entirely.

Adapter Configuration (Evaluated on HellaSwag)Accuracy
No adapter (base model)24.22%
Direct mapper conditioned on HellaSwag50.09%
Same generator conditioned on ARC40.62%
Same generator conditioned on PIQA39.84%
Shuffled HellaSwag-generated weights21.88%
Original trained HellaSwag adapter54.69%

Q2a: Generalization versus memorization — the 330 adapters derive from only 22 base checkpoints, and chunks from the same checkpoint are strongly correlated.

The count of 22 refers to the number of distinct backbone adapters available per task in the training set (22 backbones×15 tasks=330 adapters22 \text{ backbones} \times 15 \text{ tasks} = 330 \text{ adapters}). In our chunk-based formulation, we do not treat chunks from the same checkpoint as independent samples; instead, each chunk is uniquely indexed by its chunk position alongside its task, layer, and architectural configuration, and the generator is conditioned on this full identifier.


Q2b: What evidence distinguishes a transferable architecture-conditioned weight distribution from memorized checkpoint-specific patterns?

In weight space, generated adapters exhibit strong task specificity. Across all 22 models×15 tasks22 \text{ models} \times 15 \text{ tasks} (330330 total) and for each generator variant, we compared the generated weight update against all 1515 ground-truth task adapters on the corresponding backbone:

GeneratorClosest Original Adapter Has Correct TaskMean Cosine (Correct Task)Mean Cosine (Closest Wrong Task)
Chunk direct mapping329 / 330 (99.70%)0.8360.464
Graph-based mapping329 / 330 (99.70%)0.8280.471
Chunk-based flow329 / 330 (99.70%)0.8300.468

Chance-level task identification is 6.67%6.67\%.

While this weight-space task identification is an in-distribution test (demonstrating that generation cleanly separates training tasks rather than proving cross-task transfer on its own), generalization to unseen architectures is supported by our held-out Llama experiments in Sec. 4.1 and the small size/family transfer results in G1–G3 (global responses)

Q2c: Could the authors provide controlled experiments with multiple independently trained checkpoints of the same architecture, and evaluate on held-out checkpoints?

The current LoRA zoo partially addresses multi-checkpoint variation per architecture—each of the 22 architectures has 15 distinct adapters trained across different tasks—though it contains one adapter per (model, task) pair rather than seed-level replicates. (For multi-seed checkpoint density, the NNiT zoo in G3 includes up to 100 policies per architecture and is evaluated on held-out architectures.) We prioritized evaluating cross-family generalization first, and will add task-conditioned generation for unseen tasks to complement these results.

The table below compares ground-truth adapters against adapters sampled from our task- and architecture-conditioned flow (Flow) on full held-out test/validation splits without any additional fine-tuning (mean ±\pm SD across 3 sampling seeds):

ModelTaskOriginal AdapterFlow Sampled
Qwen3-0.6BHellaSwag55.19%47.83%±0.1247.83\% \pm 0.12
Qwen3-0.6BARC-Challenge53.50%54.89%±0.3854.89\% \pm 0.38
Qwen3-0.6BPIQA64.91%65.36%±0.4165.36\% \pm 0.41
SmolLM2-1.7BHellaSwag60.71%48.33%±0.0748.33\% \pm 0.07
SmolLM2-1.7BARC-Challenge54.61%55.97%±0.1555.97\% \pm 0.15
SmolLM2-1.7BPIQA71.82%71.98%±0.1471.98\% \pm 0.14
Qwen3-14BHellaSwag93.26%91.08%∗91.08\%^*
Qwen3-14BARC-Challenge92.15%92.43%±0.1092.43\% \pm 0.10
Qwen3-14BPIQA89.72%89.64%±0.0689.64\% \pm 0.06

Sampled adapters match or slightly exceed the original adapters on ARC-Challenge and PIQA across all three backbones (within −0.08-0.08 to +1.39+1.39 percentage points), while trailing on HellaSwag (−2.18-2.18 to −12.38-12.38 points), indicating that sampling fidelity varies by task.


Q2d: An ablation varying the number of source checkpoints, rather than chunks, would help.

We appreciate this suggestion and are actively running an ablation scaling the number of source models. Preliminary trends indicate that a smaller number of source models can yield strong in-distribution sampling quality, whereas out-of-distribution generalization benefits from greater source diversity. We will include the finalized numbers in the revised manuscript.


Q3: Source of the initialization gains — compare against initializations that match per-layer source statistics, or shuffle weights within tensors while preserving marginals.

We agree that Sec. 4.2 does not isolate knowledge transfer from favorable weight statistics. To directly test this, we conducted a controlled LoRA study across two backbones (Qwen3-0.6B, Qwen3-14B) and two tasks (ARC-Challenge, PIQA). All hyperparameters—pretrained backbone, adapter rank (r=8r=8), scaling, target modules, prompts, and evaluation splits (1,1721,172 ARC-Challenge test questions; 1,8381,838 PIQA validation questions) are held strictly fixed. Adapters are evaluated **zero-shot on test set ** across three seeds (11, 42, 123).

We compare Flow against three statistics-preserving controls constructed from each corresponding ground-truth adapter:

  1. Source-statistics-matched Gaussian: Each AA and BB tensor is sampled independently from a Gaussian and rescaled to match the source tensor's exact empirical mean and population standard deviation.
  2. Within-tensor entry shuffle: All entries within each AA and BB tensor are independently permuted, preserving exact per-tensor marginal distributions while breaking coordinate correspondence.
  3. Norm- and spectrum-preserving shuffle: The input columns of AA and output rows of BB are independently permuted (ΔW′=PoutΔWPin\Delta W' = P_{\text{out}} \Delta W P_{\text{in}}). This preserves A/BA/B marginals, the update's entry marginal, Frobenius norm, and singular value spectrum, altering only alignment with the frozen backbone.

1. Zero-Shot Held-In Accuracy (%)

(Mean ±\pm sample SD over 3 seeds; Base and Original Adapter are deterministic references.)

Model / TaskBaseOriginal AdapterFlowMatched GaussianEntry ShuffleNorm/Spectrum Shuffle
Qwen3-0.6B / ARC-Challenge32.8553.5054.89±0.3854.89 \pm 0.3833.22±0.3933.22 \pm 0.3932.65±0.3832.65 \pm 0.3833.05±0.2633.05 \pm 0.26
Qwen3-0.6B / PIQA62.4064.9165.36±0.4165.36 \pm 0.4162.66±0.2262.66 \pm 0.2262.53±0.2562.53 \pm 0.2562.88±0.1462.88 \pm 0.14
Qwen3-14B / ARC-Challenge46.7692.1592.43±0.1092.43 \pm 0.1046.67±0.1546.67 \pm 0.1546.81±0.2646.81 \pm 0.2646.70±0.1346.70 \pm 0.13
Qwen3-14B / PIQA81.6689.7289.64±0.0689.64 \pm 0.0681.90±0.2381.90 \pm 0.2381.88±0.2281.88 \pm 0.2281.77±0.0981.77 \pm 0.09

Comments on the paper

4 comments

General response to all reviewers

We thank the reviewers for their careful and constructive reviews. The reviews converge on four concerns: (i) what the generator is conditioned on and whether it memorizes, (ii) missing learned-generator and statistics-matched baselines, (iii) limited evidence for generalization across model families, and (iv) whether the gains persist. We ran seven additional experiments. Three are shared by all reviewers and are reported here (G1–G3); the other four are reported in the response to the reviewer who requested them.

New experimentReported inMain outcome
G1. GPT-2 size transfer vs. LoGAHGeneral responseC2W reaches 115.8–131.0 test perplexity vs. 454.3–552.2 for LoGAH; not data/compute-matched
G2. Llama size transfer (140M–3B)General responseC2W lowers perplexity vs. random init at all three sizes; single seed
G3. Robotic policies on the NNiT zooGeneral response99.55% / 99.40% / 79.90% average success on eight unseen architectures
Task-identification and task-swap checksReviewer P83q329/330 generated adapters are closest to the correct-task original
Source-statistics and shuffle controlsReviewer P83qFlow v3 beats all three controls in all four settings
MobileLLM family holdout, fixed pipelineReviewer P83qFlow beats random init; unmodified direct does not; weight copy is strongest
RPG comparison (LLaMA-7B DoRA)Reviewer HWtn76.64% vs. 76.56% (RPG) with about 4.4× less measured compute
Training loss and per-benchmark results for Figure 8Reviewer YVKWNear convergence on GPT-small; advantage remains on U4 at 5B tokens

We also state the limits of the current evidence plainly. Full-model gains are modest and depend on the generation variant, the pretraining data, and the zoo composition. Weight copying from a close source remains stronger where a close source exists. Structural-distance dependence is not yet characterized. We will revise the claims in the paper accordingly and reposition the work relative to learned weight generators (LoGAH, NNiT).

G1. GPT-2 size transfer against LoGAH

We compared C2W with random initialization and local LoGAH-T/S implementations (Zhou et al., 2024) [1]. Our zoo contains 81 GPT-2-style architectures with 7.7–28.4M parameters, 3–9 layers, hidden widths 72–256, and eight attention heads, all trained on WikiText-103. 73 architectures are used for generator training and eight for monitoring. Additional functional training used synthetic architectures up to 12 layers and width 512. The targets are GPT-2 Small, Medium, and Large; the smallest target (124M) is about 4.4× larger than the largest zoo model.

All methods use the same 100-epoch WikiText-2 fine-tuning schedule. We report test perplexity at the checkpoint selected by validation perplexity, as mean ± SD over two fine-tuning seeds (lower is better).

InitializationSmall, 124MMedium, 355MLarge, 774M
Random644.2 ± 2.1779.0 ± 4.5808.7 ± 4.6
LoGAH-T552.2 ± 0.1471.2 ± 3.5454.3 ± 2.8
LoGAH-S513.3 ± 0.7479.2 ± 1.6466.2 ± 2.4
C2W direct .131.0 ± 1.0120.8 ± 1.1115.8 ± 0.4
C2W flow138.1 ± 0.3122.0 ± 1.9119.9 ± 0.3

These results support within-family size transfer across the GPT-2 family. Two caveats apply. C2W benefits from additional WikiText-103 zoo supervision, and our local LoGAH results differ from the published ones, so this is not a data- or compute-matched reproduction. Validation-based checkpoint selection also matters: C2W direct reaches 115.8 on GPT-2 Large at its selected checkpoint, compared with 161.8 at epoch 100.

G2. Llama size transfer from a small zoo

The source zoo contains 59 Llama-style models with 8.4–27.6M parameters, 3–9 layers, and hidden widths 64–192. Each was trained for approximately 100M token positions by repeating the WikiText-2 training split, and all 59 are used to train the generator. A graph-conditioned CVAE with a frozen decoder supports two latent generators, a deterministic mapper and a flow model. They initialize unseen targets of 139.7M, 1.236B, and 3.213B parameters, up to 116.5× larger than the largest source model. Generation uses only the target configuration. Every target parameter is then trained for 100 epochs on WikiText-2.

TargetRandom init.C2W mappingC2W flow
140M661.64365.26274.08
1B414.68286.48252.32
3B547.99259.85259.70

Values are test perplexities at validation-selected checkpoints (lower is better). These are single-seed results, and the mapping–flow difference at 3B is negligible. Random and mapping use the same peak learning rate, 3×10⁻⁶. Flow uses 10⁻⁵, 6×10⁻⁶, and 3×10⁻⁶ for the three sizes, so its gains on the two smaller targets combine initialization and learning-rate changes. This is within-family size transfer; it is neither a Llama LoGAH comparison nor a family-holdout test. Absolute perplexities are not comparable across the GPT-2 and Llama tokenizers.

G3. Robotic policies on the NNiT zoo

We ran a new experiment on the NNiT pretrained zoo (Kim et al., 2026) [2], covering 81 architectures across three ManiSkill tasks.

TaskPoliciesPolicies / arch.Params / policy
PickCube-v16,4001002,568–15,752
PushCube-v16,4001002,456–15,304
StackCubeEasy-v15,27234–1002,664–16,136

Eight architectures with 1,928–10,040 parameters are held out and never used during generator training. We trained 32 generator checkpoints: four weight representations (non-pooled layer chunks, layer-pooled, graph, whole-architecture) × two generators (latent flow, deterministic configuration-to-latent mapping) × four training settings (Pick, Push, Stack, and joint task-conditioned training). The joint model combines all three zoos and conditions explicitly on the task. All generators use frozen pretrained autoencoders.

Success rates (%) of generated policies on the eight unseen architectures, without fine-tuning:

MethodMetricPickCubePushCubeStackCubeEasy
C2W, shared layer-pooled mapper (best)Average99.5599.4079.90
C2W, shared layer-pooled mapper (best)Top-1010010095.80
NNiT, our local reproductionAverage16.9578.802.60
NNiT, publishedTop-109910086

Our local NNiT reproduction underperforms the published implementation. Published top-10 scores are also not directly comparable to our all-policy averages. We therefore do not claim controlled superiority over NNiT. The result shows that a single task-conditioned C2W generator produces functional weights for unseen architectures in a second domain.

References

[1] Xinyu Zhou, Boris Knyazev, Alexia Jolicoeur-Martineau, and Jie Fu. LoGAH: Predicting 774-Million-Parameter Transformers using Graph HyperNetworks with 1/100 Parameters. arXiv:2405.16287, 2024.

[2] Jiwoo Kim, Swarajh Mehta, Hao-Lun Hsu, Hyunwoo Ryu, Yudong Liu, and Miroslav Pajic. NNiT: Width-Agnostic Neural Network Generation with Structurally Aligned Weight Spaces. ICML 2026, PMLR 306:58529–58548.

Additional baseline experiment: shared conditional LoRA generation

1. Architecture

We train a single architecture-conditioned generator that outputs complete LoRA adapters for two Qwen2.5 backbones from unlabeled task prompts, Whole-adapter representation. One sample contains all LoRA A/B factors across all layers and target projections (q, k, v, o, gate, up, down): 4,399,104 parameters for the 0.5B backbone and 18,464,768 for the 1.5B backbone. Factors are normalized and tiled into 16128 patches. Each patch becomes one latent token of 2,050 coordinates (2,048 weights plus 2 normalization statistics), giving 4,2962,050 and 9,0162,050 latents for the two backbones.

Shared CVAE codec (441M parameters). A single CVAE shared by both backbones, maps adapters to and from this latent space. It is a full-width residual Encoder with no dimensional bottleneck, and it is frozen during generator training.

Prompt- and architecture-conditioned generators. We use 4 transformer blocks with hidden width 2,304 and 18 attention heads. Self-attention runs over every adapter patch, with layer / projection / factor / position metadata identifying the architecture. Cross-attention reads token-level MiniLM-L12-v2 embeddings of the task prompts.

  • Direct mapper (574M): predicts the adapter from condition latent in one forward pass.
  • Flow (584M): starting from the condition latent generate adapter latent z from zt=(1-t)+tz and generates with 32 integration steps.

The decoded latent gives the LoRA factors (A,B) for the requested backbone.

2. Training data

Both generators are trained jointly on all domains and both backbones (20,000 updates each).

Table 1. Source adapter corpus used to train our generators.

DomainBackboneTasksCheckpointsRank r
CommonsenseQwen2.5-0.5B-Instruct6610816
MathematicsQwen2.5-1.5B-Instruct66491632
CodingQwen2.5-1.5B-Instruct77111632
Total191,970

ARC-c, GSM8K, and HumanEval are excluded from generator training: the 101 ARC-c

3. Results

Baseline. We compare against Drag-and-Drop LLMs (DnD; Liang et al., 2025), the zero-shot prompt-to-weights generator. DnD uses a separate generator for each domain and backbone: the paper configures and trains its generators per setting (common sense on Qwen2.5-0.5B, math on Qwen2.5-1.5B, coding on Qwen2.5-1.5B), and each generator's output is fixed to one backbone's LoRA layout. Our comparison therefore uses three independently trained DnD generators, one per benchmark. We run the authors' released generators locally with the same base model, tokenizer, rank/scaling, target-task prompt indices, and scorer as our methods, so the DnD numbers are a local reproduction and not copied from the published table.

One generator vs. three. In contrast, every result reported for our method comes from a single generator, trained once, jointly on all three domains and both backbone architectures . The same weights generator produces the 4.4M-parameter adapter for Qwen2.5-0.5B and the 18.5M-parameter adapter for Qwen2.5-1.5B; the target architecture is supplied only as a conditioning input.

DnDOurs
Generators used for ARC-c / GSM8K / HumanEval3 (one per domain and backbone)1 (shared)
Trainingseparate, per domainjoint, all 19 tasks
Backbone handlingfixed per generatorconditioning input

Update coefficient c. A generated adapter (A,B) with rank r and scaling is applied as

W'=Wbase+crBA,

which is equivalent to rescaling B'=cB while keeping A, r, and unchanged. c=1 is the standard LoRA update and is used for ARC-c and GSM8K. For HumanEval we use c=-3 for both of our methods, i.e. W'=Wbase-3rBA. This value was selected on HumanEval using two mapper adapters and transferred unchanged to flow; no generator or backbone is retrained. DnD adapters are applied as released.

Table 2. Results on tasks held out from generator training (%).

Methodc (ARC-c / GSM8K / HumanEval)ARC-c accuracyGSM8K accuracyHumanEval pass@1
DnD (Liang et al., 2025), released generators1/1/140.85 ± 6.4462.53 ± 1.6626.93 ± 3.76
Ours: direct mapper1 / 1 / −350.12 ± 1.3462.82 ± 1.6224.89 ± 3.73
Ours: flow1 / 1 / −350.37 ± 1.2463.57 ± 1.5525.89 ± 3.28

Mean ± population standard deviation over 10 generated adapters per method and task, each from a different conditioning prompt batch (not 10 independently trained generators).

Table 3. HumanEval pass@k under different update coefficients c (%).

Methodcpass@1pass@5pass@10
DnD (Liang et al., 2025)–26.93 ± 3.7648.49 ± 5.3356.71 ± 5.47
Ours: direct mapper+111.43 ± 0.6821.72 ± 1.4627.58 ± 2.01
Ours: direct mapper-324.89 ± 3.7348.72 ± 4.6058.20 ± 4.37
Ours: flow+111.59 ± 0.6922.12 ± 1.6927.92 ± 2.34
Ours: flow-325.89 ± 3.2849.79 ± 3.7758.55 ± 3.65

With n=20 samples per problem of which m pass, pass@k is estimated per problem as 1-n-mk/nk, then averaged over the 164 problems and over 10 adapters (32,800 completions per row). No syntax cleanup is applied. Because c=-3 is tuned on HumanEval itself, those rows are calibrated results.

  • ARC-c: both ours methods improve over DnD by more than 9 points (+9.27 mapper, +9.51 flow), with much lower variance across adapters.
  • GSM8K: differences are small (+0.29 and +1.04) and we do not claim significance.
  • HumanEval: changing c from +1 to -3 raises pass@1 by 13.46 (mapper) and 14.30 (flow) points. At c=-3, pass@1 remains 2.05 and 1.04 points below DnD, while pass@5 and pass@10 are above DnD for both variants (flow: +1.30 and +1.84).

Reference. Liang et al., Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights, arXiv:2506.16406, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.