acceptodds
Under review as a conference paper at ICLR 2027

A NIGHT AT THE LLM ZOO: DATA CONDITIONED FLOW MATCHING FOR GENERATING PRETRAINED LLM WEIGHTS

Abstract

A challenge to Weight Space Learning as it expands from task specific and vision models to Large Language Models (LLMs) is the absence of a well curated and analyzed LLM population. To address this, we train such a model zoo for application to Weight Space Generation by systematically varying the data mixture composition between web text, books, mathematics, coding, and French across populations of M, M, and M parameter GPT-2 Models, each tier with count . We analyze these populations by cosine similarity, functional diversity, Leave-One-Mixture-Out modeling, and variance decomposition. We find that the zoo contains diverse models whose variation is structured by training mixture composition. We embed the training mixture composition to condition a Flow Matching model of complete pretrained LLM weights in Gram-PCA compressed weight space. By demonstrating zero-shot generalization to held out mixtures, Continuous Ranked Probability Score skill, as well as downstream changes in test-set perplexity and general capability benchmarks, we show that our generator generalizes on the simplex of data mixtures to produce models that perform similarly to those in the zoo ex nihilo. Finally, we produce all source code and artifacts to support the continued expansion of Weight Space Learning and Generation into the LLM domain.

Primary area:
generative models
Keywords:
Weight Space Generation, Weight Space Learning, Model Zoo, Conditional Flow Matching

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_x5YXOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper trains a zoo of 300 GPT-2 models (mini, small and medium) trained on different data mixtures of 5 distinct corporas. The models are pretrained for 85% of the training and then finetuned on the different data mixtures for the last 15%. They provide an analysis into the diversity of the zoo, finding that models trained on the same mixtures are more similar than the models trained on different mixtures. Lastly they use the generation procedure from DeepWeightFlow to show the generation performance by training from this zoo.

Critical strengths

They provide a model zoo of large language models which to the best of my knowledge is a novel contribution to the scientific community and helps expand the field of weight space learning into large language models.

The authors show that a generative approach trained on this zoo can generate models conditioned on data mixtures not seen during training. With the generation outperforming nearest neighbour retrieval baseline.

Critical weaknesses

The paper is essentially training a model zoo, analysing it and using the generative approach from Deepweightflow to generate some models. And whilst the releasing of such a model zoo is valuable to the community, the analysis of the models merely confirmed that models trained on the same data were similar and models trained on different data mixtures are different, which given the shared training trajectory is not surprising. I do not find this scientific contribution substantial enough for ICLR.

Beyond that there are a number of issues in the generalization claims of the contributions. As stated in the paper the Gram-PCA, the weight encoder, is fitted on the full model zoo. Even the models later used for the evaluations. I understand that the generative model is not trained on the evaluation data however this is still a substantial benefit to the approach.

Other issues:

Shared pretrained trunk. The model zoos share a pretrained backbone, referred to as “trunk”. Effectively decomposing the model zoo into a Model Tree, where the fine-tuned models are just 15% of the total training from this trunk. This calls into question the diversity of the model tree in comparison to using independently trained models.

Generalization setting. The experimental protocol in this setup is restricted to 5 different corporas, which are used both for training and evaluation. So while this can show generalization to different data mixtures it limits the generalization claims of the paper. It's also unclear why this setup was chosen instead of using unseen corporas during evaluation.

C2ST. In section 4.3 the authors refer to Appendix B.8 in regards to identifying the requested mixture. And in Table 28 you have a table with a classifier two-sample test (C2ST) which is showing an accuracy of 98%. However it is explained that 50% accuracy represents that the generated and true distributions are truly indistinguishable. Shouldn't this then be the aim of the generative approach, and thus the numbers presented in Table 28 show that the approach is not close to that? I think the author remarks on that on line 431, however no further explanation is provided on the seemingly poor results?

Evaluation. Gold-continuation log score is used to evaluate the performance of the generated models in Table 5 but is not explained in the main text. Thus it is now clear how to interpret the results and how to compare them. Further regarding the accuracy presented in Table 32 in the Appendix, it seems to be showing barely above chance results for the generated models?

Questions for authors
  1. How does the generation performance change when the Gram-PCA is only fitted on the training models?

  2. How would the performance change if the model zoo did not share a trunk but was effectively independently trained models?

  3. Could you explain the C2ST result more, in reference to the aforementioned weakness?

  4. Why are both the real and generated models barely above chance in terms of the accuracy in Table 32?

4 comments
AuthorsOpenReview ↗

Response to Reviewer x5YX - Part 1

We thank the reviewer for an expert and candid review. The basis concern, shared by the other reviewers, led to the largest change in the revision. The C2ST and evaluation questions led to clearer reporting.

Weakness 1. Scientific contribution

We agree that "same mixture$\rightarrow$similar, different mixture$\rightarrow$dissimilar" is expected, and we do not present it as the finding. Its role is a sanity check that the zoo is valid. We make three interconnected contributions:

  1. Mixture-conditioned generation of full pretrained LM weights. Data or dataset-conditioned weight generation has so far targeted image classifiers or LoRA adapters (e.g., Soro et al., 2025b; Schürholt et al., 2022a; LoRA hypernetworks such as Charakorn et al., 2025 among others discussed in the Introduction). Full-weight generation for language models has targeted a single supervised task (BERTs on the Yelp task, Gupta et al., 2026). We generate complete pretrained LMs conditioned on their pretraining mixture, at up to 355M parameters, and beat every retrieval, interpolation, regression, and GP baseline in like settings.

  2. Quantitative structures A shared trunk predicts high raw cosine similarity, which we find (∼0.5\sim 0.5 overall). It does not predict how that similarity is organized:

    • after removing trunk drift, branches with different mixtures are near-orthogonal (−0.07 to −0.02) while replicates stay aligned (0.80–0.89, Table 1);
    • mixture explains 82–92% of variance among replicates (bias-corrected, Table 2) after only 15% of training;
    • the map from mixture to weights is non-additive, and seems to become more so with scale. Task arithmetic over vertex means degrades from 15.7 to 23.4 %ΔPPL. Averaging two different-mixture models beats both endpoints in 18/20 pairs at Mini but 5/20 at Medium, even though every interpolation path is barrier-free (Table 17). Being in one basin does not make members interchangeable.
  3. A controlled LLM zoo with an evaluation protocol (grouped LOMO, proper scoring, replicate and truncation floors, data-copying tests). As you note, no such population existed for weight-space learning.

Weakness 2. Gram-PCA fit on the full zoo

We agree; this was the most important point in the reviews. Every generative result in our revised and reuploaded manuscript uses an inductive basis (Sec. 3.3, App. B.1). For a fold targeting θi\theta_i at πi\pi_i, θi\theta_i is excluded from basis fitting, code normalization, flow training, and decoding of generated latents. For a held-out vertex, all four replicates are excluded. Under this protocol the flow obtains 3.78 / 4.11 / 4.70 %ΔPPL against 5.75 / 5.79 / 6.00 for retrieval. The flow beats every baseline (Tables 4, 23). We also scoped the claim to the interior of Δ4\Delta^4, where it holds in every case we have examined.

Other issue 1. Shared trunk / model tree

We agree that the zoo is a model tree, and that a trunk reduces diversity relative to independent pretraining. We would like to make two clarifications:
(1) The branches are not fine-tuned on a task. They continue the same pretraining objective with the same schedule (no warmup restart), differing only in data mixture over the final 15%. This mirrors how data mixtures are changed late in pretraining in practice (mid-training and annealing; e.g., OLMo et al., 2024; Blakeney et al., 2024).

(2) The design is a deliberate control. It isolates mixture-induced variation from initialization and permutation effects (barrier-free interpolation on all scored pairs, Table 17). It is also a setting where simple baselines such as retrieval and interpolation should be competitive, which makes the flow's margin over them more informative. The conclusion frames the result as holding in a controlled environment.

Branch-Train-Merge (Li et al., 2022) uses a related trunk-then-branch design for expert LMs but does not study weight-space structure or generation.

AuthorsOpenReview ↗

Response to Reviewer x5YX - Part 2

Other issue 2. Five fixed corpora

You have identified an exciting direction for future work, which this project enables. Our experiments establish generalization to unseen mixtures of five fixed corpora, not to unseen corpora themselves. Generalization to held-out datasets has been studied in vision weight generation, while generation across pretraining-corpus mixtures is unexplored. Modern LLMs are pretrained on mixtures, which makes the mixture the natural conditioning variable. We make this limitation of our claims clear in the revised text.

Other issue 3. C2ST

We agree that 0.50 is the target. Rechecking that number, we found that the submitted 0.976 was not informative, and we apologize for the error. That design scored 100 real codes against 4,000 generated ones, and 4000/4100 = 0.976 is exactly the majority-class rate. The classifier was reporting the base rate, not necessarily learning. The revision makes three changes (App. B.12, Table 33):

  1. a balanced design (100 real vs 100 generated, 50 draws);
  2. the classifier's cross-validation split grouped by held-out member, because a row-wise split puts rows that share a πi\pi_i and pushes even a real-vs-real control to 0.35–0.39;
  3. a real-vs-real control.

We report two feature sets. The first is basis-free: each model's five-domain log-perplexity profile together with π\pi. The second is the leading eight codes in a single all-member basis to fix the measurement frame. No generator is trained in or decoded from that basis alone.

C2ST accuracy (0.50 = indistinguishable)MiniSmallMedium
Function space: flow0.6500.6810.703
Function space: rank-50 projection of the real target0.6540.7140.786
Function space: real-vs-real control0.5080.5050.516
Measurement frame (c0..7,π)(c_{0..7}, \pi): flow0.5680.5660.566
Measurement frame: real-vs-real control0.4980.4880.472

In function space, generated models remain detectable at 0.65–0.70. In the code frame they are close to indistinguishable. However, the exact rank-50 projection of the real target, with no generator involved, is equally or more detectable (0.65 / 0.71 / 0.79). Most of the detectable signal therefore comes from rank truncation. The revision states this directly.

Other issue 4. Gold-continuation log score and near-chance accuracy

We have defined the gold-continuation log score in the main text and the Table 5 caption. It is the log-probability the model assigns to the correct HellaSwag ending, length-normalized (mean units per ending token), over 2,000 questions and paired between real and generated models. Unlike accuracy, it is continuous and does not saturate at chance, so it separates models that accuracy cannot.

Accuracy is near chance for real and generated models alike (Table 37, formerly Table 32): 0.252 / 0.257 / 0.273 for real interior members, against 0.25 chance. This is expected at our token budget. At 20 tokens per parameter (Chinchilla-optimal; Hoffmann et al., 2022) the zoo members see about 1.0B / 2.5B / 7.1B tokens. Small models with non-trivial HellaSwag accuracy are trained far beyond this. For example, SmolLM-135M (Allal et al., 2024) is trained on more than 50 times our token budget and reaches about 42%. Accuracy rises with scale across our zoo as expected.

On the log score, generated models score slightly higher than their real targets on the interior (Table 5). We attribute this to shrinkage: generated codes are pulled toward the conditional mean of nearby members, which acts like weight averaging. Averaging same-mixture models improves on both endpoints in 19–20 of 20 pairs (Table 17).

Q2. Performance without a shared trunk

We do not have data to answer this directly, and this gap is precisely why we started from a controlled population. We would expect some degradation but have no basis to quantify it. Our evidence is:

(1) The Mini β\beta sweep (Table 14), making 60% of training mixture-specific lowers the between/within-mixture distance ratio from 4.08 to 2.88, but mixture structure clearly survives.

(2) Independently trained networks are thought to be linearly connected only modulo permutation (Entezari et al., 2022; Ainsworth et al., 2023), so a trunk-free zoo would likely need re-basining before Gram-PCA, as in prior weight-generation work (Gupta et al., 2026).

W1: Scientific contribution. This paper is not proposing a new generation approach, but rather a new model zoo which you show, by using DeepWeightFlow (Gupta et al., 2026), can be used to train a generative model. However the authors are correct in that mixture-conditioned generation has not been used in this field before.

W2: Gram-PCA fit on the full zoo. I appreciate this change, however I do not understand the use of the wording inductive bias, shouldn't this change just amount to a proper train-test split?

Other issue 1. Shared trunk / model tree. I understand that this design is a deliberate control, however that weakens the claim of mixture conditionied generation of full pretrained LLMs. As this only applies to a low diversity zoo.

Other issue 3. C2ST I welcome the new C2ST results, which look like what i would expect for the generation in this low diversity regime.

Other issue 4. Gold-continuation log score and near-chance accuracy. If the real model are also at chance, why even use them?

Q2. Performance without a shared trunk I thank the authors for this answer and understand the need to start from a controlled population.

Nevertheless I remain unconvinced and will maintain my score.

AuthorsOpenReview ↗

Response to Reviewer x5YX

We thank the reviewer for continuing to engage with our work, and we respond to the remaining concerns they identify below.

Novelty:

The reviewer is correct that we do not propose a new generator. We see three contributions. The first is the zoo itself: a controlled population of pretrained general-purpose language models in which the pretraining mixture is the only varied factor. The second is mixture-conditioned generation of complete pretrained LM weights, which is, to our knowledge, the first full-weight generation of pretrained general-purpose language models, at scales up to 355M parameters. The third is the evaluation protocol. Its inductive LOMO design withholds the target model from the Gram-PCA fit as well as from flow training, which departs from Gupta et al. (2026). CRPS skill is a proper score that a point mass cannot optimize. The per-cell retrieval control is something a copying generator can at best tie. We regard the protocol as a durable contribution independent of the generator, since it applies to any future weight generator evaluated on this zoo.

On terminology, the reviewer is right: the inductive basis amounts to a proper train-test split for the latent representation. “Bias” would be a typo, and we have checked that it does not persist. We were simply unsure of what wording would be clearest, and are open to suggestions.

Shared trunk and scope of the claims:

We agree that the shared trunk defines the scope of our generation claims, and our revised manuscript scopes our claims in this light. Section 3.2 states that all members share architecture, optimizer, schedule, token budget, and initialization, with only π\pi and a branch seed varying. A.5 discusses the β→1\beta\rightarrow1 regime of independently trained runs, where canonicalization would become necessary and which our results do not cover. The abstract and conclusion restrict the generalization claim to the interior of the Δ4\Delta^4 mixture simplex within this population.

Within that scope, we believe the population is structured rather than degenerate. Once the drift inherited from the shared trunk is removed, cosines between branch deltas from different mixtures fall to roughly −0.14 to +0.17 (Table 1). Replicates of the same mixture stay strongly aligned (0.80 to 0.89). Functionally, members trained on different mixtures disagree on next-token predictions about twice as often as replicates do (Table 3). The variance decomposition $\eta^2_{\text{corr}}=0.82 \text{ to 0.92} $ is measured at the simplex vertices and bounds the influence of π\pi from above. On the interior, a quadratic surface on π\pi explains about 22 to 32% of held-out branch-delta variance (Table 24).

Trunk sharing is also a controlled design parameter – not a hidden confound. The branching fraction sets how much of training is shared. Our β\beta sweep (Table 14) shows displacement from the trunk rising from 16.5% to 55.9% as β\beta goes from 0.15 to 0.60. Over the same range, the between/within-mixture distance ratio falls from 4.08 to 2.88. We chose β=0.15\beta=0.15 as the smallest value that passed our domain-separation checks .

Why report HellaSwag:

We agree that accuracy alone is uninformative at these scales, which is why we report the gold-continuation log score as the primary benchmark measure (Table 5). Accuracy is quantized at 1/2000 and sits near chance for real and generated models alike, so on its own it cannot distinguish parity from a failure to measure (Appendix B.17). The log score is continuous and paired question by question, and it separates models even where the argmax does not move. It also tracks scale cleanly, from about −5.2 at Mini to −4.1 at Medium, which indicates it carries signal.

On this measure, generated models match or slightly exceed the real models at the same mixture in every stratum (Table 5). On the interior, accuracy agrees, with no stratum differing by more than one question (Table 37). The comparison of interest is generated versus real at the same mixture, not absolute capability. Our primary functional measure remains %ΔPPL\%\Delta\text{PPL} across five held-out domains (Table 4).

Regarding the models’ usefulness more broadly: they are intended as a population for studying weight space, not for deployment. They follow the Chinchilla 20:1 token-to-parameter ratio (Hoffmann et al., 2022). Small models trained for downstream utility, such as SmolLM-135M, use substantially more tokens (discussed around Table 37).

Reviewer_c2PjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how training-data mixtures relate to language-model weights. The authors construct a controlled zoo of 300 GPT-2 models across three scales (51M, 124M, and 355M), sharing the first 85% of training before branching into different mixtures of five data domains. They analyze the resulting weight-space structure and train a mixture-conditioned flow-matching generator in a 50-dimensional Gram-PCA representation. The generated codes are decoded into model weights and evaluated without further training. The reported results show closer perplexity profiles to target models than nearest-neighbor retrieval, particularly for interior mixtures. The work offers a controlled setting for studying mixture-dependent weight variation, although its scalability and practical utility remain insufficiently established.

Critical strengths

The paper presents an interesting approach to studying the effects of training-data mixtures through mixture-conditioned weight generation. The controlled model zoo could also provide a useful resource for further research on the relationship between data composition and model weights.

Critical weaknesses

Scalability remains insufficiently established. This is my main concern. First, the average relative perplexity discrepancy between generated and target models increases from 4.05% at 51M parameters to 5.11% at 355M parameters, while the advantage over nearest-neighbor retrieval narrows. Although this does not establish that the method will fail at larger scales, it raises questions about its scalability. Second, the generative experiments primarily use a 50-dimensional representation and a small generator. The appendix provides some ablations, but these do not adequately establish how representation dimensionality and generator capacity should scale with the target model. Third, the approach relies on a zoo of 100 models per scale. Sharing the first 85% of training reduces this cost, but constructing the remaining branches may still require substantial computation for larger LLMs. The paper does not establish how many models are necessary or when the upfront cost would be justified by subsequent use.

The practical value for data-mixture selection is not demonstrated. The existing comparisons evaluate the fidelity of generated models, but they do not establish whether the approach helps identify better training-data mixtures. If data-mixture selection is an intended application, an end-to-end comparison with established mixture-selection methods is needed, reporting both the final model performance and the total computational cost, including zoo construction, representation fitting, generator training, and candidate evaluation. Without such evidence, the practical benefit remains unclear.

The analysis provides limited actionable insight into data-mixture effects. The paper reports several relationships between mixture composition, weight-space structure, and model behavior, but much of the discussion remains descriptive. For example, Figure 1 shows how the advantage over retrieval and predictive skill vary with distance to observed mixtures, yet the discussion offers limited explanation of the underlying patterns or their implications for mixture design. More targeted analyses would help explain which domain interactions matter, why performance differs between interior and vertex mixtures, and whether these findings remain consistent across model scales.

The presentation makes the contribution difficult to assess. The paper is difficult to follow, with numerous metrics and experimental results but insufficient synthesis of their main implications. The relationship between the model-zoo analysis, the weight-generation experiments, and potential practical applications is not communicated clearly enough. A more focused presentation should distinguish what is demonstrated directly from what remains a prospective application.

Questions for authors

Generalization and PCA construction. The PCA basis includes models subsequently held out from generator training. Could the authors quantify performance when both the PCA representation and the generator are fitted exclusively on training models? What generalization claim remains supported when the target model’s weights are unavailable during representation construction?

Scalability. The perplexity discrepancy increases with model size, while the advantage over retrieval narrows. To what extent does this arise from PCA truncation versus limitations of the generator? Beyond the existing small-scale ablations, what evidence supports an appropriate scaling strategy for representation dimensionality, generator capacity, and zoo size?

Computational cost and sample requirements. Could the authors report the total computational cost, including zoo construction, PCA fitting, generator training, and weight decoding? How does generation quality change when fewer zoo members are available, and how many new models would need to be generated before the upfront cost becomes competitive with directly training additional branches?

Practical application and relevant comparisons. What concrete use case is the method intended to enable? If data-mixture selection is an intended application, could the authors demonstrate that it identifies competitive mixtures under a matched total compute budget relative to existing mixture-selection approaches? Otherwise, what measurable practical advantage does weight generation provide in the proposed setting?

Insights into data-mixture effects. Beyond showing that mixture composition correlates with weight-space structure, what new conclusions can be drawn about how data domains interact? For example, can the authors explain the substantially different behavior on interior versus vertex mixtures and identify findings that are consistent across model scales and useful for designing training mixtures?

3 comments
AuthorsOpenReview ↗

Response to Reviewer c2Pj - Part 1

We thank the reviewer for their thorough review. Several points (scaling decomposition, interior versus vertex behavior, cost accounting) led to new analyses that are now in the paper. For clarity in our response, we have grouped weaknesses with their matching questions.

W1a / Q2. Error grows with scale while the margin over retrieval narrows

The concern is fair, and the revised paper makes the same observation clearly, especially in Section 4.1. Two things have changed since submission. First, under the inductive protocol the trend is milder, especially on the interior mixtures:

Interior mixtures51M124M355M
Flow %ΔPPL ↓2.462.522.93
1-NN retrieval %ΔPPL4.434.254.36
Retrieval − flow+1.97+1.73+1.43
Paired t (seed 1234)5.475.414.69
CRPS skill ↑0.7740.7730.749

Second, on whether the trend comes from PCA truncation or from the generator: the evidence points mostly to truncation.

Interior, %ΔPPL51M124M355M
Rank-50 projection floor3.123.284.12
Flow2.462.522.93
Flow − floor (paired t)−0.66 (−5.2)−0.77 (−5.5)−1.19 (−7.7)

From 51M to 355M the floor rises by 1.00 point and the flow by only 0.47. The flow's advantage over the exact projection of its own target widens with scale. The likely cause is structural: transformer blocks grow from 49% to 85% of parameters across our scales, and their variance has a higher effective rank (7.2 → 8.9 → 11.3, Table 22). A fixed rank-50 basis therefore captures less of each held-out member at larger scale (projection ceiling R² 0.61 → 0.57 → 0.43, Table 24).

In this case, the floor is the L2-optimal decode, not the perplexity-optimal one. This decomposition is therefore indicative and we cannot not claim asymptotic scalability. Scaling the representation further is an open problem, and is understood as one of the primary obstacles to practical weight space generation.

W1b / Q2. Representation dimension and generator capacity

We agree we do not establish a joint scaling rule. The Mini ablation (Table 19, fixed 51K-parameter generator, all 85 inductive folds) shows that more rank is not automatically better. We report the conditional-width ratio aπa_\pi (1 = correct within-mixture spread) alongside conditional recovery:

Rank500 epochs8k epochs20k epochs
k = 50: recovery (aπa_\pi)28.7% (1.79)103.6% (0.94)117.3% (0.67)
Full rank: recovery (aπa_\pi)3.7% (2.04)23.8% (1.87)25.8% (1.85)

At full rank the generator's conditional stays about twice as wide as real within-mixture variation, even at 20k epochs. At k = 50 it converges to roughly the correct width by 8k epochs. The 20k-epoch setting was selected on functional CRPS, which is a proper score. We report aπa_\pi as a guard against collapse, not as a selection criterion (App. B.10). The potential tightness at 20k is a known cost, and it is also likely the residual that the function-space C2ST detects (response to x5YX). Table 21 explains why extra rank does not help, namely: that the first four components carry 80% of the variance but only 0.3% of the within-mixture (replicate) variance. Most replicate noise lives in non-primary components. We conclude that extra rank mostly adds member-specific noise. We use k = 50 as an empirically supported operating point, not a universal prescription.

W1c / Q3. Zoo size and total compute

Cost. Only the final 15% of training is repeated per member:

ScaleTotal stepsShared trunkSteps per branch
51M7,8546,6761,178
124M18,98816,1402,848
355M54,14246,0218,121

A 100-member zoo therefore costs about 0.85 + 100 × 0.15 = 15.85 full-run equivalents instead of 100. Measured cost (Table 13, B200 GPU-hours):

ItemGPU-hours
Trunks, all scales26.6
Branches, Mini / Small / Medium24.0 / 76.3 / 358.6
Gram-PCA fitting and encoding0.5
Generator training, all flows0.3
Evaluation69.5
β calibration and probes55.8
Total, original study611.5
Inductive revision, all jobs168.5 (+83.5 CPU-core-hours)

Each flow has about 51K parameters and trains in a few CPU-minutes. Once a zoo exists, representation and generator costs are negligible next to zoo training.

AuthorsOpenReview ↗

Response to Reviewer c2Pj - Part 2

Fewer members. Prompted by this question, we ran a controlled subsampling experiment at Mini (Table 16) among others in Appendix A.6 (Table 15, Figure 3).

In each fold, we refit both the basis and the flow on a random 50-member subsample of the training members. Interior error rises from 2.36 to 4.25 %ΔPPL. On a subset of 20 interior folds where we also decoded the floor, the rank-49 projection floor rises only from 3.34 to 3.53. On those folds the flow goes from 2.16, well below its floor, to 3.65, level with it. In weight space we refit only the basis, at N = 12 to 99 with 10 draws per size. The squared out-of-span residual of a held-out interior member falls across our scales, and is still falling at N = 99. So, N = 100 is not oversized for this task. A compute break-even against training additional branches depends on the downstream use and remains future work.

W2 / Q4. Practical value and data-mixture selection

We agree that we do not demonstrate data-mixture selection, because it is not the task we study. DoReMi and RegMix choose or rank mixtures. Our method takes a requested mixture as given and generates the corresponding pretrained weights, π↦p(θ∣π)\pi \mapsto p(\theta \mid \pi). An end-to-end comparison against mixture-selection methods would evaluate a different objective. The revision removes any framing that suggested mixture selection as a demonstrated application.

The capability we demonstrate is zero-shot synthesis of a full pretrained checkpoint at an unseen mixture, with no optimization of the generated model, at lower error than any baseline that reuses or combines existing checkpoints (Table 23). It also gives a continuous map over mixture space. We do not claim this is cheaper than mixture search or direct pretraining under matched compute.

W3 / Q5. Insight into data-mixture effects

We agree the submitted discussion was too descriptive. The revision adds the measurements behind the following findings (Tables 17, 20–23, 25, 34), each consistent across all three scales.

  1. Interior and vertices differ because of basis coverage, not difficulty in the generator. A held-out vertex takes all four of its anchors out of the basis, and no remaining mixture lies near that corner. The target's own rank-50 projection is then 23.6 / 30.2 / 33.8 %ΔPPL away (Table 20), with CRPS skill at vertices is 0.23 / 0.15 / 0.10 against about 0.77 in the interior.

  2. Domain effects are smooth but not additive. Despite all members sharing one basin, task arithmetic over vertex means is among the worst baselines (15.7 → 16.2 → 23.4 %ΔPPL across scales, Table 4 and 23). A flow trained only on the vertices fails (26.6 / 45.5 / 51.7 %ΔPPL over all 85 cells, Table 23). Averaging two models trained on different mixtures beats both endpoints in 18/20 pairs at Mini but only 5/20 at Medium (Table 17). It appears that mixtures cannot be composed by mixing their weights.

  3. Mixture entropy predicts most of the functional response in the interior. Predicting a held-out member's five-domain perplexity profile from π\pi alone gives LOMO R2R^2 of 0.16 / 0.27 / 0.37. Adding the entropy H(π)H(\pi) raises this to 0.70 / 0.72 / 0.73 (Table 25). How concentrated a mixture is matters more to the perplexity profile than its linear proportions.

  4. Domains differ in how much their own perplexity responds to their weight. Books has the weakest response at every scale (real slopes −0.36 / −0.43 / −0.47 log-PPL per unit weight, Table 34). Web and math strengthen with scale. The generator reproduces 94% / 88% / 90% of the real dose-response on interior mixtures, but captures little of the already weak books response.

We do not claim to identify causal pairwise domain interactions (e.g., code x math). That would need a design that varies pairs systematically, which our Dirichlet plan does not. The zoo is released partly to make such follow-ups possible.

W4. Presentation

We agree. The revision reports results by stratum and by scale and moves secondary diagnostics to the appendix. The text is now framed around three questions: whether mixture structures weight space, whether that structure supports generation at unseen mixtures, and where generation succeeds and fails. The scope of the claim is as follows. We demonstrate full-weight generation at unseen interior mixtures up to 355M in an inductive basis. We do not claim mixture optimization, asymptotic scaling, or compute advantages.

Q1. PCA and generator both fit only on training models

Yes; all reported generative results now follow this protocol (General Response, point 1). The flow obtains 3.78 / 4.11 / 4.70 %ΔPPL over all 85 cells against 5.75 / 5.79 / 6.00 for retrieval, and 2.46 / 2.52 / 2.93 on the interior. The claim that remains supported is zero-shot generation at unseen interior mixtures, without the target ever entering the representation.

Dear Reviewer c2Pj,

As the authors' response deadline is fast approaching, please read through their rebuttal and let them know whether your concerns have been addressed or further clarification is still required.

Thank you.

AC

Reviewer_1yTWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds a controlled zoo of GPT-2 models at 51M, 124M and 355M parameters. Each scale has N = 100 members, and the only factor that varies is the pretraining mixture over five domains (web, code, math, books, French).

Critical strengths
  1. A real gap in the literature. There is no controlled LLM zoo for weight-space learning, and varying only the data mixture is a clean design.
  2. Careful statistical hygiene. The paper uses grouped LOMO folds so replicates cannot leak, a proper scoring rule (CRPS) instead of MSE, a replicate noise floor, and a rank-truncation floor.
Critical weaknesses
  1. The central generalization claim is weaker than the abstract suggests. Zero-shot generalization on the simplex” should be restricted to interior mixtures near seen ones. The abstract’s “minimal changes” in perplexity overstates a 4–5% gap that is 4–5× the replicate floor
  2. The Gram-PCA basis is fit on all members, including the held-out ones. A fully held-out basis experiment is needed, even at Mini scale where it is cheap. Without it, the headline claim is weakly supported.
  3. The comparison to retrieval is thin, and the “beats a copier” argument has a gap. The margin is +1.7, +1.6 and +0.9 points of ΔPPL. At Medium scale it is significant in only 43 of 85 cells (t = 3.1), and it shrinks monotonically with scale.
Questions for authors
  1. Can you refit the Gram-PCA basis without the held-out members and report ΔPPL and CRPS skill?
  2. How strong are kernel or GP baselines in code space (and in weight space, with weights interpolated from the nearest members)?
  3. Is the finding independent of the shared-trunk design?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1yTW

We thank the reviewer for a precise review. It led to three substantial revisions: all generative experiments now use a fully inductive basis, the baseline comparison is greatly expanded, and the generalization claim is narrowed to the interior of the simplex.

W1. Scope of the generalization claim

We agree. The claim is limited to interior, non-degenerate mixtures, throughout the text, and the "minimal changes" wording has been revised. The conclusion names the vertices as the performance boundary.

Held-out stratum, %ΔPPLMiniSmallMedium
Flow, interior (n = 80)2.462.522.93
1-NN retrieval, interior4.434.254.36
Flow / replicate floor, interior2.4×2.3×3.0×
Flow, vertices (n = 5)24.9729.6233.09

Over all 85 cells the flow is about 4-5 times the replicate floor (Table 4). On the interior it is 2-3 times. The floor can only be measured at the vertices, and only approximates the interior mixture floor. In both cases we now report the gap rather than calling it minimal.

On "near seen ones": Fig. 1 bins the 80 interior cells, and Table 26 all 85 cells, by L1 distance to the nearest training mixture. The lower and middle terciles contain no vertices and give nearly identical error (2.14 vs 2.28 %ΔPPL at Mini). Within this zoo, error does not track distance to the nearest training mixture. Zoo size does matter, however: halving the zoo roughly doubles interior error (response to c2Pj, W1c, added in Appendix A.6, Tables 15,16 and Figure 3).

W2 / Q1. Leakage through the Gram-PCA basis

We agree; this was the most important point in the reviews. Every generative experiment now uses an inductive basis. The held-out member, or all four replicates for a held-out vertex, is excluded from basis fitting, code normalization, generator training, and decoding. Under this protocol the flow obtains 3.78 / 4.11 / 4.70 %ΔPPL against 5.75 / 5.79 / 6.00 for 1-NN retrieval. CRPS skill on the interior is 0.774 / 0.773 / 0.749. The transductive tables are replaced throughout.

W3 / Q2. Retrieval, kernel, and GP baselines

All baselines use the same training members and the same blinding as the flow. Every basis-dependent method uses the same inductive rank-50 fold basis. Your weight-space interpolation baseline is the kNN row: an inverse-distance-weighted average of the k nearest members' full weights.

%ΔPPL, all 85 cellsSpaceMiniSmallMedium
Flow (ours, 3 seeds)π→z\pi \to z3.784.114.70
1-NN retrievalπ→θ\pi \to \theta5.755.796.00
kNN, k = 2 (weight interpolation)π→θ\pi \to \theta5.915.896.27
kNN, k = 5 (weight interpolation)π→θ\pi \to \theta6.266.366.90
Quadratic surfaceπ→z\pi \to z6.496.426.54
Kernel ridgeπ→z\pi \to z7.677.818.29
GP posterior meanπ→z\pi \to z9.239.179.51
Task-arithmetic soup (vertex means)π→z\pi \to z15.6516.2023.37

The full set (projected kNN, entropy-aware GP and surfaces) is in Table 23.

You are right that the retrieval margin shrinks with scale, and our revision identifies and explains this fact (Sec. 4.4). The retrieval-minus-flow gap is +1.97 / +1.73 / +1.43 %ΔPPL. Paired over cells, using seed 1234 as in Table 27, t = 5.47 / 5.41 / 4.69, and the flow wins 58 / 56 / 51 of 80 cells. At Medium scale (previously 43 of 85, t = 3.1), the flow now wins 52 of 85 cells overall (t = 4.23).

This supports the claim that our generator is not simply memorizing. The data-copying test (App. B.16) never reaches the copying threshold.

Q3. Is the finding independent of the shared-trunk design?

We cannot establish that from these experiments. The trunk is deliberate. It cuts zoo cost approximately 6-fold, and it places all members in one basin by construction (negative interpolation barriers on all 120 scored pairs, Table 17), so mixture-induced variation is not confounded with permutation symmetries.

We do have partial evidence on robustness to the branching fraction β. In a Mini-scale sweep (N = 12 per setting, Table 14), raising β from 0.15 to 0.60 raises displacement from the trunk from 16.5% to 55.9%. The between/within-mixture distance ratio falls from 4.08 to 2.88 but stays well above 1, so mixture structure persists when most of training is mixture-specific. We did not train generators at β = 0.60, so this is only evidence about the geometry. Our claim has been clarified, that within a controlled, common-basin LLM population, mixture-conditioned structure is regular enough to support zero-shot generation at unseen interior mixtures. Independently pretrained populations, which would likely need re-basining as in prior weight-generation work, remain for future work.

Thank you for the additional experiments and clarifications. The fully inductive protocol and expanded baselines address my concerns about basis leakage and the original comparison. However, I remain unconvinced that the contribution, under its revised scope, is sufficiently strong to warrant acceptance. In recognition of the additional evidence, I would raise my assessment to between 2 and 3, while still recommending substantial revision before acceptance.

AuthorsOpenReview ↗

Response to Reviewer 1yTW

We thank the reviewer for engaging closely with the revision and for their raised assessment. We are glad the inductive protocol and expanded baselines resolve the concerns about basis leakage and the original comparison.

On the strength of the contribution under its revised scope:

The narrowing applies to the generator’s claim, not to the two contributions the reviewer identified as strengths in the original review. The first is a controlled LLM zoo for weight-space learning, which fills a gap the reviewer noted in the literature. The second is the evaluation protocol: grouped LOMO folds, a proper scoring rule, and replicate and rank-truncation floors, now extended with a fully inductive basis. Both are independent of the generator’s margin over retrieval and can be reused by any future work studying LLM weight spaces.

On top of these, the generator result under the inductive protocol is, to our knowledge, the first full-weight generation of pretrained general-purpose language models. Averaged over all 85 held-out mixtures, it has the lowest %ΔPPL\%\Delta\text{PPL} of any baseline at every scale (Table 23), and it beats retrieval on 51 to 58 of 80 interior cells.

The substantial revision the reviewer recommends has been carried out in the current manuscript. Every generative experiment was rerun under the inductive basis (168.5 additional GPU-hours, Table 13), the transductive results were replaced throughout, the baseline set was expanded, and our claims are well scoped by the evidence provided.

If there are specific further analyses the reviewer considers necessary, we would welcome the suggestion and will do our best to address it promptly.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Meta Response

We thank the reviewers for their detailed and technically engaged reviews. Three primary concerns were shared across reviewers:

(i) the Gram-PCA basis was fit on members later held out (all three reviewers)

(ii) the role and limits of the shared trunk (1yTW, x5YX)

(iii) the scope of the generalization claim and how it behaves with scale (1yTW, c2Pj)

We summarize the main revisions here and answer each review individually. Results cited by table or section number are in the reuploaded revised PDF. We will also include relevant data in our responses to individual reviewers as appropriate.

1. Every generative result now uses a fully inductive basis. For each leave-one-mixture-out (LOMO) fold, the held-out member is excluded from Gram-PCA fitting, code normalization, generator training, and decoding (Sec. 3.3, App. B.1). For a held-out vertex, all four of its replicates are excluded. We reran every generative experiment and every baseline under this protocol, and the revised tables replace the transductive ones. Our original transductive choice followed the DeepWeightFlow pipeline (Gupta et al., 2026), from which we adopt Gram-PCA.

%ΔPPL, all 85 held-out mixturesMiniSmallMedium
Flow, transductive basis, as submitted4.054.175.11
Flow, inductive basis (3 seeds)3.78 ± 0.134.11 ± 0.174.70 ± 0.13
1-NN retrieval (basis-independent)5.755.796.00
Retrieval − flow (inductive)+1.97+1.68+1.29
Replicate floor (seed noise)1.011.090.97

The controlled comparison is between the first and second rows, indicating that the transition from a transductive to inductive (held-out) basis has minimal impact. This data can be found in the context of expanded baseline methods in Tables 4 and 23.

2. Claims are scoped to the interior of the simplex. We diagnose vertex failures. On the interior mixtures the flow reaches 2.46 / 2.52 / 2.93 %ΔPPL against 4.43 / 4.25 / 4.36 for blinded retrieval. At the held-out vertices it fails (25–33 %ΔPPL). The revised paper reports the vertices separately from the interior (Tables 20, 24, 25) and names them as the performance boundary. The target's own rank-50 projection into that basis is already 23.6 / 30.2 / 33.8 %ΔPPL from it (Table 20), and the flow is within 1.4 points of that floor at every scale. The vertex failure is a basis-coverage failure.

3. Baselines are substantially expanded. All baselines receive the same training members and blinding as the flow: 1-NN and kNN (k = 2, 3, 5, in raw weights and projected), linear, quadratic and entropy-aware surfaces, kernel ridge, two Gaussian processes, and a task-arithmetic weight soup over vertex means (Table 23). The flow has the lowest error of any source at every scale.

4. The shared trunk The trunk has three roles:

  • Compute. The final 15% of training is repeated per member vs fully independent training. Trunk-plus-branch training took about 486 GPU-hours (Table 13). Scaling estimates independent runs at about 3,100.
  • A common basin by construction. All scored interpolation paths have negative LMC barriers (Table 17). Mixture-induced variation is therefore not confounded with permutation symmetries, and no re-basining is needed (App. A.5).
  • Realism. Changing the data mixture for the final phase of pretraining mirrors current practice in mid-training and annealing phases (e.g., OLMo 2, OLMo et al., 2024; Blakeney et al., 2024; Llama 3, Grattafiori et al., 2024).

The trunk limits what we can claim. Our results establish generation within a controlled trunk–branch population, not within independently pretrained, heterogeneous LLMs. We state this, together with the other boundaries raised in the reviews, throughout the manuscript.

5. Scaling. Generation error grows modestly with scale, which we determine comes mostly from the representation. Over the same range the rank-50 projection floor rises from 3.12 to 4.12, while the flow's advantage over that floor widens from −0.66 to −1.19 %ΔPPL (Table 20). Transformer blocks dominate parameter count at scale, and their variance has a higher effective rank (7.2 → 11.3, Table 20). We claim robust generation up to 355M, not asymptotic scaling.

6. Presentation. The revision reports generative results by stratum and scale. We clarify CRPS skill, gold-continuation log score, and C2ST where they are first used. We are eager to incorporate additional reviewer feedback, and have worked hard to offer a comprehensive revision and response before the end of the response period.

7. Code link. The anonymized repository link broke sometime between submission and now. We have fixed this, and hope it hasn't impacted reviewers' access to our work.

Edited to accommodate late-addition table(s).

AuthorsOpenReview ↗

Closing Meta Response

We thank the reviewers for their engagement during the discussion period, and the AC for facilitating it. As the response period closes, we briefly summarize where the discussion stands, so the full picture is available in one place.

All three reviewers identified the zoo itself as a valuable contribution. Reviewer 1yTW noted that no controlled LLM zoo exists for weight-space learning, and that varying only the data mixture is a clean design. Reviewer x5YX described it as a novel contribution that helps extend weight-space learning to large language models. Reviewer c2Pj noted it could be a useful resource for studying the relationship between data composition and model weights. Reviewer 1yTW additionally highlighted the evaluation protocol: grouped LOMO folds, a proper scoring rule, and replicate and rank-truncation floors.

Concerns resolved during discussion:

1. Gram-PCA basis leakage (raised by all three reviewers). Every generative experiment and baseline was rerun with a fully inductive basis: the held-out member (or all four replicates of a held-out vertex) is excluded from basis fitting, code normalization, generator training, and decoding. The result holds:

%ΔPPL\%\Delta\text{PPL}Mini (51M)Small (124M)Medium (355M)
Flow, interior (n=80n=80)2.462.522.93
1-NN retrieval, interior4.434.254.36
Paired tt, retrieval −- flow5.475.414.69
CRPS skill, interior0.7740.7730.749

Reviewers 1yTW and x5YX have both acknowledged this change in their follow-up comments.

2. C2ST. The submitted value of 0.976 equaled the majority-class rate of an unbalanced design. Under a balanced, member-grouped design with a real-vs-real control, generated codes score 0.570.57 against a control of ≈0.49\approx 0.49 (Table 33). In function space, the exact rank-50 projection of the real target is as detectable as the flow (0.650.65–0.790.79 vs. 0.650.65–0.700.70), so this signal comes mainly from rank truncation, not the generator. Reviewer x5YX noted the revised results are consistent with expectations.

3. Baselines and significance at scale. The flow now has the lowest error of all 19 baseline sources at every scale (Table 23), including weight-space kNN interpolation, kernel ridge, Gaussian processes, and task-arithmetic soups. At Medium, the flow now wins 52 of 85 cells (t=4.23t = 4.23), up from 43 of 85 (t=3.1t = 3.1) in the transductive initial case.

Reviewer c2Pj's concerns:

For convenience, here is where each concern is addressed in the revision:

  • Scalability. The growth in error with scale comes mainly from the representation. On the interior, the rank-50 projection floor rises from 3.12 to 4.12 %ΔPPL\%\Delta\text{PPL}, while the flow's advantage over that floor widens, from −0.66-0.66 to −1.19-1.19 (t=−5.2t = -5.2 to −7.7-7.7; Table 20).
  • Zoo size. Halving the zoo raises interior error from 2.36 to 4.25. The out-of-span residual is still falling at N=99N = 99, so N=100N = 100 is not oversized (Tables 15–16, Fig. 3).
  • Compute. Full accounting is in Table 13: 611.5 GPU-hours for the original study, plus 168.5 for the revision. The shared trunk reduces zoo cost roughly 6-fold relative to independent training.
  • Practical application. We do not claim mixture selection. The text states the demonstrated capability clearly: zero-shot synthesis of a full pretrained checkpoint at an unseen mixture.
  • Insight into mixture effects. The revision adds four findings that are consistent across scales:
    • Vertex failures reflect basis coverage, not generator difficulty (Table 20).
    • Mixture effects are smooth but non-additive. Task arithmetic over vertex means degrades from 15.7 to 23.4 %ΔPPL\%\Delta\text{PPL} with scale.
    • Mixture entropy H(π)H(\pi) raises function-space LOMO R2R^2 from 0.160.16–0.370.37 to 0.700.70–0.730.73 (Table 25).
    • Domains differ in dose-response, with books consistently the weakest (Table 34).
  • Presentation. Results are reported by stratum and scale, with secondary diagnostics in the appendix. The text is better organized around three questions: whether the mixture structures weight space, whether that structure supports generation, and where generation succeeds and fails.

Scope of our claims:

We demonstrate mixture-conditioned generation of complete pretrained language models at unseen interior mixtures, up to 355M parameters, in an inductive basis, within a controlled trunk–branch population. We do not claim generalization to independently pretrained populations, to unseen corpora, to the simplex vertices, or to asymptotic scale. We view the controlled setting as a necessary first step. Vision weight-space learning followed the same path, moving from controlled zoos (Schürholt et al., 2022b) to heterogeneous public checkpoints (Falk et al., 2025b), and the released zoo and protocol are intended to enable the analogous next steps for LLMs.

We thank the reviewers and AC again for their time and careful engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.