acceptodds
Under review as a conference paper at ICLR 2027

ParaGen-Bench: A Benchmark for Neural Network Parameter Generation

Abstract

Neural network parameter generation is a paradigm of producing the trained weights of a target network through a learned generator. It has emerged as a new direction for acquiring model parameters beyond hand-designed algorithms and backpropagation. However, the field still lacks a benchmark that enables comprehensive and standardized comparison across existing methods and provides a systematic understanding of the current state of development. To close this gap, we introduce ParaGen-Bench, the first benchmark for neural network parameter generation, covering image classification, text classification, image segmentation, reasoning, coding, multimodal understanding, and reinforcement learning. It comprises 44,000 trained checkpoints across 41 architectures and 29 datasets, each paired with 50 paraphrased prompts and structured metadata capturing training trajectories and hyperparameters. We conduct a comprehensive evaluation of ten representative methods spanning three paradigms: diffusion-based, hypernetwork-based, and large language model (LLM)-conditioned approaches. Results show that larger and more structurally complex target models exhibit greater performance gaps, parameter errors cause particularly severe downstream degradation in image segmentation and reinforcement learning, and gaps remain between generated and trained LoRA adapters across reasoning, coding, and multimodal tasks. Finally, we propose simple yet effective LLM-instructed parameter generation (LPG), a two-stage baseline that combines structure-aware weight tokenization with prompt-conditioned latent generation, taking a step toward practical prompt-conditioned weight generation.

Primary area:
datasets and benchmarks
Keywords:
Parameter Generation, Weight Space, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ju9fOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ParaGen-Bench for systematically evaluating neural network parameter generation. The benchmark comprises 44,000 checkpoints spanning seven task domains, 41 architectures, and 29 datasets, together with natural-language descriptions and training metadata. The authors compare ten representative methods using common target models and downstream evaluation settings, and introduce LPG, a baseline combining structure-aware weight encoding, a VAE, and language-conditioned latent generation. The experiments examine performance across tasks, model scales, and architectures, with additional analyses of prompt conditioning and cross-task generalization.

Critical strengths
  1. The benchmark covers classification, segmentation, language model adaptation, and reinforcement learning, including both model-weight and LoRA generation. This breadth enables evaluation beyond a small set of simple tasks and provides a useful resource for future research.
  2. The authors compare representative methods using shared target models, checkpoint pools, and downstream test sets, with repeated evaluations and documented configurations. This provides a common reference for previously fragmented evaluations and facilitates subsequent comparisons.
  3. The experiments reveal substantial performance differences across model scales, architectures, and tasks, helping identify challenging and informative evaluation settings. LPG provides a relatively simple reference method, while the additional conditioning and generalization experiments broaden the analysis.
Critical weaknesses
  1. The main experiments focus on parameter generation for known architecture–dataset combinations. The appendix includes experiments on unseen task combinations, but these primarily concern image classification. Extending this evaluation to other domains, such as LoRA adaptation, would further strengthen the benchmark’s relevance to new-task adaptation.
  2. The results demonstrate performance differences across tasks and architectures, but explanations such as the greater sensitivity of segmentation to parameter errors are mainly based on comparisons across different settings. Presenting these explanations as hypotheses requiring further controlled validation would make the conclusions more precise.
Questions for authors

In the cross-task experiments in the appendix, some methods also perform poorly on seen tasks. Could the authors briefly discuss whether the primary difficulty in these settings is learning the parameter distributions of the seen tasks, and how this should inform the interpretation of the unseen-task results?

Reviewer_SfUdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ParaGen-Bench to address the lack of standardized evaluation for neural network parameter generation methods. It aims to systematically compare the parameter generation capabilities of existing approaches. The authors construct a checkpoint dataset covering diverse tasks and model architectures, evaluate representative methods under unified target models and evaluation protocols, and analyze how model scale, architectural complexity, and task characteristics affect generation performance. In addition, the paper proposes LPG, a parameter generation baseline combining VAE-based weight encoding with language conditioning, to provide a reference for future research.

Critical strengths
  1. The benchmark provides a reasonably large and diverse dataset. The paper collects checkpoints across various tasks and model architectures, along with training metadata and task descriptions, providing a valuable resource for standardized evaluation in neural network parameter generation.

  2. The experiments offer broad coverage and useful insights. The paper evaluates multiple representative parameter generation methods and analyzes how model scale, architectural complexity, and task characteristics affect generation performance, helping to better understand the capabilities and limitations of existing approaches.

Critical weaknesses
  1. Insufficient validation of baseline adaptations and experimental reliability. To establish a unified evaluation setting, the authors make substantial modifications to several methods, including extending P-Diff to full-parameter generation, reimplementing TINA and HyperTransformer, and modifying SANE's training strategy. However, the paper lacks reproduction results under the original settings and sufficient validation of these modifications, making it difficult to distinguish the effects of the methods themselves from those of the experimental adaptations.

  2. Incomplete coverage of related work and representative methods. The paper overlooks relevant parameter generation methods, including the GHN family (e.g., GHN-2, GHN-3, LoGAH, and TAL) and PorTAL. Moreover, the classification of methods mixes generation mechanisms with conditioning types, making the categorization criteria unclear. These issues somewhat limit the benchmark's coverage of the neural network parameter generation field.

  3. Limited representativeness of the unified evaluation settings. To enable consistent comparisons, the authors primarily focus on relatively small-scale and less challenging full-parameter generation tasks, with limited evaluation on larger datasets and more complex models, such as ImageNet-1K. Some existing methods have already demonstrated parameter generation capabilities in more challenging settings. Therefore, the current experiments may not fully reflect their performance and scalability.

Questions for authors
  1. For methods that have been reimplemented or modified, such as P-Diff, TINA, HyperTransformer, and SANE, could the authors provide reproduction results under their original experimental settings and conduct ablation studies to validate the key modifications? How can the authors ensure that the observed performance differences primarily reflect the capabilities of the methods themselves rather than the effects of experimental adaptations or training configurations?

  2. The authors categorize existing methods into diffusion-based, hypernetwork-based, and LLM-conditioned approaches, but this classification mixes generation mechanisms with conditioning types. Could the authors clarify the criteria for this categorization? Additionally, why are relevant parameter generation methods, such as the GHN family (e.g., GHN-2, GHN-3, LoGAH, and TAL) and PorTAL, not included? Could the authors evaluate additional representative methods or further clarify the scope of the benchmark?

  3. The unified evaluation settings mainly focus on relatively small models and less challenging tasks, while some existing methods have demonstrated parameter generation capabilities in more challenging settings. Could the authors explain the rationale behind this choice? Could they include representative experiments on larger datasets and models, such as ImageNet-1K and ViT-Base, to examine whether the current conclusions generalize to more complex parameter generation tasks?

Reviewer_pun8OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces ParaGen-Bench, a benchmark for neural network parameter generation. It contains 44,000 checkpoints across 41 architectures, 29 datasets, 285 architecture–dataset settings and seven task domains, each checkpoint annotated with 50 natural-language prompts (30/10/10 split) and structured metadata (training configuration, optimisation trajectory, parameter statistics, downstream performance). Ten methods spanning diffusion, hypernetwork and LLM-conditioned generators are evaluated under a shared protocol: same target model, same checkpoint pool, same downstream test set, downstream metric rather than reconstruction error, and 5 seeds × 10 generations. The authors also add LPG, a two-stage baseline that trains a TokenVAE over structure-aware weight tokens and then an MLP projector from a frozen LLM prompt embedding into the VAE latent. The main findings are that gaps grow with target size and layer-type diversity, that segmentation and RL degrade far more than classification for the same parameter error, and that LoRA generation is easier.

Critical strengths
  1. The benchmark fills a real gap, and the curation is substantial. Existing model zoos cover at most two task domains and a handful of architectures, so cross-domain and cross-architecture comparison has simply not been possible. More important than the raw size is the unified evaluation: same target, same pool, same test set, downstream metric instead of reconstruction error, repeated sampling, and a sensible labelling of settings that are not scalable or structurally incompatible. That is what makes the numbers comparable at all, and I expect people will use this.

  2. The findings are actionable rather than leaderboard-only. Three stand out. (i) The BERT-tiny versus MobileNetV3 contrast at similar parameter counts (about 3.2M versus 4.2M) cleanly separates size from layer-type diversity, and the diagnosis (BatchNorm statistics, depthwise convolutions, mixed layer types) is concrete enough to act on. (ii) The task-sensitivity result — segmentation and RL degrading far more than classification — comes with a mechanistic account (per-pixel decisions, taking the maximum over action values) that explains why, which I found convincing. (iii) The LoRA objective finding, that reconstructing the full adaptation matrix beats reconstructing the two low-rank factors separately, is directly useful to anyone building these generators. I would also single out the cross-task generalisation study in Appendix B.1: it is arguably the most consequential result in the paper.

Critical weaknesses
  1. The LPG claim is a little broader than the tables support. The abstract says "competitive performance across diverse task domains", but LPG collapses on TinyImageNet (0.50), GoEmotions (8.19) and Llama on GSM8K (32.47 versus 59.07 trained). This is a wording issue rather than a methodological one — LPG works well as a reference baseline — but the claim should be scoped. Relatedly, the main benchmark measures fidelity to a single converged solution plus description-level prompt generalisation; the task-level generalisation result lives in B.1 and would be stronger in the main text, since it is the one result that really bounds what current methods can do.

  2. Baseline comparison could be tightened slightly. Training budgets differ widely across methods, two baselines are reimplementations, and the prompt encoders differ (Qwen3-VL-8B for LPG versus CLIP ViT-B/32, gte-large and MiniLM for others), so generator design and encoder quality are not fully separated. A shared-encoder control would remove this ambiguity. I would also like a rough amortisation number, since "trains once, generates many" is a stated motivation but only inference cost is reported.

Questions for authors
  1. Could you scope the LPG claim in the abstract, or add a per-domain summary row so readers can see where it is competitive and where it is not?

  2. For the prompt-conditioned comparisons, would a control with one shared frozen encoder (applied to LPG and the prompt-conditioned baselines) be feasible? It would separate generator architecture from encoder quality.

  3. Would you consider moving the cross-task generalisation study in B.1 into the main text? The contrast between dense conditioning (RPG 95.24, LPG 96.90 unseen) and sparse conditioning (most methods at chance) seems like the paper's most important scientific message.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussions

Dear Area Chair and Reviewers,

We appreciate Area Chair and Reviewers for handling our paper.

Our paper introduces ParaGen-Bench, a benchmark for neural network parameter generation with 44,000 trained checkpoints across seven task domains, 41 architectures, and 29 datasets. We evaluate ten representative methods under a unified protocol and propose LPG as a reference baseline.

We thank the reviewers for acknowledging that ParaGen-Bench fills an emerging and important research gap (pun8, SfUd, ju9f) in the lack of standardized evaluation for neural network parameter generation methods. Reviewers also acknowledge the insights and actionable findings of our paper (pun8, SfUd, ju9f), where we provide in-depth discussion on the effects of target size, layer-type diversity, and task characteristics on generation performance. Furthermore, reviewers believe that the open-source ParaGen-Bench will significantly contribute to the community and serve as a useful resource for future research (pun8, SfUd, ju9f).

We appreciate the reviewers’ constructive comments, and we have addressed them in the rebuttal accordingly. We will incorporate all the suggestions and additional experiment results presented in our rebuttal into the final version of our paper. We will also make all the data, code, and documentation public and maintain them continuously on GitHub and Hugging Face for the community.

We sincerely thank the Area Chair and Reviewers again for their time, valuable suggestions, and insightful advice, which have greatly helped us improve our paper.

Sincerely,

All Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.