- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ParaGen-Bench for systematically evaluating neural network parameter generation. The benchmark comprises 44,000 checkpoints spanning seven task domains, 41 architectures, and 29 datasets, together with natural-language descriptions and training metadata. The authors compare ten representative methods using common target models and downstream evaluation settings, and introduce LPG, a baseline combining structure-aware weight encoding, a VAE, and language-conditioned latent generation. The experiments examine performance across tasks, model scales, and architectures, with additional analyses of prompt conditioning and cross-task generalization.
Critical strengths
- The benchmark covers classification, segmentation, language model adaptation, and reinforcement learning, including both model-weight and LoRA generation. This breadth enables evaluation beyond a small set of simple tasks and provides a useful resource for future research.
- The authors compare representative methods using shared target models, checkpoint pools, and downstream test sets, with repeated evaluations and documented configurations. This provides a common reference for previously fragmented evaluations and facilitates subsequent comparisons.
- The experiments reveal substantial performance differences across model scales, architectures, and tasks, helping identify challenging and informative evaluation settings. LPG provides a relatively simple reference method, while the additional conditioning and generalization experiments broaden the analysis.
Critical weaknesses
- The main experiments focus on parameter generation for known architecture–dataset combinations. The appendix includes experiments on unseen task combinations, but these primarily concern image classification. Extending this evaluation to other domains, such as LoRA adaptation, would further strengthen the benchmark’s relevance to new-task adaptation.
- The results demonstrate performance differences across tasks and architectures, but explanations such as the greater sensitivity of segmentation to parameter errors are mainly based on comparisons across different settings. Presenting these explanations as hypotheses requiring further controlled validation would make the conclusions more precise.
Questions for authors
In the cross-task experiments in the appendix, some methods also perform poorly on seen tasks. Could the authors briefly discuss whether the primary difficulty in these settings is learning the parameter distributions of the seen tasks, and how this should inform the interpretation of the unseen-task results?
Summary of the Discussions
Dear Area Chair and Reviewers,
We appreciate Area Chair and Reviewers for handling our paper.
Our paper introduces ParaGen-Bench, a benchmark for neural network parameter generation with 44,000 trained checkpoints across seven task domains, 41 architectures, and 29 datasets. We evaluate ten representative methods under a unified protocol and propose LPG as a reference baseline.
We thank the reviewers for acknowledging that ParaGen-Bench fills an emerging and important research gap (pun8, SfUd, ju9f) in the lack of standardized evaluation for neural network parameter generation methods. Reviewers also acknowledge the insights and actionable findings of our paper (pun8, SfUd, ju9f), where we provide in-depth discussion on the effects of target size, layer-type diversity, and task characteristics on generation performance. Furthermore, reviewers believe that the open-source ParaGen-Bench will significantly contribute to the community and serve as a useful resource for future research (pun8, SfUd, ju9f).
We appreciate the reviewers’ constructive comments, and we have addressed them in the rebuttal accordingly. We will incorporate all the suggestions and additional experiment results presented in our rebuttal into the final version of our paper. We will also make all the data, code, and documentation public and maintain them continuously on GitHub and Hugging Face for the community.
We sincerely thank the Area Chair and Reviewers again for their time, valuable suggestions, and insightful advice, which have greatly helped us improve our paper.
Sincerely,
All Authors