acceptodds
Withdrawn from ICLR 2027

When Does Neural Weight Generation Work? Support, Coordinates, and Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Neural weight generation promises to reuse trained checkpoints, but when does it produce reliable samples rather than a strong best result? We study performance-weighted diffusion on CIFAR-10 CNN and ViT model zoos, varying checkpoint support, generation coordinates, and denoiser capacity. At fixed checkpoint count, replacing one trajectory with three independent runs removes –% of generated accuracy gain above chance under raw-weight coefficient diffusion, despite comparable aggregate denoising loss. Dimensional sweeps expose a normalization artifact that suppresses off-span perturbations, allowing functional outputs despite loss near the zero-predictor baseline. Greater capacity improves higher-dimensional generation, but unequal denoiser fit and exhausted training budgets leave an independent dimensionality penalty unresolved. Whitened single-trajectory span diffusion is the strongest learned custom-zoo generator evaluated, although simple baselines remain competitive and steering generally weakens with parameter-space novelty. On aligned public-zoo supports spanning hundreds of runs, full-latent U-Net diffusion recovers TiltDiff's reported accuracy scale using our own pipeline, with condition-mean Top-1@108 of – across 18 fresh-seed confirmation runs. However, distance-matched noise achieves higher mean accuracy in every run, while exponential weighting improves the upper tail at the cost of lower mean accuracy and yield. These results identify pipeline-specific sensitivities and show why weight-generation evaluation must assess typical-sample quality alongside best-of-samples performance.

Primary area:
generative models
Keywords:
Neural weight generation, optimization basins, weight-space geometry, diffusion models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FZMmOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This submission explores the application of diffusion models to weight generation for vision models (CNN and ViT). The authors explore the influence of dataset composition and architecture of the generative model on the generated samples. Potentially most interestingly, they compare average performance across sampled weights with a top-1 metric, concluding that high performance might be driven by tail events rather than the bulk of the distribution.

While the contributions that the authors claim to make are interesting, I found the manuscript exceedingly hard to parse to the extent that it impacts my ability to verify the soundness of the results (e.g., due to many undefined terms and insufficient explanations in the main text). As I was, despite repeated efforts, not able to convince myself that this paper makes a significant and sound contribution, I cannot champion this paper and recommend rejection.

The research questions that the authors ask are interesting and the contributions (as listed in the abstract and contribution section) could be impactful for the weight space learning community. Especially, the result that reported accuracy in weight generation might be driven by tail events is interesting.

I found the paper extremely hard to parse to the extent that I am unable to thoroughly verify critical parts of it. It took significant effort to understand the experimental setup and evaluation in detail. This is partially due to the large number of undefined terms throughout the manuscript, and partially due to a writing style that aims to be overly concise.

I do not have any questions to the authors that would materially affect my assessment of their work.

Critical strengths

The research questions that the authors ask are interesting and the contributions (as listed in the abstract and contribution section) could be impactful for the weight space learning community. Especially, the result that reported accuracy in weight generation might be driven by tail events is interesting.

Critical weaknesses

I found the paper extremely hard to parse to the extent that I am unable to thoroughly verify critical parts of it. It took significant effort to understand the experimental setup and evaluation in detail. This is partially due to the large number of undefined terms throughout the manuscript, and partially due to a writing style that aims to be overly concise.

Questions for authors

I do not have any questions to the authors that would materially affect my assessment of their work.

Reviewer_kPFLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when diffusion-based neural weight generation can produce functional models. Based mainly on TiltDiff, the authors investigate how checkpoint composition, coordinate normalization, and denoiser capacity affect generated model accuracy. They also compare diffusion-based generation with simple baselines such as checkpoint retrieval, interpolation, averaging, and noise perturbation.

The experiments suggest that generating weights from a single training trajectory can be easier than combining checkpoints from independent runs in the tested coefficient-diffusion pipeline. The authors also find that normalization and denoiser capacity can strongly affect generation results. Finally, experiments on a public model zoo show that performance weighting can improve best-of-sample accuracy while reducing average generated model quality.

Critical strengths
  • The authors consider several aspects of weight generation, including training checkpoint composition, representation, normalization, and model capacity. Some observations, particularly the large performance gap between single-trajectory and multi-run generation, are interesting.

  • The comparison with simple baselines such as retrieval, interpolation, averaging, and noise perturbation is useful. The distinction between best-of-sample performance and average generation quality is also worth highlighting.

Critical weaknesses
  • The main contribution is not sufficiently clear. The paper presents a collection of empirical observations rather than a coherent methodological or theoretical contribution. Many of the investigated factors, such as checkpoint alignment, dimensionality, normalization, and model capacity, are already known to affect weight-space learning. It is unclear what new general insight this work establishes beyond the specific experimental configurations.

  • The experimental scope is limited. All experiments are conducted on CIFAR-10, and most custom model experiments involve relatively small CNN and ViT architectures. The public model zoo experiments use a single CNN architecture with approximately 11K parameters. Given that neural weight generation is intended to be useful for more complex models, it is difficult to assess whether the findings generalize beyond these small-scale settings.

  • The presentation needs substantial improvement. The manuscript is difficult to follow, with many experimental settings, representations, and evaluation protocols introduced throughout the paper. The discussion frequently moves between different pipelines, making it difficult to understand which conclusions are supported by which experiments. The main findings are also expressed in rather complicated language. A clearer organization and more direct explanation of the research questions and conclusions would significantly improve readability.

Questions for authors
  • What is the main contribution of this work? The paper investigates several different factors affecting neural weight generation, but I find it difficult to identify a central research question or a general conclusion. Could the authors clarify which findings they consider the main contribution and how they advance our understanding beyond existing studies?

  • Why are all experiments restricted to CIFAR-10 and relatively small neural networks? Have the authors considered evaluating on larger architectures or additional datasets?

  • Practical advantage over simple baselines. In Sections 3.3 and 4.3, averaging, retrieval, and noise perturbation are competitive with diffusion-based generation. What is the main practical benefit of using a learned generator in these settings? Can the authors provide an experiment where the learned generator offers a clear advantage over these simpler approaches?

Reviewer_UwgTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how generation coordinates, and denoiser capacity affect neural weight generation on CIFAR-10 model zoos. It examines how conclusions change when evaluating average accuracy, functional yield, best-of-samples performance, novelty, and accuracy steering. A public-zoo experiment shows that performance weighting can improve upper-tail accuracy while substantially reducing typical-sample quality.

Critical strengths
  1. The paper addresses an evaluation problem and includes useful retrieval, interpolation, perturbation, and averaging baselines.

  2. The fixed-checkpoint-count experiments across CNN and ViT model zoos provide useful evidence that checkpoint count alone is insufficient to characterize training support. The controlled weighting comparison in Section 4.3 provides a concrete example of why best-of-samples accuracy alone is insufficient.

Critical weaknesses
  1. The geometric explanation combines evidence from different representations. In Section 3.1, single-trajectory variance concentration is measured in codec latent space, whereas the fixed-checkpoint-count generation experiments use standardized raw weights. The connection between these observations is therefore not directly established.

  2. The claim of comparable denoising losses lacks direct quantitative support in the main paper. The dimensional experiments do not clearly identify independent dimensionality effect.

Questions for authors
  1. Does the geometric explanation hold when all supports are analyzed in the same representation?

  2. How much functionality is preserved by reconstruction alone?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.