acceptodds
Under review as a conference paper at ICLR 2027

Structural Preservation of Data in Implicit Neural Representations via the Implicit Function Theorem

Abstract

Implicit Neural Representations (INRs) parameterize signals of interest as coordinate-based mappings, where each data sample is uniquely encoded in a specific set of weights. This data-to-weight association provides a convenient platform for studying Weight Space Learning (WSL), an emerging field with potential for tasks like meta-learning or transfer learning. So far, a precise theoretical explanation of how the underlying data structure is preserved in the learned weights is still missing. In this work, we employ the Implicit Function Theorem (IFT) to establish a rigorous correspondence between the data space and its latent weight representation space. We analyze a framework that maps instance-specific latent codes to INR weights via a shared hyper-network. Empirically, we construct a synthetic Fourier-curve testbed to verify data structure recovery. Furthermore, we demonstrate competitive performance against existing approaches on INR weight classification across 2D and 3D datasets using more compact networks. The present work offers a theoretical lens for future investigation into WSL.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Implicit Function Theorem, Weight Space Learning, Implicit Neural Representation, Hyper-network

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Nm8qOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper formalize the mapping from data samples to latent codes/weights in HN-INR using the Implicit Function Theorem (IFT), derive the local non-degeneracy conditions, and interpret ℓ2\ell_2 regularization and reconstruction error as two operational levers for controlling the condition number of this mapping. The authors further validate structure preservation and downstream utility with synthetic Swiss roll experiments and 2D/3D classification benchmarks.

Critical strengths
  1. The presentation is clear.
  2. The theoretical modeling is clear. The authors formulate the training of HN-INR as an implicit function, and connect local embeddings with downstream tasks via the non-degeneracy of D1ξD_1\xi. The “trivial intersection of subspaces” in Condition 2 admits a geometric interpretation.
  3. The synthetic experiments support the arguments. With known ground-truth structure on the Swiss roll dataset, the authors verify latent recovery and neighborhood preservation (kNN Jaccard index = 0.732). And classification accuracy reaches 91.0%, compared with 93.2% using ground-truth representations. Such synthetic benchmarks provide stronger evidence for the theory than real-world benchmarks.
Critical weaknesses
  1. The classification experiments cannot strongly establish the causal claim that downstream gains arise from structure preservation induced by IFT. Baseline results are taken from original papers, where training protocols, classifier capacity and data augmentation may differ. The proposed method employs a classifier with 11.5k / 2.15M parameters, while HMoE uses a Transformer+MoE architecture. The comparison is not performed under matched capacity and compute budgets. The inconsistency between κ\kappa/kNN metrics and accuracy in Table 4 also suggests that the structure-preservation metric alone does not suffice to explain performance. I think that the authors should provide ablations under identical protocols and matched model capacity, or mediation analysis to examine whether classification performance degrades when controlling for κ\kappa.
  2. The branch assumption in Proposition 1 is not empirically validated. The globally smooth extension requires that the fitting codes of the same class lie within the same connected component / branch. In practice, it remains unverified whether latents from each class reside in a single branch. Classification performance may stem from clustering across distinct branches rather than a single smooth mapping. Reporting per-class branch consistency would provide direct support for the proposition.
Questions for authors

Please refer to weaknesses.

6 comments

Thank you for recognizing our clear presentation and theoretical modeling.

W1:

  • W1.1: We agree that the connection between the Implicit Function Theorem (IFT) and our empirical results needs to be strengthened. To this end, we added two theoretical extensions in the revised Appendix. First, we establish a bi-Lipschitz bound (Appendix D): σmin⁡>0\sigma_{\min} > 0 provides an upper bound that keeps nearby samples close, while the reconstruction MSE\mathrm{MSE} gives a lower bound that prevents distant samples from collapsing. Second, we derive an explicit O(MSE)\mathcal{O}(\sqrt{\mathrm{MSE}}) bound on the residual curvature perturbation (Appendix E), which guarantees that the Hessian remains positive-definite under imperfect reconstruction. Our IFT framework is not an architecture-level modification on top of baselines to gain a few points. Instead of relying on permutation-invariant architectures, we provide an alternative mathematical instrument to analyze how data structure is preserved in weight space. Our lightweight architecture and small classifier achieve competitive results, which empirically validates that our IFT analysis leads to a well-structured weight representation.

  • W1.2: In our response to Reviewer vCtd (Q2), we summarize the parameter counts and computational costs across methods. Because of our different perspective on mathematical formulation, we do not need a large permutation-invariant classifier. This is our core message to the WSL community: analyzing the data-to-weight relationship via the IFT provides a complementary view to constructing weight quotient spaces. Regarding HMoE on CIFAR-10, the gap stems from manifold complexity and supervision. Natural images have a much higher intrinsic manifold dimension m∗m^{\ast}, which by Lemma 1 (l≥m∗l \ge m^{\ast}) demands a larger latent space and denser sampling. Moreover, HMoE use meta-learning with a large Mixture-of-Experts (MoE) Transformer architecture, and they include a classification term in their training loss that directly shapes weights with class labels. In contrast, our representations are learned purely via self-supervised reconstruction loss. This explains their stronger performance on CIFAR-10. However, once we add data augmentation (sample the manifold more densely), our method achieves comparable results.

  • W1.3: We acknowledge that our notation in Table 4 was incomplete: the reported κ\kappa is the worst-case (maximum) value across samples, reflecting a single outlier. We reported this worst-case condition number to verify that the Jacobian of the implicit function D⁡1ξ(z,X)\operatorname{D}_1 \xi(\boldsymbol{z}, \boldsymbol{X}) is full rank and satisfies the IFT condition for every sample in the dataset. Because a single worst-case outlier does not represent overall classification performance, Table 4 is not ideal for inferring any accuracy trend. Theoretically, from the closed form of D⁡g\operatorname{D}g derived in Appendix D, local structural preservation and downstream accuracy depend on two factors. First, full rank (σmin⁡(D⁡1ξ(z,X))>0\sigma_{\min}(\operatorname{D}_1 \xi(\boldsymbol{z}, \boldsymbol{X})) > 0) is a binary condition ensuring that gg locally exists. Second, the reconstruction error MSE\mathrm{MSE} directly controls sample separation in the bi-Lipschitz lower bound, while the latent dimension ll of J~⊤∈Rl×nc\tilde{\boldsymbol{J}}^\top \in \mathbb{R}^{l \times nc} limits reconstruction capacity when ll is too small. Thus, the IFT only requires D⁡1ξ(z,X)\operatorname{D}_1 \xi(\boldsymbol{z}, \boldsymbol{X}) to be full rank, and κ\kappa serves only as a numerical check against singularity. Empirically, we conducted an ablation study over ll and λ\lambda on MNIST (Appendix F). When ll is too small, κ\kappa is small, but ll cannot cover the intrinsic manifold dimension m∗m^*, leaving insufficient structure for classification. When ll is too large, reconstruction improves, but κ\kappa grows and the full-rank condition becomes less robust, causing accuracy to drop. Across a wide range of λ\lambda (from 00 to 0.30.3), κ\kappa varies substantially while accuracy remains nearly unchanged (97.597.5\\% to 97.797.7\\%), confirming our theoretical analysis.

W2: We acknowledge that our wording in Section 4 was imprecise and caused confusion regarding the branch assumption. We have revised Section 4 and Proposition 1 in the revised version (see also our response to Reviewer 7K6H, Q3/W3):

  • Local sub-components vs. label classes: In X=⨆k=1KXk\mathcal{X} = \bigsqcup_{k=1}^K \mathcal{X}_k, calling each Xk\mathcal{X}_k a "class" was misleading. Each Xk\mathcal{X}_k denotes a local connected semantic sub-component (K≫CK \gg C), and a single downstream label class can naturally span multiple disjoint branches. Proposition 1 does not require fitted latent codes of an entire label class to reside in a single branch. It simply extends the point-wise IFT to each local connected region Xk\mathcal{X}_k so that nearby samples map smoothly without shattering.
  • Clustering across branches and non-collapse: Because one label class may span multiple local branches, downstream classification naturally groups distinct branches of the same class, which is fully consistent with Proposition 1. Meanwhile, separation between distinct branches and classes is guaranteed by our global bi-Lipschitz lower bound in Appendix D: samples differing by more than the reconstruction MSE\mathrm{MSE} cannot collapse into the same latent code.

Thanks for the authors' response. My concerns on the experiments are still unresolved, so I will keep my score.

Reviewer_vCtdOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies when data geometry is preserved in representations obtained through a shared hypernetwork that generates INR weights from instance-specific latent codes. It applies the implicit function theorem to investigate the relationship between data samples and their latent representations under the HN-INR model, deriving sufficient conditions for local embeddings under exact reconstruction. Experiments report structural recovery on a synthetic dataset and classification results on image and shape INR benchmarks.

Critical strengths
  1. Evaluation across multiple datasets with repeated runs: Tables 1-2 evaluate classification across image and shape INR benchmarks, helping assess the breadth of the approach’s applicability. The authors report mean and standard deviation over five runs for their method.
  2. Illustrative synthetic experiment: Figure 3 compares the learned latent representations with the known structure used to generate the synthetic data. This provides a visual illustration of the structural preservation that the paper aims to study.
Critical weaknesses
  1. The comparison does not clearly show the benefit of the proposed INR construction: The proposed method uses newly generated INR weights, while Tables 1–2 report published baseline scores obtained on different INR weights. Therefore, the accuracy differences could come from the representations, the classifiers, or other training differences. The comparison does not isolate the benefit of the proposed INR generation method. To test that benefit, the authors should evaluate the same representative classifiers on both the proposed and reference INRs under matched settings.

  2. The claim of structural preservation under imperfect reconstruction is insufficiently supported and should be qualified: At the end of Section 5.3 (page 9, lines 450–452), the authors conclude that a suitable latent dimension and tuned regularization preserve the data structure. However, the embedding guarantee in Lemma 1(b) requires exact reconstruction throughout a neighborhood, whereas the experiments have nonzero reconstruction errors. With imperfect reconstruction, the analysis establishes a locally smooth mapping, but smoothness alone does not ensure that different samples remain distinguishable in latent space. Different nearby samples can receive the same code while reconstruction errors remain small. The results in Table 4 provide evidence for the particular synthetic experiment, but do not establish the broader conclusion. Please narrow this statement to the observed results or clarify what additional conditions guarantee structural preservation.

Questions for authors
  1. Theoretical motivation and relevance to WSL: The introduction states that "Despite empirical successes of these attempts, a solid mathematical foundation of WSL is still largely lacking". However, there is already substantial theoretical literature on WSL, as discussed in A Survey of Weight Space Learning: Understanding, Representation, and Generation (Han et al., 2026). Could the authors clarify the specific theoretical gap their work addresses and better motivate its relevance to WSL? In particular, since WSL commonly focuses on learning from the weights of existing trained models, what is the motivation for generating new, structure-preserving INR weights for classification? Which WSL tasks would benefit from this approach, rather than directly learning from existing model weights or the original data?

  2. The authors emphasize the small parameter count of their downstream classifiers (e.g., 11.5k parameters). However, the full framework also requires a shared hypernetwork, per-instance latent codes, and test-time optimization of these codes (Figure 1 and Appendix Table 5). Could the authors report the total computational cost, including hypernetwork parameters, training costs, and test-time optimization, and clarify whether the proposed approach remains computationally advantageous when compared with baselines methods under comparable accounting?

8 comments

Q1: We acknowledge that our statement in the introduction was overly broad. We will refine this phrasing in the revision. We are fully aware of the pioneering survey by Han et al. (2026), which systematically identifies many meaningful research questions and open challenges in the Weight Space Learning (WSL). We have also cited this survey in our submitted version.

  • Q1.1: Our paper does not seek to propose a new training method or architecture, as hypernetwork-based INRs (HN-INRs) are already widely used. Instead, we use the HN-INR framework as an instrument to demonstrate that the Implicit Function Theorem (IFT) can mathematically bridge the data space and the weight/latent space. Our motivation is that neural network weights are fundamentally rooted in the training process and the data used to train them. As a complementary alternative to post-hoc weight analysis, we use the IFT to characterize, verify, and guarantee how data structure is preserved in weights, which constitutes our theoretical contribution to WSL.

  • Q1.2: While many WSL methods work post-hoc on pre-trained model zoos to build symmetry quotient spaces, many representative works covered in the survey itself (such as Functa, inr2vec, MWT, and HMoE) also generate weights online during training. For tasks aiming to establish data-weight connections, adding simple constraints during training (such as a low-dimensional regularized parameterization under our IFT conditions) is beneficial.

  • Q1.3: Evaluating on INR classification is not intended to offer a replacement for conventional raw-data classifiers. Instead, it serves as the canonical testbed or a controlled case study across the WSL literature. Moreover, as noted in Han et al. (2026), WSL is closely related to areas such as continual learning and meta-learning, which also touch upon the relationship between data and weights (for example, when new data requires new weights). From this perspective, the tools and viewpoint of the IFT can also be inspiring for these broader WSL applications.

W2: We agree with your suggestion and have narrowed our conclusion in Section 5.3 to explicitly reflect these limitations, with detailed derivations provided in Appendices D and E. To formalize structural preservation under imperfect reconstruction, we establish a bi-Lipschitz guarantee for the mapping g(X)g(\boldsymbol{X}) from data space to latent weight space z\boldsymbol{z}. Intuitively, this ensures that data points cannot be torn apart in representation space (upper bound), nor can distinct samples collapse into the same code as long as their signal distance exceeds the reconstruction error floor (lower bound). Furthermore, we demonstrate in Appendix E that the higher-order residual curvature term is strictly bounded by O(MSE)\mathcal{O}(\sqrt{\mathrm{MSE}}), guaranteeing that the Hessian remains positive-definite.

Q2: In WSL, INR classification is primarily a testbed to study semantic preservation rather than an engineering benchmark for speed. In practice, all weight-space methods require test-time optimization on unseen samples, because INR classifiers take network weights/latents rather than raw pixels as input.

The following table summarizes these methods on MNIST. "Repr. Params (Training)" accounts for all parameters trained and saved to represent training data, including hypernetwork plus latent codes for ours, saved SIREN collections for post-hoc methods, and meta-initializations for meta-learning. "Classifier Params" and "Total Params" report downstream classifier size and overall model footprint respectively. "Test-Time Opt." indicates the per-sample optimization required to encode an unseen test image. Here, post-hoc methods fit an entire SIREN from scratch via Adam, meta-learning methods (MWT, HMoE) run 6 inner-loop SGD adaptation steps from their learned weight anchor, and our method freezes the hypernetwork to optimize only the compact latent code zz, which forms a well-conditioned problem solved in parallel batches. "Test Acc." gives classification accuracy on MNIST.

MethodRepr. Params TrainingClassifier ParamsTotal ParamsTest-Time Opt. (per test sample)Test Acc. (%)
Ours3.48M (2.28M Hypernetwork + 60k×20d60\text{k}\times 20\text{d} latent)0.01M (10.7k MLP)3.49M400 Adam (z∈R20z \in \mathbb{R}^{20}, frozen Hypernetwork)97.69%
MWT-L (CVPR'25)1.80M (11.9B updates per epoch)4.30M (10-layer Transformer)6.10M6 inner SGD steps (w∈R198kw \in \mathbb{R}^{198\text{k}})98.80%
HMoE-MWT (ECCV'26)0.47M (3.0B updates per epoch)11.0M (10-layer Mixture-of-Experts Transformer)11.47M6 inner SGD steps (w∈R50kw \in \mathbb{R}^{50\text{k}})99.06%
DWSNet (ICML'23)71.10M (saved SIRENs)0.55M (4-layer equivariant network)71.65M1,000 Adam (w∈R1.2kw \in \mathbb{R}^{1.2\text{k}})85.71%
NFN (NeurIPS'23)533.25M (10×10\times SIRENs)1.8M to 16M (equivariant NFN + MLP)535.1M+5,000 Adam (w∈R1.2kw \in \mathbb{R}^{1.2\text{k}})92.90%
Inr2Array (NeurIPS'23)555.25M (533.25M SIRENs + 22M NFT-Hypernetwork)38.0M (12-layer Transformer)593.3M5,000 Adam (w∈R1.2kw \in \mathbb{R}^{1.2\text{k}})98.50%

A more detailed training cost comparison will be reported in the revised Appendix.

Thank you for the detailed responses and revised manuscript. My remaining comments are:

W2: Largely resolved. I appreciate the new theoretical bounds and the more carefully qualified claims.

W1: Remains unanswered. A comparison applying the same downstream classifiers to INRs generated by different methods, under matched training settings, would help isolate the benefit of your representation construction.

Q1: If I understand correctly, your main contribution is using the IFT to characterize structural preservation between data and HN-INR representations. However, what is the practical motivation for establishing this guarantee? Could you provide a concrete use case where the IFT analysis would guide or improve weight generation? The broader applications mentioned in your response do not yet make this clear to me.

Q2: The new table provides useful parameter accounting, but does not establish a computational advantage. Could you report comparable end-to-end training costs and test-time optimization costs? This would help resolve my concern about the framework's computational efficiency.

Reviewer_7K6HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies structural preservation in weight-space learning using hyper-network-based implicit neural representations. The main idea is to treat the optimized latent code as an implicit function of the underlying data sample, and then use the Implicit Function Theorem to derive conditions under which this mapping is locally smooth and non-degenerate. Under stronger assumptions, the paper extends this to a component-wise global smooth mapping over each class. The method is evaluated on a synthetic Fourier-curve/Swiss-roll setup and several 2D and 3D classification benchmarks.

Critical strengths

I found the IFT perspective interesting and quite natural for studying how optimized latent codes vary with the underlying data.

The local embedding result is mathematically clean. Under the stated rank conditions and exact reconstruction assumption, the argument that the data manifold is smoothly embedded into the latent space seems sound.

The synthetic experiment is useful since the underlying latent geometry is known, and the learned representation recovers the Swiss-roll structure reasonably well.

The paper also checks the conditioning of the relevant Jacobian/Hessian empirically rather than only presenting the theoretical condition.

Critical weaknesses
  1. The theory establishes topological/local structural preservation, but the empirical claims are more geometric. A smooth embedding preserves the manifold structure, but it does not by itself imply preservation of distances, nearest neighbors, or local geometry. However, the experiments evaluate quantities such as k-NN Jaccard and classification accuracy. I think the paper should be more precise about this distinction. A quantitative bound on DgDg, or some local bi-Lipschitz type result, would connect the theory much better to the experiments.

  2. The non-zero residual analysis is missing an important quantitative step. The residual curvature term is EX∗(z)=2nc∑iϵi(Bi+Ci)E_X^* (z)=\frac{2}{nc}\sum_i \epsilon_i(B_i+C_i). The paper argues that reducing reconstruction error suppresses this term and improves non-degeneracy. This is intuitively reasonable, but small residuals alone are not sufficient unless the second-order terms Bi+CiB_i+C_i are also controlled. An explicit bound on ∣EX∗(z)∣|E_X^*(z)| in terms of reconstruction error and some curvature quantity would make Section 3.3 considerably stronger.

  3. The global result relies on fairly strong assumptions. Proposition 1 assumes that each class is connected and simply connected, that non-degeneracy holds everywhere along an entire connected branch of the critical set, and that the latent codes remain bounded. The experiments verify conditioning only at the fitted samples, which does not establish these assumptions along the full branch. Also, the proposition guarantees connectedness of the representation for each class, but does not imply separation between different classes. The paper mentions this, but I think the downstream classification interpretation should be somewhat more cautious.

  4. The classification results are competitive, but not clearly explained by the theory. The method performs well on several datasets, but on CIFAR-10 for example the large weight classifier reaches 56.5%, compared with 65.0% for HMoE. More importantly, it is difficult to attribute the classification behavior directly to the IFT conditions. The empirical results are supportive, but not a very direct validation of the theoretical mechanism.

Questions for authors

Can the authors derive a quantitative bound on DgDg, or a local bi-Lipschitz guarantee, to connect the IFT result with neighborhood preservation?

Can EX∗(z)E_X^*(z) be explicitly bounded using reconstruction error together with bounds on the second derivatives of the INR and hyper-network?

How can the assumptions of Proposition 1, especially non-degeneracy along the entire branch, be checked or approximated in practice?

Is there an empirical relationship between the condition number of D1ξD_1\xi and downstream neighborhood preservation/classification accuracy across samples or datasets?

2 comments

We thank the reviewer for recognizing our mathematical modeling via the Implicit Function Theorem (IFT). The questions are mainly about the theoretical details to make our IFT analysis more complete. The detailed explanations will be added into the revised version.

Q1 / W1: Our main message is not about specific geometric or topological properties, as they vary on different datasets or downstream tasks. We focus on high-level structural preservation. Intuitively speaking, nearby samples (in the data space) do not go far apart (in the weight space), while distant samples do not collapse together.

Your suggestion to examine D⁡g\operatorname{D}g and establish a bi-Lipschitz guarantee is very helpful. We provide the derived bounds in Appendix D in the revised version:

  • Upper bound (no tearing apart): Whenever the implicit Jacobian D⁡1ξ(z,X)\operatorname{D}_1 \xi(\boldsymbol{z}, \boldsymbol{X}) is non-degenerate (σmin⁡(D⁡1ξ(z,X))>0\sigma_{\min}(\operatorname{D}_1 \xi(\boldsymbol{z}, \boldsymbol{X})) > 0), the local Lipschitz constant satisfies Lmax⁡<∞L_{\max} < \infty. By the mean value inequality, ∣z1−z2∣2≤Lmax⁡∣X1−X2∣F\\|\boldsymbol{z}_1 - \boldsymbol{z}_2\\|_2 \le L_{\max} \\|\boldsymbol{X}_1 - \boldsymbol{X}_2\\|_F, guaranteeing that nearby data points are not torn far apart in the representation space.
  • Lower bound (no collapse): Accounting for non-zero reconstruction errors, we establish ∣z1−z2∣2≥1Λ(∣X1−X2∣F−nc MSE1−nc MSE2)\\|\boldsymbol{z}_1 - \boldsymbol{z}_2\\|_2 \ge \frac{1}{\Lambda} (\\|\boldsymbol{X}_1 - \boldsymbol{X}_2\\|_F - \sqrt{nc\,\mathrm{MSE}_1} - \sqrt{nc\,\mathrm{MSE}_2}). This guarantees that distinct samples exceeding the reconstruction noise floor cannot collapse into the same latent code.

Together, these two bounds provide the metric justification for our kk-NN Jaccard results.

Q4 / W4: From the closed form of D⁡g\operatorname{D}g in Appendix D, local structural preservation is influenced by two parts:

  • Full rank (σmin⁡(D⁡1ξ(z,X))>0\sigma_{\min}(\operatorname{D}_1 \xi(\boldsymbol{z}, \boldsymbol{X})) > 0) is a binary condition ensuring that gg locally exists.
  • Reconstruction error MSE\mathrm{MSE} directly affects the separation between samples in the Lipschitz lower bound. In addition, the dimension of J~⊤∈Rl×nc\tilde{\boldsymbol{J}}^\top \in \mathbb{R}^{l \times nc} matters. When ll is very small, it restricts the reconstruction capacity and leads to large MSE\mathrm{MSE}. Mathematically, IFT only requires D⁡1ξ(z,X)\operatorname{D}_1 \xi(\boldsymbol{z}, \boldsymbol{X}) to be full-rank. We report the condition number κ\kappa as a robust numerical validation against singularity, but κ\kappa does not monotonically govern downstream classification accuracy.

To examine this empirical relationship, we conduct an ablation study over ll and λ\lambda on MNIST test samples (the tables are provided in Appendix F in the revised version). In practice, the intrinsic data manifold dimension m∗m^* is typically unknown and we observe:

  • Small ll: The condition number is small, and the IFT condition is well satisfied. However, because ll cannot cover the unknown intrinsic manifold dimension, the weight space only captures partial data structure, which is insufficient for classification.
  • Large ll: Higher dimensions make it easier to fit all details in the data, continually improving reconstruction. Meanwhile, the condition number keeps growing, indicating that the full-rank condition of the Jacobian in IFT becomes less robust, and classification accuracy also drops.
  • Regularization λ\lambda: Across a wide range of λ\lambda, κ\kappa varies significantly while downstream classification accuracy remains virtually unchanged (∼97.5\sim 97.5\\% - 97.7\\%), confirming that κ\kappa does not monotonically govern accuracy once non-degeneracy holds.

The gap on CIFAR-10 comes from manifold complexity and classifier capacity. Natural images exhibit a much higher intrinsic manifold dimension m∗m^{\ast}, which by Lemma 1 (l≥m∗l \ge m^{\ast}) requires a larger latent space and higher sampling density to cover the manifold. Moreover, MWT and HMoE employ meta-learning with large Transformer and even Mixture-of-Experts (MoE) architectures. They have classification term in the loss functions, which have shaped weights by class labels. In contrast, our representations are learned purely via self-supervised reconstruction loss.

Q2 / W2: An explicit bound on ∣EX∗(z)∣2\\|E_{\boldsymbol{X}}^{\ast}(\boldsymbol{z})\\|_2 indeed makes Section 3.3 more complete. By the bounded latent code property (∣z∣2≤R<∞\\|\boldsymbol{z}\\|_2 \le R < \infty) and with hyper-network parameters v∗\boldsymbol{v}^{\ast} fixed, the generated weights w=ϕ(v∗,z)\boldsymbol{w} = \phi(\boldsymbol{v}^{\ast}, \boldsymbol{z}) remain bounded. Because both the SIREN network ff and hyper-network ϕ\phi (with SiLU activations) are C∞C^\infty smooth, their derivatives on the closed ball are continuous and bounded: ∣Bi+Ci∣2≤Mˉcurv<∞\\|\mathcal{B}_i + \mathcal{C}_i\\|_2 \le \bar{M}_{\mathrm{curv}} < \infty. For linear hyper-networks, such as in our Swiss-roll testbed, D⁡22ϕ=0\operatorname{D}_2^2 \phi = 0, so Ci=0\mathcal{C}_i = 0 identically.

In Appendix E, we derive this explicit bound on the residual curvature perturbation: ∣EX∗(z)∣2≤2MˉcurvMSE=O(MSE)\\|E_{\boldsymbol{X}}^*(\boldsymbol{z})\\|_2 \le 2\bar{M}_{\mathrm{curv}}\sqrt{\mathrm{MSE}} = \mathcal{O}(\sqrt{\mathrm{MSE}}). Since ∣λmin⁡(E)∣≤∥E∣2|\lambda_{\min}(E)| \le \|E\\|_2, as MSE→0\mathrm{MSE} \to 0 the residual curvature perturbation vanishes, causing D⁡1ξ(z,X)\operatorname{D}_1 \xi(\boldsymbol{z}, \boldsymbol{X}) to approach G+λIl≻0G + \lambda \boldsymbol{I}_l \succ 0.

Empirically, Table 3 demonstrates that even under non-exact reconstruction, the implicit function Jacobian D⁡1ξ(z,X)\operatorname{D}_1 \xi(\boldsymbol{z}, \boldsymbol{X}) consistently remains full-rank across diverse real datasets, confirming that EX∗(z)E_{\boldsymbol{X}}^{\ast}(\boldsymbol{z}) is well bounded in practice.

Q3 / W3: We agree that our phrasing in Section 4 was imprecise in a few places and caused unnecessary confusion. We will revise Section 4 carefully to clarify the following points:

  • Local sub-components Xk\mathcal{X}_k (K≥CK \ge C) vs. label classes: Calling Xk\mathcal{X}_k in X=⨆k=1KXk\mathcal{X} = \bigsqcup_{k=1}^K \mathcal{X}_k a "class" was misleading. Here Xk\mathcal{X}_k is a local connected semantic sub-component (K≫CK \gg C), and a single downstream label class can naturally span multiple disjoint local regions. Proposition 1 does not require an entire label class to lie on one branch. It simply extends the point-wise IFT to each local connected region Xk\mathcal{X}_k so that nearby samples map smoothly without shattering.
  • We agree that Proposition 1 guarantees only intra-component connectedness, whereas non-collapse between distinct classes follows from our lower bound in the response to Q1/W1.
  • While an entire continuous branch cannot be checked exhaustively, we can still verify non-degeneracy on test set with hyper-network v∗\boldsymbol{v}^{\ast} frozen (results on test set like training set in Table 3 can be seen in Appendix G in the revised version).

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

As the end of the discussion round approaches, we would like to thank all reviewers for their valuable and constructive feedback, particularly reviewers vCtd and Nm8q for their active engagement in our fruitful discussions to improve the quality of our paper.

It was a pity that we could not have the chance to have more engaging discussions about the comments raised by Reviewer 7K6H.

Here is a summary of the key statements and revisions that have been adopted in the revised version of our paper:

  • Clearer statement about the paper: Our paper is a theory-oriented work rather than a new architecture for pushing accuracy on benchmarks. Our work aims to offer an alternative lens to analyze the data-weight relationship in weight space learning. We hope this mathematical instrument can be inspiring for the newly emerging topic Weight Space Programming (Takehi et al., 2026). (vCtd-Q1)
  • Theoretical refinements & error bounds:
    • Local bi-Lipschitz bounds on DgDg formalizing non-tearing and non-collapse for better downstream classification. (7K6H-W1, Nm8q-W1)
    • Quantitative bounds on EX∗(z)E^*_X(z), the higher-order residual curvature term appearing in the implicit function Jacobian. (7K6H-W2, vCtd-W2)
    • Clarified Prop. 1 for local smooth mapping over multi-branch sub-components. (Nm8q-W2, 7K6H-W3)
  • Matched training protocols and runtime comparisons for fair evaluation. (vCtd-W1, vCtd-Q2, Nm8q-W1)

We hope this message could provide an overview and basis for the AC to coordinate discussions with the reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.