- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of editing continuous neural representations, e.g. INRs or neural ODEs. There are two general approaches: one is weight space learning, which directly maps network parameters to parameters, i.e., editing in the weight space; the other one is discrete-signal editing, which uses a separate model to operate on the discretized grid output, e.g. a network to learn to rotate an image represented as a structured grid. However, using standard CNNs or ViTs completely destroy the resolution independence that made continuous neural representations useful in the first place. This paper simply propose to use Neural Operators instead of e.g. CNNs or ViTs in the second approach. Experiments show that this workflow can rotate 27 x 27 images despite trained under 28 x 28.
Critical strengths
None
Critical weaknesses
-
Despite the continuous mathematical formulation of FNO can work with arbitrary discretizations, FNO itself cannot. See [1, 2]. For the image rotation example, if you do 512 x 512 or 1024 x 1024 in CelebA-HQ for training and then test on 64 x 64, FNO results will be very bad. Similarly, for the ODE example, if you have a very non-smooth function, DeepONet and FNOs would fail too. Evaluating on adjacent midpoints within the exact same spatial support and bandlimit is merely local interpolation. You can do INF + CNNs/ViTs + Interpolation, e.g. Fourier interpolation to achieve similar results.
-
This paper is simply applying neural operators to replace CNNs/ViTs. There is no fundamental algorithmic or architectural innovation introduced.
-
Experiments are done entirely using MNIST and a 1D synthetic harmonic oscillator. I do not see much value in these evaluations in terms of how generalizable it is across resolutions/discretizations, how generalizable it is across different smoothness levels of the functions.
[1] Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning
[2] Discretization-invariance? On the Discretization Mismatch Errors in Neural Operators
Questions for authors
-
Why is this work still called weight-space learning? I do not see any learning or editing in the weight space.
-
FNO has 16M parameters while all others have around 1M, can you do a comparison with similar model scale to compare FNO with CNN/ViT + interpolation?
3 comments
Response (2/2) to Reviewer wmAL
These results show that:
- The FNO's learned map transfers, unlike the CNN's. Run natively on the rotation at 21, 35 and 42 points, the FNO's error is 2- its training-grid error, while the CNN does worse than no edit. Both fail at 14 points, which undersamples a signal fitted at .
- On the swirl and dilation, the native FNO is better on finer grids. At 35 and 42 points it beats every interpolated pipeline by 1.25- in every seed. Interpolation is still better at 14 points, and for the dilation also at 21.
- Editors that output a function are unaffected by the grid. DeepONet's error varies by a factor of 1.16 across grids and DWSNet's by 1.02, although DeepONet is less accurate than the interpolated grid models.
We will change our claim to: "The FNO's learned map transfers across grids. On finer grids its error rises slightly with resolution, consistent with the discretization mismatch errors in [2]; on coarser grids it grows much faster, as the grid can no longer represent the signal [1]."
W2. Novelty.
This paper is simply applying neural operators to replace CNNs/ViTs. There is no fundamental algorithmic or architectural innovation introduced.
The point of the paper is in which space an edit should be defined and supervised: on the weights, on discretized signals, or on the function. We provide an operator-learning framework for model editing, along with a controlled comparison of standard architectures.
W3. Scope.
Experiments are done entirely using MNIST and a 1D synthetic harmonic oscillator. [...] how generalizable it is across resolutions/discretizations, how generalizable it is across different smoothness levels of the functions.
The submitted tasks are indeed smooth. To address this, we also added non-smooth ODE edit: the damping edit plus a preloaded spring whose force jumps at x = 0, so the target field is discontinuous. As expected, the operators' field errors rise sharply: for DeepONet and for the FNO, whose fixed output grid smears the jump. However, the neural operators nonetheless remain the most accurate learned editors. DeepONet's trajectory error is lower than DWSNet's, and both operators remove over 99% of the unedited trajectory error. This means that even on tasks that are less well aligned with the neural operators' inductive biases they can exhibit strong performance.
Q1. Title.
Why is this work called weight-space learning?
By "weight space learning" we mean the problem of processing and editing trained neural networks, which we approach through the functions they encode. We see that using "weight space learning" in the title might give more prominence to the weight space perspective than necessary and even be confusing. We will edit the title to reflect our framework more adequately.
Q2. Parameter counts.
FNO has 16M parameters while all others have around 1M, can you do a comparison with similar model scale to compare FNO with CNN/ViT + interpolation?
All models now have about 1M parameters. On MNIST these are 985k (CNN), 1.01M (ViT), 1.05M (FNO, width reduced from 128 to 32), 1.03M (DeepONet), 1.00M (MLP) and 1.14M (DWSNet). All results above use these models.
Overall I believe your results do not show operator is better than interpolation. I am not convinced by the motivation and contribution to use neural operators. I will maintain my score.
Response (1/2) to Reviewer wmAL
We thank the reviewer for their comments and for highlighting the relevant references on discretization errors.
W1. Discretization invariance.
We agree that discretization invariance is a property of the continuous formulation and not of an FNO implementation. We added an interpolation baseline. We ran the suggested comparison on grids of 14, 21, 35 and 42 points per side, with training at 28×28. We label rendering the source at n×n and applying the model as "native" and rendering at 28×28, applying the model, and interpolating its output (bilinear, bicubic or Fourier) as "interpolated". We did this for the 180∘ rotation (Table E) and for the swirl and dilation described in our reply to Reviewer 2 (Table F). The interpolation baseline shows strong performance in both experiments. We attribute this to the relatively homogeneous data and the absence of multiscale structure.
Table E. 180∘ rotation, test MSE across grids (trained at 28×28, mean over 5 seeds)
Interpolated rows use Fourier interpolation, the best of the three here. Bold marks the best of CNN, ViT and FNO in each column. The ViT's odd grids (21, 35) are zero-padded on one side to fit its 2×2 patches.
Table F. Swirl and dilation: native FNO vs. the best interpolated pipeline (best of CNN/ViT/FNO with any of the three methods; mean over 5 seeds)