acceptodds
Under review as a conference paper at ICLR 2027

Weight-Space Learning Without Weights: Leveraging Neural Operators for Model Editing

Abstract

Neural networks increasingly serve as representations of continuous signals, from images to dynamical systems. Editing these representations raises a question: should transformations act on discretized signals, network parameters, or the functions they encode? Weight-space learning operates directly on parameters, but neural representations are non-unique, while methods operating on discretized signals can depend strongly on grid resolution. Here, we formulate editing as an operator learning problem and compare our neural operator-based approach to alternatives that operate on discretized signals or weights, using rotating MNIST implicit neural representations and introducing damping into harmonic oscillator neural ODEs as examples. For fixed-angle rotation, our approach, instantiated using Fourier neural operators, performs competitively on the original grid and outperforms weight-space baselines and vision models at intermediate coordinates. Furthermore, angle-conditioned operators generalize to held-out rotations. For oscillator damping, our neural operator-based approach outperforms weight space approaches. These results support model editing as a practical approach to transforming neural representations, with advantages in generalization across sampling grids and families of transformations.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
weight space learning, neural network artifacts, neural operators, implicit neural representations, function space learning, model editing, operator learning, parameter symmetries

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wmALOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of editing continuous neural representations, e.g. INRs or neural ODEs. There are two general approaches: one is weight space learning, which directly maps network parameters to parameters, i.e., editing in the weight space; the other one is discrete-signal editing, which uses a separate model to operate on the discretized grid output, e.g. a network to learn to rotate an image represented as a structured grid. However, using standard CNNs or ViTs completely destroy the resolution independence that made continuous neural representations useful in the first place. This paper simply propose to use Neural Operators instead of e.g. CNNs or ViTs in the second approach. Experiments show that this workflow can rotate 27 x 27 images despite trained under 28 x 28.

Critical strengths

None

Critical weaknesses
  1. Despite the continuous mathematical formulation of FNO can work with arbitrary discretizations, FNO itself cannot. See [1, 2]. For the image rotation example, if you do 512 x 512 or 1024 x 1024 in CelebA-HQ for training and then test on 64 x 64, FNO results will be very bad. Similarly, for the ODE example, if you have a very non-smooth function, DeepONet and FNOs would fail too. Evaluating on adjacent midpoints within the exact same spatial support and bandlimit is merely local interpolation. You can do INF + CNNs/ViTs + Interpolation, e.g. Fourier interpolation to achieve similar results.

  2. This paper is simply applying neural operators to replace CNNs/ViTs. There is no fundamental algorithmic or architectural innovation introduced.

  3. Experiments are done entirely using MNIST and a 1D synthetic harmonic oscillator. I do not see much value in these evaluations in terms of how generalizable it is across resolutions/discretizations, how generalizable it is across different smoothness levels of the functions.

[1] Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning

[2] Discretization-invariance? On the Discretization Mismatch Errors in Neural Operators

Questions for authors
  1. Why is this work still called weight-space learning? I do not see any learning or editing in the weight space.

  2. FNO has 16M parameters while all others have around 1M, can you do a comparison with similar model scale to compare FNO with CNN/ViT + interpolation?

3 comments
AuthorsOpenReview ↗

Response (1/2) to Reviewer wmAL

We thank the reviewer for their comments and for highlighting the relevant references on discretization errors.

W1. Discretization invariance.

Despite the continuous mathematical formulation of FNO can work with arbitrary discretizations, FNO itself cannot. [...] You can do INF + CNNs/ViTs + Interpolation, e.g. Fourier interpolation to achieve similar results.

We agree that discretization invariance is a property of the continuous formulation and not of an FNO implementation. We added an interpolation baseline. We ran the suggested comparison on grids of 14, 21, 35 and 42 points per side, with training at 28×2828\times 28. We label rendering the source at n×nn \times n and applying the model as "native" and rendering at 28×2828\times 28, applying the model, and interpolating its output (bilinear, bicubic or Fourier) as "interpolated". We did this for the 180∘180^\circ rotation (Table E) and for the swirl and dilation described in our reply to Reviewer 2 (Table F). The interpolation baseline shows strong performance in both experiments. We attribute this to the relatively homogeneous data and the absence of multiscale structure.

Table E. 180∘180^\circ rotation, test MSE across grids (trained at 28×2828\times 28, mean over 5 seeds)

Model142128 (training)3542
CNN, native9.1e-21.2e-12.1e-31.8e-11.8e-1
CNN, interpolated3.5e-33.6e-3-3.7e-33.7e-3
ViT, native1.6e-36.3e-21.1e-42.6e-24.3e-3
ViT, interpolated1.7e-31.8e-3-1.8e-31.8e-3
FNO, native9.3e-21.1e-23.0e-35.8e-35.8e-3
FNO, interpolated3.9e-34.3e-3-4.4e-34.4e-3
DeepONet7.0e-37.7e-36.9e-37.9e-37.9e-3
DWSNet7.7e-67.6e-67.6e-67.6e-67.5e-6

Interpolated rows use Fourier interpolation, the best of the three here. Bold marks the best of CNN, ViT and FNO in each column. The ViT's odd grids (21, 35) are zero-padded on one side to fit its 2×22\times 2 patches.

Table F. Swirl and dilation: native FNO vs. the best interpolated pipeline (best of CNN/ViT/FNO with any of the three methods; mean over 5 seeds)

EditPipeline14213542
SwirlFNO, native1.01e-21.71e-31.22e-31.48e-3
Swirlbest interpolated2.45e-32.21e-32.32e-32.36e-3
DilationFNO, native1.05e-27.26e-45.49e-45.60e-4
Dilationbest interpolated6.52e-46.70e-46.95e-47.00e-4
AuthorsOpenReview ↗

Response (2/2) to Reviewer wmAL

These results show that:

  • The FNO's learned map transfers, unlike the CNN's. Run natively on the rotation at 21, 35 and 42 points, the FNO's error is 2-4×4\times its training-grid error, while the CNN does worse than no edit. Both fail at 14 points, which undersamples a signal fitted at 28×2828\times 28.
  • On the swirl and dilation, the native FNO is better on finer grids. At 35 and 42 points it beats every interpolated pipeline by 1.25-1.9×1.9\times in every seed. Interpolation is still better at 14 points, and for the dilation also at 21.
  • Editors that output a function are unaffected by the grid. DeepONet's error varies by a factor of 1.16 across grids and DWSNet's by 1.02, although DeepONet is less accurate than the interpolated grid models.

We will change our claim to: "The FNO's learned map transfers across grids. On finer grids its error rises slightly with resolution, consistent with the discretization mismatch errors in [2]; on coarser grids it grows much faster, as the grid can no longer represent the signal [1]."

W2. Novelty.

This paper is simply applying neural operators to replace CNNs/ViTs. There is no fundamental algorithmic or architectural innovation introduced.

The point of the paper is in which space an edit should be defined and supervised: on the weights, on discretized signals, or on the function. We provide an operator-learning framework for model editing, along with a controlled comparison of standard architectures.

W3. Scope.

Experiments are done entirely using MNIST and a 1D synthetic harmonic oscillator. [...] how generalizable it is across resolutions/discretizations, how generalizable it is across different smoothness levels of the functions.

The submitted tasks are indeed smooth. To address this, we also added non-smooth ODE edit: the damping edit plus a preloaded spring whose force jumps at x = 0, so the target field is discontinuous. As expected, the operators' field errors rise sharply: 19×19\times for DeepONet and 67×67\times for the FNO, whose fixed output grid smears the jump. However, the neural operators nonetheless remain the most accurate learned editors. DeepONet's trajectory error is 2.6×2.6\times lower than DWSNet's, and both operators remove over 99% of the unedited trajectory error. This means that even on tasks that are less well aligned with the neural operators' inductive biases they can exhibit strong performance.

Q1. Title.

Why is this work called weight-space learning?

By "weight space learning" we mean the problem of processing and editing trained neural networks, which we approach through the functions they encode. We see that using "weight space learning" in the title might give more prominence to the weight space perspective than necessary and even be confusing. We will edit the title to reflect our framework more adequately.

Q2. Parameter counts.

FNO has 16M parameters while all others have around 1M, can you do a comparison with similar model scale to compare FNO with CNN/ViT + interpolation?

All models now have about 1M parameters. On MNIST these are 985k (CNN), 1.01M (ViT), 1.05M (FNO, width reduced from 128 to 32), 1.03M (DeepONet), 1.00M (MLP) and 1.14M (DWSNet). All results above use these models.

Overall I believe your results do not show operator is better than interpolation. I am not convinced by the motivation and contribution to use neural operators. I will maintain my score.

Reviewer_gBriOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether neural representations should be edited through their weights or through the functions they represent. The authors propose using neural operators, mainly FNO and DeepONet, to map source signals to transformed signals. They test this on MNIST INR rotation and on adding damping to harmonic-oscillator trajectories. FNO performs better than the tested weight-space baselines and transfers better to intermediate sampling coordinates than the CNN and ViT baselines.

Critical strengths
  1. The question is interesting. Working with function values is a reasonable way to avoid dealing with different weights that represent the same function.

  2. I liked the evaluation at unseen coordinates. This is relevant to the motivation of the paper, and the midpoint-grid experiment shows a clear advantage for FNO over the tested image-based models.

  3. The paper includes both image and dynamical-system examples, with fairly detailed implementation settings in the appendix. The results suggest that neural operators are worth exploring for this type of problem.

Critical weaknesses
  1. I am not sure rotation is a convincing test case here. An INR can be rotated exactly by changing its input coordinates, or just its first-layer weights. This does not require a learned editor. The paper should include this simple baseline and explain what is gained by learning the transformation.

  2. For the ODE experiment, the methods seem to solve different tasks. FNO predicts a trajectory, while the weight-space baselines edit a vector field and then integrate it. The operator also receives the forcing and system parameters explicitly. This makes the large performance gap less convincing.

  3. I do not see why the arbitrary-angle experiment requires fitting target INRs for the weight-space baselines. These editors could be trained using the rendered images as supervision. It would be useful to try this before concluding that the weight-space approach is too expensive.

Questions for authors
  1. How does the proposed method compare with directly rotating the input coordinates or transforming the first-layer weights? What practical advantage does the learned editor offer on this task?

  2. Could you make the ODE comparison more closely matched in terms of both inputs and outputs? In particular, how would the methods compare when asked to handle new initial conditions for the same edited system?

  3. Could you train an angle-conditioned weight-space editor with a rendered-image loss, without fitting target INRs, and include it in the arbitrary-angle experiment?

2 comments
AuthorsOpenReview ↗

Response (1/2) to Reviewer gBri

We thank the reviewer for their detailed comments. We believe that addressing the three points raised by the reviewer has further improved our submission.

W1/Q1. Rotation has an exact solution.

An INR can be rotated exactly by changing its input coordinates, or just its first-layer weights. [...] The paper should include this simple baseline and explain what is gained by learning the transformation.

This is true. For a SIREN, rotating by α\alpha is the first-layer edit W1←W1R(α)−1W_1 \leftarrow W_1 R(\alpha)^{-1}, which for 180∘180^\circ is W1←−W1W_1 \leftarrow -W_1. Retrained solely through the rendered image, DWSNet reaches 7.6e-6 at 180∘180^\circ, better than every other editor (Table A). This is expected, as flipping the sign of W1W_1 is a linear map on the weights that respects their permutation symmetry. DWSNet can represent this transformation exactly and its inductive bias is well aligned with learning this transformation. However, this is not the case for all image transformations. To illustrate this point, we have added two additional tasks:

  • a swirl, which rotates each point by an angle that falls from π\pi at the center to 0 at radius 1. We use the the same warp for every image. Note that this is not an affine transformation.
  • a dilation (i.e., stroke thickening), which replaces at each xx the value f(x)f(x) of the signal ff with the maximum value of the signal ff over a disk of radius 0.1 centered around xx. This transformation is nonlinear.

On these edits the ranking reverses (Table B). The best affine first-layer edit removes only 22% (swirl) and 33% (dilation) of the error. Under the common training budget, DWSNet is 42×42\times and 120×120\times less accurate than the FNO, and the MLP is worse still. A SIREN of the same size fitted to each target reaches 1.9e-4 and 1.0e-4, so the difficulty is learning the edit in weight space, not representing the result. On these edits the CNN is as accurate as the FNO, or more, on the training grid.

Table A. MNIST 180∘180^\circ, 28×2828\times 28 (test MSE, mean ±\pm std over 5 seeds)

ModelMSE
CNN2.1e-3 ±\pm 1.9e-4
ViT1.1e-4 ±\pm 1.0e-4
FNO3.0e-3 ±\pm 3.6e-4
DeepONet6.9e-3 ±\pm 1.5e-3
MLP, image loss1.0e-2 ±\pm 7.5e-3
DWSNet, image loss7.6e-6 ±\pm 1.9e-6
MLP, hybrid loss (original)4.6e-2 ±\pm 6.3e-5
DWSNet, hybrid loss (original)5.2e-3 ±\pm 5.4e-3
Identity1.2e-1

Table B. Swirl and dilation, 28×2828\times 28 (test MSE, mean ±\pm std over 5 seeds)

ModelSwirlDilation
CNN6.0e-4 ±\pm 5.3e-52.2e-4 ±\pm 8.1e-5
ViT8.1e-4 ±\pm 3.9e-51.0e-3 ±\pm 2.5e-4
FNO8.8e-4 ±\pm 1.9e-52.3e-4 ±\pm 1.3e-5
DeepONet5.2e-3 ±\pm 5.2e-44.8e-3 ±\pm 6.9e-5
MLP, image loss4.6e-2 ±\pm 4.9e-53.1e-2 ±\pm 8.5e-5
DWSNet, image loss3.7e-2 ±\pm 7.5e-42.7e-2 ±\pm 3.5e-4
Best affine first-layer edit8.2e-26.6e-2
SIREN of the same size fitted to each target1.9e-41.0e-4
Identity1.0e-11.0e-1
AuthorsOpenReview ↗

Response (2/2) to Reviewer gBri

W2/Q2. The ODE methods solve different tasks.

FNO predicts a trajectory, while the weight-space baselines edit a vector field and then integrate it. The operator also receives the forcing and system parameters explicitly.

We rebuilt the experiment so that all editors solve the same task. Each one maps the source vector field to a damped vector field, with the same information (no editor receives ω\omega), the same loss and training budget, and the same evaluation. Because every editor now returns a vector field, we could also test new initial conditions (Q2), drawn from a range twice as wide as in training.

Table C. Neural-ODE damping (MSE, mean ±\pm std over 5 seeds)

ModelFieldTrajectoryNew ICs, meanNew ICs, median
FNO9.1e-4 ±\pm 6.8e-51.5e-3 ±\pm 1.8e-43.5e-2 ±\pm 4.8e-31.0e-3 ±\pm 1.2e-4
DeepONet1.2e-3 ±\pm 1.1e-42.7e-3 ±\pm 1.9e-47.1e-3 ±\pm 1.6e-31.7e-3 ±\pm 1.8e-4
MLP9.1e-2 ±\pm 7.1e-44.4e-1 ±\pm 1.5e-28.7e-1 ±\pm 3.7e-24.5e-1 ±\pm 1.3e-2
DWSNet2.5e-3 ±\pm 1.6e-44.6e-3 ±\pm 4.4e-43.5e-2 ±\pm 4.8e-33.5e-3 ±\pm 2.9e-4
DWSNet, hybrid loss2.9e-3 ±\pm 6.6e-56.3e-3 ±\pm 6.4e-45.7e-2 ±\pm 3.3e-35.3e-3 ±\pm 1.8e-4
Identity (source field)1.3e-18.4e-11.91.2
Exact damped field (integrator error)01.7e-124.2e-121.8e-13

The field error measures the edit quality directly, and the trajectory error checks that it carries over to the ODE’s solution. Both are computed on held-out systems with initial conditions from the training range, [−1,1]2[-1,1]^2. The new initial conditions are drawn from [−2,2]2[-2,2]^2, so these trajectories start outside the range seen in training. FNO and DeepONet beat both weight-space editors on field and trajectory error in every seed, and the FNO’s errors are about 3×3\times lower than DWSNet’s. All editors have higher mean error on the wider initial conditions. There, DeepONet has the lowest mean and the FNO the lowest median. The FNO’s mean comes mostly from a few high-frequency systems, for most of which the source network itself extrapolates poorly.

W3/Q3. Weight-space editors without target INRs.

These editors could be trained using the rendered images as supervision. It would be useful to try this before concluding that the weight-space approach is too expensive.

In our original experiments, the weight-space editors used a hybrid loss, consisting of the image MSE plus 0.1 times the MSE to the weights of a separately fitted rotated INR. Following the reviewer’s suggestion, we added new experiments in which MLP and DWSNet predict a change to the source weights, and the edited SIREN is rendered and compared with the exact rotation. In this case, no target INRs are needed. Trained this way, DWSNet reaches an MSE of 7.6e-6 on the 180∘180^\circ rotation, the lowest of all editors and likely the exact first-layer edit, and the MLP reaches 1.0e-2 (Table A). With the original hybrid loss, both are much worse: DWSNet reaches 5.2e-3, at least 290×290\times worse in every seed, and the MLP 4.6e-2. This is likely because many different weights give the same function. For the neural ODEs, the effect is smaller but in the same direction (field error 2.5e-3 vs. 2.9e-3, Table C).

We emphasize that this is no longer a weights-only procedure. The loss is computed on the edited function, so training requires access to the realization map Φ\Phi that turns weights into a function (Fig. 1), and gradients pass through it. This map is available whenever the architecture of the source network is known, as it is for the INRs and neural ODEs here. In this sense, weight-space editors trained this way follow our framework: the edit is defined and supervised on the function, and the weights only parameterize the output.

We also trained angle-conditioned versions on the same angle splits as the FNO (Table D). They trail the FNO in mean error on every split (DWSNet by 1.7-2.1×2.1\times, the MLP by 2.1-3.8×3.8\times). DWSNet also varies much more between seeds: its best seed matches the FNO on the sparse angles and beats it on the arc, while the other four are 1.6-2.8×2.8\times worse. We will change contribution 3 to say that neither approach needs target INRs, and that the FNO is more accurate on average and more consistent.

Table D. MNIST arbitrary angles (test MSE, mean ±\pm std over 5 seeds)

ModelHeld-out, sparse (12 angles)Held-out arc (30 angles)Training angles (0∘0^\circ, 90∘90^\circ, 180∘180^\circ, 270∘270^\circ)
FNO1.7e-2 ±\pm 9.8e-43.2e-2 ±\pm 2.4e-31.4e-2 ±\pm 9.2e-4
DeepONet4.8e-2 ±\pm 2.5e-35.4e-2 ±\pm 2.2e-34.7e-2 ±\pm 2.8e-3
MLP5.8e-2 ±\pm 3.1e-36.8e-2 ±\pm 1.9e-35.4e-2 ±\pm 1.9e-3
DWSNet2.8e-2 ±\pm 8.0e-36.7e-2 ±\pm 2.9e-22.4e-2 ±\pm 7.9e-3
Identity1.2e-11.3e-11.0e-1
Reviewer_NxbnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper is about how we can modify a signal represented by an implicit neural representation network. Authors argue that Weight-space learning for this task is tricky, because many different weights can represent the same signal. Therefore, they suggest a learned neural operator to transform the underlying implicit function. They use Fourier Neural Operator (FNO) and DeepONet for this task. They test their method on two tasks, rotating handwritten MNIST digits stored as neural networks, and adding damping to oscillator trajectories produced by neural ODEs.

Critical strengths

Authors distinguishes three places where an edit can be learned; on the network’s weights, on a discretized signal, or on the continuous function the network encodes. Then they explain the difficulties and limitations of first two methods and formulate the editing problem as an operator learning function. I think this formalization is a useful lens for the weight-space learning community and could shape how future editing tasks are posed.

Critical weaknesses

Limited novelty : The novelty lies mainly in this framing and comparison rather than in a new method. FNO and DeepONet are used off the shelf.

Only a limited number of experiments: The empirical evaluation is narrow. Only two settings are studied and they are simple tasks, which authors clarify in limitations.

Questions for authors

I am a bit unclear how the FNO takes its input. More clarification on that would be helpful. I think the paper would benefit from a pipeline figure.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Nxbn

We thank the reviewer for their comments.

W1. Novelty.

The novelty lies mainly in this framing and comparison rather than in a new method. FNO and DeepONet are used off the shelf.

We agree that the contribution is an operator-learning framework for model editing and a controlled comparison, not a new architecture, and that FNO and DeepONet are used off the shelf. This was a deliberate design choice. Our goal was to instantiate the framework with existing architectures, and to compare models with matched parameter budget. This allows us to compare approaches that operate in weight space, function space, or on discretized signals. We do think the framework is more than a relabeling, and evidently:

  • Weight-space editing does well when the edit can be expressed as a linear weight space transformation, and poorly otherwise. The 180∘180^\circ rotation is a sign flip of the first-layer weights and DWSNet is the best editor for this task. On the swirl and a dilation tasks, which cannot be expressed as linear first-layer edits, DWSNet is 42×42\times and 120×120\times less accurate than the FNO (Table B). The damping task exhibits a similar pattern: Both neural operators beat the weight-space editors (Table C).
  • Defining the edit on the function decouples the editor from the source network and from the grid. An editor that takes samples of the source, rather than its weights, can be used with any source network that can be evaluated. An editor that returns a function can be evaluated on any grid. DeepONet does both, and its error stays within 16% across grids from 14×1414\times 14 to 42×4242\times 42, while the native CNN’s grows by nearly 90×90\times (Table E, reply to Reviewer 3).

W2. Scope.

The empirical evaluation is narrow. Only two settings are studied and they are simple tasks [...]

We agree that the evaluation was somewhat narrow, and have extended it in four ways (all models with about 1M parameters, 5 seeds):

  • Two non-affine edits, a swirl and a stroke-thickening dilation, which no first-layer weight edit can reproduce. CNN, ViT and FNO are 26-140×140\times more accurate than the weight-space editors (Table B).
  • Four new grid sizes (14 to 42 points) with interpolation baselines. Interpolation wins on the rotation, the native FNO wins on the finer grids for the swirl and dilation, and editors that return a function are almost unaffected by the grid (Tables E, F).
  • Weight-space editors trained without target INRs. DWSNet becomes the best editor on the 180∘180^\circ rotation, at least 290×290\times better than with the original loss (Table A). This advantage disappears on the swirl and dilation, where the weight-space editors are far less accurate than the grid-based editors.
  • A matched neural-ODE task, in which every editor maps the source vector field to a damped one. Both neural operators beat the weight-space editors in every seed, and DeepONet is also the most accurate on wider initial conditions (Table C).

Q1. FNO input.

I am a bit unclear how the FNO takes its input. [...]

We will add a pipeline figure to the revised version of the submission. The FNO takes its input as follows.

  • MNIST: the source INR is rendered on an n×nn \times n grid over [−1,1]2[-1,1]^2. Each grid point has five input channels: the intensity, the two coordinates, and cos⁡α\cos\alpha and sin⁡α\sin\alpha of the rotation angle (constant for the fixed edits). A small pointwise network lifts these to 32 channels, followed by six Fourier layers with 12×1212\times12 modes and a pointwise projection to the edited intensity on the same grid. The same weights are used for every nn.
  • Neural ODEs: the source vector field is sampled on a 17×17×517\times17\times5 grid over a box in position, velocity and forcing. The inputs are the two field components, the three coordinates, ζ\zeta and the log of the box extents. The FNO outputs a correction on the grid. At any state, the edited field is the source network's output plus the interpolated correction, and it is integrated with RK4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.