acceptodds
Withdrawn from ICLR 2027

NOIR: Neural Operator Mapping for Implicit Representations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

This paper presents NOIR, a framework that reframes core medical imaging tasks as operator learning between continuous function spaces, challenging the prevailing paradigm of discrete grid-based deep learning. Instead of operating on fixed pixel or voxel grids, NOIR embeds discrete medical signals into shared Implicit Neural Representations and learns a Neural Operator that maps between their latent modulations, enabling resolution-independent function-to-function transformations. We evaluate NOIR across multiple 2D and 3D downstream tasks, including segmentation, shape completion, image-to-image translation, and image synthesis, on several public datasets such as Shenzhen, OASIS-4, SkullBreak, fastMRI, as well as an in-house clinical dataset. It achieves competitive performance at native resolution while demonstrating strong robustness to unseen discretizations, and empirically satisfies key theoretical properties of neural operators.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Neural Fields, Neural Operators, Medical imaging

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wjHEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents NOIR, a framework that reframes core medical imaging tasks (such as segmentation, shape completion, image-to-image translation, and image synthesis) as continuous operator learning between function spaces rather than traditional discrete grid-based learning. NOIR embeds discrete medical signals into Implicit Neural Representations (INRs) using shared dataset-level parameters and signal-specific latent modulations, and trains a Neural Operator (NO) to map between their latent spaces. The authors evaluate NOIR across five datasets in both 2D and 3D downstream tasks and provide empirical validation of its ϵ\epsilon-ReNO properties, demonstrating resolution-independent performance.

Critical strengths

The paper formulates supervised medical imaging tasks as mappings between INR-parameterized function spaces. This formulation is well motivated by the variability in spatial sampling and the frequent resampling of medical images, and makes discretization sensitivity an explicit consideration in model design.

The evaluation establishes the applicability of this method to segmentation, shape completion, image translation, and synthesis across five datasets. Covering both 2D and 3D settings, with comparisons against several architectural families, gives the medical-imaging application study substantive empirical breadth.

The cross-resolution experiments demonstrate stable segmentation performance and consistent latent representations across the evaluated input discretizations. These findings support the empirical value of the continuous representation for handling resolution changes without retraining the shared model parameters.

Critical weaknesses
  1. High Test-Time Optimization Overhead: Unlike standard feed-forward networks, NOIR requires test-time latent optimization (inner-loop gradient steps) for every new test sample. This leads to a significantly slower inference time (e.g., 1.58 seconds for fastMRI tasks), which undermines its practicality for high-throughput clinical workflows or real-time applications.
  2. Fidelity Gap in High-Dimensional Tasks: The fidelity ceiling analysis (Section 4.6) reveals a substantial gap between the output INR's theoretical reconstruction capacity and NOIR's actual end-task performance, particularly in high-dimensional tasks like fastMRI translation (PSNR 29.09 vs. 22.87). This indicates that learning the latent-to-latent mapping becomes a major bottleneck when scaling to complex high-dimensional medical data with limited paired training samples.
  3. Limited Native-Resolution Advantage: While NOIR excels in resolution-independent robustness, its performance at native full resolution is often comparable to or slightly below traditional task-specific models (such as U-Net or diffusion-based models) while demanding considerably higher computational complexity during testing.
Questions for authors
  1. For fastMRI, what evidence supports attributing the gap between direct output-INR reconstruction (29.09 dB) and NOIR prediction (22.87 dB) to latent dimensionality and limited training data? Since Table 5 evaluates alternative latent-mapping models only on segmentation, how do you distinguish mapping limitations from input representation error in this task?
  2. Does the reported inference time include output-INR decoding on the evaluation grid, in addition to latent fitting and the operator forward pass? What latency requirements underlie the statement in Section 4.5 that the method is compatible with near-real-time clinical use?
  3. Alternative Operator Architectures: Beyond residual MLPs, have the authors explored more expressive modern neural operator backbones to better handle high-dimensional latent spaces?
Reviewer_RNc7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes NOIR, a framework that represents medical signals using shared implicit neural representations and learns a mapping between their latent codes. The proposed method is evaluated on multiple tasks including segmentation, shape completion, image translation, and synthesis across five datasets. The results show robustness to reduced input resolution, with variable performance at the native resolution.

Critical strengths
  1. This work addresses a meaningful problem, as medical images can differ in spatial resolution across acquisition and processing settings.
  2. The experiments cover several tasks and imaging modalities. The results on low-resolution segmentation inputs show that the learned representations can provide useful robustness.
  3. The analysis separates output representation fidelity from latent mapping performance, and the paper also provides the inference cost of test-time latent optimization.
Critical weaknesses
  1. The methodological novelty over CORAL [1] is not sufficiently clear. CORAL already combines modulated INRs, meta-learned latent fitting, and an MLP mapping between input and output latent codes. These are also the main components of NOIR. Although CORAL is cited, there is no direct comparison or clear explanation.
  2. The claimed representation-equivalence guarantee is not established. Section 3.4 identifies nonlinear INR decoding and latent optimization with the frame synthesis and analysis operators in [2], without showing that the required assumptions hold. Small latent discrepancies across five resolutions do not establish an operator error bound. The reported segmentation error is also measured against ground truth rather than between discretizations.
  3. The resolution experiments do not sufficiently distinguish the benefit of the proposed representation from sensitivity of the baseline evaluation pipeline. Appendix C mentions resizing inputs to a fixed resolution, but the full resampling and output evaluation procedure is unclear.
  4. The claims of competitive native-resolution performance are overstated. NOIR obtains 0.85 Dice on OASIS versus 0.95 for U-Net/CNO, 0.87 on SkullBreak versus 0.98 for CNO, and 22.87 dB on fastMRI versus 25.45 dB for DDPM (Tables 1–3). Test-time latent fitting also adds substantial cost. The paper should establish when resolution robustness justifies these losses in accuracy and efficiency.

[1] Serrano, L., Le Boudec, L., Kassaï Koupaï, A., Wang, T. X., Yin, Y., Vittaut, J. N., & Gallinari, P. (2023). Operator learning with neural fields: Tackling pdes on general geometries. Advances in Neural Information Processing Systems, 36, 70581-70611.

[2] Bartolucci, F., De Bézenac, E., Raonic, B., Molinaro, R., Mishra, S., & Alaifari, R. (2023). Representation equivalent neural operators: a framework for alias-free operator learning. Advances in Neural Information Processing Systems, 36, 69661-69672.

Questions for authors

See strengths and weaknesses, please.

Reviewer_ypDJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

NOIR casts medical-imaging tasks (segmentation, shape completion, image translation, synthesis) as learning a map between function spaces. Each input and output signal is encoded as a latent modulation of a dataset-level SIREN INR. The latent comes from a hypernetwork, and its code z is fitted by a meta-learned inner loop (5 steps in training, 10 at test, starting from z = 0; Sec. 3.2, Alg. 1–2). A residual MLP is then trained to regress z_out from z_in with an L2 loss (Sec. 3.3), and the prediction is decoded through the output INR. The paper argues that NOIR is an ε-ReNO (Bartolucci et al., 2023). As evidence it reports small pairwise latent MSEs across five input resolutions on Shenzhen X-rays (Sec. 3.4, Fig. 3). Experiments cover Shenzhen and OASIS segmentation, SkullBreak completion, fastMRI PD→T2 translation, and synthesis on a 333-sample in-house phantom ultrasound set. They compare against U-Net, ViT, Attention U-Net, FNO, CNO, DiffFNO, Transolver-3 and DDPM at 10/25/100% input resolution (Tables 1–3).

Critical strengths
  1. Clear, simple, task-agnostic formulation. One pipeline (encode → latent map → decode) handles very different tasks in 2D and 3D, and the operator T can be swapped freely (Table 5).

  2. Honest positioning and limitations. The paper says plainly that NOIR is not meant to beat tuned task-specific models at native resolution (Secs. 1, 5). It reports its inference overhead and openly discusses the trade-off between latent bottleneck and fidelity.

  3. The INR-ceiling analysis (Sec. 4.6, Table 4) is a useful diagnostic. It separates how well the output INR can represent the target from how well the operator maps to it, and identifies the large-latent regime (fastMRI) as the operator's weak point.

  4. Robustness at low input resolution on some tasks is notable, e.g. OASIS at 10%: 0.85 Dice vs 0.40–0.66 for the baselines (Table 1).

Critical weaknesses
  1. Limited novelty relative to CORAL, but adapt it to medical imaging
  • Encoding inputs and outputs as meta-learned INR modulations and learning a latent-to-latent map with an MLP is essentially CORAL (Serrano et al., 2023, cited) and the functa line (Dupont et al., 2022a), applied to medical images. Closely related medical INR work also exists: MedFuncta (Friedrich et al., 2025) and "Fit pixels, get labels" (Vyas et al., 2025), both cited. The paper does not say what is methodologically new beyond moving to a new domain, and it compares against none of these INR-based methods. Without that, the contribution is mainly an application study.
  1. The ε-ReNO claim is neither established theoretically nor validated empirically
  • The theory does not apply as stated. In Bartolucci et al., ReNO is defined with frame synthesis and analysis operators, which are linear and bounded with frame bounds A and B. In NOIR, the "synthesis" operator is a nonlinear SIREN decoder, and the "analysis" operator T† is a 10-step non-convex gradient descent from z = 0 (Sec. 3.4). Neither is a frame operator, and A_Φ, B_Ψ in Eq. 2 are never defined or estimated, so the bound in Eq. 2 is not available.
  • The empirical evidence does not show ε is small. The reported ε_zin = 4.0e-6 and ε_zout = 2.2e-5 are absolute MSEs with no reference scale. Because z is initialized at 0 and updated for only 10 steps, the latents may have very small norms, which would make small pairwise MSEs uninformative. Please report a relative discrepancy.
  1. Native-resolution performance is well below baselines, and the paper's own ceiling analysis contradicts its stated limitations
  • At full resolution NOIR falls short of the best baseline on OASIS (0.85 vs 0.95 U-Net/AttU-Net/CNO), SkullBreak (0.87 vs 0.98 CNO, 0.96 U-Net), and fastMRI (22.87 dB vs 25.45 DDPM and 24.55 U-Net) (Tables 1–3). "NOIR matches the strongest baselines at full resolution" (Sec. 5) is therefore not accurate.
  • The Limitations section says the latent bottleneck "is not binding on segmentation/completion." But the output-INR ceiling itself is below the baselines: OASIS 0.91 vs 0.95, SkullBreak 0.90 vs 0.98 (Table 4). So the representation, not the operator, prevents NOIR from matching the baselines on these tasks.
  1. Internal inconsistencies in the results
  • Ultrasound SSIM exceeds its own ceiling. NOIR reaches SSIM 0.87 (Table 3), but the output INR fitted directly to ground truth reaches only 0.70 (Table 10). According to Sec. 4.6, the prediction "cannot exceed the fidelity of the representation," so one of these numbers must be wrong, or the metrics are computed differently.
  • Suspected U-Net bug on ultrasound. U-Net scores 17.5 dB PSNR but 0.84–0.89 SSIM, while every other method is around 26–31 dB (Table 3). That pattern suggests an intensity scaling or de-normalization bug; Appendix C says baseline inputs are normalized to zero mean and unit variance.
Questions for authors
  1. How exactly are low-resolution inputs given to each baseline: native low-res, or resampled to training resolution? At what resolution is each method evaluated? Can you add upsample-then-predict and resolution-augmented baselines?
  2. Can you report relative latent discrepancies (normalized by latent norm or inter-sample distance), and measure the same cross-resolution discrepancy for FNO, CNO and the other baselines, to support the claim that only NOIR is an ε-ReNO?
  3. How do you justify applying the frame-theoretic ReNO bound (Eq. 2) when the synthesis and analysis operators are a nonlinear INR and an iterative optimizer? What are A_Φ and B_Ψ?
  4. How can ultrasound SSIM (0.87) exceed the output-INR ceiling (0.70)?
  5. What is new relative to CORAL? Can you compare against CORAL and a functa-style baseline on at least one task?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.