acceptodds
Under review as a conference paper at ICLR 2027

Sampling by Target-Neighborhood Sensitivity for Accelerated Implicit Neural Representations

Abstract

Implicit Neural Representations (INRs) learn underlying continuous target functions from discrete observations, but full-coordinate training is typically time-intensive. To this end, we introduce Sampling by Target-Neighborhood Sensitivity (STaNS), a lightweight strategy that dynamically samples an informative coordinate subset for accelerated INR training. Our key insight is simple: we view each observed coordinate as a target-neighborhood (TaN) representative of the underlying continuous target function. To characterize the local informativeness of a representative, we define TaN mismatch as the discrepancy between the current INR prediction anchored at that coordinate and target values at locally offset coordinates. We then define TaN sensitivity as the worst first-order growth of this mismatch as the offset becomes infinitesimal, capturing how rapidly the anchored prediction loses consistency with the local target behavior. During INR training, STaNS dynamically updates coordinate-wise TaN scores derived from the closed-form TaN sensitivity for probabilistic sampling, prioritizing informative coordinates to accelerate training. Validated across audio, image, and shape modalities, STaNS establishes state-of-the-art INR acceleration, reducing time-to-quality by up to over existing samplers and delivering up to speedup over full-coordinate training.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Implicit Neural Representations, Adaptive Sampling, Training Acceleration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uHAKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a coordinate sampling criterion for accelerating the training of INRs. Each observed coordinate is viewed as a local representative of the underlying target function. The authors define the Target-Neighborhood (TaN) mismatch as C=∥fθ(xi)−f⋆(xi+δ)∥pC=\|f_\theta(x_i)-f^\star(x_i+\delta)\|^p, and define TaN sensitivity as the worst-case first-order growth of this mismatch as δ→0\delta\to 0. This leads to the closed-form expression Si=p∥ri∥p−2∥Jf⋆(xi)⊤ri∥S_i=p\|r_i\|^{p-2}\|J_{f^\star}(x_i)^\top r_i\|, where rir_i is the current reconstruction residual and Jf⋆(xi)J_{f^\star}(x_i) is the local Jacobian of the target signal. The target Jacobian is estimated once at initialization. During training, the static target Jacobian is combined with the evolving reconstruction residual to form coordinate-wise scores, which are subsequently mean-normalized, EMA-smoothed, transformed with a concentration parameter α\alpha, and mixed with uniform exploration controlled by λ\lambda to obtain sampling probabilities. The experiments cover 24 Kodak images, four SDF shapes, and 2,620 LibriSpeech audio clips, with comparisons against seven baselines.

Critical strengths
  1. Accelerating INR optimization under a limited coordinate budget is a reasonable and practically relevant problem. The proposed method is conceptually simple and computationally lightweight. In particular, the reported overhead is approximately 32 ms per active sampling update, substantially lower than that of NINT and EVOS.

  2. The experimental evaluation shows good attention to fairness and coverage. Reported wall-clock times include scoring and sampling overhead, and the experiments span three modalities, seven sampling baselines, three random seeds, multiple network sizes, and multiple sampling ratios. The image-domain component ablations are informative, and the visualization of how the sampled coordinates evolve during training provides an intuitive illustration of the method's behavior.

Critical weaknesses
  1. The proposed TaN sensitivity is Si=p∥ri∥p−2∥Ji⊤ri∥S_i=p\|r_i\|^{p-2}\|J_i^\top r_i\|, where ri=fθ(xi)−yir_i=f_\theta(x_i)-y_i is the current reconstruction error and Ji=Jf⋆(xi)J_i=J_{f^\star}(x_i) captures the local structure of the target signal. The paper interprets this formulation as coupling static target structure with the dynamically evolving reconstruction state, which is a reasonable motivation. However, after abstracting away the TaN terminology, the proposed criterion appears closely related to combining hard-example mining with local target-gradient or edge information. The current experiments establish that both the reconstruction residual and the target Jacobian are useful, but they do not clearly establish that the specific projected coupling Ji⊤riJ_i^\top r_i derived from TaN sensitivity is necessary.

  2. The experiments are sufficient to demonstrate the core idea, but the overall experimental setup feels somewhat dated relative to current 2026 standards. For example, the 3D reconstruction results in Figure 4 are relatively coarse and do not demonstrate performance at the level of geometric fidelity now commonly expected in modern 3D representation tasks.

Questions for authors
  1. I believe the paper should more directly justify the necessity of the specific projected coupling ∥Ji⊤ri∥\|J_i^\top r_i\|. In particular, it would be useful to include a simple baseline such as sisimple=∥ri∥p−1∥Ji∥s_i^{\mathrm{simple}}=\|r_i\|^{p-1}\|J_i\|, and compare it directly against siTaN=∥ri∥p−2∥Ji⊤ri∥s_i^{\mathrm{TaN}}=\|r_i\|^{p-2}\|J_i^\top r_i\|. Both criteria use the same dynamic residual and static target-structure information and should have comparable computational cost. Such a comparison would help determine whether the directional residual-Jacobian coupling introduced by TaN sensitivity provides benefits beyond a simpler residual-weighted local-gradient criterion.

  2. Could the authors discuss cases in which both the reconstruction residual rir_i and the target Jacobian JiJ_i have large magnitude, but Ji⊤ri≈0J_i^\top r_i\approx 0? In such a case, the TaN sensitivity can be close to zero despite both the current reconstruction error and local target variation being large. Does this situation occur in practice, particularly for vector-valued signals such as RGB images? If so, how should the resulting low sampling priority be interpreted from the perspective of optimization utility?

  3. Please also consider reporting PSNR for the audio reconstruction, or alternatively providing a more quantitative error visualization/heatmap.

3 comments
AuthorsOpenReview ↗

Response to Reviewer uHAK: Projected Coupling, NeRF Evaluation, Near-Cancellation Analysis, and Audio Evaluation

We are grateful to the reviewer for the thorough and encouraging assessment. The analyses you suggested have substantially improved the paper, and we respond to each weakness and question in turn.

W1 & Q1 (residual-Jacobian projected coupling)

We agree that the criterion is related to combining hard-example mining with local target-gradient information. Yet its form is simple and mathematically motivated (Defs. 1–2, Prop. 1), yielding the projected coupling between the residual and the signal-local Jacobian. Following the suggestion, we have added the norm-product ablation (Table 6, w/o Proj.). Against it, STaNS reduces time to PSNR = 35 dB and SSIM = 0.9 by 7.34% and 17.12%, showing that the directional coupling across output channels brings benefits beyond the norm product.

W2 (new NeRF and INR backbone evaluations)

Beyond the SDF results, we have extended the evaluation to a modern, representative 3D task by adding NeRF reconstruction experiments with both quantitative and qualitative comparisons (Sec. 4.6, Table 5, Fig. 6). STaNS achieves the best results on all metrics and checkpoints, e.g., reducing LPIPS by 13.93% at 30 seconds and 15.56% at the 5,000th iteration over the strongest competing sampler, while preserving finer details in the rendered views.

We further evaluate STaNS on different INR backbones beyond SIREN, including PE-MLP, FINER, and WIRE (Appendix A.5), where STaNS reduces time-to-quality by up to 9.58%, 19.72%, and 16.71% over the strongest competing sampler, respectively.

Q2 (near-cancellation in the residual–Jacobian projection)

We analyze this case on Kodak in Appendix A.7, defining near-cancellation pixels as those whose residual and Jacobian norms are both in the upper quartile, while the alignment ratio ∥J⊤r∥/(∥J∥∥r∥)\|J^\top r\| / (\|J\|\|r\|) is below 0.05. According to empirical results:

(1) Occurrence. It does occur for RGB images but is rare, affecting at most 0.0215% of pixels and 0.0715% of the total squared error during training (Table 13).

(2) Sampling priority and optimization utility. With residual modulation and EMA smoothing, these pixels still receive a relatively high ranking under the implemented STaNS sampler: their median percentile rank is 61–65 under the raw TaN score and 81–92 under the final sampling weight. We further replace these near-cancellation scores with the norm-product score of the w/o Proj. variant in W1 & Q1, and reconstruction is barely affected, with time-to-quality changing by at most 1.64% (Table 14).

Q3 (audio PSNR and error visualization)

We have added PSNR to the audio evaluation in Table 3, where STaNS achieves the highest value at all five checkpoints, e.g., 1.28 dB above the strongest baseline at 5 seconds. We have also added waveform error visualizations in Appendix A.4: absolute error maps on five clips (Fig. 8), and the average RMSE over all 2,620 clips versus training time (Fig. 9), where STaNS achieves the lowest error at 5, 10, and 20 seconds.

Revisions. We have added (i) the norm-product ablation (Sec. 4.7, Table 6), (ii) NeRF reconstruction experiments (Sec. 4.6) and additional INR backbones (Appendix A.5), (iii) the near-cancellation analysis (Appendix A.7), and (iv) audio PSNR (Table 3) with waveform error visualizations (Appendix A.4). All changes are highlighted in the revised PDF.

The rebuttal addressed my concerns. I remain slightly positive.

AuthorsOpenReview ↗

Thank You for the Positive Feedback

We sincerely thank Reviewer uHAK for the follow-up and for confirming that our response addressed your concerns. Your suggestions have substantially improved the paper, and we will actively engage in any further discussion during the discussion period.

Reviewer_qhYZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to sample training coordinates using the sensitivity of an anchored prediction’s mismatch to local changes in the target. Its score combines a precomputed target Jacobian with the current reconstruction residual. Experiments evaluate image, audio, and signed-distance-field fitting.

Critical strengths

It seems that the method is simple and potentially useful. Precomputing target derivatives avoids repeatedly differentiating the network to obtain structural information. Also the sampler requires no architectural changes.

Critical weaknesses

I found the spectral interpretation is to be overstated in the paper, specially for the implemented estimator. Equation 6 correctly shows that differentiation weights Fourier coefficients by frequency before summation. However, a large pointwise derivative is not equivalent to large unresolved high-frequency error. Particularly, modes can cancel and even highly oscillatory signals have stationary points where the derivative vanishes.

Experiments are quite weak. All reported network configurations use SIREN, only varying width and depth. It does not establish robustness to different spectral behavior or parameterizations. There is a large family of INRs that would strengthen the claims.

It's unclear how much of the shape improvement remains against the identical residual-only sampling pipeline?

Questions for authors

See above weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer qhYZ: Spectral Interpretation, INR Backbones, and Component Ablations

We thank the reviewer for the careful review and for noting that STaNS is simple and potentially useful. Your comments have made the paper more rigorous, and our point-by-point responses follow.

W1 (spectral interpretation)

We agree that a large pointwise derivative is not equivalent to large unresolved high-frequency error. In particular, oscillatory signals can have coordinates where the signal-local Jacobian vanishes, e.g., at signal extrema or where the modes in Eq. 6 cancel, and TaN sensitivity there becomes zero even if the residual is large. However, such coordinates can still be learned from samples on nearby flanks via the INR’s shared parameterization (analyzed by NINT, CVPR 2026). This is consistent with the toy example in Fig. 2, where the purple Jacobian band vanishes at the extrema, yet because nearby coordinates receive high TaN scores and are sampled, the high-frequency component is learned faster than under uniform random sampling.

In addition, the proposed STaNS sampler stabilizes zero TaN scores with a constant τ and enables reaching those coordinates through the uniform exploration rate λ (Sec. 3.4). The sensitivity and ablation study in Table 6 further demonstrates that the sampler is robust to the exploration rate, remaining effective even with exploration removed (λ = 0).

We have revised Sec. 3.3 to soften the spectral interpretation and to explicitly discuss the vanishing-Jacobian case, and added a note on the frequency response of the implemented finite-difference estimator in Appendix A.3.1.

W2 (INR backbones)

We have extended the evaluation beyond SIREN in Appendix A.5 (Table 10), covering parameterizations including PE-MLP, FINER, and WIRE. On all three, STaNS achieves the shortest first-hit time at every reachable threshold, reducing time-to-quality by up to 9.58%, 19.72%, and 16.71% over the strongest competing sampler, respectively. We have further extended to NeRF reconstruction (Sec. 4.6) for modern 3D representations, where STaNS achieves the best results over baseline methods.

W3 (audio and shape gains against residual-only)

We have added component ablations on both SDF and audio fitting in Appendix A.6, using the residual-only (w/o Jac.) and Jacobian-only (w/o Res.) variants within the identical STaNS pipeline.

SDF fitting at 10 s (IoU ↑ / CHD ↓, CHD in 10⁻³ units; Table 11):

SettingArmadilloLucyHappy BuddhaDragon
Jacobian-only (w/o Res.)0.959 / 5.9640.763 / 20.2370.917 / 6.7380.944 / 6.539
Residual-only (w/o Jac.)0.965 / 5.9380.845 / 15.0900.923 / 6.5200.948 / 6.395
STaNS0.970 / 5.9240.850 / 14.1310.932 / 6.3350.956 / 6.234

Audio fitting at 10 s (Table 12):

SettingSI-SNR ↑STOI ↑PESQ ↑PSNR ↑
Jacobian-only (w/o Res.)11.0270.7871.27642.46
Residual-only (w/o Jac.)11.4720.8001.28042.91
STaNS11.8850.8061.41143.19

Against the residual-only variant, STaNS achieves higher IoU and lower CHD at all 16 SDF checkpoints, with IoU gains of 0.002–0.009 and CHD reductions of up to 13.86% (Lucy, 1000th iteration), and it is better than both variants on all audio metrics and checkpoints.

Revisions. We have (i) revised the spectral interpretation (Sec. 3.3, Appendix A.3.1), (ii) added evaluations on different INR backbones (Appendix A.5) and NeRF reconstruction (Sec. 4.6), and (iii) added component ablations on SDF and audio fitting (Appendix A.6). All changes are highlighted in the revised PDF.

References:

Zhang, Chen, et al. "Ntk-guided implicit neural teaching." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer qhYZ, as the discussion period closes soon, we would like to briefly follow up on our response. In summary, we (i) clarified the spectral interpretation and discussed the vanishing-Jacobian case (Sec. 3.3, Appendix A.3.1), (ii) added evaluations on PE-MLP, FINER, and WIRE INR parameterization backbones and NeRF reconstruction (Appendix A.5, Sec. 4.6), and (iii) added residual-only and Jacobian-only ablations within the identical STaNS pipeline on SDF and audio fitting (Appendix A.6). We would be grateful to know whether these address your concerns, and we are happy to clarify any remaining points.

Reviewer_GpbUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduce Sampling by Target-Neighborhood Sensitivity (STaNS), a lightweight strategy that dynamically samples an informative coordinate subset for accelerated INR training. They observed coordinate as a target-neighborhood (TaN) representative of the underlying continuous target function. Different experiemnets are covered and achieved the state of the art results.

Critical strengths

Strong empirical performance across multiple signal modalities. The method is simple and easy to follow

Critical weaknesses

The novelty of the method is limited: Although the target-neighborhood sensitivity formulation is mathematically motivated, its resulting sampling criterion appears closely related to a conventional residual-weighted gradient-magnitude heuristic.

The computational time is faster, but what about the computational cost? Like number of parameters and Gfloop.

In addition, can this newly INR method applied to PDE?

Questions for authors

Check the Weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer GpbU: Novelty, Computational Cost, and Applicability

We thank the reviewer for recognizing the strong empirical performance and simplicity of STaNS. Your suggestions have helped us strengthen the paper considerably, and we address each concern below.

W1 (novelty)

As the reviewer notes, the TaN sensitivity formulation is mathematically motivated. We agree that the resulting criterion is simple, and we view this as a strength of the formulation, for three reasons:

(1) The form is simple and mathematically motivated. It is the closed form of the TaN sensitivity definition (Defs. 1–2, Prop. 1): how fast the mismatch between an anchored prediction and its target neighborhood grows. The derivation naturally combines the residual and the signal-local Jacobian.

(2) The form differs from the norm-product heuristic. For vector-valued signals, TaN sensitivity projects the Jacobian onto the residual direction instead of multiplying their norms. Against the simple norm-product ablation (Table 6, w/o Proj.), STaNS reduces time to PSNR = 35 dB and SSIM = 0.9 by 7.34% and 17.12%.

(3) It enables cheaper INR sampling than existing combinations. Compared with EVOS (CVPR 2025), which combines such cues through evolutionary selection (Tables 1 and 4):

MetricEVOSSTaNSReduction
Sampling overhead per active event (ms)45.7031.6830.68%
Peak VRAM (GiB)5.3021.62169.43%
Time to PSNR = 30 dB (s)26.0318.4429.16%
TFLOPs to PSNR = 30 dB327.13210.8235.55%

W2 (computation cost)

Following the reviewer's suggestion, we have added a computational cost comparison to Table 4 (Sec. 4.5).

(1) Parameters. STaNS does not modify the INR architecture and introduces no learnable parameters, so all samplers train the same network with an identical parameter count (264,707 for the 6×256 SIREN in image fitting).

(2) FLOPs and memory. STaNS requires the fewest FLOPs to reach PSNR = 30 dB, 12.14% fewer than the strongest competing baseline (E.S.). Its peak VRAM is only 0.011 GiB above Uniform Random Sampling, and 92.70% lower than NINT (CVPR 2026):

MethodUniformINTEVOSE.S.NINTSTaNS
TFLOPs to PSNR = 30 dB271.62309.83327.13239.95355.76210.82
Peak VRAM (GiB)1.6101.6105.3021.61122.1911.621

W3 (applicability to PDE; new NeRF evaluation)

STaNS belongs to the family of INR samplers that rely on the reconstruction residual against ground-truth observations (e.g., Soft Mining, INT, EVOS, NINT). In PDE solving, such observations are absent, as the solution is unknown and the network is supervised by the equation itself, so STaNS cannot be directly applied.

To broaden the evaluation, we have added NeRF reconstruction (Sec. 4.6). STaNS achieves the best results on all metrics and checkpoints, reducing LPIPS by 13.93% at 30 seconds and 15.56% at the 5,000th iteration over the strongest competing sampler.

Revisions. We have added (i) the norm-product ablation (Sec. 4.7, Table 6), (ii) the computational cost comparison (Sec. 4.5, Table 4), and (iii) NeRF reconstruction experiments (Sec. 4.6). All changes are highlighted in the revised PDF.

References:

Zhang, Weixiang, et al. "Evos: Efficient implicit neural training via evolutionary selector." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2025.

Zhang, Chen, et al. "Ntk-guided implicit neural teaching." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

Kheradmand, Shakiba, et al. "Accelerating neural field training via soft mining."Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Zhang, Chen, et al. "Nonparametric Teaching of Implicit Neural Representations." Proceedings of the 41st International Conference on Machine Learning. 2024.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer GpbU, as the discussion period closes soon, we would like to briefly follow up on our response. In summary, we (i) showed that the mathematically motivated derivation of the TaN sensitivity projected coupling outperforms the norm-product heuristic (Sec. 4.7, Table 6), (ii) added FLOPs and peak VRAM comparisons (Sec. 4.5, Table 4), and (iii) clarified the scope regarding PDEs and added NeRF reconstruction experiments (Sec. 4.6). We would be grateful to know whether these address your concerns, and we are happy to clarify any remaining points.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

Summary of Revisions. We thank all reviewers for their time and constructive feedback. For convenience, we summarize the main changes in the revised PDF (all highlighted in purple):

  • Norm-product ablation of the projected coupling (Sec. 4.7, Table 6)
  • Computational cost comparison: FLOPs and peak VRAM (Sec. 4.5, Table 4)
  • NeRF reconstruction experiments (Sec. 4.6)
  • Clarification of the spectral interpretation and discussion of the vanishing-Jacobian case (Sec. 3.3), with a note on the frequency response of the finite-difference Jacobian estimator (Appendix A.3.1)
  • Additional INR backbones: PE-MLP, FINER, WIRE (Appendix A.5)
  • TaN sensitivity component ablations on SDF and audio fitting (Appendix A.6)
  • Near-cancellation analysis of the projected coupling in TaN sensitivity (Appendix A.7)
  • Audio PSNR and waveform error visualizations (Sec. 4.4, Table 3 & Appendix A.4)

We will actively follow the discussion and respond promptly to any further points.

AuthorsOpenReview ↗

Summary of Rebuttal Discussions and Contributions

Dear Area Chair and Reviewers,

Thank you for your time and constructive feedback on our submission. As the discussion phase approaches its end, we would like to briefly summarize how we have addressed the main concerns.

Summary of Rebuttal Discussions

1. Mathematical motivation and projected coupling. TaN sensitivity is defined as the worst-case first-order growth of the mismatch between the INR prediction at a coordinate and nearby target values, and admits a closed form (Defs. 1–2, Prop. 1). This derived directional coupling matters in practice, as replacing it with a norm product slows convergence (Sec. 4.7, Table 6), while near-cancellation cases are rare and do not hinder sampling (Appendix A.7). Reviewer uHAK confirmed that the rebuttal addressed their concerns.

2. Spectral interpretation. We improved the clarity of the spectral interpretation, discussed coordinates where the signal-local Jacobian vanishes (Sec. 3.3), and added a note on the frequency response of the finite-difference estimator (Appendix A.3.1).

3. Experimental scope and cost. We added NeRF reconstruction (Sec. 4.6), evaluations on various INR backbones (Appendix A.5), component ablations on SDF and audio fitting (Appendix A.6), audio PSNR with waveform error visualizations (Table 3 & Appendix A.4), and a computational cost comparison of FLOPs and peak VRAM (Sec. 4.5, Table 4).

Summary of Contributions

1. Novel, simple, and mathematically motivated perspective for INR acceleration. STaNS views each observed coordinate as a target-neighborhood representative of the underlying continuous signal, and derives a closed-form sampling criterion, TaN sensitivity, that directionally couples the reconstruction residual with the signal-local Jacobian.

2. Broad effectiveness. Across audio, image, shape, and NeRF reconstruction, STaNS reduces time-to-quality by up to 23.41% over existing samplers.

3. Efficiency. STaNS requires no architectural changes and the fewest FLOPs to reach PSNR = 30 dB, with peak VRAM only 0.011 GiB above uniform sampling and 69.43% and 92.70% lower than EVOS (CVPR 2025) and NINT (CVPR 2026).

We hope this summary is helpful, and we will gladly respond to any remaining questions before the discussion closes.

Respectfully, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.