acceptodds
Under review as a conference paper at ICLR 2027

What Do INRs Find Difficult? Understanding and Exploiting Coordinate-Level Training Dynamics

Abstract

Implicit neural representations (INRs) are typically evaluated through aggregate reconstruction quality, obscuring how individual coordinates are learned over the course of optimization. We study this process through coordinate difficulty, defined from the reconstruction-error trajectory of each coordinate during training. Across 50 image and 50 audio signals and four INR representations, we find that coordinate difficulty exhibits substantial structure: it can be highly reproducible across random seeds, is partly predictable from local properties of the target signal, and differs systematically across representations even when they learn the same signal. We then ask whether these training dynamics are useful for optimization. Sampling coordinates according to difficulty accumulated over the complete training history improves the hardest regions of a signal but can degrade overall reconstruction, particularly for ReLU-based representations. We find that how difficulty is aggregated over time is critical: estimators that emphasize recent errors yield substantially more consistent behavior than complete-history difficulty. A simple recent-difficulty sampler improves reconstruction over uniform sampling across all eight architecture–modality settings and remains competitive with existing adaptive sampling methods under a common fixed-step evaluation protocol. Together, these results show that coordinate-level training trajectories reveal both signal-associated and representation-dependent structure hidden by aggregate reconstruction metrics, and provide an actionable signal for understanding and guiding INR training.

Primary area:
interpretability and explainable AI
Keywords:
Implicit neural representations, neural fields, training dynamics, coordinate difficulty, representation analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_v9iwOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies coordinate-level training difficulty in INRs using reconstruction-error trajectories. It examines how difficulty varies across random seeds, target-signal properties, and representations, and proposes recent-difficulty sampling to prioritize coordinates using a bounded error history. Experiments on 50 images and 50 audio signals across four architectures show that recent-history sampling improves global reconstruction over uniform sampling under fixed-step budgets, although gains over existing adaptive samplers and under equal-time budgets are not consistent.

Critical strengths
  1. Informative coordinate-level perspective on INR training. Tracking per-coordinate error trajectories reveals differences in learning behavior that final global reconstruction metrics obscure. The analysis across architectures and random seeds provides useful evidence that difficulty reflects both signal properties and representation choices, making this perspective a worthwhile contribution to understanding INR optimization.
Critical weaknesses
  1. The proposed sampling method does not yet establish a robust practical advantage. RDS turns the coordinate-level observations into a fixed-window error estimator, but its distinction from EMA is primarily the weighting of past errors. Although the 128-step window is selected on a separate development set, performance varies with the history length, and the paper provides limited guidance for choosing it under different training conditions. RDS neither consistently outperforms EMA and existing samplers nor retains its fixed-step gains under equal-time comparisons. Consequently, the evidence is stronger for an interesting diagnostic perspective than for a reliably better optimization method.
Questions for authors
  1. Temporal weighting and robustness. When EMA and window-based sampling receive comparable hyperparameter-tuning budgets, does RDS retain an advantage in global reconstruction quality on held-out signals? What evidence distinguishes the benefit of a bounded window from simply choosing an appropriate effective memory length?

  2. Applicability to indirect supervision. Does the method require directly observed targets at each sampled coordinate? For tasks such as NeRF, could difficulty instead be defined using rendered ray-level errors, and which conclusions from the coordinate-level analysis would remain applicable?

  3. Training horizon. Do the global reconstruction gains persist beyond the 640-step sampling protocol? Please clarify whether longer-run results are available and whether RDS improves the eventual reconstruction quality or primarily changes early training behavior.

  4. Value of training-history information. Given that target-only descriptors explain a substantial fraction of coordinate difficulty, have you compared RDS with sampling based on simple target statistics, such as image gradients or local audio energy? Such a comparison would help establish the additional value of tracking training errors.

Reviewer_e9QfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes to analyze which coordinates in INR inputs are challenging to learn and use a curriculum learning inspired approach to improve performance on these coordinates and overall.

Critical strengths
  • Using more model capacity for more difficult to learn coordinates is well motivated.
  • The analysis of which coordinates are considered difficult for different architectures is insightful and helps to connect individual architectures strengths to the underlying signal properties.
Critical weaknesses
  • While the analysis seems novel in the context of INRs, the paper would benefit from doing a thorough comparison to methods already proposed in different yet similarly motivated domains, in particular curriculum learning and training sample influence E.g., [1] is cited but not compared against while works such as [2] are popular in the field of curriculum learning but not cited or compared against.
  • The empirical results are clear and helpful. However, the scope of the experiments is a bit limited. While I do not expect unnecessary large scale experiments, using slightly more recent INR approaches and datasets from related work would ease comparability and showcase the actual impact of the approach other than simply being an interesting finding (see e.g. [3, 4]).
  1. Chang et al. "Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples". NeurIPS 2017.
  2. Ren et al. "Learning to Reweight Examples for Robust Deep Learning". ICML 2018.
  3. Essakine et al. "Where do we stand with implicit neural representations? A technical and performance survey". TMLR 2025.
  4. McGinnis et al. "Optimizing Rank for High-Fidelity Implicit Neural Representations". ICLM 2026.
Questions for authors
  • Have you looked into other sample influence scores (Hessian based or efficient alternatives such as TracIn [5])?
  • Which curriculum learning approach do you see best suited to work on your stated INR problem?
  1. Pruthi et al. "Estimating Training Data Influence by Tracing Gradient Descent". NeurIPS 2020.
Reviewer_M5kyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces coordinate difficulty, a per-coordinate measure defined as the mean reconstruction-error trajectory during INR training. Across 50 images, 50 audio signals, and four architectures (SIREN, ReLU-FF, ReLU-PE, dense multiresolution grid), the authors show that difficulty is (i) highly reproducible across seeds for SIREN/Grid but much less so for ReLU variants, (ii) partially predictable from target-only descriptors with reproducible residual structure, and (iii) only weakly shared across architectures once target properties are residualized out. They then show full-history difficulty-guided sampling helps hard regions but can hurt overall reconstruction (especially for ReLU models), and that temporal aggregation is key: a recent-window sampler (RDS-128) improves over uniform sampling in all eight architecture–modality settings and is competitive with Soft Mining, INT, NINT, and EVOS under a fixed-step protocol.

Critical strengths
  1. A genuinely diagnostic, analysis-first contribution. The reproducibility / target-predictability / cross-architecture transfer analyses are carefully designed (residualization, disjoint-seed controls, out-of-fold prediction) and reveal non-obvious structure — e.g., that architectures sharply disagree on hard coordinates after controlling for the target.

  2. The negative result is valuable. Showing that full-history difficulty sampling degrades ReLU representations, and tracing the fix to temporal aggregation, is a clean and actionable insight that also explains the inconsistent behavior of prior samplers (INT, NINT) across architectures.

Critical weaknesses
  1. Modest methodological novelty. The core idea — prioritizing samples by recent loss — is close to prioritized experience replay and existing error-driven INR samplers (Otonari et al., ERF). RDS is essentially a rank-based variant with a bounded window; the main novelty is the analysis, not the sampler.

  2. Mechanism left unexplained. The paper documents that full-history sampling harms ReLU models and that recent windows fix it, but offers little analysis of why (e.g., stale difficulty vs. distribution shift of hard coordinates, NTK locality of ReLU encodings). A mechanistic study would substantially strengthen the contribution.

  3. Limited scale and scope. Only 256×256 images and 4-second audio, small fixed-step budgets, no 3D/NeRF settings where adaptive sampling matters most practically; comparisons use a common fixed-step protocol without method-specific tuning, which may disadvantage baselines.

  4. Metric choice. Difficulty defined as the mean of the error trajectory conflates magnitude and persistence; alternative summaries (e.g., convergence time, area under normalized error) are not explored, leaving the definition somewhat ad hoc.

Questions for authors

refer to weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.