What Do INRs Find Difficult? Understanding and Exploiting Coordinate-Level Training Dynamics

Implicit neural representations (INRs) are typically evaluated through aggregate reconstruction quality, obscuring how individual coordinates are learned over the course of optimization. We study this process through coordinate difficulty, defined from the reconstruction-error trajectory of each coordinate during training. Across 50 image and 50 audio signals and four INR representations, we find that coordinate difficulty exhibits substantial structure: it can be highly reproducible across random seeds, is partly predictable from local properties of the target signal, and differs systematically across representations even when they learn the same signal. We then ask whether these training dynamics are useful for optimization. Sampling coordinates according to difficulty accumulated over the complete training history improves the hardest regions of a signal but can degrade overall reconstruction, particularly for ReLU-based representations. We find that how difficulty is aggregated over time is critical: estimators that emphasize recent errors yield substantially more consistent behavior than complete-history difficulty. A simple recent-difficulty sampler improves reconstruction over uniform sampling across all eight architecture–modality settings and remains competitive with existing adaptive sampling methods under a common fixed-step evaluation protocol. Together, these results show that coordinate-level training trajectories reveal both signal-associated and representation-dependent structure hidden by aggregate reconstruction metrics, and provide an actionable signal for understanding and guiding INR training.