- Rating
- 3
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes LH-NeF, a modality-agnostic framework for learning structured neural field representations. Unlike existing approaches such as Functa and ENF, which rely on per-sample optimization to obtain latent representations, LH-NeF uses a feed-forward encoder based on hierarchical grouped attention and locality-preserving coordinate ordering. The resulting tokens are decoded through spatially guided group routing, cross-attention, and FiLM modulation. The method is evaluated on images, 3D shapes, and climate data, covering reconstruction, generation, classification, and forecasting. Results show competitive or improved performance over existing neural field methods, together with substantial reductions in memory usage and support for larger batch sizes.
Critical strengths
-
A practical and well-motivated framework. The combination of spatial locality and hierarchy is a reasonable way to introduce structure into neural field representations without relying on modality-specific encoders. The encoder and renderer are designed consistently: the former produces spatially organized tokens, while the latter retrieves relevant information based on query coordinates. More importantly, replacing iterative latent optimization with feed-forward encoding addresses a clear scalability bottleneck in existing modality-agnostic methods.
-
Comprehensive experiments with convincing ablations. The evaluation covers several modalities and downstream tasks. LH-NeF improves reconstruction PSNR over ENF on CIFAR-10 (44.4 vs. 42.2) and CelebA-HQ (36.1 vs. 34.6), while achieving competitive results on ShapeNet and ERA5. The downstream results are also encouraging, particularly on CelebA-HQ generation (FID 9.7 vs. 33.8 for ENF) and CIFAR-10 classification (91.2% vs. 82.1%). The ablations provide useful evidence for the design choices. Replacing the locality-preserving tokenizer with standard HiP decreases CelebA-HQ PSNR by 6.4 dB and ShapeNet IoU by 16.9 points, indicating that spatial locality is important. The efficiency results are especially notable: LH-NeF uses approximately 42× less memory and supports 133× larger batches than ENF under the reported setting.
Critical weaknesses
-
The architectural novelty is somewhat limited. Most components, including hierarchical attention, Morton ordering, Gaussian weighting, cross-attention, and FiLM modulation, are established techniques. The main contribution is their integration into a modality-agnostic neural field framework. While the combination is effective, the methodological novelty appears relatively incremental. W2. Some efficiency and generalization claims need further validation.
-
The memory comparison is impressive, but it does not fully establish the overall training efficiency. A comparison of training throughput or time to reach comparable reconstruction quality would be useful, especially given the different implementations of the baselines. In addition, the experiments focus on signals with relatively well-defined spatial structure. Although the method is evaluated across multiple modalities, its behavior on highly irregularly sampled observations or more complex non-Euclidean domains remains unclear. Some downstream results also show limitations, such as lower ImageNet classification accuracy than Spatial Functa (71.2% vs. 80.4%) and slightly worse ERA5 forecasting performance than ENF.
Questions for authors
Q1 (Efficiency): Could the authors provide training throughput or time-to-quality comparisons against ENF under comparable reconstruction performance? This would help establish whether the substantial memory savings translate into faster end-to-end training.
Q2 (Continuity): Section 2.2 states that multi-group routing ensures continuity. However, the selected nearest groups and the FiLM conditioning may change at group boundaries. Is continuity formally guaranteed, or does the proposed design mainly improve smoothness empirically?
Q3 (Generalization): How sensitive is LH-NeF to irregular sampling density? Given that the current evaluation mainly uses structured spatial domains, it would be helpful to clarify whether the locality-preserving grouping and routing remain effective under highly non-uniform observations.
3 comments
Rebuttal pt. 2
Q2 (Continuity): Section 2.2 states that multi-group routing ensures continuity. (...) Is continuity formally guaranteed, or does the proposed design mainly improve smoothness empirically?
The reviewer is correct; continuity is not formally guaranteed, and our wording around "ensuring continuity" was too strong. Indeed, the selection of the nearest groups is a hard cut. Where the -th and -th nearest centroids swap, one group leaves the sum and another enters it. Both are at the same distance there, so they have the same weight, but this weight is not zero, since a Gaussian never vanishes, and their features differ. The output therefore jumps by that weight times the difference of the two features. Multi-group routing bounds this jump but does not remove it: the group that is swapped is the farthest of the routed groups, so its normalized weight is at most , whereas with it is the whole output that switches from one group to another. Additionally, as the reviewer intuited, the FiLM modulation is expressed in the frame of the nearest group, so its input also changes where the nearest group changes.
We have revised this sentence in Section 2.2 accordingly; it now states that multi-group routing makes the field change more smoothly, with a footnote explaining why exact continuity is not guaranteed with this change. The motivation was and still is continuity, and the effect is indeed supported empirically: replacing multi-group routing by costs 0.7 dB on CelebA-HQ and 8.4 IoU on ShapeNet16 (Table 5; sensitivity to in Appendix E.3).
Q3 (Generalization): How sensitive is LH-NeF to irregular sampling density? (...) it would be helpful to clarify whether the locality-preserving grouping and routing remain effective under highly non-uniform observations
Our grouping does not assume regular sampling, because it operates on the observations rather than on the coordinate space. A tokenizer that naively divides the space into fixed groups or regions (e.g. fixed patches) would assign the same number of tokens, i.e. the same capacity, to regions that contain very few observations and to regions that contain many. In LH-NeF, the ordered observations are instead split into groups of equal count, so every token summarizes the same number of observations, even on highly irregular signals. This is dynamic by construction: every input gets its own partition into groups, determined by where its observations lie (illustrated in Figure 11 on ShapeNet point clouds, which occupy the space irregularly).
The price of this adaptability is paid in locality: since all groups contain the same number of observations, a group in a sparsely sampled part of the signal has a proportionally larger diameter, i.e. the locality prior is weaker there. Our paper studies this scenario in Appendix B. Large groups cannot be avoided in general by any ordering (Appendix B.1), but we show that they are at most few: with Morton ordering, fewer than a fraction of the groups (a quarter in 2D, an eighth in 3D) can exceed the near-optimal diameter bound of regular grids, for any observation set, however irregular (Proposition B.3).
This also covers signals whose surface itself is sampled unevenly, as in a LiDAR scan, where nearby surfaces receive many more points than distant ones: Proposition B.3 makes no assumption on how the observations are distributed. As we acknowledge in the limitations section, evaluating empirically how the method performs on such highly irregular data is left for future work.
Some downstream results also show limitations, such as lower ImageNet classification accuracy than Spatial Functa (71.2% vs. 80.4%) and slightly worse ERA5 forecasting performance than ENF.
We agree, and we report both as such in the paper. We would like to add some context to each. Spatial Functa arranges its latents on a 2D grid aligned with the image, so it benefits from the structure of images but is restricted to 2D grids, whereas LH-NeF uses the same architecture on images, 3D shapes and the sphere. On ERA5 forecasting, LH-NeF trails ENF by 18% while improving over Functa by more than two orders of magnitude, which we consider competitive. The forecasting models also differ: we use a standard transformer as our downstream model (Appendix F), whereas ENF uses a more specialized one (PΘNITA, an SE(n)-equivariant message-passing network), so the comparison reflects the forecasting models as well as the learned representations. We hope this places the two results in context, as specific gaps in specific settings rather than general limitations of the method.
We thank the reviewer again for the detailed and constructive review. We hope that these additions and clarifications address the concerns raised, and that the reviewer will consider them in their final assessment. We are glad to answer any further questions during the discussion period.
Best regards,
The Authors
These clarifications address my main questions. I continue to favor weak acceptance.
Rebuttal pt. 1
We thank the reviewer for the careful reading of our paper and for the constructive questions. Below we address each point. In the revised manuscript, all added and modified text is marked in blue for easier reference.
We agree that the individual components are established, and we do not claim them as new. What we consider the core contribution is the answer to an open question of the field: all modality-agnostic neural field methods so far obtain their representations by per-sample optimization, which is the bottleneck that limits their scaling. Up until now, it was not known which inductive biases could be implemented a general way while also being able to compete in performance with the established optimization-based approaches. This gap is now closed: we argue that locality and hierarchy suffice, provide a construction that implements both priors without sacrificing generality, and show substantive efficiency gains, with competitive or superior performance against the baselines in most scenarios and tasks. We agree with the novelty point raised, but we approach our paper's value in identifying this limitation and resolving it with simple and well-understood components (locality-preserving orderings, nearest-neighbors routing, cross attention, etc).
Following this suggestion, we added the suggested metrics to our efficiency benchmark and extended it to 2 other datasets. Table 4 now reports training throughput and time per epoch, next to memory and largest batch size, on three datasets instead of one, together with the total parameter count of each model, on the same hardware:
*Functa is not evaluated on ImageNet1k in its original paper, so no configuration is available.
Indeed, the memory savings do translate into training speed: at each method's largest batch size, LH-NeF trains 8.8× faster than ENF and 2.3× faster than Functa on CelebA-HQ, and 5.3× and 2.6× faster on CIFAR-10. On ImageNet1k (256×256=65,536 points per image), the training step of ENF does not fit a single image in memory (it requests 56.5 GiB on a 45 GiB device), whereas LH-NeF uses 3.9 GB per image, even though LH-NeF has substantially more parameters in this case (7.7M against 817K; most of them are in the tokenizer, which amortizes the per-sample optimization). Appendix H also reports the case of subsampled points, where LH-NeF uses less memory at every number of points (2.8× to 12.2× less) and is faster from 4,096 points on; at 1,024 supervised points, ENF has the higher throughput, because the fixed cost of the tokenizer dominates our training step there.
We also note that the benchmark was re-run from scratch for this revision, and the JAX baselines now use memory preallocation with a limit of 97% of the device, the same share available to PyTorch. Our earlier runs had preallocation disabled for all measurements. This is needed to measure the memory per sample (otherwise JAX reports the block it preallocates), but when searching for the largest batch it is prone to memory fragmentation and out-of-memory errors, as the JAX documentation notes (https://docs.jax.dev/en/latest/gpu_memory_allocation.html). The re-run therefore keeps it disabled for the memory measurement and enables it for largest batch and throughput, which is a fairer (and more favorable) setting for the baselines. With memory preallocation enabled, the baselines fit larger batches (6 instead of 2 for ENF, 31 instead of 18 for Functa), so the batch size ratio of the original submission (133×) becomes 44× in the revision. Our memory result is unchanged (173 MB against 7.3 GB per sample, i.e. 42× less), and the abstract now reports training speed (up to 8.8× faster) in place of the batch size ratio. We note that we still dominate on training efficiency metrics even with this modification, as shown above.
Regarding time-to-quality: we could not provide this comparison, since it requires training curves of the baselines, i.e. training Functa and ENF from scratch on the datasets of the benchmark, which is not feasible within the rebuttal period. We therefore lead with training throughput and time per epoch as the evidence that the memory gains translate into faster training.