acceptodds
Under review as a conference paper at ICLR 2027

Neural Field Tokenizations with Hierarchy and Spatial Locality Priors

Abstract

Neural fields parameterize data as functions from coordinates to values, providing a unified framework for representation learning across modalities. Existing approaches are dominated by per-sample meta-learning, which scales poorly due to memory-intensive inner-loop optimization. The natural alternative –feed-forward encoding– typically introduces modality-specific assumptions, sacrificing the generality that makes learning with neural fields attractive. We argue that locality and hierarchy are useful priors for learning field representations that can be injected without compromising modality-agnosticism. We propose LH-NeF, a framework to learn general-purpose tokenized representations of continuous signals. A locality-preserving hierarchical encoder maps raw coordinate-value field observations to structured tokens, from which the field is reconstructed during training. By replacing meta-learning's inner loop with a single forward pass, LH-NeF uses 42x less memory and supports 133x larger batches than the strongest modality-agnostic baseline. Across images, 3D shapes, and climate fields, our learned representations match or exceed performance of modality-agnostic, modality-specific, and specialized generative neural field baselines on both reconstruction and downstream tasks.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
neural fields, tokenization, inductive bias, hierarchy

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_V8A3OpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LH-NeF, a modality-agnostic framework for learning structured neural field representations. Unlike existing approaches such as Functa and ENF, which rely on per-sample optimization to obtain latent representations, LH-NeF uses a feed-forward encoder based on hierarchical grouped attention and locality-preserving coordinate ordering. The resulting tokens are decoded through spatially guided group routing, cross-attention, and FiLM modulation. The method is evaluated on images, 3D shapes, and climate data, covering reconstruction, generation, classification, and forecasting. Results show competitive or improved performance over existing neural field methods, together with substantial reductions in memory usage and support for larger batch sizes.

Critical strengths
  1. A practical and well-motivated framework. The combination of spatial locality and hierarchy is a reasonable way to introduce structure into neural field representations without relying on modality-specific encoders. The encoder and renderer are designed consistently: the former produces spatially organized tokens, while the latter retrieves relevant information based on query coordinates. More importantly, replacing iterative latent optimization with feed-forward encoding addresses a clear scalability bottleneck in existing modality-agnostic methods.

  2. Comprehensive experiments with convincing ablations. The evaluation covers several modalities and downstream tasks. LH-NeF improves reconstruction PSNR over ENF on CIFAR-10 (44.4 vs. 42.2) and CelebA-HQ (36.1 vs. 34.6), while achieving competitive results on ShapeNet and ERA5. The downstream results are also encouraging, particularly on CelebA-HQ generation (FID 9.7 vs. 33.8 for ENF) and CIFAR-10 classification (91.2% vs. 82.1%). The ablations provide useful evidence for the design choices. Replacing the locality-preserving tokenizer with standard HiP decreases CelebA-HQ PSNR by 6.4 dB and ShapeNet IoU by 16.9 points, indicating that spatial locality is important. The efficiency results are especially notable: LH-NeF uses approximately 42× less memory and supports 133× larger batches than ENF under the reported setting.

Critical weaknesses
  1. The architectural novelty is somewhat limited. Most components, including hierarchical attention, Morton ordering, Gaussian weighting, cross-attention, and FiLM modulation, are established techniques. The main contribution is their integration into a modality-agnostic neural field framework. While the combination is effective, the methodological novelty appears relatively incremental. W2. Some efficiency and generalization claims need further validation.

  2. The memory comparison is impressive, but it does not fully establish the overall training efficiency. A comparison of training throughput or time to reach comparable reconstruction quality would be useful, especially given the different implementations of the baselines. In addition, the experiments focus on signals with relatively well-defined spatial structure. Although the method is evaluated across multiple modalities, its behavior on highly irregularly sampled observations or more complex non-Euclidean domains remains unclear. Some downstream results also show limitations, such as lower ImageNet classification accuracy than Spatial Functa (71.2% vs. 80.4%) and slightly worse ERA5 forecasting performance than ENF.

Questions for authors

Q1 (Efficiency): Could the authors provide training throughput or time-to-quality comparisons against ENF under comparable reconstruction performance? This would help establish whether the substantial memory savings translate into faster end-to-end training.

Q2 (Continuity): Section 2.2 states that multi-group routing ensures continuity. However, the selected nearest groups and the FiLM conditioning may change at group boundaries. Is continuity formally guaranteed, or does the proposed design mainly improve smoothness empirically?

Q3 (Generalization): How sensitive is LH-NeF to irregular sampling density? Given that the current evaluation mainly uses structured spatial domains, it would be helpful to clarify whether the locality-preserving grouping and routing remain effective under highly non-uniform observations.

3 comments
AuthorsOpenReview ↗

Rebuttal pt. 1

We thank the reviewer for the careful reading of our paper and for the constructive questions. Below we address each point. In the revised manuscript, all added and modified text is marked in blue for easier reference.

The architectural novelty is somewhat limited

We agree that the individual components are established, and we do not claim them as new. What we consider the core contribution is the answer to an open question of the field: all modality-agnostic neural field methods so far obtain their representations by per-sample optimization, which is the bottleneck that limits their scaling. Up until now, it was not known which inductive biases could be implemented a general way while also being able to compete in performance with the established optimization-based approaches. This gap is now closed: we argue that locality and hierarchy suffice, provide a construction that implements both priors without sacrificing generality, and show substantive efficiency gains, with competitive or superior performance against the baselines in most scenarios and tasks. We agree with the novelty point raised, but we approach our paper's value in identifying this limitation and resolving it with simple and well-understood components (locality-preserving orderings, nearest-neighbors routing, cross attention, etc).

Q1 (Efficiency): Could the authors provide training throughput or time-to-quality comparisons...

Following this suggestion, we added the suggested metrics to our efficiency benchmark and extended it to 2 other datasets. Table 4 now reports training throughput and time per epoch, next to memory and largest batch size, on three datasets instead of one, together with the total parameter count of each model, on the same hardware:

CIFAR-10CelebA-HQ 64²ImageNet 256²
FunctaENFLH-NeFFunctaENFLH-NeFFuncta*ENFLH-NeF
Total parameters ↓2.6M472K684K3.4M3.2M1.9M–817K7.7M
Memory per sample (MB) ↓340651331,5837,347173–OOM3,869
Largest batch ↑162741,372316265–011
Throughput (samples/s) ↑513.2256.11,354.8101.426.9235.8––7.3
Time per epoch (min) ↓1.52.90.63.914.91.7––2,896

*Functa is not evaluated on ImageNet1k in its original paper, so no configuration is available.

Indeed, the memory savings do translate into training speed: at each method's largest batch size, LH-NeF trains 8.8× faster than ENF and 2.3× faster than Functa on CelebA-HQ, and 5.3× and 2.6× faster on CIFAR-10. On ImageNet1k (256×256=65,536256 \times 256 = 65{,}536 points per image), the training step of ENF does not fit a single image in memory (it requests 56.5 GiB on a 45 GiB device), whereas LH-NeF uses 3.9 GB per image, even though LH-NeF has substantially more parameters in this case (7.7M against 817K; most of them are in the tokenizer, which amortizes the per-sample optimization). Appendix H also reports the case of subsampled points, where LH-NeF uses less memory at every number of points (2.8× to 12.2× less) and is faster from 4,096 points on; at 1,024 supervised points, ENF has the higher throughput, because the fixed cost of the tokenizer dominates our training step there.

We also note that the benchmark was re-run from scratch for this revision, and the JAX baselines now use memory preallocation with a limit of 97% of the device, the same share available to PyTorch. Our earlier runs had preallocation disabled for all measurements. This is needed to measure the memory per sample (otherwise JAX reports the block it preallocates), but when searching for the largest batch it is prone to memory fragmentation and out-of-memory errors, as the JAX documentation notes (https://docs.jax.dev/en/latest/gpu_memory_allocation.html). The re-run therefore keeps it disabled for the memory measurement and enables it for largest batch and throughput, which is a fairer (and more favorable) setting for the baselines. With memory preallocation enabled, the baselines fit larger batches (6 instead of 2 for ENF, 31 instead of 18 for Functa), so the batch size ratio of the original submission (133×) becomes 44× in the revision. Our memory result is unchanged (173 MB against 7.3 GB per sample, i.e. 42× less), and the abstract now reports training speed (up to 8.8× faster) in place of the batch size ratio. We note that we still dominate on training efficiency metrics even with this modification, as shown above.

Regarding time-to-quality: we could not provide this comparison, since it requires training curves of the baselines, i.e. training Functa and ENF from scratch on the datasets of the benchmark, which is not feasible within the rebuttal period. We therefore lead with training throughput and time per epoch as the evidence that the memory gains translate into faster training.

AuthorsOpenReview ↗

Rebuttal pt. 2

Q2 (Continuity): Section 2.2 states that multi-group routing ensures continuity. (...) Is continuity formally guaranteed, or does the proposed design mainly improve smoothness empirically?

The reviewer is correct; continuity is not formally guaranteed, and our wording around "ensuring continuity" was too strong. Indeed, the selection of the kk nearest groups is a hard cut. Where the kk-th and (k+1)(k+1)-th nearest centroids swap, one group leaves the sum and another enters it. Both are at the same distance there, so they have the same weight, but this weight is not zero, since a Gaussian never vanishes, and their features differ. The output therefore jumps by that weight times the difference of the two features. Multi-group routing bounds this jump but does not remove it: the group that is swapped is the farthest of the kk routed groups, so its normalized weight is at most 1/k1/k, whereas with k=1k=1 it is the whole output that switches from one group to another. Additionally, as the reviewer intuited, the FiLM modulation is expressed in the frame of the nearest group, so its input also changes where the nearest group changes.

We have revised this sentence in Section 2.2 accordingly; it now states that multi-group routing makes the field change more smoothly, with a footnote explaining why exact continuity is not guaranteed with this change. The motivation was and still is continuity, and the effect is indeed supported empirically: replacing multi-group routing by k=1k=1 costs 0.7 dB on CelebA-HQ and 8.4 IoU on ShapeNet16 (Table 5; sensitivity to kk in Appendix E.3).

Q3 (Generalization): How sensitive is LH-NeF to irregular sampling density? (...) it would be helpful to clarify whether the locality-preserving grouping and routing remain effective under highly non-uniform observations

Our grouping does not assume regular sampling, because it operates on the observations rather than on the coordinate space. A tokenizer that naively divides the space into fixed groups or regions (e.g. fixed patches) would assign the same number of tokens, i.e. the same capacity, to regions that contain very few observations and to regions that contain many. In LH-NeF, the ordered observations are instead split into groups of equal count, so every token summarizes the same number of observations, even on highly irregular signals. This is dynamic by construction: every input gets its own partition into groups, determined by where its observations lie (illustrated in Figure 11 on ShapeNet point clouds, which occupy the space irregularly).

The price of this adaptability is paid in locality: since all groups contain the same number of observations, a group in a sparsely sampled part of the signal has a proportionally larger diameter, i.e. the locality prior is weaker there. Our paper studies this scenario in Appendix B. Large groups cannot be avoided in general by any ordering (Appendix B.1), but we show that they are at most few: with Morton ordering, fewer than a fraction 2−d2^{-d} of the groups (a quarter in 2D, an eighth in 3D) can exceed the near-optimal diameter bound of regular grids, for any observation set, however irregular (Proposition B.3).

This also covers signals whose surface itself is sampled unevenly, as in a LiDAR scan, where nearby surfaces receive many more points than distant ones: Proposition B.3 makes no assumption on how the observations are distributed. As we acknowledge in the limitations section, evaluating empirically how the method performs on such highly irregular data is left for future work.

Some downstream results also show limitations, such as lower ImageNet classification accuracy than Spatial Functa (71.2% vs. 80.4%) and slightly worse ERA5 forecasting performance than ENF.

We agree, and we report both as such in the paper. We would like to add some context to each. Spatial Functa arranges its latents on a 2D grid aligned with the image, so it benefits from the structure of images but is restricted to 2D grids, whereas LH-NeF uses the same architecture on images, 3D shapes and the sphere. On ERA5 forecasting, LH-NeF trails ENF by 18% while improving over Functa by more than two orders of magnitude, which we consider competitive. The forecasting models also differ: we use a standard transformer as our downstream model (Appendix F), whereas ENF uses a more specialized one (PΘNITA, an SE(n)-equivariant message-passing network), so the comparison reflects the forecasting models as well as the learned representations. We hope this places the two results in context, as specific gaps in specific settings rather than general limitations of the method.


We thank the reviewer again for the detailed and constructive review. We hope that these additions and clarifications address the concerns raised, and that the reviewer will consider them in their final assessment. We are glad to answer any further questions during the discussion period.

Best regards,

The Authors

These clarifications address my main questions. I continue to favor weak acceptance.

Reviewer_r9oxOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LH-NeF, a modality-agnostic framework for learning neural field representations that addresses the computational limitations of existing per-sample meta-learning approaches. The method argue that locality andhierarchy are useful priors for learning field representations that can be injected without compromising modality-agnosticism. LH-NeF employs a locality-preserving hierarchical tokenizer to transform coordinate value observations into spatially structured token representations. These tokens condition a renderer that reconstructs field values at arbitrary coordinates using soft group routing and cross attention aggregation.

The framework is evaluated across images, 3D shapes, and climate fields, the authors report that the earned representations match or exceed performance of modality agnostic, modality specific, and specialized generative neural field baselines on both reconstruction and downstream tasks. LH-NeF reportedly reduces memory consumption by approximately 42x and supports 133x larger batch sizes compared with the strongest modality agnostic baseline.

Critical strengths
  1. The authors addresses an important limitation of exixting neural feild representation learning methods by replacing computationally expensive per-sample meta-learning with feed-forward encoding.
  2. The locality preserving ordering and hierarical takenization provides a practical path to incorporate spatial structure without relying on modality specific architectures.
  3. The reported 42× reduction in memory consumption and 133× increase in supported batch size are substantial improvements and represent a key strength of the work.
  4. The given results on table 1 and table 2 demonstrate the strong performance over few existing approches, specifically in image reconstruction, generation and classification the improvments are strong.
  5. The ablations supports the usefulness of the architectural choices.
Critical weaknesses
  1. Table 4 provides component ablation but the individual contibution of locality and heirararchy are not fully isolated.

  2. Table 5 focsues on peak memory consumption and max batch size and the reported improvements are substantial but the absance of traning throughput and wall clock comparisons make it difficult to understand the overall computational advantages over existing methods.

Questions for authors
  1. In table 4, can the authors also include random or raster ordering and flat tokenizer for better understanding of individual contributions of locality and heirarchy?

  2. Table 5 shows substantial improvements in memory usage and batch size, can the authors provide training time for better overall computational efficiency?

3 comments
AuthorsOpenReview ↗

Rebuttal pt. 1

We thank the reviewer for the careful reading of our paper and for the suggestions which improved our manuscript. Below we address each point. In the revised manuscript, all added and modified text is marked in blue for easier reference.

Table 5 focuses on peak memory consumption and max batch size (...) the absence of training throughput and wall clock comparisons make it difficult to understand the overall computational advantages. / Can the authors provide training time for better overall computational efficiency?

Following this suggestion, we re-ran the efficiency benchmark from scratch and extended it with metrics that better picture the consequences of the memory efficiency advantages. Table 4 now reports training throughput and time per epoch next to memory and largest batch size, on three datasets instead of one, together with the total parameter count of each model:

CIFAR-10CelebA-HQ 64²ImageNet 256²
FunctaENFLH-NeFFunctaENFLH-NeFFuncta*ENFLH-NeF
Total parameters ↓2.6M472K684K3.4M3.2M1.9M–817K7.7M
Memory per sample (MB) ↓340651331,5837,347173–OOM3,869
Largest batch ↑162741,372316265–011
Throughput (samples/s) ↑513.2256.11,354.8101.426.9235.8––7.3
Time per epoch (min) ↓1.52.90.63.914.91.7––2,896

*Functa is not evaluated on ImageNet1k in its original paper, so no configuration is available.

The memory savings translate directly into training speed. At each method's largest batch size, LH-NeF trains 8.8× faster than ENF and 2.3× faster than Functa on CelebA-HQ, and 5.3× and 2.6× faster on CIFAR-10. These gains do not come from model size: for instance, counting its tokenizer, LH-NeF has more parameters than ENF on CIFAR-10 and ImageNet, yet it is significantly the more efficient method on both. As argued throughout the paper, these gains come from replacing the per-sample optimization of the baselines, whose retained computation graph grows with the number of points of the signal, with a single forward pass. The consequence is most visible on ImageNet: with all 65,536 points of an image, the training step of ENF does not fit a single image in memory (it requests 56.5 GiB on a 45 GiB device), whereas LH-NeF uses 3.9 GB per image. Since subsampling points is common practice at this resolution, Appendix H also reports this case: LH-NeF uses 2.8× to 12.2× less memory than ENF and is faster from 4,096 points on. All times count the training steps only (forward and backward), without data loading or evaluation.

We also note that the benchmark was re-run from scratch for this revision, and the JAX baselines now use memory preallocation with a limit of 97% of the device, the same share available to PyTorch. Our earlier runs had preallocation disabled for all measurements. This is needed to measure the memory per sample (otherwise JAX reports the block it preallocates), but when searching for the largest batch it is prone to memory fragmentation and out-of-memory errors, as the JAX documentation notes (https://docs.jax.dev/en/latest/gpu_memory_allocation.html). The re-run therefore keeps it disabled for the memory measurement and enables it for largest batch and throughput, which is a fairer (and more favorable) setting for the baselines. With memory preallocation enabled, the baselines fit larger batches (6 instead of 2 for ENF, 31 instead of 18 for Functa), so the batch size ratio of the original submission (133×) becomes 44× in the revision. Our memory result is unchanged (173 MB against 7.3 GB per sample, i.e. 42× less), and the abstract now reports training speed (up to 8.8× faster) in place of the batch size ratio. We note that we still dominate on training efficiency metrics even with this modification, as shown above.

AuthorsOpenReview ↗

Rebuttal pt. 2

Table 4 provides component ablation but the individual contribution of locality and hierarchy are not fully isolated. / Can the authors also include random or raster ordering and flat tokenizer?

The locality contributions are already present in Table 5 ablations, although we realize that the wording used did not make clear which row isolates which prior, so we improved the wording in the ablation table.

Indeed, the raster-ordering ablation is already in Table 5 under the row "LH-NeF Tokenizer →\to HiP"; this is exactly our tokenizer but with raster ordering, which is the ordering of the original HiP. The hierarchy, the number of groups, the representation and the renderer are unchanged, so this row isolates cleanly the locality prior. Removing it costs 6.4 dB PSNR and 16.0 FID on CelebA-HQ, and 16.9 IoU on ShapeNet16, the largest effect in the table. In the revised manuscript, we renamed this row to "LH-NeF Tokenizer →\to HiP (raster)" for clarity. Regarding an ablation with random ordering: note that raster ordering already destroys locality along one axis, since its groups are full image rows (Section 4.3); a random ordering destroys it along all axes, so every group spans the whole image. One can therefore expect random ordering to be at a minimum, as harming as raster ordering, so we choose to leave this ablation out.

Regarding hierarchy, isolating this prior is less direct. The obvious non-hierarchical counterpart of the Hierarchical Perceiver (HiP), the Perceiver, has a single set of latents for the whole signal, i.e. a single group, so it removes locality as well (it has one global latent, much like Functa) and would confound the two priors. The ablation that removes the hierarchy while keeping locality is a single-block version of our tokenizer, which maps each of the final groups directly to its tokens, without passing through layers of fine-to-coarse groups. It keeps the Morton ordering and the same 64 groups (locality fixed), the same 64×2×2464 \times 2 \times 24 representation (latent capacity fixed), and the same renderer. It does have less capacity in the tokenizer than its multi-block counterpart, however, so this ablation cannot fully separate the hierarchy from the capacity of the tokenizer.

We ran this ablation on CelebA-HQ with the training recipe of the reference model of Table 5 (1,000 epochs). The single-block tokenizer reaches 29.7 dB test PSNR, against 36.3 dB for the hierarchical one: a drop of 6.6 dB, comparable to that of removing the locality-preserving ordering (6.4 dB). Most of the tokenizer's parameters sit in the finer blocks that this ablation removes, so the single-block variant is also smaller (0.3M against 1.3M parameters), and this number reflects the removal of the hierarchy together with that capacity, as explained above. We consider it the most direct ablation of the hierarchy that leaves locality untouched, but since it is not a clean one, we report it here and keep it out of the main text, in order to avoid claiming that hierarchy is strictly necessary, since this ablation can not fully isolate it. As explained above, the hierarchy comes with the layered grouped attention that we build on, which makes it hard to isolate on its own. Note that we do not claim that it is a necessary condition for learning neural field representations effectively and efficiently, but that, together with locality, it is a sufficient one, as the manuscript argues.


We thank the reviewer again for the detailed and constructive review. We hope that these additions address the concerns raised, and that the reviewer will consider them in their final assessment. We are glad to answer any further questions during the discussion period.

Best regards,

The Authors

I appreciate the detailed response and additional experiments.

The rebuttal addresses most of my concerns, and I will maintain my original rating.

Reviewer_SvRUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work introduces LH-NeF, an encoder-based conditional neural field with hierarchy and locality inductive biases. Different from existing encoder-based neural fields, the proposed method is modality agnostic, thus managing to avoid the disadvantages of modality-specific neural fields, while simultaneously not depending on auto-decoding or meta-learning. This is an important contribution for the field of neural field representation learning.

The presented method is clear and sound. The two inductive biases introduced, hierarchy and locality, are omnipresent inductive biases, at least in the context of computer vision (images and shapes). Hierarchy is integrated in the method through a Hierarchical Perceiver, while locality is incorporated in multiple scales: the Hierarchical Perceiver groupings, Gaussian weighting from different local groups, and FiLM layers that model distance from the groups.

Validation in standard benchmarks in the field highlights the effectiveness of the method, often outperforming modality-specific architectures, such as Spatial Functa.

Critical strengths

The proposed method is clearly presented, with well-documented and motivated design choices. LH-NeF is an important and much-needed contribution in the field.

The experimental validation is substantial, with common benchmarks being used, and the method seems to match or surpass state-of-the-art. The ablation studies examine all the most important choices the authors have made.

Critical weaknesses

It is unclear if the method can scale to larger datasets and exhibit the same state-of-the-art performance. The ImageNet example suggests that it does not scale so well, requiring 1.8M parameters to achieve a PSNR of 28.

The source code is unfortunately not shared (there is only a link renders as that link-hidden-for-double-blind-review), so the performance of the proposed method cannot be verified.

Questions for authors

Clarification questions

  1. Section 2.3 mentions that the method "adapts to the sampling density and geometry of the input". It is unclear to me, however, if and how the method can handle various densities. Figure 11 seems to contain rather uniform regions.
  2. The Gaussian -> power weight ablation in Table 4 is not explained further in the manuscript.
  3. What is the impact of the number of final blocks? Does a single final block perform well? Or is it perhaps better to have more blocks? Is there an optimal number that the authors observed?
  4. What is the impact of of group normalization to the FiLM layer being invariant to translations? Is that a desirable property? Does it lead to groups with the same size?

Suggestions

  1. Can the method handle different types of variables in the input domain, e.g. a spatio-temporal domain, or a position-orientation domain? How would the grouping work in that case?
  2. How would the method handle signals at different resolutions, e.g. the same image in high vs low resolution, and in general different resolution inputs? Are the groups optimized for a specific input resolution?

Minor

  1. The ERA5 citation references Dupont et al., but the correct reference is Hersbach et al.
8 comments
AuthorsOpenReview ↗

Rebuttal pt.1

We thank the reviewer for spending the time on carefully reading our paper and proposing questions that improve the quality and exposition of our work. Below we address every point. In the revised manuscript, all added and modified text is marked in blue for easier reference.

The source code is unfortunately not shared.

An anonymized version of the code is now available at https://anonymous.4open.science/r/iclr27-31152-2B64/. It contains one default configuration per dataset that reproduces both the reconstruction results of Table 2 and downstream task experiments of Table 3 (classification on CIFAR-10 and ShapeNet16, generation on CIFAR-10 and CelebA-HQ, forecasting on ERA5). It also contains the extended efficiency benchmark of the revised manuscript added in response to other reviewers suggestions (see below); please follow the README to run all reproductions.

It is unclear if the method can scale to larger datasets (...)

As described in L377, we chose hyperparameters by best validation loss, with no constraint on model size. Most of the 1.8M parameters sit in the MLP at the output of the renderer (hidden width 768, Table 9). The question is therefore whether a field parameterization of this size is necessary to be competitive at this scale, or whether a smaller one can reach the same competitive performance range at a comparable representation size (recall that at these latent representation sizes, LH-NeF reaches 28.0 dB, ENF 27.5 dB and Spatial Functa 28.9 dB; Table 2).

Our ImageNet-1k hyperparameter search contains such a configuration, with a field of 0.92M parameters (MLP width 512, depth 3), close to ENF's 817K and below Spatial Functa's 1.4M parameters. Its training was early-stopped automatically after 42 epochs, whereas our model in Table 2 was trained for 100, so we compare the two models at an epoch 42 checkpoint for fairness, both on the full test set:

Field #ParamsRepresentation size (groups × tokens × channels)Test PSNR at epoch 42
Model of Table 21.83M128 × 4 × 28 = 14,33627.29
Smaller field0.92M256 × 2 × 32 = 16,38427.16

With half the field parameters, this configuration is within 0.13 dB of the model of Table 2 at the same epoch. The model of Table 2 gains a further 0.7 dB between epoch 42 and the end of its 100 epochs; the smaller configuration was not trained beyond epoch 42.

We want to be precise about what this comparison shows. The two configurations come from a hyperparameter search, not from a controlled ablation, and they differ in more than the field: the smaller one uses a representation of 16,384 dimensions, whereas the model of Table 2 uses 14,336, so the representation is slightly larger and its group layout is different. This comparison therefore does not isolate the effect of the field size. What it shows is that the field size of Table 2 is not a necessary condition for our method to perform competitively on ImageNet. Indeed, there exists a checkpoint that at the same training budget, with a field half its size (comparable now to those of the baselines), performs almost the same as our competitive model in Table 2. In conclusion, this shows that the 1.8M field of Table 2 results from a search that did not optimize for size, rather than a constraint on what our method can do or not at particular parameter counts.

Since these are single runs (no std devs over multiple seeds) and for just 42 epochs, we present this comparison in this rebuttal response as supporting evidence, and leave the results of Table 2 in the manuscript unchanged. For the camera-ready version, we will run the full experiment, and we will report it in Table 2 if a competitive result at smaller size indeed holds. We note however, as shown below, that the efficiency gains do not come from more or less parameters, and we strongly dominate in memory efficiency, throughput and training time even when the total parameter count is greater than in the baselines (see below).

AuthorsOpenReview ↗

Rebuttal pt.2

As a follow-up, we would like to clarify another point; the reviewer wording "LH-NeF requires 1.8M parameters" on ImageNet made us think that Table 2 can be misleading as is. The 1.8M is the size of the field, as the Table 2 caption states. In the case of LH-NeF, our model also includes extra parameters from the tokenizer, whereas the baselines consist of the field alone, and this difference was easy to miss reading only Table 2. We apologize in advance for any confusion this might have led to, and have addressed it in the revision as we explain next. First, the caption of Table 2 now states explicitly that the count does not include the tokenizer, and points to Table 4 in Section 4, which now contains a full breakdown of total parameter counts and corresponding efficiency metrics. Second, we removed the sentence of Section 4 that referred to LH-NeF using fewer parameters (which is indeed true for the parameters of the field itself, but misleading as is), which was our only claim in the paper around parameter efficiency. The revised manuscript does not claim parameter efficiency anywhere, and our headline claims about memory efficiency and training throughput remain unchanged. Lastly, Table 4 now reports total parameter counts of all models, next to two additional training efficiency metrics as requested by other reviewers, and extended to more datasets:

CIFAR-10CelebA-HQ 64²ImageNet 256²
FunctaENFLH-NeFFunctaENFLH-NeFFuncta*ENFLH-NeF
Total #Params ↓2.6M472K684K3.4M3.2M1.9M–817K7.7M
Memory per sample (MB) ↓340651331,5837,347173–OOM3,869
Largest batch ↑162741,372316265–011
Throughput (samples/s) ↑513.2256.11,354.8101.426.9235.8––7.3
Time per epoch (min) ↓1.52.90.63.914.91.7––2,896

*Functa is not evaluated on ImageNet1k in its original paper, so no configuration is available.

Across datasets and resolutions, LH-NeF uses less memory than the baselines and trains faster wherever it runs. This includes the cases where LH-NeF is the larger model in total params: e.g. on CIFAR-10 it has 212K more parameters than ENF in total, yet it uses 19.6× less memory and trains 5.3× faster, and in ImageNet, ENF OOMs (runs out of memory; it requests 56.5 GiB on a 45 GiB device) while we fit a batch size of 11, even though our complete model has about 6.9M more parameters than ENF. As we argue throughout the paper, the reason is that the bottleneck of the baselines is not their parameter count, but their reliance on per-sample optimization, whose retained computation graph grows with the number of points of the signal. LH-NeF replaces this optimization with a forward pass of the tokenizer, whose parameters are stored once. Since subsampling is the common workaround for training neural fields efficiently on high-resolution signals, we also compared efficiency metrics under varying subsampling regimes on ImageNet, so that ENF can fit at least 1 sample in memory and a throughput comparison is available (added in Appendix H, Table 21). Overall, LH-NeF uses from 2.8× to 12.2× less memory than ENF, and is faster from 4,096 sampled coordinates on.

We also note that the benchmark was re-run from scratch for this revision, and the JAX baselines now use memory preallocation with a limit of 97% of the device, the same share available to PyTorch. Our earlier runs had preallocation disabled for all measurements. This is needed to measure the memory per sample (otherwise JAX reports the block it preallocates), but when searching for the largest batch it is prone to memory fragmentation and out-of-memory errors, as the JAX documentation notes (https://docs.jax.dev/en/latest/gpu_memory_allocation.html). The re-run therefore keeps it disabled for the memory measurement and enables it for largest batch and throughput, which is a fairer (and more favorable) setting for the baselines. With memory preallocation enabled, the baselines fit larger batches (6 instead of 2 for ENF, 31 instead of 18 for Functa), so the batch size ratio of the original submission (133×) becomes 44× in the revision. Our memory result is unchanged (173 MB against 7.3 GB per sample, i.e. 42× less), and the abstract now reports training speed (up to 8.8× faster) in place of the batch size ratio. We note that we still dominate on training efficiency metrics even with this modification, as shown above.

Overall together with a reconstruction quality on par with ENF at this scale (28.0 dB against 27.5 dB), we believe these results speak well for the scaling of our method, and we hope they address the reviewer's scaling concerns positively.

AuthorsOpenReview ↗

Rebuttal pt. 3

Section 2.3 mentions that the method "adapts to the sampling density and geometry of the input". It is unclear to me, however, if and how the method can handle various densities. Figure 11 seems to contain rather uniform regions.

Our wording here was indeed ambiguous. With "sampling density", and in Figure 11, we referred to how the observations are distributed over the coordinate space, which is uneven even when a surface is sampled evenly: a bounding box placed on the seat of a ShapeNet chair contains many points, whereas the same box around an armrest contains just a few. A tokenizer that divides the space into fixed regions (e.g. fixed patches) would assign the same number of tokens, i.e. the same capacity, to regions with very few observations and to regions with many. In LH-NeF, the ordered observations are instead split into groups of equal count, so every token summarizes the same number of observations, and every input gets its own partition into groups, determined by where its observations lie. Figure 11 illustrates this on ShapeNet point clouds, which occupy the space irregularly.

The case the reviewer raises is different and relevant: a surface that is itself sampled unevenly, e.g. a LiDAR scan with more points on nearby surfaces than on distant ones. The reviewer is right that Figure 11 does not show it, since the ShapeNet shapes are sampled evenly over their surface. Our grouping treats this case in the same way, and the price is paid in locality: since all groups contain the same number of observations, a group in a sparsely sampled part of the signal has a proportionally larger diameter, i.e. the locality prior is weaker there. The paper addresses this in Appendix B.3: with Morton ordering, fewer than a fraction 2−d2^{-d} of the groups (a quarter in 2D, an eighth in 3D) can exceed the near-optimal diameter bound of regular grids, for any observation set (Proposition B.3). So the loss of locality, while real, is confined to a bounded number of groups, although those groups can be large. As acknowledged in the limitations section, an empirical evaluation on highly irregularly sampled data such as LiDAR is left for future work.

We have revised the sentence in Section 2.3 to remove the ambiguity and to discuss this case, and we thank the reviewer for raising this observation, which improved the manuscript.

The Gaussian → power weight ablation in Table 4 is not explained further in the manuscript.

This definition was indeed missing, and we have added it to Appendix E. In this ablation, the Gaussian kernel is replaced by a "naive", inverse-distance weighting,

wg(x)=dX(x,μg)−p∑g′∈N(x)dX(x,μg′)−p,g∈N(x),w_g(x) = \frac{d_{\mathcal{X}}(x, \mu_g)^{-p}}{\sum_{g' \in \mathcal{N}(x)} d_{\mathcal{X}}(x, \mu_{g'})^{-p}}, \quad g \in \mathcal{N}(x),

where N(x)\mathcal{N}(x) are the kk routed groups. Unlike the Gaussian kernel, it has no learnable bandwidth, so it cannot adapt how quickly the weights decay with distance. Replacing the Gaussian kernel by it costs 2.3 dB on CelebA-HQ and 7.7 IoU on ShapeNet16 (Table 5).

What is the impact of the number of final blocks? Does a single final block perform well? Or is it perhaps better to have more blocks?

We believe the reviewer refers to the number of groups GG in the final block of the tokenizer, i.e. the number of local regions into which the representation is divided. If another reading was intended, please let us know and we will gladly address it.

A single group (G=1G=1) removes locality altogether: every query is routed to the same tokens, and the representation becomes one global latent. This is the regime of Functa, which falls well behind in reconstruction (e.g. 28.0 dB on CelebA-HQ against 36.1 dB for LH-NeF, Table 2). More groups are not always better either: at a fixed representation size, each additional group covers a smaller region, which exercises a stronger locality prior, but leaves fewer tokens and channels to summarize it therefore it might starve on capacity; in the limit of one group per observation, the representation is just summarized by a single float (the pixel intensity), i.e. we learn nothing.

In our experiments, we selected GG by validation among a few candidates, and the rule of thumb we used on images to narrow them down was to balance the trade-off above by using G≈NG \approx \sqrt{N}, i.e. as many groups as observations per group. For instance, the CIFAR-10 and CelebA-HQ models use 32 and 64 groups for inputs of 32232^2 and 64264^2 points respectively. We do not claim that this is optimal; the best value ultimately depends on other factors, such as the information density of the data. However, we do believe this can be useful for practitioners that apply our method, and we added this rule of thumb to the description of the tokenizer hyperparameter configuration (Appendix D.1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.