acceptodds
Withdrawn from ICLR 2027

SB-HyperGS: Spectral–Spatial Representation Capacity Reallocation for Sparse Hyperspectral Gaussian Splatting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Hyperspectral 3D reconstruction aims to recover dense spectral radiance fields from sparse spectral observations. However, directly extending 3D Gaussian Splatting typically requires assigning an independent spectral representation to each Gaussian primitive, resulting in substantial spectral parameter redundancy and limiting spatial representation capacity under a finite parameter budget. We propose a material-efficient hyperspectral Gaussian Splatting framework that replaces independent per-Gaussian spectral storage with a compact material representation. By learning shared material spectral bases and Gaussian-specific local mixing coefficients, our method reduces redundant spectral parameters and reallocates the saved representation capacity to spatial primitives. We further introduce material-aware optimization and adaptive spatial refinement to improve reconstruction from sparse spectral observations. Experiments on multi-scene hyperspectral benchmarks show that our method achieves a better quality–parameter trade-off than direct hyperspectral Gaussian Splatting. Under matched parameter budgets, our method improves reconstruction quality while substantially reducing per-primitive spectral representation overhead, demonstrating the effectiveness of compact material representations for scalable hyperspectral neural rendering.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
3D Gaussian Splatting, Hyperspectral Imaging, Sparse Reconstruction, Neural Rendering, Material Representation Learning, Spectral Compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NqLnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies hyperspectral novel-view synthesis when each training view observes only a sparse subset of spectral bands. Instead of storing an independent C-dimensional spectrum for every Gaussian, SB-HyperGS represents spectral appearance with scene-level shared spectral bases and per-Gaussian mixing coefficients, performs splatting in the low-dimensional material space, and decodes the result to arbitrary wavelengths. It further adds spectral smoothness and mixture sparsity regularization, as well as a capacity-bounded adaptive spatial refinement. Experiments on three synthetic and three real scenes compare the method with Gaussian and NeRF baselines, and with a Direct-HSI control under matched parameter budgets and matched Gaussian counts.

Critical strengths
  1. The controlled comparison with Direct-HSI isolates the spectral representation and gives clear evidence that shared spectral bases are valuable under sparse-band supervision. On real scenes with the same number of Gaussians, SB-HyperGS uses roughly half the parameters and still improves PSNR by 1.55–1.80 dB; on synthetic scenes it reduces parameters by about 74% at a cost of only 0.18–0.44 dB. This is the main evidence for the proposed representation and the main reason I see value in the paper.

  2. Rendering in the 48-dimensional material space before spectral decoding yields a compact and fast model, with about 2M parameters and more than 430 FPS for full-spectrum rendering. This efficiency is practically important for scaling hyperspectral 3D reconstruction.

Critical weaknesses
  1. The real-scene results do not fully support the central capacity-reallocation explanation. On synthetic scenes, the matched-budget advantage shrinks as the budget grows, which is consistent with the claim. On real scenes, however, PSNR decreases as the number of Gaussians increases for both representations, and the full model in Table 2 is below the fixed-topology configuration with 8192 Gaussians in Table 3. This suggests that on real data the gain may mainly come from the low-rank spectral prior of the shared bases rather than from reallocated spatial capacity, and the role of adaptive spatial refinement there is unclear.

  2. The incremental contribution over existing compact spectral representations is not established. HyperGS renders hyperspectral Gaussians through a learned low-dimensional latent space, and UnMix-NeRF represents hyperspectral radiance with a global endmember dictionary and spatial abundances, which is close to the proposed shared-bases-plus-local-coefficients design. Both are cited on page 3 but neither is compared in Tables 1–4. In addition, TensoRF-HSI and Mip-NeRF 360-HSI obtain higher PSNR than SB-HyperGS in Table 1, which the text does not discuss.

Questions for authors
  1. Could you provide a real-scene ablation of adaptive spatial refinement, and a control that separates capacity reallocation from the low-rank spectral prior? For example, Direct-HSI with a low-rank or spectral-smoothness constraint, or per-Gaussian low-dimensional coefficients without shared bases.

  2. Could you report HyperGS and/or an adapted UnMix-NeRF under the same sparse-band observation protocol?

  3. Do the Table 2 full model and the fixed-topology runs in Table 3 share the same training settings apart from refinement? What is the Gaussian capacity cap of the full model, and what is the final number of Gaussians for each row of Table 5?

Reviewer_onR2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a novel hyperspectral 3DGS framework called SB-HyperGS for reconstructing dense hyperspectral radiation fields from sparse spectral observations. Instead of storing spectra independently for each primitive, SB-HyperGS employs shared material spectral bases and Gaussian-specific local mixing coefficients to represent the spectral appearance. This design significantly reduces the overhead of repetitive spectral parameters and redistributes the saved characterization capabilities back to the spatial primitives.

Critical strengths
  1. Instead of independently storing complete spectral vectors for each Gaussian primitive, this approach significantly reduces the spectral parameter overhead for each primitive by sharing the spectral basis and local mixing coefficients.

  2. The introduced material-aware optimization combines spectral smoothness constraints with local mixing sparsity constraints. Simultaneously, an adaptive spatial refinement strategy dynamically adjusts the Gaussian distribution based on rendering errors, spatial gradients, and other metrics, effectively coordinating appearance and spatial structure modeling under sparse spectral supervision.

Critical weaknesses
  1. Lacking innovation, it is highly similar to the concept proposed by unmix-nerf. The scene-level shared material spectral basis is the endmember, and the local mixing coefficient is the abundance. The core idea of ​​reconstruction is still spectral unmixing.

  2. The core assumption of SB-HyperGS is that the hyperspectral appearance of the entire scene can be linearly combined from a finite set of "shared material spectral bases". This means that if the scene being processed contains a large number of extremely rare and unique materials with completely different spectral features, it will result in spectral loss or reduced accuracy of spectral reconstruction, and a lack of reproducibility.

Questions for authors
  1. It's highly similar to Unmix-nerf, so why not compare it? It lacks features like Unmix-nerf, HyperGS, HSI-based 3DGS, etc.

  2. Without video materials, 3D reconstruction cannot demonstrate the consistency of multiple perspectives from the images. At the same time, hyperspectral images cannot demonstrate the spectral consistency and continuity that varies with different frames or spatial variations.

  3. I am puzzled by the choice of datasets. Why not refer to the real datasets Surface Optics Dataset and BaySpec Dataset, as well as the synthetic dataset NeSpoF Dataset, which are used in most of the existing baseline papers? At the same time, there are too few hyperspectral visualization results to make it difficult to judge the advantages of the method, and there is a lack of spectral curve analysis.

Reviewer_EqutOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a memory-optimized approach for sparse hyperspectral gaussian splatting. It primarily employs a shared material spectral base to facilitate this compression, thereby reducing the number of parameters that each Gaussian needs to store.

Critical strengths
  1. By using a shared spectral base, the number of parameters stored for each Gaussian can be reduced—similar to the approach in traditional compression methods, where only the offset values for each pixel are stored—thereby reducing storage demands while maintaining accuracy.

  2. Propose to add some smoothing terms to prevent the model from overfitting during training. Also modified the original 3DGS density control method to make it better suited for hyperspectral Gaussian Splatting tasks.

Critical weaknesses
  1. There is a lack of sufficient experimental comparisons with relevant methods. This paper does not directly compare its approach with existing dedicated hyperspectral GS methods; instead, it adapts a standard RGB 3DGS method into a hyperspectral version for comparison, which weakens the persuasiveness of the experiments.

  2. The method itself also lacks sufficient contributions and fails to address some critical issues. These techniques—redundant parameter compression and interpolation-based reconstruction, as well as density control—are quite common in conventional 3DGS compression methods; the author has simply applied this strategy to the niche field of hyperspectral reconstruction under sparse observations.

Questions for authors
  1. In the ablation experiments, simply turning off spatial refinement is insufficient; it is also necessary to enable the density control of the original method. Additionally, to demonstrate that the proposed spatial refinement is more effective, the authors should compare the distribution obtained using this method with that obtained using another distribution method, both with the same number of Gaussians, and show that the proposed method achieves a higher PSNR. Only then can it be demonstrated that the proposed method is more effective.

  2. As mentioned in the weakness, add some more comparisons with relevant methods.

  3. It seems like the number of significant decimal places in the experiment tables is chosen arbitrarily.

  4. The abbreviation for the method is quite strange, and it has nothing to do with the paper’s title. Could the author explain what this name actually means?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.