acceptodds
Withdrawn from ICLR 2027

Where to Allocate Gaussians: Learning Semantic Allocation for 3D Occupancy Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Semantic 3D Gaussians provide a compact representation for 3D semantic occupancy prediction by rendering semantic primitives into a voxel volume under voxel-wise supervision. Recent methods have improved the modeling ability and efficiency of this representation through more flexible primitive shapes, geometry-guided initialization, and progressive densification. However, these advances mainly determine how primitives are represented, initialized, or added, and do not explicitly address how to select the most useful Gaussians when their total number must be limited to control memory and computation. This imbalance creates an allocation bottleneck: redundant Gaussians remain in simple regions, while difficult regions receive insufficient semantic support. We propose the Semantic Gaussian Allocation Transformer (SAGFormer), which uses Gaussian attributes and local geometric-semantic features to score candidates and select a fixed final Gaussian set. Experiments on nuScenes-SurroundOcc and SSCBench-KITTI-360 show that SAGFormer improves occupancy prediction under the evaluated protocols and yields more semantically consistent and better-utilized Gaussian representations. Under similar final counts and raw coverage, it reduces semantic mixing, strengthens class-consistent voxel support, and produces fewer unused Gaussians. The results indicate that explicit capacity allocation is a useful complement to Gaussian refinement for semantic occupancy prediction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Occupancy, Semantic Gaussian, Gaussian Budget ALlocation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cyAtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SAGFormer. The main motivation is that a compact Gaussian representation does not necessarily imply effective use of the available capacity: redundant Gaussians may remain in simple regions, while semantically difficult regions may receive insufficient support. To address this issue, SAGFormer combines Gaussian attributes with local geometric and semantic statistics, models relations among Gaussians using Transformer blocks, and generates keep, clone, split, and suppress candidates. These candidates are globally ranked and selected under a predefined Gaussian budget before voxel decoding. Experiments on nuScenes-SurroundOcc and SSCBench-KITTI-360 demonstrate strong performance.

Critical strengths

This paper identifies an interesting problem in Gaussian-based occupancy prediction. In particular, the distinction between high geometric coverage and effective capacity allocation is meaningful.

Critical weaknesses
  1. In the candidate-family ablation, introducing clone or split also increases the final number of Gaussians: the count changes from 10.38K for preserve-only to 13.32K with clone, 13.42K with split, and 16.6K with clone+split. Therefore, the reported improvements cannot clearly distinguish whether the gains come from the proposed candidate operations themselves or simply from increased representation capacity. Since the paper focuses on allocating a limited Gaussian budget, matched-count comparisons are important for validating the effectiveness of clone and split.

  2. The paper motivates SAGFormer as reallocating Gaussian capacity toward thin structures, class boundaries, and semantically difficult regions, but the gains from semantic cues are highly uneven.

  3. The proposed allocation diagnostics provide only indirect evidence for improved semantic representations. Metrics such as Mix-G, Sem-Sup, and support purity are computed based on the majority ground-truth labels of the voxels spatially covered by each Gaussian. As a result, these metrics mainly measure the semantic homogeneity of a Gaussian's spatial support rather than the correctness of its predicted semantic class. A Gaussian could therefore achieve high support purity while still predicting an incorrect semantic label.

Questions for authors

1.Could the authors provide matched-count ablations for preserve-only, clone-only, split-only, and clone+split variants?

2.Could the authors provide additional analysis on small objects, semantic boundaries, or thin structures to verify the intended semantic allocation behavior?

3.Could the authors evaluate the semantic correctness of individual Gaussians directly?

Reviewer_ggarOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SAGFormer, a method for allocating semantic 3D Gaussians for occupancy prediction. It uses geometric and semantic context to score and select Gaussian candidates under a budget. Experiments on nuScenes-SurroundOcc and SSCBench-KITTI-360 show improvements in prediction accuracy and Gaussian utilization.

Critical strengths

The paper addresses a useful problem for gaussian-based occupancy method: allocating a limited number of Gaussians for semantic occupancy prediction.

Critical weaknesses
  1. The citation formatting should be revised to follow the conference template, for example at line 267. The layout of the tables and figures needs to be improved.
  2. Improvements on foreground classes are limited. On nuScenes, semantic cues mainly benefit background classes, while several small-object classes show reduced performance.
  3. The efficiency analysis is incomplete. Rounded FPS values do not clearly show the latency overhead, and training and inference costs are not sufficiently reported.
Questions for authors

Q1: Why do semantic cues mainly improve background classes while reducing performance on several foreground classes? Can the allocation statistics help explain this behavior?

Q2: Could you report inference latency in milliseconds and the runtime of the allocation module, compared with the no-allocation baseline at matched final Gaussian counts?

Q3: What is the training overhead of decoding all generated candidates? Please report training time and peak GPU memory compared with the no-allocation baseline.

Q4: In the “w/o Semantic Cues” ablation, is the neighborhood semantic-agreement feature in Eq. (1) also removed?

Reviewer_bCeROpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to allocate a fixed budget of 3D Gaussians for semantic occupancy prediction, an under-addressed part of the Gaussian pipeline. SAGFormer encodes each Gaussian's attributes and local geometric-semantic context, routes it to keep, clone, split, or suppress, and materializes the final set through global score-based selection under a budget. It is tested on nuScenes-SurroundOcc and SSCBench-KITTI-360 with fixed-count matched comparisons, a camera-only plug-in test, and efficiency, budget, and objective ablations. The central claim is that allocation improves occupancy accuracy while reducing Gaussian count and memory.

Critical strengths
  1. Where prior Gaussian occupancy papers concentrate on primitive shape or densification, this one targets allocation under a fixed budget, and the keep/clone/split/suppress routing plus global score-based selection is a reasonable end-to-end answer. The efficiency numbers are the most persuasive part: at 16.6K Gaussians SAGFormer reaches 28.47 mIoU at 2456 MB, whereas GaussianFormer needs 25.6K Gaussians and 4093 MB for 19.10 mIoU (Table 7). The budget sweep (Table 6) shows the benefit saturates around B = 0.6, which is a useful, actionable result.
  2. The fixed-16.6K study holds the entire system constant and varies only the allocation policy, and the camera-only plug-in (Table 10) raises GaussianFormer from 19.10 to 23.67 mIoU with the same 16.6K count — a 4.57-point gain that suggests the allocation behavior transfers to a weaker, differently initialized backbone rather than being tied to the authors' own setup. The candidate-family sweep (Table 18) further shows that suppression is what closes the gap (27.69 for clone+split vs. 28.47 for the full set).
Critical weaknesses
  1. At fixed 16.6K, SAGFormer is 28.47 mIoU against 27.33 for no allocation (+1.14) and 28.03 for the strongest adapted policy (PG-Occ-style, +0.44) (Table 1). The two auxiliary objectives contribute only 0.35 mIoU combined (28.12 → 28.47, Table 5). The paper's language and figures describe fixing a "bottleneck," but the measured occupancy benefit on the camera+LiDAR setting is roughly one mIoU, and it is the camera-only plug-in rather than the headline protocol that carries the larger number.
  2. The large figures in Table 2 (51.79 / 33.17) and Table 3 (60.10 / 26.80) use one raw LiDAR sweep at inference with temporally aggregated training supervision (the 1* setting), whereas the matched results are 41.74 / 28.47 at one frame. The gap to reported GaussianFormer3D, DAOcc, and SDG-Fusion therefore combines allocation with the full Gaussian backbone and multi-sweep training, and several of those comparators are reported numbers rather than reimplementations (Gau-Occ has no public code). The paper discloses this, but it leaves the actual contribution of allocation to the benchmark result unclear.
  3. Semantic cues raise background IoU from 29.36 to 31.10 but foreground-object IoU only from 26.36 to 26.42 (p. 9), with declines on KITTI for person, parking, and terrain; the small, dynamic objects that matter most for driving benefit least. Separately, the supplementary code (86 Python files) ships with no README, requirements, or install instructions, so it cannot be set up as released.
Questions for authors

1.Can you decompose the 1* benchmark numbers (Tables 2–3) into the portion due to temporal/multi-sweep training versus the allocation module, using the same matched-control discipline as Table 1? I would like to see SAGFormer with and without allocation under the 1* setting. 2.Why is the camera-only plug-in gain (+4.57, Table 10) so much larger than the matched camera+LiDAR gain (+1.14)? Is allocation most useful when initialization is weak, and does that imply the benefit shrinks as the backbone gets stronger? 3.Can you report per-class and per-distance results for foreground/dynamic objects under the matched protocol, and whether a foreground-aware scoring term recovers those classes? 4.Will the final code release include a README, dependency list, and trained configs so the 16.6K result is reproducible? Are the reported-baseline numbers in Tables 2–3 taken verbatim from the cited papers, and under which exact settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.