- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents VGGT-Occ, a vision-only framework for 3D semantic occupancy prediction using multiple camera views and temporal observations. It combines a pretrained VGGT-Omega encoder adapted with LoRA with two principal components. Projection-Aware Deformable Attention incorporates shared 3D sampling offsets, a projection-Jacobian attention bias, and channel-wise observation fusion conditioned on image features, geometry, and temporal metadata. A sequential coarse-to-fine decoder applies cross-view attention at coarse resolutions and uses coarse-guided gated fusion with local convolutional refinement at the finest resolution. The method reports semantic mIoU of 25.36% on SurroundOcc-nuScenes, 44.47% on Occ3D-nuScenes, and 18.13% on SSCBench-KITTI-360, exceeding the methods included in the respective comparisons. Controlled ablations on SurroundOcc support the contributions of observation fusion and coarse guidance, with additional evidence that PA-DA improves performance with both VGGT-Omega and DINOv3-L backbones.
Critical strengths
1.The paper provides meaningful controlled evidence for its principal components. PA-DA improves semantic mIoU by 1.63 percentage points with VGGT-Omega and 1.24 points with DINOv3-L under the respective matched decoder settings. Full observation conditioning improves upon a feature-only gate by 0.91 points, and learned coarse fusion improves upon fixed equal weighting by 0.53 points. These comparisons substantiate benefits beyond simply adopting a stronger backbone or adding an unexamined fusion module, which is the strongest methodological argument for acceptance.
2.The complete system performs well across three benchmark protocols spanning surround-view occupancy prediction and monocular temporal scene completion. The SSCBench-KITTI-360 result is particularly consequential: with four input frames, the method improves upon the listed four-frame S2GO-Base result by 3.03 percentage points in semantic mIoU and 4.26 points in geometric IoU. Although the comparisons do not fully control backbone capacity and pretraining, the results provide useful evidence that the overall design is effective beyond a single annotation protocol.
Critical weaknesses
1.To clarify the interpretation of Stage 2, how does the full Jacobian bias compare with a depth-based bias using the same learned scales and numerical stabilization under the cameras used in these experiments? Appendix B.2 already establishes their equivalence under specific assumptions. A controlled comparison would determine whether the observed benefit requires the more general Jacobian formulation or primarily reflects depth weighting. 2.How stable are the principal improvements across independent training seeds, particularly the 0.37 percentage point Occ3D advantage and the 0.21 point contribution of Stage 2 on top of Stages 1 and 3? Please provide any available repeated-run results or uncertainty estimates, and clarify whether the reported configurations were each trained only once. 3.Does temporal projection include any compensation for independently moving objects beyond the rigid camera transformations in Equation (1)? If it does not, what evidence shows that the observation gate suppresses stale object evidence when using seven preceding nuScenes keyframes? A dynamic-class analysis with and without temporal conditioning would directly inform the claimed benefit of temporal fusion. 4.For reproducibility, what exact loss coefficients and optional class-balancing settings were used on each benchmark, and how were coarse-scale labels and validity masks constructed?
Questions for authors
see Weaknesses
3 comments
Part 2/2 — W3: Temporal Gate Analysis; W4: Reproducibility
[Part 2/2 — W3: Temporal Gate Analysis; W4: Reproducibility]
Reply to W3: Rigid Projection Scope, Dynamic-Class Ablation & Gate Behavior
(1) Rigid Projection and Dynamic-Class Ablation.
Eq. (1) uses rigid occupancy-reference-to-camera transformations only, with no explicit moving-object compensation or flow warping. Historical frames () thus aid static parallax yet project moving objects onto stale background features. Comparing Full SurroundOcc (8f) against an independently trained no-Time variant (omitting in Eq. 4 while keeping and ) isolates where temporal conditioning acts:
| SurroundOcc 8f Subset | Full (w/ ) | no-Time (Indep. Run) | Gain (Full − no-Time) |
|---|---|---|---|
Dynamic 8 Classes (7/8 improve*) | 22.36% | 21.11% | +1.25 pp |
Static 8 Classes (drive. surf., sidewalk, etc.) | 28.36% | 28.25% | +0.11 pp |
| Overall 16-Class mIoU / Geometric IoU | 25.36% / 37.24% | 24.68% / 36.71% | +0.68 pp / +0.53 pp |
*Dynamic deltas (pp): truck +2.01, bus +1.84, bicycle +1.58, motorcycle +1.52, pedestrian +1.31, car +1.17, trailer +0.76, const. veh. -0.19.
Dynamic classes contribute ~92% of the mIoU gain from temporal conditioning.
(2) Temporal Gate Analysis on 6,019 Samples.
Auditing normalized Eq. (6) weights across all 6,019 SurroundOcc val samples at the final PA-DA gate (S1) shows:
- Semantic groups: Dynamic-class queries assign 32.92% weight to vs. 28.64% for static queries (uniform valid-observation shares: 16.44% vs. 16.69%).
- Speed-stratified
carqueries (strictly matched inside 3D GT boxes; trajectory finite-difference speed):
GT car Speed (m/s) | <0.2 | 0.2–0.5 | 0.5–2 | 2–5 | 5–10 | 10–20 |
|---|---|---|---|---|---|---|
| Query-Equal Share (%) | 33.41 | 35.09 | 41.46 | 42.84 | 45.71 | 46.44 |
| Object-Frame-Equal Share (%) | 32.06 | 33.77 | 40.48 | 42.56 | 45.95 | 47.11 |
| Uniform Valid-Obs. Baseline (%) | 15.73 | 16.81 | 16.48 | 15.71 | 17.13 | 15.71 |
From <0.2 to 10–20 m/s, current-frame share rises by 13.03 pp (15.05 pp object-frame-equal), while the uniform reference shows no corresponding monotonic rise. The share of older lags ( s) falls from 49.79% to 37.40%, with 53.56% still assigned to history at 10–20 m/s. Together, the dynamic-class gains and temporal reweighting provide compelling evidence of motion-sensitive observation fusion without flow supervision.
Reply to W4: Loss Settings, Class Frequencies, Coarse Labels & Masks
(1) Actual settings for Appendix A.3, Eqs. (12)–(16). All three datasets use scale weights (0.1, 0.2, 1.0), loss coefficients (CE, sem, geo, Lovász) = (1.0, 0.25, 0.25, 1.0), Lovász start epoch 0, and CE label smoothing 0.1.
CE frequency weighting is enabled; extra Lovász frequency weighting is disabled ( at all scales). Lovász does not inherit CE frequencies. CE uses , normalized to mean 1:
| Dataset | Fixed CE frequency source | Classes / empty ID | CE weight range |
|---|---|---|---|
| SurroundOcc | Saved valid-voxel counts after densifying 28,130 training samples; original sparse label 0 treated as ignore | 17 / 0 | 0.693695–1.294716 |
| Occ3D | Fixed COTR nusc_class_frequencies table | 18 / 17 | 0.718027–1.285207 |
| SSCBench-KITTI360 | Checkpoint array identified in the config as the official SSCBench/KITTI360 frequency table | 19 / 0 | 0.714960–1.342073 |
These saved arrays are fixed and do not change with class-aware sampling. We will include the exact arrays in the released configs.
All three datasets use sem_scal_exclude_empty=True: only the empty-class loss term is excluded. Empty voxels remain negatives for other classes' precision/specificity; only 255 is ignored. Occ3D class 0 (others) participates; 17 is free.
(2) Coarse labels and validity. On Occ3D, apply mask_camera to full-resolution gt_occ (200×200×16), setting invisible voxels to 255. Generate each scale independently from this masked GT using non-overlapping 4×4×4 / 2×2×2 / 1×1×1 blocks, yielding 50×50×4 / 100×100×8 / 200×200×16 grids. No nearest-neighbor mask resizing or cascaded label downsampling is used.
For SurroundOcc, vote over valid sparse labels. For Occ3D/SSCBench, exclude empty/free and 255 from occupied-class voting; output free only if no occupied votes but valid free voxels exist, or 255 if all voxels are ignored. Ties select the smallest class ID via votes.argmax, independently of input order. Coarse validity is the OR of fine-scale validities. We will provide configs and pseudocode in Appendix A and the code release.
The author addressed most of my concerns. I still give a positive rating.
Part 1/2 — Executive Summary, Reply to W1, and Reply to W2
[Part 1/2 — Executive Summary, Reply to W1, and Reply to W2]
We thank Reviewer ksrR for the constructive review and for highlighting our controlled ablations and consistent gains across three benchmarks. We address your four points in two parts:
Executive Summary
seed 52) reaches 44.43% mIoU vs. our original 44.47% (mean 44.45 ± 0.03%, n=2)—both aboveCOTR (+Dr.Occ)(44.10%) and same-backboneOmega+DPT+COTR(40.27%; +4.20/+4.16 pp), while Stage 2's +0.21 pp is scoped as a single-run observation.no-Timemodel on SurroundOcc 8f, dynamic-class mIoU rises by +1.25 pp (22.36% vs. 21.11%, 7/8 classes up) vs. +0.11 pp on static classes, and 6,019-sample gate statistics show matchedcarqueries shift t=0 weight from 33.41% (<0.2 m/s) to 46.44% (10–20 m/s).E_start=0), with Occ3D applyingmask_cameraat200×200×16before independent4×4×4and2×2×2occupied-priority majority voting.Reply to W1: Stage 2 Jacobian Bias vs. Matched Depth-Based Weighting
We completed the requested controlled experiment on SurroundOcc. Only the Stage 2 bias is replaced; shared-parameter initialization, learned-scale parameterization, numerical stabilization, backbone, decoder, inputs, training schedule, and evaluation/checkpoint selection are identical. Stages 1 and 3 remain unchanged.
The closely matched results support the projection-scale interpretation of Stage 2. Under the zero-skew pinhole, identity image-transform, and isotropic normalization assumptions in Appendix B.2, σmin(J)=fˉ/Z. Thus sh,llog(σmin(J)+ϵ) takes the form of a stabilized log-depth prior. Both biases improve over no Stage 2 bias; their differences are only 0.04 mIoU and 0.06 IoU in these runs.
The Jacobian provides a unified geometric descriptor for point weighting and observation fusion. It incorporates the actual normalized camera projection and image transform directly, with constant per-point computation. Stage 2 uses its minimum singular value at each shifted sample; Stage 3 uses the full 2×3 Jacobian at the reference point, where geometry conditioning adds +0.34 mIoU (25.02 to 25.36, Table 5). The direct comparison clarifies the role of Stage 2, while the Stage 3 ablation supports using projection geometry throughout observation fusion.
Reply to W2: Multi-Seed Stability on Occ3D & Single-Run Scope of Ablations
Each submitted model was trained once for 30 epochs (effective BS 64, best-val EMA, shared initialization across intra-backbone ablations). We ran an independent seed (
seed 52) on Occ3D-nuScenes alongside a same-backboneCOTRbaseline (Omega + DPT + COTR, keeping COTR's encoder, group decoder/losses, and LiDAR depth loss):seed 52)