acceptodds
Under review as a conference paper at ICLR 2027

VGGT-Occ: Geometry-Guided Reasoning Across Views and Scales for 3D Semantic Occupancy Prediction

13 views

Abstract

3D semantic occupancy prediction requires consistent reasoning across camera views, temporal observations, and spatial scales. Yet reference-point projection alone does not carry camera geometry into subsequent sampling, weighting, and cross-camera aggregation. High-resolution decoding must also balance coarse 3D context with the local detail needed for fine-scale prediction. We present VGGT-Occ, a geometry-guided framework for vision-only 3D semantic occupancy prediction across views, time, and scales. Across views and time, Projection-Aware Deformable Attention (PA-DA) carries projection geometry through three stages: it projects learned 3D offsets into each view, applies a Jacobian-based attention bias, and predicts channel-wise fusion weights conditioned on semantic features, projection geometry, and temporal context to aggregate observations across cameras and time. Across scales, a sequential coarse-to-fine decoder concentrates cross-view attention at coarse resolutions and propagates coarse 3D context through gated fusion to guide high-resolution refinement. VGGT-Occ outperforms the compared state-of-the-art methods in semantic mIoU across all three benchmarks. It achieves 37.24% IoU / 25.36% mIoU on SurroundOcc-nuScenes, 44.47% mIoU on Occ3D-nuScenes, and 45.06% IoU / 18.13% mIoU on SSCBench-KITTI-360. Code will be released publicly.

Primary area:
applications to robotics, autonomy, planning
Keywords:
occupancy, VGGT

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_ksrROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents VGGT-Occ, a vision-only framework for 3D semantic occupancy prediction using multiple camera views and temporal observations. It combines a pretrained VGGT-Omega encoder adapted with LoRA with two principal components. Projection-Aware Deformable Attention incorporates shared 3D sampling offsets, a projection-Jacobian attention bias, and channel-wise observation fusion conditioned on image features, geometry, and temporal metadata. A sequential coarse-to-fine decoder applies cross-view attention at coarse resolutions and uses coarse-guided gated fusion with local convolutional refinement at the finest resolution. The method reports semantic mIoU of 25.36% on SurroundOcc-nuScenes, 44.47% on Occ3D-nuScenes, and 18.13% on SSCBench-KITTI-360, exceeding the methods included in the respective comparisons. Controlled ablations on SurroundOcc support the contributions of observation fusion and coarse guidance, with additional evidence that PA-DA improves performance with both VGGT-Omega and DINOv3-L backbones.

Critical strengths

1.The paper provides meaningful controlled evidence for its principal components. PA-DA improves semantic mIoU by 1.63 percentage points with VGGT-Omega and 1.24 points with DINOv3-L under the respective matched decoder settings. Full observation conditioning improves upon a feature-only gate by 0.91 points, and learned coarse fusion improves upon fixed equal weighting by 0.53 points. These comparisons substantiate benefits beyond simply adopting a stronger backbone or adding an unexamined fusion module, which is the strongest methodological argument for acceptance.

2.The complete system performs well across three benchmark protocols spanning surround-view occupancy prediction and monocular temporal scene completion. The SSCBench-KITTI-360 result is particularly consequential: with four input frames, the method improves upon the listed four-frame S2GO-Base result by 3.03 percentage points in semantic mIoU and 4.26 points in geometric IoU. Although the comparisons do not fully control backbone capacity and pretraining, the results provide useful evidence that the overall design is effective beyond a single annotation protocol.

Critical weaknesses

1.To clarify the interpretation of Stage 2, how does the full Jacobian bias compare with a depth-based bias using the same learned scales and numerical stabilization under the cameras used in these experiments? Appendix B.2 already establishes their equivalence under specific assumptions. A controlled comparison would determine whether the observed benefit requires the more general Jacobian formulation or primarily reflects depth weighting. 2.How stable are the principal improvements across independent training seeds, particularly the 0.37 percentage point Occ3D advantage and the 0.21 point contribution of Stage 2 on top of Stages 1 and 3? Please provide any available repeated-run results or uncertainty estimates, and clarify whether the reported configurations were each trained only once. 3.Does temporal projection include any compensation for independently moving objects beyond the rigid camera transformations in Equation (1)? If it does not, what evidence shows that the observation gate suppresses stale object evidence when using seven preceding nuScenes keyframes? A dynamic-class analysis with and without temporal conditioning would directly inform the claimed benefit of temporal fusion. 4.For reproducibility, what exact loss coefficients and optional class-balancing settings were used on each benchmark, and how were coarse-scale labels and validity masks constructed?

Questions for authors

see Weaknesses

3 comments
AuthorsOpenReview ↗

Part 1/2 — Executive Summary, Reply to W1, and Reply to W2

[Part 1/2 — Executive Summary, Reply to W1, and Reply to W2]

We thank Reviewer ksrR for the constructive review and for highlighting our controlled ablations and consistent gains across three benchmarks. We address your four points in two parts:

  • Part 1 (here): Executive Summary, Reply to W1, and Reply to W2
  • Part 2: Reply to W3 and Reply to W4

Executive Summary

  • W1 (Jacobian vs. Depth Bias): The completed matched log-depth control reaches 25.32 mIoU / 37.30 IoU, close to Jacobian's 25.36 / 37.24. Only the Stage 2 bias changes; learned-scale parameterization, stabilization, and training/evaluation settings are identical. The result supports the projection-scale interpretation, while the Jacobian provides a unified descriptor for Stages 2 and 3.
  • W2 (Seed Stability): Each submitted model was trained once (30ep EMA). An independent Occ3D seed (seed 52) reaches 44.43% mIoU vs. our original 44.47% (mean 44.45 ± 0.03%, n=2)—both above COTR (+Dr.Occ) (44.10%) and same-backbone Omega+DPT+COTR (40.27%; +4.20/+4.16 pp), while Stage 2's +0.21 pp is scoped as a single-run observation.
  • W3 (Motion & Dynamic Gate): Eq. (1) uses rigid camera transforms only. Against an independently trained no-Time model on SurroundOcc 8f, dynamic-class mIoU rises by +1.25 pp (22.36% vs. 21.11%, 7/8 classes up) vs. +0.11 pp on static classes, and 6,019-sample gate statistics show matched car queries shift t=0t=0 weight from 33.41% (<0.2 m/s) to 46.44% (10–20 m/s).
  • W4 (Reproducibility): All three benchmarks share scale weights (0.1, 0.2, 1.0) and loss weights (1.0,0.25,0.25,1.0)(1.0, 0.25, 0.25, 1.0) (E_start=0), with Occ3D applying mask_camera at 200×200×16 before independent 4×4×4 and 2×2×2 occupied-priority majority voting.

Reply to W1: Stage 2 Jacobian Bias vs. Matched Depth-Based Weighting

We completed the requested controlled experiment on SurroundOcc. Only the Stage 2 bias is replaced; shared-parameter initialization, learned-scale parameterization, numerical stabilization, backbone, decoder, inputs, training schedule, and evaluation/checkpoint selection are identical. Stages 1 and 3 remain unchanged.

Stage 2 bias (Stages 1+3 retained)mIoU (%)IoU (%)mIoU gain
None (Table 4)25.1536.89—
Log-depth (new)25.3237.30+0.17
Jacobian (ours)25.3637.24+0.21

The closely matched results support the projection-scale interpretation of Stage 2. Under the zero-skew pinhole, identity image-transform, and isotropic normalization assumptions in Appendix B.2, σmin⁡(J)=fˉ/Z\sigma_{\min}(J)=\bar f/Z. Thus sh,llog⁡(σmin⁡(J)+ϵ)s_{h,l}\log(\sigma_{\min}(J)+\epsilon) takes the form of a stabilized log-depth prior. Both biases improve over no Stage 2 bias; their differences are only 0.04 mIoU and 0.06 IoU in these runs.

The Jacobian provides a unified geometric descriptor for point weighting and observation fusion. It incorporates the actual normalized camera projection and image transform directly, with constant per-point computation. Stage 2 uses its minimum singular value at each shifted sample; Stage 3 uses the full 2×32\times3 Jacobian at the reference point, where geometry conditioning adds +0.34 mIoU (25.02 to 25.36, Table 5). The direct comparison clarifies the role of Stage 2, while the Stage 3 ablation supports using projection geometry throughout observation fusion.


Reply to W2: Multi-Seed Stability on Occ3D & Single-Run Scope of Ablations

Each submitted model was trained once for 30 epochs (effective BS 64, best-val EMA, shared initialization across intra-backbone ablations). We ran an independent seed (seed 52) on Occ3D-nuScenes alongside a same-backbone COTR baseline (Omega + DPT + COTR, keeping COTR's encoder, group decoder/losses, and LiDAR depth loss):

Occ3D Configuration (256×704, Cam-Mask)Protocol / RunsmIoU (%)
Omega + DPT + COTRSingle run, 30ep EMA; depth + group losses40.27
COTR (R50)Reported in Dr.Occ43.10
COTR + Dr.Occ (R50)Reported; adds MoGe-2 prior + region experts44.10
VGGT-Occ (orig. / indep. seed 52)Two indep. seeds, 30ep EMA (occ. losses only, 8f)44.47 / 44.43
VGGT-Occ Summary (n=2n=2)Mean ± sample std (range 0.04 pp)44.45 ± 0.03
  • Occ3D stability: The original and independent runs exceed the reported COTR (+Dr.Occ) score by +0.37/+0.33 pp, respectively, and the shared Omega+DPT+COTR baseline by +4.20/+4.16 pp. Their 0.04-point spread supports consistent full-model performance across the two independent runs. We will report the mean and sample standard deviation and include ALOcc-3D's 45.5%–50.6% results with its 16f caching and depth supervision annotated.
  • Stage 2 (+0.21 pp): Stage 2 adds +0.36 pp over S1 (23.97% → 24.33%) and +0.21 pp over S1+S3 (25.15% → 25.36%, where S3 already uses Js,nJ_{s,n}). These component ablations each use one training run.
AuthorsOpenReview ↗

Part 2/2 — W3: Temporal Gate Analysis; W4: Reproducibility

[Part 2/2 — W3: Temporal Gate Analysis; W4: Reproducibility]

Reply to W3: Rigid Projection Scope, Dynamic-Class Ablation & Gate Behavior

(1) Rigid Projection and Dynamic-Class Ablation. Eq. (1) uses rigid occupancy-reference-to-camera transformations Es,nE_{s,n} only, with no explicit moving-object compensation or flow warping. Historical frames (t<0t < 0) thus aid static parallax yet project moving objects onto stale background features. Comparing Full SurroundOcc (8f) against an independently trained no-Time variant (omitting τs,n\tau_{s,n} in Eq. 4 while keeping vs,nv_{s,n} and Js,nJ_{s,n}) isolates where temporal conditioning acts:

SurroundOcc 8f SubsetFull (w/ τs,n\tau_{s,n})no-Time (Indep. Run)Gain (Full − no-Time)
Dynamic 8 Classes (7/8 improve*)22.36%21.11%+1.25 pp
Static 8 Classes (drive. surf., sidewalk, etc.)28.36%28.25%+0.11 pp
Overall 16-Class mIoU / Geometric IoU25.36% / 37.24%24.68% / 36.71%+0.68 pp / +0.53 pp

*Dynamic deltas (pp): truck +2.01, bus +1.84, bicycle +1.58, motorcycle +1.52, pedestrian +1.31, car +1.17, trailer +0.76, const. veh. -0.19.

Dynamic classes contribute ~92% of the mIoU gain from temporal conditioning.

(2) Temporal Gate Analysis on 6,019 Samples. Auditing normalized Eq. (6) weights across all 6,019 SurroundOcc val samples at the final PA-DA gate (S1) shows:

  • Semantic groups: Dynamic-class queries assign 32.92% weight to t=0t=0 vs. 28.64% for static queries (uniform valid-observation shares: 16.44% vs. 16.69%).
  • Speed-stratified car queries (strictly matched inside 3D GT boxes; trajectory finite-difference speed):
GT car Speed (m/s)<0.20.2–0.50.5–22–55–1010–20
Query-Equal t=0t=0 Share (%)33.4135.0941.4642.8445.7146.44
Object-Frame-Equal t=0t=0 Share (%)32.0633.7740.4842.5645.9547.11
Uniform Valid-Obs. Baseline (%)15.7316.8116.4815.7117.1315.71

From <0.2 to 10–20 m/s, current-frame share rises by 13.03 pp (15.05 pp object-frame-equal), while the uniform reference shows no corresponding monotonic rise. The share of older lags (Δt≤−1.0\Delta t\le-1.0 s) falls from 49.79% to 37.40%, with 53.56% still assigned to history at 10–20 m/s. Together, the dynamic-class gains and temporal reweighting provide compelling evidence of motion-sensitive observation fusion without flow supervision.


Reply to W4: Loss Settings, Class Frequencies, Coarse Labels & Masks

(1) Actual settings for Appendix A.3, Eqs. (12)–(16). All three datasets use scale weights (0.1, 0.2, 1.0), loss coefficients (CE, sem, geo, Lovász) = (1.0, 0.25, 0.25, 1.0), Lovász start epoch 0, and CE label smoothing 0.1.

CE frequency weighting is enabled; extra Lovász frequency weighting is disabled (βr,c=1\beta_{r,c}=1 at all scales). Lovász does not inherit CE frequencies. CE uses ω~c=1/log⁡(fc+10−3)\tilde\omega_c=1/\log(f_c+10^{-3}), normalized to mean 1:

DatasetFixed CE frequency sourceClasses / empty IDCE weight range
SurroundOccSaved valid-voxel counts after densifying 28,130 training samples; original sparse label 0 treated as ignore17 / 00.693695–1.294716
Occ3DFixed COTR nusc_class_frequencies table18 / 170.718027–1.285207
SSCBench-KITTI360Checkpoint array identified in the config as the official SSCBench/KITTI360 frequency table19 / 00.714960–1.342073

These saved arrays are fixed and do not change with class-aware sampling. We will include the exact arrays in the released configs.

All three datasets use sem_scal_exclude_empty=True: only the empty-class loss term is excluded. Empty voxels remain negatives for other classes' precision/specificity; only 255 is ignored. Occ3D class 0 (others) participates; 17 is free.

(2) Coarse labels and validity. On Occ3D, apply mask_camera to full-resolution gt_occ (200×200×16), setting invisible voxels to 255. Generate each scale independently from this masked GT using non-overlapping 4×4×4 / 2×2×2 / 1×1×1 blocks, yielding 50×50×4 / 100×100×8 / 200×200×16 grids. No nearest-neighbor mask resizing or cascaded label downsampling is used.

For SurroundOcc, vote over valid sparse labels. For Occ3D/SSCBench, exclude empty/free and 255 from occupied-class voting; output free only if no occupied votes but valid free voxels exist, or 255 if all voxels are ignored. Ties select the smallest class ID via votes.argmax, independently of input order. Coarse validity is the OR of fine-scale validities. We will provide configs and pseudocode in Appendix A and the code release.

The author addressed most of my concerns. I still give a positive rating.

Reviewer_nZTEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VGGT-Occ, a geometry-guided framework for vision-only 3D semantic occupancy prediction from multi-view and multi-temporal images. The method addresses two main challenges: maintaining geometric consistency throughout 2D-to-3D feature lifting and efficiently refining dense 3D voxel representations across spatial scales.

The framework employs a pretrained VGGT-Omega encoder with LoRA adaptation to extract multi-level image features. Its first contribution, Projection-Aware Deformable Attention (PA-DA), improves 2D-to-3D feature lifting through geometry-guided sampling, attention weighting, and adaptive multi-view and temporal feature fusion. Its second contribution is a sequential coarse-to-fine decoder, which progressively increases voxel resolution through learnable patch splitting and uses coarse-guided gated fusion to balance global 3D context with fine-scale details.

The method is evaluated on three benchmarks: SurroundOcc-nuScenes, Occ3D-nuScenes, and SSCBench-KITTI-360, achieving 25.36%, 44.47%, and 18.13% semantic mIoU, respectively. Extensive ablations examine the individual PA-DA stages, observation-gate conditioning, backbone choices, and coarse-to-fine fusion strategies. The results demonstrate consistent improvements from the proposed components.

Critical strengths

S1. Clear motivation and well-formulated methodology. The paper identifies concrete limitations in existing 2D-to-3D feature lifting and multi-scale refinement. The proposed modules are well motivated, and the overall method is presented in a clear and coherent manner. The overall performance is also competitive, particularly on semantic mIoU.

S2. Comprehensive experiments and convincing ablations. The experiments cover three benchmarks, with extensive ablations examining individual PA-DA stages, observation-gate conditioning, backbone choices, and coarse-to-fine fusion. These results provide convincing evidence for the effectiveness of the proposed components. The paper is also well organized, provides detailed experimental configurations, and includes useful qualitative visualizations.

Critical weaknesses

W1. The contribution of the proposed architecture to the final SOTA performance is not sufficiently disentangled from the strong backbone, and the system-level efficiency comparison remains limited.

Table 6 shows that replacing VGGT-Omega with DINOv3-L reduces mIoU from 25.36 to 22.31, making the model less competitive than S2GO-Base (4f, 22.72) and SuperOcc-L (8f, 24.55). Although the ablations demonstrate that PA-DA improves performance under both backbones, the final performance advantage appears strongly dependent on VGGT-Omega. It remains unclear how much of the overall gain over prior methods comes from the proposed architecture versus backbone capacity and pretraining.

This concern is further amplified by the efficiency comparison in Table 8, which does not include the strongest competing methods, such as S2GO-Base and SuperOcc-L. Given the large encoder and substantial inference cost, a more comprehensive accuracy-efficiency comparison would help establish the practical advantages of the proposed design. At minimum, reporting backbone architectures, parameter counts, and computational costs alongside the main results would make the comparison more informative.

W2. Incomplete comparisons with existing state-of-the-art methods weaken the overall performance claims.

Table 3 on SSCBench-KITTI-360 omits several strong semantic scene completion methods, including Symphonies [Jiang et. al, 2024], CGFormer [Nan et. al, 2026], and FoundationSSC [Chen et. al, 2026], all reporting higher semantic mIoU than VGGT-Occ. While these methods differ in their input configurations and use of depth priors, they are relevant comparisons and should be included or explicitly discussed with appropriate modality annotations.

Similarly, Table 2 largely follows the baseline collection from Dr.Occ, and it is unclear whether more recent camera-only occupancy methods have been considered. These omissions make the claimed superiority across three benchmarks less convincing than the current presentation suggests.

Questions for authors

1. Backbone-controlled comparison and performance attribution. Table 6 shows a substantial performance drop when replacing VGGT-Omega with DINOv3-L. Could the authors provide a more controlled comparison under the same backbone, for example, by evaluating representative occupancy heads from SurroundOcc or COTR using VGGT-Omega features under comparable training settings? If such experiments are infeasible, at minimum, could the authors annotate the backbone architecture, parameter count, and input configuration for the methods in the main comparison tables?

2. Completeness and fairness of SOTA comparisons. Table 3 omits several competitive SSCBench-KITTI-360 methods, including Symphonies, CGFormer, and FoundationSSC, which report higher semantic mIoU. Could the authors include or discuss these methods with appropriate input-modality annotations? Similarly, could the authors clarify the selection of baselines in Table 2 and whether more recent camera-only occupancy methods have been considered?

3. Accuracy-efficiency trade-off against stronger baselines. Table 8 mainly compares against BEVDetOcc and GaussianFormer-2, while the strongest competitors in Table 1, such as S2GO-Base and SuperOcc-L, are absent. Could the authors provide inference latency, memory consumption, and computational cost comparisons against these methods, or explain why such comparisons are unavailable?

4. Temporal evaluation on SSCBench-KITTI-360. Table 3 only reports the 4-frame setting. Could the authors provide a single-frame result, or preferably a 1f/2f/4f comparison?

5. Geometric versus semantic occupancy performance. On SurroundOcc-nuScenes, VGGT-Occ achieves higher semantic mIoU than SuperOcc-L but lower geometric IoU (37.24 vs. 38.13). Could the authors explain this difference?

4 comments
AuthorsOpenReview ↗

Part 1/3 — Executive Summary and Reply to Q1 (W1)

[Part 1/3 — Executive Summary and Reply to Q1 (W1)]

We thank Reviewer nZTE for recognizing our clear motivation, coherent methodology, and extensive ablations. Our response is organized into three parts:

  • Part 1 (here): Executive Summary and Reply to Q1 (W1)
  • Part 2: Reply to Q2 (W2)
  • Part 3: Reply to Q3 (W1), Reply to Q4, and Reply to Q5

Executive Summary

  • Q1 (W1): Gains are not solely from backbone scale—under our exact shared VGGT-Omega (LoRA r=8) + DPT pipeline on Occ3D (retaining COTR's LiDAR depth & group losses), our downstream head exceeds COTR by +4.20 pp (44.47% vs. 40.27% mIoU); seed 52: 44.43%, alongside consistent PA-DA gains under a fixed C2F decoder on both DINOv3-L (+1.24) and Omega (+1.63).
  • Q2 (W2): Expanding Tables 1–3 with Backbone, Params, History, and Modality columns scopes our comparisons accurately: (1) highest vision-only semantic mIoU among compared methods on SurroundOcc (25.36%), with SuperOcc-L higher in binary IoU (38.13% vs. 37.24%); (2) 18.13% mIoU under stereo-free, occ.-supervised monocular image_00 on SSCBench-KITTI-360 (alongside 18.58–21.78% stereo-assisted results and VoxSAMNet's 20.21% with text guidance and depth loss); and (3) 44.47% mIoU on uncached 8f Occ3D (explicitly crediting ALOcc-3D's higher 45.5%–50.6% with 16f voxel caching and depth supervision).
  • Q3 (W1): We now include same-GPU latency and memory measurements for SuperOcc-L and ALOcc-3D. Under the shared C2F decoder, PA-DA adds only 3.92/4.47 ms (+2.7%/+2.9%); backbone encoding dominates total cost. For S2GO-Base, we report its published 19.6 FPS on RTX 4090.
  • Q4 & Q5: Monocular image_00 KITTI-360 scales cleanly across 1f (14.21%) → 2f (16.43%) → 4f (18.13% mIoU) (2f already exceeds S2GO-Base's 15.05% (4f streaming queue)), and per-class SurroundOcc scores show VGGT-Occ leads SuperOcc-L on 10/16 classes (especially smaller foreground categories), supporting stronger class-balanced semantic accuracy.

Reply to Q1 (W1): Same-Backbone Controlled Comparison & Table Annotations

Following your suggestion, we compare COTR and ours using the same pretrained VGGT-Omega-1B initialization, LoRA (r=8), DPT, inputs (8×6×256×704), and training/evaluation recipe on Occ3D (30 epochs, effective BS 64, peak lr=1e-5, camera-mask supervision/evaluation, best-val EMA). COTR retains its occupancy encoder, coarse-to-fine group decoder, group losses, and LiDAR depth loss:

Occ3D-nuScenes Pipeline (256×704)Pretraining & Supervision ProtocolmIoU (%)
COTR (reported, R50)BEVDet pretrain + BEVStereo depth loss43.10
COTR + Dr.Occ (reported, R50)Adds MoGe-2 prior and region experts44.10
Omega + DPT + COTR (controlled)Shared Omega (LoRA r=8) + LiDAR depth loss + COTR group/occ. losses40.27
Omega + DPT + Ours (controlled)Shared Omega (LoRA r=8) + occ. losses only (no LiDAR depth loss)44.47

Controlled attribution and cross-backbone evidence:

  • Same-backbone architecture contribution: COTR / +Dr.Occ scores of 43.10/44.10% are reported in Dr.Occ, not our reproductions. Under the matched Omega+DPT setup, COTR reaches 40.27% with its depth and group losses, while ours reaches 44.47% with occupancy losses only (+4.20 pp). This directly demonstrates our occupancy architecture's contribution under a shared backbone. An independent run reaches 44.43% (two-run mean: 44.45%, n=2).
  • Cross-backbone evidence on SurroundOcc (Table 6): With the C2F decoder fixed, PA-DA improves DINOv3-L by +1.24 mIoU and VGGT-Omega by +1.63 mIoU. These consistent gains demonstrate that PA-DA's contribution generalizes across backbones. The full VGGT-Omega model reaches 25.36% mIoU, exceeding SuperOcc-L (24.55%) and S2GO-Base (22.72%). We will annotate Backbone, Parameter Count, Temporal History, and Modality across Tables 1–3.
AuthorsOpenReview ↗

Part 2/3 — Reply to Q2 (W2): SOTA Completeness, Modality Audits & Separated Benchmark Tables

[Part 2/3 — Reply to Q2 (W2): Benchmark Completeness and Protocols]

Reply to Q2 (W2): Completeness & Protocol Scope of Benchmark Comparisons

(1) SSCBench-KITTI-360 Protocol Rationale. Following S2GO, we benchmark occupancy prediction from a single-camera RGB sequence (image_00). Our 1f/2f/4f comparison tests single-frame prediction and the ability to exploit temporal observations. The goal is geometry recovery and temporal fusion from monocular RGB. Stereo provides a calibrated baseline for depth estimation; supplying stereo-derived depth adds geometric information outside this protocol.

SSCBench-KITTI-360 Comparison

Stereo informationMethodBackboneTest-time input / historyIoU (%)mIoU (%)
Not usedOccFormerEfficientNet-B7Mono RGB, 1f40.2713.81
GaussianFormer-2R50 (ImageNet)Mono RGB, 1f38.3713.90
S2GO-BaseR50Mono RGB, streaming 4f queue40.8015.05
VGGT-OccVGGT-Omega (LoRA)Mono RGB, 1f41.0514.21
VGGT-OccVGGT-Omega (LoRA)Mono RGB, 2f43.1916.43
VGGT-OccVGGT-Omega (LoRA)Mono RGB, 4f45.0618.13
VoxSAMNet*Swin TransformerMono RGB, 1f47.2220.21
UsedFoundationSSCFoundationStereo ViT-LStereo RGB, 1f48.6121.78
SymphoniesR50 + MaskDINORGB + stereo depth, 1f44.1218.58
CGFormerEfficientNet-B7RGB + stereo depth, 1f48.0720.05
DISCR50 + MaskDINORGB + stereo depth, 1f47.3520.55
VisHall3DR50 + MaskDINORGB + stereo depth, 1f49.1220.95

* VoxSAMNet has not publicly released code or pretrained weights; its results are quoted from the paper.

The groups distinguish stereo RGB or stereo-derived depth from stereo-free inputs. Ours uses occupancy losses only; S2GO includes Stage-1 LiDAR/depth/RGB pretraining. The four RGB+depth methods use MobileStereoNet depth from the left/right pair; FoundationSSC uses both RGB views directly. Input sizes differ: ours uses 384×1408, S2GO 256×1408, and GaussianFormer-2 376×1408 (H×W). Table 3 rounded baseline scores to one decimal; here we retain the original reported precision and will update Table 3 and the corresponding margins.

SSCBench conclusion. Under stereo-free monocular RGB input with occupancy-only downstream supervision, VGGT-Occ leads the compared methods at 18.13% mIoU, exceeding S2GO-Base by 3.08 points (2f: +1.38).

VoxSAMNet reproducibility. Its repository is public, but neither code nor pretrained weights has been released as of this rebuttal, so its 20.21% mIoU cannot yet be independently reproduced from official artifacts. We include its paper-reported result and explicitly annotate its additional depth supervision and TGIF text guidance (Sec. 3.2, Eq. 11).

(2) Occ3D and SurroundOcc Comparisons. Table 2 followed Dr.Occ's camera-mask baseline collection. We will add ALOcc-3D (ICCV'25) and annotate each method's backbone, temporal input, and supervision.

Occ3D-nuScenes (Camera Input, Camera-Mask Supervision)

MethodBackboneDepth cuesTemporal / H×WmIoU (%)
COTR (reported in Dr.Occ)R50BEVStereo + depth loss9f + 1 stereo ref., 256×70443.10
COTR + Dr.OccR50BEVStereo + MoGe-2 prior9f + 1 stereo ref., 256×70444.10
VGGT-OccVGGT-Omega (LoRA)No depth loss8f uncached, 256×70444.47
ALOcc-3DR50Temporal stereo + depth loss16f cache, 256×70445.5
ALOcc-3DIntern-TTemporal stereo + depth loss16f cache, 256×70447.5
ALOcc-3DSwin-BTemporal stereo + depth loss16f cache, 512×140850.6

SurroundOcc-nuScenes (Vision-Only Comparison)

MethodBackboneSupervisionTemporal / H×WIoU / mIoU (%)
S2GO-BaseR50Occ. + Stage-1 LiDAR/depth/RGB4f queue, streaming, 256×70435.46 / 22.72
SuperOcc-LR50Occ. + voxel-center loss8f, 256×70438.13 / 24.55
VGGT-OccVGGT-Omega (LoRA)Occ. losses only8f uncached, 256×70437.24 / 25.36

ALOcc's official configuration uses a 16-slot voxel cache, adjacent-frame temporal stereo, BEVDet pretraining, and LiDAR depth loss. Our model uses an uncached 8-frame window and occupancy-only downstream supervision. On SurroundOcc, VGGT-Occ reaches 25.36% mIoU, exceeding SuperOcc-L (24.55%) and S2GO-Base (22.72%). On Occ3D, we will credit ALOcc's higher scores and revise the SOTA wording accordingly. The SSCBench results demonstrate strong monocular temporal learning, with 14.21→16.43→18.13 mIoU across 1f/2f/4f.

AuthorsOpenReview ↗

Part 3/3 — Reply to Q3 (W1), Reply to Q4, and Reply to Q5

[Part 3/3 — Reply to Q3 (W1), Reply to Q4, and Reply to Q5]

Reply to Q3 (W1): Accuracy–Efficiency Trade-Off Against Stronger Baselines

Beyond BEVDetOcc and GaussianFormer-2 (Table 8), we profiled SuperOcc-L and ALOcc-3D alongside our DINOv3-L and VGGT-Omega models on RTX PRO 6000 and RTX 5090 (256×704):

System Latency, Parameters & Memory (256×704)

Method / VariantParams (Tot / Train)Temporal ModePRO 6000 ms (Enc + Head)RTX 5090 ms (Enc + Head)Peak Mem (PRO / 5090)
ALOcc-3D (R50)52.9M / 52.9M1×6 + 16f cache102.77152.4210.21 / 11.64 GiB
SuperOcc-L (R50)65.5M / 65.3M*8f98.67168.777.08 / 7.77 GiB
DINOv3-L + PA-DA344.5M / 41.4M8×6 uncached259.79 (122.47+137.32)299.26 (148.25+151.01)12.14 / 13.91 GiB
Omega + PA-DA955.7M / 47.8M8×6 uncached581.08 (433.05+148.03)824.75 (663.64+161.11)13.76 / 14.68 GiB

SuperOcc-L params: official SurroundOcc-L checkpoint; trainable count follows its frozen-stage config.

SurroundOcc IoU/mIoU: SuperOcc-L 38.13/24.55; DINOv3-L 34.56/22.31; VGGT-Omega 37.24/25.36. Occ3D mIoU: ALOcc-R50 45.5; VGGT-Omega 44.47. DINOv3-L is evaluated as a SurroundOcc backbone ablation.

Memory values are reported in GiB, their original measurement unit; the GB label in Table 8 will be corrected.

Isolated PA-DA Overhead Under Shared C2F Decoder (VGGT-Omega, 640K Voxels)

Occupancy Decoder (Shared C2F)PRO 6000 Head msRTX 5090 Head msSurroundOcc IoU / mIoU
C2F + Standard Deformable Attn.144.11156.6435.14 / 23.73
C2F + PA-DA (Ours)148.03 (+3.92 ms, +2.7%)161.11 (+4.47 ms, +2.9%)37.24 (+2.10) / 25.36 (+1.63)
  • PA-DA's accuracy-overhead trade-off: Under the same C2F architecture, PA-DA adds 3.92/4.47 ms (+2.7%/+2.9% decoder latency) for +1.63 mIoU (+1.24 on DINOv3-L). Full-model memory is 12.14–13.76 GiB on PRO 6000 and 13.91–14.68 GiB on 5090. The Omega backbone contributes 74.5%–80.5% of total latency. This identifies backbone encoding as the main cost; switching to DINOv3-L reduces runtime by 55%–64%.
  • Strong-baseline coverage: We include measured SuperOcc-L latency and memory. S2GO-Base reports 19.6 FPS on RTX 4090 at 22.72 mIoU (its Table 1); a matched measurement on our two GPUs is not available.

Reply to Q4: Temporal Evaluation (1f / 2f / 4f) on SSCBench-KITTI-360

We trained 1f and 2f VGGT-Occ variants on monocular image_00 SSCBench-KITTI-360 (1408×384) under the exact same stereo-free, occupancy-supervised setup:

VGGT-Occ Temporal Setting (image_00 only, 1408×384)Completion IoU (%)Semantic mIoU (%)Gain over 1-Frame
1-Frame (1f)41.0514.21—
2-Frame (2f)43.1916.43+2.14 IoU / +2.22 mIoU
4-Frame (4f, Paper)45.0618.13+4.01 IoU / +3.92 mIoU

At 1f, VGGT-Occ exceeds GaussianFormer-2 (13.90%) and OccFormer (13.81%); 2f surpasses S2GO-Base (4f streaming queue) by +1.38 mIoU (+2.39 IoU), and 4f adds +3.92 mIoU / +4.01 IoU over 1f, supporting multi-frame lifting from monocular parallax.


Reply to Q5: Geometric IoU vs. Semantic mIoU on SurroundOcc-nuScenes

Binary IoU evaluates occupied/free geometry after merging all 16 classes; mIoU also evaluates semantic labels and weights each class equally. All 16 per-class IoU differences (VGGT-Occ − SuperOcc-L, Table 1) are:

Table 1 SubsetPer-Class IoU Differences (VGGT-Occ − SuperOcc-L, %)Metric Impact
10/16 Classes (Ours Higher)bicycle +6.20, motorcycle +4.64, pedestrian +2.42, traffic cone +2.35, trailer +2.01, barrier +1.47, manmade +0.89, truck +0.48, other flat +0.43, bus +0.11Lifts class-equal mIoU (+0.81: 25.36 vs. 24.55)
6/16 Classes (SuperOcc Higher)driveable surf. -3.01, const. veh. -1.79, sidewalk -0.97, terrain -0.86, car -0.72, vegetation -0.59Different weighting from binary IoU (37.24 vs. 38.13)

VGGT-Occ improves 10 of 16 classes, with its strongest gains on smaller foreground categories, raising class-equal semantic mIoU by 0.81 points. SuperOcc-L leads in the remaining six classes and in binary IoU. The two metrics therefore reflect different aspects of prediction quality: geometric occupancy coverage and class-balanced semantic accuracy.

Thank you for the clarifications and additional experiments. I will raise my rating to Weak Accept.

If the paper is accepted, I encourage the authors to incorporate the new results from the discussion phase into the final version, particularly by revising the discussions and comparisons across all three benchmarks, as well as the inference cost and efficiency analysis.

Reviewer_UQ9eOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces VGGT-Occ, a vision-only method for temporal multi-camera 3D semantic occupancy prediction. Its motivation is to carry projection geometry consistently through the 2D-to-3D lifting pipeline rather than applying it only when projecting reference points, and to balance coarse 3D context against fine-scale detail across spatial scales. Projection-Aware Deformable Attention (PA-DA) injects projection geometry at two levels: into the aggregation of sampling points within a single camera-time observation, through cross-view-consistent 3D offsets and a Jacobian-based attention bias, and into the fusion of features across observations, through observation-aware channel-wise gating. A sequential coarse-to-fine decoder applies different feature-propagation operations at coarse and fine voxel resolutions and combines features from adjacent resolutions through gated fusion. Extensive experiments on SurroundOcc-nuScenes, Occ3D-nuScenes, and SSCBench-KITTI-360 report the highest semantic mIoU among the compared methods, and ablation studies on SurroundOcc-nuScenes quantify the contribution of each component.

Critical strengths
  1. The motivation is well grounded and offers genuine insight. Carrying projection geometry through the temporal multi-camera fusion process, rather than confining it to reference-point projection, is a problem worth studying in 3D semantic occupancy prediction, and the three-stage formulation of PA-DA makes a coherent case for why each stage is needed.

  2. The paper is clearly organized and well written. The three stages of PA-DA and the sequential coarse-to-fine decoder are easy to follow, and the figures support the exposition rather than merely restating it.

  3. The empirical evaluation is broad, spanning three real-world driving datasets, and the ablation studies clearly isolate the contribution of each individual component.

Critical weaknesses
  1. First, in Table 8 VGGT-Occ is faster than GaussianFormer-2 on the RTX PRO 6000 (581.08 ms vs 729.48 ms) but slower on the RTX 5090 (824.75 ms vs 768.05 ms), and no explanation is given for this reversal. Second, Section 4.4 reports the latency that PA-DA adds over standard deformable attention (148.03 vs 144.11 ms on the RTX PRO 6000, and 161.11 vs 156.64 ms on the RTX 5090), but these figures appear only in the running text. They are essential to judging PA-DA as a whole, and I recommend merging them into Table 8 so that readers can see the overhead at a glance rather than reconstructing it from the prose. Third, Table 6 compares VGGT-Ω-1B and DINOv3-L in mIoU only. I recommend adding latency, parameter count, and memory for both backbones, so that the gain of 2.68 IoU and 3.05 mIoU can be weighed against the computational cost that VGGT-Ω incurs as a backbone.

  2. The name VGGT-Occ does not convey the paper's core contribution, and using a VGGT-style backbone is by now well established in this area [1-3]. I therefore suggest a name that makes the actual novelty explicit — that projection geometry is carried through sampling, attention weighting, and observation fusion when aggregating temporal multi-view features — rather than one built around the backbone.

  3. I am curious whether the observation gates γs,n\gamma_{s,n} in Eq. (6) can adaptively down-weight the inconsistency that dynamic objects induce across historical frames. The motivation comes from Table 5: in the leave-one-out comparison, removing the Time input reduces mIoU to 24.68%, whereas removing the Jacobian input reduces it only to 25.02%, so the Time conditioning carries a clearly larger share of the Stage 3 gain. A quantitative or qualitative analysis of how γs,n\gamma_{s,n} is distributed across time offsets and across dynamic versus static regions would therefore be valuable, as it would clarify what this gain actually reflects.

[1] Yan X, Pei M, Shen S. VG3S: Visual Geometry Grounded Gaussian Splatting for Semantic Occupancy Prediction[J]. arXiv preprint arXiv:2603.06210, 2026.

[2] Kim J, Lee S. Vg3t: Visual geometry grounded gaussian transformer[C]//2026 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2026: 2084-2091.

[3] Lin J, Guo X, Wang K, et al. VGOcc: Learning Visual-Geometric Gaussians for Vision-Centric 3D Driving Occupancy Prediction[J]. arXiv preprint arXiv:2607.18078, 2026.

Questions for authors

See Weaknesses

5 comments
AuthorsOpenReview ↗

Part 1/2 — Executive Summary and Reply to W1 (GPU Ranking, PA-DA Overhead & Backbone Cost)

[Part 1/2 — Executive Summary and Reply to W1 (GPU Ranking, PA-DA Overhead & Backbone Cost)]

We thank Reviewer UQ9e for the positive assessment and for recognizing our geometry-consistent motivation, clear presentation, and thorough ablations. We address all points from your review in two parts:

  • Part 1 (here): Executive Summary and Reply to W1 (GPU Ranking, PA-DA Overhead & Backbone Cost)
  • Part 2: Reply to W2 (Method Naming & Works [1–3]) and Reply to W3 (Eq. (6) Gate Analysis)

Executive Summary

  • W1 (Efficiency & Tables 6/8): VGGT-Occ is 56.70 ms slower than GaussianFormer-2 on RTX 5090 but 148.40 ms faster on RTX PRO 6000. The backbone contributes 94.63% of VGGT-Occ's own 243.67 ms cross-GPU latency reduction, explaining its greater improvement. We will tabulate PA-DA's 3.92/4.47 ms overhead (+2.7%/+2.9%) for +1.63 mIoU and both backbones' latency, parameter counts, and memory.
  • W2 (Naming & [1–3]): We will cite and contrast VGGT-based Gaussian works [1–3] (VG3S, VG3T, VGOcc), highlight our backbone-agnostic projection-aware voxel lifting (+1.24 mIoU on DINOv3-L, +1.63 on Omega, and +4.20 pp over COTR under shared Omega+DPT on Occ3D), and retain VGGT-Occ while foregrounding PA-DA and coarse-guided fusion in the revised exposition.
  • W3 (Eq. (6) Gate γs,n\gamma_{s,n}): Against an independently trained no-Time model, dynamic-class mIoU rises by +1.25 pp (7/8 classes up) vs. +0.11 pp on static classes (~92% of the Table 5 Time gain), while 6,019-sample gate statistics show normalized observation weights shift from older lags (Δt≤−1.0 s\Delta t \le -1.0\text{ s}: 49.79% → 37.40%) to t=0t=0 (33.41% → 46.44%) as car speed increases.

Reply to W1: Cross-GPU Ranking Reversal, PA-DA Overhead & Backbone Cost (Tables 6 & 8)

(1) Why Runtime Ranking Reverses Between RTX 5090 and RTX PRO 6000 (Table 8). Decomposing our same-session Table 8 / Section 4.4 timings (TensorRT 10.15, mixed FP16, batch 1) isolates the source of the platform-dependent ranking:

Model / Component (SurroundOcc)RTX 5090 msPRO 6000 msChange (PRO 6000 − 5090)
VGGT-Occ Total (8×6×256×704)824.75581.08-243.67 ms (-29.5%)
↳ VGGT-Omega-1B Backbone (48 imgs)663.64433.05-230.59 ms (94.63% of reduction)
↳ C2F + PA-DA Occupancy Decoder161.11148.03-13.08 ms (5.37% of reduction)
GaussianFormer-2 (1×6×864×1600)768.05729.48-38.57 ms (-5.0%)
Gap (VGGT-Occ − GaussFormer-2)+56.70 ms-148.40 ms-205.10 ms

From RTX 5090 to RTX PRO 6000, VGGT-Occ's latency decreases by 243.67 ms: 230.59 ms comes from the backbone (94.63%) and 13.08 ms from the decoder. GaussianFormer-2 decreases by 38.57 ms, yielding a 205.10 ms shift in the relative gap. This breakdown locates VGGT-Occ's greater cross-GPU improvement primarily in its dense frame/global-attention backbone over 48 images. The two systems have different computational structures; we will report this measured component breakdown in Section 4.4.

(2) Merging PA-DA Overhead Directly into Table 8. We will add the paired decoder latency comparison directly to Table 8, keeping the VGGT-Omega backbone and coarse-to-fine decoder structure fixed:

Decoder Variant (Shared C2F, 640K Voxels)PRO 6000 msRTX 5090 msSurroundOcc IoU / mIoU
C2F + Standard Deformable Attn.144.11156.6435.14 / 23.73
C2F + PA-DA (Proposed)148.03 (+3.92, +2.7%)161.11 (+4.47, +2.9%)37.24 (+2.10) / 25.36 (+1.63)

(3) Adding Latency, Parameters & Peak Memory for Both Backbones in Table 6. We profiled our DINOv3-L pipeline (8×6×256×704, LoRA r=8) on both GPUs and will expand Table 6 with parameter, component-sum latency, and peak inference memory columns for the two full PA-DA models:

Backbone + Attn. (8f, 256×704)Params (Tot / Train)PRO 6000 ms (Enc+Head)PRO MemRTX 5090 ms (Enc+Head)5090 MemIoU / mIoU
DINOv3-L + PA-DA344.5M / 41.4M259.79 (122.47+137.32)12.14 GiB299.26 (148.25+151.01)13.91 GiB34.56 / 22.31
VGGT-Omega + PA-DA955.7M / 47.8M581.08 (433.05+148.03)13.76 GiB824.75 (663.64+161.11)14.68 GiB37.24 / 25.36

Memory values are reported in GiB, their original measurement unit; the GB label in Table 8 will be corrected.

Expanded Table 6 weighs VGGT-Omega's +2.68 IoU / +3.05 mIoU gain directly against its 2.24×–2.76× total runtime (with the same 48-image input), while showing close trainable params (41.4M vs. 47.8M), bounded memory (~12.1–14.7 GiB), and consistent PA-DA gains on both backbones (+1.24 and +1.63 mIoU).

AuthorsOpenReview ↗

Part 2/2 — Reply to W2 (Method Naming & Related Works) and Reply to W3 (Observation Gate Analysis)

[Part 2/2 — Reply to W2 (Method Naming & Related Works) and Reply to W3 (Observation Gate Analysis)]

Reply to W2: Method Naming & Discussion of VGGT-Based Occupancy Works [1–3]

Thank you for pointing out [1] VG3S (arXiv'26), [2] VG3T (ICRA'26), and [3] VGOcc (arXiv'26), and for the suggestion on naming:

  1. Positioning vs. [1–3] & backbone-agnostic design: We will cite and discuss [1–3] in Section 2. Whereas [1–3] pair visual-geometry foundation encoders with 3D Gaussian splatting/Gaussian transformer representations, our work studies how to carry analytical camera projection geometry consistently through coarse-to-fine voxel lifting—uniting 3D offset projection (Stage 1), σmin⁡(Js,n)\sigma_{\min}(J_{s,n}) weighting (Stage 2), Js,nJ_{s,n}/time observation gating (Stage 3), and coarse-guided voxel fusion. Crucially, this design is not tied to VGGT: PA-DA lifts DINOv3-L by +1.24 mIoU and VGGT-Omega by +1.63 mIoU under the same C2F decoder (Table 6), and in a controlled Occ3D experiment sharing our exact VGGT-Omega (LoRA r=8) + DPT pipeline, our downstream head beats adapted COTR (retaining COTR's LiDAR depth & group losses) by +4.20 pp (44.47% vs. 40.27% mIoU).
  2. Method naming: We appreciate this suggestion and retain VGGT-Occ for continuity with the submitted manuscript and reported results. We will make PA-DA and coarse-guided fusion more prominent in the abstract and contribution statements.

Reply to W3: Quantitative Analysis of Eq. (6) Gates γs,n\gamma_{s,n} Across Time Offsets & Dynamic/Static Regions

Your reading of Table 5 (w/o Time: 24.68%, -0.68 pp vs. w/o Jacobian: 25.02%, -0.34 pp) pinpoints the key mechanism. Because Eq. (1) uses rigid camera motion only, past keyframes (t<0t < 0) aid static parallax yet project moving objects onto stale background features. Two quantitative analyses on SurroundOcc (8f) clarify what Eq. (6)'s Time conditioning learns:

(1) Dynamic vs. Static Class Breakdown (Full vs. Independently Trained no-Time). Comparing Full (25.36% mIoU) against an independently trained no-Time model (24.68% mIoU, omitting τs,n\tau_{s,n} in Eq. 4 while keeping vs,nv_{s,n} and Js,nJ_{s,n}) across the 8 dynamic and 8 static classes:

SurroundOcc 8f SubsetFull (w/ Time)no-Time (Indep. Run)Gain (Full − no-Time)
Dynamic 8 Classes (7/8 improve*)22.36%21.11%+1.25 pp (~92% of class-avg. gain)
Static 8 Classes (drive. surf., sidewalk, etc.)28.36%28.25%+0.11 pp
All 16 Classes mIoU / Geometric IoU25.36% / 37.24%24.68% / 36.71%+0.68 pp / +0.53 pp

*Dynamic deltas (pp): truck +2.01, bus +1.84, bicycle +1.58, motorcycle +1.52, pedestrian +1.31, car +1.17, trailer +0.76 (const. veh. -0.19).

Static mIoU changes by +0.11 pp, whereas dynamic-class mIoU rises by +1.25 pp, explaining why Time conditioning carries the larger share of Stage 3's gain in Table 5.

(2) Distribution of Normalized Eq. (6) Weights Across Regions, Speeds & Time Lags (Δt\Delta t). Across all 6,019 val samples, we audited channel-averaged normalized weights αs,n=γs,n/∑(s′,n′)∈V(q)γs′,n′\alpha_{s,n} = \gamma_{s,n} / \sum_{(s',n') \in \mathcal{V}(q)} \gamma_{s',n'} at the final PA-DA gate (S1):

  • Dynamic vs. static regions: Dynamic queries assign 32.92% weight to t=0t=0 vs. 28.64% on static queries (uniform valid-observation shares: 16.44% vs. 16.69%).
  • Distribution across all 8 keyframe lags (s0 at 0 s0\text{ s} to s7 at −3.5 s-3.5\text{ s}) vs. GT car speed (matched inside 3D GT boxes):
GT car Speedt=0t=0 (s0, 0s)s1 (-0.5s)s2 (-1.0s)s3 (-1.5s)s4–s7 (-2..-3.5s)Sum s2–s7Uniform t=0t=0
< 0.2 m/s33.41%16.80%11.13%8.82%29.84%49.79%15.73%
0.2–0.5 m/s35.09%17.32%11.04%8.53%28.02%47.59%16.81%
0.5–2 m/s41.46%17.36%9.86%7.18%24.15%41.19%16.48%
2–5 m/s42.84%16.56%9.39%6.98%24.23%40.60%15.71%
5–10 m/s45.71%17.16%9.72%6.93%20.49%37.14%17.13%
10–20 m/s46.44%16.17%9.27%6.75%21.38%37.40%15.71%

The five temporal-weight columns sum to approximately 100%; rounding can produce a 0.01% excess.

Older features receive lower average per-frame weights in every speed bin: s0>s1>s2>s3>mean(s4 ⁣: ⁣s7)s0>s1>s2>s3>\mathrm{mean}(s4\!:\!s7); the last term averages four historical frames.

While the uniform reference shows no corresponding monotonic trend (15.7%–17.1%), normalized observation weights raise t=0t=0 share by +13.03 pp (object-frame-equal: 32.06% → 47.11%) and compress distant lags (s2–s7, Δt≤−1.0 s\Delta t \le -1.0\text{ s}) from 49.79% down to 37.40% (-12.39 pp), while keeping ~16–17% on s1 (-0.5s) and 53.56% total temporal context—demonstrating adaptive temporal observation selection that complements the dynamic-class gains, without flow supervision.

Thank you for the detailed response, which addresses most of my concerns. I acknowledge the contributions regarding PA-DA and coarse-guided fusion. However, incorporating the VGGT backbone is both similar in practice to prior methods and incurs a substantial latency increase (nearly 3× on the RTX 5090). It remains questionable whether such a large latency overhead is of practical significance for a latency-sensitive task such as 3D occupancy prediction. Furthermore, GaussianFormer-2 was reported in its original paper to have a latency of approximately 300+ ms on the RTX 4090, whereas Table 8 in this paper reports a latency of 768.05 ms on the RTX 5090. It would be helpful to clarify what accounts for this discrepancy.

Reviewer_WYr5OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents VGGT-Occ, a vision-only 3D semantic occupancy prediction framework that combines Projection-Aware Deformable Attention (PA-DA) with a coarse-to-fine decoder. PA-DA incorporates 3D offset sampling, Jacobian-guided attention weighting, and observation-aware feature fusion. The coarse-to-fine decoder progressively refines voxel features using gated fusion. The method is evaluated on three occupancy benchmarks.

Critical strengths
  1. The paper is generally well-written and easy to follow. The motivation and overall architecture are clearly illustrated.
  2. The paper provides mathematical derivations of the Jacobian-based attention mechanism and reports a breakdown of backbone and decoder inference latency, which helps clarify the computational overhead of PA-DA.
Critical weaknesses
  1. The proposed method largely combines well-established techniques without introducing a sufficiently novel perspective or meaningful new insight. The main ideas, including 3D offset sampling, geometry-aware attention, and coarse-to-fine feature fusion, have already been extensively explored in 3D perception [1-4]. The paper does not clearly explain what new understanding it contributes beyond an incremental architectural combination.

  2. Table 2 omits several competitive published methods that outperform VGGT-Occ. For example, under camera-only input and camera-mask supervision:

VGGT-Occ (VGGT-Ω, 256×704): 44.47 mIoU

ALOcc-3D [4] (ResNet-50, 256×704): 45.5 mIoU

ALOcc-3D (InternImage-T, 256×704): 47.5 mIoU

ALOcc-3D (Swin-Base, 512×1408): 50.6 mIoU

These results demonstrate that stronger published methods exist, including several using much smaller backbones at the same image resolution. The claimed SOTA performance is therefore unconvincing.

Table 3 also overlooks several strong published monocular methods. VGGT-Occ achieves 18.13 mIoU using four frames, whereas VisHall3D [5] achieves 20.95 mIoU, DISC [6] achieves 20.55 mIoU, and VoxSAMNet [7] reports approximately 20.2 mIoU, using single-frame monocular inputs. These omissions substantially weaken the claim that VGGT-Occ outperforms existing methods across all three benchmarks.

  1. Tables 1–3 do not specify the backbone architectures used by competing methods, although strongly affects occupancy prediction performance. Without these details, it is difficult to assess the actual improvements attributable to the proposed method.

  2. Despite its efficiency-oriented coarse-to-fine design, VGGT-Occ uses approximately 956M parameters and requires 581 ms on RTX PRO 6000 or 825 ms on RTX 5090 per inference. In comparison, ALOcc-3D reports 45.5 mIoU at 6.0 FPS with ResNet-50, while its InternImage-T variant achieves 47.5 mIoU at 5.9 FPS. The results raise serious concerns about the practical efficiency of VGGT-Occ. A controlled accuracy-latency comparison is necessary.

  3. Under standard pinhole camera assumptions, the proposed Jacobian-based attention bias essentially reduces to a logarithmic depth prior. The paper does not compare this mechanism against simpler depth-aware or distance-aware attention. Furthermore, adding Jacobian attention to 3D sampling and observation gating improves mIoU by only 0.21 points in Table 4, making its distinct contribution questionable.

  4. The framework relies on the nearly one-billion-parameter VGGT-Ω backbone. Although PA-DA is evaluated with DINOv3-L, the paper does not sufficiently isolate the benefits of the complete proposed architecture from those of the powerful pretrained backbone. Comparisons with existing occupancy heads using the same backbone would be more convincing.

[1] Sparse4d: Multi-view 3d object detection with sparse spatial-temporal fusion

[2] Far3d: Expanding the horizon for surround-view 3d object detection, AAAI 2024

[3] Protoocc: Accurate, efficient 3d occupancy prediction using dual branch encoder-prototype query decoder, AAAI 2025

[4] Alocc: Adaptive lifting-based 3d semantic occupancy and cost volume-based flow predictions, ICCV 2025

[5] Vishall3d: Monocular semantic scene completion from reconstructing the visible regions to hallucinating the invisible regions, ICCV 2025

[6] Disentangling Instance and Scene Contexts for 3D Semantic Scene Completion, ICCV 2025

[7] Sparsity-Aware Voxel Attention and Foreground Modulation for 3D Semantic Scene Completion, CVPR 2026

Questions for authors

Please see weaknesses.

3 comments
AuthorsOpenReview ↗

Reply: Part 1/3 — Executive Summary, Reply to W1, and Reply to W6

[Part 1/3 — Executive Summary, Reply to W1, and Reply to W6]

We sincerely thank Reviewer WYr5 for the review. Our response has three parts:

  • Part 1 (here): Executive Summary, Reply to W1, and Reply to W6
  • Part 2: Reply to W2 and Reply to W3
  • Part 3: Reply to W4 and Reply to W5

Executive Summary

  • W1: Beyond projected 3D sampling, geometry/time-conditioned gating improves over feature-only gating by +0.91 mIoU. Dynamic classes contribute 92% of the gain from temporal conditioning, and current-frame weight increases with car speed, supporting motion-sensitive temporal weighting without flow supervision.
  • W6: Under shared Omega+DPT on Occ3D (retaining COTR's LiDAR depth & group losses), VGGT-Occ beats COTR by +4.20 pp (44.47% vs. 40.27% mIoU).
  • W2-W3: Comparisons distinguish inputs and supervision. VGGT-Occ reaches 25.36% on SurroundOcc and 18.13% on SSCBench, where 2f already exceeds S2GO-Base (4f streaming queue). We include ALOcc-3D's higher Occ3D scores and separate stereo-assisted SSC results from our monocular RGB setting.
  • W4 & W5: PA-DA adds only 2.7%/2.9% decoder latency. The new log-depth control reaches 25.32 mIoU, close to Jacobian's 25.36, consistent with our theory.

Reply to W1: Contribution Beyond Existing Components

We credit Sparse4D/Far3D [1,2] for projected 3D sampling and ProtoOcc/ALOcc [3,4] for related occupancy designs. Our novelty is the coordinated use of projection geometry across sampling, attention weighting, and observation fusion. The key insight is that geometric correspondence and observation reliability must be addressed together: valid projections can retrieve stale evidence for moving objects. PA-DA couples sampling locations with each camera-time observation's contribution.

Controlled evidence beyond sampling and generic gating. With the same backbone and C2F decoder, 3D sampling alone raises mIoU from 23.73 to 23.97, while full PA-DA reaches 25.36 (+1.63). With Stages 1 and 2 fixed, geometry/time conditioning improves over a feature-only gate by +0.91 (24.45 to 25.36). Separately, learned coarse fusion improves over fixed equal weighting by +0.53 (24.83 to 25.36; Tabs. 4, 5, 7).

Temporal conditioning: performance and gate behavior. Our 6,019-sample SurroundOcc validation analysis compares Full against no-Time, without flow supervision:

SurroundOcc 8f AblationFullno-TimeGain
Dynamic 8-cls mIoU (7/8 up)22.36%21.11%+1.25 pp
Static 8 / All-16 / Geo IoU28.36 / 25.36 / 37.24%28.25 / 24.68 / 36.71%+0.11 / +0.68 / +0.53 pp

The equally sized class groups imply that dynamic classes contribute 92% of the overall gain from temporal conditioning.

GT car speed (m/s)<0.20.2-0.50.5-22-55-1010-20
Learned current-frame share (%)33.4135.0941.4642.8445.7146.44
Uniform valid-observation share (%)15.7316.8116.4815.7117.1315.71

Current-frame weight rises monotonically with car speed, unlike the uniform reference; object-frame-equal aggregation shows the same trend (32.06% to 47.11%). Between the lowest and highest speed bins, the weight on observations at least 1 s old falls from 49.79% to 37.40%, demonstrating selective use of temporal context. These findings establish that geometry/time-conditioned fusion enables motion-sensitive observation selection without flow supervision. The controlled gains over feature-only gating and the adaptive temporal behavior substantiate PA-DA's contribution beyond its individual building blocks.


Reply to W6: Same-Backbone Controlled Comparison (COTR)

To isolate the occupancy head under a shared backbone, we compare COTR and ours using the same pretrained VGGT-Omega-1B initialization, LoRA (r=8), DPT, inputs (8x6x256x704), and training/evaluation recipe (30 epochs, effective BS 64, peak lr=1e-5, camera mask, best-val EMA). COTR retains its occupancy encoder, group decoder/losses, and LiDAR depth loss:

Occ3D Pipeline (256×704)Pretraining & SupervisionmIoU
COTR (reported, R50)BEVDet pretrain + BEVStereo depth loss43.10
COTR + Dr.Occ (reported, R50)Adds MoGe-2 prior and region experts44.10
Omega + DPT + COTRShared Omega (LoRA r=8) + LiDAR depth + COTR losses40.27
Omega + DPT + OursShared Omega (LoRA r=8) + occ. losses only (no depth)44.47

COTR / +Dr.Occ scores of 43.10/44.10% are reported in Dr.Occ, not our reproductions. Under the matched Omega+DPT setup, COTR reaches 40.27% with depth and group losses, while ours reaches 44.47% with occupancy losses only (+4.20 pp). This directly demonstrates our occupancy architecture's contribution under a shared backbone. An independent run reaches 44.43% (two-run mean: 44.45%), complementing PA-DA's gains on DINOv3-L/Omega (+1.24/+1.63, Tab. 6).

AuthorsOpenReview ↗

Reply: Part 2/3 — Reply to W2 (Benchmark Comparisons) and Reply to W3 (Backbone Annotations)

[Part 2/3 — Reply to W2 (Benchmark Comparisons) and W3 (Backbone Annotations)]

Reply to W2 (1/2): Omitted Camera-Only Methods on Occ3D

Thank you for highlighting ALOcc-3D [4] (ICCV'25). Table 2 followed Dr.Occ's baseline set; we will add the R50, Intern-T, and Swin-B variants with their temporal and supervision settings:

Occ3D-nuScenes (Camera Input, Camera-Mask Supervision)

MethodBackboneDepth cuesTemporal / H×WmIoU (%)
COTR (reported in Dr.Occ)R50BEVStereo + depth loss9f + 1 stereo ref., 256×70443.10
COTR + Dr.OccR50BEVStereo + MoGe-2 prior9f + 1 stereo ref., 256×70444.10
VGGT-OccVGGT-Omega (LoRA)No depth loss8f uncached, 256×70444.47
ALOcc-3DR50Temporal stereo + depth loss16f cache, 256×70445.5
ALOcc-3DIntern-TTemporal stereo + depth loss16f cache, 256×70447.5
ALOcc-3DSwin-BTemporal stereo + depth loss16f cache, 512×140850.6

SurroundOcc-nuScenes (Vision-Only Comparison)

MethodBackboneSupervisionTemporal / H×WIoU / mIoU (%)
S2GO-BaseR50Occ. + Stage-1 LiDAR/depth/RGB4f queue, streaming, 256×70435.46 / 22.72
SuperOcc-LR50Occ. + voxel-center loss8f, 256×70438.13 / 24.55
VGGT-OccVGGT-Omega (LoRA)Occ. losses only8f uncached, 256×70437.24 / 25.36

ALOcc's official configuration uses a 16-slot voxel cache, temporal stereo, BEVDet pretraining, and depth supervision. Our model uses an uncached 8-frame window with occupancy-only downstream supervision. On SurroundOcc, VGGT-Occ achieves 25.36% mIoU, outperforming SuperOcc-L (24.55%) and S2GO-Base (22.72%). On Occ3D, we will revise the SOTA wording and include ALOcc's higher results with these settings annotated.


Reply to W2 (2/2): Omitted SSCBench-KITTI-360 Methods & Protocol Rationale

Following S2GO, we benchmark occupancy prediction from a single-camera RGB sequence (image_00). Our 1f/2f/4f comparison tests single-frame prediction and the ability to exploit temporal observations. The goal is geometry recovery and temporal fusion from monocular RGB. Stereo provides a calibrated baseline for depth estimation; supplying stereo-derived depth adds geometric information outside this protocol.

SSCBench-KITTI-360 Comparison

Stereo informationMethodBackboneTest-time input / historyIoU (%)mIoU (%)
Not usedOccFormerEfficientNet-B7Mono RGB, 1f40.2713.81
GaussianFormer-2R50 (ImageNet)Mono RGB, 1f38.3713.90
S2GO-BaseR50Mono RGB, streaming 4f queue40.8015.05
VGGT-OccVGGT-Omega (LoRA)Mono RGB, 1f41.0514.21
VGGT-OccVGGT-Omega (LoRA)Mono RGB, 2f43.1916.43
VGGT-OccVGGT-Omega (LoRA)Mono RGB, 4f45.0618.13
VoxSAMNet* [7]Swin TransformerMono RGB, 1f47.2220.21
UsedFoundationSSCFoundationStereo ViT-LStereo RGB, 1f48.6121.78
SymphoniesR50 + MaskDINORGB + stereo depth, 1f44.1218.58
CGFormerEfficientNet-B7RGB + stereo depth, 1f48.0720.05
DISC [6]R50 + MaskDINORGB + stereo depth, 1f47.3520.55
VisHall3D [5]R50 + MaskDINORGB + stereo depth, 1f49.1220.95

* VoxSAMNet has not publicly released code or pretrained weights; its results are quoted from the paper.

The groups distinguish stereo RGB or stereo-derived depth from stereo-free inputs. Ours uses occupancy losses only; S2GO includes Stage-1 LiDAR/depth/RGB pretraining. The four RGB+depth methods use MobileStereoNet depth from the left/right pair; FoundationSSC uses both RGB views directly. Input sizes differ: ours uses 384×1408, S2GO 256×1408, and GaussianFormer-2 376×1408 (H×W). Table 3 rounded baseline scores to one decimal; here we retain the original reported precision and will update Table 3 and the corresponding margins.

SSCBench conclusion. Under stereo-free monocular RGB input with occupancy-only downstream supervision, VGGT-Occ leads the compared methods at 18.13% mIoU, exceeding S2GO-Base by 3.08 points (2f: +1.38).

VoxSAMNet reproducibility. Its repository is public, but neither code nor pretrained weights has been released as of this rebuttal, so its 20.21% mIoU cannot yet be independently reproduced from official artifacts. We include its paper-reported result and explicitly annotate its additional depth supervision and TGIF text guidance (Sec. 3.2, Eq. 11).


Reply to W3: Backbone Annotations Across Main Tables

We will add Backbone, Params, Resolution, Temporal History, and Modality/Supervision columns across Tables 1–3, with conclusions tied to each benchmark's settings.

AuthorsOpenReview ↗

Reply: Part 3/3 — Reply to W4 (Controlled Efficiency & Memory) and Reply to W5 (Jacobian Bias vs. Log-Depth)

[Part 3/3 — Reply to W4 (Controlled Efficiency & Memory) and Reply to W5 (Jacobian Bias vs. Log-Depth)]

Reply to W4: Practical Efficiency & Controlled Accuracy–Latency–Memory Comparison

We profiled ALOcc-3D [4] (official R50 repo) alongside our DINOv3-L and VGGT-Omega models on the same two GPUs (RTX PRO 6000 and RTX 5090, 256×704), reporting encoder/head latency breakdowns and peak inference memory:

System Latency, Parameters & Memory (256×704)

ModelParams (Tot / Train)Temporal ModePRO 6000 ms (Enc + Head)RTX 5090 ms (Enc + Head)Peak Mem (PRO / 5090)
ALOcc-3D [4] (R50)52.9M / 52.9M1×6 + 16f cache102.77152.4210.21 / 11.64 GiB
VGGT-Occ (DINOv3-L)344.5M / 41.4M8×6 uncached259.79 (122.47+137.32)299.26 (148.25+151.01)12.14 / 13.91 GiB
VGGT-Occ (VGGT-Omega)955.7M / 47.8M8×6 uncached581.08 (433.05+148.03)824.75 (663.64+161.11)13.76 / 14.68 GiB

Occ3D accuracy: ALOcc-R50 reaches 45.5% mIoU and VGGT-Omega 44.47%. SurroundOcc backbone ablation: DINOv3-L reaches 22.31% and VGGT-Omega 25.36% mIoU.

Memory values are reported in GiB, their original measurement unit; the GB label in Table 8 will be corrected.

Isolated PA-DA Overhead vs. Standard Deformable Head (VGGT-Omega, 640K Voxels)

Occupancy HeadPRO 6000 Head msRTX 5090 Head msSurroundOcc IoU / mIoU (%)
Standard Deformable Head144.11156.6435.14 / 23.73
Proposed C2F + PA-DA Head148.03 (+3.92 ms, +2.7%)161.11 (+4.47 ms, +2.9%)37.24 (+2.10) / 25.36 (+1.63)

Key efficiency takeaways:

  1. PA-DA delivers a favorable accuracy-overhead trade-off: With the backbone and C2F decoder fixed, PA-DA improves mIoU from 23.73% to 25.36% while adding only 3.92/4.47 ms (+2.7%/+2.9%) to decoder latency.
  2. The main computational bottleneck lies in backbone encoding: It accounts for 74.5%/80.5% of total latency. The system timings reflect different temporal processing strategies: VGGT-Occ jointly encodes an uncached 8-frame window (48 images), whereas ALOcc-3D encodes current images and reuses a streaming voxel cache. We will distinguish these modes in Table 8. Our C2F design concentrates cross-view attention at coarse resolutions, and the controlled timing comparison demonstrates that PA-DA adds little overhead while improving accuracy.

Reply to W5: Direct Comparison with Log-Depth Attention

We completed the requested log-depth comparison on SurroundOcc. Replacing the Stage 2 Jacobian bias with a log-depth bias yields 25.32 mIoU / 37.30 IoU, compared with 25.36 / 37.24 for Jacobian weighting:

Only the Stage 2 bias was replaced. Both variants use identical shared-parameter initialization, learned-scale parameterization, numerical stabilization, backbone, decoder, inputs, training schedule, and evaluation/checkpoint selection; Stages 1 and 3 remain unchanged.

Stage 2 bias (Stages 1+3 retained)mIoU (%)IoU (%)mIoU gain
None (Table 4)25.1536.89—
Log-depth (new)25.3237.30+0.17
Jacobian (ours)25.3637.24+0.21

The closely matched results support the projection-scale interpretation of Stage 2. As derived in Appendix B.2, under zero-skew pinhole projection, an identity image transform, and isotropic normalized focal length fˉ\bar f, σmin⁡(J)=fˉ/Z\sigma_{\min}(J)=\bar f/Z. Thus slog⁡(σmin⁡(J)+ϵ)s\log(\sigma_{\min}(J)+\epsilon) takes the form of a stabilized log-depth prior. The new control is consistent with this theoretical connection: both biases improve over no Stage 2 bias, with only 0.04 mIoU and 0.06 IoU separating them in these runs.

We retain the Jacobian as a unified geometric descriptor across point weighting and observation fusion. The analytical Jacobian incorporates the actual normalized camera projection and image transform directly. Stage 2 uses its minimum singular value at each shifted sample; Stage 3 uses the full 2×32\times3 Jacobian at the reference point, where geometry conditioning contributes +0.34 mIoU (25.02 to 25.36, Table 5).

These results clarify the contribution: projection-aware weighting provides a useful geometric prior, and the Jacobian gives it a consistent formulation shared with observation fusion. The direct comparison supports comparable performance of the two Stage 2 formulations in this setting; the reported differences are single-run observations.

We hope these controlled experiments, behavioral analyses, and benchmark/efficiency comparisons fully address your concerns, and we would be grateful if you would consider them in your final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.