acceptodds
Withdrawn from ICLR 2027

RayLift: Beyond Rank-One Lifting for Occlusion-Aware 3D Occupancy Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Camera-based 3D semantic occupancy prediction must reconstruct not only visible surfaces, but also geometry and semantics hidden behind them. Consider a pixel ray that first intersects a visible car and then an occluded bus (Fig. 1). Although the image feature at that pixel is dominated by the car, the resulting 3D representation should encode different semantics at the two depths. Standard Lift-Splat-Shoot (LSS) does not provide this distinction during lifting: it copies the same pixel feature to every depth bin and changes only its scalar weight. Consequently, the per-ray lifted representation has rank at most one. When the same depth-independent linear classifier is applied at every depth, all positive-mass depths preserve the same feature-dependent class ordering, so any class-order reversal must arise from subsequent 3D reasoning. We formalize this representational restriction as a rank-collapse obstruction. We introduce **RayLift**, which directly relaxes this restriction and complements it with directed contextual reasoning for occluded regions. Optimal-transport-routed multi-feature lifting assigns each depth bin a depth-dependent mixture of feature heads, enabling different, non-collinear feature vectors along the same ray. Causal ray-radial convolution provides a directed near-to-far pathway for propagating contextual information toward farther, occluded voxels. Visibility-conditioned temporal fusion extends this reasoning across time by conditioning historical voxel features on their predicted visibility. On Occ3D-nuScenes, RayLift achieves and , surpassing ALOcc by points in overall and points in . On Occ3D-Waymo, RayLift achieves , outperforming ALOcc by points under matched camera-only settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D semantic occupancy prediction, 3D perception, autonomous driving

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_X5SXOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper revisits LSS-based view transformation. The authors observe that standard lifting assigns the same image feature to different depth bins along a camera ray, which results in a rank-one representation. Based on this observation, they propose RayLift. The main component is Multi-Feature Lifting (MFL). MFL generates multiple feature heads from image feature and uses optimal transport to route different mixtures of these heads to different depth bins. The paper further introduces Causal Ray-Radial Convolution (CRRC) for near-to-far contextual propagation in ego-centric polar space and visibility-conditioned temporal fusion for incorporating historical features. Experiments on Occ3D-nuScenes, SurroundOcc, and Occ3D-Waymo demonstrate the state-of-the art performance of RayLift.

Critical strengths

The observation that conventional LSS-based lifting produces highly constrained per-ray representations is intuitive and well motivated. This paper also makes an effort to connect this observation to the proposed architecture.

Critical weaknesses
  1. The main improvements are not well aligned with the claimed occlusion-recovery mechanism. RayLift is motivated by recovering farther occluded semantics along multi-surface rays, but RayIoU evaluates only the first occupied surface along each ray. In Table 2, the gains are more pronounced in RayIoU than in mIoU, which does not directly support the claimed improvement on farther occluded objects.

  2. The current ablations do not isolate the necessity of the proposed optimal-transport-based MFL design. MFL introduces several changes simultaneously, including multiple learned feature heads, geometry-conditioned routing and so on. The current ablations mainly show that the complete MFL module improves performance, but do not clarify which of these ingredients is actually responsible for the gain.

  3. The analysis is performed on GT-selected multi-class rays, where different occupied depths have different semantic labels. This setting naturally favors the paper’s hypothesis that features at different depths should become less similar. However, not all depth variation corresponds to semantic variation. A single object can occupy several adjacent depth bins along the same ray, in which case maintaining similar representations across these depths may be beneficial. Since the proposed balanced routing explicitly encourages the use of multiple feature heads, it is unclear whether MFL selectively introduces diversity only when the underlying semantics change, or whether it also introduces unnecessary variation within the same object.

Questions for authors
  1. Why are the gains more pronounced in RayIoU if the main motivation is to recover farther occluded semantics?

  2. How necessary is the optimal-transport-based routing in MFL?

  3. Does MFL preserve feature consistency across depth bins belonging to the same object or semantic class?

Reviewer_WcanOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the occlusion problem in camera-based 3D semantic occupancy prediction, specifically when a single camera ray passes through multiple objects of different classes (e.g., a visible foreground car and an occluded bus behind it). The authors point out that standard Lift-Splat-Shoot (LSS) copies the same 2D pixel feature along the ray and only scales it by a scalar depth weight, which makes the per-ray lifted feature matrix rank-one (rank(Lu,v,:)≤1\text{rank}(\mathbf{L}_{u,v,:}) \le 1). In Sec. 3, they show that under a shared linear classifier without bias, scalar lifting cannot change the relative class ordering along a ray on its own, so reversing class predictions at occluded depths has to rely heavily on cross-ray context in the 3D backbone. To fix this, the authors build RayLift on top of ALOcc with three changes:Multi-Feature Lifting (MFL), which generates GG feature heads per pixel and uses 5-step Sinkhorn optimal transport (conditioned on detached geometry features) plus an auxiliary frustum segmentation loss so different depth bins along the same ray get different head mixtures.Causal Ray-Radial Convolution (CRRC), which resamples each height slice of the voxel grid into polar coordinates, runs a 1D causal dilated conv from near to far along the radius, and resamples back to Cartesian coordinates.Visibility-Conditioned Temporal Fusion, which splats ray transmittance into the voxel grid and concatenates it with historical voxel features before temporal warping. outperforming ALOcc by 6.5points under matched camera-only settings.

Critical strengths

1.The diagnosis of LSS lifting is simple, intuitive, and backed up by direct feature analysis. Instead of just adding another heavy 3D encoder, looking at the rank-1 limit of scalar lifting is a very natural way to study multi-object occlusion along a ray. What really sold me on this part is the check in Sec. 5.3 / Appendix E.4 (Table 8): measuring the singular values and inter-depth cosine similarity on 22,074 multi-class rays right before voxel pooling shows the effective rank goes from 1.00 to 3.52 and cross-class cosine similarity drops from 1.00 to 0.68. This convinces me that MFL is actually learning depth-specific features along the ray rather than just acting as extra capacity

2.Consistent improvements on strict ray-based metrics (RayIoU1m\text{RayIoU}_{1\text{m}}) with clear ray-profile visualizations. On Occ3D-nuScenes (Table 1), the method improves over ALOcc by +1.7 RayIoU overall and +1.8 on RayIoU1m\text{RayIoU}_{1\text{m}} (39.6 vs. 37.8), while keeping inference speed close to the baseline (5.2 vs. 6.0 FPS). The ray profiles in Figs. 3 and 4 show concrete examples where ALOcc repeats the front car label onto a truck or motorcycle behind it, while RayLift separates the two surfaces cleanly.

Critical weaknesses

1.The +6.5 mIoU gain on Occ3D-Waymo does not match the setup in the appendix or the occlusion motivation. The abstract and intro highlight a +6.5 mIoU jump over ALOcc on Occ3D-Waymo (36.55 vs. 30.03 in Table 7). However, checking Table 4 in Appendix D.1 and Appendix E.3 raises a few red flags. First, Table 4 shows that visibility-conditioned temporal fusion is turned off on Waymo (even though 16-frame fusion is used), MFL heads are reduced from 8 to 4, and λimg=0\lambda_{\text{img}} = 0, none of which is explained in the main paper. 2.Occ3D-Waymo evaluates only inside the official camera-visibility mask (mask_camera), which only keeps the first visible surface voxel along each ray and ignores occluded voxels behind it. Under the exact same camera-visible mask protocol on Occ3D-nuScenes (Table 5), RayLift only gains +0.7 mIoU over ALOcc. Why would a method built for occluded second surfaces jump by +6.5 mIoU on a benchmark that masks out occluded voxels, especially on flat unoccluded classes like Road (+5.9) and Walkable (+10.8), while ALOcc has a weirdly low Bicycle score (12.22 vs. 21.37 for CVT-Occ)? This makes me suspect the ALOcc baseline in Table 7 was not trained under the exact same 16-frame, 960×640960\times 640, EMA, and mask/dice loss recipe. Because the Waymo result is a major claim of the paper, I need to see an internal baseline on Waymo (RayLift with MFL and CRRC turned off under the exact same training script) to know how much of the +6.5 mIoU comes from the proposed modules. 3.Per-ray balanced OT has a flaw on single-surface rays, and key ablations against simpler alternatives are missing. Comparing CRRC against a non-causal polar conv or a standard 3D dilated conv with equal parameters, and reporting the clean baseline in Table 2 with all four components turned off at once, would make the ablation study much more convincing.

Questions for authors

1.Occ3D-Waymo setup and matched internal baseline (Table 4 & Table 7): Why is visibility-conditioned temporal fusion disabled and MFL reduced to G=4 (with lambda_img=0) on Occ3D-Waymo, even though 16-frame temporal fusion is used? More importantly, was the ALOcc baseline in Table 7 (30.03 mIoU) trained under the exact same 16-frame input, 960x640 resolution, EMA, and class-weighted mask/dice loss recipe as RayLift? What mIoU do you get on Occ3D-Waymo if you take your exact RayLift codebase and turn off MFL, CRRC, and frustum loss? Since Waymo evaluates only within the camera-visibility mask (first visible surface), seeing this matched internal baseline will directly affect my final rating.

2.Sinkhorn OT vs. plain Softmax routing (Sec. 4.1): On single-surface rays where the depth distribution v_{u,v} puts nearly all mass on one depth bin d*, doesn't the per-ray uniform head marginal (Eq. 4.3) force p(g|d*) toward 1/G across all heads? Have you compared 5-step Sinkhorn OT against a simple depth-wise Softmax router p(g|d) = softmax_g(A_{u,v,g,d} / tau) on Occ3D-nuScenes? Also, how sensitive is RayIoU to the number of heads G (e.g., G = 2, 4, 8)?

3.Clean baseline in Table 2 and CRRC alternatives: The caption of Table 2 notes that all rows keep visibility fusion and default MFL frustum supervision. How is MFL frustum supervision applied in Rows 1 and 3 where MFL is disabled, and what is the performance when all four proposed components (MFL, CRRC, frustum loss, and visibility fusion) are turned off together? Also, how does CRRC compare against a non-causal polar conv or a standard 3D dilated conv with the same parameter count?

If my questions are addressed and the ablation study is completed, I would be happy to increase my score.

Reviewer_MU41OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies a limitation of standard Lift-Splat-Shoot lifting (LSS) for camera-based 3D semantic occupancy prediction. Its core observation is that LSS copies one image feature to different depth bins along a ray with scalar weights, therefore produces a feature matrix of rank at most one. The authors proove that under a shared, depth-independent linear classifier, the scalar lifting preserves class ordering, and to seperate classes between two depths of a ray, surrounding context is neccessary. They then propose RayLift, an occlusion-aware framework for lift-splat occupancy prediction, which uses optimal-transport-routed multi-feature lifting (MFL) to construct depth-dependent features along each ray, causal radial convolution (CRRC) to propagate context from near to far, and visibility-conditioned temporal fusion to guide the integration of historical features using predicted visibility cues. Experiments report mIoU and RayIoU improvements over previous methods on Occ3D-nuScenes, SurroundOcc, and Occ3D-Waymo.

Critical strengths

The motivation is clear, natural, and supported by both theoretical analysis and direct experiment evidence. The proposed methods are closely related to the analysis of motivation.

Sec. 3 Proposition 1 shows that positive depth weights preserve class ordering under a shared, depth-independent linear classifier, which explains the limited semantic flexibility during lifting and directly inspires the proposed MFL to construct depth-dependent feature mixtures. Appendix E.4 and Table 8 analyze 22,074 multi-class rays before voxel pooling. After adopting RayLift, effective rank increases from 1.00 to 3.52, while cross-class inter-depth cosine similarity decreases from 1.00 to 0.68. These measurements show that the trained model uses the additional capacity to produce non-collinear features along a ray.

Critical weaknesses
  1. The experiments provide limited direct evidence of improved occluded semantic discrimination.

Appendix E.4 and Table 8 show increased feature rank and reduced cross-class similarity. Since the router receives relative-depth features (Appendix C.1), these changes could partly reflect positional variation. The analysis lacks farther-target prediction accuracy and a same-class comparison controlling for depth separation.

Appendix D.2 also constructs frustum supervision from the nearest projected occupied point in each image-feature cell. Improved near-surface prediction could therefore contribute to the reported gains. Table 3's 0.4 RayIoU decrease without this loss leaves its effect on farther targets unclear. Reporting farther-target accuracy and foreground-label repetition on the existing multi-class rays would directly test the motivating failure mode.

  1. The large Waymo improvement needs a matched internal baseline.

Table 7 reports a +6.52 mIoU gain on Waymo over ALOcc, the largest of reported gains. Appendix E.3 evaluates within camera-visibility and field-of-view masks. The contribution specifically associated with recovering farther occluded semantics remains unmeasured.

The Waymo configuration also reduces MFL to four heads, disables visibility conditioning, and sets λimg=0\lambda_{\mathrm{img}}=0, while retaining 16-frame fusion (Table 4; Appendix D.2). The rationale for these changes is unexplained, and the paper does not tell whether ALOcc shares the same training recipe. A same-pipeline baseline with MFL, CRRC, and frustum supervision disabled would clarify the source of the gain.

Questions for authors
  1. On the rays selected in Appendix E.4, how much does MFL improve farther-target accuracy and reduce foreground-label repetition? Does this improvement persist without frustum supervision?

  2. What Waymo result does the same training pipeline achieve with MFL, CRRC, and frustum supervision disabled? Please clarify which training settings match ALOcc and explain the dataset-specific configuration changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.