acceptodds
Withdrawn from ICLR 2027

Accurate 3D Object Detection in Super Sparse Space via Operator-level Sparse Attention

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In autonomous driving perception systems, the extreme sparsity of LiDAR point clouds poses a fundamental challenge to achieving high-accuracy 3D object detection under constrained computational budgets. Prevailing paradigms adopt a "sparse to dense" pipeline, where the dense projection introduces significant computing and memory bottlenecks. Emerging fully sparse architectures eliminate this dense projection but suffer from severe inefficiencies in both computation and memory, stemming from indiscriminate processing of all voxels and the incompatibility of standard self-attention with irregular sparse sequences. Furthermore, conventional detection heads rely on fixed-range max pooling for peak localization, which fails under the fully sparse paradigm and degrades detection performance. To address these limitations, we propose an accurate fully sparse 3D detector. Specifically, a Sparse Transformer module based on Sparse Matrix Multiplication is designed to perform attention directly over non-empty voxels and their local window neighborhoods, completely eliminating redundant computation and memory overhead at empty locations. Furthermore, a Progressive Pruning Network is introduced to hierarchically prune low-confidence background voxels during feature extraction, progressively transforming the sparse voxel set into super sparse space that retains only foreground-enriched tokens, thereby enabling stepwise computational reduction as the network deepens. To tackle the peak localization issue, we present a Dynamic Transfusion Head that replaces fixed-range max pooling with learnable bounding-box-aware dynamic pooling. Evaluated on the nuScenes dataset, our method achieves new state-of-the-art accuracy and delivers an improved accuracy-efficiency trade-off, while requiring only half the training epochs of classic methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Object Detection, Fully Sparse Architecture, Sparse Transformer, Voxel Pruning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_iBEyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DeS3, which is a fully sparse LiDAR 3D object detector. Contributions include a padding-free window attention implemented via sparse matrix kernels, a progressive pruning network that discards background voxels stage by stage, and a dynamic TransFusion head that picks object peaks using each voxel’s predicted box as its pooling window. Experiments have been conducted on the nuScenes dataset.

Critical strengths
  • The paper is well motivated, and its ablations support each component
  • It achieves strong practical efficiency, with the lowest peak memory among compared methods and an attention operator more than 2x faster than SST and DSVT, while remaining competitive in accuracy
Critical weaknesses
  • The comparison omits LION (NeurIPS 2024), which is a relevant, window-based sparse backbone evaluated on the same nuScenes benchmark.
  • The SpMM attention follows the same window-attention pattern as SST, with the main difference lying in its implementation
  • Evaluation is limited. Specifically, only nuScenes has been used for evaluation. Presenting results on Argoverse 2 or Waymo for long-range scenarios is important to motivate a fully sparse design.
  • The overall pipeline closely follows FSDv2 (sparse backbone, foreground-based voxel removal, virtual voxels, and a fully sparse head), and the virtual voxel module is adopted directly from that work. The differences from FSDv2 need to be clarified more explicitly.
Questions for authors
  • The overall pipeline closely follows FSDv2 (sparse backbone, foreground-based voxel removal, virtual voxels, and a fully sparse head), and the virtual voxel module is adopted directly from it. Please clarify the differences from FSDv2 more explicitly.
  • It is not clear why a fully sparse design is needed on a shorter-range benchmark.
Reviewer_sff6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DeS3, a fully sparse LiDAR 3D detector with three main parts:

  1. A window-based sparse attention ("SPMM") that only computes attention between non-empty voxels in the same window, using a few index tables instead of zero-padding.
  2. A progressive pruning network that drops low-confidence background voxels after each backbone stage. The pruning classifiers are trained with a re-weighted focal loss, where the weights depend on the stage and on the next stage's prediction.
  3. A "Dynamic Transfusion Head": a voxel becomes a query only if it has the highest score inside its own predicted box, and the queries are then refined with KNN cross-attention.

The backbone blocks run attention on the 2D pillar plane but keep the FFN in 3D, and the virtual-voxel module follows FSD v2.

The method reports 67.9 mAP / 72.4 NDS on nuScenes val and 69.3 mAP / 73.8 NDS on test (Tables 1 and 6), which is competitive. My main concerns are that the paper describes prior work as missing things it already has, that the ablations don't isolate the parts that are actually new, and that the text describes the efficiency results more favorably than the paper's own Figure 7 supports.

Critical strengths
  1. Cutting compute and memory in LiDAR detection without hurting accuracy matters in practice, and the overall design (prune background early, attend only over real voxels, pick queries locally) hangs together well.
  2. Against SAFDNet, the best sparse-head baseline in the tables, DeS3 improves by 1.6 mAP / 1.4 NDS on val and 1.0 mAP / 1.5 NDS on test (Tables 1 and 6).
  3. Turning off the foreground-preservation term (P2 vs. P1 in Table 3) drops mAP from 67.9 to 65.3 and raises foreground-voxel dropout from 0.12% to 1.21%, so being conservative about pruning foreground clearly matters at this threshold. The Mix block also uses about 14% fewer MACs than the pure 3D variant (389.5G vs. 454.0G) with no loss in accuracy (Table 2).
  4. SPMM is faster than the SST and DSVT implementations at 1, 5 and 10 sweeps (Figure 8), and I appreciate that the timing includes the grouping step.
Critical weaknesses

1. The paper overstates what prior work is missing.

A lot of the motivation in Secs. 1–2 rests on claims about prior work that don't hold up.

  • Padding-free sparse window attention already exists. Secs. 1, 2.1 and 3.2 describe existing sparse transformers as relying on zero-padding, and Sec. 2.1 concludes this leaves "operator-level purely sparse attention open". But Stratified Transformer [R1] (Sec. 4 and Fig. 7) already does essentially what Secs. 3.2.1–3.2.2 describe: it precomputes all valid query–key index pairs, computes the dot products by index, applies a per-query (scatter) softmax, and sums the indexed values back per query. Each step is a single CUDA kernel, and there's no dummy-token padding. The main difference I can see is the table layout: DeS3 keeps per-query offsets (Item Count, which is basically a CSR row pointer) plus a group map, instead of plain index pairs and scatter ops. That could still be a useful engineering contribution, but then the paper should say that's the contribution and show that it matters.
  • In-backbone pruning and sparse peak selection already exist too. The paper says existing fully sparse detectors process all voxels without screening (abstract, Secs. 1 and 3.3), and that they lack local peak selection and rely on global top-k plus NMS (Secs. 2.3 and 3.4; Sec. 4.4 describes the VoxelNeXt head the same way). But VoxelNeXt [R2], which is in the paper's own comparison, already prunes voxels inside the backbone based on feature magnitude (Sec. 3.1, Fig. 6). It also replaces NMS with a sparse max pooling that only runs on non-empty voxels (Sec. 3.2, Fig. 7), and reports that this works on par with NMS (its Table 10). So calling fixed-window max pooling "inapplicable" on sparse voxels (Sec. 3.4) is too strong. DeS3 is still different (a supervised pruning rule instead of a magnitude rule, and predicted-box windows instead of fixed pooling kernels), but those narrower differences are what the paper should claim and test.
  • End-to-end learned pruning exists too. Sec. 2.2 says GSP's "decoupled design limits joint optimization with the backbone". But GSP [R3] learns its pruning end-to-end from the task loss plus a sparsity regularizer; "decoupled" in that paper just means it's a plug-and-play layer. So this isn't really a limitation of GSP.

None of this makes DeS3 the same as any prior method. But it narrows the new part down to three things: the specific kernel/indexing design, the cross-stage loss weighting, and the predicted-box query selection. The paper needs direct evidence for each of them, which brings me to the next point.

2. The ablations don't isolate what's new.

  • Pruning. Tables 3 and 4 vary the loss terms and the number of stages, but there's no "same backbone, no pruning" row, and none of these ablations report latency, MACs or memory, so we can't see how much compute the pruning actually saves. There's also no SFP-only row (CA and ESP both off), which is basically a plain foreground-weighted focal loss and the natural simple baseline. Since all variants use the same threshold (R=0.2R = 0.2), turning off SFP (P2, β=1\beta = 1) moves background removal from 83.10% to 95.71%, so P1 vs. P2 compares two very different operating points, not just two loss designs. And the two terms that are specific to this paper have small effects: removing CA costs 0.5 mAP / 0.3 NDS, and removing ESP costs 0.2 mAP with no change in NDS. That could be within run-to-run noise on nuScenes val. To show the proposed weighting really helps, I'd want to see it beat a simpler pruning loss with a tuned threshold on an accuracy-vs-cost curve, ideally over a few seeds.
  • Head. The VoxelNeXt head is single-stage (Sec. 4.4 says so), while the proposed head adds a KNN cross-attention refinement stage. So the +0.7 mAP / +0.2 NDS in Table 5 could come from the extra refinement as much as from the box-based pooling. And the transplanted TransFusion head not converging only shows that this particular port didn't work; it doesn't show that dynamic pooling is the fix.
  • Operator. Figure 8 only compares against SST and DSVT, which both work on padded or fixed-size groups. Since the claimed contribution is a padding-free kernel, the baseline that matters is an existing padding-free implementation (e.g., Stratified Transformer's) on the same inputs.

3. The efficiency claims don't match the measurements.

  • Sec. 4.3 says DeS3 has "the lowest peak memory" (469.2 MB), but Figure 7 puts VoxelNeXt at about 250 MB. As written, that sentence isn't correct.
  • The same figure shows DeS3 is the slowest of the five detectors: 173.04 ms vs. roughly 75–105 ms for the others. That's about 1.6×1.6\times the latency of the next slowest one and more than 2×2\times that of VoxelNeXt, so "slightly higher latency" undersells it. To be fair, the conclusion does admit "a certain reduction in inference speed", but the abstract ("an improved accuracy-efficiency trade-off") and Sec. 4.3 paint a rosier picture than Figure 7 does. What the data show is better accuracy and lower memory than most baselines (not VoxelNeXt), at a clearly higher latency. That's a legitimate trade-off, but it should be presented as one.
  • Two more claims have no measurements behind them. The argument that compute scales with the number of valid points rather than the perception range (Sec. 5, App. B) has no end-to-end experiment; Figure 8 is operator-level and varies the number of sweeps, not the range. And "half the training epochs" (abstract, Sec. 4.2) doesn't by itself mean lower training cost, since each iteration may well be slower (inference already is). GPU-hours, or iteration counts plus per-iteration time, would settle this.

Minor. A few references list the wrong venue (all given as CVPR): [19] GSP is ICRA 2025, [32] AFDetV2 is AAAI 2022, and [36] PillarNet is ECCV 2022.

References

[R1] X. Lai et al. Stratified Transformer for 3D Point Cloud Segmentation. CVPR 2022. [R2] Y. Chen, J. Liu, X. Zhang, X. Qi, J. Jia. VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking. CVPR 2023. [R3] J. Li et al. Efficient 3D Perception on Multi-Sweep Point Cloud with Gumbel Spatial Pruning. ICRA 2025.

Questions for authors
  1. Sparse attention. What exactly is new compared with Stratified Transformer's memory-efficient attention, or other existing padding-free sparse attention code? Could you add a head-to-head operator comparison on the same inputs as Figure 8, with the same connectivity, feature dims, precision and timing boundaries?
  2. Pruning. Could you add (a) the same backbone without pruning and (b) an SFP-only (plain foreground-weighted focal loss) variant at a similar keep ratio, and report accuracy, latency and memory for each? Also, in Eq. (2) of Appendix A, wi(k)=(1−pi(k+1))δkw_i^{(k)} = (1 - p_i^{(k+1)})^{\delta_k}: is pi(k+1)p_i^{(k+1)} detached? If not, the stage-kk loss can be lowered by pushing pi(k+1)p_i^{(k+1)} up, i.e., by making the deeper stage more confident that a background voxel is foreground, which is the opposite of what you want.
  3. Head. Could you fix the query budget and the cross-attention refinement, and only swap the query selection: predicted-box pooling vs. fixed-kernel sparse max pooling (as in VoxelNeXt) vs. top-k with some local suppression? Proposal recall and the number of duplicate queries per object would show directly whether the box-based pooling is what helps.
  4. Resources and object-level preservation. Please fix the "lowest peak memory" statement and say exactly what the latency and memory numbers include. Also, PSDR is pooled over all foreground voxels, so 0.12% could still be concentrated on a few small or distant objects. Using the val pruning masks you already have, could you report how often all voxels of a GT object get pruned, broken down by distance or number of points?
Reviewer_LLQgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a fully sparse 3D object detection framework named DeS3 for autonomous driving LiDAR point clouds, aiming to address the limitations of existing fully sparse detectors in attention computation, background voxel redundancy, and target center positioning. Specifically, the authors propose SpTransformer based on sparse matrix multiplication (SPMM) to avoid the Padding overhead in traditional window attention; design Progressive Pruning Network to reduce redundant computations through multi-stage foreground prediction and voxel pruning; further propose Dynamic Transfusion Head, which uses prediction boxes to guide dynamic pooling to improve candidate target selection in the sparse space. On the nuScenes dataset, this method achieves competitive detection accuracy and low memory usage, but the end-to-end inference efficiency still has significant deficiencies.

Critical strengths
  1. The SPMM-based SpTransformer avoids the padding overhead in traditional window attention and demonstrates significant speed advantages in operator-level experiments.
  2. By implementing staged foreground supervision and adaptive loss weighting, a high background voxel removal rate is achieved while maintaining a low foreground deletion rate.
  3. The Dynamic Transfusion Head uses prediction boxes to guide dynamic peak selection, alleviating the limitations of traditional fixed-window pooling in cases of sparse voxel distribution.
Critical weaknesses
  1. The inference delay of DeS3 is as high as 173.04 ms, which is the slowest method shown in Figure 7; meanwhile, the memory usage of VoxelNeXt in Figure 7 is significantly lower than that of DeS3, but the main text claims that DeS3 has the lowest memory usage.
  2. This design mainly combines the existing local window attention, Shifted Window and sparse matrix operations, but it has not fully explained its core improvements compared to the existing sparse attention operators.
  3. The Item Map needs to construct all token pairs within the window, and its computational and storage costs depend on the density of the window occupancy; moreover, the operator efficiency experiments only use the first 20 frames of the nuScenes validation set, lacking sufficient large-scale verification.
  4. The existing experiments mainly compare different pruning losses, pruning stages and detection heads, lacking a no-pruning baseline, standard Backbone replacement and combination ablation between the three core modules, making it difficult to accurately attribute the overall performance improvement.
  5. Although the overall foreground voxel deletion rate is only 0.12%, there is a lack of analysis of distance, occlusion degree, number of target points and category dimensions, making it impossible to determine whether pruning will systematically damage the recall rate of difficult targets.
Questions for authors
  1. Figure 7 shows that the memory usage of VoxelNeXt is lower than that of DeS3, but the main text claims that DeS3 has the lowest memory usage. Can the author explain this contradiction? Additionally, since the SPMM operator has higher efficiency at the level, why does the end-to-end inference latency of DeS3 end up being the highest? Can a breakdown of the time consumption for each module be provided?
  2. Compared with existing SDDMM/SpMM-based sparse attention or standard sparse graph attention, what are the essential differences in the algorithm design of the proposed attention mechanism in this paper? Can a direct comparison be provided under the same Backbone and running environment to clearly identify the actual source of method innovation?
  3. Since the Item Map needs to store the Query-Key connections within the window, can the author provide a complete time and space complexity analysis including the index construction, attention calculation, and normalization process? Additionally, can the actual performance under different window sizes, voxel densities, and sweep quantities be tested on the complete validation set?
  4. Can the author add complete module-level ablation experiments, removing SpTransformer, Progressive Pruning Network, and Dynamic Transfusion Head respectively, and test the combinations of different modules? At the same time, report mAP, NDS, latency, and memory usage to quantify the independent contribution and additional overhead of each module.
  5. Currently, PSDR only reflects the overall proportion of deleted foreground voxels, but the removal of a few key voxels may also cause the target to be missed. Can the author further report object-level recall under different distances, target point numbers, occlusion degrees, and categories? Additionally, how sensitive is the pruning performance to the threshold R=0.2R=0.2?
Reviewer_cD7aOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents DeS3, a fully sparse LiDAR detector designed to reduce the cost of processing mostly background voxels while maintaining accurate proposal generation. It combines window-level SPMM attention over non-empty voxels, stage-wise foreground pruning, and a bounding-box-aware dynamic pooling head. On the nuScenes validation set, it achieves 72.4 NDS and 67.9 mAP. The SPMM benchmark shows a 2× speedup over SST and DSVT across 1, 5, and 10 sweeps.

Critical strengths
  1. The paper addresses several complementary bottlenecks in fully sparse detection rather than merely replacing a single backbone block, including the overhead of padded attention, the cost of processing background tokens, and duplicate or unstable candidate selection. Its integrated design is technically coherent.

2.The proposed DeS3 achieves promising detection performance on the nuScenes dataset.

Critical weaknesses

1.The proposed method DeS3 should be compared with several recent detection methods, such as ISFusion and SparseLIF.

2.The progressive pruning network in DeS3 bears some resemblance to the keypoint selection and sampling strategy in IA-SSD. The authors should clarify how their method differs from IA-SSD to better highlight its novelty. In addition, the design of the Virtual Box Generation module appears similar to the point-voting head in 3DSSD. The authors should clarify its motivation and explain how it differs from 3DSSD.

3.The progressive pruning network improves efficiency by progressively aggregating features and downsampling the point cloud. However, repeated downsampling may discard important points, particularly for sparsely observed objects with few points, potentially degrading detection performance. The authors are encouraged to analyze these cases.

4.SPTransformer further reduces the detector’s GPU memory consumption by constructing dynamic windows. However, variable-length token sequences need to be aligned for parallel computation. Does this impose constraints on window size? Larger objects may require larger windows; how does the method handle this variation in window size?

Questions for authors

1.How does the proposed DeS3 differ from IA-SSD and 3DSSD?

2.It is recommended that the authors include comparisons with more recent methods.

3.In Table 1, the proposed DeS3 performs worse than the DSVT baseline on Traffic Cone. Please analyze the reason for this decline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.