acceptodds
Under review as a conference paper at ICLR 2027

Training-Free Global Geometric Association for 4D LiDAR Panoptic Segmentation

Abstract

Dominant paradigms for 4D LiDAR panoptic segmentation are usually required to train deep neural networks with large superimposed point clouds or design dedicated modules for instance association. However, these approaches perform redundant point processing and consequently become computationally expensive, yet still overlook the rich geometric priors inherently provided by raw point clouds. To this end, we introduce Geo-4D, a simple yet effective training-free framework that unifies spatial and temporal reasoning, enabling holistic LiDAR perception over long time horizons. Specifically, we propose a global geometric association strategy that establishes consistent instance correspondences by estimating an optimal transformation between instance-level point sets. To mitigate instability caused by structural inconsistencies in point cloud observations, we propose a global geometry-aware soft matching mechanism that enforces spatially coherent point-wise correspondences grounded in the spatial distribution of instance point sets. Furthermore, our carefully designed pipeline, which considers three instance types—static, dynamic, and missing—offers computational efficiency and occlusion-aware matching. Our extensive experiments across both SemanticKITTI and nuScenes demonstrate that our method consistently outperforms state-of-the-art approaches, even without additional training or extra point cloud inputs. These results underscore the potential of inherent geometric priors in point clouds as an efficient solution for maintaining consistent instance identities over time.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Computer Vision, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YBfpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GEO-4D directly associates instance masks from a pretrained single-frame panoptic segmentation model without requiring an additional learned association network. It leverages centroid and covariance cues to match static instances, followed by iterative closest point (ICP) registration for the remaining candidate pairs. Its global geometry-aware soft matching module establishes correspondences through an entropy-regularized optimal transport solution. A short-term memory mechanism further retains unmatched instances. Experiments on SemanticKITTI and nuScenes demonstrate that this geometry-based association pipeline improves temporal consistency compared with learning-based and IoU-based alternatives.

Critical strengths
  1. A clear contribution is the use of point-cloud registration for instance association in training-free, single-scan 4D LiDAR panoptic segmentation. The method brings ICP into this setting and strengthens it with global geometric matching, rather than introducing another learned association network. This is a useful shift in design: it makes temporal association depend more directly on the geometric structure already present in the point clouds.
  2. The efficiency gains are substantial. On SemanticKITTI, GEO-4D uses 3.15 GB and 1.10 s per scan, compared with 10.4 GB and 3.51 s for two-scan Mask4Former and 15.5 GB and 3.67 s for two-scan 4D-StOP. It achieves a validation LSTQ of 72.7, comparable to Mask4Former's 72.0 and higher than 4D-StOP's 66.4.
Critical weaknesses

The main comparisons in Table 2 mix different numbers of input scans (single-scan vs. two-scan inputs) and differences in per-frame segmentation quality SclsS_{\mathrm{cls}}. Although the two-scan baseline methods inherently require overlapping scans to enable IoU-based tracking, and controlled comparisons using identical masks are provided in Table 3(b) and Appendix C.3, the presentation in the main text still makes it difficult to immediately distinguish the differences attributable to the backbone network from the gains introduced by the association algorithm. Furthermore, Appendix C.5 (Table A.6) shows that DBSCAN post-processing accounts for 0.76 seconds of the total runtime of 1.10 seconds (nearly 70%). However, the paper lacks a fine-grained runtime breakdown that would allow the individual computational costs of the 3D backbone, DBSCAN, static filtering, and the GGSM-ICP loop to be separately assessed.

Questions for authors
  1. To better clarify the efficiency and accuracy metrics reported in Tables 2 and 3 (b), could the authors provide a detailed runtime breakdown for the 3D backbone, DBSCAN, static filtering, and GGSM-ICP? Additionally, could the authors report the performance of GEO-4D when directly applied to multi-scan predictions in the main benchmark (e.g., 2-scan Mask4Former) ?

  2. Could the authors clarify the specific strategy and hyperparameters of the short-term memory bank BB, particularly the values of wmemw_{\mathrm{mem}} used for SemanticKITTI and nuScenes, as well as the Sinkhorn regularization parameter ϵ\epsilon and the number of iterations? Furthermore, since BB caches historical world coordinates without performing motion prediction, how does GEO-4D handle fast-moving objects that reappear at locations far from their last observed positions after being occluded for multiple frames?

3 comments
AuthorsOpenReview ↗

Response to Reviewer YBfp

Thank you for the insightful feedback. Below, we address your questions to each comment. We hope our responses have adequately addressed your questions.

[Q1] Detailed Runtime & Multi-scan Experiment

We additionally provide a stage-wise runtime breakdown of Geo-4D on an NVIDIA RTX 3090, as the RTX A6000 used for the original evaluation was not available for these additional measurements. Despite using different GPU hardware, the total runtime is largely consistent, with a modest increase from 1.10 to 1.22 s/scan.

SegmentationStaticDynamicMissingDBSCAN
Runtime (s/scan)0.2830.0060.0810.0640.771

Regarding multi-scan predictions, this comparison is already included in Table 3(b). Specifically, we first obtain predictions from 2-scan Mask4Former and split them into single-scan instance masks. We then apply Geo-4D to associate these instances across frames, using the same underlying predictions as the IoU-based baseline. Geo-4D improves LSTQ from 72.0 to 72.8 and SassocS_{\mathrm{assoc}} from 76.3 to 77.8, while both methods achieve the same SclsS_{\mathrm{cls}} of 68.0 due to their identical segmentation predictions. This demonstrates the benefit of our geometric association independently of segmentation quality.

[Q2-1] Hyperparameter Settings & Analysis

In this section, we provide implementation details and hyperparameter analysis. We use a memory bank size of wmem=3w_{\mathrm{mem}}=3, Sinkhorn regularization ϵ=0.2\epsilon=0.2, and 30 Sinkhorn iterations, with the same settings applied to both SemanticKITTI and nuScenes.

Memory Bank SizeLSTQ ↑SassocS_\mathrm{assoc} ↑Runtime (s/scan)
172.076.81.11
272.578.01.19
3 (Default)72.778.31.22
472.678.11.21
Sinkhorn IterationsLSTQ ↑SassocS_\mathrm{assoc} ↑Runtime (s/scan)
2072.678.01.07
30 (Default)72.778.31.22
5072.778.31.25
10072.678.11.45
Sinkhorn regularization (ε)LSTQ ↑SassocS_\mathrm{assoc} ↑
0.1072.778.3
0.20 (Default)72.778.3
0.3072.678.1
0.5072.577.8

Our ablation results show that a memory bank size of 3 achieves the best performance. While 30 Sinkhorn iterations provide the best trade-off between performance and runtime, with no further improvement from additional iterations. Performance also remains stable across ϵ=0.1\epsilon=0.1 and 0.20.2, indicating limited sensitivity to the regularization parameter. We have also updated the implementation details in Section 4.1 of the revised manuscript to clarify these hyperparameter settings.

[Q2-2] Handling Fast Moving Objects

We acknowledge the reviewer’s concern regarding fast-moving objects reappearing after prolonged occlusion. Such cases remain challenging because Geo-4D relies on historical instance geometry without explicit motion prediction.

However, associating objects across prolonged occlusions is a general challenge in temporal instance association. For example, multi-scan IoU-based methods can also struggle when occlusions eliminate shared instance observations. Furthermore, while motion prediction could help, partial LiDAR observations and changes in visible surfaces can lead to unreliable velocity estimates. Under identical instance masks, Kalman-based motion prediction with 3D Box IoU and Hungarian matching achieves an SassocS_\mathrm{assoc} of 61.9, still substantially below Geo-4D's 78.3, as shown in Table 3(a).

While our overall pipeline and geometry-aware instance association demonstrate strong effectiveness, incorporating robust motion prediction to better handle prolonged occlusions remains a promising direction for future work, as discussed in Appendix D.1.

Reviewer_YBfpOpenReview ↗

Response to the authors

Thank you to the authors for the detailed responses. They have addressed my questions. I also appreciate the authors acknowledging that incorporating reliable motion prediction is a promising direction to investigate for the long-term occlusion issue I raised. I will keep my score at 3 and lean toward acceptance.

AuthorsOpenReview ↗

Response to Reviewer YBfp

We sincerely thank you for your positive feedback and are glad that our rebuttal has addressed your questions. We also appreciate your constructive suggestions regarding long-term occlusion and motion prediction. We are grateful that you are leaning toward acceptance while maintaining your score (3). Thank you again for your thoughtful review and consideration.

Reviewer_HmbMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GEO-4D associates instance predictions from pretrained, frame-wise LiDAR panoptic segmentation networks without additional temporal training. After ego-motion compensation, it first matches likely static instances using centroid and covariance statistics. Remaining candidates are registered with ICP, using entropy-regularized optimal transport to establish point correspondences. A short-term memory bank supports reassociation after missed observations (Sec. 3.2).

The method achieves 72.7 validation LSTQ and 70.3 test LSTQ on SemanticKITTI, with a reported runtime of 1.10 seconds per scan and 3.15 GB memory usage (Table 2). It also reaches 67.9 LSTQ on panoptic nuScenes (Table 4). The appendix includes comparisons under fixed segmentation predictions, registration experiments, and an analysis of DBSCAN refinement.

Critical strengths
  1. The paper includes meaningful controls for segmentation quality. Under the same single-scan Mask4Former predictions, GEO-4D improves over CA-Net from 73.4 to 78.3 in validation association score and from 62.8 to 70.0 on the test set. Table A.4 also shows improvements with DS-Net predictions. These controls provide stronger evidence for the association method than the headline comparisons between systems with different segmentation networks. The nuScenes results extend the evaluation to a different sensor configuration and frame interval (Table 4; Appendix C.3).

  2. The ablations connect the pipeline design to both accuracy and computational cost. Table 5(a) separately evaluates histogram initialization, GGSM, and the memory bank. Table 5(b) shows that static-instance filtering reduces runtime from 1.86 to 0.96 seconds before memory-based recovery, while maintaining association quality. Appendix C.5 also reports the performance and runtime with and without DBSCAN refinement. These experiments help explain where the practical gains come from.

Critical weaknesses
  1. The comparison with the closest geometric association approach is incomplete. Section 2 distinguishes ICP-Flow mainly as a method limited to two adjacent scans. However, ICP-Flow also describes a multi-scan tracking extension in its supplementary material (Sec. 7). Training-free cluster association through rigid registration therefore already has a close precedent. The contributions to isolate here are its adaptation to panoptic identities, GGSM, and the surrounding association pipeline [1]. Table 5(a) is useful: adding GGSM to histogram-initialized ICP increases the association score from 75.4 to 76.8. However, comparisons with centroid and bounding-box association in Table 3(a) do not establish whether GGSM improves over a stronger geometric alternative. A matched ICP-Flow-style associator or a robust ICP variant, using the same masks, initialization, static filtering, and memory, would better establish the additional contribution.

  2. Important parts of GGSM remain underspecified, and its robustness explanation needs more direct support. Appendix B.2 defines generic transport marginals, but does not specify the point masses used in the experiments or the entropy regularization setting. Equation 8 also writes the cost using the original source points. It should state whether the cost is recomputed from the transformed source points at each ICP iteration; otherwise, the correspondence update is unclear. Equation 9 converts the transport plan into hard matches by taking each row's argmax. This operation need not preserve the column marginals, so the resulting correspondences do not retain all the constraints of the transport plan. Moreover, balanced transport requires all prescribed mass to be assigned, including mass belonging to outliers or non-overlapping regions. The paper should explain how these choices support its robustness claims. Figure 3 and Table A.5 provide useful evidence, but controlled tests varying overlap, outlier rate, and sampling density would make that explanation more convincing.

  3. The complexity comparison is overstated, and the runtime protocol needs clarification. Appendix B.2 treats dense Sinkhorn matching as O(n^2) and concludes that it is theoretically comparable to nearest-neighbor matching. For L scaling iterations, dense Sinkhorn requires O(L n^2) work and O(n^2) storage per registration iteration. The total association cost also depends on the ICP iterations and number of candidate instance pairs. This differs from the stated average O(n log n) nearest-neighbor cost. Table A.5 reports 1.62 seconds for ICP+GGSM versus 0.10 seconds for ICP, so the added registration cost is material. The reported end-to-end savings over learned baselines remain useful, but need a stage-level breakdown and scaling measurements. Table A.6 shows that DBSCAN changes runtime from 0.34 to 1.10 seconds, making the processing boundary especially relevant. Appendix A.1 should also clarify whether averaging the two stages means averaging total per-scan latency or averaging the stage runtimes themselves.

Questions for authors
  1. Could you compare against an ICP-Flow-style association pipeline and a robust ICP alternative under identical segmentation predictions, initialization, static filtering, and memory? Please also clarify the distinction from ICP-Flow's multi-scan extension.

  2. Please specify the OT marginals, entropy coefficient, Sinkhorn and ICP stopping criteria, and memory-window length. Is the transport cost recomputed after each transformation update? How are point-set IoU and competing greedy matches handled? Controlled overlap and outlier experiments, including a comparison between row-wise argmax and a transport-weighted rigid update, would help assess the proposed matching rule.

  3. Could you report segmentation, DBSCAN, static matching, dynamic registration, and memory-recovery times separately on stated hardware? Please clarify the two-stage timing protocol and show how association time and memory scale with point count and candidate count. Per-sequence results or uncertainty estimates would also help interpret the smaller benchmark differences, including the 0.3-point nuScenes LSTQ improvement over the two-scan Mask4Former baseline (Table 4).

3 comments
AuthorsOpenReview ↗

Response to Reviewer HmbM (1/3)

Thank you for the detailed and thorough feedback. Below, we provide our responses to each comment. We hope that our rebuttal below clarifies your concerns.

[W1, Q1] Comparison with ICP-Flow and Controlled Comparisons

As the reviewer points out, ICP-Flow estimates scene flow between pairs of nearby scans, then recovers scene flow across multiple scans by sequentially linking. This sequential strategy lacks an explicit recovery mechanism, meaning that intermediate matching failures may interrupt the tracking chain and registration errors can accumulate over time. In contrast, Geo-4D combines its memory-based strategy with object state-aware processing and global geometry-aware matching to provide a more flexible framework for maintaining consistent instance identities across scans.

To further validate the effectiveness of GGSM, we compare it against nearest-neighbor matching (ICP-Flow-style) and RobOT [1], a robust OT-based correspondence alternative. We replace only the point-wise correspondence estimation method while keeping all other association settings and the overall pipeline identical.

Correspondences StrategyLSTQ ↑SassocS_{\mathrm{assoc}} ↑SclsS_{\mathrm{cls}} ↑
Nearest-neighbor72.076.867.5
RobOT70.0 72.6 67.5
GGSM (Ours)72.778.367.5

Our empirical results show that GGSM outperforms both baselines, achieving gains of 1.5 and 5.7 points in SassocS_{\mathrm{assoc}} over nearest-neighbor matching and RobOT, respectively. To better understand these outcomes intuitively, we also provide a point correspondence visualization in the updated main paper (see Figure A.1). These results confirm that GGSM constructs geometrically consistent source–target point pairs, thereby improving instance association quality. Furthermore, applying our full pipeline to nearest-neighbor point matching improves LSTQ from 71.3 to 72.0 and SassocS_{\mathrm{assoc}} from 75.4 to 76.8, suggesting that the elaborately designed pipeline for 4D panoptic segmentation contributes to maintaining temporal instance consistency.

AuthorsOpenReview ↗

Response to Reviewer HmbM (2/3)

[W2, Q2] GGSM Implementation Details and Robustness Analysis

We thank the reviewer for the constructive feedback. We have provided the following settings and clarified them in the revised manuscript.

Implementation Settings. The memory bank retains instances from the previous 3 scans. For GGSM, the Sinkhorn regularization parameter is set to ϵ=0.2\epsilon = 0.2, with 30 Sinkhorn iterations and a maximum of 30 ICP iterations. Furthermore, at each ICP iteration, we recompute the transport cost using the currently transformed source points and update the correspondences accordingly. We have updated the main paper to include implementation details in Section 4.1 and to clarify the iterative correspondence update in Lines 243–245.

Validation and Matching. After rigid alignment, we compute nearest-neighbor distances between the transformed source and target point sets. Points whose nearest-neighbor distances are below 0.1 m are counted as inliers. We then calculate point-set IoU based on the number of overlapping points relative to the sizes of the source and target point sets. A candidate match is rejected if the minimum of the two directional IoU scores falls below 0.2, following our default configuration. For greedy matching, each source instance selects the target instance with the highest point-set IoU, provided that the score exceeds the matching threshold of 0.2. We allow multiple source instances to match the same target instance. As shown in Table A.2, greedy matching achieves 78.3 in SassocS_{\mathrm{assoc}}, compared to 77.6 with one-to-one Hungarian matching.

Marginal Constraints and Correspondence Selection. Thank you for your thoughtful feedback. We agree that the row-wise argmax in Eq. (9) does not necessarily preserve the column marginal constraints. However, this operation is applied only when selecting point pairs from the transport plan, which already satisfies the balanced OT constraints. Thus, the resulting correspondences still leverage the global geometric information captured by the transport plan. Also, we acknowledge that balanced OT does not explicitly exclude outliers or non-overlapping regions. However, our primary focus is not to improve the robustness of the OT formulation itself, but to obtain geometrically consistent point correspondences for temporal association between predicted instance masks. To verify the effectiveness of our proposed GGSM, we further present the qualitative and quantitative evidence.

Experimental Evidence. As discussed in our response to Reviewer ido3 (W3, Q2), we additionally compare GGSM with RobOT, which employs transport-weighted correspondence updates, under identical association settings. GGSM achieves 78.3 in SassocS_{\mathrm{assoc}}, substantially outperforming RobOT, which achieves 72.6. We have also updated the main paper with point-wise correspondence visualizations (see Figure A.1) to provide a more intuitive comparison between GGSM and transport-weighted target position estimation (RobOT). Together with the quantitative results, these visualizations clearly highlight the effectiveness of our correspondence strategy for instance-level association.

AuthorsOpenReview ↗

Response to Reviewer HmbM (3/3)

[W3, Q3] Complexity Analysis and Computational Efficiency

We thank the reviewer for the comment. We have updated the complexity analysis in Lines 818–819 of Appendix B.2 to explicitly account for the number of Sinkhorn iterations LL.

Efficient Implementation. We implement GGSM using PyKeOps [2]. PyKeOps supports ragged batching, allowing point clouds of varying sizes to be processed without zero-padding. We first use PyKeOps' from_matrix function to restrict transport computations to candidate instance pairs. This enables multiple independent Sinkhorn problems to be processed in parallel without unnecessary cross-instance computations. Moreover, matrix-free operations avoid explicitly storing dense cost and transport matrices. Notably, GGSM requires 0.081 s/scan, compared with 0.142 s/scan for a nearest-neighbor algorithm using PyTorch3D's knn_points under the same evaluation settings. Although GGSM has higher theoretical complexity, our optimized implementation achieves a lower measured runtime in this comparison. These optimizations improve practical efficiency without changing the asymptotic computational complexity. We also provide PyTorch-style pseudocode of our implementation in Algorithm 2 of the revised manuscript.

Runtime Breakdown. Since the NVIDIA RTX A6000 used in our original evaluation was no longer available, we conducted additional runtime profiling on an NVIDIA RTX 3090. Despite the difference in GPU hardware, the total runtime remained comparable, increasing modestly from 1.10 to 1.22 s/scan.

SegmentationStaticDynamicMissingDBSCAN
Runtime (s/scan)0.2830.0060.0810.0640.771

These results demonstrate that both GGSM and the overall Geo-4D pipeline maintain efficient runtime performance.

Analysis on the Number of Points and Candidates. To address this, we manually varying the number of maximum point counts from 300 to 3000. Specifically, we randomly subsample points from each instance when its point count exceeds the specified maximum.

Max Points3005001,0003,000 (Default)
Runtime (s/scan)0.0750.0780.0810.081
GPU Memory (GB)2.632.632.703.15

Reducing the maximum point count from 3,000 to 300 slightly decreases runtime from 0.081 to 0.075 s/scan, while reducing peak GPU memory from 3.15 to 2.63 GB.

Regarding candidate count, our efficient PyKeOps implementation processes multiple instance pairs in parallel using independent block-sparse operations. It processes an average of 216.4 candidate pairs in 0.081 seconds per scan. Although computational cost increases with the number and size of candidate pairs, this implementation avoids separate Sinkhorn calls for individual pairs, improving practical efficiency.

Two-Stage Association Protocol. For 4D-PLS, 4D-STOP, and EQ-4D-STOP, we follow their official GitHub implementations, which first generate predictions for 4D volumes and then perform temporal tracklet stitching. We report the total per-scan latency, including both stages.

Per-Scene Performance Analysis. We additionally report the mean and standard deviation of LSTQ, SassocS_{\mathrm{assoc}}, and SclsS_{\mathrm{cls}} across all 150 nuScenes validation scenes.

MetricLSTQ ↑SassocS_{\mathrm{assoc}} ↑SclsS_{\mathrm{cls}} ↑
Geo-4D (Ours)66.78 ± 2.2563.63 ± 1.8770.28 ± 5.34
Mask4Former66.78 ± 2.8161.59 ± 0.9972.47 ± 5.27

Although the overall LSTQ improvement is modest at 0.3 points, Geo-4D achieves a 2.04 point improvement in mean SassocS_{\mathrm{assoc}}, which is the primary focus of our method. Despite exhibiting greater scene-level variation than the overlap-based association approach of Mask4Former, Geo-4D still achieves higher average association quality. Moreover, its lower SclsS_{\mathrm{cls}} offsets this gain, resulting in identical mean per-scene LSTQ scores.

[1] Shen, Zhengyang, et al. "Accurate point cloud registration with robust optimal transport." Advances in Neural Information Processing Systems 34 (2021): 5373-5389.

[2] Charlier, Benjamin, et al. "Kernel operations on the GPU, with autodiff, without memory overflows." Journal of Machine Learning Research 22.74 (2021): 1-6.

Reviewer_ido3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Geo-4D, a training-free association framework for 4D LiDAR panoptic segmentation. The main idea is to associate frame-wise instance segmentation results predicted by existing 3D panoptic models, with separate handling of static, dynamic, and temporarily missing instances. Experiments on SemanticKITTI and panoptic nuScenes show competitive LSTQ and temporal association performance. Comparisons using identical segmentation predictions further support the effectiveness of the proposed method.

Critical strengths
  • The geometric association strategy is practical and does not require additional association training.
  • The proposed method achieves competitive overall performance with relatively low memory usage. The fixed-mask comparisons also support improvements from the association method.
  • The separate handling of different instance states is useful. The ablations show that static-instance filtering reduces runtime, while memory improves association.
Critical weaknesses
  • The technical contribution seems limited. The method mainly adds geometric temporal association to existing panoptic predictions, using ideas related to OT-based registration such as RobOT and ICP-based cluster matching such as ICP-Flow. The main technical difference needs to be clarified.
  • Comparisons with recent related methods are limited. The latest main baseline is Mask4Former (2024), while methods such as NextStop and D-PLS are not covered. Their differences in evaluation settings should be discussed, with matched comparisons where feasible.
  • The contribution of GGSM within the full framework is not sufficiently demonstrated. An ablation removing GGSM while retaining initialization and memory, together with a comparison against existing rigid OT matching, would better clarify its benefit.
Questions for authors
  • What is the main difference between GGSM and existing OT-based registration methods?
  • How does ICP with histogram initialization and memory perform without GGSM? How about replacing GGSM with existing rigid OT matching?
  • Could the authors clarify the differences from recent methods such as NextStop and D-PLS, and whether comparisons under matched settings are possible?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ido3 (1/3)

Thank you for the detailed questions and constructive suggestions. We hope our response fully addresses all concerns and demonstrates the strength of our contributions.

[W1, Q1] Technical Novelty of GGSM

We acknowledge that Optimal Transport (OT) has been extensively explored for point cloud registration. However, we would like to clarify the technical difference between GGSM and existing OT-based registration methods, particularly RobOT [1]. While both methods employ OT for correspondence estimation, they differ in their OT formulations and how the resulting transport plans are used for calculating point-wise correspondence, as summarized below.

MethodOT FormulationCorrespondence Estimation
RobOT [1]Unbalanced OTWeighted RobOT matching
GGSM (Ours)Balanced OTRow-wise argmax

Balanced OT enforces the marginal constraints on both source and target distributions, whereas unbalanced OT relaxes these constraints. Beyond this difference in OT formulation, the two methods also differ in how they convert the resulting transport plans into point correspondences. Weighted RobOT matching computes displacement for each source point toward a transport-weighted target positions. GGSM selects the observed target point with the highest transport mass for each source point. To further clarify the distinctiveness of our method, we describe how each method converts its transport plan into point-wise correspondences.

Let pip_i and qjq_j denote the source and target points, respectively. Let QijRobOTQ_{ij}^\mathrm{RobOT} and QijGGSMQ_{ij}^\mathrm{GGSM} denote the transport masses computed using each method's respective OT formulation.

RobOT employs weighted RobOT matching to estimate a transport-weighted displacement vector viRobOT\mathbf{v}_i^{\mathrm{RobOT}} for source point pi\mathbf{p}_i. The corresponding target positions, obtained by applying the estimated displacements to the source points, can be expressed as

\frac{ \sum_{j=0}^{J-1}\mathcal{Q}_{ij}^{\mathrm{RobOT}}\mathbf{q}_j }{ \sum_{j=0}^{J-1}\mathcal{Q}_{ij}^{\mathrm{RobOT}} }.$$ In contrast, as described in Eq. 9 of the main paper, GGSM selects correspondences from the Sinkhorn transport plan as follows: $$\hat{\mathbf{q}}_i^{\mathrm{GGSM}} = \mathbf{q}_{j^{\ast}(i)}, \qquad j^{\ast}(i) = \arg\max_j \mathcal{Q}_{ij}^{\mathrm{GGSM}}$$ This difference becomes important when multiple source points assign comparable transport weights to the same set of target points. For each source point $\mathbf{p}_i$, weighted RobOT matching computes $\hat{\mathbf{q}}_i^{\mathrm{RobOT}}$ by averaging target coordinates according to their transport weights. When different source points have similar normalized weights, these averages can lie near the same location, even if the source points are spatially separated. This concentration reduces the geometric information available for rigid registration (see our response to [W3, Q2] for details). In contrast, GGSM uses row-wise argmax to select the observed target point $\mathbf{q}_{j^\ast(i)}$ with the highest transport mass, **avoiding coordinate averaging**. To substantiate our claims, we provide additional experimental results and qualitative visualizations. As illustrated in the point-wise correspondence visualizations in the revised main paper (see Figure A.1), RobOT's estimated target positions tend to concentrate in a small region, whereas **GGSM produces more spatially distributed correspondences that better preserve the target geometry**. This observation is also supported by the controlled comparison in our response to [W3, Q2] below, where RobOT achieves lower association performance than GGSM under identical settings. Furthermore, Geo-4D offers distinct contributions beyond ICP-Flow, **not only in instance association but also in the overall pipeline design**. Our GGSM enables geometrically consistent association without being overly influenced by local proximity or outliers. Moreover, our state-aware pipeline ensures computational efficiency while leveraging historical memory retrieval to mitigate the inherent limitations of LiDAR observations caused by occlusion. We provide a more detailed empirical analysis in our response to Reviewer HmbM [W1, Q1]. We would like to emphasize that the primary contribution of this work is to **introduce a new technical route for 4D LiDAR panoptic, with GGSM as its core association mechanism**. Unlike methods relying on multi-scan networks or learned association modules, Geo-4D enables temporally consistent instance association across independently predicted scans without additional temporal training. Consequently, our framework provides an effective alternative to learned temporal association, as supported by the improvements observed under identical segmentation predictions and our ablation studies.
AuthorsOpenReview ↗

Response to Reviewer ido3 (2/3)

[W3, Q2] Compared with Existing Rigid OT

To isolate the contribution of GGSM, we conduct additional experiments by replacing GGSM with nearest-neighbor and RobOT, while keeping the segmentation predictions, histogram initialization, static filtering, memory bank, and remaining association pipeline identical.

Association StrategyLSTQ ↑SassocS_{\mathrm{assoc}} ↑SclsS_{\mathrm{cls}} ↑
Nearnest-neighbor72.076.867.5
RobOT70.0 72.6 67.5
GGSM (Ours)72.778.367.5

As shown in the table, GGSM achieves improvements of 1.5 and 5.7 points in SassocS_{\mathrm{assoc}} over the two alternative approaches, respectively. These results suggest that leveraging global geometric information enables more accurate instance association than conventional nearest-neighbor and weighted RobOT matching.

Notably, as discussed in our response to [W1, Q1], the lower performance of RobOT can be explained by its target correspondence estimation. When source points have similar transport distributions, weighted RobOT matching can map them to nearly identical locations, making rigid registration less reliable. Specifically, let pi\mathbf{p}_i and qj\mathbf{q}_j denote the ii-th source and jj-th target points, respectively, where i∈{0,…,I−1}i \in \{0,\ldots,I-1\} and j∈{0,…,J−1}j \in \{0,\ldots,J-1\}. When multiple source points have similar normalized transport weights, we can approximate

\qquad \sum_{j=0}^{J-1} b_j = 1$$ where $k$ indexes the target points for normalization, and $b_j$ represents a common normalized transport distribution across target points, independent of the source index $i$. Consequently, the target correspondences become $$\hat{\mathbf{q}}_i^{\mathrm{RobOT}} \approx \sum_{j=0}^{J-1} b_j \mathbf{q}_j = \boldsymbol{\mu}_q.$$ Thus, different source points can be mapped near the same transport-weighted correspondence $\boldsymbol{\mu}_q$, **reducing the spatial variation of correspondences needed for instance association**. This behavior is also illustrated in the visualization results provided in Appendix C.4.1 (see Figure A.1). Overall, these results support the effectiveness of GGSM in leveraging global geometric information to establish more reliable point-wise correspondences and improve instance association.
AuthorsOpenReview ↗

Response to Reviewer ido3 (3/3)

[W2, Q3] More Baseline Comparisons

We thank the reviewer for pointing out these recent works [2, 3]. We first clarify the key methodological differences below.

MethodInstance PredictionAssociation Strategy
NextStop [2]Pretrained single-scan panoptic modelBBox-based tracking
D-PLS [3]Learned multi-scan instance networkSpatio-temporal grouping
Geo-4D (Ours)Pretrained single-scan panoptic modelGlobal geometry-aware association

Unlike NextStop, which relies on bounding-box representations, Geo-4D exploits the global geometry of instance point clouds for temporal association. Furthermore, unlike D-PLS, our method requires neither multi-scan inputs nor additional instance network training. We provide the reported results on SemanticKITTI validation. The reported NextStop results are based on instance predictions from 4D-StOP.

MethodInput ScansLSTQ ↑SassocS_{\mathrm{assoc}} ↑SclsS_{\mathrm{cls}} ↑
D-PLS470.674.966.4
D-PLS269.573.066.2
NextStop169.772.766.8
Geo-4D (Ours)172.778.367.5

To provide a fair comparison of temporal association, we additionally apply NextStop's tracking method to the same pretrained segmentation network used by Geo-4D, keeping the semantic and instance masks identical. For D-PLS, the official implementation is not publicly available, and its instance prediction stage requires multi-scan inputs and additional training, preventing a direct comparison under identical panoptic prediction.

MethodInput ScansLSTQ ↑SassocS_{\mathrm{assoc}} ↑SclsS_{\mathrm{cls}} ↑
Mask4Former (NextStop)170.874.467.5
Mask4Former (Geo-4D)172.778.367.5

As observed in above table, Geo-4D achieves strong association quality using only single-scan predictions. Together with Table 3(a), these results suggest the benefit of exploiting point-level global geometry over coarse geometric cues.

[1] Shen, Zhengyang, et al. "Accurate point cloud registration with robust optimal transport." Advances in Neural Information Processing Systems 34 (2021): 5373-5389.

[2] Alkalay, Nirit, Roy Orfaig, and Ben-Zion Bobrovsky. "Nextstop: An improved tracker for panoptic lidar segmentation data." arXiv preprint arXiv:2501.06235 (2025).

[3] Steinhauser, Maik, et al. "D-PLS: Decoupled Semantic Segmentation for 4D-Panoptic-LiDAR-Segmentation." arXiv preprint arXiv:2501.15870 (2025).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Gentle Reminder: One Day Remaining in the Discussion Period

Dear Respected Reviewers (ido3 and HmbM),

Thank you very much for your time and effort in reviewing our paper and for your constructive feedback, which has been valuable in improving the quality of our manuscript.

We have provided detailed point-by-point responses to your questions and concerns, along with additional explanations and experimental results. We have also updated our manuscript accordingly, with the revisions highlighted in blue.

We understand that reviewing our detailed responses requires considerable time and effort, and we sincerely appreciate your consideration. As only one day remains in the discussion period, we would greatly appreciate it if you could review our responses and let us know whether they have adequately addressed your concerns. We would be happy to answer any additional questions.

Once again, thank you for your support, and we look forward to hearing from you.

Best regards,
The Authors

AuthorsOpenReview ↗

Author Final Remarks by Authors

Dear Reviewers, ACs, SACs, and PCs,

We sincerely thank you for your time and effort in the review process for our paper. To support your evaluation and facilitate constructive internal discussions, we have prepared the following summary of our rebuttal.

In their initial reviews, the reviewers recognized and praised our work in the following aspects: clear motivation and meaningful contributions (ido3, YBfp), strong technical design (ido3, YBfp), comprehensive experimental evaluation (ido3, HmbM), and practicality and efficiency (HmbM, YBfp). Following our rebuttal, Reviewer YBfp indicated that they were leaning toward acceptance.

The reviewers also offered constructive feedback for improvement, which we addressed and clarified in the following areas. We believe that the main concerns raised during the discussion have been substantially addressed through additional experiments and clarified exposition.

  • Technical Contribution of GGSM: We clarified why GGSM is well suited for instance association and how it differs from existing OT-based registration methods, particularly RobOT. By selecting point-wise correspondences rather than relying on transport-weighted averaging, GGSM mitigates geometric distortion and improves the stability of instance association. We further provided correspondence visualizations and additional analyses, which have been incorporated into the revised manuscript, to demonstrate the effectiveness of GGSM for instance association.
  • Detailed Latency: We provided a detailed runtime breakdown of our association pipeline, including static association, GGSM-ICP, and missing instance recovery. We further clarified how computational efficiency can be improved through an efficient PyKeOps implementation with ragged batching, which significantly reduces computational overhead. To facilitate reproducibility, we also included PyTorch-style pseudocode detailing the implementation in the revised manuscript.
  • Comparison with More Baselines: We extended our comparisons to recent 4D LiDAR panoptic segmentation methods, including NextStop and D-PLS. Our Geo-4D achieves 72.7 LSTQ and 78.3 S_assoc on SemanticKITTI, outperforming both methods in terms of segmentation and association quality, even when compared to D-PLS using multiple input scans. These results further demonstrate the effectiveness and competitiveness of Geo-4D against recent state-of-the-art methods.
  • Clarification of Implementation Details: We provided detailed explanations of Sinkhorn optimization, memory bank configuration, and instance association strategy. We also included additional robustness analyses under varying number of points and point density conditions, clarifying the design choices and validating the robustness of our geometric association framework.

We have highlighted all changes and additions in blue in the revised manuscript to facilitate the review process.

We sincerely appreciate the time and constructive feedback provided by the reviewers. We hope that our detailed responses, additional experiments, and manuscript revisions will be taken into consideration in the final evaluation. We believe these improvements further strengthen the paper and demonstrate the technical contributions and practical value of Geo-4D for 4D LiDAR panoptic segmentation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.