acceptodds
Under review as a conference paper at ICLR 2027

LoCoFormer: Rethinking Local Geometric and Semantic Context Representation for Point Cloud Transformers

Abstract

Synergistically capturing local geometric structures and semantic features remains a critical challenge in 3D point cloud analysis. Recent attempts to integrate local point operators into point cloud Transformers have made impressive progress, yet they still struggle to fully exploit local contexts. In this paper, we present LoCoFormer, an effective framework for comprehensive local geometric and semantic representation. Specifically, we identify the limitations of conventional Softmax attention and propose Bilateral Context Attention (BCA). BCA adopts delicate geometric and semantic context branches to separately capture spatial structures and implicit features, and utilizes a bilateral fusion mechanism to modulate attention weights and provide geometric priors, establishing enhanced local contextual dependencies. Furthermore, motivated by the observation that architectural design affects information acquisition and propagation, we modernize the network architecture from three aspects: , , and , which respectively determine receptive field size, feature initialization, and spatial alignment, thereby improving local context representation. Experimental results demonstrate that LoCoFormer achieves superior performance across multiple challenging benchmarks, including ModelNet40 for shape classification and S3DIS, ScanNet v2, and ScanNet200 for semantic segmentation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D point cloud analysis, Transformer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YHRwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LoCoFormer proposes Bilateral Context Attention, a dual-branch geometric/semantic attention with bilateral fusion, plus architecture modernization for point cloud classification and segmentation tasks. Experiments cover ModelNet40, S3DIS, ScanNet-v2, and ScanNet200. Overall, despite competitive indoor results, the work is weakened by the absence of outdoor road scene validation with occlusion and varying point densities.

Critical strengths
  1. The paper tackles limited local geometric context modeling in point cloud Transformers.
  2. BCA combines coordinate and feature branches via learnable geometric encodings and bilateral fusion for attention.
  3. Ablations show benefits from BCA interactions, PSS, sparse embedding, and partition pooling on indoor benchmarks.
Critical weaknesses
  1. Missing outdoor road scene validation. All classification and segmentation experiments are indoor (S3DIS, ScanNet v2/200). No SemanticKITTI, nuScenes, WHU-MLS, Waymo, or other outdoor road scene tests, so generalization to sparse, large-scale, occlusion-heavy road scenes is unproven.
  2. BCA is an incremental dual-branch attention design; PSS follows PTv3/OctFormer-style serialization, sparse-convolution embedding is standard, and partition-based pooling follows PTv2. The contribution over PTv3/LitePT is partially modular assembly.
  3. BCA is not fully evaluated in the segmentation setting. Appendix A.2 states that for semantic segmentation, BCA is used only in the first two stages, while the remaining stages use conventional Softmax attention. The paper does not ablate full-stage BCA versus this partial deployment, nor does it justify why BCA is omitted from later stages. This weakens the claim that BCA is the main source of segmentation gains and makes it difficult to separate its contribution from the architecture modernization and PTv3-style components.
  4. Gains over PTv3 are small (0.7% ScanNet v2, 1.1% S3DIS), with no outdoor baselines, few cross-dataset transfers, few robustness tests under density/occlusion changes, and no statistical variability.
Questions for authors
  1. Can LoCoFormer be evaluated on SemanticKITTI, nuScenes, or Waymo under the same protocol to verify generalization to sparse, large-scale outdoor road scenes?
  2. What is the parameter-matched gain of BCA alone versus PTv3-style attention, and why is BCA used only in the first two segmentation stages?
  3. How do PSS, sparse-convolution embedding, and partition pooling differ materially from prior work, and how does the S×3S geometric encoding scale with patch size?
4 comments
AuthorsOpenReview ↗

Response to Reviewer YHRw

Response to Question 1, Weakness 1, and Weakness 4:\textbf{Response to Question 1, Weakness 1, and Weakness 4:} Thank you for your comment. We have supplemented experiments on outdoor datasets. Please refer to Table F in the Appendix for details.

We hope you can appreciate that achieving performance improvements of 1.1 and 0.7 on the challenging S3DIS and ScanNet v2 benchmarks, respectively, represents a remarkable gain. For reference, we encourage you to consult previous top conference works [1-10] that report ablation studies on these benchmarks.

[1] Wan, Jiaxu, et al. "SP 2 T: Sparse Proxy Attention for Dual-stream Point Transformer." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[2] Zeng, Ziyin, et al. "DeepLA-Net: Very deep local aggregation networks for point cloud analysis." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

[3] Wu, Xiaoyang, et al. "Point transformer v3: Simpler, faster, stronger." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[4] Thomas, Hugues, et al. "Kpconvx: Modernizing kernel point convolution with kernel attention." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[5] Deng, Hao, et al. "Linnet: Linear network for efficient point cloud representation learning." Advances in neural information processing systems 37 (2024): 43189-43209.

[6] Duan, Lunhao, et al. "Condaformer: Disassembled transformer with local structure enhancement for 3d point cloud understanding." Advances in Neural Information Processing Systems 36 (2023): 23886-23901.

[7] Qian, Guocheng, et al. "Pointnext: Revisiting pointnet++ with improved training and scaling strategies." Advances in neural information processing systems 35 (2022): 23192-23204.

[8] Lai, Xin, et al. "Stratified transformer for 3d point cloud segmentation." 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, 2022.

[9] Wu, Xiaoyang, et al. "Point transformer v2: Grouped vector attention and partition-based pooling." Advances in neural information processing systems 35 (2022): 33330-33342.

[10] Qian, Guocheng, et al. "Assanet: An anisotropic separable set abstraction for efficient point cloud representation learning." Advances in Neural Information Processing Systems 34 (2021): 28119-28130.

AuthorsOpenReview ↗

Response to Reviewer YHRw

Response to Question 2 and Weakness 3:\textbf{Response to Question 2 and Weakness 3:} Thank you for your comment. We have provided additional ablation studies that independently modify the attention mechanism. Please refer to Table G in the Appendix for details.

To reduce memory overhead and mitigate overfitting, we employ BCA only in the first two stages, as explained in the Appendix (Section A.2).

The ablation results in Table G of the Appendix clearly demonstrate the performance gains brought by both BCA and Architecture Modernization, further substantiating the contributions of our work.

AuthorsOpenReview ↗

Response to Reviewer YHRw

Response to Question 3 and Weakness 2:\textbf{Response to Question 3 and Weakness 2:} Thank you for your comment. The proposed Architecture Modernization is inspired by previous works, but has been tailored at a fine-grained level to work synergistically with our proposed BCA. Importantly, we would like to emphasize that the motivation behind Architecture Modernization is not to pursue methodological novelty within individual components, but rather to systematically analyze and explore architectural designs of point cloud Transformers beyond the attention mechanism. To the best of our knowledge, our work is the first to pursue this objective.

The two-stage design and bilateral fusion mechanism of BCA are specifically tailored for 3D point clouds, making BCA both novel and effective. Unlike images, which consist of regularly arranged pixels, point clouds are inherently sparse, irregular, and unordered. These differences make our BCA fundamentally different from existing methods in the 2D image domain. Furthermore, the geometric context branch of BCA independently processes point coordinates, providing spatial distribution and geometric structure information. This design also distinguishes our BCA from existing methods in the 3D vision domain.

The shape of ΔM∈RS×3S\Delta M \in \mathbb{R}^{S \times 3S} can be aligned with the query, key, and value of the semantic context after being mapped through the transformation layers, as explained in Eq. (7). The patch size is only related to the receptive field of the network and does not affect the geometric encoding of each individual point.

Thanks to the authors for their improvement. I will keep my score unchanged.

Reviewer_WzRdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LoCoFormer combines Bilateral Context Attention (BCA) with serialized neighborhoods, sparse-convolution input embedding, and partition-based pooling. BCA constructs geometric encodings from flattened within-patch relative coordinates, pools queries to obtain refined queries, and performs two successive attention operations, incorporating geometric terms into the attention logits and values. The paper reports results on ModelNet40 classification and semantic segmentation on S3DIS, ScanNet v2, and ScanNet200.

Critical strengths
  1. The reported results improve upon the selected baselines across multiple datasets. Table 2 reports mIoU scores of 74.5/79.4 on S3DIS Area 5/6-fold cross-validation, 78.2 on ScanNet v2, and 36.5 on ScanNet200. Relative to the listed PTv3 results, these correspond to improvements of 1.1/1.7, 0.7, and 1.3 percentage points, respectively. ModelNet40 overall accuracy (OA) reaches 94.2%, matching the highest listed result. These results provide positive evidence of improvements over the selected baselines.

  2. There is evidence that the improvements do not arise solely from changes to the overall architecture. Table 3 reports 74.5 mIoU with BCA versus 73.5 with conventional attention and xCPE in the same overall network. Table 5 evaluates all seven nonempty combinations of the three geometric interaction locations, with the full combination performing best. Tables 6 and C also examine individual and paired architectural changes.

  3. The appendix provides useful practical information. This includes stage widths and depths, the different attention configurations used for classification and segmentation, training configurations, and latency and memory comparisons. These disclosures are useful, although important details remain unresolved.

Critical weaknesses
  1. The core BCA operator is not specified sufficiently to reconstruct its tensor operations. In Section 3.2, let a patch contain S points, let the pooled refined queries contain R tokens, and let the per-head dimension be d. The first attention logits should then have shape R × S. Under the natural row-wise interpretation of Eq. (7), the geometric encodings have S rows: consequently, Q_rfn E_q^T has shape R × S, whereas K E_k^T has shape S × S. When R differs from S, their sum in Eq. (11) is generally undefined. Similarly, Eq. (13) adds a refined value with R rows to a geometric value without specifying how the latter is reduced or aligned. The paper does not provide R, the pooling axis/kernel/stride, or the output shapes of the geometric transforms. The implementation might resolve these issues through shape-preserving pooling, additional reductions, or a different indexing convention, but none of these is defined in the manuscript.

  2. The architectural and BCA contributions need clearer separation from established designs. PSS in Section 3.3 explicitly refers to PTv3 and uses Z-order, Hilbert curves, and their variants to construct serialized neighborhoods. Replacing k-NN with serialization and expanding neighborhoods from 16 to 1024 points were also already demonstrated in PTv3. Sparse-convolution input embedding has precedent as well, including in the official PTv3 implementation, while partition-based pooling is explicitly adopted from PTv2. The architecture modernization therefore primarily integrates and evaluates existing components, leaving BCA as the main potential methodological contribution. However, pooling queries followed by two-stage aggregation and redistribution has precedents such as Agent Attention [1], and BCA's bilateral fusion is explicitly inspired by the contextual relative position encoding of Wu et al. BCA's specific differences lie in its flattened local geometric encoding, the generation of second-stage keys and values from aggregated features, and the coupling of these designs. The paper should explain what limitations of existing methods these differences address and demonstrate their value through targeted comparisons. This does not imply that BCA is equivalent to an existing mechanism, but its distinct technical contribution has not yet been sufficiently established.

  3. The scope of the claimed performance advantages needs clarification, and comparisons with recent strong methods remain incomplete. The abstract (page 1, lines 26–28) and contribution summary (page 3, lines 113–115) use the relatively broad phrase “superior performance.” The experiments mainly report improvements over specific baselines such as PTv3, LitePT, and Volt-S, while the classification comparison with PointLearner uses “competitive performance.” Thus, the paper does not explicitly claim state-of-the-art results on every benchmark, but more complete comparisons are still needed to establish the scope of its advantages. LoCoFormer reports 74.5% mIoU on S3DIS Area 5, yet the main table omits published methods including DeepLA-Net/DeepLA-120 (CVPR 2025, 75.7%), CamPoint [2] (CVPR 2025, 83.3%), and PointMC [3] (AAAI 2026, 83.6%). These are results reported in the respective papers, and differences in inputs, sampling, augmentation, and evaluation must be checked; they should not be treated as reproductions under identical settings. Nevertheless, their omission prevents the main table from fully establishing the method's relative standing. Pretrained methods such as Sonata [4] (CVPR 2025, 76.0%) and PointCSP [5] (CVPR 2026, 88.2%) should be listed separately with their pretraining data and adaptation procedures specified, rather than mixed into a comparison of backbones under identical conditions. I recommend prioritizing the missing recent supervised baselines and limiting the performance statements in the abstract and contribution summary to the comparisons supported by the evidence.

  4. Comparisons directly relevant to the motivation and the accuracy–cost trade-off remain incomplete. The main table already covers several backbones, including PTv2, PTv3, PointNeXt, OctFormer, ConDaFormer, LitePT, and Volt-S. However, SP²T and GPSFormer, which are discussed in the related work, are absent from the experiments. The former concerns proxy representations and local/global information fusion, while the latter explicitly models local geometry, making them relevant additional segmentation and classification comparisons, respectively. KPConvX would also provide a more representative modern geometric baseline than the original KPConv alone. Meanwhile, Table E shows that PTv3 and LoCoFormer have similar parameter counts, at 46.2M and 46.5M, but LoCoFormer increases latency from 51 ms to 67 ms and memory from 4.1 GB to 4.8 GB, approximately 31% and 17% increases, for a 0.7-point ScanNet mIoU gain. This demonstrates an accuracy–cost trade-off, but does not by itself establish that the trade-off is better. The paper should report point counts, numerical precision, batch size, GPU synchronization, and repeated timing procedures corresponding to the accuracy evaluation protocol, and compare the key models under the same protocol.

  5. Several experimental details require clarification. Appendix A.1 refers to the 312-scene ScanNet split as the test set, whereas the official benchmark identifies it as the validation set. The terminology should be corrected, and the authors should confirm that all baselines use the same split. The source of each result, model version, pretraining/additional data, augmentation, and test-time voting/TTA/fragment aggregation settings should be specified. The number of sampled input points and the use of normals on ModelNet40 also need clarification. Table 2 on page 8 (printed line 384) lists Point Transformer's S3DIS 6-fold mIoU as 65.4. However, Table 2 on page 6 of the original Point Transformer paper explicitly reports OA/mAcc/mIoU of 90.2/81.9/73.5; the text on page 5 and Appendix Table A.1 also confirm a 6-fold mIoU of 73.5. If this entry is taken from the original paper, it should be corrected to 73.5; if it comes from an independent reproduction, its source and setting differences should be stated. The absence of variability across random seeds for key comparisons also limits assessment of the stability of small improvements.

Questions for authors
  1. Please provide shape-annotated pseudocode for Eqs. (6)–(13), specifying the input/output shapes of each geometric embedding, the number of refined queries, the pooling axis/kernel/stride, and the reduction, broadcasting, or alignment operations in Eqs. (11) and (13). If pooling produces only one refined token, how is the second attention prevented from degenerating into a query-independent broadcast?
  2. Does PSS make any substantive algorithmic changes relative to PTv3? What operations does BCA introduce beyond existing two-stage attention and contextual position encoding? Under the same backbone and training settings, how do the two-stage branch without geometric terms, an Agent Attention [1]-style module, conventional geometric encoding, and full BCA compare in accuracy and cost?
  3. Why are recent supervised baselines such as DeepLA-120, CamPoint [2], and PointMC [3] absent from the main table? Please complete these comparisons, list pretrained results such as Sonata [4] and PointCSP [5] separately, clarify the ScanNet split, baseline sources, and evaluation procedures, and verify the Point Transformer 6-fold result. Is the key BCA-versus-xCPE comparison stable across random seeds?
  4. Can you add comparisons with SP²T, GPSFormer, and KPConvX on the corresponding tasks, or explain which settings cannot be directly aligned? Please also specify the timing protocol for Table E and explain under what resource budgets the gains justify the additional latency and memory.
2 comments
AuthorsOpenReview ↗

Response to Reviewer WzRd

Response to Question 2:\textbf{Response to Question 2:} Thank you for your comment. The PSS has been tailored at a fine-grained level to work synergistically with our proposed BCA. Importantly, PSS is only one contribution of our Architecture Modernization. We would like to emphasize that the motivation behind Architecture Modernization is to systematically analyze and explore architectural designs of point cloud Transformers beyond the attention mechanism. To the best of our knowledge, our work is the first to pursue this objective.

The two-stage design and bilateral fusion mechanism of BCA are specifically tailored for 3D point clouds. Unlike images, which consist of regularly arranged pixels, point clouds are inherently sparse, irregular, and unordered, making our BCA fundamentally different from existing methods in the 2D image domain. Furthermore, the geometric context branch of BCA independently processes the coordinate information of point clouds, providing spatial distribution and geometric structure information. This design also distinguishes our BCA from existing methods in the 3D vision domain.

Please refer to Table 5 in the revised manuscript for the corresponding ablation study.

AuthorsOpenReview ↗

Response to Reviewer WzRd

Response to Question 3:\textbf{Response to Question 3:} Thank you for your comment. We agree that the methods you mentioned are relevant to this research area. However, we would like to clarify two aspects regarding the fairness and scope of the comparisons: 1. Our LoCoFormer takes only raw point clouds as input, whereas PointMC and CamPoint additionally leverage camera perspective features, and SP$^{2}$T utilizes additional proxy point clouds. 2. Our LoCoFormer is trained in a supervised manner using a single dataset, whereas Sonata adopts self-supervised training with joint training on multiple datasets, and PointCSP employs self-supervised training with knowledge distillation. So we did not add these methods for comparison. We have supplemented the results of DeepLA, GPSFormer, and KPConvX, all of which achieve lower performance than our LoCoFormer on ScanNet v2. Please refer to Tables 1 and 2 for details.

The dataset split protocol for ScanNet has already been provided in the Appendix (Section A.1).

The results reported in our paper are stable and reliable.

The description of Table E has already been provided in the Appendix (Section A.4).

Reviewer_E4eFOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the joint modeling of local geometric structures and semantic features in point cloud Transformers. The method consists of two core parts. The first is Bilateral Context Attention (BCA): a geometric branch uses relative coordinate offsets to generate learnable geometric encodings, a semantic branch aggregates features in two stages through a refined query, and a bilateral fusion mechanism then uses the geometric encodings to modulate the attention weights and inject geometric priors into the values. The second is the overall architecture design, which replaces k-NN with a Point Serialization Strategy (PSS) to enlarge the receptive field, uses sparse convolution for raw-point embedding, and adopts partition-based pooling. Finally, the authors conduct experiments on ModelNet40, S3DIS, ScanNet v2, and ScanNet200.

Critical strengths
  1. The manuscript proposes a Bilateral Context Attention (BCA) module. This module uses a geometric branch to encode relative coordinate offsets within local regions, explicitly characterizing the spatial structure between point pairs, and a semantic branch to progressively aggregate features.

  2. The authors systematically improve the network architecture from three aspects: spatial proximity, raw-point embedding, and pooling strategy.

Critical weaknesses
  1. The novelty of this manuscript is limited. Specifically, PSS directly follows the serialization strategy of PTv3, partition-based pooling comes from PTv2, and the sparse convolution embedding is also similar to the design in PTv3. The proposed method is therefore an incremental one.

  2. The compared methods are insufficient, and some recent methods are missing. For example, DELA also adopts a serialization strategy, and CamPoint likewise adopts serialized sequences. The paper also does not compare with pre-training-enhanced PTv3 variants such as Sonata and PPT, or with recent methods such as OA-CNNs. On ModelNet40, the OA is on par with SAMBLE and PointLearner, showing no advantage. In addition, experiments on outdoor datasets (e.g., nuScenes and SemanticKITTI) are missing.

  3. Some parts of the method are not clearly described. The number and size of the refined queries QrfnQ_{\text{rfn}} after max pooling are not specified; the notation ΔE\Delta E in Eqs. (11)–(13) is inconsistent with that in Eq. (7); and the design of mixing BCA and Softmax attention across different stages lacks explanation.

  4. The model efficiency reported in the manuscript shows no advantage. Compared with PTv3, LoCoFormer has a comparable number of parameters but higher inference latency and larger memory consumption. Moreover, FLOPs are not reported, and the computational cost of BCA under large receptive fields is not analyzed separately.

  5. Figure 2 claims that the MLP embedding converges slowly and performs poorly, whereas in Table 7 the difference between MLP (74.1) and sparse convolution (74.5) is only 0.4.

Questions for authors

The novelty of the method in this manuscript is limited, its advantage over state-of-the-art methods is not evident, the method description is not sufficiently clear, and the experiments are insufficient. I suggest that the authors provide explanations for the weaknesses I have raised.

5 comments
AuthorsOpenReview ↗

Response to Reviewer E4eF

Response to Weaknesses 1:\textbf{Response to Weaknesses 1:} Thank you for your comment. The proposed Architecture Modernization is inspired by previous works, but has been tailored at a fine-grained level to work synergistically with our proposed BCA. Importantly, we would like to emphasize that the motivation behind Architecture Modernization is not to pursue methodological novelty within individual components, but rather to systematically analyze and explore architectural designs of point cloud Transformers beyond the attention mechanism. To the best of our knowledge, our work is the first to pursue this objective.

AuthorsOpenReview ↗

Response to Reviewer E4eF

Response to Weaknesses 2:\textbf{Response to Weaknesses 2:} Thank you for your comment. We agree that the methods you mentioned are relevant to this research area. However, we would like to clarify two aspects regarding the fairness and scope of the comparisons: 1. Our LoCoFormer takes only raw point clouds as input, whereas CamPoint additionally leverages camera perspective features. 2. Our LoCoFormer is trained in a supervised manner using a single dataset, whereas Sonata and PPT adopt self-supervised training and leverage joint training on multiple datasets. So we did not add these methods for comparison. We have additionally included OA-CNN in our comparisons. Its performance is substantially lower than that of our LoCoFormer (74.5 vs. 71.1 on S3DIS, 78.2 vs. 76.1 on ScanNet v2, and 36.5 vs. 33.3 on ScanNet200). Please refer to Table 2 for details. Regarding DELA, we found multiple papers with identical titles or names and thus could not identify which specific work you refer to. We would appreciate it if you could provide further details, such as the full paper title.

SAMBLE and PointLearner are among the state-of-the-art methods in this field, demonstrating the effectiveness of our approach.

We have supplemented our experiments on outdoor datasets. Please refer to Table F in the Appendix for details.

AuthorsOpenReview ↗

Response to Reviewer E4eF

Response to Weaknesses 3:\textbf{Response to Weaknesses 3:} Thank you for your comment. The refined query QrfnQ_\mathrm{{rfn}} has the same shape as the vanilla key (KK) and value (VV), i.e., RC×D\mathbb{R}^{C\times D}.

Equations (11) – (13) describe the operations for the hh-th attention head, whereas Eq. (7) presents the overall attention. Therefore, there is no inconsistency between these equations, as already clarified in the original manuscript (Section 3.2).

The hybrid use of BCA and Softmax attention is intended to reduce memory overhead and mitigate overfitting, as also explained in the Appendix (Section A.2).

Comments on the paper

1 comment
AuthorsOpenReview ↗

A summary of major changes in the revised manuscript

First, we sincerely thank the Program Chairs, Senior Area Chairs, Area Chairs, and all anonymous reviewers for their careful assessment of our manuscript. Their valuable comments and suggestions play an important role in improving the quality of the manuscript. In this revision, we have made the following changes to the original manuscript (highlighted in blue text):

  1. We have further clarified the motivation and significance of Architecture Modernization in Section 3.3 to better highlight our contributions.
  2. We have extended the comparisons in Tables 1 and 2 by including additional state-of-the-art methods suggested by the reviewers.
  3. We have added experiments on outdoor scene datasets in the Appendix (Table F) to further evaluate the generalization capability of our network.
  4. We have added independent ablation studies on the attention mechanism and Architecture Modernization in the Appendix (Table G) to further clarify and validate our contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.