acceptodds
Under review as a conference paper at ICLR 2027

Factorized Probabilistic Inference for Multimodal Few-Shot Point Cloud Segmentation

Abstract

Multimodal Few-Shot Point Cloud Semantic Segmentation (MFS-PCS) leverages Vision-Language Models (VLMs) to reduce reliance on dense 3D annotations. However, existing methods often perform inference in an entangled latent space that jointly mixes visual, textual, and geometric cues. Such a monolithic treatment obscures the distinct roles of semantic priors and geometric structure, often leading to biased prototypes and misalignment between external semantics and the intrinsic manifold of the query scene. From a probabilistic perspective, we argue that few-shot inference can be naturally factorized into semantic likelihood estimation and geometric manifold regularization. Motivated by this view, we propose Decoupling Semantic Likelihood and Geometric Priors (DSG). For semantic transfer, we reformulate support prototype construction as a Maximum Likelihood Estimation problem rather than a deterministic averaging procedure, and employ Expectation Maximization to estimate robust semantic prototypes from sparse and noisy support observations. For geometric regularization, we introduce unsupervised structural anchors that capture the latent Voronoi structure of the query manifold. These anchors act as transductive geometric priors, encouraging predictions to respect local neighborhood structure and remain coherent across geometrically related regions. By separating what a category is from where and how it appears in the query geometry, DSG better aligns external multimodal semantics with internal 3D topology. Extensive experiments on S3DIS and ScanNet show that DSG achieves new state-of-the-art performance for MFS-PCS.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal, Few-Shot, Point Cloud, Semantic Segmentation, Decoupling Semantic Likelihood, Geometric Priors

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NDfTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DSG addresses multimodal few-shot point cloud segmentation through support-derived semantic prototypes and query-derived structural anchors, each estimated by one FPS-initialized Lloyd/Hard-EM update. Improvements span S3DIS, ScanNet and multiple way/shot settings. SRI transfer and boundary evaluation provide useful supporting evidence; the appendix discloses the classical clustering equivalence and the limits of the representation analysis and high-way training.

Critical strengths
  1. Clear problem framing.
  2. Consistent main results plus stability evidence.
Critical weaknesses
  1. The probabilistic interpretation needs a clearer connection to inference. Appendix F.1 identifies the updates as classical single-step Lloyd/Hard-EM refinement. The pathways are subsequently fused through attention, while low CKA/RSA does not establish functional independence. Please define decoupling and explain its benefit to the final prediction. For fixed representations and a nonzero normalizer, the label-independent geometric factor in Eq. (3) cancels upon posterior normalization; please explain its relationship to the implemented fusion.
  2. Query-anchor benefits need more direct attribution. SRI uses the unlabeled query distribution. Transduction is legitimate here; attMPTI also uses it (Zhao et al., 2021). A support-derived-anchor control could identify the benefit of query adaptation. A capacity-matched generic query-refinement control under the same inputs and episode protocol would help distinguish the structural design from added capacity and query modeling.
  3. Robustness to noise and initialization is incompletely characterized. Modality ablations establish usefulness, but not tolerance to corrupted features or 2D-3D misalignment. One assignment-and-mean update may depend on the FPS selection and outliers. A limited alignment-noise or initialization perturbation test would help; alternatively, please delimit robustness claims to tested conditions. Fixed initialization itself does not imply random predictions.
  4. The structural prior and anchor budget need clearer limits. Table 12 already varies anchor counts, and Appendices B.1/G.3 acknowledge geometric–semantic conflicts. Anchors operate in learned feature space, not a purely spatial partition. Please explain what the existing evidence establishes about the default budget across scene complexities, and qualify boundary-preservation claims where geometry and semantics disagree.
  5. Training cost limits demonstrated scalability. Appendix E.3 reports nearly 80 GB for MM-FSS 2-way 5-shot training and identifies higher-way training as a limitation. This differs from Table 5’s 0.06-second inference overhead. Tables 6-7 offer a lower-cost 2,048-point/DGCNN configuration. Please specify the feasible way/shot range and accuracy–memory trade-off, distinguishing backbone and point-budget effects from module costs.
Questions for authors
  1. Eq. (3) writes the posterior as a product of semantic likelihood and a label-independent geometric prior. After normalization, a query-only factor cancels unless it modulates the likelihood. How is the Geometric Prior actually realized in Eq. (15)–(16): as a multiplicative prior, an additive logit bias, or an implicit cross-attention modulation?

  2. Why query-derived anchors help beyond support prototypes. SRI uses the unlabeled query distribution. Mechanistically, what does a query anchor capture that a support prototype cannot? Is the benefit from query adaptation itself, or from the added cross-attention capacity?

  3. Eq. (13)–(14) gates visual likelihood by text affinity T. If the text prior is ambiguous or misaligned with the 3D domain, does the gating amplify the error rather than suppress it? Is there a mechanism to down-weight unreliable text evidence, or does TMSA assume the VLM prior is always informative?

Reviewer_DQ9NOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DSG for multimodal few-shot point cloud semantic segmentation. DSG has two main parts. Tri-modal Semantic Alignment (TMSA) combines class prototypes from the support set with textual guidance. Structure Regularized Inference (SRI) adds structural anchors taken from the unlabeled query. Both prototypes and anchors are built with FPS initialization and one Lloyd (Hard-EM) update. The authors view this design as a factorization of semantic likelihood and geometric priors. They report gains on S3DIS and ScanNet.

Critical strengths
  1. Consistent gains. DSG beats all reported baselines in all eight dataset, way and shot settings in Tables 1 and 2. The largest gain over DA-FSS is 8.36 mIoU.
  2. Extra evidence for SRI. The appendix includes a boundary evaluation. It also adds SRI to MM-FSS and DA-FSS. These results support SRI beyond the main benchmark and the qualitative examples.
  3. Clear implementation details. Appendix F.1 explains the single-step clustering, the handling of empty clusters, and the gradient flow. The paper also separates representation dissimilarity from statistical independence. It separates episode sampling variance from training variance as well.
Critical weaknesses
  1. The probabilistic formulation does not hold. In Eq. (3), the geometric term does not depend on the label. For fixed features, it cancels out when the posterior is normalized over classes. It therefore cannot change the label posterior as the paper claims. In addition, the attention fusion and the cosine-softmax classifier in Eqs. (15) and (16) are not derived as MAP inference under this model. The main probabilistic claim is not supported, even if the architecture works well in practice.

  2. The geometric claims go beyond the theory. Appendix A.2 derives a vector quantization objective. It does not show that the cosine affinity between points and anchors approximates a Laplace-Beltrami operator. FPS runs in coordinate space, but the clustering runs in the IF and UF feature spaces. No spatial connectivity constraint is used. So feature clusters may not match connected geometric regions. The claims about topology preservation and manifold regularization need stronger support or more careful wording.

  3. The roles of FPS and clustering are not isolated. The ablation from TMSA to TMSA+SRI changes anchor construction, attention and fusion at the same time. Table 12 varies the number of anchors. This does not show whether FPS or the clustering update is needed. Two controlled baselines are missing. One uses FPS-only anchors. The other uses random initialization with the same update. Table 13 gives different results for k-means and Hard-EM, although the paper says they are equivalent under the same conditions. The paper does not state what differs between the two runs. Table 13 also concerns support prototypes, not query anchors.

  4. Novelty is limited and the decoupling claim lacks support. DSG mostly combines existing IF and UF features, a standard FPS-initialized Lloyd update, and attention-based fusion. The main new element seems to be refinement conditioned on query anchors. The paper does not show that this amounts to a new inference principle. DA-FSS already uses semantic and geometric experts, separate refinement and late arbitration. The control variants in Table 9 are not described clearly. The parameter count also grows from 11.43M to 16.01M. A coupled baseline with matched capacity is needed.

  5. Few comparisons with recent methods. The main tables include only one 2025 method (MM-FSS) and one 2026 method (DA-FSS). The authors should add results from other recent methods.

  6. Training robustness is unclear. Tables 17 and 18 vary the test episodes but keep the trained models fixed. They do not measure variance across training seeds. DSG also shows higher episode variance on ScanNet, which the paper does not discuss. On S3DIS S0S^0 in the 2-way setting, the score drops from 50.22 at 1-shot to 48.99 at 5-shot. This drop should be examined, given that the total prototype budget is fixed.

  7. Little evidence of generalization. The evaluation covers only S3DIS and ScanNet. The structural anchors are meant to capture query geometry. Tests on another dataset such as ScanNet++ or Replica would show whether this holds for other scenes and geometry. A cross-dataset transfer experiment would also help. Its label mapping and support and query construction should be clearly stated.

  8. The practical value of the episodic setting is not well argued. The setup assumes a few known target classes and densely labeled support blocks. All other classes count as background. The paper does not describe a deployment scenario. It also does not measure the labeling cost or the benefit over pretrained 3D vision-language models with simple few-shot adaptation. Comparisons under the same input and supervision would help. So would results with a larger label space or joint base and novel class recognition.

Questions for authors
  1. How can the label-independent geometric term in Eq. (3) change the normalized label posterior? How do Eqs. (15) and (16) carry out MAP inference?

  2. What assumptions and graph construction support the Laplace-Beltrami approximation in Appendix A.2?

  3. With the rest of the model fixed, how do the following query anchors compare: FPS only, random initialization plus one update, and FPS plus one or more updates? What differs between the k-means and Hard-EM runs in Table 13?

  4. Could the authors describe the Table 9 variants in detail? Could they add a coupled baseline with matched capacity and an SRI-only ablation? How exactly does DSG differ from DA-FSS?

  5. Could the authors add comparisons with more recent methods?

  6. Could the authors report several independent training runs for one representative setting? Could they also study the 1-shot and 5-shot reversal with controlled support sampling and prototype budgets?

  7. Could the authors evaluate on another dataset or across datasets, for example ScanNet++ or Replica, with a clear protocol?

  8. What deployment scenario motivates the episodic setting? What does DSG offer over pretrained 3D vision-language features with simple support-based adaptation, under similar labeling and compute budgets?

5 comments

Thank you for the detailed response and the substantial effort put into the revisions and new experiments. I have also read the reviews by ANwM and NDfT. Like ANwM, I had concerns about novelty and the component ablations. Like NDfT, I questioned the probabilistic explanation and whether the query-anchor benefits came from added capacity. I also shared their concerns about the geometric interpretation of the anchors. The revised explanations and the added SRI-only, anchor-source, and capacity-matched tests address these concerns well. The repeated training runs, cross-dataset tests, and frozen-feature comparisons also respond to several of my other requests. These additions strengthen the evidence for the method. My assessment of the paper is now more positive.

For the revised manuscript, I have a few suggestions:

  1. Please make sure the key experiments are fully included in the paper or supplementary material. A short summary with clear table references would help readers follow the evidence.

  2. Please clearly state the prototype budgets and evaluation settings. Also distinguish training-run variation from test-time initialization variation, and per-GPU memory from four-GPU totals.

  3. Please briefly clarify the settings used for the k-means and Hard-EM results in the original Table 13.

AuthorsOpenReview ↗

Follow-up Response to Reviewer DQ9N

Thank you for your positive reassessment and helpful suggestions. Next, we will respond to your suggestions:

Q1. Guide to Experiments

R1: All key experiments discussed in our previous responses are included in the revised main paper or supplementary material. We have added a “Guide to Appendix Experiments” on p. 12, the index below summarizes their purposes and locations.

Index of key experimental evidence. Tables 1, 4, and 5 are in the main text, the remaining references are in the appendix.

Experiment groupPurpose and scopeTables and figures
Component contributions and transferStandalone and complementary TMSA/SRI contributions, SRI integration into existing methods.Tables 4, 12, 24
Anchor and capacity attributionAnchor sources, parameter-matched alternatives, generic query refinement, and early coupling.Tables 32–34
Construction and budget sensitivityRefinement depth, prototype/anchor budgets, assignment space, initialization, updates, and existing shot-budget results.Tables 13–16, 37
Independent training variationIndependent meta-training runs.Table 35
Test-time sensitivity and variabilityFixed-model FPS/text perturbations, episode-set variation, and per-shot support sensitivity.Tables 17–18, 30, 22–23, 36
Cross-dataset transferBidirectional S3DIS–ScanNet transfer.Tables 8–9
Practical adaptation and label spaceFrozen-feature baselines versus the trained DSG backend, joint base/novel recognition.Tables 38, 31
Feature-space structure and boundariesSpatial connectivity of feature clusters and accuracy at semantic boundaries.Tables 28–29, 25, Figure 8
Accuracy and resource costs2,048-point results, timing, and training memory, reported separately per GPU and for four-GPU totals.Tables 6–7, 26–27, 39–40
Recent method comparisonsRecent methods, including EPSegFZ.Table 1
Model complexityParameter counts and inference latency of MM-FSS and DSG.Table 5
Historical inference interventionsFixed-checkpoint affinity routing and prediction blending, not matched decoupling controls.Table 11
Mechanistic comparisonConceptual comparison of relation objects, query usage, and refinement/fusion mechanisms.Table 41
Representation-similarity diagnosticsSemantic–structural CKA/RSA similarity.Table 10

Q2. Experimental Settings and Memory Accounting

R2: Thank you for requesting these clarifications. The settings and reporting conventions are as follows:

Prototype and anchor budgets. In the reported budget-50 configurations, Np=50N_p=50 denotes the total number of intermediate support prototypes per class, and Na=50N_a=50 denotes the number of class-agnostic query anchors. Each IF/UF branch uses these counts. The support prototypes are constructed independently within each shot and concatenated: each shot contributes Np/KN_p/K prototypes (50 in 1-shot and 10 in 5-shot). Final segmentation uses one masked-mean prototype per class. The Np=Na=10N_p=N_a=10 operating point in Table 39 is a separate resource-constrained 2-way 5-shot configuration and does not change the budgets associated with the reported budget-50 accuracy results.

Evaluation settings. The main benchmark covers 1-way/2-way and 1-shot/5-shot tasks on both S3DIS and ScanNet, using two class splits. Evaluation uses 1,000 episodes per class for 1-way tasks and 100 episodes per class combination for 2-way tasks, a S3DIS split therefore contains 1,500 two-way episodes. We average target-class IoUs and then average across the two splits. The standard configuration uses at most 20,480 points per 1 m×1 m1\ \mathrm{m}\times1\ \mathrm{m} block, alternative protocols are identified separately. (Secs. 4.1–4.2, App. F.2.)

Sources of variation. On S3DIS S0S_0, under the 2-way 1-shot setting, Table 35 reports five independent meta-training runs with shared pretrained initialization and a common evaluation set: 50.32 ± 0.11% mIoU. Under the same setting, Tables 17–18 hold the trained checkpoint fixed and vary only test-time FPS initialization, without retraining. Across the original fixed-start setting and four randomized FPS settings, mIoU is 50.34 ± 0.10% (mean ± sample SD). Tables 22–23 separately report variation across test-episode sets with trained models fixed.

GPU Memory. Table 7 reports per-GPU peak training memory for the 2,048-point/DGCNN configuration: approximately 11 GB for 2-way 1-shot and 30 GB for 2-way 5-shot. Tables 5, 39–40 report aggregate peak training memory across four RTX A6000 GPUs for the standard Stratified Transformer configuration with up to 20,480 points per block. For example, Table 5 reports peak training memory of 24.73 GB for DSG and 23.56 GB for MM-FSS under this configuration. Table 39 reports 55.21 GB at Np=Na=50N_p=N_a=50 and 21.24 GB at Np=Na=10N_p=N_a=10 for 2-way 5-shot. Table 40 reports 24.73 GB for full DSG under 2-way 1-shot.

AuthorsOpenReview ↗

Follow-up Response to Reviewer DQ9N

Q3. Settings of k-means and Hard-EM in the original Table 13

R3: The original Table 13 is Table 15 in the revised manuscript. It compares support-prototype construction on S3DIS and ScanNet under the S0S_0, 2-way 1-shot setting:

VariantInitializationAssignment spaceUpdate
k-means (XYZ assignment)XYZ-FPSXYZ coordinates1
DSG (Hard-EM)XYZ-FPSLearned IF and UF feature spaces, separately1

The prototype budget, class masks, feature aggregation, downstream architecture, and training/evaluation settings are otherwise identical.

Thus, the comparison evaluates the assignment space, rather than different clustering update rules. We have relabeled the XYZ-assignment variant and clarified this distinction in App. D.6 and revised Table 15, App. F.1 specifies the implemented update.

Under the stated GMM assumptions, Lloyd and Hard-EM are equivalent when the representations, distance metric, initialization, and update settings are identical, the two variants here differ in assignment space.

Reviewer_ANwMOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

From a probabilistic perspective, this paper decomposes few-shot inference into semantic likelihood estimation and geometric prior regularization, and proposes the DSG framework for multimodal few-shot point cloud semantic segmentation. The method contains two core modules. The first is Tri-Modal Semantic Alignment, which estimates multiple support prototypes for each class through FPS initialization and single-step Hard-EM, computes the cosine similarity between query points and prototypes, and fuses text embeddings through a text-guided attention mechanism to refine the semantic likelihood. The second is Structure Regularized Inference, which estimates structural anchors on the query features in an unsupervised manner, computes point-to-anchor affinities as the geometric prior, and then corrects the semantic predictions through cross-attention so that they conform to the intrinsic structure of the query scene. Finally, the authors conduct experiments on S3DIS and ScanNet.

Critical strengths
  1. The manuscript proposes a Tri-Modal Semantic Alignment module. This module uses FPS-initialized multi-prototype estimation to characterize the multimodal intra-class distribution, and fuses text embeddings, cross-modal features, and point cloud unimodal features through text-guided gated attention, alleviating the insufficient representativeness of a single prototype under sparse support samples.

  2. The authors propose a Structure Regularized Inference module. This module extracts structural anchors on the query set in an unsupervised manner and provides structural consistency constraints for prediction in a transductive way, so that semantic confidence propagates along the local structure of the query scene, which alleviates boundary leakage and prediction fragmentation.

Critical weaknesses
  1. The so-called “EM-based prototype estimation” is actually just a single k-means update after FPS initialization, and the authors also acknowledge that no new clustering rule is proposed. The structural anchors are likewise obtained by single-step k-means on query features, which is similar in spirit to the multi-prototype design of attMPTI and the query-guided ideas of methods such as QGE. I suggest that the authors clearly articulate the novelty of the proposed method.

  2. The anchors of the “geometric branch” are obtained by clustering learned semantic features rather than from the 3D geometry itself, and the claim in the appendix that “the affinity graph approximates the Laplace–Beltrami operator” also lacks proof. Please provide corresponding evidence.

  3. The ablation study is insufficient. Table 4 contains only three settings, namely the baseline, +TMSA, and +TMSA+SRI; the result of using SRI alone is missing, and the specific form of the baseline is not described.

  4. Recent works on few-shot point cloud semantic segmentation are missing. For example: [1] GRIP-Net: Graph Relational Interactive Prototype Network for few-shot point cloud semantic segmentation [2] CascadeProto: Cascaded Cross-Modal Prototype Purification via Entropy-Aware Learning for Few-Shot 3D Point Cloud Segmentation [3] From Coarse to Fine: Deep Prototype Refinement Network for Few-Shot Point Cloud Semantic Segmentation [4] Reasoning beyond points: A visual introspective approach for few-shot 3d segmentation [5] Biologically-inspired evolutionary domain symbiosis for few-shot and zero-shot point cloud semantic segmentation

  5. There are some problems with the writing. The symbol KK denotes both the number of support shots and the TMSA depth, and the text also contains informal expressions such as “exciting results” and “pushover geometric regularizer.” I suggest that the authors further check the writing of the manuscript.

Questions for authors

I thank the authors for their detailed response and the extensive additional experiments. The rebuttal resolves my concerns about the novelty, the interpretation of the structural anchors, the ablation study, the related work, and the writing. I believe the proposed method offers a reasonable degree of novelty and achieves state-of-the-art performance. I find the authors' response convincing, so I have raised my rating to 4. I encourage the authors to incorporate all of these revisions completely and consistently in the final version.

1 comment

I thank the authors for their detailed response and the extensive additional experiments. The rebuttal resolves my concerns about the novelty, the interpretation of the structural anchors, the ablation study, the related work, and the writing. I believe the proposed method offers a reasonable degree of novelty and achieves state-of-the-art performance. I find the authors' response convincing, so I have raised my rating to 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.