acceptodds
Withdrawn from ICLR 2027

BEYOND ARGMAX: A MECHANISTIC STUDY OF SEMANTIC RETENTION IN FROZEN FOUNDATION-MODEL COMPOSITION FOR GENERALIZED FEW-SHOT 3D SEGMENTATION

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Classical classifier-combination work distinguishes score-level fusion from hard decision voting. We revisit this distinction in a modern regime where independently pretrained, frozen foundation models are composed at inference time for generalized few-shot 3D segmentation. Our contribution is not a new sum rule; we ask a mechanistic question: *how much useful semantic information is lost when heterogeneous sources are collapsed to one class before interaction?* We construct a same-input top- retention intervention that freezes model weights, point clouds, masks, geometry, class vocabularies, scene lists, and fusion rules while varying only the semantic support retained before fusion. On 156 held-out ScanNet200 scenes, full retention reaches harmonic-mean (HM) IoU versus for the frozen top-1 control (, 95% paired scene-bootstrap CI ). The effect independently replicates on all 50 ScanNet++ validation scenes ( vs. HM; , CI ). Most of the gain appears immediately after top-1, while moderate top- can slightly outperform the unrestricted tail. The conclusion survives a GroundingDINO–SAM2.1 sparse-source replacement, alternative fusion operators, tie-policy changes, and source-weight variation. Calibration diagnostics show severe but opposite raw miscalibration of the dense and sparse sources, yet calibration alone does not eliminate the retention advantage. We therefore identify *premature semantic collapse* as a repeatable information bottleneck in heterogeneous frozen-model composition.

Primary area:
foundation or frontier models, including LLMs
Keywords:
3D semantic segmentation, foundation models, multimodal fusion, few-shot 3D segmentation, classifier combination, semantic retention, open-vocabulary 3D

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Eod3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether premature semantic information collapse through argmax operations limits the effectiveness of composing frozen foundation models for generalized few-shot 3D semantic segmentation. The paper concludes that premature semantic collapse constitutes an information bottleneck in heterogeneous frozen foundation-model composition.

Critical strengths

S1. The matched top-k intervention is reasonable. By keeping upstream predictions and fusion configurations fixed, the authors provide evidence that semantic retention affects final segmentation performance.

S2. The results on ScanNet200 and ScanNet++ show substantial improvements over top-1 fusion, supported by paired scene bootstrap confidence intervals.

Critical weaknesses

W1. Novelty is limited. This paper neither proposes a new method for probability fusion nor identifies any theoretical properties of sufficient novelty. The authors devote a considerable portion of the paper to demonstrating that a known conclusion remains valid in a new context. Although the authors acknowledge this, the paper primarily reproduces this known principle in a frozen 2D–3D foundation-model setting.

W2. Despite presenting itself as a mechanistic study, the paper relies primarily on performance comparisons across different top-k retention levels rather than explaining how retained semantic hypotheses contribute to cross-modal fusion. This paper still has additional pages available for further research. Without more targeted diagnostic experiments, the proposed explanation remains insufficiently substantiated, weakening the paper's main scientific contribution.

Questions for authors

Q1. What fundamentally new insight does this study provide beyond the well-established advantage of score-level fusion over hard decision fusion?

Q2. Why does moderate top-k retention consistently outperform full-distribution fusion on both datasets? Can the authors provide quantitative evidence explaining which retained semantic hypotheses contribute to correct predictions and which introduce harmful interference?

Q3. Can the authors independently vary semantic retention in the dense and sparse sources (Top-1/Top-1, Full/Top-1, Top-1/Full, Full/Full)? This would clarify whether the reported improvements genuinely arise from cross-source semantic complementarity or are predominantly driven by one source.

Reviewer_zd71OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the effect of retaining class probabilities when combining frozen foundation models for generalized few-shot 3D segmentation. The authors combine a pretrained 3D model (RegionPLC) with 2D segmentation models (SAM3 or GroundingDINO–SAM2.1). Instead of converting each model's prediction into a hard label before fusion, they retain the top-k class probabilities and average the distributions. Experiments on ScanNet200 and ScanNet++ show that retaining more than one class improves segmentation performance. The paper also studies different fusion rules, calibration methods, and source weights.

Critical strengths
  1. The paper provides controlled experiments where the model predictions and other inputs are fixed, and only the number of retained classes changes. This makes it easier to understand the effect of top-k retention.

  2. The authors evaluate the approach on ScanNet200 and ScanNet++, and report consistent improvements over top-1 fusion. They also test different fusion operators and replace SAM3 with GroundingDINO–SAM2.1 in a smaller experiment.

  3. The paper reports confidence intervals and includes failure cases. These help readers understand that the improvement is not consistent across all scenes.

Critical weaknesses
  1. Very limited novelty. The main finding is that retaining class probabilities before combining predictions performs better than applying argmax to each model first. This is a well-known idea in ensemble learning, as the authors also acknowledge in Section 2. The proposed method does not introduce a new fusion strategy or a new segmentation model. Equation (4) simply keeps the top-k class probabilities and averages them. While the paper shows that this approach works for 3D segmentation, I do not think applying an established idea to this setting is a sufficient contribution for ICLR.

  2. The analysis does not provide enough new insights. The paper presents itself as a mechanistic study, but most experiments compare different values of k or standard fusion and calibration methods. The results mainly confirm that argmax discards useful information. The paper does not explain which classes benefit most, when retaining more classes becomes harmful, or how the agreement between the two models affects the results. For example, Section 6 reports a failure case where the HM drops from 31.66 to 19.41 after full-distribution fusion, but does not provide a detailed explanation of why this happens. A stronger analysis of these cases would make the study more useful.

  3. The experimental setting is limited. The main experiments combine RegionPLC with SAM3 and evaluate on two indoor datasets, ScanNet200 and ScanNet++. Although the authors also test GroundingDINO–SAM2.1, this experiment uses only five ScanNet200 scenes. This is not enough to establish that the findings generalize across different foundation models and 3D segmentation settings. The paper also provides limited qualitative comparisons, making it difficult to understand where the improvements and failures occur.

  4. The top-1 baseline is weak. The paper reports that the two models disagree on 67.93% of points with sparse evidence. Since the top-1 baseline averages two one-hot predictions with equal weights, disagreements produce exact ties that must be resolved using a tie-breaking rule. Although the authors test several tie-breaking rules, this baseline remains a rather simple form of hard fusion. I would like to see comparisons with stronger confidence-based hard selection methods to better understand how much of the reported improvement comes from the information retained by top-k fusion.

  5. The presentation could be improved. The paper is relatively short, and several important results are presented mainly through tables. In particular, the failure analysis in Section 6 is brief, and the qualitative results in Figure 2 are small and difficult to examine. More detailed visualizations of successful and failed predictions would help readers understand the claimed mechanism.

Questions for authors
  1. What is the main new insight of this work beyond the established difference between decision-level and probability-level fusion? The paper acknowledges that this distinction is well known, but I do not see a clear new method or finding beyond its application to 3D segmentation.

  2. Why does full-distribution fusion significantly reduce performance in some scenes, such as scene0490_00? Can the authors analyze which classes or prediction patterns cause these failures?

  3. Can the authors provide stronger hard-fusion baselines that use the models' confidence scores to select predictions, rather than relying mainly on averaging one-hot vectors?

  4. Can the authors provide more qualitative results showing which objects or semantic classes benefit from top-k retention, and where it fails?

Reviewer_4rmtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the loss of semantic information caused by premature argmax when multiple frozen foundation models are composed for generalized few-shot 3D segmentation. The authors use RegionPLC as the dense 3D semantic source and cross-view SAM3 concept evidence as the sparse semantic source, and design a same-input top-k retention intervention experiment. On this basis, the authors analyze the effect of the retention level on base-class and novel-class segmentation performance, and examine factors including sparse-source replacement, different fusion operators, tie-breaking rules, source weights, and confidence calibration. Finally, the authors conduct experiments on ScanNet200 and ScanNet++, as well as a GroundingDINO–SAM2.1 replacement experiment on 5 scenes.

Critical strengths
  1. The manuscript designs a controlled same-input top-k retention intervention experiment. The experiment fixes all upstream evidence and fusion rules and varies only the semantic support retained before fusion, thereby cleanly isolating the effect of “premature semantic collapse.”

  2. The authors provide a fairly comprehensive robustness analysis, including sparse-source replacement, comparison of different fusion operators, changes in tie-breaking rules and source weights, and confidence calibration diagnostics, showing that the retention advantage does not stem from any particular setting.

Critical weaknesses
  1. The novelty of this manuscript is limited and incremental. Specifically, the superiority of score-level fusion over decision-level voting is a known conclusion in classical classifier-combination theory, and Proposition 1 is rather trivial. The authors do not propose a new method; the main contribution is an empirical confirmation of a known phenomenon in a specific setting.

  2. The Top-1 control is not set up reasonably. Equal-weight one-hot fusion produces ties on 67.93% of the points, so the Top-1 result is largely determined by the tie-breaking rule. Moreover, more reasonable hard-fusion baselines, such as weighted voting that retains the top-1 confidence, are missing.

  3. The task setting does not match “generalized few-shot.” The method uses no support samples at all and is essentially zero-shot open-vocabulary fusion, and its performance is clearly lower than that of GFS-VL and HOP3D.

  4. The experiments are insufficient. The GroundingDINO–SAM2.1 replacement experiment uses only 5 scenes with 10 frames per scene, so its statistical significance is limited. On ScanNet++, the sparse source covers only novel classes, making the base-class and novel-class conclusions asymmetric.

  5. The “mechanistic study” in the title is overstated. The paper does not analyze in depth which classes, which candidate ranks, or which types of spatial regions the performance gains actually come from. In addition, the caption of Figure 2 still contains draft remnants such as “file supplied with the submission.”

Questions for authors

The novelty of the method in this manuscript is limited, its core conclusion is a known phenomenon, the control setup may overestimate the size of the improvement, the task setting does not match generalized few-shot segmentation, and the experiments are insufficient. I suggest that the authors provide explanations for the weaknesses I have raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.