acceptodds
Under review as a conference paper at ICLR 2027

Tomos3D: Zero-Shot 3D Instance Segmentation from Cutting Plane Renderings

Abstract

We present Tomos3D, a zero-shot method for instance segmentation in 3D scenes. Existing approaches segment input images and then match partial, noisy masks from different viewpoints, which is the central challenge dominating their design. Instead, we take a textured mesh as input and render multiple cutting plane views of it. Their number is independent of the number of input frames. Every rendered pixel maps to a 3D triangle. Each view is segmented independently and its masks lifted directly onto the mesh. The per-view SAM 2 segmentations are then stitched into one coherent global decomposition purely by mask overlap on the mesh, with no cross-view matching. To classify the objects, we propose MaskProbe, which adapts the final attention pooling of a frozen vision-language model using projected instance masks. Multiple instances thus share one image-encoder pass per image, with pooling performed separately for each instance. For evaluation, we report the fraction of ground-truth instances recovered. Tomos3D outperforms state-of-the-art training-free methods by a large margin, 51.5 against 43.1 on ScanNet, and comes within 2.8 points of the supervised ESAM-E on its training domain. The same setting transfers without adaptation to 3RScan and SceneNN, where Tomos3D outperforms even the supervised approaches (with a score of 72.9 versus 59.0 for the best prior method). The MaskProbe recognition head improves open-vocabulary recall, outperforming even MLLM-based methods, while reducing labeling time. The code will be made publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3d instance segmentation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Xf9XOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TOMOS3D: it samples cutting planes of a textured reconstructed mesh, renders the two sides of each plane separately, generates masks with a frozen SAM 2, and then obtains 3D instance proposals at three granularities through per-pixel triangle correspondence, a surface-overlap graph, and stability clustering. In the semantic stage, MASKPROBE projects the proposals onto the original photographs and restricts the patches that take part in the readout only in the final attention pooling of the frozen SigLIP2, so that multiple instances share one image-encoder pass.

Critical strengths
  1. Cutting-plane rendering offers a proposal-generation path worth studying, and the cross-dataset results are strong. Under the paper's unified AR@500 protocol, TOMOS3D clearly above the training-free baselines evaluated in Table 1.

  2. The recognition experiments contain a well-controlled comparison. MASKPROBE and OM3D+ use the same 3D proposals, SigLIP2 encoder, selected photographs, and text prompts, so the differences in Table 2 reflect the role of the readout fairly directly. The shared photograph encoding also yields a measured speed-up of 1.47–2.70× in the reported recognition stage.

Critical weaknesses
  1. MaskProbe's mechanistic contribution is modest, and its "context-aware" explanation is not supported by the experiments. MaskProbe improves average mAR@500 by roughly 1.7 points over OM3D+ (41.4 vs. 39.7), a limited gain that is also unstable in some settings: On SceneNN with the photo prompt, SigLIP2 multi-crop is actually higher than MaskProbe (63.9 vs. 62.3); Under the GT mask setting on SceneNN, the two are on par (80.5 vs. 80.5). The paper interprets MaskProbe as a context-aware masked attention pooling, i.e., reducing background interference by restricting the readout region. Yet the current experiments do not directly support this mechanistic explanation. The SigLIP2 image encoder itself has global self-attention, and the mask acts only on the attention logits of the final pooling; all earlier tokens have already mixed in context outside the mask. The results of Fig. 11 further weaken this explanation: MASKPROBE-crop and full-frame MASKPROBE achieve almost identical mAR@500 (64.2 vs. 64.1). This indicates that, given the same region, whether the full-image context is retained has little effect on the result.

  2. The existing experiments change several factors at once: cutting-plane rendering, SAM 2.1, the three-granularity output, surface-overlap clustering on the mesh, and stability clustering. Ablations such as the number of planes cannot substitute for testing "the contribution of cutting-plane rendering itself." More notably, there is a large gap between the single-granularity results and the union of the three granularities: On ScanNet, the best single-granularity result is only 34.6 AR@500; whereas SAI3D achieves 43.1 AR@500; TOMOS3D reaches 51.5 AR@500 only through the union of the three granularities. That is, on ScanNet, single-granularity TOMOS3D is below SAI3D, and only after merging the three granularities does it gain a clear lead. The current experiments cannot answer the following: if the same SAM 2.1, three-granularity output, mesh aggregator, and clustering pipeline are kept, and only the 80 cutting-plane renderings are replaced with the same number of original recorded frames or uncut ordinary mesh renderings, how much would the result change? The paper's statement that "cutting-plane rendering is the main innovative contribution" still requires more direct experimental support.

Questions for authors
  1. Does MaskProbe's improvement genuinely come from masked attention pooling, rather than merely from the region-feature readout itself? Under the same protocol as Table 2 (same Tomos3D proposals, same selected views, same projected masks, photo prompt, 20-template ensemble, and frozen SigLIP2), please add a simple region-feature readout as a control, for example using average pooling or max pooling over the patches inside the projected mask. Please compare: MaskProbe; region average pooling; MASKPROBE-crop. If possible, further fix the target region and only change the visible background around it, and observe whether the paired recognition results change. This would help to judge whether the "context-aware masked attention pooling" emphasized by the paper is indeed the source of the performance gain, or whether MaskProbe's benefit mainly comes from a more sensible region readout or from avoiding the information dilution of multi-crop.

  2. Can the contribution of cutting-plane rendering be separated from the three-granularity output, SAM 2.1, and the aggregation pipeline? On a fixed subset of 10–20 ScanNet scenes, keep SAM 2.1, the three-granularity output, the mesh aggregator, and HDBSCAN stability clustering unchanged, and replace only the 80 cutting-plane renderings with (a) 80 uniformly sampled original RGB-D recorded frames and (b) 80 uncut ordinary mesh renderings (please state the camera and rendering settings). Report AR@500, AR50, AR90, and proposals per scene in one table. At the same time, please add a per-item comparison between the union of the three granularities and each single-granularity result, clarifying whether the final performance mainly comes from the cutting-plane views or from the expansion of the candidate set after retaining multiple granularities.

5 comments
AuthorsOpenReview ↗

Corrected response

Q1 / W1: MaskProbe's efficiency and source of its gain.

Despite MaskProbe's modest accuracy gains, its efficiency advantage remains substantial.

Efficiency. MaskProbe encodes each frame once and pools features for all visible instances; OpenMask3D encodes three crops per instance and view. This reduces encoder inputs 6–12× (3RScan: 2606→413; SceneNN: 1858→244). We repeated timing with a parallel CPU path (A100, bf16, batched OpenMask3D crops, identical encoder/masks/views, 10 ScanNet scenes):

ScanNet, per sceneOpenMask3DMaskProbe
Encoder images4026335
Labelling time27.8 s8.8 s

MaskProbe is 3.2× faster end to end. We will update Tab. 9.

Source of the gain. With GT masks, frozen SigLIP2 and the same 20-template ensemble (mAR@500):

ReadoutScanNet (312)3RScan (46)SceneNN (12)Replica (8)Mean
MaskProbe66.168.380.558.468.3
Region mean pooling (same tokens/head, uniform weights)65.667.083.563.369.9
OM3D multi-crop60.766.280.544.963.1
One crop, 15% margin60.467.280.543.763.0
One crop, no margin57.864.482.545.062.4

Mean pooling is slightly better than MaskProbe on average. Thus, the gain over crops mainly comes from region-restricted readout, not learned attention weights. Gains are smaller on SceneNN, where objects occupy more of the image: median GT area is 1.47 m² versus 0.46/0.52 on ScanNet/3RScan, and masks cover 66/110 tokens (finest/coarsest), versus 23/45 and 16/28 of 729. SceneNN has an 18-class, mostly furniture vocabulary; by object count, MaskProbe recovers 101/123 vs. 98 for multi-crop. On Replica, MaskProbe improves over multi-crop by +5.4 mAR@500 (21.5 vs. 16.1, identical proposals).

We agree that Fig. 11 does not establish a benefit from retaining full-image context: earlier encoder layers already mix context, and MaskProbe and MaskProbe-crop perform similarly. We will remove this claim. Our contribution is efficient region-restricted readout from shared image features.

Q2 / W2: Cutting-plane rendering vs. three granularities.

We ran the requested controlled ablation on 50 ScanNet val scenes. All variants use 80 views, identical SAM 2.1 settings, three levels, mesh aggregation and HDBSCAN; only the views change. We also compare cuts against uncut renders at the same plane cameras:

80 input viewsProposals/sceneAR@500AR50AR90
Cutting-plane renders30654.077.426.1
Same plane cameras, no cut32948.572.521.7
Recorded RGB-D frames39142.170.414.3
Mesh renders at frame cameras43138.164.112.1

Plane renders are orthographic, 1024², vertex-coloured. Uncut views use the same 40 planes × 2 directions, showing the full mesh without clipping. Recorded frames use original poses/intrinsics (pinhole, 1296×968); ordinary mesh renders use those cameras. Clustering was tuned on cut views, potentially disadvantaging alternatives.

80 input viewsBest single level (coarse)Three levelsGain
Cutting-plane renders37.454.0+16.6
Same cameras, no cut34.048.5+14.5
Recorded RGB-D frames29.542.1+12.6
Mesh renders at frame cameras25.438.1+12.7

At the same cameras, cutting adds 5.5 AR@500. Versus recorded frames, cut views gain 7.9 with one level and 11.9 with three, despite fewer proposals (306 vs. 391). Per level, planes beat recorded frames at the finest (35.0 vs. 17.0) and coarsest (37.4 vs. 29.5), but trail at the middle (16.5 vs. 27.0). The three-level union adds 12.6–16.6 points for every view source. These results separate the contribution of cutting-plane rendering from retaining multiple granularities. We can also provide all individual-level results for the uncut views if needed.

We retain all three levels because target granularity is unknown in zero-shot segmentation. The segmentation overhead is small: rendering and SAM passes are shared, and SAM takes ~90% of the 266 s per ScanNet scene on RTX 4090 (App. D.1). Extra levels increase semantic labelling time.

In summary, cutting-plane views and multiple granularities contribute independently. We will incorporate these ablations and the corrected MaskProbe interpretation in the revised manuscript.

The current response largely reuses text addressed to another reviewer, omits my first question entirely, and lacks an updated manuscript. My Weak Accept rating was contingent on clarifying the mechanistic attribution of your two contributions; unless these points are directly addressed and reflected in a revised PDF before the discussion period closes, I will lower my score.

First, regarding MaskProbe, please provide the region average/max pooling control requested in my initial review (which can be run quickly using your existing projected masks and cached SigLIP2 features) and clarify how you will revise the unsupported full-image context claims in light of Fig. 11.

Second, your 50-scene ScanNet comparison (note that the gain is 11.9–15.9 points, not 15.9–19.9) does not yet isolate the contribution of cutting-plane rendering. Please confirm whether all three inputs used the same 80-view budget, and report the missing metrics from my initial review (AR50, AR90, proposals per scene) alongside a breakdown of each single granularity (L0,L1,L2L_0, L_1, L_2) versus their three-level union. Furthermore, to decouple half-space clipping from global orthographic views, please compare cutting-plane renders against uncut orthographic renders under the same 80 camera poses and extents across both individual levels and their union.

Finally, please upload a revised PDF incorporating these ablations and your promised manuscript updates so they can be verified before the final decision.

Dear reviewer,

We sincerely apologize for the confusion. During the final formatting of our rebuttal, parts of the responses to different reviewers were accidentally mixed up. We have now corrected our original response, which addresses both of your questions with the requested experiments. We will also incorporate these results, clarify the contribution of MaskProbe, and revise the method presentation in the updated manuscript before the discussion closes.

We would appreciate it if you could take another look at our corrected response.

Reviewer_FkPKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free 3D scene instance segmentation method based on a 2D segmentation model, decomposing the task into two stages: class-agnostic 3D instance segmentation and semantic labelling.

Segmenting mesh renderings in the first stage is the core contribution of this paper. Existing 3D segmentation methods based on 2D segmenters operate on real input images that do not have a direct correspondence with the mesh, and subsequently lift the 2D masks back into 3D through various mechanisms. During this process, both the selection of real image frames and the lifting from 2D masks to 3D masks can introduce losses in accuracy or efficiency. Based on the insight that 2D instance segmentation relies primarily on coarse geometric shapes rather than photorealistic rendering appearance, this paper performs segmentation directly on rendered images that have an exact correspondence to the mesh, thereby bypassing the two steps that may otherwise degrade the segmentation process.

The second stage employs a frozen image-text encoder to assign semantic labels to the segmented instances, equipping the method with open-vocabulary 3D instance recognition capabilities and providing a more complete application pipeline for the first stage. However, in my view, the design of this stage builds upon existing masked attention pooling techniques and leans more toward an engineering implementation than an architectural innovation.

Through comparative experiments on three indoor scene segmentation datasets, the paper demonstrates the performance advantages of both stages over the compared baselines.

Critical strengths

The innovation regarding the target of 2D segmentation (i.e., applying the 2D segmenter to cutting-plane mesh renderings rather than recorded input frames) is sufficiently novel, and the paper convincingly justifies the rationale behind this design by analyzing the core difficulties in existing 3D segmentation pipelines.

Critical weaknesses

I find the experimental setup for the first stage (class-agnostic 3D instance segmentation) somewhat unreasonable, making the current results insufficient to support the effectiveness of the proposed design.

  1. Regarding metric selection, the authors evaluate solely using Average Recall (AR) rather than Average Precision (AP) and provide a justification for this choice, which I find acceptable. However, when comparing methods under AR, the training-free baselines assign at most one mask to any single location in a scene, whereas Tomos3D may assign multiple masks of different granularities to the same location. I consider this comparison under AR to be unfair, and segmentation results where a single spatial location can belong to multiple overlapping masks also have more limited downstream applicability. Although the supplementary material includes experiments evaluating individual granularity levels separately, if the optimal granularity varies across different test sets (e.g., L2 on 3RScan vs. L0 on SceneNN), then the selection of the granularity level itself becomes an additional dataset-specific hyperparameter.
  2. The authors claim in the Introduction that a single unified hyperparameter setting is used across all evaluations, yet state in the supplementary material that modified hyperparameters were used for two of the datasets (3RScan and SceneNN). This discrepancy can seriously mislead reviewers and weakens the reliability of the evaluation results.
  3. The 2D segmentation module in Tomos3D uses SAM 2.1, whereas some of the training-free baselines use SAM 1. To ensure a fair comparison, it would be preferable to replace their 2D segmentation modules with the same backbone model. For the first-stage experiments, if the method can ensure that each point in a single scene belongs to at most one mask (or restrict the total number of predicted masks across different methods to be identical, rather than merely bounding them under a given budget), and still outperform the baselines in both AR and AP (or at least AR) without dataset-specific hyperparameter tuning, I would be convinced that the experimental results support the effectiveness of the proposed design. Furthermore, the paper would benefit from additional validation experiments. For instance, regarding the core insight that "2D instance segmentation only relies on coarse geometric shapes and colors, whereas semantic labelling additionally depends on realistic rendering appearance," controlled experiments should be designed to verify this claim directly and better support the paper's argument.
Questions for authors
  1. Can the proposed method process all scenes across all datasets using a strictly unified set of hyperparameters without dataset-specific adjustments? (Alternatively, introducing an automated mechanism to adapt hyperparameters based on mesh properties such as tessellation density would also be reasonable.)

  2. Given that the final output consists of coexisting multi-granularity masks where the same spatial location may belong to multiple masks, what specific downstream applications can directly utilize such overlapping predictions without requiring a single coherent partition?

2 comments

Thank you for the review. We address your concerns below.

W2 / Q1: Unified hyperparameters. Most reported dataset-specific settings resulted from configuration leftovers. We reran all datasets with a unified configuration: cluster-selection tolerance 0.3, minimum cluster sizes (3, 3, 8), and mask-area floor 5×10⁻⁴. The results are shown below.

AR@500 with all three levels:

SettingScanNet3RScanSceneNN
Previous51.541.472.9
Unified51.941.470.9

The unified configuration preserves nearly all the reported performance, with a 2.0-point decrease on SceneNN and a 0.4-point increase on ScanNet.

W1 / W4: One mask per point. Each individual granularity level produces a partition: every point belongs to at most one mask. We retain three levels since the appropriate granularity is not known in advance and varies across datasets.

Our full output is therefore a multi-granularity proposal set rather than a single partition. We agree that these represent different output formats. However, matching the number of masks exactly is not necessarily a fair comparison of instance discovery: it can penalize methods for recovering additional objects. We instead evaluate under a common proposal budget using AR@K (Fig. 4, top).

Except on ScanNet, where the baselines were tuned, our coarsest level alone also outperforms the baselines at smaller proposal budgets (see Fig. 4, top row, App. C). The full multi-granularity output does not require selecting a level for each test dataset.

W3: SAM version. We replaced SAM 2.1 with SAM ViT-H, using the same checkpoint as SAM3D, without changing the remaining pipeline. Results are AR@500 (ScanNet: 50 scenes):

Method2D modelScanNet3RScan
SAM3DSAM ViT-H20.78.2
OursSAM ViT-H47.036.0
OursSAM 2.154.041.4

Even with the older SAM backbone, our method substantially outperforms SAM3D.

We also evaluated SAM 3 (without retuning, 10 ScanNet scenes only):

AR@500SAM 2.1SAM ViT-HSAM 3
Three levels53.649.750.4
Coarsest35.832.638.1
Finest34.331.031.4
Middle16.38.315.3
Recall at IoU 0.2596.795.090.5

SAM 3 improves the coarse decomposition but misses more objects. Overall, SAM 2.1 performs best in our pipeline. Importantly, the gains over SAM3D remain large with an identical 2D backbone.

W5: Role of rendering appearance. We compared different image sources while keeping the remaining segmentation pipeline fixed (50 ScanNet scenes):

SAM inputAR@500
Real RGB frames42.1
Mesh renders at the same frame cameras38.1
Cutting-plane renders54.0

Replacing photographs with mesh renderings at identical cameras reduces AR by only 4.0 points. In contrast, changing to cutting-plane views improves AR by 11.9–15.9 points. Thus, viewpoint selection has a substantially larger effect than rendering appearance for segmentation.

For semantic labelling, we compared image sources using the same masks and frozen SigLIP2 (ScanNet200, 50 scenes):

Labelling inputmAR@500
Plane renders, vertex colours7.1
Plane renders, projected photo texture12.9
Real RGB frames17.0

Recognition is much more sensitive to appearance. This supports using synthetic renderings for instance discovery and original photographs for semantic labelling.

Q2: Uses of overlapping masks. A single partition is available by selecting any granularity level. Keeping all three also preserves part–whole relationships that would be lost in a single partition. Overlapping proposals are common in instance segmentation, including Mask3D or SAM.

Our output can be used directly for open-vocabulary instance recognition, as demonstrated in the paper (see Tab. 2 and Tab. 7). It can also provide part–whole relationships. We identify a part when at least 90% of its area lies inside another mask at least twice its size.

Correct parts (%) / objects with ≥2 parts (%)ScanNet (53)3RScan (46)SceneNN (10)
Ours, three levels71.7 / 23.768.4 / 15.393.8 / 28.2
Mask3D, raw proposals67.1 / 6.157.7 / 6.759.8 / 8.7

A part is considered correct when it lies at least 90% inside a matched annotated object (IoU > 0.5). The second metric measures the proportion of recovered annotated objects containing at least two distinct parts (mutual IoU < 0.7).

Mask3D produces 2–3× more proposals per scene (518–645 versus 173–246) and is trained on ScanNet annotations. Since these datasets lack part-level annotations, the first metric measures containment consistency, not semantic part accuracy.

The resulting hierarchy has higher containment consistency and identifies part decompositions for more objects. This may be useful for applications that need both object- and part-level regions (e.g., robotics), rather than a single benchmark-specific partition.

The authors’ additional experiments with unified hyperparameters, matched SAM checkpoints, and controlled input-view comparisons substantially alleviate my concerns about the experimental setup and the sources of the performance gains. In particular, although the additional analysis of rendering appearance does not substitute for a fair comparison of the final pipeline outputs, it strengthens the empirical support for the design choices and makes the overall rationale more convincing. I retain some reservations about comparing multi-granularity proposals with single-partition outputs. Nevertheless, I recognize the contribution of the proposed training-free, multi-granularity instance proposal method and find that the response sufficiently addresses my main concerns. I therefore raise my rating from 2 (Weak Reject) to 3 (Weak Accept).

Reviewer_833nOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a zero-shot method for instance segmentation in 3D scenes. Existing approaches typically capture images from multiple viewpoints (a recorded sequence) and apply off-the-shelf foundation models, such as SAM, to obtain 2D instance masks. These potentially noisy masks are then lifted into 3D space and aggregated to obtain consistent 3D instance segmentation. In contrast to applying a 2D segmenter directly to the recorded image sequence, this paper proposes segmenting rendered views of a 3D mesh obtained through cutting planes. The main advantage of this approach is its improved robustness to occlusions, as slicing can expose regions that are otherwise hidden in the original views. The 2D segmentation masks are lifted onto the 3D mesh using a per-pixel triangle-identity buffer generated by the renderer. Finally, consistent 3D instance segmentation labels are obtained through a multi-view instance consensus step, where an affinity graph is constructed based on the overlap between lifted masks, followed by clustering to identify coherent 3D instances.

Critical strengths
  • The idea of using planar slices to address occlusions in 3D scene instance segmentation is interesting. However, the paper should cite the closely related work "Slice3D: Multi-Slice, Occlusion-Revealing, Single View 3D Reconstruction" (CVPR 2024), which employs a similar slicing-based approach to address occlusions in 3D reconstruction. The underlying motivation for using slices appears to be similar in both methods.

  • The proposed method achieves substantial improvements over previous baselines, as demonstrated in Table 1, consistently outperforming existing training-free methods.

Critical weaknesses
  • Plane Sampling: How are the cutting planes sampled? The method appears to sample a fixed number of KK planes. However, the performance could be sensitive to the locations at which these planes are sampled. An ablation study analyzing the sensitivity of the method to plane sampling locations would help provide a deeper understanding of its robustness.

  • Clarity of the Method Section: The method section would benefit from a rewrite with clearer explanations and simplified, consistent notation. Currently, new notations are introduced throughout the section without sufficient explanation, making the methodology difficult to follow. For example, in the cross-cut stitching step of Sec. 3.2, the motivation and formulation of Eq. 2 are not immediately clear. Providing a more intuitive explanation of this step, rather than relying primarily on mathematical equations, would significantly improve the readability and understanding of the proposed method.

Questions for authors
  • Although I find the idea of using slices for 3D instance segmentation interesting, I have concerns regarding the clarity of the paper's writing. The authors should improve the presentation of the method and provide a concise, well-organized summary of the notations used, possibly in the form of a table, to make the methodology easier to follow and understand.

Currently I am leaning towards rejection, particularly because of concerns regarding the presentation of the method. However, I want to see the rebuttal from the authors and feedback from my fellow reviewers before making a final decision.

3 comments

Thank you for the review. We address your concerns below.

Q1 / W2: Presentation of the method. We agree that the method section can be clearer. We will upload a revised PDF with simplified notation, definitions at first use, and a more intuitive explanation of cross-cut stitching.

W2: Intuition for cross-cut stitching (Eq. 2). Each plane is rendered from both sides. An object intersected by the plane may therefore appear as two partial masks, one from each side. The triangles intersected by the plane are visible in both renderings, providing a direct correspondence between these masks. Eq. 2 joins masks from opposite sides when they share sufficiently many cut triangles, both absolutely and relative to their sizes. Fig. 2a illustrates this process: the mesh is cut (A1), both sides are rendered (A2) and segmented (A3), and the masks are lifted to mesh triangles (A4), where they are joined through the shared cut triangles.

W1: Plane sampling and sensitivity. Plane sampling is described in Secs. 3.1 and A.1. Of the K = 40 planes, two are horizontal bird's-eye views. The remaining 38 have normals sampled uniformly on the sphere and offsets sampled uniformly within ±0.15 times the bounding-box diagonal from the mesh centroid.

Sensitivity is already evaluated in Sec. C.1 (Tab. 5). Across three independent plane draws, AR@500 is 53.4 ± 0.7 on ScanNet (50 scenes) and 41.5 ± 0.6 on 3RScan. Although the individual masks differ significantly (mean best-match IoU 0.5–0.7), the overall performance is stable.

We additionally compared alternative sampling schemes, keeping K = 40 and all other settings fixed (50 ScanNet validation scenes, one seed per alternative):

Plane samplingAR@500
Uniform normals (ours, 3 draws)53.4 ± 0.7
Fibonacci normals, stratified offsets54.4
Grid-centred offsets (3×3×3)52.7
Manhattan (room axes only)39.4

The first three schemes perform similarly. Restricting cuts to the room axes reduces AR@500 by 14 points, mostly through fragmentation of medium and large objects. AR25 drops by only 3.0 points, suggesting that objects are still found but segmented less accurately.

Removing any two of the 40 planes changes AR@500 by at most 0.4 points on ScanNet and 3RScan. Thus, no individual plane is critical. The method is robust to plane sampling, provided the sampled directions are sufficiently diverse.

Slice3D. Thank you for pointing out this work. We will cite and discuss it. Slice3D uses slices to reveal occluded geometry for single-view 3D reconstruction. In contrast, we segment an existing reconstructed mesh using a frozen 2D segmenter. Every resulting mask maps directly to the same surface, avoiding correspondence estimation between recorded views. We share the motivation of revealing otherwise occluded geometry, but address a different task with a different pipeline.

Dear Reviewer,

Thank you again for your review. Apart from the revised methodology section, do you have any remaining concerns? We would be happy to provide further details.

Best regards, The Authors

I thank the authors for the rebuttal. I see the points addressed in the revised PDF. I do not have any further concerns and hence I am willing to increase my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revised PDF

We thank the reviewers for their reviews and feedback throughout the rebuttal, which have helped us improve the submission. We have uploaded a revised PDF; here we summarize the main changes:

  • One configuration for all datasets (FkPK). Conclusions are unchanged. Results marked with a red † still use the submission configuration (due to time constraints and GPU availability) and will be updated.
  • Method, Secs. 3, 4 (833n). Added a notation table, colour-coded and de-duplicated symbols, and more explanations.
  • Related work (833n). Slice3D is cited and discussed.
  • Cutting planes, new Tab. 4 (Xf9X, FkPK). With the pipeline fixed, compares cutting-plane views with RGB frames, renders from the frame cameras, and the plane cameras withoukeept the cut (matched and whole-mesh extents). Per-level results are in Tabs. 10 and 11.
  • Recognition input, new Tab. 5 (FkPK). The recognition head on plane renders, re-textured renders and RGB frames.
  • MaskProbe readout, Tab. 14 (Xf9X). Added the uniform-weight control. We removed the claims about attention weights and surrounding context, and now present MaskProbe as an efficient region readout from shared full-frame features.
  • Labelling cost, Tab. 17 (Xf9X). Re-timed with batching.
  • New appendix ablations, App. C.1. Plane sampling schemes and removal of individual planes (Tab. 8; 833n), SAM 1 and SAM 3 in place of SAM 2.1 (Tab. 9; FkPK), and part-whole structure across levels (Tab. 12; FkPK).

Replica is not included yet: not all methods and settings have been run on it, and we prefer to add it as a complete comparison instead of tables with missing cells.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.