- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents TOMOS3D: it samples cutting planes of a textured reconstructed mesh, renders the two sides of each plane separately, generates masks with a frozen SAM 2, and then obtains 3D instance proposals at three granularities through per-pixel triangle correspondence, a surface-overlap graph, and stability clustering. In the semantic stage, MASKPROBE projects the proposals onto the original photographs and restricts the patches that take part in the readout only in the final attention pooling of the frozen SigLIP2, so that multiple instances share one image-encoder pass.
Critical strengths
-
Cutting-plane rendering offers a proposal-generation path worth studying, and the cross-dataset results are strong. Under the paper's unified AR@500 protocol, TOMOS3D clearly above the training-free baselines evaluated in Table 1.
-
The recognition experiments contain a well-controlled comparison. MASKPROBE and OM3D+ use the same 3D proposals, SigLIP2 encoder, selected photographs, and text prompts, so the differences in Table 2 reflect the role of the readout fairly directly. The shared photograph encoding also yields a measured speed-up of 1.47–2.70× in the reported recognition stage.
Critical weaknesses
-
MaskProbe's mechanistic contribution is modest, and its "context-aware" explanation is not supported by the experiments. MaskProbe improves average mAR@500 by roughly 1.7 points over OM3D+ (41.4 vs. 39.7), a limited gain that is also unstable in some settings: On SceneNN with the photo prompt, SigLIP2 multi-crop is actually higher than MaskProbe (63.9 vs. 62.3); Under the GT mask setting on SceneNN, the two are on par (80.5 vs. 80.5). The paper interprets MaskProbe as a context-aware masked attention pooling, i.e., reducing background interference by restricting the readout region. Yet the current experiments do not directly support this mechanistic explanation. The SigLIP2 image encoder itself has global self-attention, and the mask acts only on the attention logits of the final pooling; all earlier tokens have already mixed in context outside the mask. The results of Fig. 11 further weaken this explanation: MASKPROBE-crop and full-frame MASKPROBE achieve almost identical mAR@500 (64.2 vs. 64.1). This indicates that, given the same region, whether the full-image context is retained has little effect on the result.
-
The existing experiments change several factors at once: cutting-plane rendering, SAM 2.1, the three-granularity output, surface-overlap clustering on the mesh, and stability clustering. Ablations such as the number of planes cannot substitute for testing "the contribution of cutting-plane rendering itself." More notably, there is a large gap between the single-granularity results and the union of the three granularities: On ScanNet, the best single-granularity result is only 34.6 AR@500; whereas SAI3D achieves 43.1 AR@500; TOMOS3D reaches 51.5 AR@500 only through the union of the three granularities. That is, on ScanNet, single-granularity TOMOS3D is below SAI3D, and only after merging the three granularities does it gain a clear lead. The current experiments cannot answer the following: if the same SAM 2.1, three-granularity output, mesh aggregator, and clustering pipeline are kept, and only the 80 cutting-plane renderings are replaced with the same number of original recorded frames or uncut ordinary mesh renderings, how much would the result change? The paper's statement that "cutting-plane rendering is the main innovative contribution" still requires more direct experimental support.
Questions for authors
-
Does MaskProbe's improvement genuinely come from masked attention pooling, rather than merely from the region-feature readout itself? Under the same protocol as Table 2 (same Tomos3D proposals, same selected views, same projected masks, photo prompt, 20-template ensemble, and frozen SigLIP2), please add a simple region-feature readout as a control, for example using average pooling or max pooling over the patches inside the projected mask. Please compare: MaskProbe; region average pooling; MASKPROBE-crop. If possible, further fix the target region and only change the visible background around it, and observe whether the paired recognition results change. This would help to judge whether the "context-aware masked attention pooling" emphasized by the paper is indeed the source of the performance gain, or whether MaskProbe's benefit mainly comes from a more sensible region readout or from avoiding the information dilution of multi-crop.
-
Can the contribution of cutting-plane rendering be separated from the three-granularity output, SAM 2.1, and the aggregation pipeline? On a fixed subset of 10–20 ScanNet scenes, keep SAM 2.1, the three-granularity output, the mesh aggregator, and HDBSCAN stability clustering unchanged, and replace only the 80 cutting-plane renderings with (a) 80 uniformly sampled original RGB-D recorded frames and (b) 80 uncut ordinary mesh renderings (please state the camera and rendering settings). Report AR@500, AR50, AR90, and proposals per scene in one table. At the same time, please add a per-item comparison between the union of the three granularities and each single-granularity result, clarifying whether the final performance mainly comes from the cutting-plane views or from the expansion of the candidate set after retaining multiple granularities.
5 comments
The current response largely reuses text addressed to another reviewer, omits my first question entirely, and lacks an updated manuscript. My Weak Accept rating was contingent on clarifying the mechanistic attribution of your two contributions; unless these points are directly addressed and reflected in a revised PDF before the discussion period closes, I will lower my score.
First, regarding MaskProbe, please provide the region average/max pooling control requested in my initial review (which can be run quickly using your existing projected masks and cached SigLIP2 features) and clarify how you will revise the unsupported full-image context claims in light of Fig. 11.
Second, your 50-scene ScanNet comparison (note that the gain is 11.9–15.9 points, not 15.9–19.9) does not yet isolate the contribution of cutting-plane rendering. Please confirm whether all three inputs used the same 80-view budget, and report the missing metrics from my initial review (AR50, AR90, proposals per scene) alongside a breakdown of each single granularity () versus their three-level union. Furthermore, to decouple half-space clipping from global orthographic views, please compare cutting-plane renders against uncut orthographic renders under the same 80 camera poses and extents across both individual levels and their union.
Finally, please upload a revised PDF incorporating these ablations and your promised manuscript updates so they can be verified before the final decision.
Apology
Dear reviewer,
We sincerely apologize for the confusion. During the final formatting of our rebuttal, parts of the responses to different reviewers were accidentally mixed up. We have now corrected our original response, which addresses both of your questions with the requested experiments. We will also incorporate these results, clarify the contribution of MaskProbe, and revise the method presentation in the updated manuscript before the discussion closes.
We would appreciate it if you could take another look at our corrected response.
Corrected response
Q1 / W1: MaskProbe's efficiency and source of its gain.
Despite MaskProbe's modest accuracy gains, its efficiency advantage remains substantial.
Efficiency. MaskProbe encodes each frame once and pools features for all visible instances; OpenMask3D encodes three crops per instance and view. This reduces encoder inputs 6–12× (3RScan: 2606→413; SceneNN: 1858→244). We repeated timing with a parallel CPU path (A100, bf16, batched OpenMask3D crops, identical encoder/masks/views, 10 ScanNet scenes):
MaskProbe is 3.2× faster end to end. We will update Tab. 9.
Source of the gain. With GT masks, frozen SigLIP2 and the same 20-template ensemble (mAR@500):
Mean pooling is slightly better than MaskProbe on average. Thus, the gain over crops mainly comes from region-restricted readout, not learned attention weights. Gains are smaller on SceneNN, where objects occupy more of the image: median GT area is 1.47 m² versus 0.46/0.52 on ScanNet/3RScan, and masks cover 66/110 tokens (finest/coarsest), versus 23/45 and 16/28 of 729. SceneNN has an 18-class, mostly furniture vocabulary; by object count, MaskProbe recovers 101/123 vs. 98 for multi-crop. On Replica, MaskProbe improves over multi-crop by +5.4 mAR@500 (21.5 vs. 16.1, identical proposals).
We agree that Fig. 11 does not establish a benefit from retaining full-image context: earlier encoder layers already mix context, and MaskProbe and MaskProbe-crop perform similarly. We will remove this claim. Our contribution is efficient region-restricted readout from shared image features.
Q2 / W2: Cutting-plane rendering vs. three granularities.
We ran the requested controlled ablation on 50 ScanNet val scenes. All variants use 80 views, identical SAM 2.1 settings, three levels, mesh aggregation and HDBSCAN; only the views change. We also compare cuts against uncut renders at the same plane cameras:
Plane renders are orthographic, 1024², vertex-coloured. Uncut views use the same 40 planes × 2 directions, showing the full mesh without clipping. Recorded frames use original poses/intrinsics (pinhole, 1296×968); ordinary mesh renders use those cameras. Clustering was tuned on cut views, potentially disadvantaging alternatives.
At the same cameras, cutting adds 5.5 AR@500. Versus recorded frames, cut views gain 7.9 with one level and 11.9 with three, despite fewer proposals (306 vs. 391). Per level, planes beat recorded frames at the finest (35.0 vs. 17.0) and coarsest (37.4 vs. 29.5), but trail at the middle (16.5 vs. 27.0). The three-level union adds 12.6–16.6 points for every view source. These results separate the contribution of cutting-plane rendering from retaining multiple granularities. We can also provide all individual-level results for the uncut views if needed.
We retain all three levels because target granularity is unknown in zero-shot segmentation. The segmentation overhead is small: rendering and SAM passes are shared, and SAM takes ~90% of the 266 s per ScanNet scene on RTX 4090 (App. D.1). Extra levels increase semantic labelling time.
In summary, cutting-plane views and multiple granularities contribute independently. We will incorporate these ablations and the corrected MaskProbe interpretation in the revised manuscript.