- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes a method to extend feed-forward 3D reconstruction models like VGGT-Omega with the ability to perform instance and open-world class segmentation. The central contribution is a new approach that allows fine-tuning the pre-trained feed-forward reconstructor to achieve multi-view consistent segmentation without having to label multi-view data with corresponding ground-truth segments. Given a single (monocular) image, several "pseudo-views" are generated from it by taking random crops, and by mapping the instance segmentation labels and general-purpose features such as DINO to the crops. In this way, the crops are trivially labelled in a manner which is mutually consistent. The authors show that such data can be used to fine-tune the feed-forward reconstruction model to perform consistent segmentation with good generalisation performance to several standard benchmark datasets.
Critical strengths
-
The paper proposes a relatively simple way of extending strong 3D reconstructor to efficient and performant multi-view, and ultimately 3D, segmenters for both of object instance segmentation and open-world class segmentation. This has practical value, and it also demonstrates the ability of the recent class of "3D foundation models" to solve tasks beyond recovering geometry.
-
The paper (and the supplement) provide a detailed and sound technical description of the approach.
-
The paper contains experiments on a large number of datasets, assessing the proposed model against the state of the art in a number of different tasks. The performance achieved is very good, especially considering that the learned model ultimately operates in a feed-forward manner, and his thus quite efficient, particular when compared to prior alternatives that use test-time optimisation such as NeRF and 3DGS-based solutions.
-
The authors strive to give a honest representation of the contribution and experiments, noting various limitations when appropriate. This includes highlighting limitations in the fairness of certain results, particularly when apples-to-apples comparisons to prior works may be impossible.
Critical weaknesses
-
The paper cites VGGT-Segmentor as the most closely related approach (line 138). The related works clarify that it operates under somewhat different conditions (namely two-views instead of many-views). While it is not said explicitly, this could make a direct comparison not entirely trivial. Having said that, it would be good to perform this assessment as VGGT-Segmentor solves essentially the same problem, and also uses a very similar approach to training their model.
-
The paper misses citations to important background work. For instance, the use of Gram matrix for supervision (line 308) seems to be very similar to the Gram loss in DINOv3, but this is not directly acknowledged. Likewise, Eq. (3) appears to be FiLM, but this is also not credited. It is likely that most readers would implicitly impute these formulations to the correct prior works; even so, the paper should make the link explicit.
-
The authors note that many ablation are conducted on a short training schedule. For example, in Appendix C the ablated model for promotable segmentation model has mIoU 55.19 (Table 8), whereas when fully trained this model has mIoU 70.26 (Section C). In practice, the performance gap is non-negligible, and it is possible that certain conclusions about the different contributions of the method's component may not hold for a fully-trained model, or may hold in a far less significant manner.
-
The coverage map is not specified very precisely. Is this just a binary mask? What happens when you have to encode a single pixel (namely, when the prompt consists of a single point)?
Questions for authors
-
Would a direct comparison to VGGT-Segmentor be possible, at least with some work? If not, can the authors explain explicitly why not?
-
Would any conclusion change significantly if ablations were conducted on a longer training schedule, closer to full model training?
-
How is defined?
2 comments
Dear Reviewer wi4r, as the discussion deadline is very close, we kindly follow up one last time in case you have any final thoughts on our revisions and added experiments. Thank you again for your valuable feedback.
Response to Reviewer wi4r: VGGT-Segmentor comparison, citations, long-schedule ablations, and coverage-map definition
Response to Reviewer wi4r (1/4) — W1 + Q1: direct comparison with VGGT-Segmentor
We thank the reviewer for the positive and detailed review; all new results will be added to the revision.
Yes, a direct comparison is possible, and we have now run it. We evaluated the single-image VGGT-Segmentor (VGGT-S) on ScanNet++ with the same prompt (one GT source mask) and metric (mIoU over visible target views). As VGGT-S is pairwise, we extend it to N views by graph association: a chain over adjacent frames for ordered inputs (N−1 passes per object) and all pairs for shuffled inputs (N(N−1)/2). Feature Puzzles (FP) processes all N views in one pass.
† Estimated as #passes × 161.4 ms (per-pair time reported by VGGT-S, RTX 4090).
FP is better at every view count, and the gap grows with N: +3.4 / +12.3 / +18.8 mIoU (ordered) and +3.4 / +14.5 / +21.1 (shuffled) at 2 / 6 / 12 views. FP is also insensitive to view order. It needs one pass for all objects, while VGGT-S needs up to 66 passes per object at 12 shuffled views. A single VGGT-S pair is cheaper at N=2, but the pairwise cost overtakes FP as views are added.
Response to Reviewer wi4r (2/4) — W2: missing citations (Gram loss, FiLM)
Thank you for catching these attributions. Our Gram-anchoring term follows the Gram loss introduced in DINOv3, and Eq. (3) is a FiLM-style feature-wise modulation. We will cite DINOv3 at the Gram objective and FiLM (Perez et al., 2018) at Eq. (3), and describe both as established components rather than our contributions. We will also correct Eq. (3) to the residual form used in our implementation, F′=F⊙(1+γ(C))+β(C) for each view and instance, applied only to view–instance pairs that carry a prompt.
Response to Reviewer wi4r (3/4) — W3 + Q2: ablations under a longer training schedule
We agree that this needed to be verified. We retrained the full model and the three main ablations for 10,000 steps (4× the original 2,500) under otherwise identical settings, evaluated as in Table 8 (SceneNN-MV, GT prompts, 6 views, visible-view mIoU).
The full model reaches 70.26 mIoU, matching the fully trained model reported in the paper. No conclusion changes, and no gap shrinks. Removing the adapters costs 19.3 mIoU (17.1 in the short schedule), removing cross-view attention 6.8 (4.1), and removing multi-view prompting 3.5 (3.0). The ranking of components is unchanged. We will replace Table 8 with the long-schedule results and keep the short-schedule numbers in the appendix.
Response to Reviewer wi4r (4/4) — W4 + Q3: definition of the coverage map Cn,m
The coverage map C of a view–instance pair is a soft map with values in [0, 1] on the backbone patch grid (H/16×W/16), not a pixel-level binary mask. For a mask prompt, each cell stores the fraction of its 16×16 patch covered by the mask (average pooling). For a box, a cell is 1 if its patch centre lies inside the box. For positive points, the cell containing each point is set to 1. Several prompts are combined by a cell-wise maximum. Negative points do not contribute to C.
A single point therefore never vanishes: it is rasterised directly onto the patch that contains it rather than drawn at pixel level and then down-sampled. In addition, every point (and both box corners) is encoded as a SAM-style sparse token from its exact pixel coordinates (random Fourier positional encoding plus a positive/negative type embedding). These tokens join the object queries in the two-way attention, which preserves sub-patch location and negative points.
C is used twice: it modulates the tokens through FiLM (see our answer to W2), and it weights the aggregation of patch tokens that initialises the object token. We will add this definition, with an equation and a single-point example, to Appendix B.2.
We hope these new results and clarifications address the reviewer's concerns, and we are happy to provide further details.