acceptodds
Under review as a conference paper at ICLR 2027

Feature Puzzles: Single-Image Supervision for Multi-View 3D-Aware Segmentation

Abstract

Humans can easily recognize the same objects across different viewpoints of a 3D scene, whereas teaching models such multi-view-consistent perception typically requires videos or calibrated multi-view captures, which are considerably more expensive and less diverse than large-scale monocular data. In this paper, we introduce Feature Puzzles, a novel framework for learning multi-view-consistent segmentation using only single-image semantic segmentation supervision. Specifically, we propose a simple yet effective pipeline that constructs multi-view-consistent segmentation pseudo-labels through image cropping and geometric transformations. We further introduce a lightweight, plug-and-play network that leverages the multi-view-consistent image features learned by existing pretrained 3D foundation models and the generated pseudo-labels to perform multi-view-consistent segmentation. Importantly, our flexible framework supports various types of prompts provided from one or multiple views and can be readily extended to open-vocabulary 3D segmentation. Under this single-image supervision, our feed-forward model achieves the best mIoU on NVOS, remains competitive on SPIn-NeRF and ScanNet++ against baselines using per-scene optimization or in-domain multi-view training, and demonstrates zero-shot transfer to open-vocabulary 3D segmentation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Feed forward 3D reconstruction, Spatial understanding, Promptable multi-view segmentation, Spatial tracking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wi4rOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method to extend feed-forward 3D reconstruction models like VGGT-Omega with the ability to perform instance and open-world class segmentation. The central contribution is a new approach that allows fine-tuning the pre-trained feed-forward reconstructor to achieve multi-view consistent segmentation without having to label multi-view data with corresponding ground-truth segments. Given a single (monocular) image, several "pseudo-views" are generated from it by taking random crops, and by mapping the instance segmentation labels and general-purpose features such as DINO to the crops. In this way, the crops are trivially labelled in a manner which is mutually consistent. The authors show that such data can be used to fine-tune the feed-forward reconstruction model to perform consistent segmentation with good generalisation performance to several standard benchmark datasets.

Critical strengths
  • The paper proposes a relatively simple way of extending strong 3D reconstructor to efficient and performant multi-view, and ultimately 3D, segmenters for both of object instance segmentation and open-world class segmentation. This has practical value, and it also demonstrates the ability of the recent class of "3D foundation models" to solve tasks beyond recovering geometry.

  • The paper (and the supplement) provide a detailed and sound technical description of the approach.

  • The paper contains experiments on a large number of datasets, assessing the proposed model against the state of the art in a number of different tasks. The performance achieved is very good, especially considering that the learned model ultimately operates in a feed-forward manner, and his thus quite efficient, particular when compared to prior alternatives that use test-time optimisation such as NeRF and 3DGS-based solutions.

  • The authors strive to give a honest representation of the contribution and experiments, noting various limitations when appropriate. This includes highlighting limitations in the fairness of certain results, particularly when apples-to-apples comparisons to prior works may be impossible.

Critical weaknesses
  • The paper cites VGGT-Segmentor as the most closely related approach (line 138). The related works clarify that it operates under somewhat different conditions (namely two-views instead of many-views). While it is not said explicitly, this could make a direct comparison not entirely trivial. Having said that, it would be good to perform this assessment as VGGT-Segmentor solves essentially the same problem, and also uses a very similar approach to training their model.

  • The paper misses citations to important background work. For instance, the use of Gram matrix for supervision (line 308) seems to be very similar to the Gram loss in DINOv3, but this is not directly acknowledged. Likewise, Eq. (3) appears to be FiLM, but this is also not credited. It is likely that most readers would implicitly impute these formulations to the correct prior works; even so, the paper should make the link explicit.

  • The authors note that many ablation are conducted on a short training schedule. For example, in Appendix C the ablated model for promotable segmentation model has mIoU 55.19 (Table 8), whereas when fully trained this model has mIoU 70.26 (Section C). In practice, the performance gap is non-negligible, and it is possible that certain conclusions about the different contributions of the method's component may not hold for a fully-trained model, or may hold in a far less significant manner.

  • The coverage map Cm,nC_{m,n} is not specified very precisely. Is this just a binary mask? What happens when you have to encode a single pixel (namely, when the prompt consists of a single point)?

Questions for authors
  • Would a direct comparison to VGGT-Segmentor be possible, at least with some work? If not, can the authors explain explicitly why not?

  • Would any conclusion change significantly if ablations were conducted on a longer training schedule, closer to full model training?

  • How is Cm,nC_{m,n} defined?

2 comments
AuthorsOpenReview ↗

Response to Reviewer wi4r: VGGT-Segmentor comparison, citations, long-schedule ablations, and coverage-map definition

Response to Reviewer wi4r (1/4) — W1 + Q1: direct comparison with VGGT-Segmentor

We thank the reviewer for the positive and detailed review; all new results will be added to the revision.

Yes, a direct comparison is possible, and we have now run it. We evaluated the single-image VGGT-Segmentor (VGGT-S) on ScanNet++ with the same prompt (one GT source mask) and metric (mIoU over visible target views). As VGGT-S is pairwise, we extend it to NN views by graph association: a chain over adjacent frames for ordered inputs (N−1N-1 passes per object) and all pairs for shuffled inputs (N(N−1)/2N(N-1)/2). Feature Puzzles (FP) processes all NN views in one pass.

InputMethod#Views#Passes / obj.mIoU ↑Latency (s) ↓
OrderedVGGT-S2 / 6 / 121 / 5 / 1169.31 / 58.57 / 50.110.16 / 0.81 / 1.78 †
OrderedFP (ours)2 / 6 / 121 / 1 / 172.67 / 70.89 / 68.930.73 / 0.92 / 1.32
ShuffledVGGT-S2 / 6 / 121 / 15 / 6669.31 / 56.32 / 47.840.16 / 2.42 / 10.65 †
ShuffledFP (ours)2 / 6 / 121 / 1 / 172.67 / 70.86 / 68.960.73 / 0.92 / 1.32

† Estimated as #passes × 161.4 ms (per-pair time reported by VGGT-S, RTX 4090).

FP is better at every view count, and the gap grows with NN: +3.4 / +12.3 / +18.8 mIoU (ordered) and +3.4 / +14.5 / +21.1 (shuffled) at 2 / 6 / 12 views. FP is also insensitive to view order. It needs one pass for all objects, while VGGT-S needs up to 66 passes per object at 12 shuffled views. A single VGGT-S pair is cheaper at N=2N=2, but the pairwise cost overtakes FP as views are added.

Response to Reviewer wi4r (2/4) — W2: missing citations (Gram loss, FiLM)

Thank you for catching these attributions. Our Gram-anchoring term follows the Gram loss introduced in DINOv3, and Eq. (3) is a FiLM-style feature-wise modulation. We will cite DINOv3 at the Gram objective and FiLM (Perez et al., 2018) at Eq. (3), and describe both as established components rather than our contributions. We will also correct Eq. (3) to the residual form used in our implementation, F′=F⊙(1+γ(C))+β(C)\mathbf{F}' = \mathbf{F} \odot (1+\gamma(\mathbf{C})) + \beta(\mathbf{C}) for each view and instance, applied only to view–instance pairs that carry a prompt.

Response to Reviewer wi4r (3/4) — W3 + Q2: ablations under a longer training schedule

We agree that this needed to be verified. We retrained the full model and the three main ablations for 10,000 steps (4× the original 2,500) under otherwise identical settings, evaluated as in Table 8 (SceneNN-MV, GT prompts, 6 views, visible-view mIoU).

VariantShort (2.5k) mIoUΔ vs fullLong (10k) mIoUΔ vs full
Full model55.19—70.26—
w/o cross-view attention51.12−4.0763.44−6.82
w/o multi-view prompting52.17−3.0266.73−3.53
w/o adapters ("w/o LoRA")38.11−17.0850.97−19.29

The full model reaches 70.26 mIoU, matching the fully trained model reported in the paper. No conclusion changes, and no gap shrinks. Removing the adapters costs 19.3 mIoU (17.1 in the short schedule), removing cross-view attention 6.8 (4.1), and removing multi-view prompting 3.5 (3.0). The ranking of components is unchanged. We will replace Table 8 with the long-schedule results and keep the short-schedule numbers in the appendix.

Response to Reviewer wi4r (4/4) — W4 + Q3: definition of the coverage map Cn,mC_{n,m}

The coverage map C\mathbf{C} of a view–instance pair is a soft map with values in [0, 1] on the backbone patch grid (H/16×W/16H/16\times W/16), not a pixel-level binary mask. For a mask prompt, each cell stores the fraction of its 16×1616\times16 patch covered by the mask (average pooling). For a box, a cell is 1 if its patch centre lies inside the box. For positive points, the cell containing each point is set to 1. Several prompts are combined by a cell-wise maximum. Negative points do not contribute to C\mathbf{C}.

A single point therefore never vanishes: it is rasterised directly onto the patch that contains it rather than drawn at pixel level and then down-sampled. In addition, every point (and both box corners) is encoded as a SAM-style sparse token from its exact pixel coordinates (random Fourier positional encoding plus a positive/negative type embedding). These tokens join the object queries in the two-way attention, which preserves sub-patch location and negative points.

C\mathbf{C} is used twice: it modulates the tokens through FiLM (see our answer to W2), and it weights the aggregation of patch tokens that initialises the object token. We will add this definition, with an equation and a single-point example, to Appendix B.2.

We hope these new results and clarifications address the reviewer's concerns, and we are happy to provide further details.

Dear Reviewer wi4r, as the discussion deadline is very close, we kindly follow up one last time in case you have any final thoughts on our revisions and added experiments. Thank you again for your valuable feedback.

Reviewer_CQQDOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Feature Puzzles studies semantic adaptation of pretrained multiview reconstruction models using supervision from single images. It generates overlapping crops and transfers masks or DINOv3 features from a common parent image. These aligned targets support promptable segmentation and dense feature prediction through task heads built on fused backbone features. Geometric priors are inherited from the pretrained backbone. Dense features are lifted into 3D and classified through text matching. Evaluation covers prompt propagation and instance tracking alongside 3D semantic segmentation. The latter uses ground truth geometry to assess feature quality. The authors report transfer to real camera views and examine different backbones. Additional training with semantic class labels is evaluated separately.

Critical strengths

Teacher features are extracted from the complete parent image and then mapped into each crop. All synthetic views therefore inherit a common semantic target. This avoids supervision conflicts caused by independent feature extraction and directly supports the motivation of learning consistent representations across views. It also provides a practical mechanism for reusing annotations from single images. The reported transfer to real camera views supports the usefulness of this supervision design.

Critical weaknesses
  1. Existing methods already adapt geometric backbones using supervision from single images as discussed in Section 2. The paper therefore needs to establish the added value of shared parent targets and joint multiview processing more clearly. The evaluation demonstrates the combined framework across several tasks. However, the central puzzle controls also change supervision relationships or view processing. This leaves the separate contributions of these designs insufficiently established and limits the assessment of the methodological advance.
  2. Figure 3 connects multiple task heads to a shared backbone. Appendix B.4 describes the respective task training setups. However, it remains unclear whether the final semantic branches share the same trained adaptation parameters or use separate task checkpoints. The distinction between a common architecture and one model supporting both tasks is consequential. This ambiguity limits understanding of the framework’s practical reuse across tasks.
  3. The role of depth-based 3D view generation in the final training configurations remains unclear. Section 3.2 describes it as optional. Appendix A.2 disables geometric augmentation for the feature branch. Appendix A.3 also describes the construction as including 3D augmentations. The relationship between these descriptions is not explicit. This leaves uncertainty about the source of viewpoint variation during training and the contribution attributable to affine crop supervision.
Questions for authors

None.

3 comments
AuthorsOpenReview ↗

Response to Reviewer CQQD: 2×2 ablation of shared-parent targets vs. joint processing, parameter sharing, and augmentation configuration

Response to Reviewer CQQD (1/3) — W1: separate contributions of shared-parent targets and joint processing

We thank the reviewer for the careful review; all new results will be added to the revision.

We agree that the No-Puzzles control changed several factors at once. We therefore ran a controlled 2×2 ablation on the open-vocabulary branch, keeping crops, data, losses (including the warp and Gram losses) and budget identical. We vary only (i) whether each view's teacher target is pulled back from one shared parent feature map or extracted independently from each crop, and (ii) whether the NN crops are processed jointly as one sequence or independently. ScanNet 10-class mIoU / mAcc:

Per-crop targets Φ(xn)\Phi(x_n)Shared-parent targets ℓ∘πn\ell\circ\pi_n
Independent views11.42 / 23.4813.20 / 25.31
Joint views9.09 / 18.0315.70 / 29.69 (full)

Both designs contribute, and they are complementary. Shared-parent targets are the larger factor: +6.6 mIoU / +11.7 mAcc under joint processing (+1.8 / +1.8 with independent views). Joint processing adds +2.5 / +4.4, but only with consistent targets. With per-crop targets it hurts (−2.3 / −5.5): overlapping views then receive conflicting targets, and cross-view attention and the warp loss pull the student toward incompatible optima, as predicted by the well-posedness argument in Appendix A.

We run the 2×2 on the feature branch because per-crop targets would remove cross-view instance identity in the promptable branch, making that task ill-defined. We will add this table and replace the No-Puzzles discussion.

Response to Reviewer CQQD (2/3) — W2: shared adaptation parameters vs. task checkpoints

Thank you for pointing out this ambiguity. The two tasks share the architecture and the frozen VGGT-Omega backbone, but each task has its own lightweight low-rank adapters, layer-fusion weights and head. The adapters act only on the cached multi-layer tokens (layers 4 / 11 / 17 / 23) and never modify the backbone. At inference, a single backbone forward pass produces the original depth and camera outputs and caches these tokens; each task then applies its own adapters and head to the shared tokens. Adding a task therefore costs one more set of lightweight adapters and a head, not another backbone pass. The two adapter/head sets are trained separately (Appendix B.4), not jointly, and can be loaded together on one frozen backbone and run in a single pass.

We acknowledge that several sentences in the submission describe this inaccurately: adapters "in late backbone blocks", "two-pass inference", and a single representation "which both heads consume". We will revise the Abstract, Sec. 3, Appendix B.1/B.4, Fig. 3 and the ablation naming ("w/o LoRA" → "w/o adapters") to state the parameter sharing, the per-task adapters and the single-pass inference explicitly.

Response to Reviewer CQQD (3/3) — W3: role of depth-based 3D augmentation in the final configuration

We apologize for the inconsistent descriptions. The configuration used in all reported experiments is:

BranchAffine shared-parent cropsDepth-based 6-DoF aug. (MoGe-2)PhotometricPer-crop 2D geometric / occlusion
PromptableYesYesYesYes (joint on image, target, prompt)
Open-vocabularyYesNoYesNo

In the promptable branch, depth-based views are rendered from MoGe-2 parent depth, and their labels are pulled back through the same depth projection. In the open-vocabulary branch, depth-based views and per-crop geometric augmentation are disabled to keep the student–teacher correspondence pixel-exact.

Viewpoint variation therefore comes from the crop scale range and overlap curriculum in both branches, plus simulated 6-DoF motion in the promptable branch. "Exact by construction" applies to the affine views. For depth-augmented views, the rendered image and its labels come from the same depth estimate and projection, so they stay mutually aligned, but their correspondence depends on the estimated depth. We will state the per-branch configuration in Sec. 3.2 with this table, restrict the exactness claim and the "in-plane crop-and-resize" sentence in Appendix A.3 to the affine construction, and align Appendix A.2 accordingly.

We hope the new ablation and these clarifications address the reviewer's concerns, and we are happy to provide further details.

The additional ablation and implementation clarifications largely address my concerns, and I maintain my score of 3 (Weak acceptance).

Thank you for your constructive feedback and positive assessment. We sincerely appreciate your valuable suggestions, which have helped improve our work.

Reviewer_m16kOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Feature Puzzles uses overlapping crops from a shared parent image to build consistent mask and feature supervision. It adapts a pretrained VGGT-Omega backbone for joint promptable segmentation and open-vocabulary 3D features. Task-specific training uses single-image data and the geometric prior comes from the pretrained backbone.

Critical strengths

Adapting a geometric backbone with single-image semantic supervision is a vaule direction. It allows existing 2D annotations to be reused, reducing the need for real multi-view or video annotations during task-specific training. The results on real camera views support the idea.

Critical weaknesses
  1. The comparison with the closest method is still incomplete. VGGT-Segmentor also has a single-image training variant. The paper discusses the architectural differences, but gives no matched quantitative comparison.
  2. The added value of the open-vocabulary features is less clear. The model distills a strong DINOv3 teacher, but there is no direct teacher-feature baseline under the same 3D lifting and fusion setup. What the learned multi-view representation adds in semantic quality and consistency?
Questions for authors
  1. It would be useful to compare directly with the single-image-trained VGGT-Segmentor to clarify the benefit of joint processing.
  2. Add a baseline that lifts frozen DINOv3 features directly into 3D, using the same Talk2DINO mapping, geometry, and voxel fusion. Compare semantic quality and cross-view consistency to show what the learned representation adds.
  3. The segmentation accuracy in Table 3 is measured using ground-truth geometry, while the runtime test includes the model's own reconstruction. Since the paper emphasizes feed-forward 3D semantic understanding, it should report segmentation results using its own predicted depth and poses on at least one representative benchmark.
2 comments
AuthorsOpenReview ↗

Response to Reviewer m16k: VGGT-Segmentor comparison, frozen DINOv3 baseline, and end-to-end results with predicted geometry

Response to Reviewer m16k (1/3) — W1 + Q1: comparison with single-image VGGT-Segmentor

Thank you for the suggestions; all new results will be added to the revision.

We evaluated the single-image VGGT-Segmentor (VGGT-S) on ScanNet++ with the same prompt (one GT source mask) and metric (mIoU over visible target views, source excluded). As VGGT-S is pairwise, we extend it to NN views by graph association: a chain over adjacent frames for ordered inputs (N−1N-1 passes per object) and all pairs for shuffled inputs (N(N−1)/2N(N-1)/2). Feature Puzzles (FP) processes all NN views in one pass.

InputMethod#Views#PassesmIoU ↑Latency (s) ↓
OrderedVGGT-S2 / 6 / 121 / 5 / 1169.31 / 58.57 / 50.110.16 / 0.81 / 1.78 †
OrderedFP (ours)2 / 6 / 121 / 1 / 172.67 / 70.89 / 68.930.73 / 0.92 / 1.32
ShuffledVGGT-S2 / 6 / 121 / 15 / 6669.31 / 56.32 / 47.840.16 / 2.42 / 10.65 †
ShuffledFP (ours)2 / 6 / 121 / 1 / 172.67 / 70.86 / 68.960.73 / 0.92 / 1.32

† Estimated as #passes × 161.4 ms (per-pair time reported by VGGT-S, RTX 4090).

Accuracy. FP is better at every view count, and the gap grows with NN: +3.4 / +12.3 / +18.8 mIoU (ordered) and +3.4 / +14.5 / +21.1 (shuffled) at 2 / 6 / 12 views. VGGT-S degrades as errors accumulate along the chain, whereas FP loses only 3.7 mIoU from 2 to 12 views and is insensitive to view order (70.89 vs. 70.86 at 6 views).

Efficiency. VGGT-S needs up to 66 passes per object (12 shuffled views), multiplied again by the number of objects; FP needs one pass for all objects. A single VGGT-S pair is cheaper at N=2N=2, but the pairwise cost overtakes FP from about 6 shuffled or 12 ordered views (10.65 s est. vs. 1.32 s at 12 shuffled).

Thus (Q1), joint processing keeps accuracy stable as views are added or reordered, while pairwise propagation loses accuracy and adds cost with every view.

Response to Reviewer m16k (2/3) — W2 + Q2: frozen DINOv3 baseline

We lifted frozen DINOv3 ViT-B/16 features, extracted per frame, with exactly our pipeline: EmbodiedSplat protocol on ScanNet (11 scenes, stride 10), the same Talk2DINO text embeddings, 0.02 m voxel averaging and cosine-argmax classification. FP is our SA-1B-only student (no in-domain fine-tuning).

FeatureGeometry10-cls mIoU ↑15-cls mIoU ↑19-cls mIoU ↑
Frozen DINOv3 (per-frame, direct lift)GT40.8334.2430.58
FP (student)GT41.0935.8730.50
Frozen DINOv3 (per-frame, direct lift)Predicted38.1230.1428.01
FP (student)Predicted39.4532.3927.72

Semantic quality. With GT geometry the student is on par with its teacher (+0.26 / +1.63 / −0.08 mIoU on 10 / 15 / 19 classes). This is expected for a distilled model: our goal is to preserve DINOv3 semantics while making them multi-view consistent and usable end-to-end, not to exceed them.

What the student adds. Per-frame DINOv3 features of the same surface differ across views (occlusion, viewing angle, lighting), and averaging these discrepancies in 3D blurs boundaries. FP computes each view's features with all views in context through cross-view attention. The benefit shows when geometry is imperfect: with our predicted depth and poses, the student leads by +1.33 / +2.25 mIoU on 10 / 15 classes (−0.29 on 19) and degrades less from GT to predicted geometry (−1.64 / −3.48 vs. −2.71 / −4.10).

We will add this table and state the claim precisely: the student preserves the teacher's semantics, and its benefit is cross-view consistency and a single end-to-end RGB-only pipeline.

Response to Reviewer m16k (3/3) — Q3: results with predicted depth and poses

The GT-geometry evaluation in Table 3 was meant to isolate feature quality. We now also lift features with the depth and poses predicted in the same VGGT-Omega forward pass, aligned to the GT reconstruction with alignment protocol, all else identical.

Geometry10-cls mIoU ↑15-cls mIoU ↑19-cls mIoU ↑Latency, 363-frame scene (s) ↓
GT depth + poses41.0935.8730.50~23.1
Predicted (same forward pass)39.4532.3927.72~25.9
Δ−1.64−3.48−2.78+2.8

mIoU drops only moderately (−1.6 / −3.5 / −2.8), so the RGB-only pipeline remains usable without external reconstruction. Because depth and poses come from the same forward pass as the features, predicting geometry adds only ~2.8 s (~12%) on a 363-frame scene. As shown under W2, the student is also more robust than per-frame DINOv3 under predicted geometry. We will add these rows to Table 3 and mark GT vs. predicted geometry throughout the paper.

We hope these experiments address the reviewer's concerns and are happy to provide further details.

Dear Reviewer,

Thank you for your valuable suggestions. Could you please let us know whether our revisions have addressed your concerns? We would appreciate your feedback.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.