acceptodds
Under review as a conference paper at ICLR 2027

From Partial Observation to Complete Understanding: Unified 3D Semantic and Affordance Reasoning from a Single Image

Abstract

Real-world robots often receive only partial observations but require complete 3D understanding of object semantics and affordances, including in occluded regions, to guide actions. Existing methods address only parts of this problem: 2D approaches are limited to visible regions, while 3D approaches typically model semantics and affordances separately, making cascaded solutions prone to error propagation. We formulate a unified task that jointly grounds semantic and affordance queries over the full 3D object from a single image. This task involves two sources of heterogeneity. First, features derived from 2D images and 3D shapes follow different representation mechanisms. We align and fuse them via optimal transport, enforcing spatial and cross-instance consistency. Second, semantic and affordance terms, such as arm and move, can be distant in language space despite referring to the same region. We inject spatial–functional cues into text embeddings to capture such part–function correspondence while preserving part distinctions. We further construct a multi-level, dual-attribute dataset with semantic and affordance annotations at the region, instance, and category levels. Experiments in simulation and the real world demonstrate consistent improvements over 2D- and 3D-based baselines under occlusion.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Single-image 3D understanding, 3D Semantic Segmentation, 3D Affordance Segmentation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CPAGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies unified 3D semantic-part and affordance grounding from a single RGB image. The method first reconstructs a complete 3D object with SAM3D, extracts image-derived and geometry-derived features using SAM3D and Utonia, and aligns them through an optimal-transport-based fusion module. It further adapts text features to the reconstructed object through cross-attention and predicts query-conditioned 3D masks using point-text similarity. The paper also constructs a joint semantic-affordance dataset by cross-annotating PartNet and 3DAffordanceNet. Experiments on synthetic and real-world datasets report strong improvements over existing semantic and affordance segmentation baselines.

Critical strengths
  1. The problem formulation is useful and practically motivated. Predicting both semantic parts and affordances over a complete reconstructed object from a single partial observation is a meaningful setting, and the unified formulation is more useful than treating the two tasks independently.
  2. The empirical results are generally strong. The method achieves substantial gains on PartNet and 3DAffordanceNet and also transfers reasonably to real DREDS images. The constructed joint semantic-affordance supervision is also useful, and Table 5 suggests that joint training, especially, provides a sizable benefit for affordance prediction.
Critical weaknesses
  1. The contribution of the proposed OT fusion is not sufficiently supported by the current ablations. In Table 4, OT fusion improves semantic segmentation over additive fusion, but decreases affordance performance from 36.73 to 35.02 on A. and from 33.65 to 31.81 on AO. There is also no full-model ablation without OT or comparison with stronger fusion alternatives. It is therefore unclear how much OT itself contributes to the final performance.
  2. Some generalization claims are stronger than what the experiments currently demonstrate. SO./AO. are described as out-of-distribution queries, but the training-time text augmentation already contains very similar object-conditioned and compositional templates. The DREDS unseen-category results are also mixed, while only the overall average is emphasized. I would like to see genuinely unseen language evaluations and separate seen/unseen category results.
  3. The relation to closely related 2D-to-3D affordance learning work is unclear. Unlocking 3D Affordance Segmentation with 2D Semantic Knowledge (Huang et al., CVPR 2026) appears closely related in motivation, as both methods use 2D semantic knowledge to improve 3D representations for affordance segmentation. The specific alignment mechanisms are different, and this paper additionally considers joint semantic-affordance learning, but I did not find this work discussed in the related-work section. The authors should clarify the conceptual and methodological differences.
Questions for authors
  1. Can the authors provide a full-model ablation without OT and compare against stronger fusion baselines?
  2. Why are SO./AO. considered OOD given the similar training-time text templates? Can the authors also report seen vs. unseen DREDS results?
  3. How does this work differ from Unlocking 3D Affordance Segmentation with 2D Semantic Knowledge (Huang et al., CVPR 2026), which appears closely related but is not discussed?

I currently lean toward weak reject, but I may raise my score if the rebuttal addresses these points convincingly.

Reviewer_Zc3ZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates unified 3D semantic and affordance segmentation from a single image, predicting the queried region on the complete reconstructed object, including occluded surfaces. The method roughly works as follows: SAM3D reconstructs the complete object from a single image and provides image-derived features, while Utonia encodes the reconstructed geometry into shape-derived features; the two feature sets are then aligned and fused via Sinkhorn optimal transport in a shared latent space. A Spatio-Functional Adaptation module uses multi-head attention, with the object point features as keys and values, to adapt the text embeddings from a frozen Qwen3-8B encoder. A shared matching head finally outputs the semantic or affordance mask via point–text similarity. The authors also construct a multi-level dual-attribute dataset by cross-annotating PartNet and 3DAffordanceNet to support joint training.

Critical strengths
  1. The setting of achieving global understanding from partial observation is well motivated and consistent with real-world applications.
  2. The task is newly proposed and has no closely related existing methods, making the formulation reasonably novel.
  3. The method achieves strong performance on this new task.
Critical weaknesses
  1. The framework relies heavily on the capability of SAM3D, and no error analysis is provided for cases where the reconstruction is imperfect.
  2. The computational cost of the full pipeline is not reported.
  3. The dataset annotations depend on Gemini with apparently limited human verification, so the annotation quality is questionable.
Questions for authors
  1. How does the performance degrade when SAM3D produces incorrect or incomplete reconstructions? Can you provide a breakdown separating reconstruction errors from reasoning errors?
  2. The AI Use Statement says the Gemini outputs were verified, filtered, and corrected by the authors. What proportion of the annotations was actually human-verified?
  3. Both 3D-AffordanceLLM (ICLR 2025) and GREAT (CVPR 2025) are cited in the related work but excluded from the experimental comparison. Can you add comparisons under an aligned protocol, or justify why such comparisons are not feasible?
Reviewer_Z5TyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates unified semantic and affordance grounding on a complete 3D object reconstructed from a single RGB image, aligns image- and shape-derived features with optimal transport, adapts text embeddings with object-specific spatial–functional cues, and trains on a dual-attribute dataset with about 165.6K semantic and 108.0K affordance labels.

Critical strengths
  1. The paper proposes a unified framework that localizes regions on a complete 3D object from either a semantic-part query or an affordance query.

  2. It constructs a multi-level dual-attribute dataset covering 16 categories, with about 165.6K semantic labels and 108.0K affordance labels.

  3. The proposed method achieves state-of-the-art performance on both semantic and affordance queries.

Critical weaknesses
  1. Tables 1 and 2 evaluate only four templates: a semantic word, an affordance word, and each with an object name. The reported gains therefore concern category-word alignment, not text instructions of the kind used in real-world affordance grounding.

  2. Optimal-transport fusion aligns SAM3D image features with Utonia shape features, and spatio-functional adaptation aligns semantic words with affordance words on the same object. Both are built for the single-image, dual-attribute pipeline. Methods that take 3D and one text input directly, including LASO, 3D-AffordanceLLM, and GREAT, do not face these two alignment problems. The paper does not test whether either module transfers to direct 3D-plus-text or image-plus-instruction settings. In my view, transferring to real-world settings that reconstruct from images and then perform grounding would be more meaningful than remaining limited to the paper’s experimental datasets, which already provide 3D point-cloud modalities.

  3. The paper does not list which categories are used for training and which categories never enter training, so it cannot demonstrate that the method generalizes to categories outside the training domain.

Minor Weakness:

  1. Please confirm whether FDF^{D}, FUF^{U} are the same Utonia feature. If they are, please unify the notation.
Questions for authors
  1. From the experiments in Tables 1 and 2, are the quantitative queries only semantic words, affordance words, and each of them plus an object name? In the real world, affordance grounding is generally done with text instructions like those in LASO[1]. Does the method of this paper perform better on text instructions, or is it limited to aligning category-word inputs?

  2. The optimal-transport fusion and the spatio-functional adaptation both seem specific to the task designed in the paper. One aligns features before and after reconstruction, and the other aligns semantic categories with affordance categories. Many previous methods take 3D and text directly, so they do not actually need these two alignments. Is the method specific to the proposed task setting and therefore strongly limited? Can it transfer to settings that directly take 3D plus text, or an image, such as LASO[1], 3D-AffordanceLLM[2], and GREAT[3] mentioned in the related work?

  3. How does the method perform on objects outside the training domain, and does it have strong generalization ability?

[1] Li Y, Zhao N, Xiao J, et al. Laso: Language-guided affordance segmentation on 3d object[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 14251-14260.

[2] Chu H, Deng X, Lv Q, et al. 3d-affordancellm: Harnessing large language models for open-vocabulary affordance detection in 3d worlds[C]//International Conference on Learning Representations. 2025, 2025: 72009-72025.

[3] Shao Y, Zhai W, Yang Y, et al. Great: Geometry-intention collaborative inference for open-vocabulary 3d object affordance grounding[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 17326-17336.

1 comment

Thank you for the additional experiments. They partially address my concern, but my main reservations remain.

First, the LASO-style evaluation is still conducted under the authors’ own setting rather than the standard LASO[1] benchmark protocol. To demonstrate algorithmic advancement on affordance grounding, comparison with recent methods should be reported on established benchmarks such as LASO, not mainly on the proposed dataset/setting.

Second, the task significance is still unclear. The method centers on reconstructing 3D from a single image and then grounding concise semantic/affordance labels. The constructed dataset covers 16 object categories and 16 affordance types, which does not substantially broaden existing 3D affordance taxonomies. If reconstruction from 2D is partly motivated by the limited category coverage of existing 3D affordance datasets[1] relative to 2D ones[2], this paper does not expand that taxonomy, which makes the practical benefit of the reconstruction-based setting less clear to me.

Moreover, the proposed Optimal Transport fusion and spatio-functional adaptation appear closely tied to this reconstruct-then-align pipeline. The added 3D+text results show some transfer, but they still do not establish that the method is competitive or necessary under standard 3D+text affordance grounding settings such as LASO / 3D-AffordanceLLM / GREAT.

Neither the necessity of the proposed task nor the method’s effectiveness under standard comparable protocols has been sufficiently demonstrated. I therefore keep my score.

[1] LASO: Language-guided Affordance Segmentation on 3D Object, CVPR2024

[2] Learning Affordance Grounding from Exocentric Images, CVPR2022

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.