acceptodds
Under review as a conference paper at ICLR 2027

SGOP3D: A Semantic–Geometric Anchoring and Observation-Preserving Reconstruction Algorithm for Multi-Object Robotic Manipulation

Abstract

Multi-object 3D reconstruction for robotic manipulation requires consistent object identity across target admission, measured-surface fusion, and missing-geometry recovery, together with explicit rules governing the use of geometry from different sources in manipulation. We propose SGOP3D (Semantic–Geometric Anchoring and Observation-Preserving 3D Reconstruction), a semantically anchored algorithm for observation-preserving 3D reconstruction from two orthogonal views. The method combines our semantic and geometric detection networks, using class consistency, bidirectional top–front reprojection, and world-coordinate constraints to establish trusted object-level anchors. The same anchor governs measured-surface extraction and completion with our GenCompletionNet, converting complete-shape predictions into missing-geometry residuals that satisfy surface-protection and dual-view free-space constraints. Per-point provenance labels restrict contact localization to measured-source points and limit generated geometry to shape estimation and occupancy refinement. In a system-level comparison on 205 synchronized top–front observation pairs involving 20 objects arranged in two groups of 10, SGOP3D exceeds AMB3R by 36.45 and 21.10 percentage points in strict instance separation success and F@, respectively, under a shared scoring protocol. Across 400 controlled grasp–lift–hold trials with fixed detection, planning, and control, SGOP3D achieves 96% success versus 79% for the strongest geometric-input control. A separate continuous sorting demonstration with dynamic replenishment reports 19/20 first-attempt completions and 20/21 transfer-and-release completions. These results show that SGOP3D maintains object-consistency and observational-evidence constraints throughout geometric recovery and manipulation, providing object-level 3D geometry with traceable provenance for contact planning, grasping, and continuous sorting in multi-object robotic scenes.

Primary area:
applications to robotics, autonomy, planning
Keywords:
multi-object 3D reconstruction, cross-view instance association, semantic–geometric consensus, observation-preserving reconstruction, residual point cloud completion, robotic manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LSULOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SGOP3D, which reconstructs an object point cloud in a complex scene using 2 RGB-D images. Input: 2 images + a text prompt Pipeline: 1. detect whether the object is in the image and output a bounding box -> 2. target the object & correspond the object between 2 images, and only "trusted" matches are admitted -> 3. based on the measured point cloud, predict the full point cloud -> 4. grab the object. Output: object's point cloud + position for grasping

Critical strengths
  1. It is a very important research question for the robot to identify what the target object is and where it is.
  2. Interesting trusted cross-view instance anchoring: it is important to let the robot know which object is the right one to pick up instead of picking up the wrong object.
Critical weaknesses
  1. The actual inputs are 2 images with their depth, intrinsic, and extrinsic information. Please state this explicitly in the abstract. The baselines were not given the same depth and calibration, which seems unfair and should be added to the main paper instead of being stated in the appendix.
  2. Comparison is insufficient. Currently, it only has comparisons with scene-level reconstructors in Table 1; it would be better to add:
  • Comparison with object reconstruction methods such as Trellis or Hunyuan3D with depth-based scale/pose alignment;
  • Comparison with other point cloud completion networks such as PoinTr and AdaPoinTr.
  • Comparison with SceneComplete, which is shown in the supplementary video as related work, is neither cited nor compared in the paper, although it addresses a closely related task.
  1. In Table 2, "Dual + consensus" compared to "Geometry only" is a little better on false acceptance but a lot worse on Present-target acceptance, and 10.54% become unknown rate, which means the fine-tuned VLM is not significantly better than "Geometry only". Adding an ablation of "Geometry only + consensus".
  2. Since contact points are restricted to the measured point cloud in the configuration, why is the gap in Table F2 between C1 76% and SGOP3D 96% so large? Please provide a failure analysis to show how much of the gain comes from completion itself.
  3. Please clarify how the CAD ground-truth models are obtained and their accuracy, especially for natural or deformable objects, whose shapes may differ.
  4. The supplementary video stated, "All methods share inputs and calibration," which is different than what really happens.
Questions for authors

See weakness.

Reviewer_UyZqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents SGOP3D, a system for object-level 3D reconstruction from top and front RGB-D views for robotic grasping and sorting. It combines semantic and geometric detections with cross-view consistency checks to establish instance anchors, which guide measured-point fusion and shape completion. Predicted geometry is filtered using surface-protection and free-space constraints, while point-source labels restrict contact localization to measured points and reserve generated geometry for shape and occupancy estimation. Validation includes reconstruction comparisons on 205 observation pairs involving 20 objects, a three-object challenge set, admission and point-source ablations, 400 controlled grasping trials, and a continuous sorting demonstration involving 20 objects. The authors report improved instance separation and reconstruction F-score over the evaluated baselines, together with 96% grasping success versus 79% for the strongest geometric-input control.

Critical strengths
  1. The paper addresses a meaningful interface between reconstruction and robotic execution: observed and generated geometry should not carry the same evidential weight. Preserving measured surfaces and explicitly restricting contact localization by point provenance provides a coherent design principle for using shape completion in manipulation.
  2. The evaluation includes physical downstream validation beyond reconstruction metrics. Across 400 controlled grasping trials, the full system achieves 96% success compared with 79% for the strongest geometric-input control, with detection, planning, and control held fixed. This supports the practical value of the complete geometric representation, although it does not isolate individual mechanisms.
Critical weaknesses
  1. The proposed anchoring mechanism leaves a gap between instance-level association and point-level correctness. Matching detection regions across views does not establish that every depth point extracted from those regions belongs to the target, yet retained measured points remain eligible for contact localization. Without contact-instance accuracy, the paper does not adequately establish that its representation resolves the instance-consistency problem.
  2. The main reconstruction comparison mixes differences in sensor use, task-specific training, and architecture, while the source ablation does not separately test free-space filtering, surface protection, or contact-source restrictions. The admission study also lacks a geometry-only-plus-consensus baseline. Consequently, the necessity of the semantic branch and the contribution of the central constraints remain unclear.
  3. The manipulation evaluation is limited to grasping and pick-and-place sorting. These results support the method’s utility for grasp-oriented geometric perception, but do not establish its applicability to broader manipulation tasks.
Questions for authors
  1. What prevents depth points from neighboring objects or background within an admitted ROI from being retained as measured-source points and used for contact localization? Do you have measurements of measured-source contamination or contact-instance accuracy? These would help establish whether trusted anchors provide reliable target geometry.
  2. Which sensor inputs and calibration information are available to each reconstruction baseline? In the grasping comparison, how much of the improvement arises from geometric quality versus differences in geometry availability and fallback behavior? Please clarify whether existing controlled comparisons isolate these factors.
  3. What specifically makes the three-object set challenging, and are those conditions also represented in the controlled grasping trials or continuous sorting demonstration?
Reviewer_1KLGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper discusses some aspects of two-view object detection in multi-object scenes for robot manipulation.

Critical strengths

None.

Critical weaknesses

This paper discusses the problem of cross-view object detection with a focus on a very specific setup of a top and front view. This overly restrictive initial setup already greatly weakens any possible contribution over a more generic multi-view formulation.

The introduction claims "instance admission must therefore precede fusion and continue to constrain recovery." I can not confidently parse the sentence. I believe by "instance admission" the authors means that segmentation instance correspondence between views, however, the terminology used is in no way standard nor defined. "Constrain recovery" is an equally vacuous statement as we're never told what is being recovered. Here I think it might mean object 3D shape or reconstruction, but again nothing is defined. Further if this is the intended meaning, it does not follow that correspondence must be determined prior to reconstruction, one could attempt to generate reconstructions from different correspondences and then propose reconstructions and filter afterwards as a simple straw-man counterpoint to this claim.

The intro then discusses VLMs before stating "The anchor constrains measured-surface extraction..." Here again we are given an undefined term "the anchor" and some list of things it achieves without any context or connection to the previous paragraphs. It is unclear what is actually being studied by the end of the introduction.

When we look to the methods section for some help in formalizing this problem we do not find formal definitions to help clarify the problem or approach. We are again given a sentence about "cross-view admission produces a trusted anchor that constrains reconstruction" None of these terms are defined nor stated, instead we are told that a view v = {t, f} has two undefined variables, which have associated images.

We are then given a paragraph describing training psuedo targets and cad files, which does not correspond in any logical way to the material presented so far.

In the next subsection we are told about how qwen is used for Type-A discovery and Type-B presence verification. These "Type-A and Type-B" terms are found nowhere else in the paper and are not defined.

This form of gibberish mixed with poorly defined mathematical formalization, that do not connect clearly with the text or narrative of the problem or approach continue throughout the remainder of the approach section. It is entirely unclear what is actually being proposed or implemented once we reach the end of the proposed approach.

The experiments are thus incomprehensible given we have not basis to understand what is being studied.

Questions for authors

Nothing to ask.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.