acceptodds
Withdrawn from ICLR 2027

DORIC: Digital-Gene-based Object Understanding from a single RGBD Image via Chain-of-Thought Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding 3D object knowledge and functionality in the physical world is a fundamental challenge for embodied AI. The recently proposed Digital Gene offers a promising explicit representation of objects via executable programs for object structure and function understanding. However, three major practical obstacles remain: 1) inaccurate gene grounding, 2) missing task-oriented reasoning for function utility, and 3) inflexible functional programs. To address these, we introduce DORIC, a framework that pairs MLLM's semantic and image reasoning with our proposed gene grounding modules for precise physical and numerical analyses. DORIC decomposes object-understanding requests into sub-tasks, each focusing on a single object and processed in a four-stage Chain-of-Thought (CoT) pipeline: I. Object and Structure Identification, II. Gene Selection and Grounding, III. Part-Level Program Selection, and IV. Object-Level Program Synthesis. By integrating multi-modal reasoning with specialized grounding modules, our contributions are four-fold: 1) a hybrid architecture for precise gene grounding involving MLLM and specialized modules, 2) a task-oriented four-stage CoT pipeline for function utility reasoning, 3) an in-context functional program synthesis paradigm, and 4) superior effectiveness and robustness demonstrated by extensive experiments.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Object Understanding, Object Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_gqZaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce DORIC, a framework that leverages LLMs, alongside prior work (Digital Gene), to generate programmatic descriptions of objects for geometric understanding and physical interaction. The paper describes the proposed system and evaluates it's effectiveness.

Critical strengths

Motivation: The problem is well-defined and articulated. The preliminary and introductory sections highlight the motivation and challenges well, making the rest of the paper follow nicely as a logical conclusion.

Straight-forward: The proposed system is intuitive and straight forward. Conceptually it is quite simple, which makes the results particularly impressive and the provided ablations shed insight into the importance of each component.

Clear writing: The writing is clear and easy to follow. My only issue is with Figure 2 (see Weaknesses). Otherwise, the figures are clear and helpful.

Critical weaknesses

Extensibility: The proposed method appears quite rigid to change. Specifically, the authors define a list of parts and concept templates (L229 - 263 concept template). Then, for each concept template a "specialized parameter estimation module is trained" (L229). This training requires the object concept to be represented in the ConceptFactory dataset. Thus, one can imagine that given the set of 263 templates, one can construct examples that fall out of distribution of these templates. For these objects/scenes, the method is definitionally unable to capture these objects. While 263 is a large number, the authors need to do more to convince the reader that this set is sufficient to capture a significant distribution of objects (including the weird and irregular geometries we encounter in the real world!). If it cannot, that is also ok, but the authors should explicitly discuss this limitation and show cases where objects fall OOD.

Ablation on concept templates: Slightly related to the previous point, it would be interesting to investigate the ability of DORIC to represent objects when there are fewer concept templates to compose. Specifically, are there some concept templates that are crucial to accurate modeling? Is the DORIC pipeline & the LLMs it uses able to compose templates in creative ways when the bespoke "percect" template is missing? This analysis would be valuable and insightful to establish the robustness of the approach.

Figure 2: This figure is difficult to follow. There are many arrows, an exhaustive amount of text, and tiny sub-figures. These things make the logical flow hard to keep up with. I recommend the authors to simplify this figure with a schematic (e.g. showing code snippets or pseudocode) and make the logical flow of the figure more salient. I appreciate the attention to detail, but a full working example can go in the Supplementary, the purpose of this figure should be to give an intuitive understanding of your method.

Questions for authors

Please see Weaknesses.

Reviewer_UcEHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DORIC combines MLLM-based task decomposition and program synthesis with geometric parameter estimation and pose estimation. From a single RGBD observation, it fits objects to Digital Gene templates and composes predefined part-level functions into executable interaction programs.

Critical strengths

The explicit representation connects semantic planning with geometric computation and makes intermediate outputs inspectable. The paper evaluates both reconstruction and interaction, including real-robot demonstrations. Replacing intermediate predictions with ground truth provides useful evidence that pose estimation is a major bottleneck.

Critical weaknesses

The contribution of the four-stage reasoning pipeline is not isolated. Without comparisons using the same geometric modules and function library, it is unclear whether the gains come from the proposed reasoning process or better geometric estimates.

The claims about functional understanding and generalization exceed the evidence. Much of the functional knowledge is predefined in templates, while evaluation provides limited evidence on unseen structures or novel task compositions.

The comparisons also leave uncertainty. The geometry baseline uses RGB rather than RGBD, and correcting its scale does not remove this information difference. Simulation success improves from 33.7% to 38.7%, while the much higher real-world success rate lacks sufficiently detailed trial counts and matched baselines.

Questions for authors

With geometry and pose estimates held fixed, how does the four-stage pipeline compare with direct program generation or a fixed interaction policy?

Can the system handle held-out template combinations and tasks requiring new compositions of part-level functions?

How does DORIC compare with a baseline given the same depth information and geometric tools?

What are the object counts, trial counts, and baseline results for the real-world evaluation, and how is training–test asset overlap excluded?

Reviewer_96dyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new method DORIC which uses LLM and specialized models to infer the information for an object-centric structured representation called Digital Gene. The main pipeline of DORIC includes subtask decomposition, object and structure identification, geometric parameters inference, as well as part-level and object-level program synthesis. To support the effectiveness of this method, the authors conduct experiments on both simulation and real-world settings. Overall, DORIC achieves better performance than GeneVLM on geometry and pose understanding and other baselines on object interaction understanding.

Critical strengths
  • DORIC provides a sound modular pipeline to solve the problem of estimating the parameters in Digital Gene, so that this Digital Gene representation can be leveraged for further reasoning and planning. GeneVLM directly fine-tunes a VLM to infer all the information. DORIC, on the other hand, leverages existing strong foundation models, existing off-the-shelf perception models and another trained perception model for geometric parameter estimation to solve this task, which enables more accurate Digital Gene estimation.

  • The experiments are well-designed to support the effectiveness of DORIC. Section 5.1 supports the geometry and pose understanding ability of DORIC comparing with GeneVLM. Section 5.2 shows the improvements on robot-object interaction tasks compared with existing baselines by leveraging DORIC and Digital Gene, which demonstrates the practical contribution of DORIC.

Critical weaknesses

Major weaknesses:

  • Unclear positioning of the contribution. Although the entire pipeline for Digital Gene parameter estimation is sound, I have two concerns to the positioning of this paper. (1) What's the advantage of DORIC and Digital Gene for robot tasks? In VoxPoser, we've already seen that VLMs with structured representation can achieve promising performance for complex robot tasks. The necessity and practical advantages of DORIC with Digital Gene haven't been well-discussed in the paper. I'm not fully convinced that leveraging DORIC for Digital Gene estimation provides new ability to solve the robot tasks compared with existing methods, such as VoxPoser. (2) The pipeline design lacks new insights. The pipeline of identifying subtasks and objects first, then using modular models to estimate parameters for each part is well-known. CoPA has discussed a very similar pipeline. Only transferring this pipeline to Digital Gene can't really support enough contribution.

  • Limited Scope of the method. In the real world robot applications, it's far from enough to only handle the rigid and articulated objects. I won't consider the robot-object interaction with rigid and articulated objects as one of current bottlenecks. I wonder if DORIC with Digital Gene is a suitable design for soft-body manipulation. It's unclear to me how to define the geometric parameters, the part decomposition for soft-body objects and how to synthesize the programs for soft-body objects. Another scope is object-object interaction. For example, if we want to use a hanger to hook the handle of a cup, and pull it closer, this program can't be synthesized with the current DORIC method. These scope limitations set a clear boundary of what tasks can be solved, which limits the ability of this pipeline for solving general robotics problem.

  • Limited part-level programs. The granularity of part isn't constant. For example, open the drawer and tighten the screws on the drawer handle require different part decomposition, which is highly task-related. If the part-level programs are pre-defined and DORIC only selects the part-level programs from the pre-defined library, the diversity of the tasks that DORIC can solve is limited.

Minor weaknesses:

  • Presentation. I wonder if CoT is the correct terminology to describe DORIC? Typical CoT means generating intermediate reasoning steps so that it provides stronger ability on complex reasoning tasks. The method proposed by this paper is more like a modular pipeline, which leverages designed rules to connect different models. Using CoT in the title and the paper may mislead the readers. I suggest to call it a workflow or a harness, which might be more accurate.
Questions for authors
  1. Can the authors discuss the advantage of DORIC with Digital Gene for robot tasks comparing with methods like VoxPoser? A experiment shows that DORIC unblocks new ability that was lacking in VoxPoser would be great.
  2. Given previous modular pipelines like CoPA, what are the new insights and technical contributions that DORIC bring to the community? I'd like to see more insights rather than a new modular pipeline for Digital Gene.
  3. I'd like the authors to show experiments of 1. handling soft-body objects, and 2. object-object interaction with DORIC.
  4. Please also discuss whether a pre-defined part-level program library is enough for diverse robot tasks. If a task is defined on a fine-grained part which isn't in the pre-defined part set, how to handle this case?
Reviewer_nBaVOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to identify an object's structure from a single RGBD image and use it to generate manipulation programs that a robot can execute. DORIC builds on the existing Digital Gene representation. It uses a multimodal large language model to identify parts, select templates, and compose manipulation programs, while specialized modules estimate the parts' geometry, positions, and orientations. The experiments cover geometric reconstruction, simulated manipulation across 21 object categories, and real-world manipulation. Table 1 shows that the Chamfer distance drops from 0.066 to 0.044 compared with GeneVLM. Tables 2 and 8 show that the manipulation success rate increases from 33.7% to 38.7% compared with A3VLM, a gain of 5.0 percentage points, or about 14.8% relative improvement.

Critical strengths
  1. The method achieves better manipulation results across a range of object categories.In Table 8, DORIC has a higher success rate than A3VLM in all 21 categories. Table 3 also shows that adding parameter refinement raises the success rate from 35.4% to 38.7% and reduces geometric reconstruction error. Appendix F.2 provides real-world experiments as well. These results support the method's practical value and show that more accurate parameter estimates can help the robot complete the task.

  2. The authors give a fairly clear account of why manipulation fails.In Table 3, after segmentation and geometric parameters have already been replaced with ground truth, also providing the correct part positions and orientations raises the success rate from 42.5% to 81.8%. Appendix G includes examples involving occlusion, inaccurate shape estimates, and incorrect orientation estimates for symmetric objects. These analyses show that even when an object's shape is reconstructed reasonably well, errors in its estimated position or orientation can still cause manipulation to fail. This helps explain a major limitation of the current method.

Critical weaknesses

It is unclear whether any test objects also appeared in training.Section 4.2 states that all 4,380 objects in ConceptFactory are used for training, while Sections 5.1 and 5.2 evaluate on 785 objects from GAPartNet and 1,170 objects from PartNet-Mobility, respectively. However, Appendix E.3 and Table 6 only list the number of objects in each category. They do not explain whether the training and test objects overlap, and the data splits used to retrain the baselines are also unclear. ConceptFactory itself includes objects from sources such as PartNet-Mobility, so different dataset names do not necessarily mean different objects (ConceptFactory paper, Section 4). This is not confirmed data leakage. Still, if the test objects were used during training, the current results would not be enough to show that the method also works well on unseen objects, and the fairness of the baseline comparisons would also need to be checked.

Questions for authors
  1. Could you clarify whether the training and test sets contain any of the same object models? Please provide an object ID mapping or explain how duplicates were removed, and clarify whether the baselines used the same splits. If the test objects were not seen during training and the comparison settings are consistent, I would view the experimental reliability and overall contribution more favorably. If there is overlap, the results on seen and unseen objects need to be reported separately.

  2. What does program synthesis add compared with predefined manipulation rules? Have these two approaches been compared using the same predicted geometry, positions, and orientations? Alternatively, do any of the existing experiments include new tasks that require recombining part-level functions? This would help distinguish gains from more accurate object recognition and localization from gains due to task reasoning and program synthesis. If most of the benefit comes from the former, the method still has value, but the claims about the adaptability of program synthesis should be narrowed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.