acceptodds
Withdrawn from ICLR 2027

Task-Grounded Open-Vocabulary 3D Scene Graph Incremental Generation for Task Planning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

For a robot operating in an indoor 3D scene, a planning task is inherently local: it is defined over a small subset of entities, relations, and states that the goal depends on, rather than over the scene as a whole. Robotic task planning therefore demands task-grounded, on-demand scene understanding—yet existing 3D scene graph pipelines do the opposite, consuming all frames and exhaustively enumerating objects and relations before any task is known, on a closed scene graph with predefined categories and structure. This global, task-agnostic construction is mismatched to planning: in the 2D-to-3D reconstruction process, more frames accumulate larger back-projection and registration errors, which instead amplify object recognition bias; pose drift corrupts the association of small interaction elements; and computation is wasted on task-irrelevant regions. We propose Task-Grounded Open-Vocabulary 3D Scene Graph Incremental Generation (TG-OSG), which reframes the scene graph into an open representation generated on demand. TG-OSG (1) builds a coarse open scene graph from a few dominant-viewpoint frames, (2) incrementally refines it by acquiring additional frames only for task-relevant regions and discovering novel entities and relations, and (3) terminates once the task is determined to be executable—yielding an execution sequence—or infeasible. We evaluate on three public 3D scene graph datasets together with a self-built simulation for task-grounded scene graph generation, and demonstrate significant improvements and advantages over existing closed and open SGG methods on both open scene graph generation and the corresponding task planning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Open-World Scene Understanding, 3D Scene Graph Completion, Task-Grounded Scene Reasoning, Task Planning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bWy3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed a task-grounded open-vocabulary 3D scene graph Incremental Generation method. The core idea is to reframes the scene graph into an open representation generated on demand. The method is evaluated on three public 3D scene graph datasets together with a self-built simulation. It demonstrates some improvements and advantages over existing closed and open SGG methods.

Critical strengths
  1. The distinction between task-agnostic construction and task-grounded construction is interesting. Task-grounded and on-demand 3DSG generation is a worthwhile research direction, and the proposed framework connects incremental scene understanding with downstream planning.

  2. The reported improvements are significant. The method achieves a success rate of 37.96%, compared with 22.22% for the strongest baseline. It also improves R@50 from 12.44% to 24.91% across the three datasets.

Critical weaknesses
  1. Some claims appear broader than the implemented observation mechanism. The abstract describes acquiring additional observations only for task-relevant regions, and Figure 2 suggests task-dependent frame selection. However, Appendix B.2 specifies a fixed Pose-FPS ordering independent of task text and graph predictions. The method therefore appears to combine incremental graph construction with task-solvability checks and early stopping. Please clarify this distinction and align the abstract and Figure 2 with the implemented mechanism.

  2. The positioning relative to prior task-grounded or on-demand scene graph construction could be strengthened. In particular, Clio [1] already studies task-dependent selection of scene content and granularity, with incremental graph construction. As additional context, recent work such as [2] and [3] may also be useful to discuss.

Related work:

[1] Clio: Real-time Task-Driven Open-Set 3D Scene Graphs

[2] FOUND-IT: Foundation-model-first Task-driven 3D Scene Graphs with Granularity on Demand

[3] Just-In-Time Scene Graph Growth: Combating Perceptual Saturation in Long-Horizon Robotics

Questions for authors
  1. Could you clarify whether the task influences which additional frames are selected, or only whether the next fixed Pose-FPS batch is processed? If task-dependent frame selection is implemented, please describe it clearly. Otherwise, should the observation mechanism be understood as task-triggered expansion with task-dependent early stopping?
Reviewer_2mPqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposed a task-driven approach to build a 3D scene graph. Given a task in natural language and a set of posed RGB-D observations, the approach selects a subset of the observations to build a representation that allow a robot to complete the task. The approach prioritizes observations using camera viewpoint diversity, and then sequentially processes subsets of observations. 3D objects and relations are extracted from observations and associated to the existing scene graph using a semantic and geometric criterion. The construction stops when the task becomes solvable according to an external symbolic planner.

Critical strengths
  • The idea of using 3D scene graphs for planning is sound and has been proven effective in related work.
  • The goal of informing the representation and data acquisition with task information is a worthwhile pursuit.
  • The paper idea is easy to grasp from the writing, despite the fact that the presentation can be largely improved.
Critical weaknesses
  • Incorrect statements: (1) The statement that reconstructing the entire scene induces large pose errors is incorrect and confuses localization/odometry/SLAM with semantic mapping and object understanding. Common SLAM approaches would not reason over objects or the scene graph, hence they would be agnostic to the scene graph construction. (2) the statement that completeness does not imply sufficiency is clearly inaccurate. Completeness does imply sufficiency (by definition of completeness), but of course it might be more than what’s needed for the task.
  • The presentation is confusing: the abstract and intro hint to the fact that the goal is retaining only task-relevant aspects in the 3D scene graph; but then the second paragraph of the introduction reframes the goal in terms of “what should be observed new” (an active perception problem); then the methodology proposes an approach for observation selection which is task agnostic and distance-based. The paper requires major rewriting to make the narrative and problem formulation coherent across the manuscript. The other major issue is that key details are left out from the main paper and postponed to appendix, which makes the paper not self-contained. For instance, the method description never mentions the notion of budget, which instead is one of the key aspects of the experiments.
  • Novelty and contribution: The proposed approach is described in Sections 3.3 to 3.5, which is roughly half a page of content. While I understand some of the material is postponed to appendix, the reader is left to wonder what is the main contribution supporting the paper.
  • Soundness: the information collection, encoded in the permutation pi, is not really described in the main paper. If I read the details in Appendix A.1 the selection is purely spatial, which questions the task-driven nature of the representation. As discussed in [S3], the scene representation is really coupled with task planning and in many cases the decoupling assumed in this paper (essentially, the task plan is given and one only has to assess feasibility) is not applicable in practice.
  • Incomplete literature review: the literature review completely overlooks related work on incremental scene graph construction [S1], task-driven and open-vocabulary scene graph construction [S2-S4], and planning over scene graphs. Moreover, the literature review lacks any review of work on active perception, which is related to the idea of active information gathering in the paper.
  • Minor: Figures are low-resolution and one would expect from Figure 4 to show a representation grounded in 3D space rather than an abstract graph.
  • Experiments: Table 1 shows substantially improved performance in terms of object recall but it is unclear (1) if this is for a specific task / set of objects, or for all objects in the scene, and (2) where the advantage is coming from, considering that the contribution of the paper is the task-driven nature of the approach, which is not expected to increase recall or precision across all objects.

[S1] Hughes N, Chang Y, Hu S, et al. Foundations of spatial perception for robotics: Hierarchical representations and real-time systems. The International Journal of Robotics Research. 2024;43(10):1457-1505. [S2] D. Maggio et al., "Clio: Real-Time Task-Driven Open-Set 3D Scene Graphs," in IEEE Robotics and Automation Letters, vol. 9, no. 10, pp. 8921-8928, Oct. 2024, doi: 10.1109/LRA.2024.3451395. [S3] Y. Chang et al “ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis”, CVPR, 2025. https://openaccess.thecvf.com/content/CVPR2025/html/Chang_ASHiTA_Automatic_Scene-grounded_HIerarchical_Task_Analysis_CVPR_2025_paper.html [S4] D. Maggio et al., FOUND-IT: Foundation-model-first Task-driven 3D Scene Graphs with Granularity on Demand, arxiv May 2026. https://arxiv.org/abs/2605.25371

Questions for authors
  • can the authors spell out the novelty of the approach, besides the spatial selection/ordering of the observations, and the stopping based on the task?
  • can the authors clarify where the advantage seen in Figure 1 is coming from?
Reviewer_5DscOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Task-Grounded Open-Vocabulary 3D Scene Graph Incremental Generation (TG-OSG), which incrementally constructs a 3D scene graph from RGB-D observations and stops once the graph is sufficient to complete the task. The authors argue that conventional task-agnostic scene graph generation unnecessarily processes entire scenes, increasing computation and reconstruction errors.

The method uses a fixed, task-independent camera ordering to progressively process observations, updates a persistent scene graph, and checks task solvability at each stage. Other works, such as “Think while you map” (https://arxiv.org/abs/2606.31471), explore incremental scene graph construction as exploration occurs, or use the incremental scene graph to aid exploration (Li et. al. CorL 2021 https://proceedings.mlr.press/v164/li22e/li22e.pdf). Its primary contribution is therefore task-solvability-based early termination of incremental scene graph construction.

Experiments report a task success rate of 37.96% vs 22.22% for the strongest baseline, and better scene graph recall. However, methodological issues make it difficult to attribute these improvements to the proposed contribution or establish its advantages over simpler alternatives.

Critical strengths

Terminating perception once sufficient information is available for planning could reduce unnecessary computation, particularly when scene understanding is expensive. The method combines open-vocabulary perception, persistent scene graph updates, and symbolic planning, with reported improvements across three datasets. The paper's most defensible contribution is applying task-solvability-based early stopping to incremental open-vocabulary scene graph construction over a predetermined observation sequence.

Critical weaknesses

The paper frames task-agnostic scene graph construction as inefficient because it processes information irrelevant to an individual task. However, scene graphs are typically used to provide reusable representations supporting multiple downstream tasks. The paper does not evaluate this reuse or demonstrate why a persistent graph is preferable to other methods, such as direct VLM-based grounding and planning for its single-task setting. Figure 2 indicates task-relevant detection and reasoning, whereas the baselines primarily perform task-agnostic scene graph construction. Appendix B does not clarify whether task information conditions detection, potentially giving TG-OSG an additional advantage unrelated to incremental generation. Furthermore, Figure 3(a) reports only aggregate baseline success rates without clearly specifying how the baselines are evaluated, with the assumption being that they are evaluated at L3. The paper lacks ablations isolating task conditioning, perception, and persistent graph updates. In particular, the substantial scene graph improvements using static L3 (33.33%) cannot be attributed to early stopping. Since observations follow a fixed ordering, the method essentially processes increasingly large subsets of pre-existing frames until a symbolic plan is found. The paper does not establish whether this adaptive procedure outperforms constructing a scene graph once from a smaller, fixed subset of informative frames. The authors attribute the improvement from 33.33% static L3 success to 37.96% dynamic success to avoiding accumulated reconstruction errors, but provide no direct error analysis. Comparisons against fixed-budget and batch-processing alternatives are essential to demonstrate that incremental termination adds value beyond frame subsampling. The method assumes access to an existing RGB-D sequence and camera poses, rather than addressing the harder robotic problem of acquiring informative observations. Their frame sampling maximizes viewpoint diversity rather than task-relevant information gain or visibility of occluded objects, though the authors explicitly call this out as a limitation. Furthermore, no runtime savings are reported, with the closest metric being the frame count budget, leaving performance as the main axis of comparison. The authors do not establish that this mechanism provides meaningful advantages over fixed-budget frame selection, nor that an explicit scene graph is necessary for the evaluated setting. The reported improvements are further complicated by unclear task conditioning and baseline protocols.

As a result, the paper currently demonstrates an integrated system with promising reported metrics, but does not convincingly validate the novelty, necessity, or practical benefits of its central methodological contribution.

Questions for authors

Does TG-OSG use task information during object detection or relation prediction? If so, do the baselines receive equivalent task information? Can the authors provide a task-agnostic TG-OSG ablation (such as by removing the solve check)? How were the baselines evaluated across the L0–L3 frame sets? Can the authors report per-stage results for baselines as well as TG-OSG? What are the specific factors in your method that actually need to increased performance? Can the authors provide evidence that accumulated reconstruction errors explain the degradation observed at L3? Why is a persistent scene graph necessary for the evaluated tasks? Can the authors compare against direct VLM-based planning or demonstrate benefits from reusing the graph across multiple tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.