acceptodds
Under review as a conference paper at ICLR 2027

TaoGraph: Task-Oriented Scene Graph Learning for Embodied Planning

Abstract

Embodied task planners are crucial for robotic navigation and manipulation. Among them, scene-graph-based approaches represent environments by semantically organizing objects into a spatial topology, supporting reasoning about object interactions and action prerequisites. However, constructing scene graphs that provide the spatial and procedural information needed for planning remains challenging. First, generic spatial representations lack task-aware adaptation, making it difficult to distinguish task-relevant spatial cues and construct relations needed for planning. Second, conventional graph supervision lacks sufficient procedural semantic guidance, making it difficult to infer action prerequisites and determine valid action orders. To address these challenges, we propose **TaoGraph**, a new task-oriented scene graph learning framework that transforms multi-view observations into task-oriented scene graphs through two key components. Semantics-Induced Spatial Reorganization (SISR) constructs task-conditioned spatial representations by modulating generic spatial features according to task instructions and scene semantics, preserving task-relevant information for graph generation. Change-Aware Procedure Distillation (CAPD) introduces procedure-guided graph supervision by distilling procedure-enriched graph predictions with guidance from discrepancies between two graph teachers, transferring procedural knowledge for action-order reasoning. Experiments demonstrate that SISR preserves task-relevant cues for graph reconstruction, while CAPD improves sequence-sensitive reasoning with comparable overall accuracy. Results on BLINK and MMSI-Bench further demonstrate TaoGraph's generalization to broader scene understanding and reasoning tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Task Planning, Scene Graph, Scene Understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yuFdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose TaoGraph, a method that dynamically reorganizes spatial features based on the 'task instruction' and the overall semantics of the scene. TaoGraph uses Semantics-Induced Spatial Reorganization, which reorganizes generic spatial characteristics based on the task instruction and the overall semantics of the scene, and Change-Aware Procedure Distillation, which induces learning based on the prediction differences between two teacher models, allowing the model to effectively learn the procedural knowledge required for action sequence reasoning. As a result, in task planning reasoning metrics and graph generation quality metrics, the proposed method has improved performance compared to the prior research MomaGraph and other closed VLMs.

Critical strengths
  • TaoGraph dynamically reorganizes spatial features based on the 'task instruction' and the overall semantics of the scene. As a result, spatial cues essential for the task goal remain and unnecessary information is deleted, so the graph composition is flexibly adapted to the task.
  • The proposed Change-Aware Procedure Distillation is a procedure-guided graph supervision method that distills knowledge through procedure-enriched graph predictions. It induces learning based on the prediction differences between two teacher models, making the model effectively learn the procedural knowledge required for action sequence reasoning.
  • In the task planning reasoning metrics and graph generation quality metrics, the proposed methodology has better performance compared to the prior research MomaGraph and other closed VLMs.
Critical weaknesses
  • Metrics such as Sequence-Sensitive Subset, Sequence Planning, Node F1, Spatial F1, and Edge F1 may show better performance simply because TaoGraph is more overfitted to the output of this context. There is insufficient evidence that this actually leads to better spatial knowledge or better downstream tasks compared to the baseline (especially non-finetuned ones).
  • To verify generalization on unlearned external benchmarks (External Evaluation), external benchmarks called BLINK and MMSI-Bench are used, but even these might be favorable conditions for a model trained to output a specific form of graphs.
  • Tests were conducted in the ManiSkill 3D simulation environment. Although it provides a qualitative demo where the scene graph extracted by the model (e.g., the open/close relationship between the handle and the microwave oven, sequential operation of two stove knobs) is input to the planner and the robot actually determines and executes the interaction targets and action sequences, there is no verification on whether this leads to a meaningful improvement compared to baselines such as MomaGraph.
Questions for authors

Has it been verified in what setup, for which downstream task, for what reason, and by how much having better graph generation or a graph better aligned with the instruction actually leads to improvements?

8 comments
AuthorsOpenReview ↗

Response to Reviewer yuFd (1/4)

W1 : Metrics such as Sequence-Sensitive Subset, Sequence Planning, Node F1, Spatial F1, and Edge F1 may show better performance simply because TaoGraph is more overfitted to the output of this context. There is insufficient evidence that this actually leads to better spatial knowledge or better downstream tasks compared to the baseline (especially non-fine-tuned ones).

We thank the reviewer for this comment. We agree that it is important to assess whether TaoGraph’s reported gains can be explained by adaptation to the target scene graph format. To address this concern, we conducted two sets of additional experiments. First, we fine-tuned Qwen2.5 and Qwen3 for scene graph generation on the same training data used for TaoGraph and compared their graph generation and downstream planning performance. Second, we evaluated spatial and functional relations using metrics that require each relation to be assigned to the correct pair of objects.

1. Fine-tuned baselines

We applied supervised fine-tuning for scene graph generation (Graph SFT) to Qwen2.5-VL-7B and Qwen3-VL-8B using the same scene graph training data as TaoGraph. On the same graph validation split, the results are as follows (%).

MethodNode F1Spatial F1Edge F1
Qwen2.5-VL-7B (non-fine-tuned)32.3431.800.00
Qwen2.5-VL-7B + Graph SFT53.6458.224.14
Qwen3-VL-8B (non-fine-tuned)37.9246.510.00
Qwen3-VL-8B + Graph SFT68.1363.413.82
Ours81.5977.1342.58

We further compare the performance of these methods on MomaGraph-Bench (%):

MethodSeq. Sub.Seq. Plan.Non-Seq.
Qwen2.5-VL-7B + Graph SFT67.0764.0077.13
Qwen3-VL-8B + Graph SFT68.2965.3380.29
Ours75.6173.3384.18

Graph SFT improves both Qwen baselines on all three graph generation metrics, showing that task adaptation matters. TaoGraph scores higher than both fine-tuned baselines on all reported graph generation and downstream planning metrics.

2. Relation triplet evaluation

To assess whether individual relations are assigned to the correct object pairs, we follow prior work on scene graph evaluation [1][2] and represent each relation as a (subject, relation, object) triplet, such as (handle, openorclose, microwave) for a functional relation and (handle, in_front_of, microwave) for a spatial relation. The table below reports micro-averaged precision, recall, and F1 (%) separately for functional and spatial triplets [3].

MethodFunctional triplet PFunctional triplet RFunctional triplet F1Spatial triplet PSpatial triplet RSpatial triplet F1
Qwen2.5-VL-7B (non-fine-tuned)0.000.000.000.000.000.00
Qwen2.5-VL-7B + Graph SFT33.9023.2627.5918.8313.8815.98
Qwen3-VL-8B (non-fine-tuned)8.966.987.846.203.834.73
Qwen3-VL-8B + Graph SFT60.5650.0054.7831.8427.2729.38
Ours76.8161.6368.3963.1651.6756.84

Compared with Qwen3-VL-8B + Graph SFT, TaoGraph is 13.61 and 27.46 percentage points higher in functional and spatial triplet F1, respectively. This shows that the improvement lies in recovering relations on the correct object pairs, not merely in whether relation types appear at the graph level.

In summary, after the baselines receive the same scene graph fine-tuning, TaoGraph still recovers more accurate object-pair relations and achieves higher task accuracy. We will add the above fine-tuned baselines and relation metrics to the revised manuscript.

References

[1] Lu C, Krishna R, Bernstein M, et al. Visual relationship detection with language priors[C]//European conference on computer vision. Cham: Springer International Publishing, 2016: 852-869.

[2] Xu D, Zhu Y, Choy C B, et al. Scene graph generation by iterative message passing[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017: 3097-3106.

[3] Anderson P, Fernando B, Johnson M, et al. Spice: Semantic propositional image caption evaluation[C]//European conference on computer vision. Cham: Springer International Publishing, 2016: 382-398.

AuthorsOpenReview ↗

Response to Reviewer yuFd (2/4)

W2 : To verify generalization on unlearned external benchmarks (External Evaluation), external benchmarks called BLINK and MMSI-Bench are used, but even these might be favorable conditions for a model trained to output a specific form of graphs.

We thank the reviewer for this comment. We agree that it is important to examine whether the choice of external subtasks and the evaluation protocol favor models trained to generate graphs in the target format. To address this concern, we explain how the selected subtasks relate to the information needed for embodied task planning and report a controlled comparison to assess whether TaoGraph’s gains persist when the Qwen baselines also receive scene graph training.

1. Rationale for the external evaluation

First, BLINK and MMSI-Bench are general-purpose public benchmarks used in prior evaluations of vision-language models (e.g., LLaVA-OneVision [1], LLaVA-Interleave [2], and ViLaSR [3]). We use them only for external evaluation (Appendix D.1) and score each question by the correctness of the final multiple-choice answer (Section 4.1).

Second, TaoGraph is designed for embodied task planning. It builds a task-oriented scene graph from multi-view observations and a task instruction. The graph captures task-relevant objects, interactive parts, and their functional and spatial relations, which help the planner identify interaction targets and reason about action order. We selected external subtasks that probe similar information, such as functional correspondence and relative object positions, to test whether the learned graph representation transfers beyond MomaGraph-Bench (Section 4.2.4). Tasks that rely on other visual cues may benefit from representations tailored to those cues.

2. Comparison with Fine-tuned baselines

We evaluate the same Graph SFT baselines as in our response to Weakness 1 on BLINK and MMSI-Bench. The table below reports the accuracy (%).

MethodFunc. Corr.Obj. Loc.Spatial Rel.Obj.–Obj.
MomaGraph-R120.0040.1679.0228.72
Qwen2.5-VL-7B + Graph SFT26.1554.1079.7234.04
Qwen3-VL-8B + Graph SFT23.8549.1882.5224.47
Ours*33.8558.2079.7235.11

* Ours uses Qwen2.5-VL-7B-Instruct as its vision-language model backbone.

Compared with Qwen2.5-VL-7B + Graph SFT, TaoGraph achieves higher accuracy on Functional Correspondence, Object Localization, and object–object positional reasoning, and matches it on Spatial Relation.

We will add a description of the external evaluation setting and the above control results to the revised manuscript.

References

[1] Li B, Zhang Y, Guo D, et al. Llava-onevision: Easy visual task transfer[J]. arXiv preprint arXiv:2408.03326, 2024.

[2] Li F, Zhang R, Zhang H, et al. Llava-interleave: Tackling multi-image, video, and 3d in large multimodal models[C]//International Conference on Learning Representations. 2025, 2025: 81182-81199.

[3] Wu J, Guan J, Feng K, et al. Reinforcing spatial reasoning in vision-language models with interwoven thinking and visual drawing[J]. Advances in Neural Information Processing Systems, 2026, 38: 143297-143330.

AuthorsOpenReview ↗

Response to Reviewer yuFd (3/4)

W3 : Tests were conducted in the ManiSkill 3D simulation environment. Although it provides a qualitative demo where the scene graph extracted by the model (e.g., the open/close relationship between the handle and the microwave oven, sequential operation of two stove knobs) is input to the planner and the robot actually determines and executes the interaction targets and action sequences, there is no verification on whether this leads to a meaningful improvement compared to baselines such as MomaGraph.

We thank the reviewer for this comment. We agree that a quantitative comparison is needed to determine whether TaoGraph improves task performance in simulation. To address this concern, we compared TaoGraph with MomaGraph-R1 in ManiSkill.

The experiments follow the process in Appendix F.2 and cover 3 tasks: turning on the kitchen sink faucet (spanning 8 kitchen layouts and 5 faucet models), and the two tasks in Figure 7, opening a microwave oven and sequentially rotating two stove knobs, with 57 configurations in total. In each configuration, the execution success rate is evaluated on two decisions, the action and the interaction target. The results are as follows (%).

Task# TrialsMomaGraph-R1Ours
Faucet8055.0080.00
Microwave oven2080.0085.00
Two stove knobs14100.00100.00

In addition, we conduct a test specifically on spatial relations in the same configurations: each configuration contains one multiple-choice question on the spatial relation between objects in the scene, evaluated by accuracy. The results are as follows (%).

Task# QuestionsMomaGraph-R1Ours
Faucet4020.0032.50
Microwave oven1010.0020.00
Two stove knobs728.5742.86

We will add the setup and results of the above simulation comparison experiments to the revised manuscript.

Reviewer_zbCLOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TaoGraph, a task-oriented scene graph learning framework for embodied task planning, targeting limitations of existing methods in capturing task-relevant spatial relations and action prerequisites. It introduces Semantics-Induced Spatial Reorganization (SISR) to construct task-conditioned spatial representations from task instructions and scene semantics, and Change-Aware Procedure Distillation (CAPD) to incorporate procedural knowledge into graph supervision for better action-order reasoning.

Experiments on MomaGraph-Bench show improved task-relevant graph reconstruction and sequence-sensitive planning, while evaluations on BLINK and MMSI-Bench further demonstrate generalization to functional and spatial reasoning tasks.

Critical strengths
  1. The paper is well written and easy to follow, with clear and visually appealing figures and tables.
  2. The paper is technically substantial. The two proposed modules directly address the two identified limitations of existing methods, and the ideas are straightforward yet effective.
Critical weaknesses

The comparative evaluation is somewhat limited.

  1. Although the paper compares against both open-source and closed-source models, it includes only one method from prior work as a direct baseline and they are original VLM models, what about some professional designed VLM-based methods, e.g. GRID2024@IROS and others.
  2. The paper emphasizes improving task relevance in scene graph construction, it would be valuable to include comparisons with similar graph construction methods from multi-view inputs, e.g. DSM2025RAL, to better prove the advantages of the proposed approach.
Questions for authors

The paper includes simulation experiments in the Supplementary_Material, which is good. However, it would be helpful to clarify whether the method has also been evaluated on a real robotic platform.

4 comments
AuthorsOpenReview ↗

Response to Reviewer zbCL (1/3)

W1 : Although the paper compares against both open-source and closed-source models, it includes only one method from prior work as a direct baseline and they are original VLM models, what about some professional designed VLM-based methods, e.g. GRID2024@IROS and others.

We thank the reviewer for suggesting additional baselines. We agree that including more VLM-based scene graph generators would strengthen the evaluation. We therefore added R1-SGG [1], SceneGraphVLM-PSG [2], and LLaVA-SpaceSGG [3] to our MomaGraph-Bench comparison, which already included MomaGraph-R1.

GRID (IROS 2024) [4] addresses task planning given an existing scene graph. It takes a natural-language instruction, a scene graph, and a robot graph as input and predicts subtasks specified by predefined actions and target objects. TaoGraph addresses the preceding stage of generating task-oriented scene graphs from multi-view observations, with the goal of capturing information that is useful for downstream planning. Since our comparison evaluates the graphs produced by each method, we selected R1-SGG, SceneGraphVLM-PSG, and LLaVA-SpaceSGG as direct baselines for scene graph generation.

Performance on each task type and overall on MomaGraph-Bench (%).

ModelsSpatialObjectActionAffordanceEffectSequenceOverall
R1-SGG19.9328.9440.7595.87100.0036.0050.27
SceneGraphVLM-PSG12.7514.2938.1195.87100.0033.3344.79
LLaVA-SpaceSGG10.1316.4838.4995.8799.4034.6744.64
Ours56.5492.3185.6699.0899.4073.3383.65

Our method achieves higher overall accuracy than R1-SGG, SceneGraphVLM-PSG, and LLaVA-SpaceSGG. Across task types, the gap mainly comes from Spatial Reasoning, Object Selection, Action Reasoning, and Sequence Planning, with the largest gap on Object Selection. The former four types of questions require the graph to contain task-relevant objects, interactive parts, and their spatial and functional relations, whereas the three added methods generate objects and relations for general scenes, and the advantage of our method is also mainly reflected on these tasks.

We will add the above comparison results and a discussion of GRID to the revised manuscript and cite the original GRID paper.

References

[1] Chen Z, Wu J, Lei Z, et al. Compile scene graphs with reinforcement learning[J]. arXiv preprint arXiv:2504.13617, 2025.

[2] Makarov V, Gizetdinov M, Yudin D. SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models[J]. arXiv preprint arXiv:2605.13667, 2026.

[3] Xu M, Wu M, Zhao Y, et al. Llava-spacesgg: Visual instruct tuning for open-vocabulary scene graph generation with enhanced spatial relations[C]//2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2025: 6362-6372.

[4] Ni Z, Deng X, Tai C, et al. Grid: Scene-graph-based instruction-driven robotic task planning[C]//2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024: 13765-13772.

AuthorsOpenReview ↗

Response to Reviewer zbCL (2/3)

W2 : The paper emphasizes improving task relevance in scene graph construction, it would be valuable to include comparisons with similar graph construction methods from multi-view inputs, e.g. DSM2025RAL, to better prove the advantages of the proposed approach.

We thank the reviewer for suggesting additional multi-view baselines. We agree that comparing methods that construct scene graphs from multi-view observations helps assess how well their graphs support downstream task reasoning. We therefore added ConceptGraphs (ICRA 2024) [1] and OpenFunGraph (CVPR 2025) [2] as baselines on MomaGraph-Bench.

DSM (RA-L 2026) [3], suggested by the reviewer, is relevant to our work because it also constructs a scene representation from multi-view observations. It builds a semantic map that incorporates appearance, physical, and affordance attributes to support 3D visual grounding. We could not locate a public implementation of DSM to evaluate on MomaGraph-Bench. We therefore used the publicly available implementations of ConceptGraphs and OpenFunGraph to compare TaoGraph with other multi-view scene graph construction methods. Once the DSM implementation is publicly released, we will promptly evaluate it on MomaGraph-Bench and include the results in the revised manuscript.

Performance on each task type and overall on MomaGraph-Bench (%).

ModelsSpatialObjectActionAffordanceEffectSequenceOverall
ConceptGraphs21.5721.2541.1395.87100.0036.0049.13
OpenFunGraph32.3553.8557.7494.5097.6248.0061.90
Ours56.5492.3185.6699.0899.4073.3383.65

The overall accuracy of our method is 34.52 and 21.75 percentage points higher than that of ConceptGraphs and OpenFunGraph, respectively. Across task types, the gap mainly comes from Spatial Reasoning, Object Selection, Action Reasoning, and Sequence Planning, with the largest gap on Object Selection (71.06 and 38.46 percentage points, respectively); on Affordance Reasoning and Effect Prediction, all three methods are above 94% and perform similarly. Between the two added methods, OpenFunGraph, whose graphs contain interactive parts and functional relations, is higher than ConceptGraphs on all of the former four task types, which is consistent with the importance our work places on interactive parts and functional relations. Both methods construct graphs of the whole scene, whereas our method generates task-oriented scene graphs around the task instruction and achieves higher performance on these tasks.

We will add the above comparison results and a discussion of DSM to the revised manuscript and cite the original DSM paper.

References

[1] Gu Q, Kuwajerwala A, Morin S, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning[C]//2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024: 5021-5028.

[2] Zhang C, Delitzas A, Wang F, et al. Open-vocabulary functional 3d scene graphs for real-world indoor spaces[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 19401-19413.

[3] Xie Q, Liang Z, Li F, et al. DSM: Constructing a Diverse Semantic Map for 3D Visual Grounding[J]. IEEE Robotics and Automation Letters, 2026, 11(5): 6344-6351.

AuthorsOpenReview ↗

Response to Reviewer zbCL (3/3)

Q : The paper includes simulation experiments in the Supplementary_Material, which is good. However, it would be helpful to clarify whether the method has also been evaluated on a real robotic platform.

We thank the reviewer for this question. The robot experiments in the submitted version were conducted in the ManiSkill simulation environment and do not yet include an evaluation on a real robotic platform. We are currently conducting real-robot experiments. Since deploying and debugging the real-robot platform takes some time, the results cannot be provided in this response yet. We plan to report the real-robot results during the rebuttal period and include them in the revised manuscript.

In Appendix G, we present two real-robot demonstrations of TaoGraph on a Franka Research 3 arm. Following the Graph-then-Plan protocol, TaoGraph generates a task-conditioned scene graph from two camera views and the task instruction. The planner uses the graph and instruction to select an interaction target and action. The selected action is then executed on the robot via teleoperation.

Spatial task: The instruction was to place the white stick second closest to a foam box into the box. The scene graph represented the three sticks as separate instances and linked the selected stick to the box. The robot picked up the middle stick and placed it in the box. This example shows how the graph represents the target and destination, although it does not explicitly encode the full distance ordering.

Functional task: The laptop charger was already connected, but the power strip was switched off. The planner selected pressing the switch, and the laptop displayed its charging indicator after the press. The scene graph visualized in Appendix G shows the switch and its control relation to the power strip, illustrating how the representation can capture the role of an interactive part.

Appendix G provides the experimental setup, scene graph visualizations, and execution keyframes. Given the limited time during the review period, our real-robot testing was limited to these two preliminary demonstrations. We plan to conduct a more systematic evaluation with a broader range of tasks and repeated trials across different scene configurations.

Reviewer_mT1eOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies task-oriented scene graph learning for embodied learning. Semantics-Induced Spatial Reorganization (SISR) uses task and scene semantics to modulate geometric features before graph-token construction. Change-Aware Procedure Distillation (CAPD) trains a procedure-enriched graph teacher, uses its generated graphs as student targets, and increases exposure to samples where its canonicalized graph differs from a standard teacher's graph. The paper reports graph reconstruction, multiple-choice planning, external visual reasoning, and sequence-sensitive results.

Critical strengths
  1. The problem is well motivated, and the representation pipeline is coherent.
  2. SISR has useful empirical support.
  3. The CAPD ablations and evaluation definitions are informative.
  4. The structure of the gap analysis is clear.
  5. The experimental improvements are impressive.
Critical weaknesses
  1. The design of both modules lacks clear analysis and reasoning over learning mechanism as motivation.
  2. It is doubtful whether the classification task in CAPD introduces additional information, which may cause unfair comparison.

Minor issues:

  • Figure 3 gives an illustrative improvement involving an interactive part, but the results do not establish how often changes improve procedure-relevant relations, rather than reflecting neutral differences, errors, or ordinary variation between separately trained teachers.
  • Table 6 reports only the three planning results. The conclusion states that CAPD "improves graphs' support for action-order reasoning", but what supports that sentence is downstream accuracy,
Questions for authors
  • Are Table 6 variants matched in optimizer updates and effective token budget? Can you provide a matched-budget uniform/random-duplication comparison with the same procedure-enriched targets?
  • On the changed samples, how often does the procedure teacher improve versus degrade graph correctness, particularly procedure-relevant relations? Is the signal stronger than disagreement between two standard teachers? Are all graph sources evaluated by the same frozen planner/checkpoint? What is the procedure head's validation accuracy?
10 comments
AuthorsOpenReview ↗

Response to Reviewer mT1e (1/6)

W2 : It is doubtful whether the classification task in CAPD introduces additional information, which may cause unfair comparison.

We thank the reviewer for raising this concern about the fairness of the comparison. To address it, we clarify how procedure-category classification contributes to CAPD’s supervision design and present a controlled ablation in which the baseline receives the same auxiliary classification supervision.

1. Role of classification in CAPD

Existing scene graph methods differ in how they organize supervision, and the improvement of CAPD is at this level as well:

MethodSupervision
R1-SGG [1]Supervised fine-tuning (SFT) for scene graph generation, followed by reinforcement learning that refines the matching of objects and relations
MomaGraph-R1 [2]Reinforcement learning to train a task-oriented scene graph generation model
Ours (CAPD)Beyond graph-generation supervision, an auxiliary procedure-category objective trains the procedure teacher; the teacher's generated graphs supervise the student; training exposure is allocated according to the differences between the two teachers' graphs

Conventional graph supervision does not explicitly target the interactive parts and functions needed to determine action order, so CAPD incorporates this procedural knowledge into graph supervision through the auxiliary procedure-category objective.

2. Source of procedure labels

We derive procedure-category labels using deterministic rules based on the task instruction and the function type, action type, and functional relations in the annotated training graph (Appendix C). The labels provide an additional training signal derived entirely from the existing graph training data, without additional manual annotations, action trajectories, or downstream question and answer data.

We apply the auxiliary classification loss only when training the procedure teacher (Eq. 5). The CAPD student learns from the teacher’s generated graphs. At test time, it generates scene graphs directly from multi-view observations and the task instruction.

3. Ablation with matched supervision

We add the auxiliary procedure-category objective to Qwen2.5-VL-7B + Graph SFT (i.e., Qwen2.5-VL-7B fine-tuned on the graph-generation training data). The results are as follows (%).

MethodSeq. Sub.Seq. Plan.Non-Seq.
Qwen2.5-VL-7B + Graph SFT67.0764.0077.13
+ Procedure Auxiliary Loss70.7368.0077.29
Ours75.6173.3384.18

The auxiliary procedure-category objective improves the Seq. Sub. and Seq. Plan. accuracy of this baseline by 3.66 and 4.00 percentage points, respectively, showing that this training signal is indeed effective. Meanwhile, the baseline that receives the same category supervision remains 4.88 and 5.33 percentage points below the full model on Seq. Sub. and Seq. Plan., respectively.

In summary, the classification task is a component of the proposed supervision scheme, and the full model retains its advantage when the baseline receives the same category supervision. In the revised manuscript, we will clarify how the procedure labels are derived and used in CAPD and report the ablation results.

References

[1] Chen Z, Wu J, Lei Z, et al. Compile scene graphs with reinforcement learning[J]. arXiv preprint arXiv:2504.13617, 2025.

[2] Ju Y, Liang Y, Wang Y J, et al. Momagraph: State-aware unified scene graphs with vision-language models for embodied task planning[C]//International Conference on Learning Representations. 2026, 2026: 96361-96386.

AuthorsOpenReview ↗

Response to Reviewer mT1e (2/6)

Minor issue 1 : Figure 3 gives an illustrative improvement involving an interactive part, but the results do not establish how often changes improve procedure-relevant relations, rather than reflecting neutral differences, errors, or ordinary variation between separately trained teachers.

We thank the reviewer for this comment. We agree that Figure 3, as a single illustrative example, is not sufficient to establish how common the changes are or what their nature is. To answer how often changes improve procedure-relevant relations, rather than reflecting neutral differences, errors, or ordinary variation, we add analyses from the following two aspects.

1. Significance analysis of procedure-relevant changes

We compare the graphs of the two teachers against the ground-truth graphs of the training split and analyze only procedure-relevant changes, i.e., samples whose action type, function type, or functional relations (interactive part, target object, relation type) change; there are 219 such samples. For each sample, we compute the difference in procedure score, Δ = score of the procedure teacher − score of the standard teacher, where the procedure score consists of three terms: action type, function type, and functional-relation F1.

Ordinary variation between two separately trained models has no fixed direction, so the numbers of improved and degraded samples would be roughly equal. We therefore use three complementary paired tests to examine whether Δ is systematically shifted to one side:

TestStatisticResult
Sign testimproved 115 / degraded 63 / tied 41p = 1.2×10⁻⁴
Wilcoxon signed-rank test (also accounts for the magnitude of the change)z = 4.95p = 7.6×10⁻⁷
Paired sign-flip permutation test (20,000 permutations; no distributional assumption)mean Δ = +0.112p < 1×10⁻⁴

Effect sizes: among the samples with Δ ≠ 0, 64.6% are improvements (bootstrap 95% CI: 57.3%–71.3%, excluding 50%); the mean Δ is +0.112 (95% CI: 0.070–0.154, excluding 0); the ratio of improved to degraded samples is 1.83.

Testing each field separately (counting only the samples in which that field actually changes; numbers are improved / degraded) gives the same direction:

  • Action type: 44 / 20, p = 0.004
  • Function type: 20 / 1, p = 2×10⁻⁵
  • Functional relations: 79 / 48, p = 0.008

In summary, the three tests agree: the procedure-relevant changes are significantly shifted toward better agreement with the ground truth, rather than being random differences.

2. Sequence Planning case analysis

We examined all seven Sequence Planning questions that the full model answered correctly and the model without CAPD answered incorrectly. In each case, the graph changes concerned the interaction between an interactive part and its target object.

In two cases, the graphs differed in the action type or the interactive part. In bench_001108, the action type changed from press to pull, consistent with the correct answer, "Pull the knob, then open the door." In bench_001048, the model without CAPD represented the interactive part with two nodes, knob1 and knob2, whereas the full model used a single knob node, as the question and its options refer to it. In two further cases (bench_000257 and bench_001063), the part, action type, and functional relation were the same, and the graphs differed in the spatial relations on the edge between the interactive part and its target object. Each option in these questions is a two-step sequence that first acts on the interactive part and then operates the target object, and these spatial relations are the graph's description of where the part acted on in the first step is located relative to that object. We have added these four examples to Appendix F.3 of the revised manuscript on OpenReview.

We will add the above significance analysis and case analysis to the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer mT1e (3/6)

Minor issue 2 : Table 6 reports only the three planning results. The conclusion states that CAPD "improves graphs' support for action-order reasoning", but what supports that sentence is downstream accuracy,

We thank the reviewer for this comment. We understand that the reviewer raises an important and reasonable concern about the evidence supporting our statement that CAPD “improves graphs’ support for action-order reasoning.” As we explain below, our downstream planning experiments can, to some extent, establish a link between improvements in the generated scene graphs and gains in action-order reasoning performance.

Table 6 reports multiple-choice accuracy on MomaGraph-Bench under the Graph-then-Plan process, with the serialized scene graph serving as the planner’s only visual evidence (Appendix A). Seq. Plan. measures accuracy on the benchmark’s Sequence Planning category, whose questions ask for the correct sequence of actions. Seq. Sub. covers a broader set that includes Sequence Planning questions and other questions with explicit ordering cues (Appendix D.2).

For example, a question about opening a kitchen cabinet asks whether to push, pull, turn, or press the knob before opening the door. The scene graph can help identify the required first action by representing the knob, its functional relation to the cabinet, and the associated action.

To assess how the generated graphs affect action-order reasoning, we varied only the input graphs within each controlled comparison, holding the planner checkpoint, questions, prompts, and decoding settings fixed. The table reports accuracy (%).

ComparisonPlannerSeq. Sub.Seq. Plan.
w/o CAPD → Ours (Table 6)each model's own checkpoint68.29 → 75.6165.33 → 73.33
w/o CAPD graphs → CAPD graphsfrozen: w/o CAPD68.29 → 73.1765.33 → 70.67
w/o CAPD graphs → CAPD graphsfrozen: CAPD71.95 → 75.6169.33 → 73.33
standard teacher's graphs → procedure teacher's graphsfrozen: standard teacher65.85 → 70.7362.67 → 68.00
standard teacher's graphs → procedure teacher's graphsfrozen: procedure teacher69.51 → 71.9566.67 → 69.33

In all four frozen-planner comparisons, switching to the CAPD graphs or the procedure teacher's graphs improves both Seq. Sub. and Seq. Plan. That is, the sequence metrics in Table 6 directly evaluate action-order reasoning, and with the planner held fixed, these metrics improve when the graphs are replaced; together, the two support the statement in the conclusion.

We will add the above clarification and the frozen-planner results to the revised manuscript.

Comments on the paper

1 comment

Dear reviewers, the authors have posted their responses to your reviews. Please have a look and discuss ASAP as the author-reviewer discussion period ends at November 18 at 11:59pm AoE

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.