acceptodds
Under review as a conference paper at ICLR 2027

ChronoGraph: Functional 4D Scene Graphs with Vision-Language Models for Interaction Understanding and Grounded Planning

Abstract

Embodied agents must determine where to act, anticipate the resulting scene changes, and interpret observed outcomes to guide subsequent actions. This requires connecting 4D interaction understanding, which explains how past actions changed the scene, with spatially grounded planning, which determines how and where to act toward a goal and anticipates the resulting scene changes. We introduce ChronoGraph, a functional 4D scene graph that links actions on affordance parts to semantic and geometric state changes. By representing observed and anticipated transitions in the same form, it provides a shared basis for understanding and planning. We construct ChronoGraphBench through an automatic data engine that converts human-interaction videos and simulated robot trajectories into graph-annotated questions for training and evaluating Vision-Language Models (VLMs) on both tasks. Using these annotations, we train ChronoGraphVLM by adapting pretrained VLMs in two stages. Graph-as-Chain-of-Thought supervised fine-tuning teaches the models to reconstruct observed transitions and predict future ones as graph traces before answering. Subsequent joint 4D graph reinforcement learning directly rewards graph properties and answer correctness. Experiments across model scales show improvements over the corresponding pretrained baselines and zero-shot transfer to VLM4D. Real-world demonstrations further show that graph-based planning and affordance grounding support mobile manipulation through existing robot skills without additional fine-tuning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Scene Graph, Scene Understanding, Spatial Intelligence

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5EtjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ChronoGraph, a functional 4D scene graph representation that models objects, affordance parts, actions, semantic states, spatial relations, and action-induced state changes. Based on this representation, the authors construct ChronoGraphBench from HD-EPIC, Hoi!, and RoboCasa, and generate graph-annotated questions for interaction understanding and spatially grounded planning. ChronoGraphVLM is trained with graph-as-chain-of-thought supervised fine-tuning followed by joint 4D graph reinforcement learning. The paper reports improvements over pretrained baselines on ChronoGraphBench, zero-shot transfer to the egocentric split of VLM4D, and qualitative demonstrations on a mobile manipulation platform.

Critical strengths
  1. The paper proposes a coherent functional 4D scene graph that connects objects, affordance parts, actions, semantic states, and action-induced state changes. This is a meaningful formulation for bridging interaction understanding and grounded planning.
  2. The empirical setup spans egocentric human videos, simulated robot data, and a real-robot demonstration, with component ablations that provide useful evidence for the proposed training design.
Critical weaknesses
  1. The OOD evaluation is limited to VLM4D, which is still a closely related 4D understanding benchmark. Without before/after evaluation on broader spatial-understanding or VQA benchmarks, it is unclear whether post-training preserves the model's existing general capabilities, causes forgetting, or generalizes beyond the particular 4D task family.
  2. The benchmark relies centrally on a half-automatic scene-graph construction pipeline, but the paper provides limited quantitative validation of the resulting graph quality and no systematic error analysis on ChronoGraphBench. This limits confidence that the reported gains reflect robust functional 4D reasoning rather than sensitivity to annotation noise, templates, or the benchmark distribution.
Questions for authors
  1. Can the authors evaluate ChronoGraphVLM and the corresponding base models on several general spatial reasoning (such as VSI-Bench/EmbSpatial) and general VQA ( such as BLINK/V*) benchmarks? This would clarify both generalization and catastrophic forgetting.
  2. What fraction of automatically generated graph annotations are sent to human review, and how often do human reviewers modify nodes, edges, state transitions, 2D/3D boxes, or QA answers? Please report these statistics separately for HD-EPIC, Hoi!, and RoboCasa. Moreover, Can the authors provide a quantitative annotation audit, including semantic graph accuracy, temporal transition consistency, object identity consistency, 2D/3D grounding error, and QA validity, together with randomly sampled failure cases?
  3. Can the authors provide a systematic error pattern analysis on ChronoGraphBench, separating graph-generation errors from final-answer errors, so that we can know more about the boudnaries of model capacity, and what kind of capacities can(not) be improved through training on ChronoGraphBench?
Reviewer_9QokOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ChronoGraph, a functional 4D scene graph that represents objects, actionable parts, semantic states, geometry, and action-conditioned transitions Gt→atGt+1G_t \xrightarrow{a_t} G_{t+1}. It constructs ChronoGraphBench from human interaction videos and simulated robot trajectories, comprising 1,669 episodes, 5,143 graph states, and 15,094 QA pairs covering interaction understanding and static- or history-conditioned planning. ChronoGraphVLM is trained through graph-as-chain-of-thought supervised fine-tuning followed by GRPO with graph and answer rewards. The reported overall benchmark improvements are 24.0 and 24.1 points over the corresponding 9B and 4B initializations, with respective gains of 6.6 and 7.3 percentage points on the egocentric VLM4D split. Three qualitative Spot demonstrations illustrate integration with existing manipulation skills.

Critical strengths
  1. A useful connection between interaction understanding and planning. Representing observed and anticipated interactions through the same affordance-centered graph is well motivated. The annotations connect actionable parts with semantic and geometric consequences, providing richer supervision than action labels or static spatial relations alone. Combining human recordings and simulated robot trajectories makes the resulting resource potentially useful across perception and embodied reasoning.

  2. Substantial empirical improvements with informative training ablations. Improvements appear at both model scales and extend beyond the newly constructed benchmark. Table 3 provides useful evidence for structured supervision: graph-as-CoT SFT reaches 69.9 on ChronoGraphBench and 64.2% on VLM4D, versus 67.0 and 49.4% for answer-only SFT. Adding graph rewards subsequently improves the full model beyond answer-reward-only RL. These results support the value of graph supervision and suggest benefits beyond fitting the benchmark's answer distribution.

Critical weaknesses
  1. The accuracy and individual contributions of the generated 4D representation are insufficiently characterized. Table 3 provides useful evidence that graph supervision and graph rewards improve downstream performance. However, it does not isolate the contributions of explicit temporal transitions and metric geometry. Appendix E reports improvements in graph-quality rewards during RL, but does not clearly specify whether these statistics are measured on held-out episodes; Appendix F provides qualitative comparisons. A separate held-out evaluation of reconstructed and predicted graphs, including semantic transitions and geometric errors, would make the central representation claim substantially more convincing. The current evidence supports the combined training framework more strongly than it identifies which elements of the functional 4D representation drive its benefits.

  2. Dependence on interaction history and robustness during execution remain incompletely validated. The history-conditioned planning average improves substantially, from 51.4 to 66.8 for the 9B model, and this is a meaningful strength. However, Section 4.2 does not report the development experiments supporting its assertion that longer-horizon questions require temporal evidence. Comparisons against text-only, latest-frame-only, or temporally perturbed inputs would clarify how much improvement depends on interpreting interaction history. Separately, Section 6.5 demonstrates integration with robot skills through three qualitative tasks, but provides no repeated-trial success rates or controlled execution comparisons. These demonstrations establish feasibility while leaving the reliability and comparative benefit of the approach during sequential execution uncertain.

Questions for authors
  1. Contribution clarification and controlled comparison: What specifically distinguishes ChronoGraph from MomaGraph's action-conditioned graph updates? Can you provide comparisons using the same backbone, training examples, and comparable optimization budgets against semantic-only temporal graphs and static graphs, isolating temporal transitions and metric geometry?

  2. Representation validation: Can you report held-out graph accuracy separately for observed and predicted states, including entity/relation correctness, semantic transitions, and 3D localization or displacement errors? What quantitative audit establishes the reliability of the automatically reconstructed geometric annotations?

  3. Planning-evaluation clarification: How does performance change when history-conditioned questions receive only the goal and text, only the latest available frame, or temporally shuffled frames? How are distractors and reference graph continuations checked to avoid penalizing alternative actions or successor states that also satisfy the goal?

  4. Execution evidence: How many robot trials were attempted, with what task-success rates, interventions, and failure modes? Are comparisons available against the pretrained model or a graph-free variant using the same robot skills?

These requests concern evidence needed to assess the current claims, rather than suggestions to expand the scope of future work.

Reviewer_TtbBOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ChronoGraph, a functional 4D scene graph that connects interaction understanding and grounded planning through action-conditioned semantic and geometric state transitions. The authors construct ChronoGraphBench with 1,669 interaction samples and 15,094 VQA pairs from human interaction videos and simulated robot trajectories. They further train ChronoGraphVLM using Graph-as-Chain-of-Thought supervised fine-tuning followed by reinforcement learning with graph- and answer-level rewards. Experiments show improvements across model scales, zero-shot transfer to VLM4D, and qualitative real-world mobile manipulation demonstrations.

Critical strengths
  1. Representing affordance-level actions and their induced semantic and geometric changes within the same 4D graph is an interesting formulation. It provides a common representation for reconstructing observed interactions and predicting future scene states, rather than treating interaction understanding and planning as separate problems.

  2. ChronoGraphBench provides fine-grained supervision that combines affordance grounding, spatial changes, and action-conditioned state transitions. The combination of human interaction recordings and simulated robot trajectories, together with static- and history-conditioned planning tasks, makes the benchmark useful for studying interaction-aware reasoning.

  3. The ablations in Table 3 provide meaningful evidence for graph-guided training beyond in-domain improvements. In particular, Graph-as-CoT SFT achieves 64.2% on VLM4D compared with 49.4% for answer-only SFT, while adding graph rewards further improves cross-benchmark accuracy from 65.4% to 67.1%.

Critical weaknesses
  1. Since the central contribution is the modeling of action-conditioned 4D graph evolution, I would like to see a comparison against a simpler static functional scene graph without explicit state transitions. Table 3 validates graph-guided supervision and rewards, but does not isolate how much the temporal transition representation itself contributes.

  2. The benchmark questions are generated using 62 templates, while the visual-dependence filtering in Section 4.2 is applied only to atomic action questions. Extending the text-only evaluation to history-conditioned and static-conditioned planning would help determine how much of the benchmark requires visual reasoning rather than relying on goal descriptions or task priors.

  3. The quality of generated functional 4D graphs is mainly illustrated through qualitative examples and the graph reward progression during training. I would appreciate a quantitative evaluation of node and relation accuracy, action-conditioned state transitions, and 3D localization on the held-out set, independently of the final VQA scores.

  4. For 2D/3D grounding, the evaluation assigns full credit to predictions anywhere inside the target bounding box, although the questions ask for precise interaction centers. Reporting center localization error and breaking down performance by affordance size would provide a more informative assessment of spatial grounding precision.

  5. The real-world manipulation results in Figure 3 are encouraging, but they cover three qualitative demonstrations using existing robot skills. Repeated trials with success rates and a graph-free planning baseline would help distinguish the contribution of the learned graph representation from the underlying planning and execution pipeline.

Questions for authors

pls see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.