acceptodds
Under review as a conference paper at ICLR 2027

CausalScene: Explicit Intervention-Response Structure for Counterfactual Reasoning in Embodied Agents

14 views

Abstract

3D scene graphs capture where objects are, but not what happens when an agent acts on them. We introduce **CausalScene**, built on the *Causal 3D Scene Graph* (C3SG), which augments spatial scene graphs with directed, typed intervention-response edges (Aff, Fc, Tmp, Co), each carrying a strength estimated from simulation and validated through held-out intervention experiments. A **Causal Edge Predictor** (CEP) populates these edges by fusing cross-attention over the scene, a physics prior from 50,000 rigid-body simulations, and distilled LLM commonsense. A **Causal Query Language** (CQL) then delegates traversal to a graph engine, so every answer arrives with the intervention-response chain that produced it, making path accuracy measurable. On five 3D and physical-reasoning benchmarks (CLEVRER, CRAFT, ScanQA, SQA3D, EmbodiedScan) against 29 baselines, CausalScene leads on every one: 61.2% on CLEVRER counterfactual QA (+8.8 pts), 58.7% on CRAFT (+8.6), 54.3% on ScanQA (+5.1), 57.1% on SQA3D (+5.3), 62.4% on EmbodiedScan (+10.3), and 58.3% on RLBench (+5.6). On a real Franka Emika Panda across 120 tabletop trials, intervention consistency is 58.7% (vs. 63.4% on a held-out simulator), and human raters find the chains plausible (3.6/5) and correct (54.2%). Finally, we show that under 96% label skew the predictor collapses below a constant baseline and that a cost-sensitive two-stage detector recovers F1 to 0.687.

Primary area:
causal reasoning
Keywords:
Causal Scene Graphs, 3D Scene Understanding, Embodied AI, Counterfactual Reasoning, Intervention-Response Dynamics, Physics-based Reasoning, Causal Query Language, Scene Graph Generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_z17UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CausalScene, a framework for reasoning about the physical consequences of actions in 3D scenes. Its core representation, the Causal 3D Scene Graph, augments spatial relations with typed, weighted intervention-response and contextual edges. A Causal Edge Predictor combines scene-wide attention, simulation-trained physics priors, and distilled LLM knowledge. A Causal Query Language executes graph traversal to produce answers with explicit supporting paths, while dynamic graph updates support sequential manipulation. The authors evaluate the framework on CLEVRER, CRAFT, ScanQA, SQA3D, EmbodiedScan, and RLBench. The paper also investigates predictor collapse under severe class imbalance and proposes a two-stage cost-sensitive detector to mitigate it.

Critical strengths
  1. The reasoning process is easier to inspect. The system returns the graph path it actually traverses, such as “removing the tray → the cup loses support → the cup falls.” When errors occur, they can be traced to object recognition, relation prediction, or path selection, rather than leaving only an unverifiable verbal explanation.
  2. The method integrates complementary information sources. Geometry describes the current scene, physical simulation provides priors on action consequences, LLMs contribute commonsense knowledge about long-tail interactions, and scene-wide attention accounts for the influence of a third object. This combination may help address the limitations of purely visual or language-based reasoning.
Critical weaknesses
  1. Estimating mass, friction, and density from images is inherently challenging, making manipulation based on these estimates unreliable. Moreover, the method’s physics prior is derived primarily from rigid-body simulations, limiting its applicability to deformable objects, complex contact interactions, and unseen materials.
  2. Comparisons with directly relevant methods such as ALOE [1] and VRDP [2] are missing. The current results are therefore insufficient to establish the method’s superiority on CLEVRER counterfactual question answering.
    [1] Ding, David, et al. "Attention over learned object embeddings enables complex visual reasoning." Advances in neural information processing systems 34 (2021): 9112-9124.
    [2] Ding, Mingyu, et al. "Dynamic visual reasoning by learning differentiable physics models from video and language." Advances in Neural Information Processing Systems 34 (2021): 887-899.
Questions for authors

The consequences of an action depend on its type, direction, magnitude, and other parameters. Gently pushing, pulling away, or lifting the same object may produce different effects. How are actions and their parameters encoded in the graph? How does the system retrieve the corresponding relations and response strengths for different actions applied to the same object?

Reviewer_kA2TOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CausalScene, a framework for counterfactual reasoning in embodied agents. The authors extend conventional 3D scene graphs by introducing explicit intervention-response relations through a Causal 3D Scene Graph (C3SG). A Causal Edge Predictor (CEP) combines cross-attention, simulation-based physics priors, and distilled LLM knowledge to predict causal relationships between objects. The framework further introduces a Causal Query Language (CQL) to support structured graph traversal and interpretable causal reasoning. Experiments on multiple 3D understanding, physical reasoning, and robotic manipulation benchmarks demonstrate improvements over existing baselines. The authors also conduct cross-simulator and real-robot experiments to evaluate the consistency of predicted causal relationships.

Critical strengths
  1. The paper addresses an important limitation of conventional 3D scene graphs, which primarily capture spatial relationships rather than the physical consequences of actions. Introducing explicit intervention-response relations is a reasonable direction for improving counterfactual reasoning in embodied agents. The attempt to provide traceable reasoning paths is also potentially valuable for robotic planning and decision-making.

  2. The authors evaluate the proposed framework across multiple benchmarks covering physical reasoning, 3D scene understanding, and robotic manipulation. The inclusion of cross-simulator validation, real-robot intervention experiments, and component ablations is valuable. In particular, the real-robot experiments provide useful evidence for assessing whether predicted intervention-response relations correspond to actual physical consequences.

Critical weaknesses
  1. The proposed framework mainly combines existing techniques, including 3D scene graphs, cross-attention, physics simulation, and LLM knowledge distillation. The core technical contributions are not sufficiently distinctive, and it remains unclear whether the complex architecture is necessary to achieve the reported improvements.

  2. The cumulative path-strength criterion defined in Theorem 1 appears inconsistent with the individual edge-thresholding procedure described in Algorithm 1, raising concerns about the theoretical guarantees. Additionally, Figure 1 is poorly organized, with inconsistent font sizes, crowded modules, and unclear visual hierarchy, making the core contributions difficult to identify.

Questions for authors

Q1. Definition 3 and Theorem 1 define valid causal paths using cumulative path strength, whereas Section G.1 and Algorithm 1 appear to use individual edge-strength thresholds during graph traversal. Could the authors clarify which criterion is actually implemented and explain how the stated theoretical guarantees hold under this implementation? Q2. Could the authors clarify the relationship between the 62.4% edge prediction accuracy reported on EmbodiedScan in Table 4 and the class-imbalance results in Section 5.1? Are these results obtained using different evaluation settings, metrics, or model configurations? Is the two-stage cost-sensitive detector included in the main experimental results? Q3. Given the relatively complex architecture, could the authors provide additional comparisons with simpler intervention-aware scene graph methods to better isolate the contribution of the proposed causal representation? Furthermore, how do the authors interpret the relatively modest real-world intervention consistency and path accuracy in terms of the reliability of the proposed framework?

Reviewer_9H5pOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper upgrades general 3D scene graphs into causal scene graphs, using physics simulation, commonsense knowledge, and a dedicated query language to construct and trace intervention-response chains between objects.

Critical strengths
  1. This paper identifies an important existing gap: current 3D scene graphs encode spatial configurations but do not encode the action outcomes that embodied agents need. Augmenting scene graphs with directed, typed intervention-response edges is an idea that makes sense.

  2. Graph traversal is delegated to CQL, while the LLM is only responsible for parsing and verbalizing. In principle, this makes the reasoning chain inspectable and supports path-level evaluation.

Critical weaknesses

[IMPORTANT]: The manuscript already violates ICLR rules: there is no “AI use statement.” I would suggest that the AC consider desk-rejecting this paper.

  1. There are many errors and unreasonable points in the mathematical proofs. For example, Theorem 6 proves an upper bound on PA (path accuracy): the paper claims that for a path of length k, its PA is less than or equal to the product of the intervention consistencies of its edges, i.e., PA ≤ IC_1 × IC_2 × … × IC_k. However, the reasoning in the proof runs opposite to the direction of an upper bound: the proof states that the inequality accounts for cases in which some edges are predicted correctly by chance. But correct predictions by chance can only increase PA. If PA increases by luck, the product may fail to bound it, and it would no longer be an upper bound. Therefore, the manuscript’s argument actually explains why this upper bound may not hold. This theorem has other problems as well; for instance, the two sides of the product do not seem to measure the same thing. PA in Definition 4 describes “whether the entire predicted path is exactly the same as the GT,” where both the nodes and the edge types must match. Definition 5, on the other hand, describes “whether the predicted outcome actually occurs after the action is performed.” The two measure different things, so I do not think the IC in Definition 5 can be used directly to bound PA. In addition, Assumption 2 assumes that “the strengths σ of the edges are independent of each other,” but this does not imply that “whether each prediction is correct” is independent, and the latter should be what the authors intend to use in Theorem 6.
  2. Could the authors provide PA stratified by path length k, together with the product of the per-edge intervention consistencies measured on the same paths? This would directly verify whether the bounds in Theorem 6 and Corollary 9 hold empirically. If they do not, please clarify whether the theorem or the reported results should be corrected.
  3. Line 1525 states: “We fine-tune GPT-4V+Proj. on the CLEVRER training split using LoRA with rank r = 16 and α = 32.” Later in the same paragraph, it states: “Fine-tuning was performed on 4 A100 GPUs for approximately 12 hours.” What confuses me is that OpenAI has not released the weights of GPT-4V, which can only be accessed through the API. Even if it could be fine-tuned, it would not be possible with GPUs and GPU hours on this scale. Therefore, I would like to ask the authors: what was fine-tuned, and how?
  4. I think the experimental results on ScanQA and SQA3D should be added to Table 7, because these two benchmarks are the tasks that least require causal edges. This would also help clarify why intervention-response edges improve their performance by more than 5%, which is currently unclear.
  5. ScanQA typically uses metrics such as EM@1, BLEU, and CIDEr. Why does the paper report only accuracy? I think it is necessary to provide these metrics for a fair comparison.
  6. The supplementary material seems to be a different version of the manuscript, because its content is inconsistent with the main text. For example, line 273 of the manuscript states: “On 500 real-world video events, the PPM achieves 74.2% state-transition accuracy, compared with 69.8% for a visual-only baseline (+4.4 points),” but the same experiment is described in the supplementary material as: “On 500 real-scene events annotated by human video review, the PPM achieves 79.4% state-transition accuracy vs. 74.1% for a visual-only baseline (+5.3 pts).” The state-transition accuracy changes from 74.2% to 79.4%, and the visual-only baseline numbers also differ (69.8% vs. 74.1%). This is only one example; there are many similar cases that the authors need to explain.
Questions for authors

As mentioned in the Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.