acceptodds
Under review as a conference paper at ICLR 2027

Probabilistic Knowledge Graphs for Embodied AI: Joint Inference via Message Passing over Entities, Relations, and Affordances

Abstract

Embodied agents that operate in open worlds must keep structured beliefs about objects, their relations, and their actionable properties while observations remain partial and noisy. Recent work encodes such beliefs as knowledge graphs whose nodes and edges carry uncertainty, yet inference stays local. Each entity is filtered independently, beliefs are never propagated along graph edges, and relation evidence is not fed back into entity states. As a result coupled errors are underestimated and the model cannot return calibrated marginals that tell the agent when another observation is worth its cost. We treat the embodied knowledge graph as a factor graph and perform joint inference by message passing. Observation, relation, temporal, and affordance potentials define a joint distribution over entities, relations, and affordances. Closed form Gaussian messages handle the linear potentials, and a variational linearization of the relation energy yields messages that close the loop between entities and relations. The inferred marginals are calibrated and drive active perception and robust anomaly detection. Across three question answering benchmarks, embodied scene understanding, affordance prediction, active perception, and physical adversarial attacks, ProbKG improves accuracy and calibration over a broad set of baselines, needs fewer observations to reach a fixed confidence, and recovers scene semantics faster under attack.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Knowledge Graph, Embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2523OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formalizes the embodied knowledge graph as a factor graph. It gives explicit observation, relation, temporal, and affordance potentials together. Authors try to derive the message passing inference in closed form for the linear Gaussian potentials and through a second order variational linearization for the relation potential. The method is well evaluated on three question answering benchmarks, embodied scene understanding, affordance prediction, and physical adversarial attacks.

Critical strengths
  1. The motivation for jointly reasoning about entities, relations, and affordances is great. Relational evidence can help refine uncertain entity states. The factor-graph formulation provides an explicit framework for combining observational, relational, temporal, and affordance constraints.

  2. Experiments cover a broad range of downstream tasks, including QA, relation inference, manipulation, and active perception.

Critical weaknesses
  1. Section 3.3 and Eq. (16) describe fully factorized mean-field updates based on the exponential of an expected log factor, whereas the derivation leading to Eq. (18) appears to marginalize the neighboring Gaussian belief, yielding a correction. These operations generally lead to different updates, and it is unclear how Eq. (18) follows from the stated ELBO or whether an additional approximation is introduced. Since this correction underlies the claimed propagation of neighbor uncertainty, clarifying this connection would help assess the method’s probabilistic interpretation.
Questions for authors
  1. Could authors clarify how the relation message in Eq. (18) follows from the expected-log mean-field update in Eq. (16)? In particular, does the neighbor marginalization introduce an additional approximation, and how does the resulting update relate to the stated ELBO?
Reviewer_nrp3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

To address local inference in embodied knowledge graphs, where entities are filtered independently and relation posteriors do not feed back into entity states, this paper proposes PROBKG, which performs joint inference via variational message passing. PROBKG treats the knowledge graph as a factor graph and uses a second-order linearization of the relation energy to propagate the mean and covariance of entity beliefs and yield calibrated relation beliefs.

Critical strengths

This paper addresses an important problem in embodied AI: how uncertainty about entities and their relations can inform perception and decision-making.

Critical weaknesses
  1. The writing is overloaded with concepts and lacks a clear logical progression, making it difficult for readers to follow the authors’ argument. For example, “A second order linearization of the relation energy propagates both the mean and the covariance of entity beliefs, closes the loop between entities and relations, and yields calibrated relation beliefs.”
  2. The related work discussion and experimental comparisons focus mainly on classical methods, while coverage of recent studies directly relevant to the paper’s central problem is quite limited.
  3. The mathematical justification of the proposed inference procedure is insufficiently established. For example, Eq. (16) uses a mean-field update, while Eq. (18) follows from marginalizing the factor. These operations generally give different updates.
Questions for authors
  1. Please see the weaknesses.
  2. Based solely on the manuscript, there are strong signs of AI-assisted figures and writing, yet the paper lacks an adequate AI use statement.
1 comment

Thank you for your detailed response. It has resolved some of my questions. However, I still have some remaining concerns. For example, when the authors compare with top-venue methods, they refer to "[method] (adapted)." What does "adapted" mean here? In addition, I consider that a considerable amount of revision would be needed to express the logic of this paper more clearly. Therefore, I am inclined to maintain my current score.

Reviewer_5JsSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

I write this review as a person who is not an expert in learning on graphs, but is familiar with embodied AI applications. Please take my opinions accordingly.

As per my understanding, the work is motivated by having a structured model of their environment to enable good task execution. One way to build such a structured model is a graph over the entities in the agent’s environment, where each node can describe various attributes about the entity, such as object type and affordances, and edges can describe relations between these entities. A challenge in the real world is to construct this graph without ground-truth supervision, from just noisy observations. If such an embodied knowledge graph (as the authors call it) exists, it can help task execution fairly well, as also validated by their experiments on knowledge-based question answering and embodied scene understanding. The core contribution over prior work is that instead of updating each node or edge individually, they allow updates from one node to propagate to another connected node to reflect the global state of things, as opposed to local confidence for just an object. They do this using a probabilistic graphical model. They use the uncertainty over their graph from this model to collect better observations (active perception) and use it for anomaly detection.

Critical strengths
  • The core point of the work, which is that instead of updating node beliefs individually, we need to consider updating in relation to other nodes it's connected to, makes a lot of sense. Updating beliefs in isolation is unrealistic in physical environments where things are bound to affect each other.
  • Some empirical results and ablations support the claim that joint inference improves performance. Specifically, it's cool to see (as expected) that without message passing, the performance drops to basically the independent filtering baseline.
Critical weaknesses
  • In various results (Table 3, Table 4, Figure 4 (a,c)), the gains are relatively minor. It seems that there was not much discussion around how meaningful the gain is on these benchmarks in the draft.
  • Perhaps it's a naive question, but since the authors are tracking a more global state of things, does it incur extra computational cost to update the graph?
  • (Minor) Figure 4, when printed, is basically not visible.
Questions for authors
  • Besides the points raised in the weaknesses, one question for the authors is that the concept is theoretically interesting and it makes sense, but the evaluations chosen here do not show its effect as clearly, making it look unconvincing. What application, as practitioners, should we be looking for to be convinced that going beyond local updates is useful? I’d be willing to improve my rating if I can be convinced that the work in its current form can include an eval that justifies the idea more strongly. Note that this should not be taken as an incentive to improve scores on the current benchmark, but more to find better use cases for this method.
  • How exactly is the manipulation task in AI2THOR implemented? There was no description of the action space or how the graph is used to guide the predictions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.