- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a structured Theory-of-Mind framework that models human mental states as dynamic belief graphs to capture multiple interdependent beliefs and their temporal evolution. It combines LLM-derived semantic embeddings with an energy-based factor graph to model belief dependencies, employs variational learning with an ELBO objective to infer latent beliefs from observation-action trajectories without belief-level supervision, and uses action-specific self-attention for action prediction. Experiments on real-world wildfire evacuation datasets and standard ToM benchmarks demonstrate improvements in action prediction and alignment with human-reported belief states.
Critical strengths
The paper addresses a practically important problem: human decisions are influenced by multiple interdependent beliefs that evolve with new observations. The proposed framework provides a reasonable connection between this motivation and its technical components, using factor graphs for belief dependencies, temporal transitions for belief evolution, and variational learning to address the lack of belief-level supervision.
Critical weaknesses
-
The paper emphasizes learning time-varying belief interactions, but the pairwise potential in Eq. (5) is computed from LLM embeddings of belief pairs without explicitly conditioning on current observations or previous belief states. It is therefore unclear whether the model learns genuinely dynamic interaction strengths or primarily captures time-varying marginal beliefs through the unary potentials.
-
Although the factor graph models a joint distribution over interdependent beliefs, the action-specific self-attention module in Eq. (6) receives only marginal belief probabilities. Different joint distributions can share identical marginals, potentially losing dependency information relevant to decisions.
Questions for authors
- How are time-varying belief dependencies explicitly modeled? In Eq. (5), the pairwise potential appears to depend only on the semantic representations of belief pairs. Could the authors clarify how these interactions vary with observations and previous belief states, and distinguish changes in pairwise potentials from changes in the resulting joint belief distribution?
- Why does the action prediction module use only marginal belief probabilities rather than the joint belief distribution? Since the factor graph explicitly models belief dependencies, marginalization may discard information about joint configurations. Could the authors explain this design choice and evaluate whether incorporating joint belief conf?
2 comments
Thank you to the author for the clarification. These explanations have made some technical details of the paper clearer, and I believe it is a standard paper; therefore, I tend to maintain my score(3).
We thank the reviewer for the thoughtful questions and constructive feedback. We address each point below and will incorporate the corresponding clarifications into the revised manuscript.
For weakness 1. We would like to clarify that our belief interaction modeling explicitly considers observations, and the pairwise interaction strengths change over time as new observations arrive. In our survey setup, each time step corresponds to a new observation, such as observing nearby fire cues, receiving an official evacuation warning, or receiving a warning from a familiar person. Specifically, in Eq. (5), the current observation ot and previous belief state bt−1 are the conditioning inputs defined in Eq. (1), while the pairwise potential ϕij corresponds to the belief interaction term in Eq. (2). These equations define how the current observations and previous belief states are incorporated into the belief distribution, together with the pairwise interactions. We will update the explanation of Eq. (5) to explicitly clarify how these terms are connected and where the conditioning takes place.
We further provide empirical evidence that the pairwise interaction strengths ϕij change over time as new observations arrive. For example, when residents receive an official warning after observing nearby fire cues, the interaction between home damage and death of others increases by 14.35%, while the interaction between personal injury and personal death decreases by 11.74%. The interaction between home damage and neighborhood damage also increases by 11.21%. This suggests that official warnings strengthen the connections between property-related and broader community risks, while the relationship between personal injury and death becomes less pronounced. After receiving a warning from a familiar person, these interactions change again by −2.22%, +1.78%, and −0.89%, respectively, suggesting that additional social information further adjusts these belief relationships, although the changes are smaller than those following official warnings.
For weakness 2. In our action predictor, the action-specific self-attention mechanism captures how multiple beliefs jointly influence actions by learning the dependencies among individual beliefs' contributions to action prediction. As described in Section 3.2 (Eq. 6), our action-specific self-attention mechanism learns how the contribution of one belief to an action depends on the activation of other beliefs. Specifically, the attention weight At,j[i,k] captures how belief k influences the contribution of belief i when predicting action j. By learning these interdependencies directly through self-attention, the action predictor can capture how beliefs jointly influence decisions using marginal belief probabilities, without evaluating all 2K configurations of the full joint belief distribution, substantially reducing computational costs. We will further explain this design choice of action predictor in Section 3.2 (Action Generation), explaining why we use marginal belief probabilities with self-attention instead of the full joint distribution for action prediction.
For question 1. Please see our response to Weakness 1, where we address this concern in detail. We will update Section 3.2 (Action Generation)
For question 2. Please see our response to Weakness 2, where we address this concern in detail.