acceptodds
Withdrawn from ICLR 2027

Beyond Understanding: Learning Neuro-Symbolic Representations for Agentic Intelligence

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal foundation models learn powerful representations across heterogeneous modalities, but representations optimized for semantic understanding may be insufficient for agentic intelligence, where agents must reason over evolving environments, predict future states, and make goal-directed decisions. We introduce Neuro-Symbolic Multimodal Representation Learning for Agentic Intelligence, a paradigm that jointly learns neural and symbolic representations capturing semantic, relational, temporal, causal, and action-relevant structure. Our framework forms an evolving internal world representation in which symbolic structures guide neural learning, prediction, reasoning, and action, while new observations continuously refine the symbolic state. We propose an evaluation framework that shifts multimodal representation learning from semantic alignment toward prediction and decision utility, measuring compositional generalization, robustness to multimodal uncertainty, temporal prediction, and downstream decision-making. Our work establishes a foundation for multimodal representations designed not only for understanding the world, but for acting within it.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Neurosymbolic AI, Multimodal representation, Agentic Intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_22ipOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Neuro-Symbolic Multimodal Representation Learning (NSMR), a framework that integrates neural perceptual features, symbolic relations, temporal world states, and uncertainty estimation for agentic decision-making. The method combines goal-conditioned multimodal fusion, symbolic structure induction, temporal prediction, and action-aware representation learning. Experiments on CLEVRER, NExT-QA, ALFWorld, and additional benchmarks report improvements in reasoning, compositional generalization, robustness, and task success.

Critical strengths
  1. Relevant Research Motivation. The paper addresses an important distinction between semantic understanding and decision-relevant representation learning. Incorporating relational, temporal, and action-relevant information into agent representations is a meaningful research direction.

  2. Broad Evaluation Scope. The evaluation attempts to cover multiple aspects of agentic intelligence, including compositional reasoning, temporal prediction, robustness, and sequential decision-making. Such a comprehensive evaluation could be valuable if the experimental protocols and comparisons were adequately established.

Critical weaknesses
  1. Questionable Experimental Validity. Several baselines are fundamentally mismatched with the evaluation tasks. Laytrol is an image generation method, while Agent-X is an evaluation benchmark, yet both are reported as direct baselines without explaining their adaptation. Tables 10 and 11 also report inconsistent CLEVR accuracies (NSMR: 93.4% vs. 90.8%). These issues seriously undermine the credibility of the experimental results.

  2. Insufficient Methodological Rigor and Novelty. NSMR primarily combines established components without clearly demonstrating a novel technical mechanism. Moreover, Eq. 16 introduces an apparent circular dependency with Eq. 2, while Theorem 3 assumes a different objective from Eq. 22. These inconsistencies weaken the theoretical justification and reproducibility of the method.

Questions for authors

List above

Reviewer_9cwdOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

NSMR combines neural features, a symbolic graph, temporal state, and uncertainty into a representation for agents. It trains this representation with reconstruction, dynamics, symbolic consistency, and action-related objectives. The paper reports results on CLEVRER, NExT-QA, ALFWorld, and other benchmarks.

Critical strengths

Preserving information needed for decisions and future prediction is a reasonable motivation for agent representations.

Critical weaknesses

I could not determine which pretrained encoders are used, how entities and relations are extracted, or which symbolic rules are actually implemented. Eq. (24) requires distances between optimal decisions, but the paper does not explain where these targets come from for each dataset. Eq. (27) allows learned or predefined rules without specifying the choice used in the experiments. Appendix A gives dimensions and training hyperparameters, but these do not resolve how the method is implemented or what additional supervision it receives.

Laytrol scores 73.9 on CLEVRER in Table 1 and 78.9 in Table 10; its NExT-QA score is 68.1 in Table 3 and 73.6 in Table 10. No change of setting is identified to explain these differences. It is also unclear which dataset and modalities are used for the degradation experiment in Table 4 and the two-to-five-modality comparison in Table 18. These details need to be clarified before the reported gains can be assessed.

Figures 1 and 2 are low-resolution raster images with small, blurred labels. Section 3 lists numerous equations but leaves the actual computation unclear: graph construction, symbolic reconstruction, and rule execution are represented by abstract operators without concrete implementation details. The notation also changes between Eqs. (2) and (5), making the formulation harder to follow. No references are provided in the methodology to identify the adopted components or distinguish them from the proposed contributions. Despite the extensive mathematical notation, I could not reconstruct a concrete algorithm or training procedure from this description.

Questions for authors

See weaknesses

Reviewer_2s5mOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that representations trained for semantic alignment are not enough for agents and proposes NSMR, a representation that combines a neural embedding, a symbolic entity–relation graph, a temporal world state, and an uncertainty estimate. The model uses goal-conditioned fusion and symbolic reasoning, and is trained with neuro-symbolic consistency, dynamics, action-conditioned prediction, and action-aware geometry losses. It reports gains over four to five baselines on CLEVRER, NExT-QA, and ALFWorld, along with results on compositional generalization and modality degradation.

Critical strengths
  • The motivation is valid and well argued. The paper clearly explains why semantic similarity does not imply decision similarity, which makes a reasonable case for building relational, temporal, and action-relevant structure into agent representations.

  • The evaluation is well chosen. The paper evaluates representations on compositional generalization, robustness to missing or conflicting modalities, temporal prediction, and downstream decisions, using benchmarks that progress from CLEVRER to NExT-QA to ALFWorld. This design is useful, even though the experiments here cannot be verified.

Critical weaknesses
  • The method is not specified well enough to implement or assess. The paper does not name the backbone, define how the uncertainty u_t is computed, or say where d_A(i, j) in L_geo comes from. Nor does it explain how per-entity features (Eq. 12) are obtained when Eq. 8 produces a single fused vector. L_geo also closely resembles bisimulation-metric objectives (e.g., DBC), which the paper does not discuss.

  • The baselines are less suited, and the numbers are inconsistent. Most baselines were not designed for these tasks: Laytrol is a layout-control method for diffusion models, and Agent-X is a benchmark. Their adaptation is not described, and task-specific methods such as NS-DR/DCL on CLEVRER or ReAct/Reflexion on ALFWorld are missing. Laytrol scores 73.9/68.1 on CLEVRER/NExT-QA in Tables 1 and 3 but 78.9/73.6 in App. Table 10, and NSMR’s full-modality accuracy in Table 4 (88.4) matches none of its main-benchmark scores.

Questions for authors
  1. Which backbone and training data are used for each benchmark? What supervision do the relation predictor, the symbolic graph, and d_A receive, and does d_A require expert actions?

  2. How were the baselines adapted to CLEVRER, NExT-QA, and ALFWorld, and will code and evaluation scripts be released?

  3. Which dataset does Table 4 use, and how are the “missing” and “conflicting” conditions built? Does NSMR receive any visual input in ALFWorld, which App. Table 9 lists as “Language + Action”?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.