acceptodds
Withdrawn from ICLR 2027

Thinking with Spatial–Social Code for Social Reasoning in Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Real-world social reasoning is essential for intelligent systems to understand human behavior, anticipate needs, and coordinate actions with people. Inferring beliefs, goals, and intentions requires connecting the visual context of an interaction with each participant's history of observations and actions. The central challenge is bridging observable interaction and latent social states: visual events acquire social meaning through a participant's access to information, prior experience, and ongoing goals. A representation must therefore capture not only the scene, but also how it informs different participants' perspectives. We argue that explicit code offers a computational medium for organizing and using these dependencies through stable references, agent-specific records, and update operations. We introduce Spatial-Social Code (SSC), which links physical situations, visual evidence, and inferred social states in a persistent, updateable representation. Spatial Code organizes entities, spatial relations, physical events, and perceptual-access evidence. Social Code links these records to participants' information and hypotheses about their beliefs, goals, and intentions. Their connections make explicit how situated experiences inform social inference, supporting both queryable reasoning and social memory over time. We evaluate SSC on cross-agent video question answering, collaborative visual grounding, and embodied interaction, demonstrating improvements across multiple tasks. Together, these capabilities provide a computational interface for social reasoning grounded in evolving physical situations and individual experience.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
social reasoning, theory of mind, multimodal reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Kp1nOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Spatial–Social Code, a representation framework connecting physical events and perceptual evidence with participant-specific observations, beliefs, goals, and intentions. Spatial and social records share entity, temporal, and evidence references. In interactive settings, the framework maintains private agent states through observations, action feedback, and delivered messages. The paper evaluates SSC on video question answering, collaborative grounding, and embodied interaction. Its central contribution is an explicit interface for distinguishing changes in the physical world from changes in what individual participants know.

Critical strengths

Well-motivated problem formulation. Distinguishing world state from individual information histories is essential for social reasoning. The separation between observations, beliefs, and received reports is conceptually sound.

Useful representation design. Agent attribution, timestamps, evidence references, and update semantics make intermediate states inspectable. The manuscript appropriately acknowledges that structural validation does not establish the semantic correctness of inferred mental states.

Evaluation across multiple tasks. SSC improves MA-EgoQA overall accuracy from 48.71% to 51.92% and ToM accuracy from 41.28% to 47.66%. MMToM-QA accuracy increases from 73.33% to 78.33%.

Promising evidence for persistent state maintenance. Dynamic SSC succeeds in 24/36 Habitat runs, compared with 8/36 for Direct and 10/36 for Static SSC. Task-level bootstrap analysis appropriately accounts for repeated seeds within tasks.

Transparent reporting of limitations and regressions. The manuscript includes negative grounding results and task-level failures rather than reporting only favorable examples.

Critical weaknesses

The contribution of code representation is not sufficiently isolated. The strongest evidence supports explicit information organization and persistent updating. Information-equivalent prose achieves 92.5% paired correctness versus 95.0% for SSC, while the online comparison is 7/12 versus 8/12 successes. Removing attribution or evidence links removes relevant information, so those ablations do not establish an advantage specific to code.

Main comparisons contain inference-pipeline confounds. The MMToM-QA pipeline includes candidate-conditioned evidence, rule-based bridging, Bayesian scoring, and conditional frame refinement. MA-EgoQA includes an additional raw-video audit. Sharing a backbone and video clips does not ensure matched inference compute or reasoning procedures.

Component ablations are not aligned with the main evaluation. The 70-question diagnostic uses a different backbone and different readout procedures across conditions. Consequently, improvements cannot be cleanly attributed to individual representation components. Generalization evidence remains limited. Habitat evaluation uses only 12 development tasks, while CoMind uses inspected train-development examples. Independent, frozen test sets are needed. Moreover, the strongest interaction results rely on private text inputs and deterministic updates, which provide narrower evidence than end-to-end visual social reasoning.

Semantic grounding is insufficiently evaluated. An evidence identifier establishes traceability, not whether the cited observation supports a belief. The paper should measure attribution errors, unsupported social inferences, and error propagation.

Novelty requires stronger positioning against recent structured reasoning methods. AutoToM (2025), PDDL-Mind (2026), and Thinking with Spatial Code (2026) already establish substantial precedent for explicit agent modeling, state tracking, and code-based visual reasoning. The distinctive contribution should be demonstrated through matched comparisons.

Presentation could better reflect the evidence hierarchy. Some figures use small text, and results from substantially different protocols are placed together. The narrative should emphasize that evidence for state updating is stronger than evidence for code-specific superiority.

Questions for authors

NA

Reviewer_RD5VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors address the challenge of social reasoning for multi-agent interactions in video, where answers depend on not only the current world state (visual evidence) but also what each participant observed previously / was told (past social context). The paper proposes Spatial-Social Code (SSC) a framework with several components: (1) A VLM "spatial compiler" that outputs a database of entities, relations, events, etc. with time stamp indices; (2) a "social compiler" that outputs agent observations + hypotheses (belief, goal, intention) that are linked to evidence IDs; and (3) a reasoner LLM that takes these compiled outputs for final answer output. The authors evaluate their method on four benchmarks (MMToM-QA, MA-EgoQA, CoMind / SCOIA, Habitat/EnactTom), showing improvements over some settings (particularly ones related to theory of mind). There are some concerns regarding the experiments/results denoted below, so preliminary rating for this work is BR pending rebuttal.

Critical strengths
  • Evaluations across the four settings show nontrivial gains in certain theory-of-mind relevant categories, with the same base model (largely, Gemini 3).
  • Further analysis seems interesting for how the proposed state representation can help with visual reasoning beyond vanilla videoQA settings.
Critical weaknesses

The core contribution (centering around the code representation) does not seem to be clearly tested for the video QA setting. For example, on MMToM-QA, the SSC method also leverages other techniques (Appendix B: "candidate-conditioned evidence, rule-based bridging, Bayesian scoring, and conditional frame refinement.") that could be contributing significantly to the final difference between the "direct" baseline (which doesn't seem to get any of these additional elements). All of those elements have been subjects of other related work individually (especially frame refinement), and it's unclear what the code representation actually adds. Similarly, on MA-EgoQA, it seems that the Direct baseline gets only one model call, while the proposed method gets multiple model calls during compilation -- it seems like there is a more compelling baseline where this is controlled for. Note that while there is some analysis of code vs. plain text in a text-only setting, it's not clear how this extends to the main visual reasoning part.

Questions for authors

Please address the comments above in critical weaknesses, especially regarding ablations and controlled baselines - thanks

Reviewer_TQeDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper applies a social reasoning representation from a paper on text-based ToM reasoning onto video ToM problems.

Critical strengths

It's natural to expect improvements from translating proven text-based ToM representations to video ToM applications.

Critical weaknesses
  1. The manuscript is littered with AI-generated prose that should have been rewritten by the authors. In the first 70-80% of the paper, the prose is extremely wordy and hard to understand at the same time. This takes away from the core purpose of the manuscript, i.e., to communicate the authors' scientific findings.
  2. After reading this paper, I have yet to understand the method, the experiment, and any findings.
  3. There is a severe lack of experiments, most notably the number of model families. They are mostly Gemma.
  4. It's obvious that this method should work reasonably well, but the work only measures effectiveness without studying the concrete effect or mechanisms, and how well it might work/limitations thereof.
Questions for authors

All questions below should have been featured prominently in the main paper. If they are genuinely and substantially addressed in a future work, a much improved, human-written manuscript may merit acceptance.

  1. What is your method exactly? What is the pipeline?
  2. What is your representation? 3 How well does your method work? What are the upper and lower bounds?
  3. What are the limitations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.