acceptodds
Withdrawn from ICLR 2027

JANUS: Spatial Belief and Evidence Imagination for Embodied Active Perception

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Active perception is an essential capability for embodied agents, providing the ability to selectively acquire the visual evidence required for downstream decision-making. Recent approaches leverage multimodal large language models (MLLMs) to decide where to observe next. However, without persistent memory and explicit spatial representations, these approaches struggle to maintain spatial consistency across observations and identify informative regions for further exploration. In this work, we seek to unlock the memory and associative reasoning capabilities of MLLMs, investigating whether an agent can efficiently accomplish visual perception tasks through a sequence of selective egocentric observations. To this end, we introduce JANUS, a framework that couples persistent spatial belief with imagination-guided next-view decisions. Specifically, we develop a Hierarchical Spatial Belief module that organizes accumulated real observations into a hierarchical representation over scenes, regions, objects, and their attributes, while preserving confidence estimates and links to the supporting visual evidence. We further introduce a Belief-Conditioned Evidence Imagination module that conditions on the current spatial belief to anticipate informative candidate observations, using predicted latent evidence to reactivate relevant past observations and guide next-view selection. Together, these modules connect accumulated observations with future information needs, enabling belief-guided decisions about where to observe next. Experiments on challenging perception benchmarks demonstrate the effectiveness of JANUS for embodied active perception.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Evidence-driven active observation, Embodied multimodal agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YyHeOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents the Janus framework to represent belief and history for Multimodal Large Language Model (MLLM) guided exploration. The Janus framework consists of two parts, the Hierarchical Spatial Belief module and the Belief-Conditioned Evidence Imagination module. The Hierarchical Spatial Belief module builds a spatial graph for information retrieval for the query. The Belief-Conditioned Evidence Imagination module predicts the next feature for next-view selection. Experiments show that such modules could enhance MLLMs performance on embodied geo navigation benchmarks. The ablation study verifies the effect of each specific module.

Critical strengths

I cannot identify the strength since many details are missing.

Critical weaknesses
  1. [Poor Writing] The symbols in the method section lack definitions and explanations. I will list some examples below:
  • In section 4.2, what is the shape of the visual feature ziz_i? How do you encode it?
  • For line 231, what is the confidence score? How do we get it? And how do we get the supporting-view references?
  • For line 251-252, how do you revise the claim? More specifically, how do you update the confidence score and the references?
  • Equation 6, what is TθT_\theta and FθF_\theta? If they are networks, do you train them or just use pre-trained checkpoints?
  • Equation 7, how do you compute U?

These definitions and explanations are critical for the understanding and evaluation of this paper.

  1. [Lack of Baseline] The result table only shows LLM vs. LLM + Janus without other baselines that use memory for LLM on these tasks. The author discusses several previous works that use memory but does not compare to them in the experiments. It is unclear whether the improvement is brought by the proposed belief and imagination mechanism or simply historical information.
Questions for authors

See critical weakness.

Reviewer_AJCMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses embodied active perception by controlling camera pan, tilt and zoom from a fixed robot pose. It consists of two core modules: Hierarchical Spatial Belief (HSB), which organizes real observations into scene, region, object and attribute layers with associated confidence scores and provenance of evidence; and Belief Evidence Imagination (BEI), which predicts visual features for candidate viewpoints to retrieve historical observations and select the next viewing direction. The authors evaluate the framework on ERGeoBench and H*Bench across question answering, geolocalization, object search, and directional path search tasks.

Critical strengths
  1. The framework explicitly enforces an information boundary between imagination and factual evidence. Predicted features are restricted to planning and retrieval signals, and fact updates must be grounded in real observations. This design mitigates hallucination propagation in active perception and carries greater methodological merit than simply appending historical images or generating hypothetical future views.
  2. Partial ablation results suggest promising empirical gains. In Table 3, HOS improves from 55.00% to 61.25% and HPS improves from 29.38% to 33.13% with a fixed SFT actor.
Critical weaknesses
  1. There is a disconnect between the core claims and the implemented system. Page 3 states that the full candidate retrieval and reranking loop is framed as a future extension, and belief calibration and contradiction-aware revision are listed as future work on page 9. Nevertheless, the method section introduces these capabilities as integral parts of the complete framework. Implementation details for HSB construction, entity merging, confidence update rules. It is therefore unclear whether observed gains stem from the proposed belief-imagination interaction or simpler memory interventions and action scoring.
  2. Experimental results may conflict with the conclusions.The module ablations in Table 3 are helpful but do not validate the unique mechanism of “predictive features serving as queries for retrieving real evidence”. Missing baselines include equal-budget vanilla retrieval and viewpoint scoring without imagination.
Questions for authors
  1. Which implementation and evaluation version do Tables 1–3 correspond to, relative to the claims on page 9? Which results share identical samples, observation budgets and final controller logic?
  2. Which belief revision and candidate retrieval operations are actually implemented? Please fully specify algorithms, training data and inference interfaces for HSB, BEI and the answer router.
  3. What is the measured magnitude of BEI negative transfer? Under equal observation and compute budgets, does BEI yield stable improvements over HSB augmented with ordinary retrieval?
Reviewer_ghwJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents JANUS, a framework for embodied active perception designed to enhance sequential observation decisions via persistent spatial memory and evidence imagination. It consists of two core modules: the Hierarchical Spatial Belief (HSB), which organizes observations into structured spatial representations, and Belief-Conditioned Evidence Imagination (BEI), which predicts latent visual features for candidate viewpoints to facilitate evidence retrieval and next-view selection. The authors evaluate JANUS on ERGeoBench and H* Bench over six tasks using multiple MLLM backbones. The results show improvements in question answering, geolocation, and directional search, with component ablations suggesting complementary benefits from HSB and BEI.

Critical strengths
  1. Well-motivated design. HSB organizes observations and beliefs hierarchically, while BEI uses predicted visual evidence to guide further observations. The combination makes sense.

  2. Broad evaluation. The framework is evaluated on multiple tasks across two benchmarks and several MLLM backbones, showing improvements across different settings.

Critical weaknesses
  1. Missing implementation details. The belief and confidence updates in HSB are not clearly explained. For BEI, details on the latent world model's encoding, training data, and key functions are not clearly explained, making the method difficult to reproduce.

  2. Unclear link between prediction and action selection. The paper does not clearly explain how predicted latents help select more informative viewpoints.

  3. Issues with reporting and presentation. It is unclear how the QA All score in Table 2 is calculated from the individual tasks. Figure 3 also contains an apparent copy-paste error, and some references have incomplete information.

Questions for authors
  1. Method implementation. Could the authors clarify how confidence is updated and beliefs are revised in HSB, as well as how the training data, feature encoding, and latent world model are implemented in BEI? In particular, how are the utility function (U) and retrieval function (r) in Equation (7) computed, and how do the predicted latents influence the final action selection?

  2. Experimental results. Could the authors clarify how the QA All score in Table 2 is calculated? For example, Qwen-VL Max and GPT-4o show similar accuracy on FP, SA, and CS, but their QA All scores differ by 6.48 percentage points. It is unclear how this difference arises from the subtask results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.