acceptodds
Under review as a conference paper at ICLR 2027

AWAM: Active Trajectory-Conditioned View Imagination for Robust Robot Manipulation

Abstract

Rapid progress in embodied intelligence has enabled increasingly capable robots to follow language instructions and manipulate diverse objects. However, strong results in structured environments do not automatically transfer to open-world conditions, where clutter, occlusion, and changing viewpoints can obscure the spatial relationships required for action. To address this perceptual gap, a virtual active-perception paradigm is introduced. AWAM, an Active World-Action Model actively plans where to observe and imagines what will be seen along the planned trajectory before acting. Task-related virtual camera motion is predicted and used as a structured query into a geometry-conditioned world model, producing a reusable spatial prior for manipulation, which is later compressed into a fixed memory and fused with live observations through cross-attention for diffusion-based action prediction. AWAM achieves competitive success rates with 75.9% on LIBERO-Occ, 62.4% on LIBERO-Plus, 76.5% on RLBench-OG Occlusion1, and 94.4% in real-robot trials. These results support improved robustness to the evaluated occlusions and distribution shifts, addressing part of the gap between structured manipulation and less controlled environments.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Active Perception, Embodied AI, World Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pBskOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AWAM, a framework that constructs a geometry-conditioned world memory to support a downstream action policy. A language-conditioned trajectory generator first predicts virtual camera poses from the initial observation and task instruction. Partial 3D geometry reconstructed from the initial RGB-D observations is then rendered from these viewpoints and processed by a world model. The resulting intermediate predictive features are compressed into a cached memory. During execution, the action policy queries this memory using live observations and predicts actions in closed loop. Experiments across multiple simulation benchmarks and real-world manipulation tasks demonstrate improved overall performance over the evaluated baselines.

Critical strengths
  1. The paper addresses robotic manipulation under partial observability. Constructing memory from task-conditioned virtual viewpoints is a useful approach to supporting action prediction. Caching this memory once per episode and querying it using live observations provides a coherent way to combine initial-scene priors with current sensory feedback.
  2. The evaluation covers both simulation benchmarks and real-world manipulation tasks, with encouraging performance gains in both settings.
  3. The ablation study includes trajectory-only conditioning, sampled viewpoints, and geometric warping, providing useful evidence of the contributions and roles of the main components.
  4. The paper explicitly acknowledges the limitations of keeping the initial-scene memory fixed throughout execution.
Critical weaknesses

Major Comments:

  1. Training data and evaluation protocols are insufficiently specified. Appendix A.1.2 states that world-model supervision is obtained by fixing the initial simulator state and rendering additional calibrated views without advancing the dynamics. However, the amount and source of this data, and its separation from evaluation states, are unclear. The paper should document the training data, dataset sizes, and splits for each module and benchmark.

  2. The role of the demonstrated camera trajectory needs stronger analysis. The trajectory predictor is supervised by demonstrated wrist-camera paths. The trajectory-only ablation feeds these predictions to a separate live policy, but does not evaluate direct Cartesian tracking or closed-loop trajectory execution. These would be useful comparisons for assessing the added value of the world-model branch. Moreover, the trajectory objective imitates demonstrations rather than optimising visibility or information gain. The paper should provide evidence that the predicted paths improve task-relevant observation coverage.

  3. The computational cost needs clarification. The paper should report initialisation latency, replanning latency, and end-to-end task time. Since avoiding physical inspection from different viewpoints is a key motivation, a breakdown of the costs of trajectory prediction, rendering, world-model feature extraction, memory construction, and action inference would help assess the practical benefit.

  4. The gains over the closest occlusion baselines are modest. AWAM achieves 75.9% versus 74.0% for VIM on LIBERO-Occ, and 76.5% versus 75.0% for TVVE on RLBench-OG. Uncertainty estimates would help assess the reliability of these improvements. Furthermore, RLBench-OG Occlusion1 involves both training and testing on occluded configurations, so these results should not be interpreted as zero-shot transfer to unseen occlusions.

  5. The accuracy of the memory’s representation of hidden objects remains unclear. The memory improves task performance, but generated views depend on initial observations and learned priors. The reported hole rates measure missing geometry rather than prediction accuracy. Hidden-object localisation errors and tests with varying target positions behind occluders would help. For example, if different block positions behind the box produce identical initial observations and instructions, the model has no evidence to distinguish them. Successful execution may instead rely on live observations once the target becomes visible, which does not establish that the initial memory predicted its location correctly.

Minor Comments:

  1. Section 4.3 should specify exactly how trajectory poses are encoded and injected into the trajectory-only policy, and how warp-only features are produced.
Questions for authors
  1. Could you specify the data sources, dataset sizes, and training/evaluation splits for each module and baseline?
  2. Could you explain how variants (b) and (d) in Table 5 are implemented?
  3. Would it be possible to compare AWAM with direct or receding-horizon tracking of the predicted trajectory poses?
  4. Could you quantitatively evaluate prediction accuracy in hidden regions and analyse representative failure cases?
  5. Could you report the computational cost of each module, including initialisation and per-step inference latency?
Reviewer_mkNdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AWAM, an active world-action model for robotic manipulation under occlusion and partial observability. The main idea is to predict a task-conditioned virtual camera trajectory from the initial RGB-D observations and language instruction, and then use a geometry-conditioned world model to generate complementary views of the scene. The resulting features are stored in a persistent memory, which the diffusion-based policy can access together with live visual observations during execution. This allows the policy to use imagined scene information without requiring additional physical exploration. Experiments report success rates of 75.9% on LIBERO-Occ, 62.4% on LIBERO-Plus, 76.5% on RLBench-OG Occlusion1, and 94.4% on real-world manipulation tasks.

Critical strengths
  1. The paper presents an interesting way to combine virtual active perception with closed-loop manipulation. Instead of physically moving the robot to explore occluded regions, AWAM predicts a virtual camera trajectory and uses a world model to construct additional scene information. This is also different from approaches that mainly predict future observations resulting from robot actions. I find the separation between memory construction and online observation-conditioned retrieval (Figs. 1–2, Sec. 3) particularly useful. It provides a reasonable framework for incorporating imagined viewpoints into manipulation without introducing extra exploration actions.

  2. The experiments cover several benchmarks and show encouraging improvements in robustness. AWAM is evaluated on three simulation benchmarks and three real-world tasks. On LIBERO-Plus, it achieves 62.4% success compared with 53.6% for π0 (Table 3). The improvement under camera perturbations is especially noticeable, with success increasing from 13.8% to 53.4%. The real-world results are also promising. Table 4 reports 85 successful trials out of 90 for AWAM, compared with 73 out of 90 for Motus. These results suggest that predictive visual memory can be useful when observations are incomplete or affected by viewpoint changes.

  3. The ablation studies provide useful evidence for the contributions of the main components. In Table 5, removing predictive memory reduces LIBERO-Occ success from 75.9% to 64.4%. Replacing the learned camera trajectory with uniformly sampled viewpoints lowers performance to 70.2%, while removing generative prediction reduces it to 69.0%. These results indicate that both task-conditioned viewpoint selection and generative completion contribute to the final performance. They also suggest that the improvements cannot be explained entirely by the underlying policy or geometric warping alone.

Critical weaknesses
  1. The novelty and specific benefit of the active trajectory design could be better established. Related methods such as TVVE (Bai et al., 2026) have already explored task-aware virtual viewpoints, while VIM (Li et al., 2026) uses imagined complementary observations to address occlusion. AWAM extends these ideas by generating views along a predicted trajectory and storing their features in memory. However, as described in Sec. 3.2, the trajectory predictor is trained to imitate demonstrated wrist-camera paths rather than explicitly optimizing visibility or information gain. This makes it unclear whether the learned trajectory is actually better at revealing task-relevant occluded regions. Table 5 compares it with uniformly sampled spherical viewpoints, but this is a relatively simple baseline. I would like to see comparisons with other task-aware multi-view selection strategies under the same number of generated views. Such experiments would help clarify what is gained specifically from continuous trajectory prediction.

  2. The comparisons with existing methods do not fully account for differences in sensing and model resources. AWAM uses calibrated RGB-D observations from two cameras, a pretrained trajectory generator, and an adapted WorldStereo model. The baselines may not have access to the same sensing information, auxiliary supervision, or pretrained resources. This is particularly relevant because the improvements over the strongest specialized baselines are relatively small. For example, AWAM achieves 75.9% versus 74.0% for VIM on LIBERO-Occ (Table 1), and 76.5 ± 1.1% versus 75.0 ± 1.2% for TVVE on RLBench-OG (Table 2). While the internal ablations in Table 5 are helpful, they do not fully resolve these differences across methods. A more controlled comparison would make it easier to judge how much of the improvement comes from trajectory-conditioned predictive memory rather than additional sensing, pretraining, or model capacity.

  3. There is limited quantitative evidence about the reliability of the imagined observations. A central motivation of AWAM is to recover information about occluded or unobserved parts of the scene. However, Fig. 4 mainly provides qualitative examples, and the paper does not report quantitative measurements of generated-view accuracy, target visibility, or hallucination errors. The average geometric hole rate of 39.29% also suggests that a considerable amount of scene content needs to be generated rather than directly reconstructed. I think this makes the reliability of the imagined views an important question. For example, how often does the world model recover the correct object geometry, and how does this change with occlusion severity? It would also be useful to examine whether better virtual-view predictions actually lead to higher manipulation success. Without such analysis, it remains difficult to distinguish the benefit of accurate spatial reconstruction from that of learned scene or action priors.

  4. The computational cost and the use of fixed predictive memory need further discussion. AWAM generates 21 virtual viewpoints using a relatively large pretrained world model, but the paper does not provide initialization latency, peak GPU memory usage, or a detailed comparison of inference costs (Sec. 3 and Appendix A.1). Since the memory is built before execution, these costs may matter in practical deployments. Another limitation is that the memory is constructed only once and cannot explicitly account for objects being moved later in the episode. This may be particularly relevant for long-horizon tasks, where the scene can change substantially after the initial observations. The weaker result on LIBERO-Occ Long (41.0% compared with 44.2% for Fast-WAM in Table 1) is consistent with this concern, although it does not establish the cause. I think further analysis of both computational overhead and sensitivity to scene changes would help clarify the practical trade-offs of the method.

  5. There are a few minor presentation issues. Some figures contain small labels or crowded connections that make them harder to read at the paper's actual display size. In particular, the arrows and connectors around the trajectory predictor, coarse rendering, and memory-attention modules in Fig. 2 could be better aligned. Figs. 1, 3, and 4 would also benefit from clearer labels. I also noticed a few typographical issues, such as "cross-attentionwhere" in Appendix A.1.3 and "criterion.Example" in Appendix A.4.1. These are relatively easy to fix but would improve the overall presentation.

Questions for authors
  1. Could the authors clarify the pretraining data, model sizes, sensing modalities, and computational resources used by AWAM and the reproduced baselines? It would be helpful to understand whether the performance differences remain similar when these factors are more closely controlled.

  2. How does the learned camera trajectory compare with other task-aware viewpoint selection methods under the same generation budget? Beyond the comparison with uniform sampling, can the authors quantify whether the predicted trajectory improves target visibility or spatial reconstruction, especially in heavily occluded scenes?

  3. What are the initialization latency, peak GPU memory usage, and world-model inference costs for constructing the 21-view memory? How do these compare with the strongest baselines? I would also be interested in how robust the fixed memory is when objects move or the scene changes significantly during execution.

1 comment
Reviewer_mkNdOpenReview ↗

Official Comment to Authors

Thanks for the responses, which fully address my concerns. I raised my rating from 2 to 3.

Reviewer_y76zOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AWAM, a virtual active perception framework for robust robot manipulation under occlusion and partial observability. The method first predicts a task-conditioned virtual camera trajectory from initial RGB-D observations and language instructions. A geometry-conditioned world model then imagines visual features along the predicted trajectory, which are compressed into a persistent world memory. During execution, the policy retrieves relevant information from this memory through cross-attention with live visual observations and generates actions using a diffusion transformer. The method is evaluated on three simulation benchmarks, including LIBERO-Occ, LIBERO-Plus, and RLBench-OG Occlusion1, as well as three real-world manipulation tasks, demonstrating improved robustness under occlusion and distribution shifts.

Critical strengths
  1. Interesting and practical virtual active perception framework. The paper addresses an important challenge in robotic manipulation: incomplete visual observations caused by occlusion and unfavorable camera viewpoints. Instead of requiring additional physical camera movements, AWAM predicts task-conditioned virtual camera trajectories and leverages a world model to imagine complementary observations. The idea of compressing these predicted multi-view features into persistent memory is also practical, as it allows the policy to reuse the predictive information throughout execution without repeatedly invoking the world model.

  2. Comprehensive simulation and real-world evaluations. The authors evaluate AWAM on three simulation benchmarks covering occlusion and distribution shifts, together with three real-world manipulation tasks. The method achieves competitive performance across these settings, particularly on LIBERO-Plus and real-world tasks. The ablation studies are also informative, demonstrating the contributions of learned camera trajectories, generative view imagination, and predictive memory. These results provide reasonable evidence for the effectiveness of the overall framework.

Critical weaknesses
  1. The necessity of trajectory-based view planning is not sufficiently justified. While predicting a continuous virtual camera trajectory is presented as a key contribution, the trajectory predictor is trained using demonstrated wrist-camera trajectories rather than explicitly optimizing information gain, target visibility, or task success. It remains unclear whether the continuous trajectory formulation provides substantial advantages over simpler task-conditioned viewpoint selection methods. The current ablation compares learned trajectories with uniformly sampled spherical viewpoints, but does not include stronger alternatives that predict task-relevant viewpoints independently. Moreover, the improvement over the closely related TVVE method on RLBench-OG is relatively modest (76.5% vs. 75.0%). Additional comparisons would help better establish the benefit of trajectory-conditioned view imagination.

  2. Limited analysis of the reliability of imagined visual information. A central assumption of AWAM is that the world model can provide useful information about regions that are not visible in the initial observations. However, such predictions rely on learned priors and may not accurately reflect the actual scene, particularly when the target object is fully occluded. Although the paper provides qualitative visualizations of generated views, it lacks quantitative evaluations of reconstruction accuracy, geometric consistency, or the impact of hallucinated information on manipulation performance. In addition, the world memory remains fixed throughout an episode, potentially becoming outdated as objects move. This limitation may be particularly relevant to long-horizon tasks, where AWAM performs worse than Fast-WAM on the LIBERO-Occ Long suite (41.0% vs. 44.2%).

Questions for authors
  1. Could the authors further clarify the advantages of predicting a continuous virtual camera trajectory over directly predicting a set of task-conditioned viewpoints? A comparison with stronger task-aware viewpoint selection baselines would help establish the necessity of trajectory-based planning.

  2. Could the authors provide quantitative evaluations of the generated virtual views, particularly regarding the accuracy of occluded object reconstruction and geometric consistency? It would also be interesting to understand how inaccurate or hallucinated predictions affect downstream manipulation performance.

  3. Since the world memory is constructed only once at the beginning of each episode, have the authors investigated how its usefulness changes as the scene evolves? Could periodically updating the memory improve performance on long-horizon tasks?

  4. Could the authors report the computational overhead of virtual trajectory prediction, world-model inference, and memory construction? Although these operations are performed only once per episode, their initialization latency may still affect practical deployment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.