- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AWAM, a framework that constructs a geometry-conditioned world memory to support a downstream action policy. A language-conditioned trajectory generator first predicts virtual camera poses from the initial observation and task instruction. Partial 3D geometry reconstructed from the initial RGB-D observations is then rendered from these viewpoints and processed by a world model. The resulting intermediate predictive features are compressed into a cached memory. During execution, the action policy queries this memory using live observations and predicts actions in closed loop. Experiments across multiple simulation benchmarks and real-world manipulation tasks demonstrate improved overall performance over the evaluated baselines.
Critical strengths
- The paper addresses robotic manipulation under partial observability. Constructing memory from task-conditioned virtual viewpoints is a useful approach to supporting action prediction. Caching this memory once per episode and querying it using live observations provides a coherent way to combine initial-scene priors with current sensory feedback.
- The evaluation covers both simulation benchmarks and real-world manipulation tasks, with encouraging performance gains in both settings.
- The ablation study includes trajectory-only conditioning, sampled viewpoints, and geometric warping, providing useful evidence of the contributions and roles of the main components.
- The paper explicitly acknowledges the limitations of keeping the initial-scene memory fixed throughout execution.
Critical weaknesses
Major Comments:
-
Training data and evaluation protocols are insufficiently specified. Appendix A.1.2 states that world-model supervision is obtained by fixing the initial simulator state and rendering additional calibrated views without advancing the dynamics. However, the amount and source of this data, and its separation from evaluation states, are unclear. The paper should document the training data, dataset sizes, and splits for each module and benchmark.
-
The role of the demonstrated camera trajectory needs stronger analysis. The trajectory predictor is supervised by demonstrated wrist-camera paths. The trajectory-only ablation feeds these predictions to a separate live policy, but does not evaluate direct Cartesian tracking or closed-loop trajectory execution. These would be useful comparisons for assessing the added value of the world-model branch. Moreover, the trajectory objective imitates demonstrations rather than optimising visibility or information gain. The paper should provide evidence that the predicted paths improve task-relevant observation coverage.
-
The computational cost needs clarification. The paper should report initialisation latency, replanning latency, and end-to-end task time. Since avoiding physical inspection from different viewpoints is a key motivation, a breakdown of the costs of trajectory prediction, rendering, world-model feature extraction, memory construction, and action inference would help assess the practical benefit.
-
The gains over the closest occlusion baselines are modest. AWAM achieves 75.9% versus 74.0% for VIM on LIBERO-Occ, and 76.5% versus 75.0% for TVVE on RLBench-OG. Uncertainty estimates would help assess the reliability of these improvements. Furthermore, RLBench-OG Occlusion1 involves both training and testing on occluded configurations, so these results should not be interpreted as zero-shot transfer to unseen occlusions.
-
The accuracy of the memory’s representation of hidden objects remains unclear. The memory improves task performance, but generated views depend on initial observations and learned priors. The reported hole rates measure missing geometry rather than prediction accuracy. Hidden-object localisation errors and tests with varying target positions behind occluders would help. For example, if different block positions behind the box produce identical initial observations and instructions, the model has no evidence to distinguish them. Successful execution may instead rely on live observations once the target becomes visible, which does not establish that the initial memory predicted its location correctly.
Minor Comments:
- Section 4.3 should specify exactly how trajectory poses are encoded and injected into the trajectory-only policy, and how warp-only features are produced.
Questions for authors
- Could you specify the data sources, dataset sizes, and training/evaluation splits for each module and baseline?
- Could you explain how variants (b) and (d) in Table 5 are implemented?
- Would it be possible to compare AWAM with direct or receding-horizon tracking of the predicted trajectory poses?
- Could you quantitatively evaluate prediction accuracy in hidden regions and analyse representative failure cases?
- Could you report the computational cost of each module, including initialisation and per-step inference latency?
Official Comment to Authors
Thanks for the responses, which fully address my concerns. I raised my rating from 2 to 3.