- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors contribute a method to condition a perception layer of a model-based planning system with task information. Their goal is to use knowledge from a task to enable the perception layer to learn representations that ignore task-irrelevant information.
One of the core assumptions of the method is that it is possible to obtain a distractor-free version of any observation without training an additional component. (The methodology uses a pipeline with off-the-shelf components.)
The validation centers on experiments conducted in three simulated environments. These experiments have the distractors causally isolated from the actual task. E.g., in Push-T, the distractor objects do not interact (collide) with the task's object.
Critical strengths
-
The manuscript presents a synthetic data-driven methodology to address an important problem. This could allow training of the perception layer without relying on costly on-policy rollouts.
-
The experiments measure and validate some of the properties one would want out of "configurable perception". The authors do probing on physical quantities and a causal intervention to establish (to a meaningful, but not overwhelming extent) that the query does carry information and it is used.
Critical weaknesses
1.Experiments are limited by distractors being causally/physically inert. The model is never tested against objects that could interact with the task object, which is one of the central difficulties of multi-object scenes that are the focus of your evaluation; in this situation distractors would become non-distractors depending on the dynamics (e.g., if an object interacts with the task object). This makes the evaluation of limited value for testing the core claim because the "distractors" in the evaluation are of a very specific kind. It seems that the method could incorporate more realistic distractors, but a significant extension to the core experiments would be necessary to show so.
2. Availability of "distractor-free" synthetic versions of any given observation is a strong assumption. A core part of the contribution is the idea of using distractor-free versions of observations, which are synthetically augmented with distractors, but the paper does not show the methodology could be applied (in principle) to a realistic setting. My main concern here is that "augmenting" data in a way that generates realistic/in-distribution distractors is a core challenge faced by the method, but it is not explored and instead is worked around. For instance, distractors in the real world exhibit things like object permanence and can interact with "non-distractors". The fact that you use completely different techniques to generate this data (overlaying, tiling and an image-edit model) I believe supports my claim that this is non-trivial and a solution to a core challenge has not been identified.
3. The dynamics model is not evaluated in isolation. Nothing in the paper measures the dynamics model's prediction accuracy directly; every result either probes the encoder alone or measures the full pipeline (encoder+dynamics+planning) at once. We cannot tell, whether planning gains come from better dynamics modeling (the "world model") versus a representation space that's merely more CEM-friendly.
Questions for authors
- Would you consider running an experiment where distractors are real objects and the augmentation pipeline used for the real world data is used?
- E.g., a variation of any of your simulated environments where distractors are other objects and the image-edit model is used.
- This would be direct evidence for your core claim, whereas your current experiments are only indirect (the core experiments test on highly specific artificial distractors and with varying generation pipelines, as described above.)
- Do you have any evidence that the learned representation encodes the correct information also when the distractors are more complex? E.g., in situations where distractors become non-distractors, and vice versa?
- How do you think this approach compares to "emergent" configurable perception which might arise during, e.g., RL? I would imagine it is (in principle) possible to learn configurable perception without generating the synthetic data you propose generating, but perhaps much more inefficiently. Having data that quantifies the efficiency bought by your method would be valuable.
4 comments
W2 & Q1: Clean Target Construction and Realistic Distractor Evaluation
We agree that constructing reliable noisy–clean training pairs in realistic scenes is challenging. This is precisely why we adopt a clean-to-noisy augmentation strategy: we start from existing observations and construct noisy inputs by adding visual distractors. Similar to the denoising objective in diffusion models, our strategy does not require perfectly distractor-free original observations: they serve as clean targets relative to the added distractors, allowing the configurable encoder to learn to suppress the injected task-irrelevant information while preserving task-relevant content.
We have already validated the scalability of this approach on real-world data (Sec. 6 and Appx. D), using existing egocentric (e.g., Ego4D[1], EPIC-KITCHENS-100[2]), robot manipulation (e.g., DROID[3], Open X-Embodiment[4]), internet activity and instructional videos (e.g., ActivityNet[5], COIN[6]). Following the same pipeline, we treat original observations as clean targets and generate noisy inputs by adding visual distractors through image editing. Across four visual backbones, our configurable encoder reduces distractor AP by 17.9 percentage points on average while largely preserving task-core AP, with qualitative results shown in Figs. 10 and 22.
Furthermore, following your suggestion, we use the same image-editing model as our real-world data pipeline to generate real objects (e.g., dinosaurs, ducks, and bottles), which are then extracted and inserted into Two-Room, Push-T, and OGBench-Cube as static or moving distractors. We directly evaluate the existing checkpoints through closed-loop planning. Representative results with two inserted objects () are shown below.
| Environment | Method | Static | Moving |
|---|---|---|---|
| Two-Room | ConfigWM | 86.7 ± 4.1 | 87.3 ± 5.0 |
| DINOv2-WM | 76.7 ± 4.1 | 80.7 ± 3.4 | |
| Push-T | ConfigWM | 27.3 ± 0.9 | 28.7 ± 3.4 |
| DINOv2-WM | 6.7 ± 3.4 | 7.3 ± 5.0 | |
| OGBench-Cube | ConfigWM | 65.3 ± 9.6 | 62.7 ± 9.0 |
| DINOv2-WM | 52.0 ± 5.9 | 50.7 ± 4.1 |
As shown, ConfigWM achieves higher planning success than DINOv2-WM under both static and moving distractors across all three environments at . These results further support the applicability of our image-edit-based augmentation approach to simulated environments and the robustness of ConfigWM to more diverse visual distractors. We have included the corresponding visualizations and complete experimental results in Appx C.10 of the revised paper.
[1] Ego4D: Around the World in 3,000 Hours of Egocentric Video. CVPR 2022
[2] Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100. IJCV 2022
[3] DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. RSS 2024
[4] Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration. ICRA 2024
[5] ActivityNet: A Large-Scale Video Benchmark for Human Activity Understanding. CVPR 2015
[6] COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis. CVPR 2019
Q3: Data Efficiency of Configurable Perception
As you suggest, configurable perception could potentially emerge through RL, where task-relevant representations are learned implicitly from reward signals and interactions. In contrast, our approach explicitly learns task-conditioned representations from synthetic paired supervision, without requiring task rewards or on-policy interactions during perceptual pretraining.
To further investigate its data efficiency, we conducted an additional experiment on Two-Room , varying the proportion of original trajectories available for Stage-A training while keeping the training steps and full Stage-B data unchanged. All data fractions were independently retrained under the same training and evaluation protocol, including the full dataset. The planning success rates (%) are shown below.
| Stage-A Data | Available Trajectories | Planning Success (%) |
|---|---|---|
| 0.1% | 10 | 74.7 ± 7.4 |
| 0.5% | 50 | 85.3 ± 8.1 |
| 1% | 100 | 94.7 ± 0.9 |
| 10% | 1,000 | 95.3 ± 3.4 |
| 100% | 10,000 | 95.3 ± 4.1 |
Notably, using only 1% of the original trajectories for Stage-A training achieves 94.7% planning success, comparable to 95.3% with the full dataset. These results suggest that effective configurable perception can be learned with relatively limited paired supervision. We have incorporated the complete results into Appx C.9 of the revised paper.
Dear Reviewer hLiM,
As the discussion period draws to a close, we would greatly appreciate any feedback you may have on our rebuttal and revised manuscript. We would be happy to address any remaining questions or concerns.
Thank you again for your time and consideration!
Best regards,
Authors
Dear Reviewer hLiM,
As today is the final day of the discussion period, we would like to sincerely thank you for the many helpful suggestions you provided on our paper, especially your perspective on the connection between configurable perception and RL, which helped us recognize another potential advantage and application of our method that we had not fully considered before. Following your suggestions, we have conducted additional experiments and provided further clarifications. We hope our responses have helped address your concerns. Regardless, we truly appreciate your valuable feedback on our paper.
Thank you once again for your careful review and constructive comments.
Best regards,
The Authors
We sincerely thank the reviewer for the thorough assessment and insightful feedback. Following your suggestions, we have substantially extended our evaluation with physically interacting objects, image-edit-generated visual distractors, planner-free physical-state prediction, and a Stage-A data-efficiency study. We hope these additional evaluations and analyses help alleviate your concerns and strengthen the experimental evidence supporting our approach.
W1 & Q2: Task-Relevant Representations under Physical Interactions
Thank you for your insightful comments. Our original evaluation setting deliberately introduces non-interacting visual distractors to clearly distinguish task-relevant from task-irrelevant information. Since these distractors are introduced through observation-space augmentation without affecting the queried target or the underlying task dynamics, they are task-irrelevant by construction. The original observations therefore preserve the task-relevant information, naturally serving as clean targets for Stage-A and encouraging the encoder to retain task-relevant information while filtering out irrelevant visual variations.
Meanwhile, we agree that evaluating physically interacting objects is an important extension of our controlled setting. Following your suggestion, we further introduce a physically collidable blue ball in Two-Room to investigate whether ConfigWM can capture interaction-relevant information while the configurable encoder remains frozen with the query "red ball". Contact-window prediction error measures the mean Euclidean distance between predicted and ground-truth red-ball positions around collision events. As shown below, training with physical collisions reduces the contact-window prediction error by 64.5% at one step and 43.4% at five steps compared with training without collisions.
Furthermore, linear probing of the learned state embeddings yields a lower position MSE for the non-queried blue ball, decreasing from 0.859 to 0.695 (19.0%) compared with training without collisions. These results suggest that ConfigWM can faithfully follow language-guided configurable perception while further adapting its latent representations to preserve physically relevant information through downstream dynamics learning. We have incorporated these results into Appx C.8 of the revised paper.
W3: Evaluating World Model Prediction Accuracy
Planning success alone does not directly measure the world model's prediction accuracy. To directly evaluate world model prediction accuracy independently of CEM planning, we conducted additional open-loop physical-state prediction experiments across all three environments. Specifically, we evaluate the predicted states against ground-truth simulator states using frozen linear probes. We report Euclidean position RMSE across different distraction levels and prediction horizons (h1/h5 corresponding to 5/25 simulator steps), as shown below.
As shown, ConfigWM consistently achieves lower physical prediction errors across evaluated distraction levels and both prediction horizons. These results demonstrate that ConfigWM's improvements extend beyond CEM-friendly representations to more accurate physical-state prediction. We have incorporated these additional results into Section 5.8 of the revised paper.