- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes AWARE, a method for selecting actions that may expose errors in a robotic world model before execution. With the visual history and task conditions held fixed, it generates 64 perturbations to the first action and selects the candidate that produces the largest change in the frozen model's prediction relative to the nominal action. The selected actions are then evaluated offline in a simulator by comparing the model's predicted action response with the environment's actual response. Across three environment–model combinations and four perturbation budgets, the paper reports higher average response mismatch than random selection from the same candidate pool. The authors also fine-tune the model using selected actions and simulator supervision. This repair improves some component metrics but does not reduce overall response error on held-out episodes.
Critical strengths
AWARE requires neither additional training nor access to future environment outcomes during selection. Across three environment–model combinations and four budgets, it selects actions with higher average response mismatch than random selection from the same pool. This is a positive empirical result and provides initial support for using the heuristic to select offline test cases.
Critical weaknesses
-
The response-energy score is not sufficiently justified as a proxy for action-response mismatch. A large predicted response may be accurate, while an important action effect that the model fails to predict may receive a low score. Section 3.4 and Appendix A provide a conditional explanation, but the paper needs stronger evidence that these conditions hold in the tested settings.
-
Beating random selection does not establish an advantage over simple alternatives. Candidates vary in magnitude and may be clipped to action bounds, so larger executed perturbations could produce both larger model responses and larger mismatch. Comparisons with selecting the largest executed perturbation, and with directions at the same executed magnitude, would help determine whether AWARE adds useful information beyond perturbation size.
-
Task relevance and use on physical robots are not established. The paper does not show whether larger pixel-level action-response mismatch leads to poor action choices or task failure. Evaluation and supervised repair also rely on paired environment outcomes from the same initial state. How comparable responses would be obtained and used on physical robots remains unclear.
-
Repair does not reduce overall response mismatch on episodes not used for repair. On the episode-disjoint development set in Table 3, response RMSE rises from 0.09833 to 0.10222–0.11001 across all three repair configurations. Improvements in direction or magnitude do not establish better overall response accuracy. The experiment therefore does not demonstrate that AWARE-selected samples enable effective repair across episodes.
-
Figure 2 is difficult to read. The small labels and robot outlines are blurry. The figure should be redrawn or replaced with a clearer, higher-resolution version.
Questions for authors
Beyond higher average mismatch than random selection, what evidence shows that response energy identifies directions with large actual errors? Have the authors examined candidates with high response but low error, or low response but high error, within the same pool, including cases that AWARE misses? What evidence supports the directional alignment conditions in Section 3.4 in the actual evaluation settings?