- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the robustness of latent world models against perturbations on observations under the white-box setting. It proposes two types of attacks, including state hallucination which forces the model to initiate its rollouts from an erroneous state, and dynamic shift which tampers with the model's transition dynamics. It also introduces TA-MDP framework to formalize the robustness of world models, proving that while a robust predictor alone is insufficient to guarantee robustness, a robust encoder is sufficient. Based on these theorems, it employs adversarial training using a method called FARE to fine-tune the encoder on task-specific data. Experimental results demonstrate that proposed attacks severely degrade the performance of world models, as well as downstream planning and policy, and FARE significantly mitigates these adverse effects.
Critical strengths
- This paper focuses on an important but underexplored problem that world models serve planning, policy evaluation, and policy learning in embodied AI, and a compromised world model can affect multiple downstream systems, so their robustness is well worth studying. And it also presents a complete study including two types of attacks considering state and transition, and TA-MDP framework and extended FARE for encoder-based adversarial training.
- This paper conducts comprehensive experiments covering four diverse world models, evaluating the attack performance on both the world models and downstream systems such as policy eval, action planning, and policy learning, as well as the effectiveness of the FARE defense. It further provides numerous comparisons and ablations and also reports inconclusive findings.
Critical weaknesses
- Table 1 notes ‘scale is environment-specific’, but lacks further elaboration. It could lead to confusion when making horizontal comparisons of similar conditions within the table, for example, 'Dino WM appears most vulnerable in the Push-T task, whereas Le WM shows greater weakness in the Wall task' (line 348). It would be helpful to provide a brief clarification regarding the units themselves or the quantities being compared here, although some illustrations are shown in Appendix.
Questions for authors
1.Can FARE be compared with other defense methods, that is, other than adversarial training, to further prove its effect and priority?
2 comments
Thanks for the authors' response. It is helpful to have such a clearer illustration of the definition and a more comprehensive experiment on defence performance.
Response to Reviewer oAqm
Thank you so much for your positive assessment of our paper, we greatly appreciate it. In the following response we aim to answer your questions and hopefully increase your confidence in our results. If you have any additional questions please feel free to ask us, otherwise please see our responses to the other reviewers for further details on how we improved our paper.
W1 “Table 1 notes ‘scale is environment-specific’, but lacks further elaboration.”
This is a fair critique. Therefore we’ve standardized all of our reported metrics so they’re all on the same scale. We’ve also added some additional interpretable metrics which measure distances directly in latent space rather than over state probes. Our updated metrics, Hr and Dr, are all normalized with respect to the distance between the true and adversarial target latents, and are therefore all on the same scale as well. We have found these latent space metrics to be more reliable, since the linear probes themselves often come with a high level of inaccuracy even given clean inputs (see our updated Figure 4 results on r2Dreamer).
Q1 “Can FARE be compared with other defense methods, that is, other than adversarial training”
This is a fair point, therefore in Table 11 and 12 we include additional evaluation against the JPEG compression defense [1] when targeted by a BPDA style “Expectation Over Transformation” attack [2]. The results show that the defense is able to mitigate the attack’s effect to an extent, but is unable to reach the complete nullification of FARE against this attack across Dino, Le WM, and r2Dreamer on Wall, Push-T, and Reacher respectively. The defense’s effectiveness also seems to be environment or world model dependant, as it works well on Le WM Push-T, but fares much worse on Dino Wall and r2Dreamer Reacher. The defense also fails to prevent the impact of state hallucination from reaching planning, as success rates drop from 86% -> 24% on Le WM Push-T and episodic returns in r2Dreamer reacher drop from 805 -> 91. It is also possible that more complicated defense techniques, such as certified robustness, can be adapted to world models, however such adaptation would require its own analysis and paper to be properly explored and implemented.
[1] Guo et. al. “Countering Adversarial Images Using Input Transformations”
[2] Athalye et. al. “Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples”