- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper formulates adversarial robustness evaluation for world-model agents as a finite-budget search problem and proposes WMAttack, which combines Representation-Guided Attack Retrieval (RGAR) with Self-Correcting Attack Search (SCAS). RGAR retrieves historically effective attack configurations using representation and behavioral similarity, while SCAS uses current-task feedback and an LLM-assisted proposer to refine candidate selection under a scout–confirm evaluation protocol; controlled experiments cover 26 Atari and 20 DeepMind Control tasks with DreamerV3, with additional evaluations on other world-model architectures.
Critical strengths
The careful attempt to turn adversarial evaluation into a controlled search protocol rather than reporting only a hand-tuned attack, including scout/confirm allocation and an independent held-out testing stage.
Critical weaknesses
Matched online episodes do not imply matched attack computation: a configuration with more optimization steps or restarts can consume substantially more victim forward/backward evaluations within the same episode, and WMAttack additionally invokes retrieval and LLM components.
Questions for authors
-
The matched-budget evaluation is a substantial improvement, but the budget is defined primarily in terms of online attack episodes. Could the authors additionally report total victim forward evaluations, backward/gradient evaluations, wall-clock time, and LLM cost for WMAttack and each search baseline?
-
The controlled experiment uses a common discrete configuration menu and a fixed 10-pixel perturbation radius within each attack family. How does WMAttack perform when attack family, perturbation radius, optimization steps, restarts, and allocation policy are searched jointly, rather than under the more restricted controlled menu? This would better support the broader formulation of automated attack-configuration search presented in the paper.
-
Please further analyze the sensitivity of the search outcome to the utility weights assigned to return degradation, policy disagreement, runtime, and return variance. In particular, does changing these weights alter which configuration is ultimately selected or the ranking among search methods? A Pareto-style analysis would be more informative than relying on a single scalarization.
-
The historical ablation is somewhat intriguing because SCAS without RGAR is not uniformly beneficial in the historical evaluation, whereas the complete system performs strongly. Could the authors explain more clearly why self-correction can become ineffective or detrimental without a good retrieval initialization?
-
The LLM-specific contribution appears statistically positive but relatively modest compared with the overall effect of the full search pipeline. Could the authors provide a stronger deterministic, LLM-free baseline using the same candidate construction, feedback features, and evaluation budget?
-
The theoretical result decomposes finite-budget suboptimality into coverage, selection, and estimation terms, but the assumptions are fairly generic and do not directly establish why RGAR or SCAS should improve these quantities in realistic world-model evaluation. Could the authors more explicitly distinguish what is formally guaranteed from what is empirically observed?