acceptodds
Under review as a conference paper at ICLR 2027

Attacking Action Discriminability in Latent World Models

Abstract

Latent world models support model-based control by learning action-conditioned state transitions in representation space. A key requirement of such models is that their predicted dynamics faithfully reflect the consequences of different actions, allowing downstream planners to distinguish alternative future trajectories. Many existing attacks are tied to specific action sequences or trajectories, whereas multiple alternative action sequences may reach the same goal, making persistence across plan variations and repeated replanning challenging. We instead target the action dependence of the latent dynamics itself, aiming to produce perturbations that are less tied to particular action sequences and remain persistent across multi-step rollout and replanning. Specifically, our attack operates at two complementary levels: action sensitivity suppression and multi-step discrepancy propagation. For the former, we construct action ensembles and directly reduce the dispersion of their predicted latent states, weakening the model's sensitivity to action variations. To further suppress the propagation of action-induced discrepancies over future rollout steps, we derive a finite-horizon sensitivity bound and reduce the dominant sensitivity of the latent transition dynamics. Additionally, we steer the attacked predictions away from their clean counterparts to induce erroneous rather than merely insensitive predictions. The proposed attack is evaluated on multiple JEPA-style latent world models across two control environments, demonstrating that suppressing action sensitivity in latent dynamics can effectively disrupt future-state prediction and downstream planning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
adversarial attack, latent predictive world models, action-conditioned dynamics, model-based planning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fC8vOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

AIDA attacks the initial visual observation of a latent world model. It makes predictions under different actions more similar while pushing them away from clean predictions. A third loss suppresses the state-transition Jacobian’s spectral norm to limit how action-induced differences grow during rollout. The paper tests the attack on LeWM, DINO-WM, and PLDM using PushT and OGBench Cube.

Critical strengths
  1. The attack targets a capability that planners depend on: distinguishing the consequences of different actions. Making one prediction inaccurate may leave other useful plans available; reducing differences across actions attacks the comparison itself. The sensitivity analysis gives a reason for including the propagation loss.

  2. The attack achieves the lowest task success rate in all six tested model–environment combinations. On Cube, LeWM success falls from 79.0% to 24.2%, and DINO-WM from 80.0% to 29.6%. The loss ablations and action-sensitivity measurements help explain the results, while the MPPI evaluation checks that the effect is not limited to CEM.

Critical weaknesses
  1. The information available to the attacker is unclear. Section 4.1 uses recorded future actions to construct action ensembles and subsequent clean observations for teacher-forced prediction. Access to model parameters and gradients does not by itself provide this information. The paper should state whether these trajectories are assumed to be available and what attack setting this represents. The evaluation also needs clarification: Cube cases are filtered for clean LeWM success, yet the reported clean success rate is 79%, and the separation between hyperparameter tuning and final evaluation is not clear.

  2. The results do not establish that action discriminability remains impaired after the controller receives fresh observations. Equation (8) and Figure 6 concern predictions within a latent rollout. Figure 5 shows that task performance remains affected under different replanning settings, but this could also reflect damage caused by the initial incorrect actions. Both effects matter, but they are different explanations. The paper should distinguish them and either add evidence after clean observations are reintroduced or narrow the persistence claim.

Questions for authors
  1. Does attack generation require the recorded future actions and the three subsequent clean observations? If so, how are these made available to the attacker? Can the attack also be constructed using only current and past observations?

  2. Why is clean LeWM success on Cube 79% after filtering for clean success? Please clarify whether filtering and evaluation use different seeds or settings, whether clean and attacked runs are paired, and whether tuning and evaluation use separate data.

  3. After a fresh clean observation enters the controller, does the model still produce less distinguishable predictions across actions, or is the remaining performance loss mainly due to the earlier actions?

  4. For the propagation loss, which history variables are differentiated, and at which prediction steps is the Jacobian evaluated?

5 comments
AuthorsOpenReview ↗

Response to reviewer fC8v regarding to Weakness 1, Questions 1 (Part 1/3)

We thank the reviewer for the insightful comments.

Information Needed for Attacks

"W1: The information available to the attacker is unclear. Section 4.1 uses recorded future actions to construct action ensembles and subsequent clean observations for teacher-forced prediction. Access to model parameters and gradients does not by itself provide this information. The paper should state whether these trajectories are assumed to be available and what attack setting this represents ..." and "Q1: Does attack generation require the recorded future actions and the three subsequent clean observations? If so, how are these made available to the attacker? Can the attack also be constructed using only current and past observations?"

Thank you for highlighting this ambiguity. We clarify that our main experiments adopt an oracle-assisted white-box setting, where the attacker is assumed to have access to a short future trajectory following the target frame, in addition to the model parameters and gradients. The expert actions guide action sampling, while future clean observations provide reference latents and reduce the influence of the world model's prediction errors. We have explicitly stated these additional assumptions in the revised threat model. To evaluate AIDA without future information, we additionally test an oracle-free variant that replaces expert actions with randomly sampled actions and future clean observations with model-generated rollout contex. This variant requires only current frame and achieves 29.8%/12% SR on Cube/PushT, compared with 24.2%/8.4% in the main setting. These results shown in the table below demonstrate that AIDA remains effective without access to future trajectories or expert actions. We have included these results and clearly distinguish the two settings in section 3.1 and 4.3.

MethodCubePushT
SR (%) ↓ASR (%) ↑FGD (cm) ↑SR (%) ↓ASR (%) ↑FPE ↑FAE (°) ↑
AIDA24.273.2814.828.491.57257.3724.40
AIDA (Oracle-free)29.866.6713.3412.086.89209.9224.59

AuthorsOpenReview ↗

Response to reviewer fC8v regarding to Weakness 1, Questions 2 (Part 2/3)

Evaluation

"W1: ... The evaluation also needs clarification: Cube cases are filtered for clean LeWM success, yet the reported clean success rate is 79%, and the separation between hyperparameter tuning and final evaluation is not clear." and "Q2: Why is clean LeWM success on Cube 79% after filtering for clean success? Please clarify whether filtering and evaluation use different seeds or settings, whether clean and attacked runs are paired, and whether tuning and evaluation use separate data."

Thank you for pointing out these ambiguities. Clean-success filtering was performed on a larger pool of episodes, while final evaluation reran only the selected successful episodes. Despite using the same seed, the different episode sets led to different random number generator states and CEM samples, resulting in a 79% clean SR. Importantly, clean and attacked evaluations are paired, using identical random seeds and random sampling sequences for each episode. To further ensure that our results are not specific to a particular evaluation seed, we further evaluate AIDA, TRAP and UADA/TMA using five different random seeds. Results are shown in the table below:

MethodSR (%) ↓ASR (%) ↑FGD (cm) ↑
TRAP74.20 ± 2.0221.20 ± 1.835.891 ± 0.261
UADA/TMA39.90 ± 1.7157.47 ± 1.899.90 ± 0.26
AIDA (Ours)26.00 ± 4.1771.57 ± 5.2914.35 ± 0.68

Across 5 seeds, ours consistently outperforms UADA/TMA. These results suggest that AIDA's effectiveness is not tied to a particular evaluation seed.

We acknowledge that the sensitivity analyses of loss weights and action-sampling noise scale were conducted on the original evaluation set. To verify that the selected hyperparameters generalize beyond this set, we additionally evaluate AIDA and other baselines on 200 new test cases for each dataset, using the same hyperparameters without further tuning.

World ModelMethodOGBench-CubePushT
SR ↓ASR ↑FGD ↑SR ↓ASR ↑FPE ↑FAE ↑
LeWMClean83.00–5.0086.00–28.436.51
TRAP76.5016.875.4880.5013.3726.446.82
UADA/TMA39.5058.439.5610.5087.79255.5123.13
Ours25.5070.4814.079.0089.53245.4622.76
DINO-WMClean79.00–4.5338.00–38.2813.75
TRAP80.5016.464.6623.5052.6351.4220.25
UADA/TMA51.0046.207.7517.0068.4257.8325.47
Ours33.5063.2910.2312.5075.0083.3129.71
PLDMClean46.50–8.6974.00–93.0010.58
TRAP42.0026.888.9072.009.4682.5411.43
UADA/TMA35.0052.6910.3113.0082.43266.4427.03
Ours31.0059.1411.988.0090.54325.4829.09

Across all three world models on both Cube and PushT, AIDA consistently achieves the lowest success rates and highest attack success rates, demonstrating its effectiveness over the adapted baselines. These additional experiments demonstrate that the attack performance is not specific to the original evaluation cases. The above experiments have all been added to section A.7.

AuthorsOpenReview ↗

Response to reviewer fC8v regarding to Weakness 2, Questions 3 and 4 (Part 3/3)

Impact of Clean Re-observation

"W2: The results do not establish that action discriminability remains impaired after the controller receives fresh observations. Equation (8) and Figure 6 concern predictions within a latent rollout. Figure 5 shows that task performance remains affected under different replanning settings, but this could also reflect damage caused by the initial incorrect actions. Both effects matter, but they are different explanations. The paper should distinguish them and either add evidence after clean observations are reintroduced or narrow the persistence claim. " and "Q3: After a fresh clean observation enters the controller, does the model still produce less distinguishable predictions across actions, or is the remaining performance loss mainly due to the earlier actions?"

Thank you for this important distinction. In the official planner, fresh clean observations are reintroduced at each replanning stage. Therefore, the performance degradation in Fig. 5 already reflects replanning with clean observations. We acknowledge that the subsequent failures are primarily due to the consequences of earlier incorrect actions. To further investigate this distinction, we compare the official planner with a variant that directly carries over the predicted latent from the previous stage instead.

SettingCube SR (%) ↓ASR (%) ↑FGD (cm) ↑PushT SR (%) ↓ASR (%) ↑FPE ↑FAE (°) ↑
Clean re-observation25.570.4814.079.089.53245.4622.76
Latent carryover16.080.7214.780.599.42291.4422.94

Without clean re-observation, ASR increases by 10.24 percentage points on Cube and 9.88 points on PushT. Fresh clean observations partially mitigate the downstream impact of adversarial latent predictions. Follow your suggestion, we have revised the abstract, introduction, related work and experimental discussion to distinguish reduced action discriminability within attacked latent rollouts from downstream performance degradation across replanning stages. Experiment results have been included in section A.8.


Propagation Loss

"Q4: For the propagation loss, which history variables are differentiated, and at which prediction steps is the Jacobian evaluated?"

While the main text uses a single-frame formulation for simplicity, our implementation accounts for the full temporal context. We compute the joint Jacobian Jz=∂(z^1,z^2,z^3)∂(z0adv,z1,z2)J_z = \frac{\partial (\hat{z}_1, \hat{z}_2, \hat{z}_3)}{\partial (z_0^{\mathrm{adv}}, z_1, z_2)}, differentiating the concatenated predictions at steps 1–3 with respect to the entire three-frame latent history while keeping the action embeddings fixed. All three predictions are obtained in a single forward pass. The spectral norm is estimated at the current perturbed history in each attack iteration, and the penalties are averaged across sampled action sequences. Only the initial observation is perturbed, while the remaining two history frames are kept unchanged during optimization. Related explanation has been added to section A.9 in the revised paper.

Reviewer_YrieOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a novel method to adversarially attack latent world models, such that they are unable to plan effectively. The core idea is to increase the action insensitivity of the world model by perturbing the initial observation shown to the model. The proposed loss encourages the different action sequences to lead to similar representations that deviate from the clean predictions. Additionally, the method encourages the differences between action sequences to grow less over the course of the rollout. The proposed method, AIDA, is evaluated on multiple different JEPA-style latent world models in OGBench-Cube and PushT, and AIDA outperforms the chosen baselines by reducing the success rate the most.

Critical strengths
  • The proposed method achieves good results in three different world models in different environments, and the ablation study confirms adding L_disp and L_prop improve the performance of the method even further.
  • The proposed model only needs to apply the perturbation to the initial frame, though frequent replanning (Figure 5) then reduces the strength of the attack.
Critical weaknesses
  • The threat model is unusually permissive, as oracle information from the test episode is used. My understanding of the paper is that AIDA uses the test episode’s expert action sequence A_g, and future clean observations from the same trajectory for creating the adversarial perturbation. The paper highlights that it needs no planner, but this most likely depends on access to A_g.
  • Given the above, the selection of baselines does not appear to make much sense. Neither original baseline was designed for the threat model chosen in this paper, as they are universal and do not use information about the test episode. Comparing the ablation results in Figure 4 with those in Table 1, even the naivest version of the proposed method (L_dev) outperforms all the baselines. TRAP uses predicted reward and value, whereas the latent world models in this paper use goal distance. UADA/TMA attacks VLA policies, which have no world model. AIDA applies a full image perturbation to the first observation, and the baseline methods apply universal patches to every frame in their original form.
Questions for authors
  • Could you please clarify, which oracle information does the method have access to?
  • How does AIDA perform without this oracle information? For instance, with A_g replaced by random actions or the planners plan, or without the future context frames.
  • How does AIDA compare to other baselines that create per-instance attacks with white-box access to the model?
5 comments
AuthorsOpenReview ↗

Response to reviewer Yrie regarding Weakness 1, Questions 1 and 2 (Part 1/2)

We appreciate the reviewer's careful reading and constructive feedback.

Oracle Information

"W1: The threat model is unusually permissive, as oracle information from the test episode is used. My understanding of the paper is that AIDA uses the test episode’s expert action sequence A_g, and future clean observations from the same trajectory for creating the adversarial perturbation. The paper highlights that it needs no planner, but this most likely depends on access to A_g.", "Q1: Could you please clarify, which oracle information does the method have access to?" and "Q2: How does AIDA perform without this oracle information? For instance, with A_g replaced by random actions or the planners plan, or without the future context frames."

We clarify that the main AIDA setting does use two forms of episode-specific oracle information: (1) the expert action sequence AgA_g, which is used to guide the selection of action samples, and (2) future clean frames, which provide reference latents and reduce the effect of the predictor’s own modeling error. This information assumption is also comparable to the adapted baselines: TRAP requires planner-related information, while UADA/TMA require the correct action and goal image. To directly test whether AIDA depends on such oracle information, we additionally evaluate a fully oracle-free variant on the same 500 data as in table 1 of our paper. We replace AgA_g with randomly sampled actions and replace the future clean context with model-generated rollout context. The attack remains effective as in table below:

MethodCubePushT
SR (%) ↓ASR (%) ↑FGD (cm) ↑SR (%) ↓ASR (%) ↑FPE ↑FAE (°) ↑
AIDA24.273.2814.828.491.57257.3724.40
AIDA (Oracle-free)29.866.6713.3412.086.89209.9224.59

Removing all oracle information causes a moderate performance drop. ASR decreases by 6.61% on Cube and 4.68% on PushT. Nevertheless, AIDA still remains highly effective, achieving SR of 29.8% and 12% ASR, respectively. These results show that the oracle information improves the attack but is not necessary for AIDA to remain effective. We have revised the threat-model description in section 3.1 to clearly distinguish the main oracle-assisted setting from this oracle-free variant and included the oracle-free results in section 4.3.

AuthorsOpenReview ↗

Response to reviewer Yrie regarding Weakness 2 and Question 3 (Part 2/2)

Baselines

"W2: Given the above, the selection of baselines does not appear to make much sense. Neither original baseline was designed for the threat model chosen in this paper, as they are universal and do not use information about the test episode. Comparing the ablation results in Figure 4 with those in Table 1, even the naivest version of the proposed method (L_dev) outperforms all the baselines. TRAP uses predicted reward and value, whereas the latent world models in this paper use goal distance. UADA/TMA attacks VLA policies, which have no world model. AIDA applies a full image perturbation to the first observation, and the baseline methods apply universal patches to every frame in their original form." and "Q3: How does AIDA compare to other baselines that create per-instance attacks with white-box access to the model?"

Thank you for raising this valuable suggestion. As no existing attacks specifically addressed our threat model as far as we know, we adapted methods originally developed under different settings. Our adaptations preserve their core attack mechanisms while aligning the perturbation setting and optimization interface with the JEPA-based world models studied here. Specifically, TRAP retains its tail-aware score suppression and gating mechanisms, replacing reward/value scores with goal-distance-based planning costs. This adapts the scoring interface to our planner without changing its underlying attack principle. The action-disruption and targeted-manipulation principles of UADA/TMA are adapted to the world-model planning setting, while the optimization interface is adapted from VLA policy outputs to action-conditioned latent predictions. All baselines are optimized per instance under the same first-observation, full-image perturbation constraint as AIDA.

To examine the effect of the original perturbation setting, we additionally evaluate TRAP using universal patches on 500 episodes. This yields SRs of 80.4% on Cube and 81.2% on PushT, with ASRs of 12.21% and 9.13%, respectively.

Following the reviewer's valuable suggestion, we further introduce GRAT (NeurIPS 2024), a gradient-robust feature-space attack that optimizes per-instance full-image adversarial perturbations by accounting for uncertainty in feature-space gradients. We adapt it to generate per-instance perturbations with white-box access to the world model, providing an additional baseline that directly attacks learned representations without relying on a VLA policy or reward/value predictor.

World ModelMethodCube SR ↓ (%)ASR ↑ (%)FGD ↑ (cm)PushT SR ↓ (%)ASR ↑ (%)FPE ↑ (px)FAE ↑ (°)
LeWMGRAT34.6063.5412.1526.4071.4388.7819.80
DINO-WMGRAT41.6054.758.9011.4074.7584.0628.35
PLDMGRAT30.4057.3312.2716.2079.33207.0423.79
AverageGRAT35.5358.5411.1118.0075.17126.6323.98
AverageOurs25.6769.1013.1810.5385.60219.3727.26

GRAT also demonstrates strong attack effectiveness, while our method consistently achieves better average performance across all metrics on both OGBench-Cube and PushT. These additional comparisons strengthen the evaluation, and we have incorporated the results and clarified the baseline adaptations in the revised paper.

We sincerely thank the reviewer for the constructive comments, which have helped us improve the paper, particularly in clarifying the threat model, evaluating the method without oracle information, and strengthening the baseline comparisons.

We would like to kindly follow up on our rebuttal and ask whether the reviewer has any remaining concerns. We would be happy to provide further clarification or discussion. Thank you again for your time and consideration.

Reviewer_pMUAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a method that targets the transition function's sensitivity to input actions, causing the model's outputs to predict similar latent states even with different action sequences. This makes the attack more general by not relying on reducing the sensitivity to action sequences observed in the data, which a planner can get around by resampling or replanning. The paper instead goes after the action sensitivity property the planner depends on to predict distinct future latents.

Critical strengths
  1. Focusing the attack on action discriminability is an effective strategy for attacks on action-conditioned WMs since the future predictions depend strongly on that aspect. Planners compare candidates, so attacking the model’s ability to tell actions apart makes more sense than attacking one trajectory. It also means the attack doesn’t need the planner’s candidate set. The evidence mostly supports this. The attack transfers from CEM to MPPI.
  2. The paper covers three models and multiple environments as well as considering different types of planners.
Critical weaknesses
  1. Most of the gain may come from a generic attack. The L_dev only ablation is a feature-space attack: it pushes latents away from the clean prediction. It already reaches 32% success on the Cube subset. The paper’s contribution, dispersion and propagation, add roughly 9 more points, which is an improvement but it's not clear why this is the case
  2. Connected to the first weakness, the paper doesn't analyse the effect of the attack on CEM. CEM ranks candidates and keeps the top ones, so shrinking all predictions uniformly toward one point would not change which candidate wins.
Questions for authors
  1. Can you analyze the effect of the attack on the planning (reversing winning candidates, etc.)
  2. Can you analyze the effect of the attack on an interpretable latent space? For that, you can use a simple MLP model that predicts the true state directly (i.e., the position of the T in the pushT environment) instead of the visual observation. This will provide a much more interpretable analysis of the perturbation effect.
4 comments
AuthorsOpenReview ↗

Response to Reviewer pMUA regarding weaknesses 1, 2 and Q1 (Part 1/2)

We thank the reviewer for the insightful and constructive comments.

Effectiveness of Dispersion and Propagation Losses

"W1: Most of the gain may come from a generic attack. The Ldev\mathcal{L}_{dev} only ablation is a feature-space attack: it pushes latents away from the clean prediction. It already reaches 32% success on the Cube subset. The paper’s contribution, dispersion and propagation, add roughly 9 more points, which is an improvement but it's not clear why this is the case"

We agree that feature deviation is an effective attack objective. To isolate the contributions of the other two components, we conducted an additional ablation using only (Ldisp+Lprop\mathcal{L}_{disp}+\mathcal{L}_{prop}), completely removing Ldev\mathcal{L}_{dev}, and it achieves an SR of 39.6% ±\pm 2.07% over five random seeds. Results show that suppressing action discriminability alone can also effectively disrupt downstream planning.

To further assess the generality of these objectives, we further incorporate Ldisp+Lprop\mathcal{L}_{disp}+\mathcal{L}_{prop} into GRAT, another feature-space attack baseline. On a Cube subset, this reduces SR from 40% to 28%. The results provide additional evidence that the benefits of our objectives are not specific to their combination with Ldev\mathcal{L}_{dev}, as shown in the table below:

MethodOGBench-CubePushT
SR ↓FGD ↑SR ↓FPE ↑FAE ↑
GRAT40.011.6031.084.6821.82
GRAT + Ours28.011.8217.0146.4224.89

The objectives target different properties of latent dynamics: Ldev\mathcal{L}_{dev}pushes predictions away from clean dynamics, whereas Ldisp\mathcal{L}_{disp} reduces the differences among action-conditioned predictions and Lprop\mathcal{L}_{prop} attenuates their propagation across rollout steps. These objectives address distrinct but complementary properties of latent dynamics, achieving the strongest attack performance when combined. We have included the new experiments and analysis in section 4.3 and A.6 of the revised paper to better clarify their respective contributions.


Effect of the Attack on the Planning

"W2: Connected to the first weakness, the paper doesn't analyse the effect of the attack on CEM. CEM ranks candidates and keeps the top ones, so shrinking all predictions uniformly toward one point would not change which candidate wins.", "Q1: Can you analyze the effect of the attack on the planning (reversing winning candidates, etc.)"

Following your suggestions, we further analyse the actual impact on CEM. Two candidate pools are examined: one from the initial CEM iteration and one from the final iteration of clean CEM planning, where candidates have concentrated around promising action regions. For each pool, the same 300 action sequences are evaluated under both clean and adversarial observations, ensuring a direct comparison of candidate rankings and elite selection:

MetricInitial PoolFinal Clean Pool
Attacked Rank of Clean Winner160.1 / 300156.3 / 300
Clean Rank of Attacked Winner200.5 / 300240.8 / 300
Spearman Rank Correlation-0.0790.022
Top-30 Overlap2.80 / 301.07 / 30
Elite Replacement Rate (%)90.7096.40

Results show that candidate rankings are largely disrupted, with Spearman correlations of -0.079 and 0.022 at two pools. Also, the top-30 elite sets are almost entirely replaced, with replacement rates of 90.7% and 96.4%. Additionally, the adversarially selected winners rank only 200.5/300 and 240.8/300 under clean evaluation. The results demonstrate that AIDA substantially alters candidate rankings and elite selection, disrupting the optimization process of CEM. The experiments and analysis have been added to section A.2 in the revised paper.

AuthorsOpenReview ↗

Response regarding Question 2 (Part 2/2)

Attack Effects in an Interpretable State Space

"Q2: Can you analyze the effect of the attack on an interpretable latent space? For that, you can use a simple MLP model that predicts the true state directly (i.e., the position of the T in the pushT environment) instead of the visual observation. This will provide a much more interpretable analysis of the perturbation effect."

Following your valuable suggestions, we analyse the effect of our attack in an interpretable state space using an MLP probe to decode predicted latent into physical state of PushT, including the position and orientation of T and the position and velocity of the agent. We roll out clean and adversarial latents under the same set of action sequences and measure the pairwise state differences across actions. As shown in the table below, our attack consistently reduces action-induced state differences across five rollout action blocks. For example, at the end of the fifth block, the average pairwise distance in T-block position decreases from 46.64 to 19.37 pixels, while the agent position difference decreases from 131.28 to 91.96 pixels. These findings demonstrate that the loss of action discriminability extends beyond latent features to interpretable physical-state predictions and persists across multi-step rollouts.

BlockT Position (px)T Angle (°)Agent Position (px)Agent Velocity (px/s)
115.05 → 7.1919.17 → 13.7166.09 → 37.0831.88 → 12.76
225.35 → 11.7628.90 → 19.2096.07 → 56.5739.65 → 17.66
335.11 → 14.6233.87 → 19.61115.49 → 74.1444.41 → 22.13
441.02 → 16.7337.62 → 23.01124.05 → 85.2546.56 → 23.68
546.64 → 19.3736.48 → 25.49131.28 → 91.9647.37 → 24.86

Corresponding results and analysis have been added to the revised paper section 4.3.

Thanks to the authors for the reply. All my concerns have been addressed. I keep my rating for acceptance.

Thank you for your positive feedback. We greatly appreciate your constructive comments and the time you have devoted to reviewing our work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response and Summary of Revisions

We sincerely thank all reviewers for their valuable feedback and constructive suggestions. We have carefully addressed the concerns raised in the reviews and revised the manuscript, including additional experiments, analyses, and clarifications. All revisions are highlighted in blue in the updated manuscript. The major changes are summarized below:

  1. Contribution Beyond Deviation Loss. According to reviewer pMUA, we have updated the ablation study in Section 4.3 (page 9) to include an experiment without the deviation loss, further examining the independent contribution of our action discriminability objectives. We have also added an experiment in Appendix A.6 (page 16), where we incorporate our action discriminability objectives into GRAT to evaluate whether they can further improve the effectiveness of an existing feature-space attack.

  2. Planner-Level Analysis. Following the suggestions of reviewer pMUA, we analyze how AIDA affects CEM candidate rankings and elite selection, providing direct evidence of its impact on the planning optimization process. Results are shown in Appendix A.2 (page 13, 14).

  3. Interpretable State-Space Analysis. Following the suggestions of reviewer pMUA, We use an MLP probe to decode latent predictions into physical states and demonstrate reduced action discriminability across multi-step rollouts in an interpretable state space. Results are added to Section 4.3 (page 9).

  4. Threat Model Clarification and Oracle-Free Evaluation. Following the suggestions from reviewers Yrie and fC8v, we clarify the additional information assumed in our main setting in Section 3.1, page 4 and introduce an oracle-free variant that requires no recorded future trajectories or actions, demonstrating that AIDA remains effective under Oracle-Free scenarios. The results are presented in Section 4.3 (page 10).

  5. Additional Baselines. Following the suggestions from reviewer Yrie, we include GRAT as an additional per-instance full-image perturbation baseline. The results have been added to Section 4 (page 7), with its corresponding analysis in Section 4.2 (page 8).

  6. Evaluation. Following the suggestions from reviewer fC8v, we clarify the evaluation robustness in Section 4.1 (page 7), and conduct evaluations across five evaluation random seeds. To assess hyperparameter generalization, we conducted additional experiments with AIDA and baselines on 200 new episodes per task without further tuning. These results are in Appendix A.7 (page 16, 17).

  7. Clarification of Attack Effects Across Planning Stages. Following the suggestions from reviewer fC8v, we have refined the relevant descriptions throughout the manuscript to more clearly distinguish the effects within latent rollouts from downstream planning failures after clean re-observation. We have also added an experiment in Appendix A.8 (page 17) comparing clean re-observation with latent carryover to further investigate these effects.

  8. Methodological and Implementation Clarifications. Following the suggestions from reviewer Yrie and fC8v, we clarify the Jacobian computation in Appendix A.9 (page 18) and baseline adaptations in Section 4.1 (page 7).

Thank the reviewers again for your constructive feedback, which has greatly helped us improve the quality of our paper. We would appreciate it if you could let us know whether our rebuttal has addressed your concerns, or if there are any remaining questions we could clarify. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.