acceptodds
Withdrawn from ICLR 2027

VSAD: Valuation-Sensitivity Attacks and Robust Defense for Latent World-Model Planning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model-based reinforcement learning (MBRL) achieves high sample efficiency by planning in the latent space of a learned world model. However, planners select actions by ranking candidate trajectories according to their predicted returns, making them vulnerable to local valuation errors.We show that these errors can be amplified by the planning process and identify valuation sensitivity as a characteristic signature of fragile high-value trajectories. Specifically, trajectories that receive spuriously high predicted returns exhibit substantial changes in valuation under small perturbations of the input observation, which correspondingly alter the initial latent state. They are also frequently associated with abnormal control patterns, such as action jitter and excessive impact, whereas successful trajectories exhibit more stable valuations. We further demonstrate that this sensitivity can be exploited at test time. In the white-box setting, bounded latent-state perturbations manipulate trajectory rankings and promote unreliable action sequences into the planner’s elite set. In the black-box setting, NES-based observation perturbations directly alter the target planner’s trajectory selection and induce similar control degradation. Based on this finding, we propose a training-free, model-agnostic defense that re-evaluates each candidate trajectory under multiple perturbed observations and ranks trajectories using a conservative score combining mean predicted return, return variance, and action smoothness. The defense suppresses trajectories whose high scores depend on unstable value estimates, without modifying the world model or requiring additional training data. Experiments show that it substantially reduces the effectiveness of both white-box and black-box attacks while preserving normal performance in clean environments. These results establish valuation sensitivity as a mechanistic signature of fragile trajectories, an exploitable attack surface, and a practical signal for robust planning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Model-based reinforcement learning, Adversarial attacks, Latent-space planning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_fttaOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This manuscript studies how sensitive value estimates are in latent world-model planning. It proposes a white-box attack on the latent state and a black-box attack on observations. Both aim at disrupting trajectory selection, along with a training-free defense that scores candidate trajectories using perturbed-return statistics and action smoothness. Experiments over 50 Meta-World tasks report attack effectiveness, defense performance, and what each scoring term contributes.

Critical strengths

S1. The paper studies a real vulnerability in world-model planning, and it pairs attack design with a matching test-time defense. S2. The experiments span 50 tasks and includes component ablations, so there is some evidence for how the proposed approach behaves.

Critical weaknesses

W1. It seems that both the methodological novelty and the conceptual consistency are unestablished. I could not tell how this differs from prior relevant studies, nor whether the mechanisms as described match what was actually implemented. W2. The experimental results does not carry the weight of the central claims. The reported metrics are inconsistent with each other, the diagnostic results are ambiguous, and the attacker is not trained under equal settings across experiments.

Questions for authors
  1. The proposed method is not clearly distinguished from existing work on planner overfitting, perturbation-based evaluation, and conservative trajectory scoring.
  2. Some entries in Table 1 do not add up under the paired-episode ASR of Equation (7). For undefended TD-MPC2-48M under white-box attack, 86.1% clean success and 81.6% ASR together require at least 15.8% of episodes to still succeed, yet the reported attacked success is 12.4%.
  3. Figure 1 does not support the interpretation the paper draws from it. In panel (b), both groups show a negative prediction-minus-environment return gap, which points to underestimation, not the claimed overestimation — unless the sign or the return comparison is wrong. Panel (c) shows increased sensitivity in only 54% of tasks, with substantial overlap between the clean and poisoned evaluations.
  4. The black-box objective in Equation (4) targets action matching. That is not the same as valuation instability, nor as reversing the trajectory ranking.
  5. The abstract, Section 4, the conclusion, and Appendix C all speak of a variance penalty, while Equation (6) and the sensitivity definition use standard deviation.
  6. Appendix A trains the standalone white-box attacker for 3,000 steps with 10,000 environment steps, but the defense experiments use only 500 attack-training steps and 1,000 warm-up steps. The defense improvement reported there may therefore be partly due to a weaker attacker rather than to the defense itself.
Reviewer_4oKhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that in model-based reinforcement learning, planners aggressively select action sequences with high predicted returns, making them vulnerable to local valuation errors. The authors propose valuation sensitivity as a signal for identifying potentially overestimated trajectories, based on the finding that their predicted returns are sensitive to small perturbations of the input observation. They then develop both attack and defense methods based on this finding.

Critical strengths
  1. The paper proposes a simple and intuitive approach to measuring trajectory valuation stability. The proposed defense is training-free and does not require modifying the pretrained world model, making it easy to apply to existing planners.
  2. The paper reports substantial improvements across multiple models and tasks, with clear and easy-to-understand illustrations.
Critical weaknesses
  1. The motivation is not sufficiently supported by the experiments. In Figure 1(b), both groups have negative average gaps, indicating underestimation rather than the claimed overestimation. Although failed trajectories have a higher gap, this may simply be due to their lower true returns. The result only shows that successful trajectories are underestimated more severely but does not prove that failed trajectories are overestimated. In Figure 1(c), the criteria for dividing the tasks into 45 failure tasks and only 5 success tasks are unclear, and the sample size of only 5 success tasks is too small to draw reliable statistical conclusions. Moreover, only 54% of tasks show higher sensitivity under poisoning, which is not strong evidence for the authors' claim. The paper does not clearly establish the relationship between valuation overestimation and valuation sensitivity, which weakens its core motivation.
  2. The comparison with existing defense methods is limited. The paper only compares against two baselines without evaluating other relevant robust planning or defense methods. This makes it difficult to assess the advantages of the proposed approach over existing methods.
Questions for authors

Please see the Critical Weaknesses

Reviewer_GwFmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates valuation sensitivity as a vulnerability in latent world-model planning. The authors show that small perturbations to latent states or observations can alter the predicted returns of candidate trajectories and influence action selection. They introduce white-box latent-state and black-box observation-space attacks targeting trajectory rankings. To mitigate these attacks, they propose a training-free test-time defense that re-evaluates candidate trajectories under multiple perturbations and combines return statistics with an action-smoothness penalty. Experiments on several world-model planners demonstrate substantial recovery of task performance under the evaluated attacks, including a defense-aware white-box attack.

Critical strengths

1.The paper identifies trajectory-ranking manipulation as a meaningful attack surface in latent world-model planning. 2.The proposed defense substantially improves robustness under the evaluated attacks, supported by component ablations and defense-aware attack experiments.

Critical weaknesses

1.Although the paper compares against DD and Q-Ensemble, the proposed mean-variance and action-smoothness scoring remains insufficiently distinguished from conventional risk-sensitive MPC objectives, particularly without matched-compute LCB or CVaR-based planning comparisons. 2.The proposed interpretation of valuation sensitivity as an indicator of world-model estimation errors is insufficiently validated, since high sensitivity may also arise from genuinely unstable environment dynamics rather than inaccurate value predictions. 3.The defense introduces considerable computational overhead, with black-box attack evaluation time increasing from approximately 6 to 10 minutes per episode. Although this cost is reported, the paper lacks a per-planning-step latency breakdown and matched-compute comparisons to establish its practical efficiency. 4.Although the paper evaluates a defense-aware white-box attack, its ASR increases from 8.8% to 20.3%, indicating sensitivity to attacker adaptation. The robustness limits remain uncertain without stronger attack-budget sweeps or evidence of convergence against the full defended planner.

Questions for authors

1.How does VSAD compare with mean-variance, lower-confidence-bound, or CVaR-based planning under identical computational budgets? 2.Beyond the reported episode-level wall-clock measurements, what is the per-planning-step latency overhead of the defense, and how does robustness compare with alternative planners under matched compute budgets? 3.Can the authors distinguish valuation sensitivity caused by model estimation errors from sensitivity caused by genuinely unstable environment dynamics?

Reviewer_GTnCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the vulnerability of latent world-model planners to errors in candidate-trajectory valuation. It identifies sensitivity of predicted returns to small state perturbations as a potential indicator of fragile trajectories and evaluates both internal latent-state and observation-space perturbations. The proposed training-free defense re-evaluates candidate trajectories under perturbations and ranks them using mean predicted return, a sensitivity penalty, and action smoothness. Experiments evaluate frozen TD-MPC2 models and NEWT across 50 Meta-World tasks.

Critical strengths
  • The paper addresses an interesting robustness problem in model-based RL. Studying how perturbations manipulate candidate-trajectory rankings provides a useful perspective beyond measuring changes in the final action or task return alone.

  • The defense is simple and training-free. It modifies the trajectory-scoring rule without retraining the world model or requiring attack labels. The reported results also suggest that clean performance is largely preserved.

  • The evaluation covers a substantial manipulation benchmark. The stated protocol includes all 50 Meta-World tasks, 50 episodes per task, and three pretrained checkpoints, providing useful coverage across tasks and model sizes.

  • The paper includes useful ablations and a defense-aware evaluation. The scoring ablations examine the contributions of sensitivity and smoothness, while Table 2 considers an attacker that knows the defense. The paper also acknowledges the privileged access required by the white-box setting, high black-box query costs, and lack of formal robustness guarantees.

Critical weaknesses
  • Some reported success rates and ASRs are inconsistent with the stated metric definition. Assuming Table 1's Clean column provides the paired reference used in Eq. (7), the equal episode counts imply Sattack≥Sclean(1−ASR)S_{\mathrm{attack}} \geq S_{\mathrm{clean}}(1-\mathrm{ASR}), with rates expressed as fractions. The undefended TD-MPC2-48M white-box result therefore requires at least 15.8% attacked success, but reports 12.4%. NEWT's defended black-box result similarly requires at least 73.9%, versus 70.5% reported. Rounding cannot explain these discrepancies, and the pairing, aggregation, and reference values need to be reconciled.

  • The white-box comparison may use differently trained attacks. Appendix A reports 3,000 training steps and 10,000 environment steps for the standalone attack, but only 500 training steps and 1,000 collection steps for the defense configuration. It is unclear which attack checkpoint produced each row in Tables 1–3. If the compared results use these different configurations, the reported improvement cannot be attributed solely to the defense. Matching the perturbation bound does not resolve this issue; the black-box parameters are explicitly unchanged, so this concern is specific to the white-box evaluation.

  • Robustness to adaptive attacks is not sufficiently established. In the black-box setting, the defense uses random observation perturbations, but the paper does not clarify whether these draws are shared or resampled across attack queries. Such randomness could increase the variance of attack estimates and potentially overstate robustness. Explicitly accounting for defense randomness is therefore important [1, 2], yet the manuscript does not explain how this is handled or report query-budget sensitivity and convergence analyses. Keeping attack-side parameters unchanged supports comparability but does not resolve this concern. The only explicitly defense-aware experiment (Table 2) is white-box, limited to TD-MPC2-48M, and described only at a high level. It increases the reported defended ASR from 8.8% to 20.3%, although the defended planner still substantially outperforms the undefended one (77.2% versus 10.8% success). These results provide useful but limited evidence, and a clearer evaluation accounting for defense stochasticity is needed.

  • The evidence for valuation sensitivity as a mechanism of failure is not fully convincing. Section 4 defines the valuation gap as predicted minus environment return and interprets Figure 1(b) as systematic overestimation, yet the annotated gaps for both groups are negative. The authors should clarify the sign convention and whether the compared returns use consistent horizons, discounting, and continuation policies. Negative gaps do not rule out ranking errors, but they do not support the stated absolute-overestimation interpretation. More generally, sensitivity alone does not establish prediction error, and a per-candidate analysis relating sensitivity, valuation error, and realized performance would better support the proposed explanation.

  • Important baseline and evaluation details remain under-specified. Appendix B defines Denoising Defense through the operator ADDA_{\mathrm{DD}} without specifying its aggregation rule, making the baseline difficult to reproduce from the manuscript or distinguish from Mean-Only. The black-box interface permits observation queries and action feedback, whereas Appendix A uses reward-based activation; the source of this reward information should be clarified. The manuscript should also specify the observation normalization and which coordinates are perturbed. Finally, although Appendix A reports the values of β\beta, λ\lambda, and the noise scales, it does not explain how they were chosen or what the defense start fraction (set to 0.5) means. These details are needed to reproduce the evaluation and assess the fairness of the comparisons.

[1]: Tramer, Florian, et al. "On adaptive attacks to adversarial example defenses." Advances in neural information processing systems 33 (2020): 1633-1645.

[2]: Athalye, Anish, et al. "Synthesizing robust adversarial examples." International conference on machine learning. PMLR, 2018.

Questions for authors
  1. Are the Clean column and ASR computed from the same paired episodes? Please provide the paired success/failure counts and reconcile the TD-MPC2-48M white-box and NEWT defended black-box results with Eq. (7).

  2. Which attack-training configuration and checkpoint produced each defended and undefended white-box result in Tables 1–3? Can the authors provide comparisons under matched attack-training and data-collection budgets?

  3. Are the defense's random perturbations shared or resampled across attack queries, and how does the evaluation account for this stochasticity? Please clarify the construction of the defense-aware attack in Table 2 and provide evidence that the reported robustness is not an artifact of noisy attack estimates, including randomness-aware evaluation, query-budget sensitivity, and convergence analysis.

  4. Why are the valuation gaps in Figure 1(b) negative under the stated definition? Are predicted and environment returns comparable, and can the authors quantify whether sensitivity predicts candidate valuation error and failure across tasks?

  5. What is the exact Denoising Defense aggregation rule, and how were the baseline and proposed defense hyperparameters selected? In particular, how were β\beta, λ\lambda, and the noise scales chosen, and what does the defense start fraction of 0.5 control? Please also clarify the reward-based activation condition, observation normalization, and which observation coordinates are perturbed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.