- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces LatentGauge, an inference-time attack on planning with latent world models. The central observation is that accurate latent prediction and preservation of the latent marginal distribution do not uniquely determine the geometry used by a downstream planner. The authors exploit this degree of freedom by applying a shared, radius-dependent rotation to predicted latent endpoints and goals. The transformation is constructed to preserve the Gaussian reference distribution while changing local distances and therefore the ranking of candidate action sequences.
The proposed attack uses calibration candidate pools from the planner, including predicted endpoints, goals, scores, and the native candidate-selection structure. For CEM, the method specifically targets pairs around the elite-selection boundary. A kernel parameterization controls the radial rotation profile, and the coefficients are optimized under a displacement constraint to reverse candidate margins while preserving the reference latent distribution.
The paper provides a substantial theoretical analysis connecting the transformation to the induced latent metric, the Gauss-Newton geometry of planning, candidate ranking, CEM elite-set changes, and propagation of these changes through later CEM updates. Empirically, the method is evaluated across DINO-WM, temporally straightened representations, PLDM, and LeWorldModel settings. LatentGauge consistently reduces planning success and produces considerably more ranking flips than identity, orthogonal, random-profile, and random-direction controls.
I find the central perspective interesting: prediction correctness and marginal-distribution preservation do not necessarily imply robustness of the geometry used for planning. However, I have concerns regarding the degree of methodological novelty and the strength of the attack-access assumptions.
Critical strengths
1. Interesting identification of a planning-specific blind spot.
The strongest conceptual contribution is the observation that correct world-model predictions and preservation of a regular latent marginal are insufficient to guarantee stable planning decisions. The paper separates prediction, distributional validity, and scoring geometry in a clean way. I think this is a useful perspective for evaluating latent world models.
2. Strong mathematical characterization of the mechanism.
The paper does considerably more than simply demonstrate empirical score changes. It derives the induced metric of the transformation, relates it to the action-side Gauss-Newton matrix, derives an exact candidate-cost shift, and analyzes how boundary crossings affect CEM elite selection and subsequent moment updates. This makes the failure mechanism unusually transparent compared with many empirical adversarial attacks.
3. Careful controlled validation.
The authors include useful controls such as identity transformations, global orthogonal rotations, random kernel profiles, and random-direction optimization. The controlled experiments support the specific claim that spatially varying measure-preserving geometry, rather than arbitrary latent movement, causes the observed ranking changes. The comparison between top-one and elite-boundary calibration is also informative because it directly connects the attack objective to the planner's native update rule.
Critical weaknesses
1. The individual methodological ingredients appear to have limited novelty.
This is my main concern. Measure-preserving transformations and latent non-identifiability are established ideas, Gaussian-preserving spiral/radial transformations have already appeared in related representation-learning work, and attacking rankings or imagined trajectories in world-model planners has also been explored. Kernel interpolation, constrained optimization, and pullback-metric analysis are likewise standard tools.
The paper combines these ingredients in a thoughtful way and adapts them specifically to planner scoring, but I am not convinced that the resulting attack constitutes a sufficiently new methodological contribution for ICLR. In my view, the strongest novelty is the formulation and analysis of the connection between these existing concepts rather than the attack construction itself.
2. The attacker requires unusually strong access to the planning pipeline.
The calibration procedure uses predicted latent endpoints, encoded goals, clean candidate scores, complete candidate pools, native elite counts, and planner-specific candidate boundaries. In CEM, the attack explicitly extracts the candidates at positions and around the elite boundary. The method therefore appears closer to a strong gray-box or planner-interface attack than to a generally applicable deployment-time adversary.
This matters because an attacker able to insert an arbitrary differentiable transformation directly between the world model and its scoring function already has substantial control over the decision pipeline. The paper would benefit from a much clearer justification of realistic scenarios in which this precise access is available while direct modification of the scorer or planner is not.
3. The practical relevance of preserving the Gaussian reference marginal is not fully established across all evaluated representations.
The strongest theoretical guarantee assumes preservation of the reference Gaussian law. However, several evaluated representations, including pretrained DINO features and other learned representations, are not necessarily distributed according to this exact reference law. The paper acknowledges this distinction and separately evaluates learned-feature detectability, but this weakens the interpretation that the transformation is generally "distribution preserving" in realistic victim representations.
The theoretical blind spot is therefore cleanest for explicitly Gaussian-regularized models such as LeWM. For the broader experiments, I would like stronger evidence that preservation of the theoretical reference distribution corresponds to actual stealth or validity under the empirical latent distribution.
4. Experimental scale is somewhat limited for an attack making broad claims about latent world-model planning.
Although the paper covers several model families and tasks, the primary checkpoint evaluation uses only 50 held-out goal pairs and three attack-calibration seeds. Furthermore, some comparisons to other attacks are inherited from different source protocols rather than being fully reproduced under a single matched threat model.
The results demonstrate the proposed mechanism, but stronger evidence across independently trained checkpoints, more environments, and matched implementations of competing attacks would make the generality claim considerably stronger.
Questions for authors
1. Threat-model clarification.
What concrete deployment scenario gives an attacker access to predicted latent endpoints, encoded goals, complete candidate pools, candidate scores, and CEM elite counts, while preventing the attacker from directly manipulating the scoring function or planner? I think clarifying this distinction is important for assessing the practical significance of the attack.
2. Novelty relative to existing Gaussian-preserving transformations.
Can the authors state more precisely what is mathematically new in the transformation itself relative to prior Gaussian-preserving spiral/radial constructions? If the transformation family is largely inherited, I think the paper should more explicitly position the contribution as planner-specific optimization and analysis rather than a new transformation family.
3. Novelty relative to existing world-model planning attacks.
Could the authors provide a matched experimental comparison against TRAP and/or WMAttack under the same victim models, attack information, and computational budget? Literature-reported numbers from different protocols make it difficult to determine how much additional attack capability LatentGauge provides.
4. Empirical latent distribution.
For representations that are not explicitly Gaussian-regularized, how closely does the empirical latent distribution match the reference distribution that the transformation preserves? Could the authors report distributional discrepancy before and after the transformation under the actual learned-feature distribution?
5. Reduced-access ablation.
How does the attack perform if the attacker does not know the native elite boundary or does not observe latent endpoints directly? A reduced-information experiment would help determine whether the attack reveals a broadly exploitable weakness or primarily a vulnerability under privileged planner access.