acceptodds
Withdrawn from ICLR 2027

ΔWAM: Distilling Action Tangent Fields into World Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

World Action Models (WAM) improve robot policies by augmenting sparse action supervision with dense future prediction. However, much of the predictable future is dominated by appearance and scene persistence rather than action-dependent dynamics. We observe that several recent WAM designs, including optical flow, motion-centric representations, and latent actions, can be understood from a common perspective in which world supervision becomes more efficient as it contains a higher proportion of action-relevant variation. Based on this insight, we introduce Action Tangent Fields, which reformulate world supervision through a local Taylor expansion of how actions induce changes in future dynamics. We represent future dynamics in Residual VAE space, where the future latent remains recoverable from the current latent and its residual, and use a strong action-conditioned world model (ACWM) to probe the local correspondence between action variations and residual-world variations. This local first-order structure is distilled into the WAM to guide its denoising supervision toward dynamics that are more tightly coupled to action, rather than merely predictable from appearance. Across LIBERO-Plus, RoboTwin, and RoboTwin2.0-Plus, our method consistently improves robustness to lighting, background, layout, and other environmental perturbations. Despite using no large-scale embodied pretraining, it achieves stronger robustness under several distribution shifts than pretrained policies. We further distill multi-step VideoDiT denoising into a single step for efficient inference. Our results suggest that effective WAM supervision should remain information-rich while concentrating its predictive capacity on the directions along which actions change the future.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, World Action Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LvhFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how future prediction supervision in world-action models can better support action learning. The method uses future latent residuals relative to the current observation as prediction targets. It constructs an Action Tangent Subspace (ATF) by perturbing demonstrated actions and comparing the futures generated by an action-conditioned teacher model. While retaining supervision from the ground-truth future, the method assigns additional weight to prediction errors within this subspace. The teacher is only used to construct supervision offline, and actions are generated through simple forward passes during deployment.

The authors evaluate the method on multiple simulation benchmarks and two real-world robot platforms. The method provides a concrete way to use the teacher's local responses to action changes for policy training. However, the reliability of these teacher responses and the independent contribution of the proposed supervision structure still need further validation.

Critical strengths
  1. The way teacher knowledge is incorporated is reasonable.
    The method keeps the ground-truth future from demonstrations as the prediction target and only uses teacher responses to adjust the weights of prediction errors. This reduces the risk of directly learning from incorrect synthetic futures while preserving supervision over the full future latent representation.

  2. The method has initial empirical support and a relatively simple deployment process.
    The ablation without ATF suggests that the auxiliary objective may be useful, and the experiments cover different manipulation settings. Teacher queries and subspace construction are performed offline, so the 16B teacher is not needed during deployment.

Critical weaknesses
  1. The reliability and task relevance of the teacher's local responses are not sufficiently verified, and the impact of incorrect supervision directions remains unclear.
    ATF captures the teacher's responses to action perturbations, but these responses are not guaranteed to reflect real environment dynamics or changes that matter for task success. In particular, during contact establishment or slipping, the teacher may predict plausible futures but provide incorrect local responses. Fixing the generation conditions helps reduce sampling noise, but does not verify physical correctness. Although the ground-truth future remains the prediction target, incorrect subspaces may still cause the model to overemphasize irrelevant directions and affect action learning through shared representations. The paper does not directly verify whether the teacher's directions match real action responses or explain how unreliable directions can be identified and suppressed.

  2. The current comparisons are insufficient to isolate the contribution of the core designs.
    The improvements from ATF may also come from stronger auxiliary supervision or general motion priors. The paper lacks comparisons with mismatched subspaces and standard residual prediction losses with comparable supervision strength. In addition, the contributions of residual parameterization and additional robot video adaptation are not clearly separated. The residual representation is recoverable, but this property alone does not demonstrate that the supervision is more action-selective. Therefore, the current evidence is insufficient to attribute the overall improvements to the proposed local action structure. A more careful attribution analysis would help address my concerns and could improve my score.

Questions for authors

Key Questions That May Affect My Evaluation

  1. Are the teacher's directions physically correct and useful for control?
    Could the authors execute perturbed actions from the same initial simulation states to check whether ATF captures real changes in the environment, especially around contact establishment and slipping? I would also like to know whether unreliable teacher directions can cause performance to drop below the version without ATF, and whether effective-rank filtering or loss weighting can reduce this risk. Are there any safeguards in the method to prevent such problems?

  2. Do the improvements depend on local action structures matched to each sample?
    Under the same model, data, and training settings, does the correct ATF outperform ATF subspaces exchanged across samples and standard residual prediction losses with comparable supervision strength? These comparisons would directly affect my assessment of the core method's contribution.

  3. How much do residual representations and additional training each contribute?
    Under the same video adaptation data and training budget, how do absolute and residual latent representations perform with and without ATF? Please also clarify the teacher adaptation data and its relationship to the policy training and test sets.

  4. What are the actual deployment latency and offline preparation costs?
    Does the reported 0.43 seconds in Figure 4 include the full inference process, including image encoding, VideoDiT, and four ActionDiT steps? Please clarify the inference hardware, input settings, real-world action execution frequency, and replanning frequency. A baseline comparison under the same conditions would also be helpful. In addition, what are the costs of adapting the 16B teacher and generating ATF for each training window? These details would help assess the efficiency advantages and practical applicability of the method.

Separate Reporting Checks: Not Used in My Current Score, but Require Clarification

The following issues concern the statistical definitions and traceability of the reported results. I do not use them as reasons for my current score or rejection recommendation. However, they are more important than minor presentation issues, and I ask the authors to check each one and correct them if necessary.

  1. Aggregation rules for RoboTwin 2.0-Plus.
    Please clarify whether the Total column in Table 2 includes Original and whether the same rule is applied to all models. The Total for ΔWAM matches the average of all eight columns, while the Total for Fast-WAM matches the average of the seven perturbation categories. Under a consistent aggregation rule, the difference between the two models would be approximately 4.3 or 4.8 percentage points, rather than the reported 6.6 percentage points. Please verify the aggregation rules and check the reported improvements and baseline rankings.

  2. Task categories and the definition of Original in LIBERO-Plus.
    Please explain how the 1,564 Original instances in Table 5 relate to the seven official perturbation categories, and whether this metric is directly comparable to the Original results on standard LIBERO reported in baseline papers. Please provide the task IDs, category mappings, and per-instance results. Also, please check the difference between the total success counts of 8,804 and 8,802 in Tables 4 and 5.

  3. Actual scoring rules for Move Ingredients.
    Page 14 states that each correct step contributes one-third of the total score, but Table 6 includes a single-run score of 0.6. Please clarify whether this is due to truncated display or different weights for different stages. Also, please confirm whether Table 3 is calculated using the original unrounded scores.

Reviewer_hagEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies World Action Models (WAMs), robotic policies trained with both action supervision and future video prediction, and its core insight is that many predictable visual elements in future observations are independent of robot actions, so future prediction supervision ought to concentrate on the components of the future that are causally affected by actions. The proposed approach, ∆WAM, consists of three main components: a residual-VAE target formulation that isolates predictive targets by removing static first-frame latent information, Action Tangent Fields (ATF) which precompute and cache a basis capturing how futures change under small action perturbations using a large pre-trained action-conditioned world model, and ATF-guided post-training for a policy architecture that combines multiple loss terms, including an action flow-matching objective and an anisotropic loss that penalises prediction error along the ATF basis directions. At inference, the model runs a small fixed number of forward passes and update steps. Empirical evaluation across several simulated robotic benchmarks shows strong overall task success rates compared with relevant prior baselines, with ablation results confirming the benefit of including more action perturbation directions for building the ATF basis; the authors also note that the method’s performance degrades under camera and layout distribution shifts.

Critical strengths
  1. A clear, simple idea: weight world-model errors along directions an ACWM identifies as responsive to action, while keeping real futures as targets. This is safer than training on synthetic rollouts.

  2. The design is easy to deploy: the ACWM is used offline only, inference is a single pass, and there is no video decoding at test time.

  3. Broad evaluation covering three simulation benchmarks and two real platforms, including 20-DoF dexterous hands.

  4. Large robustness gains over Fast-WAM on LIBERO-Plus.

  5. Transparent reporting: per-task counts, per-trial real-world scores, and open acknowledgement of weaknesses.

Critical weaknesses
  1. Attribution. About 76% of the LIBERO-Plus gain comes before the ATF loss is added. Mid-training data, the residual target, and the baseline setup are confounded, and the Fast-WAM baseline looks weak.

  2. Missing key controls: random, shuffled, or PCA subspaces in place of Q; λ sweeps; no validation of the ACWM tangents against simulator ground truth; no seeds or CIs.

  3. Aggregate metrics are defined inconsistently:

——LIBERO-Plus Total includes Original instances; without them ∆WAM scores 85.8, which ties π0.5.

——Fast-WAM's RoboTwin 2.0-Plus Total is computed differently from the other rows (75.0 under the common definition, not 72.7).

  1. There are some overclaims in the paper, e.g.

——"Improves robustness to camera/layout" (camera and layout regress on some benchmarks);

——"distillation" (no procedure is described);

——"no embodied pretraining" (DROID/BridgeV2 mid-training and a robot-pretrained Cosmos teacher);

——"distilling tangent fields" (the student never learns a Jacobian).

  1. Technical gaps:

——the causal-VAE mismatch in the broadcast reference;

——normalizing by ‖∆A‖ has no effect after orthonormalization;

——perturbation size, rank threshold, λ values, and offline compute are unreported;

——the input and conditioning at t_v = 0 are unclear.

——The real-world gains are mostly within trial noise. Wipe Board degrades more than the baseline under appearance shift, and Fast-WAM is missing on Cook Meal.

Questions for authors

Q1. How was Total computed for each baseline? Please report all methods under one definition, with and without Original instances, and correct the Fast-WAM RoboTwin 2.0-Plus Total.

Q2. Can you provide controls for Fast-WAM + the same mid-training, absolute vs. residual targets (K = 0), and random/shuffled/PCA subspaces of the same rank, along with per-dimension K-ablations and ≥3 seeds?

Q3. How do the ACWM tangents compare with simulator-executed perturbations? What are the action-to-seed signal-to-noise ratio, subspace stability, perturbation magnitude, and the distribution of r_t and its tolerance?

Q4. How are I_t and the future frames encoded by the causal VAE? Does B = E(repeat(I_t)) behave differently?

Q5. What exactly is the input at t_v = 0, and what do the 1/2/4-step settings mean? Is there any distillation?

Q6. What are λ_A, λ_R, and λ_T, and how sensitive are results to them? What is the total offline compute (ACWM post-training, probing, number of windows, storage)?

Q7. Why were 264 LIBERO-Goal instances repeated, and what are the results under first-run selection? Where does the 8804 vs. 8802 discrepancy come from?

Q8. Can you report Fast-WAM on Cook Meal, more trials per cell with CIs, and a no-ATF real-world ablation?

Reviewer_D9tkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The submission tries to remove unrelated background details and autonomous object movements in a representation which they then use to train a policy to be more robust and generalizable. The methodology has three steps: Firstly, they change the VAE embeddings that the diffusion model would generate by subtracting the embeddings of the first frame from them, so it only includes variations comparing to the first frame rather than encoding all visual details of frames in its latents each time. This step is designed to remove static parts of scene and background. Then, they use an action conditioned world model to simulate multiple futures, each with slightly perturbed action conditionings, which provides them with the action tangent field (ATF), which represent the areas in the scene where the robot actions can have an effect on. Finally, they train their policy on this modified latent and using these cached ATFs, so their policy model can be more robust to non-relevant information. Their approach shows remarkable improvements in benchmarks such as Libero-plus, showing that the approach is working for the intended datasets and manipulation tasks.

Critical strengths

The action tangent field idea is interesting and a good formulation of previous works trying to manually design such spaces using optical flow or focusing on moving objects. The empirical results on the experiment section also share considerable improvement and state of the art performance which supports the generality of the method.

Critical weaknesses

There are two main weaknesses in the approach:

  1. The Residual VAE formulation: the authors describe in Sec 3.1 that they subtract the first frame's VAE latents from those of the future frames. They motivate this design to remove static parts of the scene, specially in the background areas. However there are three flaws here: firstly, the 3D VAEs of video diffusion models are usually causal, meaning that future frames embeddings also have access to the first frame, so they could already be handling such residual behaviour implicitly, which doesn't make sense to repeat again explicitly. Secondly, the first frame in WAN is usually encoded individually with no temporal compression, while future frames are (so every 4 temporal frames is compressed into one temporal token), so subtracting the compressed tokens with the non-compressed one might result in some issues. Finally, even the motivation only applies to limited scenarios where the camera and the scene are mostly static, and when there is camera motion this would become less relevant, like in driving scenarios or where the camera is mounted on the robot itself, which can be viewed in table 1 where camera perturbation results in a significant drop in performance from 98.2 to 60.9, while the drop is much less for Cosmos Policy. The RoboTwin experiment (table 2) also shows similar trend, where the model collapses to 19.1 performance from the original 92.2, ranking the worst model across all the baselines!

  2. The second weakness is in the action tangent field formulation. The method conditions an action conditioned world model on different perturbed actions and use their decoded predictions and variations in them to identify regions in the video that can be changed due to the change of action. They argue this would locate regions of interest (although locally) that the policy should focus on. This however builds on the assumption that given the same starting seed and starting state (image), and slightly perturbed actions, all generated results are the same except the ones caused by the change in conditioning (e.g. where the robotic arm would be in the scene). This however does not necessarilly hold in a 16B diffusion model, and even slight changes in the action conditioning can change the whole behaviour of the model in some aspects; in short there might be spurious correlations the model has learned between action representations and non-relevant features in the scene. And example could be the trajectory of an autonomous agent in the scene (like a cat!) could be changed when you perturb the conditioning, hence the cat would also enter the action tangent field, while obviously not controllable/relevant to the manipulation task.

Questions for authors

Regarding the main weaknesses above, I have two questions for the authors:

  1. Could you address the first weakness, specially by providing controlled ablation studies to isolate the role of the Residual VAE in the performance of your model, and under various settings (static/dynamic camera)?

  2. Although there might be spurious correlations in the model, they might not be very severe. So an experiment evaluating these spurious correlations, perhaps visualizing where the action tangent field focuses on with heatmaps on multiple samples would be useful (so both quantiative and qualitative experiments would be appreciated).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.