acceptodds
Under review as a conference paper at ICLR 2027

CAD-WAM: Dexterous Manipulation Generation via Contact-aware World Action Model

19 views

Abstract

Generating 4D hand interaction trajectories from a single egocentric image and a language instruction requires anticipating how hand–object contact evolves across different hand regions. Most existing methods map vision and language directly to actions without explicitly imagining how the interaction unfolds, and their global conditions leave fine-grained contact under-constrained, so contact emerges only as an implicit by-product of global motion. We present CAD-WAM, a contact-aware world action model for dexterous generation which jointly imagines future egocentric video and hand actions, grounding trajectory generation in imagined interaction dynamics rather than static observations. Our key idea is to recast regional contact evolution from an implicit consequence of global motion into an explicit, predictable discrete prior for fine-grained refinement. Specifically, a coarse-to-fine dual-stream Mixture-of-Transformers (MoT) based network is proposed for both dexterous and imagination generation, where high-noise steps establish global interaction dynamics and a contact-aware action expert injects predicted regional contact skill codes at low-noise steps to refine finger articulation and contact transitions. The codes are learned by a spatiotemporal VQ-VAE over six palmar regions and predicted from the instruction and the initial contact state, requiring no future observation at inference. The anatomical contact atlas engine further extracts vertex-level contacts from egocentric videos into six anatomically defined regional atlases. The detailed experiments are conducted to show the effectiveness of our method. On HoloAssist and ARCTIC, CAD-WAM reduces MPJPE by over 39% and wrist translation error by over 30% relative to the strongest baseline. Code will be released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Hand Object Interaction Generation, Hand Interaction Trajectory Generation, World Action Model.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_z5f1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The proposed method, named CAD-WAM, generates hand trajectories from an initial egocentric image, a language instruction, and an initial contact atlas. It combines a pretrained video model with an action-generation expert, then refines motion using predicted discrete contact codes for six anatomical hand regions. An asymmetric attention mask allows auxiliary video supervision while restricting the refinement branch’s access to future-video tokens.

Critical strengths
  1. The reported trajectory accuracy is strong. In detail, Table 1 reports MPJPE reductions of 39.3% on HoloAssist and 44.9% on ARCTIC relative to the best listed baseline. Improvements also appear in articulation and rotation metrics.

  2. CAD-WAM proposes an useful anatomical representation. Regional contact codes provide a plausible intermediate representation between task instructions and finger coordination.

  3. The resaerch topic is hot and meaningful. Introducing WAM into dexterous hand manipulation generation is promising in this area.

Critical weaknesses
  1. The central contact claim lacks direct validation. Pseudo-contact labels are derived from 2D hand–object overlap, which does not establish physical 3D contact. The paper does not quantitatively establish contact-code prediction quality or resulting contact accuracy. THe lack of object geometry at inference does not preclude object-aware evaluation where annotations are available.

  2. The pipeline may be fragile. In detail, the entire anatomical contact atlas (ACAE) depends on pseudo contact labels obtained from HaMeR results plus SAM2 segmentation, rather than physical contact measurements from dedicated sensors. The whole contact-code learning pipeline is contaminated once hand reconstruction fails. The paper only briefly acknowledges this risk and provides no perturbation experiments under reconstruction noise, so the model’s tolerance to hand-registration errors is unknown. Inference requires high-quality hand reconstruction on the first frame, which strongly limits practical applicability.

Questions for authors
  1. Only hand-pose metrics are evaluated, and contact quality metrics (e.g., contact F1) are absent.
Reviewer_CaFXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CAD-WAM, a contact-aware world action model for generating dexterous hand-object interaction trajectories. The method combines regional contact priors with a hierarchical video-action generation framework, using coarse-to-fine refinement to improve hand articulation and motion accuracy. Experiments on HoloAssist and ARCTIC demonstrate substantial improvements over existing methods.

Critical strengths

S1.Reasonable technical design. The integration of regional contact priors with hierarchical world-action modeling is well motivated. The proposed components provide a structured approach to improving hand trajcetory generation.

S2.Promising experimental results. The method achieves substantial improvement across multiple hand trajectory metrics on two benchmarks. The ablation studies also provide evidence supporting the effectiveness of the proposed components.

Critical weaknesses

W1.Although contact-aware generation is the central contribution, the evaluation primarily focuses on hand pose and trajectory errors. These metrics do not fully establish whether the generated motions preserve accurate hand-object contact dynamics.

Questions for authors

Q1.Can the authors provide more direct evidence that the proposed contact priors improve hand-object interaction consistency, beyond improvements in hand trajectory accuracy?

Q2.Could the authors further clarify the independent contributions of contact-aware modeling and future video imagination relative to the underlying WAM backbone?

Reviewer_QLeMOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses generating 4D (MANO) hand interaction trajectories from a single egocentric image, a language instruction, and a first-frame palmar contact atlas, arguing that fine-grained contact is under-constrained when it emerges only implicitly from global motion.

CAD-WAM combines a dual-stream Mixture-of-Transformers world action model (a Wan2.2-5B video expert plus an action DiT) with a low-noise contact-action expert that is conditioned on six regional contact skill codes; the codes come from a spatiotemporal VQ-VAE over six palmar contact atlases (built from HaMeR/SAM2 pseudo-contacts) and are predicted from the instruction and initial contact, and an asymmetric attention mask lets future-video supervision reach the refinement stream without the actions attending to future video.

On HoloAssist and ARCTIC it reduces MPJPE by 39.3% and 44.9% relative to the best retrained baseline (LatentAct, LatentAct-Diff, MEgoHand), with ablations on contact representation, codebook size, atlas layout, attention mask, and losses.

Critical strengths
  1. Large and consistent accuracy gains over retrained baselines. Under the same splits, with large-displacement samples retained rather than filtered (Appx. A.1--A.2), CAD-WAM lowers all six metrics on both datasets (Table 1), e.g., MPJPE 13.35 →\rightarrow 8.10 cm on HoloAssist and 11.65 →\rightarrow 6.42 cm on ARCTIC, and the advantage also holds under LatentAct's filtered protocol (Table 6). Because HoloAssist uses unseen verb--noun combinations and ARCTIC a cross-subject split, this indicates a substantially stronger trajectory generator than prior image-and-text-conditioned methods.

  2. An explicit, anatomically structured contact prior that consistently helps. Discretizing per-region contact evolution into skill codes that are predictable from the instruction and the initial contact (Sec. 3.2) is a clear way to make contact a usable conditioning signal without future observations. The ablations consistently favor the design choices: image-atlas codes over vertex codes (Table 2), local over global atlases (Fig. 5), codebook size 64 over 16 (Table 4), and the auxiliary video and reference-space losses (Table 5).

Critical weaknesses
  1. Most of the gain appears to come from the large pretrained world-action backbone, and the "imagination" is not used at inference. The WAM baseline without any contact component already reaches 8.74 cm MPJPE on HoloAssist (Table 2), about 35% lower than the best external baseline (13.35 cm), while all contact-specific components together bring it to 8.10 cm. None of the baselines builds on a pretrained video-generation model (and their sizes are not reported), and no other video-prior or world-action baseline is compared (e.g., the joint video--action denoising approach of Wang et al. (2026), which the paper discusses). It is therefore unclear how much of the headline 39--45% improvement is due to the proposed contact modeling. Moreover, future-video tokens are omitted at inference and the refinement stream is never allowed to attend to them (Sec. 3.3, ACAAM), so the trajectory is not actually conditioned on imagined video at test time; the claim of "grounding trajectory generation in imagined interaction dynamics" (abstract) is overstated. It is also not stated whether the baselines receive the first-frame contact atlas that CAD-WAM uses, or how the predicted translation offset Δp\Delta p is anchored.

    • Suggestion: add a video-prior or world-action baseline (e.g., Wang et al., 2026) and report baseline sizes, revise the "imagined interaction dynamics" claim to match inference, and state the baselines' inputs and how Δp\Delta p is anchored.
  2. The evaluation does not test the paper's central "contact-aware" claim. All six reported metrics (MPJPE, PA-MPJPE, MPVE, PA-MPVE, MWTE, MRE; Sec. 4.1) compare a prediction with a single ground-truth trajectory. There is no measure of contact correctness or contact transitions, no diversity or multi-sample evaluation, and no variance across seeds, even though the method is generative and its motivation is fine-grained contact. The contact labels themselves are pseudo-labels from 2D hand--object mask overlap (Sec. 4.2) whose quality is not validated, and the predicted codes' accuracy (indexer accuracy) is not reported. In addition, the ablation numbers do not compose: individual variants reach 8.28--8.36 cm (Table 2, Fig. 6, Tables 4--5) while the full model is 8.10 cm, without a clear account of which configuration each number corresponds to.

    • Suggestion: add contact-specific metrics (e.g., contact accuracy or transition timing), multi-sample evaluation with variance, a validation of the pseudo contact labels with indexer accuracy, and a mapping of each ablation number to its configuration.
Questions for authors

Clarifications (could change my assessment):

  1. Do LatentAct, LatentAct-Diff, and MEgoHand receive the same first-frame contact atlas as CAD-WAM? Relative to what is the predicted translation offset Δp\Delta p defined, e.g., a given ground-truth initial wrist position?
  2. How does a strong video-prior baseline without the contact components compare, e.g., a world action model such as Fast-WAM or a joint video--action denoiser with the same Wan2.2 backbone? Is the WAM (Baseline) row in Table 2 exactly such a model, and how does it compare with the external baselines on ARCTIC?
  3. Since future-video tokens are omitted at inference, what role does video "imagination" play at test time? Does generating and conditioning on future video at inference help or hurt?
  4. How accurate is the contact prior indexer (per-region code accuracy), and how accurate are the ACAE pseudo-contacts against any available ground-truth contact (e.g., ARCTIC provides 3D object geometry)?

I am open to revisiting my rating if the rebuttal addresses these points well.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.