- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the experience reuse of world action models. The authors argue that WAMs struggle to capture underlying cross-task semantic relationships that could guide target action prediction. To this end, the authors propose Action Experience Dictionary to encode historical physical action trajectories. To model action-related motion and reduce reliance on irrelevant background cues, the authors propose motion-aware transition loss. Experiments show the effectiveness of the proposed method
Critical strengths
- Overall the paper is well-written and relatively easy to understand.
- The experiments show the effectiveness of the proposed method.
Critical weaknesses
-
The proposed method can not be distinguished from prefix tuning, or to be clearer, adding more learnable parameters to the model. The proposed AED is essentially an embedding layer. The dictionary encoder is also essentially more learnable parameters. As the losses are standard loss functions for world action models, no structural prior was introduced for these modules. Therefore there is no evidence that the proposed modules do learn the expected knowledge.
-
The proposed method rely on historical experiences, while the authors did not explain how to cold-start the model.
-
FAST tokenizer tokenizes physical trajectories described by rot6d, which hardly contains any semantic meaning such as grasping or carrying. Therefore, if FAST tokenizer outputs an ID, it does not necessarily mean "grasping". The soundness of the proposed method is suspicious.
-
The proposed AED does not require large-scale pretraining. Therefore, there is no guarantee that it can learn a wide range of semantics used by robot manipulation tasks.
Questions for authors
-
Please clarify what the AED and dictionary encoder has learnt through experiments. Even qualitative visualization is better than no further explanation.
-
Please explain how to start the model when the model has no historical experiences.
-
Does the cooperation of pre-defined FAST tokenizer and AED trained from scratch have concrete meanings? When FAST tokenizer outputs a certain ID, what does this ID mean to AED?
23 comments
Response to Reviewer ymMg: Clarification on cold start (2/5)
2. Cold start. Appendix C.1, “Model and Representation Configuration,” states that AED contains 2,048 valid entries plus a separate PAD entry; at cold start, the PAD entry represents the empty action history. Initialization details are given in Sec. 4.1 and Appendix C.1: the video expert is initialized from pretrained Wan2.2 weights (Sec. 4.1; Appendix C.1), the ActionDiT action expert is initialized by linear interpolation of the video-expert parameters (Appendix C.1), and the AED action embeddings are randomly initialized (Sec. 3.2).
Response to Reviewer ymMg: Clarification on FAST+ tokenizer (3/5)
3: FAST+ tokenizer and AED. a) A high-level skill can comprise multiple motion patterns, each indexed by a FAST+ ID, an single ID therefore denotes a discretized action pattern. A high level skill such as “grasp” would be represented by a sequence of ID. FAST+ maps each action segment to a discrete ID squence based on its motion pattern; AED uses that ID as an address to retrieve a trainable embedding from its shared dictionary. Thus, when FAST+ outputs ID , it tells AED which dictionary vector to retrieve and share with other occurrences of ID across tasks. b) In Fig. 5 and Fig.9 we provide the visualization results.
Response to Reviewer ymMg: Clarification on AED (1/5)
1. AED is not an unconstrained prefix tuning method. a) AED is a shared dictionary of 2,048 learnable entries, and pretrained FAST+ action-token indices retrieve these entries. Thus, occurrences assigned the same index across tasks are tied to the same learned vector: this imposes a specific cross-task parameter-sharing structure. The retrieved action embeddings are then conditioned on historical visual features and compressed into eight prefix tokens; they are not eight independent, freely learned prefix vectors. The dictionary contains approximately 2.10M parameters, but its defining property is the indexing and sharing constraint, not parameter count alone. MT is a separate, explicitly defined objective, not a relabeling of the standard WAM flow-matching loss. In Eq. (7), the model uses action-expert hidden states conditioned on the AED representations to predict visual-feature changes over randomly sampled temporal intervals. This directly couples reusable action representations to their visual consequences and provides action-conditioned transition supervision. It is jointly optimized with the standard flow-matching objective as the total loss in Eq. (10).