- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
UniAction proposes a unified conditioning interface for heterogeneous actions in video world models. It represents actions as sparse sets of elements, each defined by a discrete semantic identity and a spatiotemporal coordinate . Semantic identities index a learned codebook, while coordinates are mapped into the pretrained video DiT's native 3D RoPE space. Action tokens interact with video tokens through shared self-attention, with an action-specific LoRA applied only to action tokens. It further introduces cardinality-calibrated attention to account for differences in action token counts. Experiments on four action types (force, camera control, hand pose, and robot pose) demonstrate better action following than VACE and cross-attention conditioning baselines. A jointly trained model also outperforms action-specific experts across all four types. Ablations support the importance of pretrained positional encoding and attention calibration.
Critical strengths
-
Simple and effective design. Reusing pretrained positional geometry enables accurate action conditioning with fewer trainable parameters and lower inference overhead. The ablation studies provide convincing support.
-
Promising joint-training results. A single model outperforms action-specific experts across all four action types, suggesting benefits from heterogeneous action data.
Critical weaknesses
-
Limited novelty and missing direct baseline. Shared-attention conditioning with aligned positional encodings has been explored in OminiControl [1] and FullDiT [2]. The paper lacks a dense in-context conditioning baseline with shared RoPE, making it unclear whether improvements come from the proposed sparse representation or established conditioning mechanisms.
-
Unification is limited to the conditioning interface. Each action type requires manually designed semantic labels and spatial mappings, with no learned alignment across action types. The method unifies how actions are injected, rather than their representations.
-
Insufficiently controlled comparisons. UniAction directly leverages pretrained 3D RoPE for explicit grounding, whereas cross-attention must learn the corresponding action-video spatial alignment from limited data. Its poor performance (Table 1) therefore does not establish an inherent advantage of the proposed conditioning mechanism. Moreover, although data exposure and total compute are matched, the jointly trained model updates its shared parameters approximately 4× more often than each expert, making it unclear whether the gains reflect cross-action transfer or additional optimization.
[1] Tan, Zhenxiong, et al. "Ominicontrol: Minimal and universal control for diffusion transformer." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.
[2] Ju, Xuan, et al. "Fulldit: Multi-task video generative foundation model with full attention." arXiv preprint arXiv:2503.19907 (2025).
Questions for authors
-
Novelty and stronger baselines. How does UniAction differ technically from OminiControl and FullDiT? Could the authors compare against these in-context conditioning approaches, particularly a dense conditioning baseline using shared RoPE, to isolate the contribution of sparse action elements?
-
Cross-attention baseline fairness. Could the cross-attention baseline be provided with equivalent explicit spatiotemporal grounding? Does the performance gap persist with more training data or optimization steps?
-
Scope of unification. Since each action type requires manually designed semantic identities and spatial mappings, the method appears to unify the conditioning interface rather than the action space itself. Could the authors clarify this distinction and the scope of their unification claims?
-
Joint training (Table 2). Do the gains over action-specific experts persist when the experts receive the same number of optimizer updates as the jointly trained model? This would help distinguish cross-action transfer from additional optimization.
4 comments
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns.
I agree that the proposed conditioning scheme is effective, particularly under limited action-labeled data. However, much of its advantage seems to come from explicitly grounding action tokens in the pretrained spatialtemporal representation, making action conditioning easier to learn from limited data.
My main concern remains the broader claim of unified action learning. While the additional joint-training experiments are encouraging, the gains over the longer-trained experts are relatively modest, and it remains unclear whether they reflect transferable physical dynamics or simply the benefits of multi-task training.
Overall, I appreciate the additional results, but I still feel that the broader claims about unified action learning and cross-action transfer are not fully supported. I therefore maintain my rating of 2.
Thank you for the follow-up, and for agreeing that the conditioning scheme is effective under limited action-labeled data.
1. Grounding in the pretrained spatiotemporal representation. The dense in-context baseline you asked for tests this (Table R2). Its condition tokens are also grounded in the pretrained spatiotemporal representation, through the backbone's shared RoPE, and it trails UniAction on every form: 0.4849 / 0.4960 / 0.2483 / 0.4733 vs 0.7630 / 0.6371 / 0.4119 / 0.6521. Grounding alone is therefore not sufficient. The precise, sparse representation and the semantic identities account for this gap, which is the "contribution of sparse action elements" that your Q1 asked us to isolate. In addition, our advantage also comes from cardinality calibration, which addresses the bias from differing action representation granularities (Table 3), and from joint training on heterogeneous action data (Tables 2 and R1).
2. Unified action learning. UniAction represents every action as a set of elements {(s_i, p_i)} (Eq. 2), which makes the tasks of the four actions structurally identical: they are four input forms of one task, p(V | I_0, T, A) (Eq. 1), and can even be composed within one video (Table R4). Joint training in UniAction is therefore training on a single task with multiple forms of input. Table 2 shows the additional advantage of training the actions jointly, and the experts trained four times as much (Table R1) further show that, within this one task, training on the other three actions helps more than three times more training on a single action's own data. We therefore attribute these gains to transferable physical dynamics: what the model learns from one action about how the world responds to actions carries over to the others (Sec. 4.5).
We hope this clarifies both points.
To verify transferable physical dynamics directly, we take transfer from hand actions to robot actions as an example, evaluated on the robot test set of Table 1. We start either from the 4× hand expert (Table R1), which has never seen a robot action, or from scratch, and evaluate zero-shot and after a brief training on robot actions:
| Table R6: Robot action following ↑ | Zero-shot | Robot training, 2% |
|---|---|---|
| From the hand expert | 0.4977 | 0.6030 |
| From scratch | 0.0768 | 0.0978 |
The 2% is relative to the training of the 4× robot expert (Table R1), which reaches 0.6335.
Without seeing a single robot action, the hand expert already follows robot actions, with 0.4977 against 0.0768 from scratch. On top of this, with only 2% of the robot expert's training, it reaches 0.6030, 95% of the robot expert, while the model trained from scratch on the same data stays at 0.0978. These results show that what the model learns from one action about how the world responds to actions transfers well to other actions (Sec. 4.5). The gains of unified training can therefore be attributed to transferable physical dynamics.
We sincerely thank the reviewer for the detailed and constructive review.
W1 / Q1. Comparison with Generic Conditioning Mechanisms
Thank you for pointing us to these two works. OminiControl and FullDiT condition on dense, grid-aligned images or videos; as with VACE, every action must then be rendered first, although not every action has a visual form. UniAction conditions on a sparse set of action elements, each a semantic identity at a spatiotemporal coordinate (Eq. 2): actions with and without a visual form enter one model in the same way, at a cost that scales with the action: a dense condition adds as many tokens as the video itself, 32,760 for an 81-frame 480×832 clip, while UniAction adds 2 for a force, about 80 for a camera command and up to about 1,000 for a hand or robot pose.
As you suggested, we added the dense in-context baseline with shared RoPE, following OminiControl and FullDiT, with UniAction's backbone, data and number of updates:
It trails UniAction on every form, which isolates the contribution of sparse action elements. Supplementary C now reports this baseline, and Sec. 2 now cites both works.
W2 / Q3. Semantic Representation and Extensibility
We would like to clarify that UniAction unifies the representation itself: whatever its raw form, every action can be expressed as a set of elements {(s_i, p_i)} (Eq. 2). Its semantic identities and spatial mapping follow naturally from the action's own structure (e.g., the joints of a hand, the tail and head of a force vector). Every semantic identity indexes a token in one shared codebook space, which learns through training how the action corresponds to changes in the world. This unified learning therefore aligns the action types and lets them benefit one another (Table 2; see W3 / Q4 below).
W3 / Q2. Explicit Spatiotemporal Grounding
For giving Cross-Attn the pretrained spatiotemporal grounding (3D RoPE), we ablated this in Table 3's Self-attn → Cross-attn variant: action tokens keep 3D RoPE and only the injection changes to cross-attention, and hand and robot pose drop from 0.4266 / 0.7004 to 0.0533 / 0.0733. This shows that the key to UniAction successfully reusing the pretrained spatiotemporal modeling capabilities is its spatiotemporal grounding shared with the video (Sec. 3.2).
Following your suggestion, we further trained the Cross-Attn baseline to 1.5× and 2× its original number of updates:
This shows that the Cross-Attn baseline struggles to learn effective action control from limited data.
W3 / Q4. Benefits of Unified Training
Table 2 compares UniAction with an MoE hard-routed by action form: each expert is as large as the whole UniAction, the MoE as a whole uses UniAction's training samples and total compute, and the four experts are isolated from one another. Table 2 thus shows the gain of moving from isolated training to unified training.
We then trained each expert four times as much, all on its own action form, so the MoE uses four times UniAction's total compute. From the view of a single action form, each expert's training on that form's data is four times UniAction's, while UniAction spends three quarters of its training on the other three forms. Even so, UniAction still leads:
This further strengthens the conclusion that unified training benefits UniAction. What UniAction learns from the other three heterogeneous action forms is worth, for each form, at least quadrupling the training on that form's own data. On robot pose, four times the training lifts the expert only from 0.6310 to 0.6335, while UniAction reaches 0.6521. Repeated training on a form's own data saturates, while replacing this repeated training with data of the other three heterogeneous actions still improves that form, which reflects the cross-action transfer of unified training.
Supplementary C now includes this table.
We hope our responses have addressed your concerns.