acceptodds
Under review as a conference paper at ICLR 2027

UniAction: Unified Action Conditioning for World Models

Abstract

Actions that condition world models are highly heterogeneous, and thus existing action-conditioned world models are typically built for individual action types, fragmenting already limited action-annotated data across separate models. We introduce UniAction, a unified action-conditioning framework that represents any action as a sparse set of action elements, each with a discrete semantic identity and a continuous spatiotemporal grounding coordinate. By mapping these coordinates directly into the pretrained video model's native positional space, UniAction allows heterogeneous actions to condition a shared world model within the same spatiotemporal structure. We further introduce cardinality-calibrated attention to address the bias from differing action representation granularities. We evaluate UniAction on four representative action forms: force, camera control, hand pose, and robot pose. UniAction outperforms conditioning alternatives in action following, and a single jointly trained model surpasses the action-specific experts, showing that heterogeneous action data benefit from unified model and joint training.

Primary area:
generative models
Keywords:
world models, action conditioning, video generation, unified modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1CfCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UniAction proposes a unified conditioning interface for heterogeneous actions in video world models. It represents actions as sparse sets of elements, each defined by a discrete semantic identity and a spatiotemporal coordinate (t,y,x)(t,y,x). Semantic identities index a learned codebook, while coordinates are mapped into the pretrained video DiT's native 3D RoPE space. Action tokens interact with video tokens through shared self-attention, with an action-specific LoRA applied only to action tokens. It further introduces cardinality-calibrated attention to account for differences in action token counts. Experiments on four action types (force, camera control, hand pose, and robot pose) demonstrate better action following than VACE and cross-attention conditioning baselines. A jointly trained model also outperforms action-specific experts across all four types. Ablations support the importance of pretrained positional encoding and attention calibration.

Critical strengths
  • Simple and effective design. Reusing pretrained positional geometry enables accurate action conditioning with fewer trainable parameters and lower inference overhead. The ablation studies provide convincing support.

  • Promising joint-training results. A single model outperforms action-specific experts across all four action types, suggesting benefits from heterogeneous action data.

Critical weaknesses
  • Limited novelty and missing direct baseline. Shared-attention conditioning with aligned positional encodings has been explored in OminiControl [1] and FullDiT [2]. The paper lacks a dense in-context conditioning baseline with shared RoPE, making it unclear whether improvements come from the proposed sparse representation or established conditioning mechanisms.

  • Unification is limited to the conditioning interface. Each action type requires manually designed semantic labels and spatial mappings, with no learned alignment across action types. The method unifies how actions are injected, rather than their representations.

  • Insufficiently controlled comparisons. UniAction directly leverages pretrained 3D RoPE for explicit t,x,yt,x,y grounding, whereas cross-attention must learn the corresponding action-video spatial alignment from limited data. Its poor performance (Table 1) therefore does not establish an inherent advantage of the proposed conditioning mechanism. Moreover, although data exposure and total compute are matched, the jointly trained model updates its shared parameters approximately 4× more often than each expert, making it unclear whether the gains reflect cross-action transfer or additional optimization.


[1] Tan, Zhenxiong, et al. "Ominicontrol: Minimal and universal control for diffusion transformer." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[2] Ju, Xuan, et al. "Fulldit: Multi-task video generative foundation model with full attention." arXiv preprint arXiv:2503.19907 (2025).

Questions for authors
  1. Novelty and stronger baselines. How does UniAction differ technically from OminiControl and FullDiT? Could the authors compare against these in-context conditioning approaches, particularly a dense conditioning baseline using shared RoPE, to isolate the contribution of sparse action elements?

  2. Cross-attention baseline fairness. Could the cross-attention baseline be provided with equivalent explicit spatiotemporal grounding? Does the performance gap persist with more training data or optimization steps?

  3. Scope of unification. Since each action type requires manually designed semantic identities and spatial mappings, the method appears to unify the conditioning interface rather than the action space itself. Could the authors clarify this distinction and the scope of their unification claims?

  4. Joint training (Table 2). Do the gains over action-specific experts persist when the experts receive the same number of optimizer updates as the jointly trained model? This would help distinguish cross-action transfer from additional optimization.

4 comments

We sincerely thank the reviewer for the detailed and constructive review.

W1 / Q1. Comparison with Generic Conditioning Mechanisms

Thank you for pointing us to these two works. OminiControl and FullDiT condition on dense, grid-aligned images or videos; as with VACE, every action must then be rendered first, although not every action has a visual form. UniAction conditions on a sparse set of action elements, each a semantic identity at a spatiotemporal coordinate (Eq. 2): actions with and without a visual form enter one model in the same way, at a cost that scales with the action: a dense condition adds as many tokens as the video itself, 32,760 for an 81-frame 480×832 clip, while UniAction adds 2 for a force, about 80 for a camera command and up to about 1,000 for a hand or robot pose.

As you suggested, we added the dense in-context baseline with shared RoPE, following OminiControl and FullDiT, with UniAction's backbone, data and number of updates:

Table R2: Action following ↑ForceCameraHandRobot
VACE (Table 1)0.60460.49970.29380.5245
Dense in-context0.48490.49600.24830.4733
UniAction (Table 1)0.76300.63710.41190.6521

It trails UniAction on every form, which isolates the contribution of sparse action elements. Supplementary C now reports this baseline, and Sec. 2 now cites both works.

W2 / Q3. Semantic Representation and Extensibility

We would like to clarify that UniAction unifies the representation itself: whatever its raw form, every action can be expressed as a set of elements {(s_i, p_i)} (Eq. 2). Its semantic identities and spatial mapping follow naturally from the action's own structure (e.g., the joints of a hand, the tail and head of a force vector). Every semantic identity indexes a token in one shared codebook space, which learns through training how the action corresponds to changes in the world. This unified learning therefore aligns the action types and lets them benefit one another (Table 2; see W3 / Q4 below).

W3 / Q2. Explicit Spatiotemporal Grounding

For giving Cross-Attn the pretrained spatiotemporal grounding (3D RoPE), we ablated this in Table 3's Self-attn → Cross-attn variant: action tokens keep 3D RoPE and only the injection changes to cross-attention, and hand and robot pose drop from 0.4266 / 0.7004 to 0.0533 / 0.0733. This shows that the key to UniAction successfully reusing the pretrained spatiotemporal modeling capabilities is its spatiotemporal grounding shared with the video (Sec. 3.2).

Following your suggestion, we further trained the Cross-Attn baseline to 1.5× and 2× its original number of updates:

Table R3: Action following ↑ForceCameraHandRobot
Cross-Attn (Table 1)0.16180.53060.06780.0859
Cross-Attn, 1.5× updates0.22580.50780.07120.0724
Cross-Attn, 2× updates0.22000.50480.06970.0883
UniAction (Table 1)0.76300.63710.41190.6521

This shows that the Cross-Attn baseline struggles to learn effective action control from limited data.

W3 / Q4. Benefits of Unified Training

Table 2 compares UniAction with an MoE hard-routed by action form: each expert is as large as the whole UniAction, the MoE as a whole uses UniAction's training samples and total compute, and the four experts are isolated from one another. Table 2 thus shows the gain of moving from isolated training to unified training.

We then trained each expert four times as much, all on its own action form, so the MoE uses four times UniAction's total compute. From the view of a single action form, each expert's training on that form's data is four times UniAction's, while UniAction spends three quarters of its training on the other three forms. Even so, UniAction still leads:

Table R1: Action following ↑Total computeForceCameraHandRobot
Experts (Table 2)1×0.54360.51820.37010.6310
Experts4×0.75470.62050.40140.6335
UniAction1×0.76300.63710.41190.6521

This further strengthens the conclusion that unified training benefits UniAction. What UniAction learns from the other three heterogeneous action forms is worth, for each form, at least quadrupling the training on that form's own data. On robot pose, four times the training lifts the expert only from 0.6310 to 0.6335, while UniAction reaches 0.6521. Repeated training on a form's own data saturates, while replacing this repeated training with data of the other three heterogeneous actions still improves that form, which reflects the cross-action transfer of unified training.

Supplementary C now includes this table.

We hope our responses have addressed your concerns.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns.

I agree that the proposed conditioning scheme is effective, particularly under limited action-labeled data. However, much of its advantage seems to come from explicitly grounding action tokens in the pretrained spatialtemporal representation, making action conditioning easier to learn from limited data.

My main concern remains the broader claim of unified action learning. While the additional joint-training experiments are encouraging, the gains over the longer-trained experts are relatively modest, and it remains unclear whether they reflect transferable physical dynamics or simply the benefits of multi-task training.

Overall, I appreciate the additional results, but I still feel that the broader claims about unified action learning and cross-action transfer are not fully supported. I therefore maintain my rating of 2.

Thank you for the follow-up, and for agreeing that the conditioning scheme is effective under limited action-labeled data.

1. Grounding in the pretrained spatiotemporal representation. The dense in-context baseline you asked for tests this (Table R2). Its condition tokens are also grounded in the pretrained spatiotemporal representation, through the backbone's shared RoPE, and it trails UniAction on every form: 0.4849 / 0.4960 / 0.2483 / 0.4733 vs 0.7630 / 0.6371 / 0.4119 / 0.6521. Grounding alone is therefore not sufficient. The precise, sparse representation and the semantic identities account for this gap, which is the "contribution of sparse action elements" that your Q1 asked us to isolate. In addition, our advantage also comes from cardinality calibration, which addresses the bias from differing action representation granularities (Table 3), and from joint training on heterogeneous action data (Tables 2 and R1).

2. Unified action learning. UniAction represents every action as a set of elements {(s_i, p_i)} (Eq. 2), which makes the tasks of the four actions structurally identical: they are four input forms of one task, p(V | I_0, T, A) (Eq. 1), and can even be composed within one video (Table R4). Joint training in UniAction is therefore training on a single task with multiple forms of input. Table 2 shows the additional advantage of training the actions jointly, and the experts trained four times as much (Table R1) further show that, within this one task, training on the other three actions helps more than three times more training on a single action's own data. We therefore attribute these gains to transferable physical dynamics: what the model learns from one action about how the world responds to actions carries over to the others (Sec. 4.5).

We hope this clarifies both points.

To verify transferable physical dynamics directly, we take transfer from hand actions to robot actions as an example, evaluated on the robot test set of Table 1. We start either from the 4× hand expert (Table R1), which has never seen a robot action, or from scratch, and evaluate zero-shot and after a brief training on robot actions:

Table R6: Robot action following ↑Zero-shotRobot training, 2%
From the hand expert0.49770.6030
From scratch0.07680.0978

The 2% is relative to the training of the 4× robot expert (Table R1), which reaches 0.6335.

Without seeing a single robot action, the hand expert already follows robot actions, with 0.4977 against 0.0768 from scratch. On top of this, with only 2% of the robot expert's training, it reaches 0.6030, 95% of the robot expert, while the model trained from scratch on the same data stays at 0.0978. These results show that what the model learns from one action about how the world responds to actions transfers well to other actions (Sec. 4.5). The gains of unified training can therefore be attributed to transferable physical dynamics.

Reviewer_paF1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UniAction, a unified action-conditioning framework for video world models that accommodates heterogeneous action types within a shared generative architecture. The paper represents actions as sparse sets of elements, each characterized by a discrete semantic identity and a continuous spatiotemporal coordinate. These elements are integrated into the pretrained video model's native positional encoding system, enabling interactions between action and video tokens through shared self-attention. Additionally, cardinality-calibrated attention is introduced to mitigate conditioning bias caused by different numbers of action tokens. Experiments on force, camera control, hand pose, and robot pose demonstrate improvements over visual-based and cross-attention-based conditioning methods.

Critical strengths
  1. This paper proposes a solution to a significant limitation of existing "action-conditioned world models," which typically require the design of specialized architectures for different types of actions. By representing heterogeneous actions through semantic identities and spatiotemporal grounding, the approach provides a concise and flexible interface for conditioning. Integrating action coordinates into the existing positional encodings of a pre-trained backbone network allows the model to reuse established spatiotemporal representations without the need for additional, action-specific encoders.

  2. The proposed cardinality-calibrated attention mechanism is designed to address the issue of imbalanced attention distribution that arises from variations in the number of action tokens. This approach is concise in its formulation and requires no additional learnable parameters. Furthermore, compared to other evaluated methods, this mechanism achieves superior action-following performance while requiring fewer trainable parameters and incurring lower inference overhead. These characteristics make it highly practical for use in large-scale video world models.

  3. These experiments encompass four distinct forms of action representation: Force, Camera Control, Hand Pose, and Robot Pose. Evaluation methods include automated action-following metrics, perceptual similarity, and human assessment. Ablation studies further confirm the importance of shared positional encoding and base-value calibration.

Critical weaknesses
  1. Although the paper proposes UniAction as a general framework for heterogeneous actions, its quantitative experiments are limited to four predefined action types. Each action type still requires manual design to map it to semantic identifiers and spatiotemporal groundings. Consequently, the effectiveness of the framework in handling action representations that are significantly different or previously unseen remains unclear. Furthermore, the experiments on compositional action generalization presented in Appendix H are primarily qualitative and lack quantitative evaluation.

  2. Although Table 2 demonstrates improvements over action-specific experts, the gains are relatively modest for hand pose and robot pose (Hand: 0.3701->0.4119 and 0.0934->0.0910; Robot: 0.6310->0.6521 and 0.1286->0.1164).

  3. The proposed cardinality-calibrated attention is motivated by differences in the number of action tokens. However, Table 3 shows substantial improvements from calibration mainly for force and camera control, whereas the results for hand pose and robot pose remain nearly unchanged or slightly decrease.

Questions for authors
  1. Could the authors provide quantitative evidence demonstrating the model's performance in executing action instructions when faced with combinations of action types it has not encountered before?

  2. Table 2 shows that joint training enhances the model's ability to execute actions compared to expert models trained for specific actions. However, the improvements in hand pose and robot pose estimation are relatively limited. Could the authors clarify whether these performance gains primarily stem from the transfer of knowledge regarding physical dynamics across different action types, or from the increased diversity of the training data?

1 comment

We sincerely thank the reviewer for the detailed and constructive review.

W1. Heterogeneous and New Actions

We chose four representative heterogeneous actions of very different forms to instantiate UniAction (Sec. 4.2): an instantaneous physical vector at one point (force, 2 elements), a global command (camera control), and dense per-frame trajectories of a human hand and a robot arm (hand and robot pose, up to about 1,000 elements), each from its own dataset. All of them are expressed in one representation, a set of elements {(s_i, p_i)}, each a semantic identity at a spatiotemporal coordinate (Eq. 2). UniAction can thus unify the representation of heterogeneous actions.

As with the four existing heterogeneous actions, a new form of action needs only new identities (s_i) in the shared codebook and grounding coordinates (p_i), either given by the action or assigned. We also evaluate composing the four actions, a combination unseen in training; please see our response to Q1 below for the quantitative composition results.

W2 / Q2. Benefits of Unified Training

Table 2 compares UniAction with an MoE hard-routed by action form: each expert is as large as the whole UniAction, the MoE as a whole uses UniAction's training samples and total compute, and the four experts are isolated from one another. Table 2 thus shows the gain of moving from isolated training to unified training.

We then trained each expert four times as much, all on its own action form, so the MoE uses four times UniAction's total compute. From the view of a single action form, each expert's training on that form's data is four times UniAction's, while UniAction spends three quarters of its training on the other three forms. Even so, UniAction still leads:

Table R1: Action following ↑Total computeForceCameraHandRobot
Experts (Table 2)1×0.54360.51820.37010.6310
Experts4×0.75470.62050.40140.6335
UniAction1×0.76300.63710.41190.6521

On hand and robot pose, the experts rise only from 0.3701 / 0.6310 to 0.4014 / 0.6335, still below UniAction's 0.4119 / 0.6521. Repeated training on a form's own data saturates, while replacing this repeated training with data of the other three heterogeneous actions still improves that form. The gains therefore stem from both: the added diversity is the data of the other three heterogeneous actions, and what these data bring is knowledge of physical dynamics transferred across action types.

W3. Cardinality Calibration

We would like to clarify that this is exactly what calibration should do: it places different actions on a common attention scale, so that a force with 2 tokens controls generation as effectively as a hand pose with up to about 1,000. Its offset log(N_ref/N), with N_ref = 784, is about 6 for a force's 2 tokens and near 0 for the pose forms, whose token counts are at the scale of N_ref. It therefore lifts the sparse forms (force 0.5186 → 0.7190, camera 0.4966 → 0.5842) and leaves the dense ones unchanged (hand 0.4303 → 0.4266, robot 0.6995 → 0.7004; Table 3).

Q1. Compositional Action Generalization

Thank you for this suggestion. We composed the four forms in 16 test videos, each containing all four forms, a combination unseen in training since each training sample contains one action form, and scored each form with its own action-following metric:

Table R4: Action following under composition ↑ForceCameraHandRobot
VACE0.2400.3930.2660.651
Cross-Attn0.0580.2100.2220.149
UniAction0.7570.7110.3960.877

Under composition, UniAction leads on every form: each action element carries its own semantic identity and its own coordinate in the video's space-time, so actions given together still act where and when they are meant to. Supplementary H now reports this table.

We hope our responses have addressed your concerns.

Reviewer_sLfLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UniAction proposed to create a shared conditioning interface for video world models, and turn each action into a sparse set of elements. Action tokens are combined with video tokens in self-attention and are trained with an action-token only LoRA. Experiments with visual-based (VACE) and token-based (Cross-Attn) conditioning mechanisms demonstrate that UniAction performs better in four action forms. The authors also propose cardinality-calibrated attention, and show that one jointly trained model outperforms separate experts for each action form.

Critical strengths
  • The core idea is quite straightforward and makes sense: instead of rendering actions into videos or learning an encoder for each action, the action tokens are placed directly in the backbone's pretrained RoPE space. It is also the cheapest of the compared methods.
  • The comparison with VACE and Cross-Attn is fair (same backbone, data and training budget), and UniAction gets the best action following on all four action forms, in both the automatic and the human evaluation.
  • The ablations in Table 3 are convincing: replacing RoPE or moving the action tokens to cross-attention almost breaks hand and robot pose control.
Critical weaknesses
  • The biggest weakness and the main question for this paper is the joint training result (Table 2). Each expert is only updated on its own 10K clips, while the shared LoRA of the joint model gets 4× more updates, so the gain may simply come from longer training. Also, the paper attributes the gain to "shared learning of world dynamics", but the video branch is frozen and the LoRA only touches action tokens, so I feel it is not clear what dynamics are shared.
  • The evaluation is not fully convincing. There are no seeds or confidence intervals, and the test sets are small (e.g., 13 scenes for camera control). The camera score maps a cosine similarity to [0, 1] without giving the mapping. If it is (1 + cos)/2, a video that ignores the command would already score about 0.5, close to VACE and Cross-Attn. On hand pose, the human ratings of UniAction and VACE are almost the same, so the claim of better visual quality should be toned down.
  • Each action form still needs a hand-designed mapping (e.g., fixed frame locations for camera commands), and it is not clear how this works for actions without a location in the frame.
Questions for authors
  • Does the joint model still beat the experts if each expert gets the same number of updates? I would consider raising my score if it does.
  • What are the exact action following formulas, and what does a video generated without any action score?
  • How sensitive are the results to N_ref, and does the location of the camera tokens matter?
2 comments

We sincerely thank the reviewer for the detailed and constructive review.

W1 / Q1. Benefits of Unified Training

Table 2 compares UniAction with an MoE hard-routed by action form: each expert is as large as the whole UniAction, the MoE as a whole uses UniAction's training samples and total compute, and the four experts are isolated from one another. Table 2 thus shows the gain of moving from isolated training to unified training.

We then trained each expert four times as much, all on its own action form, so the MoE uses four times UniAction's total compute. From the view of a single action form, each expert's training on that form's data is four times UniAction's, while UniAction spends three quarters of its training on the other three forms. Even so, UniAction still leads:

Table R1: Action following ↑Total computeForceCameraHandRobot
Experts (Table 2)1×0.54360.51820.37010.6310
Experts4×0.75470.62050.40140.6335
UniAction1×0.76300.63710.41190.6521

This further strengthens the conclusion that unified training benefits UniAction. What UniAction learns from the other three heterogeneous action forms is worth, for each form, at least quadrupling the training on that form's own data. On robot pose, four times the training lifts the expert only from 0.6310 to 0.6335, while UniAction reaches 0.6521. Repeated training on a form's own data saturates, while replacing this repeated training with data of the other three heterogeneous actions still improves that form, which reflects the cross-action transfer of unified training.

W1. Shared Learning of World Dynamics

Thank you for raising this point. What is shared is "how the world responds to actions" (Sec. 4.5). Precisely because the video branch is frozen, everything learned in action training, the correspondence between actions and changes in the world, lies in the action pathway, and this pathway (one codebook and one action-branch LoRA) is shared by all four actions. Each action therefore also learns how the world responds to actions from the data of the other three heterogeneous actions, which is where the gain of unified training comes from (Table R1).

W2. Evaluation

Seeds, intervals and test size. As suggested, we report three sampling seeds and 95% intervals. Each seed tests every method on 410 videos (camera control: 13 scenes × 8 commands = 104), so three seeds give 1,230 videos per method.

Table R5: Action following ↑ (mean ± sd, 3 seeds)ForceCameraHandRobot
VACE0.5930 ± 0.01150.4913 ± 0.00850.2815 ± 0.01110.5449 ± 0.0198
Cross-Attn0.1987 ± 0.03320.5260 ± 0.00650.0619 ± 0.00740.0795 ± 0.0055
UniAction0.7595 ± 0.02180.6097 ± 0.02540.4045 ± 0.00650.6519 ± 0.0014
UniAction − VACE, 95% interval[0.0725, 0.2429][0.0866, 0.1843][0.0992, 0.1371][0.0894, 0.1665]
UniAction − Cross-Attn, 95% interval[0.5327, 0.6694][0.0538, 0.1673][0.3090, 0.3811][0.5029, 0.6235]

UniAction leads on every form, and every 95% interval of its gain over VACE and Cross-Attn excludes zero.

Camera mapping. It is (1+c)/2(1+c)/2, now in Supplementary C. Because the eight commands form four opposite pairs, ignoring the command scores exactly 0.5 by construction: 0.5 is a built-in no-response floor, and UniAction's 0.6371 lies well above it.

Visual quality. In the human evaluation, UniAction receives the highest visual-quality rating on force (4.00 vs 3.40 for VACE), camera control (3.68 vs 3.55) and robot pose (3.99 vs 3.73), and is on par with VACE on hand pose (3.42 vs 3.43; Table 1). This is what Sec. 4.4 states: UniAction receives "comparable or better overall visual quality".

W3. Semantic Representation and Extensibility

UniAction expresses every action in one representation, a set of elements {(s_i, p_i)} (Eq. 2), whose semantic identities and spatial mapping follow naturally from the action's own structure (e.g., the joints of a hand, the tail and head of a force vector). Camera commands are actions without a location in the frame: they "have no intrinsic spatial grounding", so each direction is assigned a fixed canonical location, and commands sharing it "are distinguished by their semantic identities". Supplementary A now states: "Actions without an intrinsic spatial location can be incorporated in the same way as camera control (Sec. 4.2), with a fixed canonical location and their own semantic identities."

Q2. Evaluation Metrics

Supplementary C gives each formula, with C(x)=min⁡(1,max⁡(0,x))C(x)=\min(1,\max(0,x)):

  • Force: sF=C(cos⁡(f^,f))s_F=C(\cos(\hat f,f)), and 0 without a valid motion response.
  • Camera: sC=(1+c)/2s_C=(1+c)/2, where cc is the cosine between the commanded direction and the camera displacement relative to the no-action generation.
  • Hand / robot: per tracked point and frame, with commanded displacement Δc\Delta c (r=∥Δc∥r=\lVert\Delta c\rVert) and measured displacement Δp\Delta p, s=C((r−∥Δp−Δc∥)/(r−τ))s=C\big((r-\lVert\Delta p-\Delta c\rVert)/(r-\tau)\big) with tolerance τ\tau, averaged over frames, then points; thresholds and the stationary case are in Supplementary C.

Without any action, the same model falls far below UniAction: force 0.2178, camera 0.5000 (by construction), hand 0.0911 and robot 0.0801, vs 0.7630, 0.6371, 0.4119 and 0.6521.

Q3. Reference Cardinality and Spatiotemporal Grounding

N_ref. Calibration makes the attention an action receives independent of how many tokens express it: a force has 2 elements, a hand pose up to about 1,000. N_ref is the fixed reference cardinality shared by all actions, and training and inference use the same N_ref. For any value of N_ref, each action's total attention mass becomes N_ref · mean_i exp(ℓ_i), independent of its token count, allowing actions with sparse tokens "to control generation just as effectively". Table 3 tests exactly this balance: without calibration, force (sparse) drops by 0.20 and camera (sparse) by 0.09, while hand (dense) and robot (dense) are unchanged.

Camera-token location. Camera tokens sit at fixed canonical locations that we assign: for example, the keys W and I, A and J, S and K, and D and L each share one location, and are distinguished by their semantic identities (s_i).

We hope our responses have addressed your concerns.

Comments on the paper

1 comment

General Response

We sincerely thank all reviewers for their careful and constructive reviews, and for finding the design "simple and effective" (1CfC) and "concise and flexible" (paF1), the ablations "convincing" (1CfC, sLfL), and the comparison with VACE and Cross-Attn "fair (same backbone, data and training budget)" (sLfL).

1. Benefits of Unified Training (1CfC, paF1, sLfL)

Table 2 compares UniAction with an MoE hard-routed by action form: each expert is as large as the whole UniAction, the MoE as a whole uses UniAction's training samples and total compute, and the four experts are isolated from one another. Table 2 thus shows the gain of moving from isolated training to unified training.

We then trained each expert four times as much, all on its own action form, so the MoE uses four times UniAction's total compute. From the view of a single action form, each expert's training on that form's data is four times UniAction's, while UniAction spends three quarters of its training on the other three forms. Even so, UniAction still leads:

Table R1: Action following ↑Total computeForceCameraHandRobot
Experts (Table 2)1×0.54360.51820.37010.6310
Experts4×0.75470.62050.40140.6335
UniAction1×0.76300.63710.41190.6521

This further strengthens the conclusion that unified training benefits UniAction. What UniAction learns from the other three heterogeneous action forms is worth, for each form, at least quadrupling the training on that form's own data. On robot pose, four times the training lifts the expert only from 0.6310 to 0.6335, while UniAction reaches 0.6521. Repeated training on a form's own data saturates, while replacing this repeated training with data of the other three heterogeneous actions still improves that form, which reflects the cross-action transfer of unified training.

2. Semantic Representation and Extensibility (1CfC, paF1, sLfL)

UniAction unifies the representation itself: whatever its raw form, every action can be expressed as a set of elements {(s_i, p_i)} (Eq. 2). Its semantic identities and spatial mapping follow naturally from the action's own structure (e.g., the joints of a hand, the tail and head of a force vector). Every semantic identity indexes a token in one shared codebook space, which learns through training how the action corresponds to changes in the world. This unified learning therefore aligns the action types and lets them benefit one another (point 1 above).

As with the four existing heterogeneous actions, a new form of action needs only new identities (s_i) in the shared codebook and grounding coordinates (p_i), either given by the action or assigned; actions without a location in the frame, such as camera commands, take a fixed canonical location (Supplementary A).

Revisions (blue in the PDF)

  • Sec. 2: OminiControl and FullDiT.
  • Sec. 4.5: experts trained four times as much.
  • Supplementary A: actions without a spatial location.
  • Supplementary C: metric formulas; no-action reference; dense in-context baseline; Cross-Attn with more updates; experts with four times the compute.
  • Supplementary H: quantitative composition.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.