acceptodds
Under review as a conference paper at ICLR 2027

CompWorld: Compositional Action Control for Video World Models

Abstract

Interactive video world models increasingly serve as controllable simulators for domains such as gaming, camera-controlled navigation, and robot manipulation, where video generation models must support heterogeneous action types such as keyboard commands, mouse motion, camera poses, and robot-arm trajectories. However, most existing video world models build action-control modules for partic- ular action types, leaving different action types in isolated representations that are difficult to combine reliably without paired multi-action supervision. We propose CompWorld, a framework for compositional action control in video world models. CompWorld keeps the original action streams as precise inputs, while projecting them into a text-aligned unified action space that uses the pretrained text space of a text-conditioned video generation model as a shared interface prior. With action type-specific encoders, learned “not provided” placeholders, and Action-to-Text Fusion, CompWorld enables different action types to condition the video generator through Fused-Text Cross-Attention and compose at inference time. We evalu- ate CompWorld in compositional and separate action-control settings, including two compositional benchmarks—keyboard plus camera pose and robot-arm plus camera pose—that pair heterogeneous controls only at inference, without paired multi-action training annotations. Experiments across keyboard, mouse, camera- pose, and robot-arm controls show that CompWorld improves compositional action control, action following, and video quality over FrameInject, TokenConcat, and GameFactory, and yields a more semantically aligned conditioning interface.

Primary area:
generative models
Keywords:
video world model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wvWAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies compositional action control for video world models when the specific heterogeneous control types to be combined are not jointly annotated during training. CompWorld encodes raw actions with type-specific encoders, represents unavailable action streams with learned placeholders, and fuses action tokens into pretrained text features through Action-to-Text Fusion. The resulting features condition a video diffusion model through its inherited text cross-attention.

Critical strengths
  1. Combining heterogeneous action streams that were not jointly annotated during training addresses a practical limitation of interactive video models. The keyboard-plus-camera and arm-plus-camera settings make this question concrete and extend the evaluation beyond ordinary control on a single data source.

  2. Preserving numerical action inputs while using a pretrained text-conditioning interface is a plausible way to share information across control types. The study includes several injection baselines, GameFactory, public-backbone experiments, and relevant ablations.

Critical weaknesses
  1. The central claim of simultaneous control is not sufficiently separated from camera following. In the arm-plus-camera benchmark, Table 2 reports camera-trajectory errors and FID/FVD, but no independent assessment of adherence to the commanded arm trajectory under camera motion. The RT-1 task text also describes the intended manipulation, so a plausible manipulation combined with good camera following does not by itself establish that the arm-action stream is being followed. Figure 4 and the supplementary video provide qualitative support, but no systematic test holds the text and camera fixed while varying the arm command. Table 1's paired fidelity provides additional evidence for keyboard-plus-camera composition, although it likewise does not independently isolate adherence to each stream. This gap matters because successful joint conditioning is the paper's principal contribution. A controlled action-change analysis could address it without requiring an executable robot environment.

  2. The camera-following evidence needs clearer temporal alignment and registration accounting. In the released Wan evaluator, run_glomap returns poses without retaining the original frame identifiers, and compute_cam_distance truncates the predicted and reference pose arrays to their common length before comparing them. If reconstruction registers different frame subsets, corresponding array entries need not refer to the same video time. Separately, Table 2 averages over clips registered for every method, but does not report the retained count or per-method registration rates. These details affect the interpretation of a metric central to the conclusions.

Questions for authors
  1. Independent evidence of joint action adherence. Can the authors provide a controlled comparison that holds the initial frame, text, camera trajectory, and sampling seed fixed while changing the arm command, together with evidence that the resulting arm/gripper motion changes appropriately? An existing targeted analysis or an independently assessed set of examples would be useful. How is adherence to both streams checked in the keyboard-plus-camera setting? These requests concern the current joint-control claim, rather than a new task-success benchmark.

  2. Camera evaluation alignment. Does the evaluation used for the reported results match reconstructed poses by their original frame identifiers and use a consistent temporal reference when registration is incomplete?

  3. Registration coverage. How many of the 300 arm-plus-camera clips remain in the common-success set in Table 2, and what are the registration rates for each method? Please clarify how failures are treated and provide uncertainty estimates for the camera-error differences on the matched evaluation set, if available.

3 comments
AuthorsOpenReview ↗

Response to Reviewer wvWA (1/3): arm adherence and interventions (W1, Q1)

We thank the reviewer for suggesting a concrete test of joint control without requiring an executable robot environment, and for checking camera registration and temporal alignment in our released code. Below we respond point by point; all results have been added (in blue) to the revised PDF.

Q1a (Weakness 1 & Question 1 — Controlled arm intervention): "Can the authors provide a controlled comparison that holds the initial frame, text, camera trajectory, and sampling seed fixed while changing the arm command, together with evidence that the resulting arm/gripper motion changes appropriately?"

A1a: On 30 held-out RT-1 scenes, we hold the initial frame, RT-1 task text, moving camera trajectory, and sampling seed fixed while changing only the 7-D arm command: (i) scaling its per-step displacement deltas by 0.5×/1.0×/1.5×0.5\times / 1.0\times / 1.5\times, and (ii) mirror-flipping its lateral (left/right) component. After removing background camera motion, we track the 2D gripper trajectory with CoTracker [1] and report its displacement relative to the unedited (1.0×1.0\times) run and the fraction of scenes where net lateral motion under the mirrored command matches the commanded left/right sign (Lateral flip acc.; 50% chance baseline):

MethodDisplacement ratio (0.5x / 1.0x / 1.5x)Lateral flip acc. ↑
FrameInject0.89 / 1.00 / 1.0550.0% (15/30)
TokenConcat0.85 / 1.00 / 1.0953.3% (16/30)
GameFactory0.82 / 1.00 / 1.1256.7% (17/30)
CompWorld w/o Text-Align0.86 / 1.00 / 1.0853.3% (16/30)
CompWorld0.68 / 1.00 / 1.2870.0% (21/30)

FrameInject (0.89 / 1.00 / 1.05, 50.0%), TokenConcat (53.3%), GameFactory (56.7%), and w/o Text-Align (53.3%) stay near 1.00 and the 50% chance level, whereas CompWorld scales monotonically (0.68 / 1.00 / 1.28) and matches the mirrored direction in 70.0% of scenes (p=0.021p = 0.021, one-sided binomial test vs. 50%). Because the fixed initial scene, workspace bounds, and 3D-to-2D perspective projection pull extreme 2D ratios toward 1.00 for all methods, we compare the modulation range across methods (0.60 for CompWorld vs. 0.16–0.30 for baselines and 0.22 for w/o Text-Align; Table 2(b), Sec. 4.2).

Q1b (Weakness 1 — Independent arm adherence on Table 2): "In the arm-plus-camera benchmark, Table 2 reports camera-trajectory errors and FID/FVD, but no independent assessment of adherence to the commanded arm trajectory under camera motion. The RT-1 task text also describes the intended manipulation..."

A1b: Across all 300 clips of Table 2 (50 scenes ×\times 6 camera trajectories), we initialize CoTracker [1] from the shared first frame, remove background camera motion, and report the mean 2D gripper trajectory L2 error against the recorded RT-1 episode normalized by the 320x256 image diagonal (EE-TrajErr, Table 2(a)). To separate the arm command from the instruction text, Text-only keeps the first frame, instruction, and camera command unchanged and replaces only the arm command with its "not provided" placeholder:

MethodEE-TrajErr (%) ↓RotErr ↓TransErr ↓Cam ↓
FrameInject25.41.40310.01610.469
TokenConcat24.61.33611.90412.351
GameFactory24.01.22512.93613.249
CompWorld w/o Text-Align24.41.63513.51714.023
CompWorld, Text-only (no arm command)25.91.1829.74010.095
CompWorld19.61.1509.6019.940

EE-TrajErr is averaged over all 300 clips; the camera columns use Table 2's 245-clip matched set (see A3). Removing the arm command (Text-only) raises CompWorld's EE-TrajErr from 19.6% to 25.9% while leaving camera following nearly unchanged (Cam 10.095 vs. 9.940). Under the composed commands, all baselines remain close to this no-arm level (24.0–25.4%), while CompWorld has the lowest error on both streams (19.6% and 9.940).

AuthorsOpenReview ↗

Response to Reviewer wvWA (2/3): keyboard+camera and pose alignment (Q1, Q2)

Q1c (Weakness 1 & Question 1 — Stream-wise adherence in keyboard+camera): "Table 1's paired fidelity provides additional evidence for keyboard-plus-camera composition, although it likewise does not independently isolate adherence to each stream... How is adherence to both streams checked in the keyboard-plus-camera setting?"

A1c: In Table 1 (100 held-out GF-Minecraft clips), the camera input is converted from mouse pitch/yaw and contains rotation only (zero translation), whereas player locomotion is commanded by the keyboard WASD stream (coupled with the current view heading). Splitting Table 1's Cam into rotation error (RotErr) and translation error (TransErr) against the GLOMAP [2] reconstruction of the ground-truth video therefore isolates adherence to the camera stream in RotErr and, together with frame-paired PSNR and LPIPS against the recorded joint video, reflects keyboard-driven locomotion in TransErr (added to Appendix G, Table 9):

MethodRotErr (camera stream) ↓TransErr (keyboard-driven) ↓Cam (Table 1) ↓PSNR ↑LPIPS ↓
FrameInject15.1418.3528.0715.540.6307
TokenConcat14.8918.2627.9115.720.6237
GameFactory16.1820.7231.0815.960.6132
CompWorld w/o Fix14.6218.1227.6315.640.6290
CompWorld w/o Placeholder15.3818.9628.8516.220.6105
CompWorld w/o Text-Align15.7619.5429.6015.960.6169
CompWorld14.5517.4826.9916.620.6074

CompWorld has the lowest error on both RotErr (14.55) and TransErr (17.48) among all baselines and ablations, consistent with its best paired fidelity (PSNR 16.62, LPIPS 0.6074). Within-keyboard composition is further tested on held-out key combinations in Appendix E (Table 7), where CompWorld also leads on every metric.

Q2 (Weakness 2 & Question 2 — Camera evaluation alignment): "In the released Wan evaluator, run_glomap returns poses without retaining the original frame identifiers, and compute_cam_distance truncates the predicted and reference pose arrays... Does the evaluation used for the reported results match reconstructed poses by their original frame identifiers and use a consistent temporal reference when registration is incomplete?"

A2: Thank you for checking the released code. In short: Table 2 is unaffected because it evaluates only on clips with complete 20/20 registration for every method; in the reference-video script (evaluate_reference_camera_pose.py), incomplete registrations were paired by sorted array position and anchored at each video's first registered frame (mode="left") rather than matched by frame ID, which affects <8%<8\% of clips:

  • Table 2 (commanded trajectory, evaluate_video_camera_pose): It averages over the 245 clips with full 20/20 keyframe registration across all five methods (see A3). When all 20 keyframes are registered (estimated_poses_len == 20), filename-sorted poses match the 20 sampled keyframes [0,4,8,…,76][0, 4, 8, \dots, 76] one-to-one, frame 0000.png is the shared reference in relative_pose(..., mode="left"), and the commanded trajectory is sampled at the same indices, so incomplete registration never enters Table 2.
  • Reference-video tables (Tables 1, 3, 4, 6, and 7, evaluate_reference_camera_pose.py): get_frames_from_video extracts 20 keyframes at a 4-frame stride per clip. For >92%>92\% of evaluated clips, both generated and reference videos register all 20 keyframes (20/20), so sorted array positions already match by keyframe index and share frame 0 as the reference. Evaluating on this full 20/20 subset changes Cam by at most 0.08 in Tables 1, 3, 4, and 6 and at most 0.02 in Table 7, with all method rankings unchanged.
AuthorsOpenReview ↗

Response to Reviewer wvWA (3/3): registration and uncertainty (W2, Q3)

Q3 (Weakness 2 & Question 3 — Registration coverage and uncertainty): "How many of the 300 arm-plus-camera clips remain in the common-success set in Table 2, and what are the registration rates for each method? Please clarify how failures are treated and provide uncertainty estimates for the camera-error differences on the matched evaluation set, if available."

A3: For each of the 300 arm+camera clips per method in Table 2, GLOMAP [2] registers all 20 sampled frames (Full 20/20), a subset (Partial 2–19), or fails (<2 poses). Below are the per-method registration counts alongside the 95% paired bootstrap confidence intervals for the camera-error differences (Δ=CompWorld−Baseline\Delta = \text{CompWorld} - \text{Baseline}) on the 245-clip common evaluation set (added to Appendix G, Table 10):

MethodFull (20/20) ↑Partial (2–19)Failed (<2)ΔRotErr [95% CI]ΔTransErr [95% CI]ΔCam [95% CI]
FrameInject92.0% (276)6.7% (20)1.3% (4)-0.253 [-0.52, +0.02]-0.415 [-1.17, +0.34]-0.529 [-1.30, +0.24]
TokenConcat94.0% (282)4.3% (13)1.7% (5)-0.186 [-0.48, +0.11]-2.303 [-3.03, -1.58]-2.411 [-3.17, -1.65]
GameFactory92.7% (278)3.7% (11)3.7% (11)-0.075 [-0.33, +0.18]-3.335 [-4.08, -2.59]-3.309 [-4.06, -2.56]
CompWorld w/o Text-Align95.3% (286)2.3% (7)2.3% (7)-0.485 [-0.77, -0.20]-3.916 [-4.69, -3.14]-4.083 [-4.87, -3.30]
CompWorld96.0% (288)2.3% (7)1.7% (5)---

Out of 300 clips per method, 245 clips (81.7%) achieve Full 20/20 registration across all five Table 2 methods simultaneously; Table 2's camera metrics (RotErr, TransErr, Cam) average over this common set, while FID and FVD use all 300 clips per method. Both failed (<2 poses, 1.3%–3.7%) and partially registered (2–19 poses, 2.3%–6.7%) clips are excluded for all methods so that every method is scored on the same 245 complete 20-keyframe trajectories. CompWorld has the highest full-registration rate (96.0%, 288/300) and the fewest non-full registrations (12/300, vs. 14–24/300 for the other methods).

On the 245 matched clips, CompWorld's reductions in TransErr and Cam are statistically significant (95% CIs below zero, in bold) against TokenConcat, GameFactory, and w/o Text-Align, and its RotErr reduction is significant against w/o Text-Align. Against FrameInject, the mean differences also favor CompWorld but the CIs include zero; FrameInject reaches this camera accuracy while barely following the arm command (EE-TrajErr 25.4% vs. 25.9% without any arm command; lateral flip accuracy 50.0%, Part 1/3), matching the one-stream-dominates failure in Fig. 2(b), whereas CompWorld follows both streams simultaneously (19.6% EE-TrajErr, 70.0% flip accuracy, and 9.940 Cam). We have noted this in Sec. 4.2 and added the registration and alignment tables to Appendix G.

Together, the fixed-seed arm interventions and 300-clip end-effector tracking (A1a, A1b), the stream-wise keyboard+camera breakdown (A1c), and the alignment and registration accounting with paired CIs (A2, A3) address both of the reviewer's main concerns; we are happy to discuss any further questions.

References

[1] Karaev et al., CoTracker: It is Better to Track Together, ECCV 2024.

[2] Pan et al., Global Structure-from-Motion Revisited (GLOMAP), ECCV 2024.

Reviewer_1AK5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CompWorld, a framework for compositional action control in video world models. It addresses the setting where heterogeneous controls are annotated separately during training but must operate jointly at inference. CompWorld encodes precise action inputs into text-dimensional tokens, uses learned placeholders for unavailable action types, and fuses action information into pretrained text representations. The resulting features condition video generation through frozen text–video cross-attention, using the pretrained text space as a shared interface for composition.

The main contributions are this shared conditioning architecture and benchmarks testing keyboard–camera and robot-arm–camera composition without paired annotations for those combinations during training. Validation includes comparisons with FrameInject, TokenConcat, and GameFactory, separate evaluations on P2P, SpatialVID, and RT-1, and experiments with an internal 1B backbone and Wan2.1-1.3B. Results generally show improved control adherence and visual quality, although FVD improvements are not uniform. Ablations examine text-space alignment, interface freezing, and missing-action placeholders; additional experiments assess unseen keyboard combinations and semantic alignment through text-anchor retrieval and counterfactual probes.

Critical strengths
  1. The paper studies a useful setting: learning different action controls from separate datasets and combining them at inference without paired annotations. Using the pretrained text space as a shared interface is a sensible approach, and the method retains the original action inputs for precise control. This could reduce the data collection needed to support new combinations of controls.

  2. The evaluation covers two compositional settings, several baselines, and two backbones. I find the text-alignment ablation particularly useful: removing the pretrained text interface increases arm–camera trajectory error from 9.940 to 14.023, with relatively little change in visual quality. This provides evidence that the text interface helps the model follow controls, rather than simply generate better-looking videos.

Critical weaknesses
  1. The paper does not directly test whether the shared text interface preserves fine-grained robot action information. Although the experiments use RT-1 data, video quality and camera trajectory metrics do not establish whether the generated arm motion follows the commanded trajectory or gripper state. Evaluating end-effector trajectory error, gripper-state consistency, or responses to small changes in the action input would help address this gap. Without such evidence, it remains unclear whether the interface supports precise robot control beyond producing visually plausible motion.
Questions for authors

Aligning action representations through a shared text space to facilitate transfer is interesting. However, can text interface preserve the fine-grained information needed for precise robot-arm control? Could you extend discussion with latent action model (LAM).

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1AK5 (1/2): arm control and trajectory error (Q1, W1)

We thank the reviewer for the constructive comments and for highlighting our w/o Text-Align ablation (Cam 9.940 vs. 14.023 with similar FID/FVD) as evidence that the pretrained text interface specifically helps control adherence. Below we address each point directly; all new results and discussions have been added (in blue) to the revised PDF.

Q1 (Question 1, part 1): "Aligning action representations through a shared text space to facilitate transfer is interesting. However, can text interface preserve the fine-grained information needed for precise robot-arm control?"

A1: Routing actions through the pretrained text space does not quantize continuous actions into discrete words. Raw 7-D RT-1 actions (3-D end-effector translation deltas, 3-D rotation deltas, and the gripper command) are mapped by a type-specific MLP with temporal position embeddings directly to continuous dd-dimensional action tokens Uarm,τ∈Rw×d\mathbf{U}_{\text{arm},\tau} \in \mathbb{R}^{w \times d} at each latent timestep τ\tau. Action-to-Text Fusion then adds a continuous residual to the text tokens, Yτfuse=Y+CrossAttnA2T(Y,Uτ,Uτ)\mathbf{Y}^{\text{fuse}}_\tau = \mathbf{Y} + \mathrm{CrossAttn}_{\text{A2T}}(\mathbf{Y}, \mathbf{U}_\tau, \mathbf{U}_\tau), which is a continuous linear combination of action value vectors WVUτW_V \mathbf{U}_\tau updated at every latent timestep τ∈{0,…,n}\tau \in \{0,\dots,n\}. The pretrained text space serves as the shared coordinate system expected by the frozen cross-attention layers, while the per-timestep residual carries continuous action magnitudes and directions. While fine-grained, high-frequency control remains hard for pretrained text-to-video models (Sec. 5), the empirical question specific to our interface is whether routing through the pretrained text space loses arm-control information compared with bypassing it (w/o Text-Align) or injecting actions directly under the same backbone and training data. We test this directly in A2 and A3.

Q2 (Weakness 1 — End-effector trajectory error): "The paper does not directly test whether the shared text interface preserves fine-grained robot action information... Evaluating end-effector trajectory error... would help address this gap."

A2: To measure arm adherence directly under simultaneous camera motion across all 300 clips of Table 2 (50 held-out RT-1 scenes ×\times 6 camera trajectories), we track the end-effector (gripper) in each generated clip and in the recorded RT-1 episode with CoTracker [1], stabilize the background to remove the commanded camera motion, and compute the mean 2D trajectory L2 distance normalized by the 320x256 image diagonal (EE-TrajErr). Because RT-1 instructions describe the intended manipulation, we also include a Text-only control for CompWorld, in which the arm command is replaced by its learned "not provided" placeholder while keeping the first frame, instruction, and camera command identical:

MethodEE-TrajErr (%) ↓RotErr ↓TransErr ↓Cam ↓
FrameInject25.41.40310.01610.469
TokenConcat24.61.33611.90412.351
GameFactory24.01.22512.93613.249
CompWorld w/o Text-Align24.41.63513.51714.023
CompWorld, Text-only (no arm command)25.91.1829.74010.095
CompWorld19.61.1509.6019.940

EE-TrajErr is averaged over all 300 clips; the camera columns are evaluated on Table 2's 245-clip matched set.

Removing the arm command (Text-only) raises CompWorld's trajectory error from 19.6% to 25.9%, confirming that the arm motion is driven by the arm command rather than by the instruction text alone. Under the same composed commands, FrameInject (25.4%), TokenConcat (24.6%), GameFactory (24.0%), and w/o Text-Align (24.4%) remain close to the 25.9% obtained without any arm command, whereas CompWorld achieves the lowest error on both the arm stream (19.6%) and the camera stream (Cam 9.940). Removing the arm command also leaves CompWorld's camera error nearly unchanged (10.095 vs. 9.940), showing that composing the arm stream does not degrade camera following.

AuthorsOpenReview ↗

Response to Reviewer 1AK5 (2/2): arm perturbations and LAMs (W1, Q1)

Q3 (Weakness 1 — Responses to action perturbations): "Evaluating... gripper-state consistency, or responses to small changes in the action input would help address this gap. Without such evidence, it remains unclear whether the interface supports precise robot control..."

A3: On 30 held-out RT-1 scenes, we hold the first frame, task instruction, moving camera trajectory, and sampling seed fixed, and perturb only the 7-D arm command: (i) we scale its per-step displacement deltas by 0.5×/1.0×/1.5×0.5\times / 1.0\times / 1.5\times; (ii) we mirror its lateral (left/right) velocity component. After background stabilization, we report 2D end-effector displacement relative to the unedited (1.0×1.0\times) trajectory and the fraction of scenes whose net lateral displacement under the mirrored command matches the commanded left/right sign (Lateral flip acc.; 50% chance baseline):

MethodDisplacement ratio (0.5x / 1.0x / 1.5x)Lateral flip acc. ↑
FrameInject0.89 / 1.00 / 1.0550.0% (15/30)
TokenConcat0.85 / 1.00 / 1.0953.3% (16/30)
GameFactory0.82 / 1.00 / 1.1256.7% (17/30)
CompWorld w/o Text-Align0.86 / 1.00 / 1.0853.3% (16/30)
CompWorld0.68 / 1.00 / 1.2870.0% (21/30)

The baselines barely respond to the arm edits: their displacement ratios stay close to 1.00 (modulation range 1.5×1.5\times minus 0.5×0.5\times of only 0.16–0.30) and their lateral sign accuracy under the mirrored command stays near chance (50.0–56.7%). CompWorld scales its 2D arm displacement monotonically with the commanded amplitude (0.68 / 1.00 / 1.28, modulation range 0.60) and matches the mirrored lateral direction in 70.0% of scenes (p=0.021p = 0.021, one-sided binomial test against 50%). Because the initial frame and task prompt remain fixed to the original scene, while tabletop contact, workspace bounds, and 3D-to-2D perspective projection pull extreme 2D displacement ratios toward 1.00 for every method, the key comparison is across conditioning interfaces under identical data and training: bypassing the pretrained text pathway in CompWorld w/o Text-Align increases EE-TrajErr from 19.6% to 24.4%, narrows the amplitude modulation range by 2.7×2.7\times (from 0.60 down to 0.22, i.e., 0.86–1.08), and drops lateral flip accuracy from 70.0% to 53.3%. Thus, under the same backbone and training data, routing through the pretrained text interface preserves substantially more arm-control information than a parameter-matched unaligned latent (w/o Text-Align) or direct injection. Both tables have been added to Sec. 4.2 (Table 2(a)–(b)).

Q4 (Question 1, part 2): "Could you extend discussion with latent action model (LAM)."

A4: We have expanded the third paragraph of Sec. 2 (Related Work) to discuss latent action models (LAMs) such as Genie [2], AdaWorld [3], and LAPA [4] along three points:

First, regarding supervision and grounding, LAMs discover latent actions from unlabeled video via inverse and forward dynamics reconstruction. This removes the need for action labels during pretraining, but the learned latent space is not grounded in explicit physical commands: executing a specific 6-DoF camera trajectory or 7-D robot-arm command still requires learning a supervised command-to-latent mapping from labeled data. Second, regarding compositional control, inverse-dynamics LAMs infer a single latent transition from frame-to-frame pixel changes, so when the camera and robot arm move simultaneously, global viewpoint motion and local arm motion are entangled in one latent rather than factored into independently controllable streams. Even when explicit action streams map into a learned shared space without text alignment (w/o Text-Align), control adherence drops substantially on both benchmarks (Cam 9.940→14.0239.940 \to 14.023 and EE-TrajErr 19.6%→24.4%19.6\% \to 24.4\% on arm+camera; Cam 26.99→29.6026.99 \to 29.60 on keyboard+camera) compared with anchoring the interface to the pretrained text space. Third, the two paradigms are complementary: a pretrained LAM latent can be added to CompWorld as an additional action slot with its own placeholder, allowing unlabeled video priors to combine with explicitly labeled action streams.

Together, the 300-clip end-effector trajectory error (A2), the controlled amplitude and directional perturbations against w/o Text-Align and injection baselines (A3), and the LAM discussion (A4) address the reviewer's questions on fine-grained robot-arm control; we are happy to discuss any further questions.

References

[1] Karaev et al., CoTracker: It is Better to Track Together, ECCV 2024.

[2] Bruce et al., Genie: Generative Interactive Environments, ICML 2024.

[3] Gao et al., AdaWorld: Learning Adaptable World Models with Latent Actions, ICML 2025.

[4] Ye et al., Latent Action Pretraining from Videos, ICLR 2025.

Thanks for detailed response. Most of my concerns are solved and I lean to acceptance.

Thank you for your constructive feedback and positive assessment. We appreciate your time and thoughtful review.

Reviewer_5zDjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper considers action-controlled video generation models. In particular, it introduces CompWorld, which allows various action streams to control video generation by projecting them in a unified text-anchored action space. This allows different action types to be composed at inference, enabling a rich control signal, even if in training video generation is conditioned on one action type at a time. The unified action space is aligned with the text encoding space of a pre-trained text-to-video model. Experiments across several different control types (including keyboard controls, mouse, camera pose, and robot arm controls) show the improvements brought by the proposed approach.

Critical strengths
  • The presented model is able to compose different control signals together at inference time, even if these were not seen in conjunction during training.

  • Experiments are conducted across different datasets and different types of control signal

  • The approach is build on pre-trained text-to-video model, which seems a good idea because of the abundant data for that task. Finetuning the model for action-controlled video generation therefore requires less data by leveraging the pertained model.

  • Experiments include comparison to prior work and a number of ablations of the proposed approach.

Critical weaknesses
  • The training data used in the experiments is not described in sufficient detail. I did not see a clear list with how many videos with what spatiotemporal resolution from which source were used to finetune the pre-trained Wan model.

  • The main text misses detail on how FVD / FID metrics are computed, critically: from how many samples. The reported values are extremely high, and require explanation.

Questions for authors
  • In Section 4.3 I was not sure to understand the difference between the experiments on the internal backbone and the Wan2.1 backbone. In Section 4.1 it seemed that the internal backbone was based on Wan2.1 ? What data and data volume was the internal backbone (pre)trained on?

  • Related to the previous point, it would be interesting to analyze the performance difference between using the internal and Wan2.1 backbone by comparing the results across tables 3 and 4.

  • It would be interesting to add a baseline besides TokenConcat and FrameInject, based on GameFactory: one could channel-concat the control embedding with the text embedding Y. One can still keep the cross-attention parameters from the pretrained model fixed, but simply expand the Q,K,V projection matrices to leverage the action embeddings. This is similar to the current attention fusion approach, but simplified to a “sum” fusion on the Q,K,V projections. This baseline would reveal the importance of the proposed attention-fusion mechanism vs the overall approach of jointly training from different action modalities and enabling inference time composition.

  • It would be useful to know the number of parameters in the trained models: encoder/decoder/backbone/ additional action-related parameters.

  • The eval protocol for the Robot-arm + camera pose composition experiment (last paragraph of page 7), was not completely clear. What data was trained on? How is eval ground truth established?

  • I would suggest to reduce the qualitative figures 4 and 5, and optimize the layout of fig 6 in order to expand section 4.5 to be more self-contained. Currently it cannot be appreciated without referring to the supplemental.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 5zDj (1/3): training data and FID/FVD (W1, W2)

We thank the reviewer for the positive assessment and concrete suggestions. Below we respond to each point; all additions have been incorporated (in blue) into the revised manuscript.

W1: "The training data used in the experiments is not described in sufficient detail... how many videos with what spatiotemporal resolution from which source were used to finetune the pre-trained Wan model."

AW1: We have added the dataset summary table below to Appendix C (Table C.1). No training clip carries both action types of a composed pair: in keyboard+camera (Table 1), keyboard and camera annotations never appear on the same training clip; in arm+camera (Table 2), arm control is learned only from RT-1 and camera control only from SpatialVID-HQ.

SourceAction type(s)Train clipsEvaluated test clips (gen. / ref.)ResolutionFrames (Int. 1B / Wan)Used in
GF-Minecraft, keyboard onlyKeyboard19,696-640x35277 / -Tab. 1 (train)
GF-Minecraft, keyboard+mouseKeyboard + camera pose (from mouse)0 (eval only)100 / 100640x35277 / -Tab. 1 (test)
Context-as-MemoryCamera pose9,900 (x2)*-640x35277 / -Tab. 1 (train)
P2P (8 games)Keyboard + mouse12,69150 / 50640x35277 / 81Tabs. 2, 3, 4, 6
SpatialVID-HQCamera pose13,59850 / 50640x35277 / 81Tabs. 2, 3, 4, 6
RT-1Robot arm10,22050 / 50; 300 / 50 (Tab. 2)320x25677 / 81Tabs. 2, 3, 4, 6

*Repeated 2×2\times in the Table 1 training manifest to balance the 19,696 GF-Minecraft keyboard clips. For P2P, SpatialVID-HQ, and RT-1 (36,509 train clips in total, used to fine-tune both the internal 1B model and Wan2.1-1.3B), source videos are preprocessed into 81-frame clips at native frame stride, from which Internal 1B samples 77 contiguous frames (20 latent steps) and Wan2.1-1.3B uses all 81 frames (21 latent steps). Evaluated test clips are fixed subsets sampled from the held-out data (from the 5% test split in Sec. 4.3 for P2P, SpatialVID-HQ, and RT-1). The within-keyboard held-out study in Appendix E uses GF-Minecraft splits with 14,566 and 5,262 training clips and 100 / 100 test clips.

W2: "The main text misses detail on how FVD / FID metrics are computed, critically: from how many samples. The reported values are extremely high, and require explanation."

AW2: FID [1] is computed frame-wise with pytorch-fid (InceptionV3 pool3, 2048-D) over all frames, and FVD [2] is computed video-wise with the official Google Research I3D implementation (Kinetics-400, 400-D) on 16 uniformly sampled frames per clip at 224x224. For sample sizes, Table 1 compares 100 generated vs. 100 reference clips; Tables 3 and 4 use 50 vs. 50 per subset (150 vs. 150 in the merged Table 6); and Table 2 compares 300 generated clips against the 50 recorded RT-1 clips from the same held-out scenes.

Three factors inflate the absolute values. First, empirical Fréchet distances have a strong upward bias at small sample sizes (N=50N = 50–100100) [3]: for FVD, each clip yields one 400-D feature vector, so the empirical covariance has rank at most N−1≪400N - 1 \ll 400; for FID, frames within a clip are strongly correlated, so the effective sample size is far below the total frame count. Even with real videos only, splitting the 50 real RT-1 test clips into two disjoint 25-clip subsets gives a real-vs-real FID of 90.2 and FVD of 832.0, showing that at this scale absolute values are strongly affected by sample size and subset variance, not only by generation quality (similarly, when the three N=50N = 50 subsets in Table 3 are pooled into the N=150N = 150 merged set in Table 6, CompWorld's FVD goes from 935.85 / 576.28 / 481.05 to 363.00). Because this offset depends on the dataset and NN, absolute values should only be compared within a table. Second, Minecraft gameplay and tabletop robot arms differ substantially from the ImageNet and Kinetics domains used to train the feature extractors. Third, in Table 2, 250 of the 300 generated clips follow moving camera trajectories while all 50 reference RT-1 videos have a static camera, raising FVD for every method. Within each table, all methods use the exact same clips, references, and protocol. We have added the protocol and sample sizes to Sec. 4.1 and the scale analysis to Appendix C.

AuthorsOpenReview ↗

Response to Reviewer 5zDj (2/3): backbones and Sum-Fusion baseline (Q1-Q3)

Q1: "In Section 4.3 I was not sure to understand the difference between the experiments on the internal backbone and the Wan2.1 backbone... What data and data volume was the internal backbone (pre)trained on?"

A1: We clarify that the internal backbone is not based on Wan2.1. It is a separate in-house ~1B latent text-to-video DiT (Sec. 3.1: causal 3D VAE with 4×4\times temporal compression, plus DiT blocks with 2D/3D attention, text cross-attention, and FFN), pretrained on a proprietary general-domain text-video corpus without action labels; because this pretraining corpus and its exact scale are not publicly released, we additionally evaluate all methods and release code on the open-weight Wan2.1-1.3B checkpoint [4] (Tables 4 and 6). All other tables use the internal backbone. We have separated the two descriptions in Sec. 4.1.

Q2: "Related to the previous point, it would be interesting to analyze the performance difference between using the internal and Wan2.1 backbone by comparing the results across tables 3 and 4."

A2: Because the two setups differ in backbone architecture, pretraining corpus, text encoder, rollout length (81 vs. 77 frames), and fine-tuning hyperparameters (Sec. 4.1, Appendix B), absolute scores shift between Tables 3 and 4, while comparing Tables 3, 4, and 6 shows three main patterns:

  1. On the merged benchmark (Table 6), all methods have lower absolute PSNR and higher LPIPS, FID, and FVD on the 81-frame Wan2.1 setup than on the 77-frame Internal 1B setup (e.g., CompWorld PSNR 13.47 vs. 16.09, FVD 485.32 vs. 363.00), with lower PSNR on all per-subset cells as well (Tables 3 vs. 4).
  2. CompWorld achieves the best score on all five merged metrics on both backbones (Table 6) and on every per-subset metric on Internal 1B (Table 3). On Wan2.1-1.3B (Table 4), it leads 10 of 14 cells, with its largest margins on P2P (PSNR 11.98 vs. 11.25, LPIPS 0.5356 vs. 0.5530, FVD 1063.38 vs. 1223.60 against the best baseline) and a consistent though narrower Cam lead (merged 13.37 vs. 13.95, vs. 9.69 vs. 11.24 on Internal 1B). The four cells it does not lead are all won by GameFactory: SpatialVID LPIPS (0.5670 vs. 0.5713) and RT-1 PSNR, LPIPS, and FVD (17.13 vs. 16.62, 0.2950 vs. 0.2987, 706.70 vs. 710.57); CompWorld retains the best RT-1 FID (49.90 vs. 51.82).
  3. Freezing the pretrained text interface helps on both backbones: w/o Fix is worse than CompWorld on every per-subset metric in both Table 3 and Table 4.

We have added this cross-backbone analysis to Sec. 4.3.

Q3: "It would be interesting to add a baseline besides TokenConcat and FrameInject, based on GameFactory: one could channel-concat the control embedding with the text embedding Y... simplified to a 'sum' fusion on the Q,K,V projections..."

A3: Thank you for suggesting this baseline [5]. Under our Sec. 4.4 ablation setup (Table 1), we channel-concatenate the encoded action representation with Y\mathbf{Y} at each latent timestep, keeping the pretrained cross-attention weights frozen and expanding the projection matrices with zero-initialized weights for the action channels (an additive fusion on the projections). Action encoders, placeholders, the frozen interface, disjoint training data, and schedules match CompWorld, isolating the fusion operator:

MethodCam ↓PSNR ↑LPIPS ↓FID ↓FVD ↓
FrameInject28.0715.540.630791.48843.31
TokenConcat27.9115.720.623789.66730.16
GameFactory31.0815.960.613287.75686.56
CompWorld w/o Text-Align29.6015.960.616998.35838.36
Sum-Fusion (new)27.9216.110.611893.25867.82
CompWorld26.9916.620.607484.74781.96

The comparison separates the two factors. First, holding all else fixed, our attention-based Action-to-Text Fusion improves over Sum-Fusion on every metric (Cam 26.99 vs. 27.92, PSNR 16.62 vs. 16.11, LPIPS 0.6074 vs. 0.6118, FID 84.74 vs. 93.25, FVD 781.96 vs. 867.82), because content-dependent cross-attention lets each text token selectively attend to relevant action slots and window tokens at each timestep rather than applying a uniform offset. Second, additive fusion alone already improves paired fidelity over all baselines and w/o Text-Align (PSNR 16.11, LPIPS 0.6118), while remaining on par with the best baseline on Cam (27.92 vs. 27.91 for TokenConcat) and trailing on FID/FVD; the attention-based fusion is what brings the further gains in control adherence and distributional quality. We have added Sum-Fusion to Table 1 and Sec. 4.4.

AuthorsOpenReview ↗

Response to Reviewer 5zDj (3/3): parameters, arm+camera, and layout (Q4-Q6)

Q4: "It would be useful to know the number of parameters in the trained models: encoder/decoder/backbone/additional action-related parameters."

A4: Parameter counts on the public Wan2.1-1.3B backbone (added to Appendix B, Table B.1) are:

ModuleParametersIn CompWorld
3D causal VAE (encoder + decoder)126.9Mfrozen
Text encoder (umT5-XXL)5,680.9Mfrozen
DiT backbone1,419.0M1,135.6M trained; 283.4M text cross-attention frozen
Additional action-related parameters (action encoders, placeholders, Action-to-Text Fusion)16.26M (1.15% of the DiT)trained

On the internal 1B backbone, the added action modules similarly account for 0.84% of the DiT parameters. CompWorld w/o Text-Align uses a parameter-matched cross-attention module, so its parameter count is identical to CompWorld's.

Q5: "The eval protocol for the Robot-arm + camera pose composition experiment (last paragraph of page 7), was not completely clear. What data was trained on? How is eval ground truth established?"

A5: The experiment uses the same model jointly trained on P2P (keyboard+mouse), SpatialVID-HQ (camera pose), and RT-1 (robot arm) as evaluated in Table 3. No training clip contains both arm and camera annotations: arm control is learned only from RT-1 (with the camera slot filled by its learned "not provided" placeholder) and camera control only from SpatialVID-HQ (with the arm slot filled by its placeholder).

At evaluation, 50 held-out RT-1 scenes ×\times 6 commanded camera trajectories (5 moving trajectories: orbit left, orbit right, dolly in, dolly out, truck left; and 1 static trajectory, shared across all methods) produce 300 clips per method at 320x256. Each clip is conditioned on the initial frame, the RT-1 instruction as the only prompt, the recorded 7-D arm actions, and the commanded camera trajectory. Because no real video records an RT-1 manipulation episode under a moving camera, each stream is evaluated against its own reference: camera adherence (RotErr, TransErr, Cam) compares the GLOMAP-reconstructed trajectory [6] of the generated video against the commanded camera trajectory using scale-normalized relative poses (Sec. 4.1), and arm adherence (2D end-effector trajectory error, Table 2(a)) compares the background-stabilized gripper trajectory against the recorded RT-1 episode. FID and FVD use the 50 recorded RT-1 clips as references. We have clarified this protocol in Sec. 4.2.

Q6: "I would suggest to reduce the qualitative figures 4 and 5, and optimize the layout of fig 6 in order to expand section 4.5 to be more self-contained..."

A6: Thank you for the layout suggestion. In the revised PDF, we have compacted Figs. 4–6 and moved the anchor construction and metric definitions from Appendix F into Sec. 4.5 so that the section is self-contained without consulting the appendix.

References

[1] Heusel et al., GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (FID), NeurIPS 2017.

[2] Unterthiner et al., Towards Accurate Generative Models of Video: A New Metric & Challenges (FVD), arXiv 2018.

[3] Chong & Forsyth, Effectively Unbiased FID and Inception Score and Where to Find Them, CVPR 2020.

[4] Wang et al., Wan: Open and Advanced Large-Scale Video Generative Models, arXiv 2025.

[5] Yu et al., GameFactory: Creating New Games with Generative Interactive Videos, ICCV 2025.

[6] Pan et al., Global Structure-from-Motion Revisited (GLOMAP), ECCV 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.