- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies compositional action control for video world models when the specific heterogeneous control types to be combined are not jointly annotated during training. CompWorld encodes raw actions with type-specific encoders, represents unavailable action streams with learned placeholders, and fuses action tokens into pretrained text features through Action-to-Text Fusion. The resulting features condition a video diffusion model through its inherited text cross-attention.
Critical strengths
-
Combining heterogeneous action streams that were not jointly annotated during training addresses a practical limitation of interactive video models. The keyboard-plus-camera and arm-plus-camera settings make this question concrete and extend the evaluation beyond ordinary control on a single data source.
-
Preserving numerical action inputs while using a pretrained text-conditioning interface is a plausible way to share information across control types. The study includes several injection baselines, GameFactory, public-backbone experiments, and relevant ablations.
Critical weaknesses
-
The central claim of simultaneous control is not sufficiently separated from camera following. In the arm-plus-camera benchmark, Table 2 reports camera-trajectory errors and FID/FVD, but no independent assessment of adherence to the commanded arm trajectory under camera motion. The RT-1 task text also describes the intended manipulation, so a plausible manipulation combined with good camera following does not by itself establish that the arm-action stream is being followed. Figure 4 and the supplementary video provide qualitative support, but no systematic test holds the text and camera fixed while varying the arm command. Table 1's paired fidelity provides additional evidence for keyboard-plus-camera composition, although it likewise does not independently isolate adherence to each stream. This gap matters because successful joint conditioning is the paper's principal contribution. A controlled action-change analysis could address it without requiring an executable robot environment.
-
The camera-following evidence needs clearer temporal alignment and registration accounting. In the released Wan evaluator,
run_glomapreturns poses without retaining the original frame identifiers, andcompute_cam_distancetruncates the predicted and reference pose arrays to their common length before comparing them. If reconstruction registers different frame subsets, corresponding array entries need not refer to the same video time. Separately, Table 2 averages over clips registered for every method, but does not report the retained count or per-method registration rates. These details affect the interpretation of a metric central to the conclusions.
Questions for authors
-
Independent evidence of joint action adherence. Can the authors provide a controlled comparison that holds the initial frame, text, camera trajectory, and sampling seed fixed while changing the arm command, together with evidence that the resulting arm/gripper motion changes appropriately? An existing targeted analysis or an independently assessed set of examples would be useful. How is adherence to both streams checked in the keyboard-plus-camera setting? These requests concern the current joint-control claim, rather than a new task-success benchmark.
-
Camera evaluation alignment. Does the evaluation used for the reported results match reconstructed poses by their original frame identifiers and use a consistent temporal reference when registration is incomplete?
-
Registration coverage. How many of the 300 arm-plus-camera clips remain in the common-success set in Table 2, and what are the registration rates for each method? Please clarify how failures are treated and provide uncertainty estimates for the camera-error differences on the matched evaluation set, if available.
3 comments
Response to Reviewer wvWA (2/3): keyboard+camera and pose alignment (Q1, Q2)
Q1c (Weakness 1 & Question 1 — Stream-wise adherence in keyboard+camera): "Table 1's paired fidelity provides additional evidence for keyboard-plus-camera composition, although it likewise does not independently isolate adherence to each stream... How is adherence to both streams checked in the keyboard-plus-camera setting?"
A1c: In Table 1 (100 held-out GF-Minecraft clips), the camera input is converted from mouse pitch/yaw and contains rotation only (zero translation), whereas player locomotion is commanded by the keyboard WASD stream (coupled with the current view heading). Splitting Table 1's Cam into rotation error (RotErr) and translation error (TransErr) against the GLOMAP [2] reconstruction of the ground-truth video therefore isolates adherence to the camera stream in RotErr and, together with frame-paired PSNR and LPIPS against the recorded joint video, reflects keyboard-driven locomotion in TransErr (added to Appendix G, Table 9):
| Method | RotErr (camera stream) ↓ | TransErr (keyboard-driven) ↓ | Cam (Table 1) ↓ | PSNR ↑ | LPIPS ↓ |
|---|---|---|---|---|---|
| FrameInject | 15.14 | 18.35 | 28.07 | 15.54 | 0.6307 |
| TokenConcat | 14.89 | 18.26 | 27.91 | 15.72 | 0.6237 |
| GameFactory | 16.18 | 20.72 | 31.08 | 15.96 | 0.6132 |
| CompWorld w/o Fix | 14.62 | 18.12 | 27.63 | 15.64 | 0.6290 |
| CompWorld w/o Placeholder | 15.38 | 18.96 | 28.85 | 16.22 | 0.6105 |
| CompWorld w/o Text-Align | 15.76 | 19.54 | 29.60 | 15.96 | 0.6169 |
| CompWorld | 14.55 | 17.48 | 26.99 | 16.62 | 0.6074 |
CompWorld has the lowest error on both RotErr (14.55) and TransErr (17.48) among all baselines and ablations, consistent with its best paired fidelity (PSNR 16.62, LPIPS 0.6074). Within-keyboard composition is further tested on held-out key combinations in Appendix E (Table 7), where CompWorld also leads on every metric.
Q2 (Weakness 2 & Question 2 — Camera evaluation alignment): "In the released Wan evaluator,
run_glomapreturns poses without retaining the original frame identifiers, andcompute_cam_distancetruncates the predicted and reference pose arrays... Does the evaluation used for the reported results match reconstructed poses by their original frame identifiers and use a consistent temporal reference when registration is incomplete?"
A2: Thank you for checking the released code. In short: Table 2 is unaffected because it evaluates only on clips with complete 20/20 registration for every method; in the reference-video script (evaluate_reference_camera_pose.py), incomplete registrations were paired by sorted array position and anchored at each video's first registered frame (mode="left") rather than matched by frame ID, which affects of clips:
- Table 2 (commanded trajectory,
evaluate_video_camera_pose): It averages over the 245 clips with full 20/20 keyframe registration across all five methods (see A3). When all 20 keyframes are registered (estimated_poses_len == 20), filename-sorted poses match the 20 sampled keyframes one-to-one, frame0000.pngis the shared reference inrelative_pose(..., mode="left"), and the commanded trajectory is sampled at the same indices, so incomplete registration never enters Table 2. - Reference-video tables (Tables 1, 3, 4, 6, and 7,
evaluate_reference_camera_pose.py):get_frames_from_videoextracts 20 keyframes at a 4-frame stride per clip. For of evaluated clips, both generated and reference videos register all 20 keyframes (20/20), so sorted array positions already match by keyframe index and share frame 0 as the reference. Evaluating on this full 20/20 subset changes Cam by at most 0.08 in Tables 1, 3, 4, and 6 and at most 0.02 in Table 7, with all method rankings unchanged.
Response to Reviewer wvWA (3/3): registration and uncertainty (W2, Q3)
Q3 (Weakness 2 & Question 3 — Registration coverage and uncertainty): "How many of the 300 arm-plus-camera clips remain in the common-success set in Table 2, and what are the registration rates for each method? Please clarify how failures are treated and provide uncertainty estimates for the camera-error differences on the matched evaluation set, if available."
A3: For each of the 300 arm+camera clips per method in Table 2, GLOMAP [2] registers all 20 sampled frames (Full 20/20), a subset (Partial 2–19), or fails (<2 poses). Below are the per-method registration counts alongside the 95% paired bootstrap confidence intervals for the camera-error differences () on the 245-clip common evaluation set (added to Appendix G, Table 10):
| Method | Full (20/20) ↑ | Partial (2–19) | Failed (<2) | ΔRotErr [95% CI] | ΔTransErr [95% CI] | ΔCam [95% CI] |
|---|---|---|---|---|---|---|
| FrameInject | 92.0% (276) | 6.7% (20) | 1.3% (4) | -0.253 [-0.52, +0.02] | -0.415 [-1.17, +0.34] | -0.529 [-1.30, +0.24] |
| TokenConcat | 94.0% (282) | 4.3% (13) | 1.7% (5) | -0.186 [-0.48, +0.11] | -2.303 [-3.03, -1.58] | -2.411 [-3.17, -1.65] |
| GameFactory | 92.7% (278) | 3.7% (11) | 3.7% (11) | -0.075 [-0.33, +0.18] | -3.335 [-4.08, -2.59] | -3.309 [-4.06, -2.56] |
| CompWorld w/o Text-Align | 95.3% (286) | 2.3% (7) | 2.3% (7) | -0.485 [-0.77, -0.20] | -3.916 [-4.69, -3.14] | -4.083 [-4.87, -3.30] |
| CompWorld | 96.0% (288) | 2.3% (7) | 1.7% (5) | - | - | - |
Out of 300 clips per method, 245 clips (81.7%) achieve Full 20/20 registration across all five Table 2 methods simultaneously; Table 2's camera metrics (RotErr, TransErr, Cam) average over this common set, while FID and FVD use all 300 clips per method. Both failed (<2 poses, 1.3%–3.7%) and partially registered (2–19 poses, 2.3%–6.7%) clips are excluded for all methods so that every method is scored on the same 245 complete 20-keyframe trajectories. CompWorld has the highest full-registration rate (96.0%, 288/300) and the fewest non-full registrations (12/300, vs. 14–24/300 for the other methods).
On the 245 matched clips, CompWorld's reductions in TransErr and Cam are statistically significant (95% CIs below zero, in bold) against TokenConcat, GameFactory, and w/o Text-Align, and its RotErr reduction is significant against w/o Text-Align. Against FrameInject, the mean differences also favor CompWorld but the CIs include zero; FrameInject reaches this camera accuracy while barely following the arm command (EE-TrajErr 25.4% vs. 25.9% without any arm command; lateral flip accuracy 50.0%, Part 1/3), matching the one-stream-dominates failure in Fig. 2(b), whereas CompWorld follows both streams simultaneously (19.6% EE-TrajErr, 70.0% flip accuracy, and 9.940 Cam). We have noted this in Sec. 4.2 and added the registration and alignment tables to Appendix G.
Together, the fixed-seed arm interventions and 300-clip end-effector tracking (A1a, A1b), the stream-wise keyboard+camera breakdown (A1c), and the alignment and registration accounting with paired CIs (A2, A3) address both of the reviewer's main concerns; we are happy to discuss any further questions.
References
[1] Karaev et al., CoTracker: It is Better to Track Together, ECCV 2024.
[2] Pan et al., Global Structure-from-Motion Revisited (GLOMAP), ECCV 2024.
Response to Reviewer wvWA (1/3): arm adherence and interventions (W1, Q1)
We thank the reviewer for suggesting a concrete test of joint control without requiring an executable robot environment, and for checking camera registration and temporal alignment in our released code. Below we respond point by point; all results have been added (in blue) to the revised PDF.
A1a: On 30 held-out RT-1 scenes, we hold the initial frame, RT-1 task text, moving camera trajectory, and sampling seed fixed while changing only the 7-D arm command: (i) scaling its per-step displacement deltas by 0.5×/1.0×/1.5×, and (ii) mirror-flipping its lateral (left/right) component. After removing background camera motion, we track the 2D gripper trajectory with CoTracker [1] and report its displacement relative to the unedited (1.0×) run and the fraction of scenes where net lateral motion under the mirrored command matches the commanded left/right sign (Lateral flip acc.; 50% chance baseline):
FrameInject (0.89 / 1.00 / 1.05, 50.0%), TokenConcat (53.3%), GameFactory (56.7%), and w/o Text-Align (53.3%) stay near 1.00 and the 50% chance level, whereas CompWorld scales monotonically (0.68 / 1.00 / 1.28) and matches the mirrored direction in 70.0% of scenes (p=0.021, one-sided binomial test vs. 50%). Because the fixed initial scene, workspace bounds, and 3D-to-2D perspective projection pull extreme 2D ratios toward 1.00 for all methods, we compare the modulation range across methods (0.60 for CompWorld vs. 0.16–0.30 for baselines and 0.22 for w/o Text-Align; Table 2(b), Sec. 4.2).
A1b: Across all 300 clips of Table 2 (50 scenes × 6 camera trajectories), we initialize CoTracker [1] from the shared first frame, remove background camera motion, and report the mean 2D gripper trajectory L2 error against the recorded RT-1 episode normalized by the 320x256 image diagonal (EE-TrajErr, Table 2(a)). To separate the arm command from the instruction text, Text-only keeps the first frame, instruction, and camera command unchanged and replaces only the arm command with its "not provided" placeholder:
EE-TrajErr is averaged over all 300 clips; the camera columns use Table 2's 245-clip matched set (see A3). Removing the arm command (Text-only) raises CompWorld's EE-TrajErr from 19.6% to 25.9% while leaving camera following nearly unchanged (Cam 10.095 vs. 9.940). Under the composed commands, all baselines remain close to this no-arm level (24.0–25.4%), while CompWorld has the lowest error on both streams (19.6% and 9.940).