- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MoFa-World, an egocentric world model that supports separate and joint control of object interaction and egomotion. Its central contribution is MoFa-LAM, a two-stage latent-action model: it first learns regional motion factors through frame reconstruction using predicted displacements, without action labels, masks, or optical-flow supervision; it then uses limited interaction-mask supervision to aggregate these factors into interaction and egomotion representations. MoFa-World incorporates these representations through separate cross-attention and spatial-modulation pathways, respectively.
Validation includes frozen-representation probes for interaction classification, robot action regression, and camera-pose estimation on datasets unseen during latent-action training. These experiments demonstrate complementary functional specialization and improvements over AdaWorld and a vanilla-LAM-pretraining variant. World-model experiments further evaluate cross-scene action transfer, adaptation to explicit controls, and composition of controls learned from different datasets. Ablations examine pretraining, mask supervision, and the number of regional factors. Overall, the results support the usefulness of motion-factorized pretraining, although evidence for cross-scene transfer and joint control is primarily qualitative.
Critical strengths
-
The paper study an important and interesting: disentangling object interaction from egomotion, with the potential to make better use of large-scale unlabeled video data and improve action transfer, data-efficient adaptation, and compositional control in world models.
-
Under identical mask supervision, motion-factorized pretraining outperforms vanilla LAM pretraining, improving interaction classification from 49.04% to 59.51% and reducing camera-rotation error from 0.381° to 0.348°. Probes on three datasets unseen during LAM training support complementary functional specialization, while downstream experiments demonstrate improved control accuracy. Combining controls learned from different datasets without additional joint training further suggests composability, although this evidence is primarily qualitative.
Critical weaknesses
-
It is unclear whether the gains require explicit disentanglement or mainly come from mask-guided visual representation learning. The vanilla-LAM variant uses the same masks, but there is no comparison using a shared action representation with the same masked reconstruction losses. A semantic-alignment baseline such as Olaf-World is also missing. These comparisons would help establish what the proposed decomposition adds beyond stronger representation learning.
-
The method relies on a complex pipeline of motion-factorized pretraining, mask-guided routing and aggregation, and separate conditioning pathways. This design, together with manual mask filtering, raises questions about how readily it can scale to large, diverse video collections. Yet the results mainly demonstrate functional specialization, leaving unclear whether this complexity yields reliably independent control.
Questions for authors
Do you expect the proposed explicit disentanglement approach to scale to large, unlabeled video collections? In particular, how much manual mask filtering was required, and what evidence suggests that the interaction/background split remains useful in more diverse scenes with independently moving objects?
5 comments
Response to Reviewer Uuu4 (2/3)
[W2-a, Q] Scalability and manual mask filtering
Thank you for raising this practical question. Yes, we expect MoFA to scale to larger unlabeled video collections: pretraining on raw video requires no masks. Only the 15k-step fusion stage uses masks; inference is mask-free. Even at the 1% mask setting, MoFA achieves higher HOI4D accuracy and lower camera rotation error than AdaWorld (Appendix B.1.2). In Stage 2, MoFA-LAM uses 0.34% more trainable parameters than v-LAM.
Results at the 1% mask budget.
| Model | HOI4D Top-1 (%) ↑ | Camera RRE (°) ↓ |
|---|---|---|
| AdaWorld | 46.72 | 0.426 |
| MoFA (1% mask data) | 53.94 | 0.384 |
Manual filtering. The manual filter requires only one decision per clip: accept or reject its automatically generated masks. The masks require no manual correction or editing, so screening took about 5 seconds per clip. Checking the masks used in our experiments took about 8–10 person-hours. Given this modest effort, we do not expect mask screening to be a major barrier to scaling. As VLMs and AI agents advance, we expect agents to automate the entire screening process by using a VLM to inspect each clip and accept or reject its masks.
We have clarified the mask budgets in Section 4.5 (Table 3) and documented the manual filtering procedure and screening time in Appendix A.4.
[W2-b] Reliability of independent control
Thank you for raising this point. We agree that branch specialization alone does not establish whether the added complexity yields reliable independent control of interaction and camera motion.
To test independent control directly, we changed one command at a time and checked whether the other task still met its criterion. Across 60 scenes, we crossed two interaction commands with two egomotion commands (camera turns), generating 240 videos per method (480 total). Within each scene, we matched the initial frame, text, seed and sampling settings.
For the target interaction command, we report the percentage of videos that Qwen3-VL scored 4 or higher. Egomotion success requires a camera turn with an estimated yaw change of at least 5° in the commanded direction. For each switch, we consider only trials in which the unchanged task initially met its criterion and report the percentage that failed afterward. We count both switch directions.
| Method | Interaction score ≥ 4 ↑ | Egomotion success ↑ | Interaction failure after egomotion switch ↓ | Egomotion failure after interaction switch ↓ |
|---|---|---|---|---|
| Ours (v-LAM) | 82.50% | 67.50% | 9.09% | 3.70% |
| MoFA-World | 90.00% | 96.67% | 7.41% | 1.72% |
Among interactions initially scored ≥4, switching the egomotion command pushed 7.41% below 4 for MoFA-World and 9.09% for v-LAM. Among turns that initially met the yaw criterion, switching the interaction command caused 1.72% to miss it for MoFA-World and 3.70% for v-LAM. Thus, for both methods, the unchanged task usually still met its criterion after the switch.
We have added the crossed-control protocol to Appendix C.2 and reported the results in Table 15.
Response to Reviewer Uuu4 (3/3)
[Q] Usefulness of the interaction/background split in scenes with independently moving objects
We appreciate the reviewer's question about whether the interaction/background split remains useful in scenes with independently moving objects. As noted in Limitations, our current model focuses on egocentric manipulation. The interaction/background split is appropriate for most of these scenes because it groups regions involved in the manipulation separately from the surrounding scene.
Egomotion on EPIC-Kitchens. We tested whether the background branch remains useful for estimating camera rotation when background motion is high. We measured background motion as the motion remaining outside the hands after compensating for camera motion. This measure can reflect both independent object motion and parallax. We split 606 clips at the median (303 per group). The background branch had lower rotation error than the interaction branch in both groups, including the high background motion group (0.430° vs. 0.702°).
| Camera RRE (°) ↓ | Low background motion | High background motion |
|---|---|---|
| AdaWorld | 0.307 | 0.523 |
| Ours (v-LAM), BG | 0.275 | 0.466 |
| MoFA-LAM, BG | 0.245 | 0.430 |
| MoFA-LAM, FG | 0.427 | 0.702 |
Independently moving objects on MOVi. When several objects move independently, a useful alternative to the interaction/background split is to group motion by object. Stage 1 learns regional motion factors without masks or predefined categories, so Stage 2 can assign those factors to individual objects using their masks. To explore the potential for per-object control, we adapted MoFA on 500 MOVi scenes [1] with one instance mask per object. The held-out examples illustrate the resulting groups.
In Section 5 (Limitations), we have clarified the scope of our evaluation and identified control of independently moving objects as a direction for future work. We have added the EPIC-Kitchens results to Appendix C.5 (Table 20) and the MOVi examples to Appendix D (Figure 16).
[1] Klaus Greff et al. Kubric: A Scalable Dataset Generator. CVPR, 2022.
We hope our responses have addressed the reviewer's concerns. If anything remains unclear or if you have further questions, we'd be happy to clarify and discuss them in more detail.
Response to Reviewer Uuu4 (1/3)
We thank the reviewer for recognizing the importance of separating object interaction from egomotion, the gains of motion-factorized pretraining under identical mask supervision, and the complementary specialization on datasets unseen during LAM training. We have included the additional results and clarifications in the revised manuscript.
[W1-a] Shared representation with the same masked reconstruction losses
We appreciate this valuable suggestion to include a shared-latent baseline.
MoFA outperforms the requested shared-latent control on all four probes. We use a single 64D shared latent to match the total dimensionality of MoFA’s two 32D branches. The shared latent reconstructs a single RGB frame, with the same foreground and background losses applied to their respective regions. Both models start from the same pretrained MoFA-LAM checkpoint and use the same masks for 15k stage-2 steps.
With the same pretrained checkpoint, masked losses and total latent dimensionality, MoFA-LAM’s task-specific 32D branches outperform the shared 64D latent on all four probes, including 6.91 percentage points higher HOI4D accuracy and 22.5% lower camera rotation error. These results support explicitly separating interaction and egomotion even when the shared representation receives the same regional supervision.
We have added the shared-latent comparison to Section 4.5 (Table 4) and described its masked reconstruction losses in Appendix B.1.1.
[W1-b] Comparison with Olaf-World semantic alignment
Thank you for suggesting this comparison. We did not compare with Olaf-World in the main paper because the two methods address different questions: MoFA separates interaction from camera motion when both occur in the same transition, while Olaf-World learns consistent representations of similar actions across contexts.
Following your suggestion, we evaluated Olaf-World’s alignment objective on a single shared 64D latent, matching the total dimensionality of MoFA-LAM’s two 32D branches. The baseline uses SeqΔ-REPA to align the shared latent with changes in frozen V-JEPA2 features. MoFA-LAM outperforms this baseline on all four probes.
MoFA rows report the task-specific branch first (FG for action, BG for camera), with the other branch in parentheses.
Although MoFA and Olaf-World have different research goals, their approaches can be combined. We applied semantic alignment separately to each MoFA branch using teacher features pooled over its corresponding region. The combined model scores better on three of four probes than MoFA alone, including lower action-regression error (0.634 to 0.597), with a small increase in rotation error (0.348° to 0.350°).
We have added the Olaf-style alignment comparison to Section 4.5 (Table 4) and the results for each branch to Appendix B.1.1 (Table 9).