acceptodds
Under review as a conference paper at ICLR 2027

MoFa-World: Motion-Factorized Latent Actions for Interactive World Models

Abstract

Egocentric world models provide a promising foundation for interactive environment simulation, yet existing approaches tend to focus on egomotion control, with object-interaction control constrained by the scarcity of densely annotated interaction videos. Although latent action models (LAMs) learn action representations from videos without action labels, they often entangle object interaction and egomotion in a single representation. We believe that pixel-space motion patterns can reveal distinct action semantics, providing a basis for learning decoupled representations. Building on this insight, we introduce MoFA-LAM, which pretrains region-level latent actions through pixel-space motion factorization and aggregates them into interaction and egomotion representations through region-specific reconstruction with limited mask supervision. Probing experiments demonstrate the functional specialization of the two representations and the benefits of motion-factorized pretraining over vanilla LAM pretraining. We further introduce MoFA-WORLD, an egocentric world model conditioned on these representations through distinct pathways. It supports cross-scene action transfer and adaptation to dataset-specific controls, including joint use of interaction and egomotion controls learned from different datasets. Together, these results support pixel-space motion factorization as a way to learn decoupled control representations for egocentric world models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Interactive World Models, Motion-Factorized Latent Actions

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Uuu4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MoFa-World, an egocentric world model that supports separate and joint control of object interaction and egomotion. Its central contribution is MoFa-LAM, a two-stage latent-action model: it first learns regional motion factors through frame reconstruction using predicted displacements, without action labels, masks, or optical-flow supervision; it then uses limited interaction-mask supervision to aggregate these factors into interaction and egomotion representations. MoFa-World incorporates these representations through separate cross-attention and spatial-modulation pathways, respectively.

Validation includes frozen-representation probes for interaction classification, robot action regression, and camera-pose estimation on datasets unseen during latent-action training. These experiments demonstrate complementary functional specialization and improvements over AdaWorld and a vanilla-LAM-pretraining variant. World-model experiments further evaluate cross-scene action transfer, adaptation to explicit controls, and composition of controls learned from different datasets. Ablations examine pretraining, mask supervision, and the number of regional factors. Overall, the results support the usefulness of motion-factorized pretraining, although evidence for cross-scene transfer and joint control is primarily qualitative.

Critical strengths
  1. The paper study an important and interesting: disentangling object interaction from egomotion, with the potential to make better use of large-scale unlabeled video data and improve action transfer, data-efficient adaptation, and compositional control in world models.

  2. Under identical mask supervision, motion-factorized pretraining outperforms vanilla LAM pretraining, improving interaction classification from 49.04% to 59.51% and reducing camera-rotation error from 0.381° to 0.348°. Probes on three datasets unseen during LAM training support complementary functional specialization, while downstream experiments demonstrate improved control accuracy. Combining controls learned from different datasets without additional joint training further suggests composability, although this evidence is primarily qualitative.

Critical weaknesses
  1. It is unclear whether the gains require explicit disentanglement or mainly come from mask-guided visual representation learning. The vanilla-LAM variant uses the same masks, but there is no comparison using a shared action representation with the same masked reconstruction losses. A semantic-alignment baseline such as Olaf-World is also missing. These comparisons would help establish what the proposed decomposition adds beyond stronger representation learning.

  2. The method relies on a complex pipeline of motion-factorized pretraining, mask-guided routing and aggregation, and separate conditioning pathways. This design, together with manual mask filtering, raises questions about how readily it can scale to large, diverse video collections. Yet the results mainly demonstrate functional specialization, leaving unclear whether this complexity yields reliably independent control.

Questions for authors

Do you expect the proposed explicit disentanglement approach to scale to large, unlabeled video collections? In particular, how much manual mask filtering was required, and what evidence suggests that the interaction/background split remains useful in more diverse scenes with independently moving objects?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Uuu4 (1/3)

We thank the reviewer for recognizing the importance of separating object interaction from egomotion, the gains of motion-factorized pretraining under identical mask supervision, and the complementary specialization on datasets unseen during LAM training. We have included the additional results and clarifications in the revised manuscript.


[W1-a] Shared representation with the same masked reconstruction losses

We appreciate this valuable suggestion to include a shared-latent baseline.

MoFA outperforms the requested shared-latent control on all four probes. We use a single 64D shared latent to match the total dimensionality of MoFA’s two 32D branches. The shared latent reconstructs a single RGB frame, with the same foreground and background losses applied to their respective regions. Both models start from the same pretrained MoFA-LAM checkpoint and use the same masks for 15k stage-2 steps.

MethodHOI4D Top-1 (%) ↑AgiBot NMSE ↓Camera RRE (°) ↓Camera RTE ↓
AdaWorld, no masks46.720.8010.4260.092
Ours (v-LAM)49.040.6930.3810.091
Shared latent (64D) + masked losses52.600.6920.4490.093
MoFA-LAM, task branch (other branch)59.51 (55.50)0.634 (0.752)0.348 (0.577)0.087 (0.098)

With the same pretrained checkpoint, masked losses and total latent dimensionality, MoFA-LAM’s task-specific 32D branches outperform the shared 64D latent on all four probes, including 6.91 percentage points higher HOI4D accuracy and 22.5% lower camera rotation error. These results support explicitly separating interaction and egomotion even when the shared representation receives the same regional supervision.

We have added the shared-latent comparison to Section 4.5 (Table 4) and described its masked reconstruction losses in Appendix B.1.1.


[W1-b] Comparison with Olaf-World semantic alignment

Thank you for suggesting this comparison. We did not compare with Olaf-World in the main paper because the two methods address different questions: MoFA separates interaction from camera motion when both occur in the same transition, while Olaf-World learns consistent representations of similar actions across contexts.

Following your suggestion, we evaluated Olaf-World’s alignment objective on a single shared 64D latent, matching the total dimensionality of MoFA-LAM’s two 32D branches. The baseline uses SeqΔ-REPA to align the shared latent with changes in frozen V-JEPA2 features. MoFA-LAM outperforms this baseline on all four probes.

MethodHOI4D Top-1 (%) ↑AgiBot NMSE ↓Camera RRE (°) ↓Camera RTE ↓
Olaf-style alignment (shared 64D)53.650.7100.4160.090
MoFA-LAM59.51 (55.50)0.634 (0.752)0.348 (0.577)0.087 (0.098)
MoFA-LAM + Olaf-style alignment60.66 (54.93)0.597 (0.756)0.350 (0.577)0.085 (0.098)

MoFA rows report the task-specific branch first (FG for action, BG for camera), with the other branch in parentheses.

Although MoFA and Olaf-World have different research goals, their approaches can be combined. We applied semantic alignment separately to each MoFA branch using teacher features pooled over its corresponding region. The combined model scores better on three of four probes than MoFA alone, including lower action-regression error (0.634 to 0.597), with a small increase in rotation error (0.348° to 0.350°).

We have added the Olaf-style alignment comparison to Section 4.5 (Table 4) and the results for each branch to Appendix B.1.1 (Table 9).

AuthorsOpenReview ↗

Response to Reviewer Uuu4 (2/3)

[W2-a, Q] Scalability and manual mask filtering

Thank you for raising this practical question. Yes, we expect MoFA to scale to larger unlabeled video collections: pretraining on raw video requires no masks. Only the 15k-step fusion stage uses masks; inference is mask-free. Even at the 1% mask setting, MoFA achieves higher HOI4D accuracy and lower camera rotation error than AdaWorld (Appendix B.1.2). In Stage 2, MoFA-LAM uses 0.34% more trainable parameters than v-LAM.

Results at the 1% mask budget.

ModelHOI4D Top-1 (%) ↑Camera RRE (°) ↓
AdaWorld46.720.426
MoFA (1% mask data)53.940.384

Manual filtering. The manual filter requires only one decision per clip: accept or reject its automatically generated masks. The masks require no manual correction or editing, so screening took about 5 seconds per clip. Checking the masks used in our experiments took about 8–10 person-hours. Given this modest effort, we do not expect mask screening to be a major barrier to scaling. As VLMs and AI agents advance, we expect agents to automate the entire screening process by using a VLM to inspect each clip and accept or reject its masks.

We have clarified the mask budgets in Section 4.5 (Table 3) and documented the manual filtering procedure and screening time in Appendix A.4.


[W2-b] Reliability of independent control

Thank you for raising this point. We agree that branch specialization alone does not establish whether the added complexity yields reliable independent control of interaction and camera motion.

To test independent control directly, we changed one command at a time and checked whether the other task still met its criterion. Across 60 scenes, we crossed two interaction commands with two egomotion commands (camera turns), generating 240 videos per method (480 total). Within each scene, we matched the initial frame, text, seed and sampling settings.

For the target interaction command, we report the percentage of videos that Qwen3-VL scored 4 or higher. Egomotion success requires a camera turn with an estimated yaw change of at least 5° in the commanded direction. For each switch, we consider only trials in which the unchanged task initially met its criterion and report the percentage that failed afterward. We count both switch directions.

MethodInteraction score ≥ 4 ↑Egomotion success ↑Interaction failure after egomotion switch ↓Egomotion failure after interaction switch ↓
Ours (v-LAM)82.50%67.50%9.09%3.70%
MoFA-World90.00%96.67%7.41%1.72%

Among interactions initially scored ≥4, switching the egomotion command pushed 7.41% below 4 for MoFA-World and 9.09% for v-LAM. Among turns that initially met the yaw criterion, switching the interaction command caused 1.72% to miss it for MoFA-World and 3.70% for v-LAM. Thus, for both methods, the unchanged task usually still met its criterion after the switch.

We have added the crossed-control protocol to Appendix C.2 and reported the results in Table 15.

AuthorsOpenReview ↗

Response to Reviewer Uuu4 (3/3)

[Q] Usefulness of the interaction/background split in scenes with independently moving objects

We appreciate the reviewer's question about whether the interaction/background split remains useful in scenes with independently moving objects. As noted in Limitations, our current model focuses on egocentric manipulation. The interaction/background split is appropriate for most of these scenes because it groups regions involved in the manipulation separately from the surrounding scene.

Egomotion on EPIC-Kitchens. We tested whether the background branch remains useful for estimating camera rotation when background motion is high. We measured background motion as the motion remaining outside the hands after compensating for camera motion. This measure can reflect both independent object motion and parallax. We split 606 clips at the median (303 per group). The background branch had lower rotation error than the interaction branch in both groups, including the high background motion group (0.430° vs. 0.702°).

Camera RRE (°) ↓Low background motionHigh background motion
AdaWorld0.3070.523
Ours (v-LAM), BG0.2750.466
MoFA-LAM, BG0.2450.430
MoFA-LAM, FG0.4270.702

Independently moving objects on MOVi. When several objects move independently, a useful alternative to the interaction/background split is to group motion by object. Stage 1 learns regional motion factors without masks or predefined categories, so Stage 2 can assign those factors to individual objects using their masks. To explore the potential for per-object control, we adapted MoFA on 500 MOVi scenes [1] with one instance mask per object. The held-out examples illustrate the resulting groups.

In Section 5 (Limitations), we have clarified the scope of our evaluation and identified control of independently moving objects as a direction for future work. We have added the EPIC-Kitchens results to Appendix C.5 (Table 20) and the MOVi examples to Appendix D (Figure 16).

[1] Klaus Greff et al. Kubric: A Scalable Dataset Generator. CVPR, 2022.


We hope our responses have addressed the reviewer's concerns. If anything remains unclear or if you have further questions, we'd be happy to clarify and discuss them in more detail.

Reviewer_fmfiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MOFA-WORLD for separately controlling viewpoint motion and object interaction in a first-person interactive world model. The method consists of three stages: first, MOFA-LAM learns multiple region-level latent actions from videos without action labels, where each latent variable describes local pixel motion through region assignment and shared 2D displacement; second, using interaction region masks from about 10% of the videos, these motion factors are fused into an interaction latent and an egomotion latent through region reconstruction; finally, the authors feed the two latent variables into the Wan2.2-TI2V-5B video diffusion model through different conditioning paths, and map discrete action instructions to the corresponding latent space. The experiments include interaction classification, robot action regression, camera pose estimation, cross-scene action transfer, dataset-specific control adaptation, and cross-dataset joint control. The results show that the two latent variables are functionally complementary to some extent, and that motion-factorized pretraining improves probing tasks and downstream control performance compared with vanilla-LAM pretraining. The paper also demonstrates separated and compositional control of viewpoint motion and object interaction through qualitative results.

Critical strengths

Strength 1 — The idea of decoupled control is clear and practically meaningful. The paper combines local pixel motion factors, sparse mask supervision, and two different conditioning paths to form an interaction latent and an egomotion latent. Probing experiments show that the interaction latent performs better on interaction classification and robot action regression, while the egomotion latent performs better on camera pose estimation. This functional complementarity directly supports the paper's core claim about control decoupling. Strength 2 — The experiments cover multiple levels and include useful comparisons. The paper simultaneously evaluates latent variable probing, cross-scene transfer, explicit control adaptation, and cross-dataset joint control. In particular, Ours (v-LAM) uses the same mask-guided RGB reconstruction supervision as the full model, allowing the role of motion-factorized pretraining to be compared to some extent in isolation.

Critical weaknesses

Weakness 1 — The decoupling mechanism lacks sufficient identifiability and collapse analysis. In Section 3.2 and Equations (1)–(2), the first stage mainly learns region assignments and shared displacements through a next-frame reconstruction loss, without explicit constraints on factor diversity, mutual exclusivity, or load balancing. The router in the second stage adopts hard grouping, but the paper does not report the utilization of each region-level latent variable in the two branches, assignment stability, or whether multiple latent variables redundantly represent the same motion. The flow field visualization in Figure 10 provides intuitive evidence, but it is still insufficient to prove that what is learned is a stable semantic factor rather than an arbitrary motion decomposition that can reduce reconstruction error. Weakness 2 — Statistical and reproducibility evidence for comparisons and downstream evaluations remains insufficient. The paper mainly compares AdaWorld, Ours (v-LAM), and the full model, and does not directly compare with recent factorized latent-action methods mentioned in Related Work. The results in the tables also do not report means, standard deviations, or confidence intervals over multiple runs. Cross-scene transfer and joint control are mainly demonstrated through qualitative videos. HOI4D action accuracy relies on sequence-level scoring by Qwen3-VL, but the scoring prompts, the number of evaluation samples, human verification, or inter-rater agreement are not provided. In addition, the world model training, adaptation, and test splits for MIND and HOI4D need clearer specification.

Questions for authors

Question 1: On whether motion factors collapse Please report the utilization rates of the 8 region-level latent variables in the first stage, region assignment entropy, displacement similarity, or other diagnostic results that guard against factor redundancy/collapse. Is it possible for the second-stage router to assign most latent variables to the same branch? If there is no mask, mask noise, or variation in mask quality, do the behaviors of the two branches remain stable? Question 2: On mask data and data splits Please provide the actual number of videos used for each dataset, the number of videos with masks, the training/validation/test splits, and the mask filtering criteria. In particular, please specify whether the videos used for world model pretraining, control adaptation, and final testing in MIND and HOI4D overlap in terms of clips, scenes, objects, or participants. Question 3: On baseline fairness and statistical reliability Are AdaWorld, Ours (v-LAM), and MOFA-WORLD identical in training steps, number of trainable parameters, number of adaptation parameters, random seeds, and computational budget? Why was no direct comparison conducted with FLAM, OTF-LAM, DiLA, or other recent factorized latent-action methods? Please explain whether these methods cannot be compared due to differences in interfaces or data settings. Question 4: On Qwen3-VL evaluation and the RPE metric Please provide the prompts used by Qwen3-VL when evaluating action sequences, the number of samples, whether blind evaluation was performed, whether human rating references were used, and the rating consistency results. For camera pose evaluation, please explain what information the translational scale error still retains after Sim(3) alignment is performed on each video, and report the results without this alignment or a sensitivity analysis. Question 5: On quantitative results for cross-scene transfer Figures 5, 11, and 12 mainly present qualitative transfer results. Please specify how many reference videos and target scenes are included in the cross-scene experiments, and provide interaction completion rate, viewpoint trajectory error, or other quantitative transfer metrics.

9 comments
AuthorsOpenReview ↗

Response to Reviewer fmfi (1/6)

We thank the reviewer for recognizing the complementary roles of the two branches and the value of the matched-mask comparison. We appreciate your assessments of clarity (Presentation: 3) and informativeness (Informativeness: 3). Below we address your concerns about Soundness: 2. We have included the additional results and clarifications in the revised manuscript.


[W1, Q1-a] Utilization and redundancy of the eight stage-1 motion factors

We appreciate the reviewer's suggestion to check whether any motion factors go unused or predict the same motion. Each of the eight slots predicts how a region moves from one frame to the next. We evaluated the original checkpoint on 6,000 frame transitions from 400 clips (100 per training source), selected independently of model outputs.

Slot utilization. We counted each pixel toward the slot with the highest weight and called a slot active if it covered more than 1% of the pixels in that transition. On average, 7.09 of the eight slots were active.

Slot12345678
Transitions where slot was active (%)69.3399.9296.6082.2270.9299.8094.6095.50
Pixels assigned to slot (%)3.6229.337.154.446.2524.2716.838.10

Normalized entropy measures how evenly the assignments are spread: 0 means all weight goes to one slot, and 1 means an even split across all eight. The average was 0.485 for an individual pixel’s weights and 0.799 for the slots’ shares of the whole frame. In other words, a pixel tends to favor particular slots, while different parts of the frame use several slots.

Displacement similarity. For pairs of active slots that both predicted motion greater than 0.1 pixels, we compared their motion vectors. Mean cosine similarity was 0.243 (1 means the same direction). Only 4.34% of these slot pairs predicted nearly identical displacements (within 0.5 pixels).

We then kept the pixel assignments fixed and replaced each active slot’s motion with the most similar motion predicted by another active slot. For all eight slots, this swap made the reconstructed next frame less accurate on average. The mean increase in squared pixel error ranged from 4.22 × 10⁻⁴ to 14.86 × 10⁻⁴ across slots. All eight cluster-bootstrap 95% confidence intervals were above zero.

We have added the slot utilization and redundancy analysis to Appendix C.3, with per-slot utilization in Table 16.


[W1, Q1-b] Router utilization and stability

On EgoDex, GR1 and OpenAoE, each branch received at least two of the eight factors on every evaluated transition. Across these manipulation datasets, the mean number assigned to interaction ranged from 3.07 to 4.02. On the navigation-only MIND dataset, the router could assign all eight slots to egomotion. Across 165 transitions, 86.5% of slot assignments went to egomotion and 13.5% to interaction.

SourceClipsInteraction factors, mean ± SD /8Both branches nonemptySlot stayed in the same branch (%)Expected by chance (%)
EgoDex1003.067 ± 0.614100.00%97.97%92.42%
GR1224.024 ± 0.550100.00%96.59%93.00%
OpenAoE123.167 ± 0.860100.00%95.98%88.21%
MIND111.079 ± 1.40142.42%96.02%81.35%

Per-slot utilization. Each cell gives the percentage of transitions in which that slot went to interaction (the remainder went to egomotion).

SourceTransitionsSlot 1Slot 2Slot 3Slot 4Slot 5Slot 6Slot 7Slot 8
EgoDex1,500100.00%0.33%100.00%0.07%0.00%84.67%21.60%0.00%
GR1330100.00%0.00%92.42%93.33%13.33%0.00%100.00%3.33%
OpenAoE180100.00%0.00%100.00%8.33%0.00%76.67%31.67%0.00%
MIND16541.82%0.00%30.30%0.00%0.00%21.21%14.55%0.00%

Assignment stability. We evaluated 145 clips (2,175 transitions) held out of stage-2 training. Slots stayed in the same branch on adjacent steps within a clip 95.98–97.97% of the time. The rate exceeded each dataset’s chance baseline, with gaps ranging from 3.59 percentage points on GR1 to 14.67 on MIND.

We have reported branch assignments and their stability in Appendix C.3 (Tables 17–18).

AuthorsOpenReview ↗

Response to Reviewer fmfi (2/6)

[Q1-c] Robustness to mask quantity, noise and quality

We appreciate this valuable feedback. During training, MoFA with a 1% mask budget outperformed AdaWorld on both probes, and perturbing mask boundaries preserved the branches' distinct roles.

No masks. MoFA needs no masks at inference. In stage-2 training, removing masks removes the supervision that assigns each branch its role. We have no zero-mask MoFA training result.

Mask quantity. HOI4D Top-1 decreased by 0.38 percentage points from the nominal 10% setting to the 5% setting. At 1%, MoFA still outperformed AdaWorld on both probes. The available clip pool also varies with the mask budget.

Training setting (probe seed 0)HOI4D Top-1 (%) ↑Camera RRE (°) ↓
MoFA, 10% masks59.510.348
MoFA, 5% masks59.130.363
MoFA, 1% masks53.940.384
AdaWorld baseline (no masks)46.720.426

Mask quality. We retrained stage 2 for 15k steps with random 1–5 pixel dilation or erosion, fixed within each training window. Even with noisy mask boundaries, the interaction branch (FG) remained better at recognizing actions, and the egomotion branch (BG) remained better at estimating camera rotation.

Metric (fixed encoders, mean ± SD over five probe seeds)Clean masksPerturbed boundaries
HOI4D FG Top-1 (%) ↑59.5662 ± 0.131759.5639 ± 0.0507
HOI4D BG Top-1 (%)55.5542 ± 0.118555.6733 ± 0.0766
Camera BG RRE (°) ↓0.347407 ± 0.0008260.355296 ± 0.000735
Camera FG RRE (°)0.575485 ± 0.0009010.562212 ± 0.001364
Camera BG RTE ↓0.086713 ± 0.0000580.086703 ± 0.000059
Camera FG RTE0.098267 ± 0.0000670.097011 ± 0.000044

We have reported the mask ablations in Section 4.5 (Table 3) and the five-seed boundary perturbation results in Appendix B.1.2 (Table 12).


[W1] Motion factors and action semantics

We agree that Figure 10 shows how regions move, but the flow alone cannot tell us what action they represent. Stage 1 learns regional motion factors. Stage 2 then groups these factors into separate representations for interaction and egomotion. We therefore test whether the interaction representation supports action recognition and retrieval across clips.

Effect of pretraining. With the same stage-2 mask supervision, HOI4D action accuracy reached 59.51% with motion-factorized pretraining, compared with 49.04% with vanilla-LAM initialization and 49.59% with random initialization. Thus, the benefit of motion-factorized pretraining extends beyond frame reconstruction.

Interaction retrieval across clips. For each of 10,997 held-out HOI4D action events (737 clips), we averaged its features and retrieved the five closest training events by cosine similarity from a pool of 32,934. We used action labels only to score the matches.

RepresentationTop-5 matches with the same HOI4D interaction label (%)95% clip-bootstrap CI
AdaWorld34.76[33.89, 35.61]
v-LAM, interaction branch36.36[35.48, 37.24]
MoFA-LAM, interaction branch63.97[62.82, 65.13]

With MoFA’s interaction branch, 63.97% of retrieved events shared the query’s interaction label, compared with 36.36% for the v-LAM interaction branch.

We have added the retrieval results to Appendix B.1.1 (Table 10). Section 4.5 (Table 3) reports the pretraining comparison.

AuthorsOpenReview ↗

Response to Reviewer fmfi (3/6)

[W2, Q2] Data, masks and splits

Mask-guided fusion (stage 2). The mask-budget ablation uses nominal clip budgets of 1%, 5%, and 10% of all clips from each source. At 10%, the quota covers training and validation together. At 1% and 5%, it sets the training quota, and both runs reuse the 10% validation split. The default set comprised 7,269 clips (6,907 training and 362 validation). Stage 1 uses raw RGB video from the same four sources.

SourceTotal source clipsStage 2 clips (train + val)Training clipsValidation clips
OpenAoE2,51025123912
GR14,42044242022
EgoDex63,5506,3556,038317
MIND2,21022121011
Total72,6907,2696,907362

World-model training.

SourceTraining clipsValidation clips
MIND20710
GR136219
EgoDex4,364218
HOI4D2,300150

Control adaptation.

DatasetTraining clipsValidation clipsTable 1 test clips per method
HOI4D2,47627250
MIND1,6309650

Mask filtering. MIND used all-background masks for all 221 retained clips and required no manual review. For each other source, we screened until its quota was met, retaining complete masks aligned to the RGB frames and covering the intended regions. Screening took about 5 seconds per clip. Checking the masks used in our experiments took about 8–10 person-hours.

Table 1 evaluation set. All methods used the same 50 HOI4D clips and 50 MIND clips, with matched initial frames, control sequences and generation seeds.

Train–test split criteria. For both HOI4D and MIND, we held out entire scenes before sampling test clips. We checked the source-video IDs against the world-model training and control-adaptation manifests and verified that neither training set contained a test scene or its source videos. On HOI4D, object-instance and participant IDs were not split criteria.

We have clarified the mask-budget denominator in Section 4.1 and Appendix A.3, and documented mask filtering and dataset splits in Appendix A.4 (Tables 5–6).


[Q3-a] Fairness of the comparison

MoFA-World and v-LAM used identical step counts at every training stage. AdaWorld followed the same step counts except that it skipped the 15,000 mask-guided steps. All three methods used the same evaluation seeds.

Training stepsAdaWorldOurs (v-LAM)MoFA-World
LAM pretraining steps150,000150,000150,000
Additional training steps with masks015,00015,000
World-model checkpoint (step)20,00020,00020,000
Adaptation steps on HOI4D2,0002,0002,000
Adaptation steps on MIND2,0002,0002,000

Trainable parameters. All three methods trained the same number of parameters during world-model training and on each adaptation task. Compared with v-LAM, MoFA-World used 0.0011% more trainable parameters in Stage 1 and 0.34% more in Stage 2.

Training stageAdaWorldOurs (v-LAM)MoFA-World
Stage 1 (LAM pretraining)474,708,032474,708,032474,713,408
Stage 2 (mask-guided training)N/A474,708,800476,336,512
World-model training167,054,976167,054,976167,054,976
HOI4D interaction adaptation48,785,15248,785,15248,785,152
MIND egomotion adaptation4,245,8244,245,8244,245,824

Seeds. We used the same five probe seeds (0–4) and the same generation seed for each test case across methods. All three methods used the same training seed.

We have documented the training steps, seeds, and trainable parameter counts in Appendix A.4 (Table 7).

Reviewer_yDVgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper learns separate latent controls for egomotion and object interaction in egocentric video. MoFa-LAM first learns regional motion factors through frame reconstruction, then uses limited mask supervision to aggregate them into two action representations. MoFa-World incorporates these representations through separate conditioning pathways. Evaluation includes frozen-feature probes, action transfer, adaptation to explicit commands, and composition of controls learned on different datasets.

Critical strengths

The comparison with vanilla-LAM pretraining is the most convincing part of the paper. Both variants receive the same mask supervision, yet motion-factorized pretraining improves interaction classification from 49.04% to 59.51% and also improves camera-pose estimation. This gives the proposed motion representation empirical support beyond the qualitative generation examples. The probes are conducted on datasets unseen during LAM training, which makes the observed specialization more meaningful.
I also like the cross-dataset control composition. Appendix A.1 explains how the interaction and egomotion adapters are trained separately and combined without further joint training. The examples are limited, but this is a useful demonstration of why separating the controls matters in practice, rather than simply producing two differently named embeddings.

Critical weaknesses

The evidence for independent control is weaker than the evidence for useful specialization. Figure 4 measures how well information can be decoded from each representation; it does not measure whether changing one control leaves the other behavior intact. Figure 9 does provide separate-conditioning examples, so this question is not ignored. What remains unclear is how consistently the separation holds across scenes, particularly when camera motion and manipulation occur together. A quantitative crossed-control test would make the composability claim considerably more convincing.
The interaction evaluation also needs firmer validation. The 72% versus 52% result in Table 1 is the fraction of videos receiving a full score from Qwen3-VL, not an independently measured task-success rate. Appendix B.2 gives a sensible rubric, but I could not find the evaluation sample count or agreement with human judgments. Without these, it is difficult to judge the reliability of an important downstream result.

Questions for authors
  • Across a fixed set of scenes, what happens to object interaction when only the egomotion control changes, and vice versa? Is there a quantitative measure of these unintended effects?
  • How many clips underlie Table 1, how are they separated from training and adaptation data, and has the interaction judge been checked against blinded human ratings?
4 comments
AuthorsOpenReview ↗

Response to Reviewer yDVg

We thank the reviewer for recognizing the benefit of motion-factorized pretraining under identical mask supervision, the probes on datasets unseen during LAM training, and the practical value of composing controls learned on different datasets without joint training. We have included the additional results and clarifications in the revised manuscript.


[W1, Q1] Quantitative crossed-control test

Thank you for this suggestion. We agree that switching one command while holding the other fixed tests independent control more directly than probing alone.

Across 60 scenes, we crossed two interaction commands with two egomotion commands (camera turns), generating 240 videos per method (480 total). Within each scene, we matched the initial frame, text, seed and sampling settings.

For the target interaction command, we report the percentage of videos that Qwen3-VL scored 4 or higher. Egomotion success requires a camera turn with an estimated yaw change of at least 5° in the commanded direction. For each switch, we consider only trials in which the unchanged task initially met its criterion and report the percentage that failed afterward. We count both switch directions.

MethodInteraction score ≥ 4 ↑Egomotion success ↑Interaction failure after egomotion switch ↓Egomotion failure after interaction switch ↓
Ours (v-LAM)82.50%67.50%9.09%3.70%
MoFA-World90.00%96.67%7.41%1.72%

After switching the egomotion command, 7.41% of initially qualifying MoFA-World interactions scored below 4. After switching the interaction command, 1.72% of initially successful turns failed. We have added the crossed-control protocol and results to Appendix C.2 (Table 15).


[W2, Q2] Agreement with blinded human ratings

We agree that “Success” was an imprecise label in Table 1. We renamed it “Full-score rate” to reflect what it measures: the percentage of videos that Qwen3-VL scores 5/5 (72% for MoFA and 52% for v-LAM). Our evaluation draws on the VLM-Action-Judge in §4.3 of ActWorld [1].

User study. We randomly sampled 15 HOI4D cases from Table 1 and compared all three methods (45 videos). For each video, we used the original Qwen3-VL score from the run that produced Table 1. 20 raters evaluated instruction completion on a 1–5 scale, with an option to mark a video unjudgeable. Raters viewed each case’s three videos side by side, with case order and method positions randomized and both method names and VLM scores hidden.

Individual score agreement. Of 900 individual ratings (20 raters × 45 videos), 5 were marked unjudgeable. We compared each of the 895 valid ratings with Qwen’s score for the same video. 34.19% (306/895) matched exactly, and 80.22% (718/895) differed by at most one point, including exact matches.

Method comparison. Both human raters and Qwen gave MoFA-World the highest mean score. On these 45 videos, the MoFA advantage over v-LAM was 1.54 points with human ratings and 0.40 with Qwen. Against AdaWorld, the corresponding gaps were 1.41 and 0.47 points.

JudgeAdaWorldOurs (v-LAM)MoFA-World
Human2.8322.6984.240
Qwen3-VL3.333.403.80

We have added the user study results to Section 4.4 (Table 2) and the detailed scoring protocol to Appendix B.2.

[1] Zhexiao Xiong et al. ActWorld: From Explorable to Interactive World Model via Action-Aware Memory. arXiv:2606.17730, 2026.


[Q2] Evaluation size and separation from training data

We appreciate the reviewer's reminder to report the sample counts and clarify how test scenes were held out. We have reported the test set sizes in Section 4.4 and detailed the scene-level split in Appendix A.4, with clip counts by training stage in Table 6.

Size. Table 1 uses 50 HOI4D clips and 50 MIND clips. The set is fixed and shared by all methods, with matched initial frames, control sequences and generation seeds.

Separation. We constructed the test pool at the scene level before sampling. We checked the source-video IDs against both the world-model training and control-adaptation manifests and verified that no evaluation scene or source video appeared in either training set.


We hope our responses have addressed the reviewer's concerns. If anything remains unclear or if you have further questions, we'd be happy to clarify and discuss them in more detail.

Thanks for adding the additional experiments. Those new results resolve most of my questions.

Additionally, please consider release the annoynmous repository to facilitate reproductivity, even during the discuss phase. My score stays at 3.

We are glad that the additional experiments addressed most of your questions. Thank you for suggesting that we make an anonymous repository available during the discussion period to facilitate reproducibility. We are checking whether we can prepare the repository before the discussion period ends and will share the link here if it is ready in time.

Thank you again for suggesting that we share the repository. As the discussion draws to a close, we are still organizing the code and preparing the documentation. We are committed to releasing our code publicly to support reproducibility and foster further developments in this direction.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response

We thank all three reviewers for their thoughtful feedback. They highlighted the gains from motion-factorized pretraining under identical mask supervision (yDVg, Uuu4, fmfi), the complementary roles of the interaction and egomotion branches (Uuu4, fmfi), and the value of composing controls learned from different datasets without additional joint training (yDVg, Uuu4). In response to the reviewers’ questions, we have added the following to the revised manuscript: new experiments on motion representations, control, and evaluation reliability, and clarifications on the method’s scalability and scope.

1. Explicit separation under matched supervision. (Uuu4) Starting from the same pretrained checkpoint, we trained one shared 64D latent or two 32D branches with the same masks, reconstruction losses, and training steps. The separate branches outperformed the shared latent on all four probes, raising HOI4D action accuracy from 52.60% to 59.51%. Applying Olaf-style semantic alignment to each branch further improved three of the four metrics (Section 4.5, Table 4).

2. Control under command switches. (yDVg, Uuu4) Across 60 scenes and 480 generated videos, we changed one command at a time. For MoFA-World, after an egomotion switch, 7.41% of initially qualifying interactions fell below the scoring threshold. After an interaction switch, 1.72% of initially successful egomotion trials failed (Appendix C.2, Table 15).

3. Agreement between human and Qwen3-VL ratings. (yDVg, fmfi) In a blinded study, 20 raters scored 45 videos from 15 randomly sampled HOI4D test cases on the same five-point scale as Qwen3-VL. We compared each of the 895 valid individual ratings with the original Qwen3-VL score for that video: 34.19% matched exactly and 80.22% differed by at most one point. Both humans and Qwen3-VL gave MoFA-World the highest mean interaction score (Section 4.4, Table 2).

4. Motion factor utilization and redundancy. (fmfi) Across 6,000 frame transitions, 7.09 of eight slots were active on average. We kept pixel assignments fixed and replaced each active slot’s predicted motion with the closest prediction from another active slot. The swap increased the average next-frame reconstruction error for every slot (Appendix C.3).

5. Feasibility of large-scale video training. (Uuu4, fmfi) Pretraining uses raw video without masks. Only the fusion stage uses mask supervision, and even with a 1% mask budget, MoFA outperformed AdaWorld on both action recognition and camera rotation estimation. Masks are generated automatically. Manual screening requires only an accept-or-reject decision per clip, with no mask editing. Current vision-language agents could make these accept-or-reject decisions at scale.

6. Scope of the interaction/background split. (Uuu4) Our current model focuses on egocentric manipulation. The split is well suited to scenes where the interaction is localized to the hands and the objects they manipulate, while apparent motion in the surrounding scene comes mainly from camera movement. Our EPIC-Kitchens experiments show that the background branch remains useful for estimating camera rotation even in more complex scenes with stronger background motion (Appendix C.5, Table 20). Our exploratory experiments on MOVi show that MoFA can also group motion by object in scenes with several independently moving objects. Controlling independently moving objects separately remains a direction for future work (Section 5 and Appendix D, Figure 16).

Taken together, these findings support our central insight: pixel-space motion patterns help distinguish object interaction from egomotion. MoFA-LAM uses limited mask supervision to group learned regional motion factors into two specialized latent actions. Through separate conditioning pathways, MoFA-World can transfer controls across scenes, adapt to dataset-specific commands, and combine controls learned on different datasets without additional joint training.

AuthorsOpenReview ↗

Summary of Contributions and Rebuttal Responses

Dear Reviewers, AC, SAC, and PC,

We thank the reviewers for their thoughtful feedback and the chairs for guiding the discussion.

Paper Overview

Our key insight is that distinct action semantics can be reflected in different motion patterns in pixel space. MoFA-LAM first learns region-level latent actions through pixel-space motion factorization. It then uses limited mask supervision and region-specific reconstruction to aggregate these factors into separate latent actions for object interaction and egomotion. MoFA-World uses these representations through distinct conditioning pathways, supporting cross-scene action transfer, dataset-specific control adaptation, and composition of controls learned on different datasets without additional joint training.

Initial Ratings

The initial ratings were 3 (yDVg), 2 (Uuu4), and 2 (fmfi). During the discussion, Uuu4 and fmfi each raised their rating to 3. All three reviewers now rate the paper 3 (weak acceptance).

Consensus on Strengths

All three reviewers highlighted the benefits of motion-factorized pretraining over vanilla LAM pretraining under identical mask supervision. Uuu4 and fmfi recognized the complementary specialization of the interaction and egomotion representations. yDVg and Uuu4 highlighted the value of composing controls learned on different datasets.

Main Concerns and Rebuttal Actions

  1. Explicit separation under matched supervision (Uuu4, fmfi). Two 32D branches outperformed a shared 64D latent on all four probes, raising HOI4D accuracy from 52.60% to 59.51%. Applying Olaf-style alignment to each branch further improved three of the four metrics (Table 4). MoFA-LAM also outperformed FLAM, OTF-LAM, and DiLA on all four probes (Table 8).

  2. Redundancy of regional motion factors (fmfi). Replacing each active slot’s motion prediction with the closest prediction from another active slot increased mean next-frame reconstruction error for all eight slots (Appendix C.3). The reviewer noted that this result substantially reduced their concern about redundancy.

  3. Control under command switches (yDVg, Uuu4). Across 60 scenes, 7.41% of MoFA-World interactions that initially scored at least 4/5 fell below that threshold after an egomotion switch. After an interaction switch, 1.72% of initially successful camera turns failed (Table 15).

  4. Human validation of interaction scores (yDVg, fmfi). In a blinded study, 20 raters scored 45 videos from 15 HOI4D test cases. Of the 895 valid individual ratings, 34.19% matched Qwen3-VL exactly and 80.22% were within one point. Both human raters and Qwen3-VL gave MoFA-World the highest mean interaction score (Table 2).

  5. Mask supervision and the scope of the learned controls (Uuu4, fmfi). Pretraining requires no masks. With a 1% mask budget during fusion, MoFA-LAM achieved higher action accuracy and lower camera rotation error than AdaWorld (Appendix B.1.2). Our current model focuses on object interaction and egomotion in egocentric manipulation videos. Grouping motion factors with per-object masks could extend this approach to controlling independently moving objects (Section 5 and Appendix D).

We have incorporated these results into the revised manuscript. We are committed to releasing our code publicly to support reproducibility and foster further developments in this direction. We appreciate your time and consideration.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.