acceptodds
Under review as a conference paper at ICLR 2027

Segment to Focus: Guiding Latent Action Models in the Presence of Distractors

Abstract

Latent action models (LAMs) offer a promising path to pre-training embodied agents on large amounts of action-free video. They infer latent actions between consecutive observations that can later be decoded to ground-truth actions using a small number of labels. However, recent work has shown that this recipe fails in the presence of action-correlated visual distractors, such as dynamic backgrounds, camera shake, or other moving objects. In these scenarios, the reconstruction objective drives latent actions to encode exogenous motion instead of agent-controlled dynamics, resulting in policies that underperform when fine-tuned. We observe that endogenous and exogenous factors are typically spatially separated in pixel space: control-relevant change is concentrated on the agent, while distractor motion occurs elsewhere. We exploit this observation by restricting the reconstruction objective to agent pixels, while the model still receives the full observation as input. We call this method MaskLAM; it obtains the agent mask zero-shot from off-the-shelf segmentation foundation models (e.g., SAM) and requires no architectural changes, auxiliary losses, or action labels during pre-training. Across two continuous-control benchmarks (Distracting Control Suite, Distracting Meta-World), MaskLAM reduces normalized linear-probe MSE by up to and improves normalized return by up to over LAPO. On real robot video (DROID), it yields the most action-predictive latents among all compared LAMs.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
latent action models, representation learning, visual distractors, exogenous distractors, video pretraining, segmentation, imitation learning, continuous control, behavior cloning, robot learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_x5d9OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper proposes MaskLAM, a simple modification to latent action model training that aims to reduce the influence of visual distractors. Instead of reconstructing the entire next observation, the method uses agent masks obtained from pretrained segmentation models to restrict the reconstruction loss to agent pixels, while keeping the full observation available as input. The authors compare loss masking with input masking and evaluate the approach on multiple datasets. The results show improvements in latent action prediction and downstream control performance over several baselines.

Critical strengths

The paper addresses a relevant problem in latent action learning and proposes a simple modification that requires no changes to the underlying LAM architecture. The results consistently show improvements over label-free baselines, and the comparison between loss masking and input masking provides some practical insights into the importance of preserving contextual information.

Critical weaknesses

In its current form, I find the contribution too limited for acceptance, while the writing is clear and the method works empirically well.

  1. Limited novelty and heavy reliance on external spatial priors : The core idea is to use a pretrained segmentation model to identify the agent and apply the reconstruction loss only to those pixels. While effective, this seems like a fairly straightforward use of existing segmentation and masked reconstruction techniques. The method does not discover which factors are endogenous or exogenous, but instead relies on an external segmentation model and an explicit prompt identifying the agent. In this sense, the method shifts the supervision requirement from action labels to spatial priors.

  2. The theoretical framing and claimed contribution seem stronger than what is actually shown : The paper describes loss masking as a "training-time intervention" based on the Ex-BMDP factorization, but the actual operation is simply restricting the reconstruction loss to selected pixels. This does not identify the endogenous/exogenous factorization itself. Proposition 1 also seems rather straightforward, which tells Bayes-optimal predictor with additional context cannot perform worse than one without it. It does not establish that the proposed unsupervised training recovers better actions or that loss masking is generally superior to input masking.

Questions for authors
  1. Since MaskLAM relies on a pretrained segmentation model and prompts identifying the agent, how should we compare its supervision and computational requirements with methods that use action labels? Could the authors provide a more complete comparison of these costs?

  2. Given that Proposition 1 does not establish recovery of action-relevant latent variables, what evidence supports interpreting MaskLAM as separating endogenous and exogenous dynamics, rather than simply improving action predictability through a more focused reconstruction objective? Could the authors provide a more direct analysis of whether exogenous information is actually removed from the learned latent actions?

4 comments

Response to Reviewer x5d9 (1/2)

We thank Reviewer x5d9 for recognizing the relevance of the problem, the fact that the method needs no architectural change, and the practical insight from comparing loss masking with input masking. The review raises three concerns, summarized here and answered below.

  • Novelty and spatial priors. The method is intentionally simple, and the shift of supervision from action labels to a spatial prior is its thesis.
  • Theoretical framing and evidence. Thanks to your comment, we added a new linear analysis that proves recovery of the action (Appendix V), and a direct test shows what is removed from the latent action (Table C in R2).
  • Supervision and compute cost. MaskLAM trades operator and robot time for GPU time, which can be paid afterwards for any video.

Tables A to E are in the comments R1 and R2 under the general comment.

W1: Novelty and reliance on spatial priors.

"a fairly straightforward use of existing segmentation and masked reconstruction techniques"

We discuss Segmentation Dreamer [22] and related masked-reconstruction methods in Section 2. As noted there, they train on action- and reward-labeled transitions, whereas MaskLAM brings target masking to action-free video, where the reconstruction target is the only signal that defines the latent action. For this setting, LAOM [31] concluded that supervision is required, and its follow-up [32] argues that "segmentation, while simple, is not enough". Prior work counters distractors with additional supervision or model components, such as action labels [31], an optical-flow decoder [7], a slot model [23], language conditioning [6], or a vision-language model [32]. We show that none of this is needed for a visible agent. With SAM masks and no action labels, MaskLAM's latents are more action-predictive than those of LAOM-Labels with 128k labels on DMW and on the hard DCS setting (Tables 1 and 17). The linear analysis backs this with theory (W2/Q2 below).

"The method does not discover which factors are endogenous or exogenous, but instead relies on an external segmentation model and an explicit prompt" / "the method shifts the supervision requirement from action labels to spatial priors"

This is the thesis of the paper. The conclusion states that the supervision can shift from action labels, which are scarce, to segmentation masks, which are cheap and available zero-shot. What the model learns beyond the mask is in W2/Q2 below.

W2/Q2: Theoretical framing and evidence.

"describes loss masking as a 'training-time intervention' based on the Ex-BMDP factorization ... This does not identify the endogenous/exogenous factorization itself."

We use the Ex-BMDP framework to describe the problem. The agent mask supplies one part of the factorization, the agent itself, and it is the only prior. Which context outside the mask matters is learned from data, which we see empirically in the results below (Fig. 5, Table 21, Table C in R2). Contribution 1 will name the mask as a spatial prior.

"Proposition 1 also seems rather straightforward ... It does not establish that the proposed unsupervised training recovers better actions"

As the paragraph after Proposition 1 states, it concerns the information available to the inverse model and does not guarantee what training recovers. Your comment motivated us to extend the linear analyses of [49, 25] from the full-frame objective to a masked objective. The analysis, which is Appendix V of the uploaded revision, shows the following under spatial separability, with no assumption on the strength of the distractor.

  • Under the full-frame loss the distractor is a target that the latent action is rewarded for carrying. Under the masked loss, exogenous content in the latent action can only increase the loss.
  • At every global minimizer of the masked loss, the action is an exact linear function of the latent action and the current frame, and with a tight bottleneck the latent action contains no exogenous content. The full-frame loss loses the action once the distractor has more variance [49].
  • When the latent action is larger than the action, as in our experiments, the minimum-norm solution, which weight decay selects, contains no exogenous content.
  • Input and loss masking differ when the context moves the agent. With a masked input the model sees only how the agent moved, so its latent action entangles the action with the motion caused by the context.

A simulation in the style of [49] agrees with these results. The analysis is linear and assumes a fixed mask, and it identifies the action up to a function of the current frame (as in any linear LAM). Whether the conclusions carry over to nonlinear models is what our experiments already tested.

Response to Reviewer x5d9 (2/2)

W2/Q2: Theoretical framing and evidence (continued).

"or that loss masking is generally superior to input masking"

Section 6.1 reports that the two are statistically indistinguishable on DCS and DMW. We chose them because they are the established distractor benchmarks, and during experimentation we discovered that most of the change in both is driven by the agents themselves. This motivated the Atari experiments, where the environment moves the agent and the two variants differ (Table 21). The linear analysis now proves this case (Appendix V). A model with masked input sees only how the agent moved, so its latent action encodes the sum of the action and the motion caused by the context. A model with masked loss sees the context and can subtract its effect. For a self-actuated agent the two agree.

"what evidence supports interpreting MaskLAM as separating endogenous and exogenous dynamics, rather than simply improving action predictability through a more focused reconstruction objective? Could the authors provide a more direct analysis ...?"

It uses endogenous context. The mask covers only the agent, but the inverse model attends to the ground under the cheetah and to the floes and logs that carry the agent in Atari (Fig. 5). Hiding these regions from the model, as input masking does, lowers NR from 0.94 to 0.33 (Table 21).

It discards exogenous information. Your request for a more direct analysis led to a useful new experiment for the revision. Our datasets render every simulator state twice, with and without the video background. We infer the latent action for the same transition before and after swapping in the video background of another episode and measure how far it shifts, on a scale S where 0 means unchanged and 1 means as different as two unrelated transitions (Table C in R2).

  • Replacing the background moves LAPO's latent action most of the way to an unrelated transition (S = 0.85). MaskLAM's moves by S = 0.03 with simulator masks and S = 0.13 with SAM masks.
  • LAOM-Labels, trained with 128k action labels, is at S = 0.69 on average, although the background's motion is barely decodable from its latent action (last column of Table C in R2). Action labels make the action decodable, but on average they do not make the latent action invariant to the exogenous change.

MaskLAM is therefore more than a focused reconstruction objective. Its latent action depends on the endogenous part of the scene and is nearly independent of the exogenous part. The test covers distractors outside the mask, and the hard DCS setting covers changes on the agent itself (Table 17).

Q1: Supervision and compute cost.

"how should we compare its supervision and computational requirements with methods that use action labels? Could the authors provide a more complete comparison of these costs?"

We think the fair way to compare them is by what it costs to scale each kind of supervision to more data. For robots, action labels come from teleoperation, so every labeled second needs a human operator and a robot. This is why robot datasets are small and expensive. DROID took 50 data collectors 12 months for 350 hours of data [21]. This cost grows with the data, has to be paid again for every new embodiment, and has to be paid at recording time. If the ground-truth actions were not recorded, they cannot be recovered afterwards at any cost, which is the case for most existing video, for example of humans or animals. For humans, pseudo-actions could be obtained with pose estimation, but it is unclear how good a substitute they are for ground-truth actions, and running a pose estimator on every frame is likely as expensive as running a segmentation model. The supervision of MaskLAM does not grow linearly with the data. It is one coarse box per task on DCS and DMW, and a single text prompt for all of DROID. What MaskLAM needs instead is compute for the masks, which we list in Table 3. So MaskLAM trades operator and robot time for GPU time, which can be paid afterwards for any video. We agree that the segmentation model and the prompt are still a form of supervision, but a much cheaper and more scalable one than teleoperating a robot, especially for utilizing already available, unlabelled video data.

Presentation.

Your review describes the writing as clear, while the presentation score is 1. If specific parts were hard to follow, we would be grateful for pointers and will revise them.

We hope our answers address your concerns. If they do, we would be grateful if you would consider raising your score. We are happy to run any further experiments during the rebuttal discussion if time allows.

I appreciate the authors' thoughtful responses and the effort they have put into addressing my concerns. Most of my questions have been adequately resolved. I also acknowledge that the presentation score was a mistake on my part, and I have therefore raised it from 1 to 2.

However, regardless of the effort demonstrated during the rebuttal or the number of additional experiments conducted, I respectfully maintain that the core idea does not offer a sufficiently strong contribution to warrant acceptance at ICLR. Therefore, I will maintain my original overall score.

Thank you for reading our response and for correcting the presentation score. We are glad that most of your questions are resolved. Your comments led to the linear analysis (Appendix V) and the background-swap test, and both improved the paper.

We respect that we weigh the contribution differently, and we do not want to argue the point again. We only want to say briefly where we see it. Masked reconstruction targets were known in model-based RL with action and reward labels [22], and object-centric latent action learning [23] masks the input together with the target and needs action labels to select the agent slot. What had not been shown is that masking the loss alone, with the full frame as input and zero-shot masks, is enough. It matches 128k action labels in LAOM’s own training setup, and it keeps the lowest probe error under the color and camera changes for which [32] argued that segmentation would not suffice (Table 17). In a linear model we prove that the masked loss recovers the action for any distractor strength, that weight decay favors a latent action free of exogenous content, and that loss masking differs from input masking when the environment moves the agent (Appendix V). The background-swap test and the Atari experiments confirm this for the trained models.

That a single change to the loss achieves this, where other approaches add action labels [31], a flow decoder [7], slot models [23] or a vision-language model [32], is to us the result of the paper and not a shortcoming. The mask also states whose action is to be learned, which action-free video alone leaves open when several things move.

Reviewer_L7ctOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MaskLAM, a simple modification to latent-action learning that restricts the forward dynamics reconstruction loss to agent pixels identified by a pretrained segmentation model. The inverse and forward dynamics models retain access to the full observation, preserving contextual information while reducing reconstruction pressure from distractors. The method is evaluated on Distracting Control Suite, Distracting Meta-World, and DROID, with targeted Atari experiments examining the benefits of retaining context outside the agent mask.

Critical strengths
  • The paper addresses an important failure mode with a simple and effective intervention. Reconstruction-based latent-action models can encode distracting visual changes instead of action-relevant information. MaskLAM provides a straightforward way to mitigate this problem without action labels during pretraining, and the improvements over LAPO are supported by the experiments.

  • The empirical evaluation is reasonably thorough. The paper covers locomotion and manipulation, includes real robot video, and examines latent dimensionality, action-label budgets, imperfect mask boundaries, and partial occlusion. Comparisons between ground-truth and SAM masks support the practical applicability of the approach beyond ideal segmentation.

  • The ablations provide useful understanding of the method. The experiments isolate the contribution of the masked reconstruction loss, while the analysis and Atari results explain why retaining contextual information can matter when the environment influences the agent’s motion. The paper also clearly states the assumptions and limitations of its approach.

Critical weaknesses
  • Conceptual novelty and significance are limited. Segmentation-guided reconstruction is closely related to existing task-relevant reconstruction methods, including Segmentation Dreamer [22], and object-centric latent-action learning [23]. The principal contribution is an effective application and empirical analysis of this idea within latent-action learning. The theoretical discussion provides useful motivation but does not constitute a substantial new theoretical result.

  • The method has meaningful scope restrictions. It assumes a visible, segmentable agent and does not address general egocentric video, where camera motion can affect the entire observation. Spatial masking also does not eliminate nuisance changes occurring within the agent region, such as appearance or illumination changes. These limitations restrict its applicability to uncurated video, although they are explicitly acknowledged by the authors.

  • The real-data evidence is encouraging but remains limited to representation quality. DROID demonstrates improved action predictability, but the evaluation uses randomly held-out episodes and does not establish improved downstream control or generalization to unseen scenes. More generally, improvements in probe error do not always translate into the best policy performance: MaskLAM’s DCS return trails LAOM-Labels, while its DMW success is close to LAPO-Slots. The evidence supports a useful action-label-free alternative rather than uniform superiority across metrics.

  • Some baseline details need clarification. The reported dataset-subset sizes and optical-flow masking convention leave ambiguities in interpreting the LAOF and LAPO-Slots comparisons. These questions affect the relative performance claims, while the controlled comparisons against LAPO provide convincing evidence for the core intervention.

The contribution is modest, but the central empirical claims are adequately supported. The method is simple, useful, and evaluated with reasonably thorough experiments and ablations. Its principal scope limitations are clearly acknowledged. I therefore lean toward weak acceptance (due to the limited novelty and significance).

Questions for authors
  • Could you clarify the training-data budgets in Tables 4–8? MaskLAM uses the full dataset, whereas LAOF lists 128k transitions for pretraining and policy training, and LAPO-Slots lists 128k for policy training. Do these entries describe the actual runs, and how were the budgets selected and convergence verified?

  • Could you clarify the optical-flow coordinate convention in Figure 32? The target is written as (M_{t+1}\odot\mathrm{RAFT}(o_t,o_{t+1})), whereas standard forward flow is indexed in source-frame coordinates. Is this a notation issue, or is the future-frame mask explicitly aligned with the flow? Clarifying this would strengthen the failure analysis in Appendix U.

  • How much of the improvement requires precise segmentation? A comparison against coarse bounding-box loss masks would help establish whether the benefit primarily comes from approximate foreground localization or from accurately identifying agent pixels. This would also clarify the simplest practical implementation of the method.

2 comments

Response to Reviewer L7ct (1/2)

We thank Reviewer L7ct for recognizing the importance of the failure mode, the breadth of the evaluation, and the ablations that explain when context outside the mask matters. The review raises four weaknesses and three questions, summarized here and answered below.

  • Novelty and significance. Prior work argued that segmentation is not enough for latent action learning under distractors. We show that a mask on the target is, and the revision adds a theoretical analysis (Appendix V).
  • Scope. Sections 4 and 7 state the scope, which all label-free methods we compare against share.
  • Policy performance. The DCS return gap to LAOM-Labels is now localized and originates in additional label-free modifications (Tables A and B in R1). For DROID we agree, and the easy-to-hard shift on DCS is our closest evidence for generalization (Table 17).
  • Baseline details. The budget entries and the mask index of the flow target are typos. Thank you for spotting both.
  • Precise segmentation. We ran the bounding-box comparison you suggested, and boxes keep about half of the gain (Table E in R2).

Tables A to E are in the comments R1 and R2 under the general comment.

W1: Novelty and significance.

"closely related to existing task-relevant reconstruction methods, including Segmentation Dreamer [22], and object-centric latent-action learning [23]" / "The principal contribution is an effective application and empirical analysis of this idea within latent-action learning."

We discuss Segmentation Dreamer (SD) [22] in Section 2. As noted there, SD and related methods train on action- and reward-labeled transitions, whereas MaskLAM brings target masking to action-free video. The same holds for earlier mask-restricted losses such as robot-aware control [*1] and FOCUS [*2], which also train with ground-truth actions. We will expand this discussion in the revision.

A latent action model has no actions, so the reconstruction target is the only signal that defines the latent action. For this setting, LAOM [31] concluded that supervision is required, and its follow-up [32] argues that "segmentation, while simple, is not enough". Prior work counters distractors with additional supervision or model components (Section 2), such as action labels [31], an optical-flow decoder [7], language conditioning [6], or a vision-language model [32]. We show that none of this is needed for a visible agent. With no action labels, MaskLAM's probe NMSE matches that of LAOM-Labels with 128k labels on DCS and is lower on DMW and in the hard DCS setting (Tables 1 and 17).

Beyond that, the paper shows where and how the mask has to be applied.

  1. The mask restricts the target, and the input stays complete. Object-centric latent action learning [23] selects the agent in the input on an object-centric slot and needs action labels to choose that slot. Masking the input is our ablation. It ties with loss masking on DCS and DMW, where most change is driven by the agents themselves, and fails on the Atari games, where the environment moves the agent (Table 21).
  2. The mask only needs to cover the agent. MaskLAM's inverse model still sees the scene and finds the relevant context itself (Fig. 5). Adding the manipulated object to the mask leaves the probe NMSE unchanged (Table 22).

"The theoretical discussion provides useful motivation but does not constitute a substantial new theoretical result."

Proposition 1 was meant as the reason for masking the loss and not the input. Thanks to Reviewer x5d9 (W2/Q2), the uploaded revision goes further and extends the linear analyses of [49, 25] from the full-frame objective to a masked objective (Appendix V). We show that the masked objective recovers the action exactly under arbitrary distractors outside the mask, and that with standard weight decay it favors a latent action free of exogenous content.

W2: Scope.

"assumes a visible, segmentable agent and does not address general egocentric video" / "does not eliminate nuisance changes occurring within the agent region, such as appearance or illumination changes"

We state both limitations in Section 7. They are shared by all label-free methods we compare against. In egocentric video the camera motion changes every pixel, so every latent action would absorb it. Under the moving camera and stronger color randomization of the hard DCS setting, MaskLAM still has the lowest probe NMSE of all methods (Table 17), so we think it is beneficial in practice even with this limitation.

References

[*1] Hu et al. (2022). Know Thyself: Transferable Visual Control Policies Through Robot-Awareness. ICLR.

[*2] Ferraro et al. (2023). FOCUS: Object-Centric World Models for Robotics Manipulation. arXiv:2307.02427.

Response to Reviewer L7ct (2/2)

W3: Policy performance.

"The real-data evidence is encouraging but remains limited to representation quality." / "does not establish improved downstream control or generalization to unseen scenes"

We agree. On DROID we report latent-action quality on held-out episodes and no downstream control. For generalization, the closest evidence is the easy-to-hard shift on DCS, where MaskLAM is trained on easy data only and remains at the level of LAOM-Labels on hard data (Table 17, Fig. 10).

"improvements in probe error do not always translate into the best policy performance" / "The evidence supports a useful action-label-free alternative rather than uniform superiority across metrics."

Linear action predictability is the common measure of latent action quality [31, 32, 49], and to our knowledge the community has not yet found a metric that aligns with downstream return directly. We regard a low probe error as necessary for good control, but not as sufficient.

The one case where MaskLAM trails in return is the comparison with LAOM-Labels on DCS, and we investigated where this gap comes from (Reviewer TuZz, W2/Q2). Linear action decodability does not degrade from the inverse dynamics model to the latent policy, and the gap appears only in closed loop (Table A in R1). We traced it to Stage 1 training, where LAOM-Labels uses several additional modifications to LAPO, e.g. image augmentations, a larger inverse dynamics model and a longer training schedule with warm-up. With MaskLAM's two changes in this training setup and the label loss switched off, the model reaches the DCS return of LAOM-Labels with 128k labels (mean NR 0.570 vs. 0.566, Table B in R1).

On DMW, the differences appear when labels are scarce. At 2k labels MaskLAM has the highest mean NSR of all methods, including LAOM-Labels (NSR 0.75 with SAM masks against 0.69, Fig. 19b). Among methods that use no action labels in pre-training, MaskLAM has the highest return on both benchmarks (Table 1).

W4/Q1: Training-data budgets.

"The reported dataset-subset sizes ... leave ambiguities in interpreting the LAOF and LAPO-Slots comparisons." / "Do these entries describe the actual runs, and how were the budgets selected and convergence verified?"

Thank you for spotting this. The entries are typos and do not describe the actual runs. We re-checked the configurations and logs of every run. Every baseline uses the full datasets (9M transitions per task for DCS and 1M for DMW) for all stages except the final decoder stage. 128k is the label budget of the final decoder stage, which is identical for every method. Table 2 confirms this. An end-to-end LAOF run on DCS takes 11 h 10 m and a MaskLAM run 11 h 8 m, whereas two epochs over a split 70 times smaller would take minutes. We will correct Tables 7 and 8 in the revision.

Each baseline starts from its published hyperparameters, and we adjusted learning rate, batch size and training length to our larger datasets (Section E). We verified convergence on the held-out loss and the action probes. In the last quarter of training the losses and probes change less than 1% for all methods.

W4/Q2: Optical-flow convention.

"The target is written as Mt+1⊙RAFT(ot,ot+1)M_{t+1}\odot\mathrm{RAFT}(o_t,o_{t+1}) ... Is this a notation issue, or is the future-frame mask explicitly aligned with the flow?"

It is also a notation issue, a typo in the mask index. The training code multiplies the forward flow RAFT(ot,ot+1)\mathrm{RAFT}(o_t,o_{t+1}) with the source-frame mask MtM_t, which is also how LAOF defines its target. We will correct Figures 32 and 33, their captions and the text of Section U.

Q3: Bounding-box loss masks.

"How much of the improvement requires precise segmentation? A comparison against coarse bounding-box loss masks would help ..."

It depends on what else changes inside the box. Our hypothesis, based on the LAM theory of [49], is that a box works when the agent dominates the change (variance) inside it, because the agent fills the box or the background is static, and fails when the exogenous change inside the box exceeds the agent's.

We ran the experiment you suggested on two DCS and two DMW tasks. The tight bounding box of the simulator mask replaces the pixel mask as loss mask and input channel in Table E in R2.

Boxes are better than LAPO on all four tasks and keep about half of the gain of pixel masks. This fits the picture above. With a video background, the more background the target contains, the higher the error, so the order is pixel mask, box, full frame. Approximate localization therefore already helps, but good results need a segmentation mask, which does not have to be perfect (Fig. 6c). We will add the table to the appendix.

We hope our answers address your concerns. If they do, we would be grateful if you would consider raising your score. We are happy to run any further experiments during the rebuttal.

Reviewer_TuZzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MaskLAM, a simple modification to latent action learning from action-free video. It restricts next-frame reconstruction loss to agent pixels using segmentation masks, while retaining the full scene as input to preserve useful context. Masks also serve as additional input channels. The central contribution is applying spatial supervision directly to the reconstruction objective to reduce pressure to encode distractor motion.

Validation covers Distracting Control Suite, Distracting Meta-World, DROID robot videos, and two Atari games. With SAM masks, MaskLAM improves normalized return over LAPO from 0.076 to 0.358 on DCS and normalized success from 0.636 to 0.832 on DMW. DROID action-probe NMSE improves from 0.848 to 0.798, without downstream robot evaluation. Ablations examine loss masking, input masking, temporal horizons, mask errors, latent dimensionality, and label budgets.

Critical strengths

A simple intervention with convincing controlled evidence. Masking the reconstruction loss produces substantial improvements over LAPO, and component ablations identify loss masking as the principal contributor. This offers a practical improvement without introducing a separate auxiliary objective.

Careful examination of context and segmentation dependence. Comparisons against input masking, particularly in Atari environments where platforms move the agent, substantiate why retaining scene context can matter. Ground-truth versus SAM masks and mask perturbations further support practical usability.

Critical weaknesses

Baseline comparisons appear confounded by unequal data access. Table 4 specifies full datasets for MaskLAM’s representation and latent-policy training, whereas Table 7 lists only 128,000 samples for LAOF in both stages; Table 8 similarly limits LAPO-Slots’ behavior-cloning data. These differences could materially affect the claimed advantage over competing methods, particularly because their original gains are not reproduced here.

Evidence for practical control gains remains narrower than the representation results. DROID provides only a modest probe improvement and no robot rollouts. Moreover, MaskLAM has better probe error but substantially lower DCS return than LAOM-Labels, while its DMW success with SAM masks is close to LAPO-Slots. Thus, improved linear action predictability does not establish a consistent downstream advantage.

Questions for authors

Clarification: Are the dataset sizes in Tables 4, 7, and 8 correct? If so, how do LAOF and LAPO-Slots perform when given the same unlabeled representation-learning and latent-policy training data as MaskLAM?

Clarification: Can you localize the DCS gap between MaskLAM’s stronger action-probe scores and LAOM-Labels’ stronger returns—for example, through action-decoding error from inferred latents versus predicted policy latents? This would help establish whether the remaining limitation arises in the representation or policy-learning stages.

3 comments

Response to Reviewer TuZz (1/2)

We thank Reviewer TuZz for recognizing the controlled evidence for a simple intervention and our examination of context and segmentation dependence. The review raises two concerns, summarized here and answered below.

  • Baseline fairness. LAOF and LAPO-Slots were trained on the full datasets, and the 128k entries are typos in our tables.
  • Downstream control. The diagnostic you proposed shows that linear action decodability does not degrade from the inverse dynamics model to the latent policy. The DCS gap arises in Stage 1 training and closes when our masked target intervention is applied to LAOM's training setup without labels (Tables A and B in R1).

Tables A to E are in the comments R1 and R2 under the general comment.

W1/Q1: Baseline fairness.

"Baseline comparisons appear confounded by unequal data access." / "Are the dataset sizes in Tables 4, 7, and 8 correct? If so, how do LAOF and LAPO-Slots perform when given the same unlabeled representation-learning and latent-policy training data as MaskLAM?"

Thank you for spotting this. Table 4 is correct. The 128k entries in Tables 7 and 8 are typos. LAOF and LAPO-Slots were trained on the full datasets. We re-checked the configurations and logs of every run. Every baseline uses the full datasets (9M transitions per task for DCS and 1M for DMW) for all stages except the final decoder stage. 128k is the label budget of the final decoder stage, which is identical for every method. Table 2 confirms this. An end-to-end LAOF run on DCS takes 11 h 10 m and a MaskLAM run 11 h 8 m, whereas two epochs over a split 70 times smaller would take minutes. Table 1 is therefore already the equal-data comparison you ask for, and we will correct Tables 7 and 8 in the revision.

"particularly because their original gains are not reproduced here"

We analyze this in Section U. LAOF's paper evaluates on LIBERO [*1] and Procgen [*2], whose backgrounds are mostly static, with no video playing behind the agent. In DCS and DMW the background is a moving video, so the distractors sit directly at the segmentation boundary and their flow bleeds into the agent mask (Figs. 32 and 33). LAPO-Slots does improve over LAPO on DMW in our runs, in line with its paper (Table 1). On DCS the agent does not bind reliably to the same slot in every run (Figs. 34 to 37), and the method relies on a stable agent slot. Its authors themselves name the dependence on the underlying object-centric model as a limitation [23].

References

[*1] Liu et al. (2023). LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. NeurIPS Datasets and Benchmarks.

[*2] Cobbe et al. (2020). Leveraging Procedural Generation to Benchmark Reinforcement Learning. ICML.

Response to Reviewer TuZz (2/2)

W2/Q2: Downstream control evidence.

"MaskLAM has better probe error but substantially lower DCS return than LAOM-Labels" / "Can you localize the DCS gap ... through action-decoding error from inferred latents versus predicted policy latents?"

Thank you for proposing this diagnostic, which turned out to be very informative. Table A in R1 gives the held-out error after Stages 1 and 2 and the closed-loop return for both methods.

We first followed the action information through the stages. Linear action decodability does not degrade (Table A in R1), as MaskLAM's probe NMSE stays roughly the same from the latents inferred by its inverse dynamics model (Stage 1) to the latents predicted by its latent policy (Stage 2). The gap is thus a closed-loop effect that is not visible in the quality of the latent actions.

To find the stage where this behavior arises, we kept LAOM-Labels' later stages fixed and replaced only its resulting Stage 1 model with MaskLAM's. The downstream return then matched that of MaskLAM, so the difference arises in Stage 1 training. In Stage 1, LAOM-Labels uses several additional modifications to LAPO, e.g. image augmentations, a larger inverse dynamics model, labels and a longer training schedule with warm-up (Tables 4 and 6). MaskLAM changes only the target and the mask input, so that their effect can be checked directly.

We then tested whether the gap can be closed without labels. We put MaskLAM's two changes, the agent-masked pixel target and the mask as an input channel, into LAOM-Labels' published training setup and switched its label loss off. Without any action label, this model (MaskLAOM@GT) reaches the return of LAOM-Labels with 128k labels (mean NR 0.570 vs. 0.566, Table B in R1). Without the mask, the same label-free setup reaches a mean NR of only 0.263. In this setup, the masked target raises the return as much as the 128k labels do. A new appendix section will report this analysis.

"its DMW success with SAM masks is close to LAPO-Slots"

At 128k labels the comparison is quite saturated, because plain LAPO already solves four of the ten tasks (Table 16). With 2k labels, MaskLAM with SAM masks reaches a mean NSR of 0.75 against 0.52 for LAPO-Slots, and MaskLAM is the best of all methods (Fig. 19b).

"improved linear action predictability does not establish a consistent downstream advantage"

Linear action predictability is the common measure of latent action quality [31, 32, 49], and to our knowledge the community has not yet found a metric that aligns with downstream return directly. We regard a low probe error as necessary for good control, but not as sufficient. LAOM [31] and its follow-up [32] note the limitation of the metric themselves. A linear probe shows whether the true actions are present in the latent action, but not whether exogenous content is encoded next to them. Our background-swap test measures this second property. The latent action of LAOM-Labels still moves with the background (S = 0.69 on average, against S = 0.03 for MaskLAM, Table C in R2), which is consistent with its higher probe NMSE despite the higher return.

Still, among methods that use no action labels in pre-training, MaskLAM has the highest return on both benchmarks (Table 1).

We hope our answers address your concerns. If they do, we would be grateful if you would consider raising your score. We are happy to run any further experiments in the rebuttal period.

Reviewer_TuZzOpenReview ↗

Post-rebuttal assessment and rating

Thank you for the detailed response and for conducting the stage-wise diagnostic suggested in my review. The clarification that LAOF and LAPO-Slots used the full unlabeled datasets addresses my data-access concern; the erroneous budget entries should be corrected consistently in the manuscript.

The stage-wise probe results and Stage-1 replacement experiment provide a more informative explanation of the gap between action decodability and closed-loop return. The MaskLAOM@GT comparison further strengthens the downstream evidence by showing strong performance within the LAOM training setup without action labels. However, this modified configuration uses ground-truth masks and should be distinguished from the original MaskLAM@SAM setting. The background-swap experiment also supports reduced sensitivity to external distractors, while downstream DROID control remains unevaluated.

I am maintaining my rating of 3 (Weak acceptance), with increased confidence in the empirical case. The rebuttal substantially addresses my original questions and strengthens the evidence for the proposed intervention. The distinction between the new GT-mask configuration and the practical SAM-mask setting, together with the limited real-robot control evidence, still limits a stronger recommendation. Please incorporate the corrected budgets, diagnostic protocol, and new results into the revised manuscript.

Reviewer_ajkcOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary
  1. Authors propose MaskLAM, which restricts reconstruction loss to agent pixels for latent-action learning under visual distractors.
  2. They evaluate action prediction and downstream control on DCS and DMW, and action prediction on DROID.
Critical strengths
  1. Substantial improvements in latent-action quality. MaskLAM consistently achieves lower action-prediction error than existing label-free baselines across DCS and DMW, without action supervision during pretraining. The improvement also extends to real-world DROID videos (Table 1).

  2. Strong empirical validation and ablations. The paper isolates the contribution of loss masking, input masks, and multi-step inverse dynamics. It also demonstrates robustness to imperfect segmentation masks and substantial downstream improvements over LAPO (Figures 6–7, Table 1).

Critical weaknesses
  1. Limited methodological novelty. Segmentation-guided reconstruction to ignore visual distractors was already explored in Segmentation Dreamer [1]. Moreover, input masking performs comparably to loss masking on both primary benchmarks (Figure 7), limiting the novelty of the proposed approach.

  2. Incomplete and potentially unfair baseline comparisons. The paper compares against action-supervised LAOM-Labels but omits the label-free LAOM variant [2]. Furthermore, LAOF [3] uses only 128k training transitions, while MaskLAM uses up to 9M per task, making the comparison difficult to judge as fair.

References

[1] Kim et al. (2025). Make the Pertinent Salient: Task-Relevant Reconstruction for Visual Control with Distractions. RLC. https://arxiv.org/abs/2410.09972

[2] Nikulin et al. (2025). Latent Action Learning Requires Supervision in the Presence of Distractors. ICML. https://proceedings.mlr.press/v267/nikulin25a.html

[3] Bu et al. (2025). LAOF: Robust Latent Action Learning with Optical Flow Constraints. https://arxiv.org/abs/2511.16407

Questions for authors
  1. Novelty over prior work Segmentation Dreamer [1] already uses segmentation-masked RGB reconstruction targets to suppress visual distractors. Beyond removing the reconstruction penalty outside the mask, what is the substantive novelty of MaskLAM, and how much does this specific change contribute to the reported improvements?

  2. Baseline fairness. Why is the label-free LAOM variant omitted? Additionally, would LAOF remain substantially worse if trained on the same amount of data as MaskLAM, rather than 128k transitions?

  3. Downstream policy performance. Despite substantially better latent-action prediction, why does MaskLAM achieve only marginally higher success than LAPO-Slots on DMW (0.832 vs. 0.827) and significantly lower return than LAOM-Labels on DCS (0.358 vs. 0.566)? What explains this discrepancy?

Reference

[1] Kim et al. (2025). Make the Pertinent Salient: Task-Relevant Reconstruction for Visual Control with Distractions. RLC. https://arxiv.org/abs/2410.09972

3 comments

Response to Reviewer ajkc (1/2)

We thank Reviewer ajkc for recognizing the improvements in latent-action quality on DCS, DMW and real DROID video as well as the ablations and the robustness to imperfect masks. The review raises three concerns, summarized here and answered below.

  • Novelty. Section 2 positions MaskLAM against Segmentation Dreamer. The new result is that a mask on the target suffices for latent action learning without action labels, which prior work argued it would not. The revision adds a linear analysis (Appendix V).
  • Baseline fairness. LAOF was trained on the full data. The 128k entry is a typo. Following your request we ran LAOM (label-free). It is the strongest label-free baseline on DCS (Table D in R2), and we will add it to Table 1.
  • Downstream performance. The small DMW margin is a ceiling effect at 128k labels. Your question about the DCS gap to LAOM-Labels led to a new analysis. The gap arises in Stage 1 training and closes when our masked target runs in LAOM-Labels' training setup without labels (Tables A and B in R1).

Tables A to E are in the comments R1 and R2 under the general comment.

W1/Q1: Novelty over prior work.

"Segmentation-guided reconstruction to ignore visual distractors was already explored in Segmentation Dreamer [1]." / "Beyond removing the reconstruction penalty outside the mask, what is the substantive novelty of MaskLAM?"

We discuss Segmentation Dreamer (SD) [22] in Section 2. As noted there, SD and related methods train on action- and reward-labeled transitions, whereas MaskLAM brings target masking to action-free video. The same holds for earlier mask-restricted losses such as robot-aware control [*1] and FOCUS [*2], which also train with ground-truth actions. We will expand this discussion in the revision.

SD is a Dreamer agent trained online with ground-truth actions and rewards, and its mask decides what the world model represents. A latent action model has no actions, so the reconstruction target is the only signal that defines the latent action. For this setting, LAOM [31] concluded that "without any supervision latent action learning may not be able to learn actions useful for efficient pre-training". A follow-up from the same group [32] argues in its Appendix A that "segmentation, while simple, is not enough".

Prior work counters distractors with additional supervision or model components (Section 2), such as action labels [31], optical flow [7], slot models [23], language conditioning [6] or vision-language models [32].

We show that none of this is needed for a visible agent. A mask on the target is enough and tolerates distractors on the agent. With no action labels, MaskLAM's probe NMSE matches that of LAOM-Labels with 128k labels on DCS and is lower on DMW (Table 1), and it stays lower under the color randomization and camera motion of the hard DCS setting (Table 17).

Beyond the masked loss itself, the paper establishes what the mask has to cover and where it has to be applied.

  1. The mask only needs to cover the agent. SD reports that its return on cheetah-run depends on including the ground in the mask (its Appendix C). MaskLAM's inverse model still sees the scene, and its saliency falls on the ground under the cheetah (Fig. 5). Adding the manipulated object to the mask leaves the probe NMSE unchanged (Table 22).
  2. It matters where and how the mask is applied. Having an agent mask is not sufficient by itself. LAOF applies an agent mask to a flow target, and LAPO-Slots selects the agent in the input on an object-centric slot. Both stay behind MaskLAM in probe NMSE on DCS and DMW (Table 1). We show that the mask has to restrict the pixel reconstruction target, while the inverse model keeps the full frame as input.

The uploaded revision also adds a theoretical result (Appendix V), which is motivated by a comment from Reviewer x5d9 (W2/Q2). We extend the linear analyses of [49, 25] from the full-frame objective to a masked objective. The masked objective recovers the action exactly under arbitrary distractors outside the mask, and with standard weight decay it favors a latent action free of exogenous content.

References

[*1] Hu et al. (2022). Know Thyself: Transferable Visual Control Policies Through Robot-Awareness. ICLR.

[*2] Ferraro et al. (2023). FOCUS: Object-Centric World Models for Robotics Manipulation. arXiv:2307.02427.

Response to Reviewer ajkc (2/2)

W1/Q1: Novelty over prior work (continued).

"input masking performs comparably to loss masking on both primary benchmarks (Figure 7), limiting the novelty of the proposed approach"

This is correct for DCS and DMW. We chose them because they are the established distractor benchmarks, and during experimentation we discovered that most of the change in both is driven by the agents themselves, so pixels outside the mask say little about the action, as Proposition 1 predicts. This motivated the Atari experiments on Frostbite and Frogger, where floes and logs carry the agent. There, loss masking raises NR from 0.33 to 0.94, and input masking fails on Frogger (Fig. 7b,c and Table 21). We consider this result part of the contribution.

W2/Q2: Baseline fairness.

"omits the label-free LAOM variant [2]" / "Why is the label-free LAOM variant omitted?"

We will add LAOM (label-free) to Table 1. We had compared against LAOM-Labels because the LAOM paper [31] presents it as the stronger variant. Its authors report that LAOM (label-free) "performs better, but not significantly" than LAPO (caption of their Figure 8). For the new baseline we ran LAOM-Labels' configuration with its label loss switched off, so that no action label reaches the latent model. On DCS, LAOM (label-free) is the strongest of the label-free baselines, ahead of LAPO, LAOF and LAPO-Slots in probe NMSE and NR. It stays behind MaskLAM in both (mean NR 0.26 vs. 0.38, Table D in R2 under the general comment).

"LAOF [3] uses only 128k training transitions, while MaskLAM uses up to 9M per task" / "would LAOF remain substantially worse if trained on the same amount of data as MaskLAM?"

Thank you for spotting this. The 128k entries in Table 7 are typos. LAOF was trained on the full datasets. We re-checked the configurations and logs of every run. Every baseline uses the full datasets (9M transitions per task for DCS and 1M for DMW) for all stages except the final decoder stage. 128k is the label budget of the final decoder stage, which is identical for every method. Table 2 confirms this. An end-to-end LAOF run on DCS takes 11 h 10 m and a MaskLAM run 11 h 8 m, whereas two epochs over a split 70 times smaller would take minutes. Table 1 is therefore already the equal-data comparison you ask for, and we will correct Table 7 in the revision.

Q3: Downstream policy performance.

"why does MaskLAM achieve only marginally higher success than LAPO-Slots on DMW (0.832 vs. 0.827)"

At 128k labels the comparison is quite saturated. Plain LAPO already solves four of the ten tasks, so the mean compresses the differences (Table 16). The differences appear when labels are scarce, which is the regime latent actions are meant for (Fig. 19b). With 64 times fewer labels MaskLAM loses little (NSR 0.83 to 0.75 with SAM masks), while the NSR of LAPO-Slots falls from 0.83 to 0.52. At 2k labels MaskLAM has the highest mean NSR of all methods, including LAOM-Labels (NSR 0.69).

"significantly lower return than LAOM-Labels on DCS (0.358 vs. 0.566)? What explains this discrepancy?"

Your question, which Reviewers TuZz and L7ct also raised, led us to investigate where this gap comes from. The details are in our answer to Reviewer TuZz (W2/Q2), with tables in R1 under the general comment.

Linear action decodability does not degrade from the inverse dynamics model (Stage 1) to the latent policy (Stage 2), and the gap appears only when the decoded policy runs in closed loop (Table A in R1). We traced it to Stage 1 training, where LAOM-Labels uses several additional modifications to LAPO, e.g. image augmentations, a larger inverse dynamics model and a longer training schedule with warm-up. We then put MaskLAM's two changes, the agent-masked pixel target and the mask as an input channel, into LAOM-Labels' training setup and switched its label loss off. Without any action label, this model reaches the DCS return of LAOM-Labels with 128k labels (mean NR 0.570 vs. 0.566, Table B in R1), while the same setup without labels and without the masked target reaches a mean NR of 0.26. In this setup, the masked target raises the return as much as the 128k labels do. A new appendix section will report this analysis.

We hope our answers address your concerns. If they do, we would be grateful if you would consider raising your score. We are happy to run any further experiments in the rebuttal period.

We would like to share a result that was not yet available when we posted our response. It concerns your question why MaskLAM trails LAOM-Labels in DCS return (Q3) and is in the updated Table B in R1.

MaskLAOM, our masked target in LAOM-Labels’ training setup without labels, now also runs with SAM masks. It reaches a mean normalized return of 0.566 (one seed so far), the same as LAOM-Labels (0.566) and close to the version with ground-truth masks (0.570).

Together with the label-free LAOM baseline you asked for (Table D) and the corrected data budgets, we hope this covers the points of your review: the comparison is on equal data, the strongest label-free baseline is included, and the return gap of MaskLAM closes without labels in the pipeline of LAOM.

If any concern remains open, we would be glad to address it in the time left.

Comments on the paper

2 comments

Summary of our response and of the revision

We thank all four reviewers for their careful reviews. They agree that masking the reconstruction loss is a simple change that improves latent-action quality, and that the central empirical claims are supported. Their questions led to five new results and to insights we did not have at submission. Details are in the individual replies. The new tables are in the comments R1 and R2 under this one, and the list of changes to the paper is at the top of R1. Unless stated otherwise, every new result follows the protocol of the paper.

Additions

  1. A theory for the masked loss. (prompted by Reviewer x5d9, W2/Q2, also raised by L7ct) We extend the linear analyses of [49, 25] from the full-frame objective to a masked objective in Appendix V. We show that at every global minimum of the masked objective the action is exactly decodable from the latent action and the current frame, under arbitrary distractors outside the mask, and that weight decay favors solutions whose latent action is free of exogenous content. We also show that input masking cannot separate the action from context-induced motion that the agent pixels do not reveal, whereas loss masking can.

  2. A test of what the latent action contains. (requested by Reviewer x5d9, W2/Q2) We hold the transition fixed, replace only the background, and measure how far the latent action moves on a scale S, where 0 means unchanged and 1 means as different as two unrelated transitions. LAPO's latent action moves by S = 0.85, LAOM-Labels by S = 0.69 and MaskLAM's by S = 0.03. Action labels make the action decodable, but on average they do not make the latent action invariant to the exogenous change. (Table C in R2)

  3. The origin of the DCS return gap to LAOM-Labels. (diagnostic proposed by Reviewer TuZz, W2/Q2, question also raised by ajkc and L7ct) We traced where this gap comes from (full answer to Reviewer TuZz, W2/Q2). Linear action decodability does not degrade from the inverse dynamics model to the latent policy, and the gap appears only in closed-loop rollouts. It originates in Stage 1 training, where LAOM(-Labels) uses several additional modifications to LAPO, e.g. image augmentations, a larger inverse dynamics model and a longer training schedule that make the latent action more robust in closed loop. We therefore put MaskLAM's two changes, the agent-masked pixel target and the mask as an input channel, into LAOM-Labels' training setup and switched its label loss off. Without any action label, this model (MaskLAOM@GT/MaskLAOM@SAM) reaches the DCS return of LAOM-Labels with 128k labels (mean NR 0.570/0.566 vs. 0.566). LAOM (label-free) reaches a mean NR of 0.263. In this setup, the masked target raises the return as much as the 128k labels do. (Tables A and B in R1)

  4. How precise the mask has to be. (experiment suggested by Reviewer L7ct, Q3) Bounding-box masks keep roughly half the gain of segmentation masks. Approximate localization therefore already helps, but good results need a segmentation mask. (Table E in R2)

  5. A stronger label-free baseline. (requested by Reviewer ajkc, W2/Q2) We ran LAOM (label-free) and will add it to Table 1. On DCS it is the strongest of the label-free baselines, but stays behind MaskLAM in probe NMSE and NR. (Table D in R2)

Clarifications

  1. All methods saw the same unlabeled data. (ajkc, TuZz, L7ct) We thank the reviewers for spotting this. The "128,000" entries for LAOF Stages 1 and 2 (Table 7) and LAPO-Slots Stage 4 (Table 8) are typos in our tables. We re-checked the configurations and logs of every run. 128k is the label budget of the final decoder stage for every method. Table 2 is consistent with that. An end-to-end LAOF run on DCS takes 11 h 10 m and a MaskLAM run 11 h 8 m.

  2. Novelty relative to Segmentation Dreamer [22]. (ajkc, L7ct, x5d9) Segmentation Dreamer or similar methods train on action- and reward-labeled transitions, and MaskLAM brings target masking to action-free video, where the reconstruction target alone defines the latent action. For this setting, LAOM [31] concluded that supervision is required, and its follow-up [32] argues that "segmentation, while simple, is not enough". We show that for a visible agent it is enough, since one mask on the target does the job for which prior work added action labels [31], optical flow [7], slot models [23], language conditioning [6] or vision-language models [32]. We will expand the paragraph in Section 2, and the uploaded revision backs the empirical findings with theory (Appendix V).

We also thank Reviewer L7ct for spotting a typo in Figures 32 and 33, where LAOF's flow target uses the source-frame mask MtM_t, not Mt+1M_{t+1}. Neither typo affects any result, and we will correct both in the revision.

Results R2: background-swap test, LAOM (label-free) and bounding-box masks

Table C. Background-swap test (Stage 1 models of Table 1, 4 DCS tasks, 20,000 held-out transitions per model).

Our datasets render every simulator state twice, with and without the video background, with identical state, action and agent mask. For the same transition we infer the latent action zz under both renders and report S=E∥z−z′∥2/(2 tr Cov(z))S = \mathbb{E}\lVert z - z'\rVert^2 / (2\,\mathrm{tr}\,\mathrm{Cov}(z)). S=0S = 0 means the latent is unchanged, and S=1S = 1 means it differs as much as between two unrelated transitions. Because the plain render is outside the training distribution, we also swap in the video background of another episode.

The last column checks whether the background's own motion can be read out of the latent action. It is the R2R^2 of a linear probe from zz to the frame-to-frame change of the background pixels (agent pixels removed, 8×8 average-pooled, top 16 principal components), fitted on 200 episodes and evaluated on 50 others.

Stage 1 modelSS, swap to plain backgroundSS, swap to another videoBackground change R2R^2
LAPO0.62 ± 0.030.85 ± 0.020.21 ± 0.02
LAOM-Labels (128k labels)0.45 ± 0.050.69 ± 0.010.05 ± 0.00
MaskLAM architecture, unmasked loss0.44 ± 0.050.56 ± 0.030.06 ± 0.01
MaskLAM@GT0.005 ± 0.0030.03 ± 0.010.02 ± 0.00
MaskLAM@SAM0.05 ± 0.010.13 ± 0.010.02 ± 0.00

Values are means over the four tasks. The unmasked-loss row keeps the mask as an input channel. The same probe recovers the action with R2R^2 between 0.78 and 0.84 for the masked and the supervised models.

Table D. LAOM (label-free). LAOM-Labels' configuration with its label loss switched off, on the same data, schedule and augmentations.

DCS, probe NMSE (lower is better)

Methodcheetah-runhopper-hophumanoid-walkwalker-runMean
LAPO0.384 ± 0.0220.643 ± 0.0670.677 ± 0.0380.342 ± 0.0070.511 ± 0.034
LAOM (label-free)0.252 ± 0.0050.375 ± 0.0070.414 ± 0.0030.372 ± 0.0120.353 ± 0.007
MaskLAM@GT0.117 ± 0.0010.185 ± 0.0260.353 ± 0.0060.169 ± 0.0050.206 ± 0.010
MaskLAM@SAM0.115 ± 0.0040.260 ± 0.1430.385 ± 0.0010.170 ± 0.0010.233 ± 0.037
LAOM-Labels0.165 ± 0.0020.193 ± 0.0070.311 ± 0.0030.194 ± 0.0030.216 ± 0.004

DCS, normalized return (higher is better)

Methodcheetah-runhopper-hophumanoid-walkwalker-runMean
LAPO0.206 ± 0.0210.004 ± 0.0010.038 ± 0.0120.056 ± 0.0040.076 ± 0.010
LAOM (label-free)0.613 ± 0.0890.295 ± 0.0360.043 ± 0.0150.102 ± 0.0120.263 ± 0.038
MaskLAM@GT0.759 ± 0.0310.274 ± 0.0380.076 ± 0.0120.402 ± 0.0380.378 ± 0.030
MaskLAM@SAM0.738 ± 0.0340.216 ± 0.1050.065 ± 0.0090.413 ± 0.0140.358 ± 0.041
LAOM-Labels0.942 ± 0.0020.543 ± 0.0120.045 ± 0.0030.736 ± 0.0580.566 ± 0.019

DMW, preliminary results on two tasks (probe NMSE / NSR)

Methodpushpick-place
LAPO0.481 ± 0.003 / 0.172 ± 0.0460.547 ± 0.011 / 0.056 ± 0.021
LAOM (label-free)0.383 ± 0.023 / 0.480 ± 0.0260.263 ± 0.029 / 0.377 ± 0.064
MaskLAM@GT0.144 ± 0.008 / 0.929 ± 0.0220.129 ± 0.001 / 0.777 ± 0.051
MaskLAM@SAM0.137 ± 0.006 / 0.907 ± 0.0470.132 ± 0.008 / 0.857 ± 0.029
LAOM-Labels0.207 ± 0.006 / 0.525 ± 0.0480.141 ± 0.003 / 0.443 ± 0.108

Table E. Bounding-box masks in place of pixel masks. Stage 1 only, probe NMSE (lower is better), two DCS and two DMW tasks.

Stage 1 modelcheetah-runwalker-runpushpick-placeMeanGain kept
LAPO (no mask)0.384 ± 0.0220.342 ± 0.0070.481 ± 0.0030.547 ± 0.0110.439 ± 0.0110%
MaskLAM with bounding-box masks0.213 ± 0.0010.220 ± 0.0040.376 ± 0.0100.365 ± 0.0200.294 ± 0.00949%
MaskLAM@GT (pixel masks)0.117 ± 0.0010.169 ± 0.0050.144 ± 0.0080.129 ± 0.0010.140 ± 0.004100%

The bounding-box variant uses the tight bounding box of the simulator mask as loss mask and as input channel. Everything else is as in MaskLAM@GT. Gain kept is the share of the mean NMSE reduction from LAPO to MaskLAM@GT that the variant retains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.