acceptodds
Under review as a conference paper at ICLR 2027

FOCWM: Factorized Object-Centric World Model for Latent-Action Dynamics

Abstract

Existing object-centric world models typically decompose scenes into object slots, yet each slot remains a homogeneous representation in which geometry, shape, and appearance are entangled. Such entanglement requires inverse dynamics models to implicitly extract action-relevant cues from object representations, complicating latent-action inference. To address this challenge, we propose a Factorized Object-Centric World Model (FOCWM), which introduces a structured object representation together with adaptive latent-action inference for object-centric latent dynamics modeling. Specifically, we decompose each object slot into three complementary factors: spatial geometry, occupancy, and appearance. We further propose an action-oriented adaptive fusion mechanism that integrates these factors into object-wise representations for latent-action inference, allowing each factor to contribute adaptively according to the underlying dynamics. The inferred latent actions, together with the factorized object states, are then used to predict future object states. Experiments on synthetic and real-world datasets show that FOCWM consistently improves future prediction performance and downstream behavior learning compared with existing object-centric world models. The source code will be made publicly available.

Primary area:
learning on time series and dynamical systems
Keywords:
object-centric learning, world model, latent action, video prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TJXPOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a slot-attention-based object-centric world model, termed FOCWM, where each slot is factorized into geometry (position, scale, depth), occupancy (visibility) and appearance factors. The dynamic module operates over the latent slots along with learned latent actions. The model is evaluated on video prediction and downstream imitation learning on various datasets.

Critical strengths
  • Novelty: my impression is that the authors try to bring the attributes of the latent particles (DLP) to the slot-attention framework. I think that it makes a lot of sense and overall a novel idea. This mechanism can compensate, for example, when several particles overlap and can actually be aggregated to a single entity. I find this a novel contribution to the self-supervised object-centric family. I also find the latent action per-attribute a novel idea.
  • Results: the proposed method outperforms PlaySlot and is comparable or slightly better than LPWM.
  • Detailed appendix.
  • A promise for an open-source code.
Critical weaknesses
  • Writing: overall, I have a good understanding of the method; however, I believe the paper could benefit from a writing revision to make the method section clearer. For example, in line 216, a patch of size LL is introduced. It is not clear where it came from and what the role of this in the slot-attention framework is. Typically, the particle approach relies on glimpses/patches, so the connection should be made clearer. If this stitching part is inspired by Lin et al. or DLP, it should be clearly mentioned. In addition, up to the method section, the authors mention “action fusion” multiple times, and it is not clear what the readers should understand from this term.
  • Complexity and multi-stage: to my understanding, the model is trained in two stages, where in the second stage the representation is frozen. Compared to end-to-end models such as Dreamer or LPWM, the representation is not trained to be predictable by the dynamics modules. That also means that training this model online (e.g. for model-based applications) would be challenging. I find this a major limitation of the model.
  • Stochasticity: to my understanding, all the video prediction experiments in the paper are based on first extracting latent actions from GT videos, and then using them to condition the dynamics module. Can latent actions be sampled from the fixed unit Normal prior? Can this model be used for stochastic video prediction?
  • Limited benchmark: the model is only evaluated on 4 datasets with very small number objects.
  • Limitations: no discussion of limitations and future work. I can think of several limitations, as mentioned in my review, that I believe should be explicitly mentioned.
Questions for authors
  • Stability: can the authors please address the stability of the object decomposition across runs? Slot-attention is notoriously hard to train with unstable slot decompositions between runs, and I’m interested in whether the additional attributes actually help with stable decomposition.
  • Scenes with a large number of objects: how would the proposed method perform on datasets with larger numbers of objects, like LanguageTable or BAIR datasets, what would the slot decompositions look like with the proposed method?
  • LPWM training: was LPWM trained with similar reconstruction losses (perceptual + MSE)?
3 comments

We sincerely thank the reviewer for the thorough and constructive feedback. We appreciate the careful assessment of our work and the valuable suggestions. We address each concern and question in detail below.

W1:

The local patch of size L is generated by the RGB and alpha decoders from object slots, rather than extracted as an input glimpse by Slot Attention. The spatial features determine how these patches are transformed and placed onto the image canvas for reconstruction. This design is related to patch-based object rendering approaches such as DLP, and we will clarify this connection. We will also replace the potentially ambiguous term "action fusion" with "action-oriented feature fusion," which specifically refers to adaptively combining spatial, alpha, and RGB features for latent-action inference.

W2:

Our two-stage design separates representation learning from dynamics learning. The first stage establishes functionally specialized object features through dedicated decoding pathways. Freezing the representation then provides stable feature targets for training the IDM and FDM, which are supervised by both feature prediction and image reconstruction losses. We acknowledge that this does not explicitly optimize the representation for predictability and may limit online adaptation compared with end-to-end approaches.

W3:

FOCWM supports stochastic video prediction without access to future observations.

Specifically, we introduce a learned history-conditioned Gaussian prior (Appendix A.6, Eqs. 33–34). The prior and posterior are jointly trained, with the prior learning to approximate the posterior through KL divergence. At inference time, latent actions can be sampled from this learned prior to generate future frames autoregressively, as illustrated in Figure 8.

We additionally evaluated this history-only setting on Sketchy using FVD (lower is better). FOCWM achieves 125.61, compared with 85.45 for LPWM. Although LPWM performs better in stochastic generation, these results confirm that FOCWM supports quantitative history-only video prediction. Improving its stochastic generation quality remains an important direction for future work.

W4:

Although our evaluation is limited to four datasets with relatively few objects, these benchmarks cover diverse manipulation scenarios across simulated and real-world environments, including multi-object interactions and dual-arm manipulation.

Furthermore, we have obtained stable object decompositions on the LanguageTable dataset. FOCWM achieves a PSNR of 30.5, SSIM of 0.95, and LPIPS of 0.039, compared with 29.5, 0.94, and 0.037 for LPWM, respectively. Overall, our method achieves comparable video prediction performance to LPWM on this additional benchmark. These results provide preliminary evidence that our representation can generalize beyond the evaluated benchmarks.

W5:

FOCWM has several limitations. First, although it performs well on the evaluated manipulation datasets, its object-centric decomposition may become less reliable in complex real-world scenes with densely packed objects, severe occlusions, and intricate object interactions. Second, FOCWM adopts a two-stage training strategy in which the representation model is frozen during dynamics learning. While this simplifies optimization, it limits joint adaptation between object representations and dynamics and may hinder online learning. Future work will explore more robust object decomposition in complex scenes and end-to-end optimization of representation and dynamics learning.

Q1:

In our experiments, FOCWM consistently produces stable object decompositions across independent training runs. Furthermore, Figure 5 demonstrates temporally consistent object assignments across video frames in both simulated and real-world environments.

Q2:

We additionally conducted experiments on both LanguageTable and BAIR, using 12 and 15 slots, respectively.

On LanguageTable, FOCWM achieves stable object-level slot decompositions across consecutive frames. Quantitatively, FOCWM achieves a PSNR of 30.5, SSIM of 0.95, and LPIPS of 0.039, compared with 29.5, 0.94, and 0.037 for LPWM, respectively, demonstrating comparable video prediction performance. Figure 5(c) shows the decomposition results.

On BAIR, however, we observed lower-quality slot decompositions despite increasing the number of slots to 15. We hypothesize that this is related to the greater scene complexity, including frequent occlusions and overlapping or stacked objects, which make consistent object separation more challenging.

These results suggest that FOCWM can generalize to scenes with more objects, while robust object decomposition in heavily occluded and cluttered environments remains an important direction for future work.

Q3:

LPWM was trained with both perceptual and MSE reconstruction losses, following its official implementation and training configurations to ensure a fair comparison.

Reviewer_TJXPOpenReview ↗

Thank you for your effort!

I thank the authors for their significant effort during the rebuttal phase, it is appreciated. I acknowledge that I have read the other reviews and their responses as well. My questions have been answered and my impression of the paper remains positive.

Overall, given my familiarity with the field, I find the main contribution lies in improving the slot-attention framework with stable training and additional attributes, bringing it closer to the particles representations but overcoming several limitations with it. I believe the community would benefit from access to a better slot-attention framework, as proposed in this work. However, as for the downstream applications (video prediciton, decision-making), I don't think the results (and the new additions as well) are entirely convincing, and the margins are very small. Mainly, the results on LanguageTable suggest that several objects can still be modeled with a single slot (so the decomposition is not entirely object-centric), and the fact that the model struggles with more objects (BAIR) emphasizes the limitations of the slot-attention framework, where other representations already work with complex scenes. So, the argument that the representation is more "compact" than, e.g., the particles, is nice and all, but if it does not work on more complex datasets that other methods already scale to, I don't find this a compelling argument to use the method.

Considering the limited scale of ratings this year, I believe my current score reflects my impression of the work, and I'm happy to keep it.

We sincerely thank the reviewer for their thoughtful feedback, constructive suggestions, and positive assessment of our work. We greatly appreciate the time and effort you have devoted to reviewing our paper and engaging in the discussion throughout the rebuttal process. We also appreciate your recognition of our main contributions and understand your remaining concerns regarding the downstream applications and scalability. Your feedback is valuable and will help us further improve our work and guide future research. Thank you again for your careful consideration and encouragement.

Reviewer_36jzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes FOCWM for learning object-centric latent actions and visual dynamics from videos without action annotations. The method first extracts object slots using temporally recurrent Slot Attention. Separate projection heads then map each foreground object into spatial, alpha, and RGB features, which are decoded into position, scale and depth ordering, local opacity, and local appearance, respectively. An inverse dynamics module adaptively fuses these features and infers object-wise Gaussian latent actions from consecutive states. A forward dynamics module combines object states with latent actions to autoregressively predict future representations and reconstruct video frames. The paper evaluates video prediction on BlockPush, ButtonPress, PlaceBreadSkillet, and Sketchy, and evaluates behavior policies trained from expert demonstrations on the first two simulated tasks.

Critical strengths
  1. Structuring each object representation according to rendering functions is reasonably motivated.
  2. Video prediction and behavior learning results show consistently positive trends under the reported evaluation protocols, achieving SoTA compared with other baselines.
Critical weaknesses
  1. The independent contributions of factorization and adaptive fusion are not sufficiently isolated. Table 3 provides limited support for adaptive fusion. Relative to directly using all three feature groups, Sketchy PSNR increases only from 29.23 to 29.45, LPIPS decreases from 0.063 to 0.061, and ButtonPress success increases from 79.2 ± 1.9% to 80.3 ± 2.1%. These numbers suggest a possible modest improvement but do not establish that it is robust and practically meaningful.

  2. Limited evaluation without future observations. Table 1 uses latent actions inferred from ground-truth transitions, while history-only prediction is evaluated only qualitatively. Can you provide some quantitative results?

  3. Unclear source of prediction gains. The reported metrics do not sufficiently separate improvements in representation and decoding quality from improvements in dynamics modeling.

  4. Limited control-task coverage. Control evaluation covers only two simulated tasks, with gains of 2.7 and 2.6 percentage points over LPWM, leaving broader generalization untested.

Questions for authors

See weaknesses

4 comments

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the careful assessment of our work and the valuable suggestions regarding factorization, adaptive fusion, prediction evaluation, and control generalization. We address each concern in detail below.

W1:

First, factorization provides the primary improvement. Compared with holistic-slot inputs, directly combining factorized features improves ButtonPress success from 75.5% to 79.2% (+3.7 percentage points), demonstrating the benefit of structured features for latent-action inference and downstream behavior learning.

Second, adaptive fusion provides a complementary improvement. Although its incremental gains are modest, it consistently improves the reported metrics over direct feature combination. Furthermore, Table 5 shows that the learned weights assign different importance to spatial, occupancy, and appearance features across four datasets, supporting the motivation for action-oriented feature fusion.

W2:

We additionally evaluated history-only prediction on Sketchy using FVD (lower is better). FOCWM achieves an FVD of 125.61, compared with 85.45 for LPWM.

While LPWM performs better in this setting, we believe the difference may be related to two aspects of the model design.

First, representation–dynamics optimization. FOCWM adopts a two-stage training strategy, where the factorized object representations are frozen during dynamics learning. In contrast, LPWM jointly optimizes object representations and stochastic dynamics, allowing its representations to adapt to prediction objectives.

Second, differences in training objectives. Although both methods learn latent-action distributions, FOCWM primarily supervises predicted object states through feature-level MSE and image reconstruction losses, whereas LPWM additionally regularizes predicted particle-state distributions through KL divergence. This probabilistic dynamics objective may be advantageous for stochastic future generation.

However, FOCWM outperforms LPWM in posterior-conditioned prediction (Table 1), demonstrating its strength in latent-action-conditioned dynamics modeling. Improving history-only stochastic prediction remains an important direction for future work.

W3:

Table 3 provides a controlled comparison in which the pretrained representation model and decoder remain unchanged, while the IDM inputs and fusion strategies are varied. Under these settings, Sketchy PSNR improves from 28.98 to 29.45 dB, indicating that the improvements are not solely attributable to reconstruction quality.

Furthermore, our additional object-level evaluation shows that FOCWM achieves a lower Centroid ADE than LPWM (4.876 vs. 4.958 pixels), providing evidence that the benefits extend beyond appearance reconstruction to object-level motion consistency.

Together, these results support the effectiveness of combining structured object representations with latent-action dynamics modeling, which is the central objective of FOCWM.

W4:

Although the control evaluation is limited to two simulated tasks, ButtonPress and BlockPush involve distinct manipulation challenges, including precise contact-based interaction and multi-object pushing. We will explore broader control generalization across more diverse tasks and environments in future work.

We sincerely appreciate your thoughtful comments and constructive suggestions regarding the independent contributions of factorization and adaptive fusion, history-only prediction, dynamics modeling, and control-task generalization.

In our responses, we have further clarified the distinct roles of factorization and adaptive fusion, supported by additional explanations of the ablation results and learned feature weights across four datasets.

Additionally, following your valuable suggestion, we have conducted a quantitative evaluation of history-only prediction using FVD and provided a detailed discussion of the observed results and differences in training strategies and objectives. We have also clarified how our controlled ablation studies isolate the effects of latent-action modeling and provided additional object-level evaluation results to assess motion consistency.

Furthermore, we have discussed the scope of our current control evaluations and acknowledged the importance of extending our approach to more diverse tasks and environments.

We would be grateful to hear whether our responses, additional experimental results, and clarifications have adequately addressed your concerns.

We remain fully open to further discussion and would be happy to address any remaining questions or suggestions.

Thank you again for your valuable time and thoughtful consideration.

Thanks for your effort.

The authors present adaptive fusion as a distinct contribution (see introduction #2), but the current experiments do not establish that dynamically adjusting the fusion weights yields a robust or practically meaningful improvement. I will keep my original scores

Thank you for this insightful comment. To further investigate the effectiveness of adaptive fusion, we have added ablation experiments on the PlaceBreadSkillet dataset, where objects exhibit more complex motion patterns and dynamic interactions. We have also included row-wise relative performance changes in the updated Table 3 to make the improvements of different fusion strategies more explicit. The new results show that adaptive fusion consistently improves performance over directly combining all factorized features on PlaceBreadSkillet, achieving gains of 5.14% in PSNR, 2.27% in SSIM, and 2.61% in LPIPS. Moreover, adaptive fusion also brings meaningful improvements on Sketchy, with a 0.75% increase in PSNR and a 3.17% improvement in LPIPS, while maintaining the same SSIM. These results provide additional evidence that adaptively adjusting fusion weights is beneficial across different scenarios, with more pronounced advantages in complex dynamic environments, supporting the practical value of the proposed adaptive fusion strategy. We have updated Table 3 in the revised manuscript to reflect these findings.

Reviewer_i1fKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary
  1. The authors present a slot-based representation that is used to train an IDM and FDM jointly from video data.
  2. The authors' main contribution is factorizing the slots into spatial, occupancy, and appearance components that are decoded separately providing structure to the slot representations.
  3. The authors primarily evaluate their method with ground-truth future frames fed to the IDM to predict latent actions and then use the FDM conditioned on the latent actions to reconstruct the GT future frames.
  4. The authors report visual metrics on four manipulation datasets, task success on two simulated robotics tasks, and inference runtime/GPU memory.
Critical strengths
  1. Consistent perceptual improvements over baselines. The method improves PSNR, SSIM, and LPIPS over PlaySlot and LPWM across four manipulation datasets, including real-world video. Together these results indicate the usefulness of the learned representations.

  2. Substantial inference efficiency gains. The reported evaluations show approximately 13× faster inference on BlockPush and 5× on Sketchy compared with LPWM, alongside lower GPU memory consumption. Achieving better/comparable perceptual quality with substantially reduced computational requirements gives the approach practical value.

Critical weaknesses
  1. The evaluation does not isolate improvements in object geometry and motion. The proposed representation separates spatial, occupancy, and appearance features, but the quantitative video evaluation relies on PSNR, SSIM, and LPIPS, which conflate errors in these properties. Consequently, it remains unclear whether the gains reflect more accurate object dynamics or primarily better appearance reconstruction. An object-level evaluation should compare matched objects in ground-truth and predicted videos. For example, using validated SAM3 masks and report mask overlap alongside centroid/trajectory error; position-aligned mask comparisons could further distinguish shape errors from motion errors.

  2. The advantage over holistic-slot inputs is limited, and adaptive fusion adds little further benefit. In the paper’s own Sketchy ablation, replacing the holistic-slot IDM input with directly combined factorized features improves PSNR by only 0.25 dB and reduces LPIPS by 0.002. Even with adaptive fusion, the total improvement over holistic-slot input is only 28.98 to 29.45 PSNR and 0.065 to 0.061 LPIPS, providing limited evidence of a substantial image-quality benefit from the proposed factorization. Adaptive fusion itself contributes just 0.22 dB PSNR, a 0.002 LPIPS reduction, and 1.1 percentage points of ButtonPress success, despite being presented as a core contribution. Downstream improvements over LPWM are also modest (2.7 percentage points) on ButtonPress and (2.6 points) on BlockPush; with statistical significance not established. Together, these results insufficiently justify the claimed importance of the factorized IDM inputs and adaptive fusion.

Questions for authors
  1. Baseline fairness. Were the comparisons with PlaySlot and LPWM conducted using comparable model capacities, data splits, and training budgets? Please report each method’s configuration, parameter count, slot/particle count, latent-action dimensionality, training budget including pretraining, and tuning/checkpoint-selection procedures; without these details, it is hard to judge the fairness.

  2. Limited robotics gains. Why do the proposed improvements yield only +2.7 percentage points on ButtonPress and +2.6 on BlockPush over LPWM, and are these gains statistically reliable? Provide per-seed results, establish comparable policy capacities and training budgets, and identify which failures the proposed method actually resolves.

  3. Weak evidence for the core contributions. Given that even the reported ablations show only small gains over holistic-slot inputs and direct feature combination, what evidence justifies factorization and adaptive fusion as core contributions? Are these gains reproducible across seeds and benchmarks, and do they translate into better object geometry and motion when evaluated using mask overlap and trajectory errors?

  4. Efficiency attribution. Do the timings cover the complete encoder–IDM–FDM–decoder pipeline under comparable precision and caching settings? How much of the reported advantage is explained by differences in model size, slot/particle count, or implementation, and what evidence attributes it to the proposed design?

4 comments

We sincerely thank the reviewer for the thoughtful evaluation and constructive feedback. We appreciate the valuable suggestions, which have helped us clarify our contributions and strengthen the presentation of our work. We address the concerns and questions in detail below.

W1:

We have conducted an additional object-level evaluation on Sketchy. We use SAM2.1 to obtain object masks from ground-truth and reconstructed videos. We report three complementary metrics: (1) mask IoU, measuring spatial overlap; (2) centroid ADE, measuring positional error over time; and (3) position-aligned mask IoU, measuring visible silhouette agreement after compensating for centroid displacement. FOCWM achieves better results than LPWM across all three metrics: higher Mask IoU (89.441% vs. 89.292%), lower Centroid ADE (4.876 vs. 4.958 pixels), and higher Position-aligned Mask IoU (89.765% vs. 89.632%). These results indicate that FOCWM maintains competitive object-level geometric and motion accuracy, with consistent advantages across the evaluated metrics.

W2:

We would like to clarify that factorization and adaptive fusion serve distinct but complementary purposes. First, the benefit of factorization extends beyond perceptual metrics. As shown in Table 3, replacing holistic-slot inputs with directly combined factorized features improves ButtonPress success from 75.5% to 79.2% (+3.7 percentage points), while adaptive fusion further increases it to 80.3% (+4.8 points overall). These results suggest that structured features are beneficial not only for video prediction but also for downstream behavior learning, which is a key objective of our work. Second, adaptive fusion is designed to exploit complementary object factors rather than independently account for the entire improvement. Table 6 shows that the learned weights consistently emphasize spatial information while incorporating occupancy and appearance cues across all four datasets. This supports the motivation for action-oriented feature integration. Our contribution lies in combining compact object-centric representations with explicit within-object structure for latent-action dynamics, rather than maximizing gains in individual perceptual metrics.

Q1:

First, data splits and evaluation protocols. All methods were evaluated on the same test sequences, with identical conditioning and prediction horizons.

Second, model configurations. We have added the Table 5 in appendix (A.3.4) to summarize the relevant details.

Third, training and model selection. For LPWM, we followed the official implementation and its recommended hyperparameters. For PlaySlot, we used the same hyperparameter settings as our method.

Q2:

FOCWM primarily improves object representations and latent-action dynamics rather than directly optimizing the control policy. Since downstream success also depends on policy learning, action decoding, and environment interactions, improvements in representation quality do not necessarily translate proportionally into task success.

Q3:

We would like to clarify that factorization and adaptive fusion serve distinct but complementary roles.

First, factorization provides the primary improvement. As shown in Table 3, directly combining factorized features improves ButtonPress success from 75.5% to 79.2%, while adaptive fusion further increases it to 80.3% (+4.8 percentage points overall). Figures 6–7 further demonstrate the functional specialization and controllability of the learned representations.

Second, the benefits extend beyond appearance reconstruction. Following the reviewer's suggestion, we conducted additional object-level evaluations. FOCWM outperforms LPWM in Mask IoU (89.441% vs. 89.292%), Centroid ADE (4.876 vs. 4.958 pixels), and Position-aligned IoU (89.765% vs. 89.632%). These results provide additional evidence of improved geometric and motion consistency.

Finally, regarding robustness, Table 1 demonstrates consistent prediction improvements across four benchmarks, while Table 2 reports downstream performance over three random seeds. These results support the effectiveness of the overall framework, with factorization providing the main structural contribution and adaptive fusion offering an additional refinement.

Q4:

All methods were evaluated on the same NVIDIA A100 GPU, using identical evaluation sequences, batch sizes, and prediction horizons. The efficiency gains primarily stem from the overall architectural design. Compared with LPWM, which models numerous latent particles, FOCWM maintains a compact representation with seven foreground slots and one background slot, reducing the number of entities involved in temporal and relational modeling. Compared with PlaySlot, FOCWM reconstructs objects using localized patches rather than full-image decoding, further reducing computational and memory overhead.

We sincerely appreciate your thoughtful comments and constructive suggestions regarding the effectiveness of our factorized representations, adaptive fusion, baseline fairness, and efficiency analysis.

In our responses, we have provided further explanations of the motivations behind our designs and clarified the contributions of factorization and adaptive fusion.

Additionally, we have conducted new object-level evaluations, including Mask IoU, Centroid ADE, and Position-aligned Mask IoU, to further assess the geometric and motion consistency of our approach. We have also provided additional details on baseline configurations, evaluation protocols, and computational efficiency.

We would be grateful to hear whether our responses and additional experimental results have adequately addressed your concerns.

We remain fully open to further discussion and would be happy to address any remaining questions or concerns.

Thank you again for your valuable time and thoughtful consideration.

Reviewer_i1fKOpenReview ↗

Response

I thank the authors for their detailed response.

The authors have demonstrated that the factor decomposition does in fact achieve the separation of geometry and appearance in slot representation, as shown in Fig 6 and 7. However, the benefit of the proposed method over holistic slots, as ablated on Sketchy and Button Press in table 3, show that adaptive fusion barely helps with highly marginal gains, while the factorization improves success rate by ~3-4 pts on Button press, its benefit in Sketchy is again marginal. The authors' new results on mask IoU and centroid error also show extremely marginal gains over LWPM, and it is unclear if it improves or hurts relative to holistic slots.

Given these results, i think the approach is interesting but the significance of the proposed factorization and adaptive fusion over simpler slot based alternatives has not been demonstrated downstream. I therefore would like to maintain my current score.

Thank you for the thoughtful feedback.

We have extended the ablation study in Table 3 by adding the PlaceBreadSkillet dataset, which involves more complex object motion and dynamic interactions. We have also included row-wise relative performance changes to make the contributions of different feature components and fusion strategies more explicit.

The additional results provide further evidence of the downstream benefits of our approach. Specifically, compared with the Holistic-slot baseline, directly combining all factorized features ([Spatial, Alpha, RGB]) improves PSNR by 0.75%, SSIM by 3.53%, and LPIPS by 2.54% on PlaceBreadSkillet. More importantly, adaptive fusion further improves these metrics to 23.93 PSNR, 0.90 SSIM, and 0.112 LPIPS, corresponding to improvements of 5.93%, 5.88%, and 5.08%, respectively, over the Holistic-slot baseline. This suggests that the benefits of adaptive fusion become more pronounced in scenarios involving complex object dynamics.

We also clarify that the benefits of factorization are not limited to PlaceBreadSkillet. On Sketchy, combining all factorized features improves PSNR, SSIM, and LPIPS over Holistic slots, while on ButtonPress, it improves the success rate from 75.5% to 79.2% (a gain of 3.7 percentage points). Adaptive fusion further increases the success rate to 80.3%.

Regarding Mask IoU and centroid error, we acknowledge that the improvements over LWPM are relatively modest. These metrics primarily evaluate geometric localization quality, whereas downstream task performance also depends on how effectively the learned representations support action prediction and control. We therefore view these geometric metrics as complementary evidence rather than the sole measure of the proposed method's effectiveness.

We have updated Table 3 to provide a more comprehensive evaluation and clarify the practical benefits of feature factorization and adaptive fusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.