acceptodds
Under review as a conference paper at ICLR 2027

FORCE: Factorized Object Relations for Counterfactual Editing in Video World Models

Abstract

World models predict how scenes evolve, but they are difficult to interrogate: given a model that forecasts a video, one cannot easily ask what would have hap- pened had a particular interaction between two objects been different. We present Factorized Object Relations for Counterfactual Editing (FORCE), a world model that infers pairwise interactions from raw video without supervision and repre- sents each as an explicit latent per object pair, so that a relation can be removed, changed, transferred or introduced directly. Objects are extracted by slot attention over frozen self-supervised features; a relation encoder infers each pair latent from a short observation window, and an interaction head turns it into a force. The latent is held fixed across the rollout, so an edit persists rather than being recomputed, and every edit acts on the model’s own quantities with no mapping to simulator parameters. We evaluate in a simulated environment where the true outcome of an edit is available, scoring the change each edit induces against the change the simulator produces under the same operation. On held-out clips, FORCE’s ed- its made from video agree with the simulator’s response at 0.19 to 0.37 on our effect-based score; trained and scored on exact object positions instead, the same architecture reaches 0.57 to 0.85. The gap therefore lies not in the learned rela- tional dynamics but in perception: what bounds an edit is how precisely objects can be located from video, and the precision editing needs lies just beyond what current object-centric perception delivers. Closing that gap is a concrete target for making counterfactual world models work from video.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, object-centric learning, counterfactual reasoning, relational reasoning, slot attention, video prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bbuWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an object-centric world model designed to support explicit counterfactual interventions on pairwise interactions inferred from video. It proposes to encode each object pair's interaction into a persistent latent representation, which is held fixed throughout the rollout, and supports four types of interventions without requiring a learned mapping to the simulator's ground-truth parameters. It also proposes an effect-based evaluation metric that compares the change between edited and factual rollouts against the corresponding change produced by the simulator. Experiments on a controlled two-dimensional spring system show that the proposed model achieves non-trivial counterfactual editing performance from video.

Critical strengths
  1. The paper addresses a meaningful limitation of existing object-centric world models that interactions are often recomputed from the current object states at each step, making it difficult to intervene on an individual relation. By introducing a persistent pairwise latent representation, this paper provides a direct mechanism for manipulating interactions independently of object states.
  2. The proposed evaluation that measure the difference between edited and factual predictions is well motivated. Ordinary rollout errors can obscure whether a model correctly captures the effect of an intervention, and the effect-based evaluation provides a useful perspective for assessing counterfactual world models and distinguishing intervention quality from general forecasting accuracy.
Critical weaknesses
  1. A central motivation is that an explicit learned relation representation enables more meaningful counterfactual editing. However, Table 2 shows that distance-only selection performs comparably to or better than FORCE. It raises a question that how much of the observed performance comes from the persistent relation representation, and how much comes from the assumed spring dynamics and the hand-designed editing rules? Although the distance-based selector cannot predict an edited rollout, the current evidence does not clearly isolate the value added by the learned relation encoder.
  2. The evaluation uses a small synthetic environment with four objects, two persistent spring interactions, central forces, and a fixed integration rule. This setting substantially simplifies the challenges of relational reasoning, and the paper explicitly excludes transient interactions such as collisions. Consequently, it remains unclear whether the proposed approach generalizes to more diverse physical relations in real world scenarios.
  3. The paper attributes much of the performance gap to noisy object localization and initial velocity estimation. The controlled noise sweep provides evidence that the model is sensitive to state-estimation errors. However, the reported threshold is obtained in a particular synthetic environment, with a particular object scale, image resolution, dynamics and model configuration. It is unclear how transferable this threshold is to other settings.
Questions for authors

1.What is the specific benefit of the learned relation encoder over simpler relation-selection strategies? Can the authors provide experiments that isolate the contribution of the learned relation representation beyond the fixed interaction dynamics and the hand-designed editing rules?

  1. How well does the method generalize beyond the current spring-system environment? Can the authors evaluate it on environments with different interaction types, such as repulsive forces, frictional interactions, collisions, or more complex multi-object constraints? It would also be useful to evaluate varying numbers of objects, denser interaction graphs, and held-out dynamical parameter regimes.

  2. Can the evaluation account for both the magnitude and the temporal evolution of counterfactual effects? Since the current metric relies on cosine similarity, it may not adequately penalize errors in the magnitude of predicted changes.

  3. How robust are the conclusions about perception errors across different experimental settings? The paper identifies a performance transition at specific noise scale and attributes a part of the remaining gap to initial velocity estimation. How does this threshold vary with image resolution, object size, observation-window length, and dynamical regime?

  4. The paper assumes that relations persist over time. Could the authors clarify how the current representation behaves when a relation becomes active or inactive during a rollout, or when the interaction depends on contact and geometry?

Reviewer_vHrxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FORCE, an object-centric world model that represents pairwise interactions as explicit, editable relation latents inferred from video. The framework supports removing, changing, transferring, and introducing interactions without accessing simulator parameters during editing. Experiments in a spring-based simulated environment evaluate counterfactual effects and analyze how object perception errors limit editing accuracy.

Critical strengths
  • The paper addresses an interesting problem of editing object interactions directly within a relationally factored world model. The factorized relation representation provides an intuitive mechanism for persistent counterfactual interventions.
  • The experiments systematically investigate how object-state estimation errors affect counterfactual prediction, including controlled noise experiments and ground-truth state ablations.
Critical weaknesses
  • The proposed framework shares substantial similarities with prior relational world models (e.g., V-CDN), which already support counterfactual prediction through relation editing. While FORCE enables direct editing of learned force coefficients, this relies on a predefined force formulation and simulator-matched dynamics. It remains unclear what substantial methodological advantages FORCE offers over prior approaches.
  • The main evaluation metric measures directional agreement between predicted and ground-truth counterfactual changes but does not penalize errors in their magnitudes. Although Figure 4 analyzes performance across different effect magnitudes, it remains unclear whether FORCE accurately predicts the scale of the interventions' effects.
Questions for authors
  • Could the authors clarify the main methodological advantages of FORCE over prior relational counterfactual models such as V-CDN, beyond directly editing force coefficients?
  • Could the authors provide magnitude-sensitive evaluation metrics to assess whether FORCE accurately predicts the scale of counterfactual effects, in addition to their directions?
Reviewer_sooAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds a world model that learns pairwise relations between objects from video. Objects come from slot attention over frozen DINOv2 features. For each pair, a relation encoder reads 16 frames of motion and outputs one latent. This latent is held fixed during the rollout. A linear head turns it into two spring coefficients, and a fixed integrator that matches the simulator moves the state forward. Four edits remove, change, transfer or introduce a relation by writing these coefficients. Each edit is scored by the cosine between the change the model predicts and the change the simulator produces. The test uses four objects with two random springs in a 2D box.

Critical strengths
  1. The core design is simple and works. The relation latent for each pair is held fixed after it is inferred. So an edit stays in effect for the whole rollout. It also needs no mapping to the simulator’s parameters.
  2. The evaluation and the error analysis are careful. The score compares the change an edit causes, and the paper uses several controls. The paper also finds the main limit on performance. It is how precisely objects are located, and the start velocity matters most.
Critical weaknesses
  1. The importance and practical use of the problem are not clear. The paper mainly argues that world models should be open to counterfactual questions. But it does not say in which real setting one needs to change the relation between two objects. The experiments also do not show that such edits help a downstream task, such as planning or control. So it is hard to judge the practical value of this ability. I suggest adding a simple downstream task, such as planning with the edited rollouts. Another option is to give concrete use cases in the introduction.
  2. The setting is fairly simple, and generalization is not tested. All experiments use one setting with four objects, two springs, no damping and synthetic shapes. The held out clips come from the same setting, and there is no real video. On the method side, the force form and the integrator follow the simulator. The model mainly learns two spring coefficients. So the claim that the dynamics are not the bottleneck partly comes from the design itself. The force can only act along the line between two centers. Hinges, ropes and contacts are outside this scope. The 1.7 pixel threshold may also depend on object size, speed and spring strength. I suggest testing generalization by changing the number of objects, the number of springs, the spring strength and the damping. It would also help to replace the spring form with a learned force function and to compare with NRI on the same slot tracks.
Questions for authors
  1. Can you show one downstream task that benefits from editing a relation, such as planning or control?
  2. How does FORCE perform from video if the spring form is replaced by a learned force function? How does NRI perform on the same slot tracks?
  3. Do the results and the 1.7 pixel threshold still hold with more objects, more springs, damping or a different object size?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.