acceptodds
Under review as a conference paper at ICLR 2027

Steering Physics in Video World Models

Abstract

Video world models learn representations that support prediction and planning, and a growing body of work finds that these representations capture physics and dynamics. For example, velocity and acceleration can be read from the representations of V-JEPA 2. However, knowing that a quantity is encoded does not tell us where it is represented, and without that we cannot steer it. In this work, our goal is to edit the representation so that the decoded video carries out a requested physical instruction, such as making a ball move faster. Using simulation datasets, we investigate velocity and acceleration, both linear and angular. We first observe that translation and rotation trace very distinct shapes in the latent space, and that rotation splits along two axes. This structure motivates two steering writers, an additive writer that steers translation, and an orientation field that steers rotation. Measured from the motion in the decoded video, the additive writer controls both the direction and the speed of translation, with direction errors below and speed correlations of in 2D and in 3D. For rotation, the orientation field sets how fast the object spins, and the requested and decoded spin rates correlate at , holding at when the object's appearance changes. The same field, fitted only on objects spinning at a constant rate, also makes objects speed up or slow down their spin, with no refitting. Finally, we use an edited representation to steer a simulated robot arm, which strikes a ball within of the requested speed on all 200 test commands. Together, these results show that a video model's latent space can serve as a two-way interface to physics, in which physical quantities can be measured, set, and used as goals to plan actions in the real world.

Primary area:
interpretability and explainable AI
Keywords:
Video World Models, Latent-Space Steering, Physical Control, World Model Interpretability, V-JEPA, Robot Control

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YwiyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The reviewer uses "human" to denote my hand-written opinions in the review and the reviewer uses "Opus" to denote Opus 5.5. The reviewer uses the incognito mode so no data will be shared to Claude.

Human:

This paper proposes a way to edit physical properties based on latent editing. This proposed approach is inference-only and is based on pre-trained V-JEPA 2 model. This paper finds that translation traces open paths in latent space, while rotation revisits a recurring orientation path at a command-dependent rate. Based on these findings, a two-writer system is proposed to edit and track two types of velocities.

Critical strengths

Human:

  1. This paper is well written, and experimental findings strongly support their model design.

  2. Experimental results are clean and inspiring.

  3. The proposed two-writer system is tested on various platforms, including a robotic simulation. The variety of tests is good.

Critical weaknesses

Human:

The reviewer thinks this paper is a borderline paper, and would update his score after reading the rebuttal.

  1. This paper is an inference only paper, built on frozen video world models. It's hard to imagine such a method would transfer to real-world examples beyond simulation.

  2. The simulation environment is a bit too easy. Although they are good for pure analysis, it's hard to see whether the proposed approach can really work in complex environments, especially when the objects are not centered or there are multiple objects.

  3. Concerns about missing baselines. This paper is like a pure study paper without compairng to prior state-of-the-arts. Although this could be a good blog, the reviewer doubts whether this contribution is enough for ICLR.

The following concern from Opus is worth noticing, and is verifited by the reviewer. Opus:

Competing steering methods are missing. Translation baselines are a random projection and a PCA projection of the oracle target edit. Rotation baselines are shuffled commands. None of these is an alternative writer that works from the command alone, as the authors acknowledge (B.4). The only real comparison is the additive vs. conditioned writer in latent space (Table 5), and B.5 tests only a single-probe version of Joseph et al. The paper would be much more convincing with comparisons against: the multi-probe steering of Joseph et al. (2026), evaluated through the same decoder; difference-of-means steering vectors indexed by command; a nonparametric baseline that retrieves the nearest training scene's latent displacement for the same (q_a, q_b); a direct nonlinear regressor (e.g., an MLP) from (H_a, q_a, q_b) to ΔH. Without these, it is hard to tell whether the specific writer designs, rather than the paired-clip supervision, are responsible for the results.

Questions for authors

Opus:

What is the correlation between commanded and decoded angular speed within each rotation direction?

How does the additive writer compare with Joseph et al.'s multi-probe steering and with difference-of-means or nearest-neighbor edit baselines, all through the same decoder?

How much does performance drop when q_a, the object center, and the initial orientation are estimated from H_a rather than taken from the simulator?

How well is the rest of the scene (background, object appearance and size) preserved after editing?

Human:

Is there a trainable approach to steer physics in video world models?

5 comments

Thank you for the specific suggestions. We ran the requested comparisons and additional tests on scene complexity, initial state, and scene preservation.

W1 Generalization beyond simulation

We keep V-JEPA 2 frozen and fit a writer, a method that changes its internal features to produce the requested motion. We now estimate starting motion directly from the video's representation, without simulator values. Accuracy remains nearly unchanged, as shown in Q3. This makes the method less dependent on the simulation setup. We will evaluate real-video steering separately.

W2 Off-center and multiple objects

We tested two independently moving objects at random positions. Our writer reduced target direction error from 42.5° to 11.7°, comparable to the nearest-neighbor results of 9.9° and 10.4°. This shows that the target can be steered when it is off-center and another object is moving nearby. The result concerns target motion, not interactions between objects.

W3 Missing baselines

We added all four suggested methods, plus a neural adapter. All use the same video pairs, decoder, and tracker. The full comparisons are in Q2. Our writer is strongest in mixed-appearance 2-D and matches the strongest methods in 3-D direction control, with better 3-D speed control.

Q1 Spin speed within each direction

SettingBoth directionsCounterclockwiseClockwise
Our writer, one look0.9420.460.59
Our writer, mixed looks0.9450.460.67
Decoder given correct motion, one look0.9770.750.76
Decoder given correct motion, mixed looks0.9790.780.75

Our writer reliably controls spin direction. The within-direction results should be read alongside the decoder's own measurement ceiling of about 0.75, even when given the correct motion. We will report these separate correlations and their uncertainty intervals.

Q2 Comparison with competing steering methods

We evaluated Joseph et al.'s multi-probe method, average-change edits, nearest-neighbor edits, a small neural network, and a neural adapter. All use the same decoder and held-out test pairs.

Method2-D ↓Mixed looks ↓3-D ↓Spin corr. ↑
No edit45.1°46.0°44.0°N/A
Joseph et al., multi-probe9.2°12.3°8.3°−0.09
Average change13.3°13.3°15.3°0.05
Nearest neighbor, 1 / 510.5° / 8.3°14.1° / 10.5°12.5° / 8.4°0.81 / 0.94
Small neural network9.6°11.1°8.4°0.61
Neural adapter24.3°18.7°22.9°−0.03
Our writer6.45°7.09°9.28°0.942
Decoder given correct motion1.91°4.16°1.93°0.977

The first three columns report decoded direction error. Lower is better.

Our writer improves on the multi-probe method by 2.8° in 2-D and 5.2° with mixed looks. In mixed looks, it also beats five-neighbor retrieval by 3.4°. These differences have 95% confidence intervals excluding zero. In 3-D, direction results are statistically comparable, but our writer has the highest speed correlation at 0.86, versus 0.75–0.81 for the strongest alternatives.

For rotation, five-neighbor retrieval matches our writer only when given the object's starting orientation. Removing orientation drops its spin correlation from 0.94 to 0.02. This supports the need to condition rotation edits on orientation.

Q3 Estimating starting state

TestSimulator inputEstimated input
2-D direction error ↓5.55°5.76°
3-D direction error ↓9.67°9.40°
Spin correlation ↑0.9420.945

Translation uses our raw-feature writer. Linear probes estimate starting speed, object center, and orientation from the video's representation. The results show no meaningful loss in steering accuracy.

Q4 Preserving the rest of the scene

We compared edited and unedited decoded videos. Our raw-feature writer preserves object area within 2%, with brightness changes of 0.001 in 2-D and −0.002 in 3-D. Background similarity is 32.8 / 31.3 dB in 2-D / 3-D, close to the correct-motion reference at 33.5 / 30.4 dB. By comparison, nearest-neighbor editing reduces 2-D object area to 84%, and the adapter reduces 3-D area to 86%.

Q5 Trainable methods

Yes. Our writers are fitted from paired videos, without training through the decoder. We also trained a small neural network and an adapter, shown in Q2. The paper further tests per-scene optimization through the decoder, reaching 5.07° acceleration direction error and 0.94 magnitude correlation. Our fitted writer can instead be reused across scenes.

Thank you again for your constructive review. We wanted to kindly follow up on our previous response, as we have added several experiments directly addressing your main concerns.

In particular, we evaluated the competing steering methods you suggested under the same experimental setup. Our method achieves 6.45° direction error in 2-D, outperforming Joseph et al.'s multi-probe baseline (9.2°), and 7.09° in mixed-appearance scenes (versus 12.3°). We also demonstrated steering in off-center, multi-object scenes and showed that estimating the initial physical state from latent representations introduces no meaningful performance degradation.

Since you mentioned being open to reassessing the paper after the rebuttal, we would greatly appreciate your thoughts on whether these new results address your main concerns. We would be happy to clarify anything further before the discussion period ends.

The reviewer's concerns have not been addressed. So he would maintain his score. Details will be given in his final review.

Reviewer_ZootOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
3
Read review
Paper summary

Whereas past work finds that physical properties are identifiable in latent space, the present work demonstrates they are precisely editable. With V-JEPA 2 as the world model, they isolate linear/angular velocity and acceleration by controlled simulation videos and examine their latent trajectories following Musa et al. 2026. The trajectories show that linear velocity and acceleration exhibit 'fan out', flower-like shapes and angular velocity and acceleration move in an overlapping bowtie-like pattern -- this implies rotation and translation exhibit distinct structure in latent space, motivating distinct steering interventions. They propose an 'additive writer' for editing linear properties and an orientation field for manipulating angular properties. Both steering methods are geometrically constrained in their constructions because they are not learned end-to-end. For each scene each steering method learns how latent representations change under different motion settings (qaq_a vs. qbq_b). They achieve high correlation with the command and low direction error on their controlled simulation videos, even in mixed-appearance 2D and rendered 3D. Finally, the steering methods successfully control simulated head-on ball striking with a paddle, kinematic robot arm, and actuated robot arm. Together, the experiments suggest latent physics steering in video world models is possible in controlled settings and may extend to actions in simulation.

Critical strengths

This work is perhaps one of the first to manipulate physics in video world models rather than simply identify it. Moreover, their investigation nicely motivates having two distinct steering interventions for translation and rotation.

The ablations are extensive. Throughout the work, the authors ablate their design choices for the writers, goal use, and decoders. Careful effort lead to their reported solution.

Critical weaknesses

The writing is confusing. I understand that nine pages restricts detail given the large volume of content, but the paper makes liberal use of self-references and bespoke terminology. In particular, frequent references to appendix material makes Section 6 difficult to read beyond its surface level takeaways. Additionally, I admit that I do not fully understand the section about 'unwrapping' the rotation to reveal its orientation (Section 4 and Figure 4). To elaborate on the bespoke terminology point, the paper introduces the terms 'writers', 'conditioned writer', 'scene', 'motion setting', 'readout', 'edit', 'true edit', 'predicted edit', 'target decode', and more. Perhaps this is unavoidable to some extent, but truly understanding the content requires considerable time and effort. Generally, the writing style demands frequent 'jumping' around different sections to fully grasp the information; I believe 'linearizing' the content would greatly improve the work.

The authors examine only one use case of their work and only up to two spatial dimensions. It remains unclear whether the method supports interactions beyond the ball pushing experiment, even in simulation. The work is encouraging as a proof of concept but it does not fully realize its ultimate purpose - if physics are steerable for video models, the most exciting application is for non-simulated video. The authors acknowledge this as a limitation, but even slight evidence that the method generalizes to arbitrary video would be exciting.

Questions for authors

Can the authors further elaborate why custom decoders are necessary? On pp. 3, the text motivates these decoders by asserting, "pixel losses alone... decodes to a ball moving about four times too slowly" but I cannot find the paper section that shows evidence for this claim. On pp. 7, the text reads, "Steering also works with ViT-H and ViT-g encoders and their own decoders, but larger encoders do not reliably improve control". As I understand it, the linked section in appendix F3 ablates different encoders but keeps the decoders fixed to the bespoke ones in this work. Am I reading this wrong?

On pp. 15, the text reads, "Each steering decoder ... predicts each frame through a patch-level output head. ... Its training loss combines ten terms, namely a robust pixel loss, a foreground loss, SSIM, MS-SSIM, temporal consistency, state, trajectory, and velocity losses, a per-frame centroid loss, and a spread penalty. A pixel loss on its own can be minimized by a blurred, nearly static reconstruction, and the per-frame centroid term, which carries the largest weight, rules that out by requiring the object’s position to be correct in every frame." Could these properties be reliably estimated for non-simulated video? The limitations section identifies using the original decoders as a future direction and I am curious to understand its feasibility.

Are the writers trained per single-appearance, mixed-appearance, and rendered 3D? My understanding is that each scene has a distinct writer. Is each writer trained distinctly for each visual style depicted in Figure 5?

Can the authors explain their choice of layers 6,12,18,23 for ViT-L, 8,16,24,31 for ViT-H, and 10,20,30,38 for ViT-g? Several papers in the related work section report that different layers encode physics more strongly than others, is this choice informed by that?

One comment: I believe you might be missing a closely related work "The Invisible Hand of Physics - When Video Diffusion Models Know More Than They Show (2026)" by Esmati et al. The paper demonstrates physical plausibility is linearly decodable from diffusion transformers and uses V-JEPA as a baseline. The paper is closely related to the work you do cite, "Interpreting Physics in Video World Models (2026)" by Joseph et al. I am not affiliated with either aforementioned work in any capacity.

3 comments

Thank you for your helpful comments. We will make the paper easier to follow and clarify the scope of our results.

W1 Writing and explanation of rotation

We will present the paper in a clearer order, moving from the patterns found in the model to our editing methods and their evaluation. We will define terms once and make Section 6 readable without frequent appendix references.

We will also explain unwrapping more simply. An angle normally jumps from +180° to −180° after a full turn. Removing that jump lets us track continuous rotation. We will add a figure showing why the edit must depend on the object's current orientation.

W2 More complex scenes and real video

We added a test with two objects moving from random positions. Our writer, which changes the model's internal features to produce a requested motion, reduced the off-center target's direction error from 42.5° to 11.7°. This tests steering of one object while another is moving nearby. It does not test collisions.

Our perspective-rendered 3-D setting achieves 9.28° direction error and 0.861 speed correlation. We also test three robot systems, including an actuated Franka arm. We can now estimate initial motion from the video representation without simulator inputs and with no meaningful loss of accuracy. Q2 explains how we plan to extend the measurements to real video.

Q1 Decoder losses and encoder scaling

We use custom decoders because steering must be measured in reconstructed video. We trained three decoders with identical data and training budgets, changing only the loss.

Decoder lossRendered / true speed ↑Direction error ↓
Pixels only0.11Not measurable
Without frame-position term0.4358.5°
Full loss0.8113.6°

Pixel loss alone produces nearly static motion. We will add these results and correct the statement on page 3. Under this budget, rendered speed was about nine times too slow, not four.

Each encoder also has its own trained decoder and fitted writer. We do not reuse the ViT-L decoder for larger models.

MeasureViT-LViT-HViT-g
2-D direction error ↓5.8°11.4°6.9°
Acceleration direction error ↓10.2°18.5°19.8°
Spin correlation ↑0.9420.6320.535
Spin correlation with correct motion ↑0.9770.8860.831

These results show that larger encoders do not consistently improve steering. The final row also shows differences in how accurately their decoders reconstruct spin. We will clarify this setup in Appendix F.3.

Q2 Decoder losses on real video

Pixel and image-similarity losses need no labels. Tracking tools such as CoTracker or SAM 2 could supply object positions and motion for the remaining losses. Our decoder tests show why accurate position tracking matters.

We have also built an initial tracking and labeling step for real Physics-IQ clips. We will describe how this supports future real-video evaluation.

Q3 Training across appearances

We do not fit a different writer for each appearance in Figure 5. A rotation writer trained on one look also works on mixed looks without refitting, with correlations of 0.942 and 0.945. Translation uses one writer fitted across mixed appearances. The 3-D renderer has its own writer and decoder. We will clarify this near Figure 5.

Q4 Choice of layers

We select layers at similar relative depths for a fair comparison across encoder sizes. ViT-L uses 6, 12, 18, 23, ViT-H uses 8, 16, 24, 31, and ViT-g uses 10, 20, 30, 38.

Our tests show that reading a physical property from a layer is different from changing it. In 3-D, layers 6 and 12 read velocity with R² ≥ 0.991, but editing layer 23 has a much larger effect on decoded motion. This motivates editing all four layers together.

C1 Related work

We will cite Esmati et al. (2026) alongside Joseph et al. (2026) and distinguish reading physical properties from deliberately changing them in decoded video.

Thank you again for your detailed and constructive review. We wanted to follow up on our previous response, where we provided additional decoder ablations, clarified the encoder-scaling experiments and training across appearances, and outlined how we will improve the presentation of the rotation analysis and Section 6.

We also added experiments on multi-object scenes and steering without simulator-provided initial states, addressing some of your concerns about the scope of the evaluation.

We would greatly appreciate any further feedback on whether our clarifications have resolved your main questions. We are happy to address any remaining issues before the discussion period ends.

I appreciate the authors’ careful response to my questions. I will address each.

RE: W1

The discontinuity at the angle makes sense, thank you for explaining this.

Re: W2

That is encouraging. Collisions are difficult, so this is understandable. Is this test of two objects a 2D render?

The perspective rendered setting is also encouraging. Are these results for a single video instance, or are these averages? I am assuming a single instance but clarification in the manuscript would be helpful.

RE: Q1

This is a compelling result. I feel this supports the custom loss function and alleviates my concerns.

RE: Q2

“Could supply object positions and motion for the remaining losses” is speculative. I recognize that this is a large engineering effort but I maintain that results on real videos would greatly strengthen the work. A single instance of the setup working end to end on real videos would raise my score to a strong accept.

RE: Q3

Thank you for clarifying this. This resolves my misunderstood limitation that each writer is coupled to a specific set of inputs.

RE: Q4

This is a sensible choice. I would encourage the authors to add this reasoning to the appendix, or anywhere deemed appropriate, if they have not already.


Had I given the paper a weak reject, I would have raised it to weak accept. I am unfortunately unable to raise my overall score based on these results. However, conditioned on the authors improving the presentation as stated and adding the clarifications, I raise my score on presentation from a 1 to a 2. I appreciate the time and effort the authors have invested in the rebuttal period, and remain open to additional results.

Reviewer_d7VkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates post-hoc control of physical motion in the latent space of a pretrained, frozen V-JEPA 2 model. The authors first analyze the latent geometry of translational and rotational motion, finding distinct structures that motivate two editing methods: an additive writer for translation and an orientation-conditioned field for rotation. Experiments on synthetic videos show that these methods can modify decoded velocity and acceleration without changing the pretrained encoder. The authors further demonstrate that edited latent representations can serve as goals for a simulated robot striking task.

Critical strengths
  • The analysis of translational and rotational latent trajectories provides a reasonable motivation for designing different editing operators. The comparison between additive and configuration-conditioned writers across six motion types supports this design choice, making the method more interpretable than a generic latent editing approach.
  • Rather than only measuring whether physical properties can be recovered from edited representations, the paper evaluates the actual motion in decoded videos using object tracking. This is a meaningful step toward demonstrating physical controllability, since changes in probe predictions do not necessarily correspond to changes in decoded motion.
Critical weaknesses

Unclear advantage over direct physical conditioning. The paper aims to introduce post-hoc controllability to a pretrained model through latent editing. However, the benefits of this approach over simpler alternatives, such as direct physical conditioning with lightweight adapters, remain unclear. Additional comparisons are needed to assess whether the proposed latent editing formulation provides additional advantages in controllability, generalization, or training efficiency.

Limited evidence for generalizable physical control. The experiments rely on simplified synthetic scenes and task-specific decoders trained with explicit physical supervision. This makes it difficult to determine how much of the observed controllability originates from the pretrained V-JEPA 2 representations rather than the additional supervised components. Furthermore, the robot control experiment is relatively simple, and directly conditioning the policy on numeric physical goals achieves the same success rate as the latent-goal approach. These limitations weaken the evidence for the broader usefulness of the proposed control interface.

Questions for authors
  • What advantages does latent editing offer over directly conditioning a frozen pretrained model on physical quantities through lightweight adapters? Could the authors provide a comparison under similar training data and parameter budgets?

  • Does latent editing genuinely control physical dynamics, or does it primarily modify the visible motion trajectories? Could the authors evaluate counterfactual physical interactions, such as changing an object's velocity before a collision and testing whether the resulting motion of both objects remains physically consistent?

  • Given that the latent-goal policy and the numeric-goal policy both achieve 100% success in the robot striking experiment, what additional benefit does the latent-goal representation provide?

3 comments

Thank you for your thoughtful review. We ran new comparisons and controls to test where our method works and why.

W1 Advantage over direct physical conditioning

A writer is a method that changes the video model's internal representation to produce a requested motion. We compared our writer with a neural adapter trained on the same video pairs. Our writer controls direction and speed more accurately, with far fewer trained command weights. Full results appear in Q1.

Our edits can also be used by different models that already read V-JEPA 2 representations. We show this with both a video decoder and a robot control system.

W2 Evidence that control comes from the pretrained model

We repeated our 2-D velocity test with a randomly initialized encoder. Both encoders used the same architecture and training budget. Each had its own trained decoder and writer.

EncoderNo edit ↓After editing ↓Correct motion decoded ↓Share of possible gain
Pretrained V-JEPA 244.7°13.6° [10.4°, 17.1°]9.1°87%
Random weights44.6°40.5° [30.5°, 51.8°]18.2°16%

Direction error on held-out scenes. Both runs use the raw-feature writer, without feature scaling or magnitude axes.

Editing greatly improves motion with the pretrained encoder, but has little effect with random weights. The decoder is trained only on unedited video and never receives a steering command. This supports the role of the pretrained representation in steering.

We also tested an off-center target alongside a second moving object. Direction error fell from 42.5° to 11.7°. This result is discussed further in our response to Ywiy W2. The robot experiment is addressed in Q3.

Q1 Comparison with a neural adapter

We trained a neural adapter using the same video pairs as our writer. Both were tested through the same decoder and object tracker.

MeasureNeural adapterOur writer
2-D direction error ↓24.3°6.45°
3-D direction error ↓22.9°9.28°
2-D speed correlation ↑0.240.75
3-D speed correlation ↑0.350.86
Spin-speed correlation ↑−0.030.942
Trained command weights1.35 million572 in 2-D, 988 in 3-D

Our writer fits in seconds on a CPU. It also uses fixed directions found from the training data, so the weight counts are not a complete comparison of model size.

We tested how much training data each method needs. Because each setting requires a new fit, this test uses the raw-feature version of our writer.

Training scenes2550125500
Neural adapter25.9°24.7°26.0°24.3°
Small neural network14.6°10.0°10.8°9.6°
Our raw-feature writer13.0°9.4°6.5°5.55°

2-D direction error on held-out scenes. Lower is better.

Our writer is at least as accurate as these alternatives at every tested data size. The advantage is clearest from 125 scenes onward. At 25 and 50 scenes, the uncertainty intervals overlap with the small neural network.

Q2 Physical dynamics and collisions

Our acceleration and rotation tests show that an edit changes how motion develops over time, rather than simply moving an object to a new position.

We also implemented the suggested two-ball collision setting. The current decoder does not resolve both objects and their speeds well enough for a reliable analysis. We therefore make no claim about collision control and plan to evaluate this separately with a decoder suited to that task.

Q3 Benefit of an edited representation as a robot goal

The robot test asks whether a robot can follow a goal stored in an edited video representation. On the actuated Franka arm, using either the edited representation or a numerical goal gives 100% success at 2% tolerance over 200 commands. Shuffling the goals lowers success to 28%, showing that the robot follows the specific goal it receives.

This shows that a robot can use an edited representation as its goal. The result establishes a useful goal format for models that use V-JEPA 2 features. We will clarify the comparison with numerical goals in Section 6.1.

Thank you again for your thoughtful feedback. We wanted to briefly follow up on our previous response, particularly regarding your questions about the advantages of latent editing and the contribution of pretrained representations.

We added a comparison against a trainable neural adapter using the same video pairs and decoder. Our method achieves 6.45° versus 24.3° direction error in 2-D, with substantially better speed control. We also compared against a randomly initialized encoder under matched training conditions: steering reduces direction error from 44.7° to 13.6° with pretrained V-JEPA 2, but only from 44.6° to 40.5° with random weights.

We believe these results substantially strengthen the evidence for our main contribution. We would appreciate hearing whether these additional experiments address your primary concerns, or if there are any outstanding questions we could clarify before the discussion period closes.

Thank you for your detailed responses and additional experiments. My main concerns have been largely addressed. The new comparisons demonstrate the advantages of the proposed structured writers over neural network-based latent editing, while the random-initialization experiments provide stronger evidence that the method leverages meaningful representations learned during V-JEPA 2 pretraining. Furthermore, the acceleration and rotation experiments provide evidence that the edits affect temporal motion dynamics rather than merely changing object positions.

Nevertheless, the experimental settings remain relatively simple, and the current writer designs primarily exploit the geometric structures of translation and rotation. Extending the approach to more complex physical interactions, diverse scenes, and broader downstream applications remains an important direction for future work.

Overall, considering the strengths of the paper and the additional evidence provided in the rebuttal, I am inclined to revise my rating to 3: Weak Accept.

Comments on the paper

1 comment

We thank all reviewers for their thoughtful feedback. Their questions helped us sharpen the paper’s central claim and clarify what our experiments establish.

In controlled settings, the distinct structures of translation and rotation in frozen V-JEPA 2 representations guide edits that change how objects move in decoded video. In response to the reviews, we ran three sets of experiments.

Compared against five alternative steering methods. We evaluated all four suggested baselines and a trained neural adapter under a shared evaluation setup. Our method reduced direction error relative to Joseph et al.’s multi-probe baseline from 9.2° to 6.45° in 2D and from 12.3° to 7.09° across mixed appearances, with 95% confidence intervals excluding zero for both improvements. In 3D, direction accuracy was comparable to the strongest baselines, and our method achieved the highest speed correlation (0.86).

Tested the contribution of pretraining and the decoder. Under matched training conditions, steering reduced direction error from 44.7° to 13.6° with pretrained V-JEPA 2 and from 44.6° to 40.5° with the same architecture at random initialization. This supports the role of the learned representation. A separate decoder ablation showed why the motion-aware loss matters for evaluation. Pixel-only training recovered just 11% of true motion speed, compared with 81% using the full loss.

Tested steering in less constrained scenes. In a scene with two independently moving objects, steering an off-center target reduced its direction error from 42.5° to 11.7°. Estimating the initial state directly from the video representation yielded similar accuracy to using simulator-provided values. We also measured scene preservation and reported spin-speed correlations separately for clockwise and counterclockwise motion. Additional evaluations.

We will incorporate these results and revise the presentation to make the link between the representation analysis, the editing methods, and the decoded motion easier to follow. We will also clarify that the robot experiment shows that a policy can use an edited representation as a goal, with performance matching numerical goals in that task. End-to-end steering on real video and reliable control of collisions remain open questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.