- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The reviewer uses "human" to denote my hand-written opinions in the review and the reviewer uses "Opus" to denote Opus 5.5. The reviewer uses the incognito mode so no data will be shared to Claude.
Human:
This paper proposes a way to edit physical properties based on latent editing. This proposed approach is inference-only and is based on pre-trained V-JEPA 2 model. This paper finds that translation traces open paths in latent space, while rotation revisits a recurring orientation path at a command-dependent rate. Based on these findings, a two-writer system is proposed to edit and track two types of velocities.
Critical strengths
Human:
-
This paper is well written, and experimental findings strongly support their model design.
-
Experimental results are clean and inspiring.
-
The proposed two-writer system is tested on various platforms, including a robotic simulation. The variety of tests is good.
Critical weaknesses
Human:
The reviewer thinks this paper is a borderline paper, and would update his score after reading the rebuttal.
-
This paper is an inference only paper, built on frozen video world models. It's hard to imagine such a method would transfer to real-world examples beyond simulation.
-
The simulation environment is a bit too easy. Although they are good for pure analysis, it's hard to see whether the proposed approach can really work in complex environments, especially when the objects are not centered or there are multiple objects.
-
Concerns about missing baselines. This paper is like a pure study paper without compairng to prior state-of-the-arts. Although this could be a good blog, the reviewer doubts whether this contribution is enough for ICLR.
The following concern from Opus is worth noticing, and is verifited by the reviewer. Opus:
Competing steering methods are missing. Translation baselines are a random projection and a PCA projection of the oracle target edit. Rotation baselines are shuffled commands. None of these is an alternative writer that works from the command alone, as the authors acknowledge (B.4). The only real comparison is the additive vs. conditioned writer in latent space (Table 5), and B.5 tests only a single-probe version of Joseph et al. The paper would be much more convincing with comparisons against: the multi-probe steering of Joseph et al. (2026), evaluated through the same decoder; difference-of-means steering vectors indexed by command; a nonparametric baseline that retrieves the nearest training scene's latent displacement for the same (q_a, q_b); a direct nonlinear regressor (e.g., an MLP) from (H_a, q_a, q_b) to ΔH. Without these, it is hard to tell whether the specific writer designs, rather than the paired-clip supervision, are responsible for the results.
Questions for authors
Opus:
What is the correlation between commanded and decoded angular speed within each rotation direction?
How does the additive writer compare with Joseph et al.'s multi-probe steering and with difference-of-means or nearest-neighbor edit baselines, all through the same decoder?
How much does performance drop when q_a, the object center, and the initial orientation are estimated from H_a rather than taken from the simulator?
How well is the rest of the scene (background, object appearance and size) preserved after editing?
Human:
Is there a trainable approach to steer physics in video world models?
5 comments
Thank you again for your constructive review. We wanted to kindly follow up on our previous response, as we have added several experiments directly addressing your main concerns.
In particular, we evaluated the competing steering methods you suggested under the same experimental setup. Our method achieves 6.45° direction error in 2-D, outperforming Joseph et al.'s multi-probe baseline (9.2°), and 7.09° in mixed-appearance scenes (versus 12.3°). We also demonstrated steering in off-center, multi-object scenes and showed that estimating the initial physical state from latent representations introduces no meaningful performance degradation.
Since you mentioned being open to reassessing the paper after the rebuttal, we would greatly appreciate your thoughts on whether these new results address your main concerns. We would be happy to clarify anything further before the discussion period ends.
The reviewer's concerns have not been addressed. So he would maintain his score. Details will be given in his final review.
Thank you for the specific suggestions. We ran the requested comparisons and additional tests on scene complexity, initial state, and scene preservation.
W1 Generalization beyond simulation
We keep V-JEPA 2 frozen and fit a writer, a method that changes its internal features to produce the requested motion. We now estimate starting motion directly from the video's representation, without simulator values. Accuracy remains nearly unchanged, as shown in Q3. This makes the method less dependent on the simulation setup. We will evaluate real-video steering separately.
W2 Off-center and multiple objects
We tested two independently moving objects at random positions. Our writer reduced target direction error from 42.5° to 11.7°, comparable to the nearest-neighbor results of 9.9° and 10.4°. This shows that the target can be steered when it is off-center and another object is moving nearby. The result concerns target motion, not interactions between objects.
W3 Missing baselines
We added all four suggested methods, plus a neural adapter. All use the same video pairs, decoder, and tracker. The full comparisons are in Q2. Our writer is strongest in mixed-appearance 2-D and matches the strongest methods in 3-D direction control, with better 3-D speed control.
Q1 Spin speed within each direction
Our writer reliably controls spin direction. The within-direction results should be read alongside the decoder's own measurement ceiling of about 0.75, even when given the correct motion. We will report these separate correlations and their uncertainty intervals.
Q2 Comparison with competing steering methods
We evaluated Joseph et al.'s multi-probe method, average-change edits, nearest-neighbor edits, a small neural network, and a neural adapter. All use the same decoder and held-out test pairs.
The first three columns report decoded direction error. Lower is better.
Our writer improves on the multi-probe method by 2.8° in 2-D and 5.2° with mixed looks. In mixed looks, it also beats five-neighbor retrieval by 3.4°. These differences have 95% confidence intervals excluding zero. In 3-D, direction results are statistically comparable, but our writer has the highest speed correlation at 0.86, versus 0.75–0.81 for the strongest alternatives.
For rotation, five-neighbor retrieval matches our writer only when given the object's starting orientation. Removing orientation drops its spin correlation from 0.94 to 0.02. This supports the need to condition rotation edits on orientation.
Q3 Estimating starting state
Translation uses our raw-feature writer. Linear probes estimate starting speed, object center, and orientation from the video's representation. The results show no meaningful loss in steering accuracy.
Q4 Preserving the rest of the scene
We compared edited and unedited decoded videos. Our raw-feature writer preserves object area within 2%, with brightness changes of 0.001 in 2-D and −0.002 in 3-D. Background similarity is 32.8 / 31.3 dB in 2-D / 3-D, close to the correct-motion reference at 33.5 / 30.4 dB. By comparison, nearest-neighbor editing reduces 2-D object area to 84%, and the adapter reduces 3-D area to 86%.
Q5 Trainable methods
Yes. Our writers are fitted from paired videos, without training through the decoder. We also trained a small neural network and an adapter, shown in Q2. The paper further tests per-scene optimization through the decoder, reaching 5.07° acceleration direction error and 0.94 magnitude correlation. Our fitted writer can instead be reused across scenes.