- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Action Jets, a local polynomial representation for efficiently predicting the outcomes of candidate actions. The method first probes a pretrained action-conditioned generator to estimate action Jacobians, then trains a geometry encoder to predict these first-order responses from visual context. A correction head trained on recorded transitions supplies linear corrections and higher-order coefficients. At inference time, multiple candidate actions can share the extracted coefficients, allowing their future latent states to be evaluated through polynomial operations rather than repeated generative inference.
The paper evaluates the approach on Kubric, BridgeData V2, and DL3DV, studying prediction quality, computational efficiency, cross-generator tangent agreement, and offline candidate-action selection. It also introduces a coefficient-based radius intended to characterize the useful range of the local approximation.
Critical strengths
-
An interesting design for action-conditioned world models. Predicting reusable, state-dependent coefficients is a compelling way to exploit the shared visual context across candidate actions. This offers a useful perspective on improving world models for search: changing the representation of the predictor, rather than only accelerating individual generations.
-
A well-motivated combination of supervision sources. Using a generator to obtain local action responses, while learning corrections from recorded transitions, is an appealing approach. It allows the model to benefit from generator-provided supervision while retaining the opportunity to correct its inaccuracies using real observations.
-
Broad evaluation with informative controls and limitations. The experiments cover synthetic scenes, robot manipulation, and camera motion. Copy-frame controls, probe-scale analyses, decoder ablations, and cross-generator comparisons help characterize the method. I also appreciate the reporting of negative results, including disagreement between generators on BridgeData and the source generator’s advantage at longer horizons and in policy consistency.
-
Promising computational advantages for candidate evaluation. The reported results suggest that sharing state-dependent computation can substantially reduce the cost of evaluating multiple actions. This is relevant to settings where the number of candidates that a planner can consider is limited by world-model inference cost.
Critical weaknesses
-
The interpretation of the validity radius could be clearer. Section 3.4 defines the radius through the relative magnitudes of linear and higher-order terms. This is intuitive, and the empirical validation is helpful, but its relationship to prediction error against real observations remains somewhat unclear. The paper would benefit from explicitly distinguishing an empirical reliability indicator from a region with stronger accuracy guarantees.
-
The independent contribution of generator-derived supervision is not fully isolated. The combination of generator-derived first-order responses and real-data corrections is interesting. However, it is difficult to determine how much the derivative supervision contributes compared with training the same polynomial predictor directly on recorded transitions. Such a baseline would help establish the value of this component.
-
The evidence for planning utility is currently limited to offline evaluation. The candidate-selection experiments demonstrate potential under a computational budget, but prediction metrics and action error relative to demonstrations do not directly establish improved task success. A small closed-loop experiment would strengthen this aspect. Alternatively, the conclusions could more explicitly distinguish the demonstrated offline selection capability from prospective control applications.
Questions for authors
-
The interpretation of the validity radius could be clearer. Section 3.4 defines the radius through the relative magnitudes of linear and higher-order terms. This is intuitive, and the empirical validation is helpful, but its relationship to prediction error against real observations remains somewhat unclear. The paper would benefit from explicitly distinguishing an empirical reliability indicator from a region with stronger accuracy guarantees.
-
The independent contribution of generator-derived supervision is not fully isolated. The combination of generator-derived first-order responses and real-data corrections is interesting. However, it is difficult to determine how much the derivative supervision contributes compared with training the same polynomial predictor directly on recorded transitions. Such a baseline would help establish the value of this component.
-
The evidence for planning utility is currently limited to offline evaluation. The candidate-selection experiments demonstrate potential under a computational budget, but prediction metrics and action error relative to demonstrations do not directly establish improved task success. A small closed-loop experiment would strengthen this aspect. Alternatively, the conclusions could more explicitly distinguish the demonstrated offline selection capability from prospective control applications.
2 comments
Thank you for your reply. Since our original score was already quite high, I'll keep the score.
Thank you for the positive and detailed review.
1. The radius. We agree and adopt this framing. r1∗ compares the Action Jet's own terms, so it is an empirical reliability indicator rather than a region with accuracy guarantees. Its relation to real error is the calibration in Sec. 4.3 and App. B.4, summarized in the general response. The revision states this in the abstract, introduction, Sec. 3.4, Sec. 4.3 and the conclusion.
2. Derivative supervision. Recorded actions co-vary across axes, so regressing image change on them alone entangles the per-axis fields, while generator probes answer isolated single-axis queries at any state (Sec. 3, App. C). Planning mostly queries actions that were not demonstrated, and Bridge has no ground truth for them. The matched comparison in the paper is the IRASim-basis and WorldGym-basis pair, which share the architecture, corrector recipe and real transitions and differ only in the probed generator. Fitting the corrector to the same real transitions does not repair the WorldGym tangent (anchoring +0.249 against −0.011, App. Table 17), so the probed tangent decides whether the Action Jet anchors on real data. On Kubric, with the encoder, latent and decoder fixed, the probed tangent alone, the 2-jet and the 3-jet reach 22.80, 25.33 and 26.27 dB PSNR at gap 4 (App. Tables 9 and 25). We will add the comparison you suggest, the same polynomial trained on recorded transitions alone, on BridgeData V2 in the camera-ready version.
3. Offline selection and closed loop. The general response reports the closed-loop experiment on PushT (Sec. 4.4, App. F). With the same CEM planner the Action Jet plans 22× faster than its source generator, succeeds on 84.7% of 300 episodes against 56.7% for the generator at equal planning time, and ties LeWM. The conclusion now separates the offline selection on Bridge from closed-loop control on PushT.
We hope these answers and the closed-loop results address your points.