acceptodds
Under review as a conference paper at ICLR 2027

Action Jets: Charting Visual Action Manifolds with Generative Oracles

Abstract

World models are a powerful means of reasoning and planning for a variety of applications including robotics. However, many of these world models such as video generation models are high dimensional and computationally expensive which limits the number of action candidates that can be evaluated by a planner. We show that this cost is avoidable for planners over short horizons which only local information with respect to the current state. For a fixed state, the frames reachable by varying a low-dimensional action, a camera movement or an end-effector command, lie on a low-dimensional manifold. We estimate the local shape of that manifold directly via a network trained to estimate how each image region moves per unit of each action axis supervised by data collected from samples by the generator. A correction head fit to trajectory data estimates the shape of this manifold region up to a third-order Taylor polynomial. We call this network estimating the Taylor polynomial describing the manifold shape an action jet and compute its radius of validity in closed form. By traversing this approximation of the local manifold shape, we only have to perform inference once per state for any number of candidate actions. We achieve over 100x speedup of same length trajectory rollouts compared to the source generators, and this gap widens with the number of candidates a planner evaluates. We measure against state-of-the-art action-conditioned generators: IRASim and WorldGym on BridgeData V2, a real 7-DoF manipulation benchmark, and DepthSplat, LagerNVS and FrameCrafter on DL3DV scenes. Because only the first-order term comes from the generator and the rest from the data, the action jet also predicts held-out frames more accurately than IRASim, the generator that supplied its tangent, at short horizons and at roughly x less compute.

Primary area:
generative models
Keywords:
world models, generative models, manifold hypothesis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jJmwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Action Jets, a local polynomial representation for efficiently predicting the outcomes of candidate actions. The method first probes a pretrained action-conditioned generator to estimate action Jacobians, then trains a geometry encoder to predict these first-order responses from visual context. A correction head trained on recorded transitions supplies linear corrections and higher-order coefficients. At inference time, multiple candidate actions can share the extracted coefficients, allowing their future latent states to be evaluated through polynomial operations rather than repeated generative inference.

The paper evaluates the approach on Kubric, BridgeData V2, and DL3DV, studying prediction quality, computational efficiency, cross-generator tangent agreement, and offline candidate-action selection. It also introduces a coefficient-based radius intended to characterize the useful range of the local approximation.

Critical strengths
  1. An interesting design for action-conditioned world models. Predicting reusable, state-dependent coefficients is a compelling way to exploit the shared visual context across candidate actions. This offers a useful perspective on improving world models for search: changing the representation of the predictor, rather than only accelerating individual generations.

  2. A well-motivated combination of supervision sources. Using a generator to obtain local action responses, while learning corrections from recorded transitions, is an appealing approach. It allows the model to benefit from generator-provided supervision while retaining the opportunity to correct its inaccuracies using real observations.

  3. Broad evaluation with informative controls and limitations. The experiments cover synthetic scenes, robot manipulation, and camera motion. Copy-frame controls, probe-scale analyses, decoder ablations, and cross-generator comparisons help characterize the method. I also appreciate the reporting of negative results, including disagreement between generators on BridgeData and the source generator’s advantage at longer horizons and in policy consistency.

  4. Promising computational advantages for candidate evaluation. The reported results suggest that sharing state-dependent computation can substantially reduce the cost of evaluating multiple actions. This is relevant to settings where the number of candidates that a planner can consider is limited by world-model inference cost.

Critical weaknesses
  1. The interpretation of the validity radius could be clearer. Section 3.4 defines the radius through the relative magnitudes of linear and higher-order terms. This is intuitive, and the empirical validation is helpful, but its relationship to prediction error against real observations remains somewhat unclear. The paper would benefit from explicitly distinguishing an empirical reliability indicator from a region with stronger accuracy guarantees.

  2. The independent contribution of generator-derived supervision is not fully isolated. The combination of generator-derived first-order responses and real-data corrections is interesting. However, it is difficult to determine how much the derivative supervision contributes compared with training the same polynomial predictor directly on recorded transitions. Such a baseline would help establish the value of this component.

  3. The evidence for planning utility is currently limited to offline evaluation. The candidate-selection experiments demonstrate potential under a computational budget, but prediction metrics and action error relative to demonstrations do not directly establish improved task success. A small closed-loop experiment would strengthen this aspect. Alternatively, the conclusions could more explicitly distinguish the demonstrated offline selection capability from prospective control applications.

Questions for authors
  1. The interpretation of the validity radius could be clearer. Section 3.4 defines the radius through the relative magnitudes of linear and higher-order terms. This is intuitive, and the empirical validation is helpful, but its relationship to prediction error against real observations remains somewhat unclear. The paper would benefit from explicitly distinguishing an empirical reliability indicator from a region with stronger accuracy guarantees.

  2. The independent contribution of generator-derived supervision is not fully isolated. The combination of generator-derived first-order responses and real-data corrections is interesting. However, it is difficult to determine how much the derivative supervision contributes compared with training the same polynomial predictor directly on recorded transitions. Such a baseline would help establish the value of this component.

  3. The evidence for planning utility is currently limited to offline evaluation. The candidate-selection experiments demonstrate potential under a computational budget, but prediction metrics and action error relative to demonstrations do not directly establish improved task success. A small closed-loop experiment would strengthen this aspect. Alternatively, the conclusions could more explicitly distinguish the demonstrated offline selection capability from prospective control applications.

2 comments

Thank you for the positive and detailed review.

1. The radius. We agree and adopt this framing. r1∗r_1^{\ast} compares the Action Jet's own terms, so it is an empirical reliability indicator rather than a region with accuracy guarantees. Its relation to real error is the calibration in Sec. 4.3 and App. B.4, summarized in the general response. The revision states this in the abstract, introduction, Sec. 3.4, Sec. 4.3 and the conclusion.

2. Derivative supervision. Recorded actions co-vary across axes, so regressing image change on them alone entangles the per-axis fields, while generator probes answer isolated single-axis queries at any state (Sec. 3, App. C). Planning mostly queries actions that were not demonstrated, and Bridge has no ground truth for them. The matched comparison in the paper is the IRASim-basis and WorldGym-basis pair, which share the architecture, corrector recipe and real transitions and differ only in the probed generator. Fitting the corrector to the same real transitions does not repair the WorldGym tangent (anchoring +0.249 against −0.011, App. Table 17), so the probed tangent decides whether the Action Jet anchors on real data. On Kubric, with the encoder, latent and decoder fixed, the probed tangent alone, the 2-jet and the 3-jet reach 22.80, 25.33 and 26.27 dB PSNR at gap 4 (App. Tables 9 and 25). We will add the comparison you suggest, the same polynomial trained on recorded transitions alone, on BridgeData V2 in the camera-ready version.

3. Offline selection and closed loop. The general response reports the closed-loop experiment on PushT (Sec. 4.4, App. F). With the same CEM planner the Action Jet plans 22× faster than its source generator, succeeds on 84.7% of 300 episodes against 56.7% for the generator at equal planning time, and ties LeWM. The conclusion now separates the offline selection on Bridge from closed-loop control on PushT.

We hope these answers and the closed-loop results address your points.

Thank you for your reply. Since our original score was already quite high, I'll keep the score.

Reviewer_JmmGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the acceleration of evaluation of action-conditioned video prediction models under the assumption of local action conditioning. It argues that the local changes in the latent prediction given action perturbation could be approximated with a low-dimensional manifold conditioned on the current state, and uses a neural network to learn the mapping from a state to its corresponding manifold. In essence, this paper is trying to perform a model distillation: distilling the large prediction model to a smaller local model with Taylor expansion, so that different actions are evaluated physically instead of with a large neural network.

Critical strengths
  • The method is interesting and novel. Instead of distilling the large model into a model with a similar structure, it proposes to design a physical structure (Taylor expansion) for the local responsiveness, which leads to a higher efficiency. The advantage (being faster to evaluate multiple planning candidates) is also very important to the community.
  • The method is evaluated on three domains with different action axes which show some generalizability of the proposed approach.
Critical weaknesses
  • The experiment results are not significant enough to show the advantage of the proposed method. In Table 3, on gap3/gap4, the baseline is better. On the policy result, the baseline is also better. Only on gap1 and gap2, the proposed is better - yet it's almost indistinguishable from the "copy-frame" trivial baseline. Is the proposed method prediction real motion, or it's just predicting smaller motion compared to the baseline?
  • Despite the major advantage of the paper being "faster to evaluate multiple planning candidates", it fails to provide any evidence on closed-loop planning. It is not clear whether the trained model can be used for any planning tasks. Moreover, from the results, the model will have accumulated error for local motion planning. Therefore, whether this proposed model is useful for a long-horizon real-world planning task is also questionable.
Questions for authors
  • Why is the radius of validity in L248 computed in this way? Is this more like a heuristics?
  • Is it possible to use the learned world model to conduct any closed-loop control experiments, like using MPC?
  • Why is there no baseline comparison on the kubric dataset? How is it compared to the generator and the copy-frame reference?
2 comments

Thank you for the review. The general response reports the closed-loop results (Q2) and our treatment of the radius.

W1. The Action Jet predicts real, action-dependent motion. Figs. 7 and 14 and the rollouts in the supplementary material show the predicted arm moving with the ground truth over the evaluated gaps, while WorldGym stays nearly static. A new paragraph in Sec. 4.3 collects the measurements.

  1. Copy-frame control (Table 1). Most of a Bridge scene is static, so the copy frame is a strong control. Neither generator beats it on PSNR at gap 1 (IRASim 23.34 and WorldGym 23.80 against 24.48 dB), while the Action Jet with its fine-tuned decoder does (24.57) and stays above it at every gap, by 0.9 to 1.0 dB from gap 2 on. On LPIPS the copy frame leads at every gap against all models, both generators included.
  2. Candidate selection (Table 2). A predictor that holds the frame gives every candidate the same prediction and selects at random-of-K (0.038 at K = 32). The Action Jet selects 0.025 and IRASim 0.030.
  3. Policy consistency (Table 1). Octo's actions on the Action Jet's predictions are closer to its actions on the real futures than its actions on the copy frame are (ADE 1.30 against 1.58, FDE 1.66 against 1.99).
  4. Motion size (App. Table 11). Binned by the size of the demonstrated action, the Action Jet beats the copy frame on LPIPS in every bin up to 1.7 r1∗r_1^{\ast}. Its margin is smallest for the smallest motions and largest near r1∗r_1^{\ast}.
  5. Kubric and PushT. On Kubric the Action Jet beats the copy frame at every gap by 4.8 to 5.6 dB PSNR, with lower LPIPS (App. Table 9). On PushT the block enters the planner's cost only through the Action Jet's prediction, and an episode succeeds only when the block reaches its goal position and angle. Planning with the Action Jet succeeds on 84.7% of 300 episodes, as often as LeWM.

IRASim leads by 0.6 to 1.0 dB from gap 3 and on policy consistency, as Sec. 4.3 states. Our claim is short-horizon accuracy at a cost that lets a planner score many candidates (Tables 2 and 3).

W2. The general response shows the Action Jet planning in closed loop. In MPC the Action Jet is re-extracted from each new observation, so prediction error does not accumulate across replanning steps. On PushT it is also re-extracted at every CEM iteration, at the planner's current mean plan, and one polynomial maps a whole 25-step plan to its outcome, so error does not compound across the steps of a plan either. Within a Bridge rollout the error grows with the step, and r1∗r_1^{\ast} tracks where the Action Jet remains useful (App. B.4). Our claims concern short horizons. For a long-horizon task the Action Jet would score short-horizon candidates from a proposal policy or subgoals, as with Octo in Fig. 3, and the revised conclusion states this scope.

Q1. Along a unit action direction uu, the Action Jet's displacement is exactly a cubic, P(r)=αr+βr2+γr3P(r)=\alpha r+\beta r^2+\gamma r^3. By the triangle inequality its nonlinear part is at most p2r2+p3r3p_2 r^2+p_3 r^3, with pip_i the norms of the coefficients. r1∗r_1^{\ast} is the step at which this bound reaches the linear term p1rp_1 r, the positive root of p3r2+p2r−p1=0p_3 r^2+p_2 r-p_1=0. Inside r1∗r_1^{\ast} the nonlinear terms cannot exceed the linear term. This is the classical criterion for where a truncated Taylor expansion is dominated by its leading term, the analogue of a trust-region radius in uncalibrated visual servoing.

  • It is computed from the extracted coefficients, without rendering.
  • It depends on the direction.
  • It is invariant to a uniform rescaling of the response, so a model that commands less motion does not get a larger radius.

It is a heuristic about the Action Jet rather than a bound on its error, and the revised Sec. 3.4 says so. App. B.4 calibrates it against real frames, as summarized in the general response.

Q2. Yes. The general response reports CEM-based MPC on PushT, with the source generator, the Action Jet and LeWM on the same episodes.

Q3. Kubric served to certify the probe against autodiff Jacobians and to compare corrector designs, so its table compared Action Jet variants. App. Table 9 now adds copy-frame and generator rows through the same decoder. The Action Jet leads the copy frame at every gap, is ahead of its generator at gap 1 (31.80 against 30.63 dB PSNR) and trails it from gap 2.

We hope these results address your concerns, and we would be grateful if you would reconsider your score.

Thanks for your rebuttal! I have read it and will take it into consideration in the later discussion stage.

Reviewer_nm6BOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Action Jets for approximating the local shape of the action-conditioned data manifold with a third-order Taylor polynomial. A geometry encoder learns the action Jacobian from generator probes, while a correction head is fitted to recorded trajectories. The resulting coefficient fields can be reused to evaluate multiple candidate actions efficiently. The paper also introduces a closed-form radius of validity intended to indicate the range of actions over which the approximation remains accurate.

Critical strengths
  1. By reusing a local predictive model across candidate actions, the method reduces the cost of short-horizon prediction. The reported speedups suggest that this design could be useful when many alternatives need to be evaluated under a limited computational budget.
  2. The paper is generally well written and logically organized, with ample figures and tables that illustrate the method and summarize the results.
Critical weaknesses
  1. The claimed radius of validity is not established as a bound on prediction error. Section 3.4 and Eq. (8) compare terms within the learned polynomial without controlling its discrepancy from the true transition. Omitted higher-order terms can cause arbitrarily large errors within the computed radius. The empirical calibration in Appendix B.4 is useful, and the supplementary code acknowledges this distinction, but neither establishes the stronger interpretation in the main text. The radius should therefore be presented as an empirically calibrated heuristic unless additional error bounds are provided.
  2. Evidence for iterative planning remains limited. Table 4 shows improved agreement with demonstrated actions, but whether this translates into better task performance in the setting illustrated in Figure 4 remains unclear.
Questions for authors
  1. Could you add a small closed-loop evaluation, in simulation or on a robot, comparing task success rate (SR) for the same policy with and without Action Jets re-ranking?
  2. Could you specify the hardware and evaluation configurations used for Table 5, including network-call counts for composed rollouts and which computations are shared across candidates?
  3. Could you extend the ablations to compare the same third-order model with and without teacher derivative supervision, alongside a linear-in-action baseline under matched settings?
2 comments

Thank you for the careful review. The general response reports the closed-loop results and our treatment of the radius.

W1. Agreed. The radius compares the polynomial's terms with each other, so it describes the Action Jet and not its distance to the true transition, as the supplementary code notes. The revised Sec. 3.4 says so, states that omitted higher-order terms can make the error large inside r1∗r_1^{\ast}, and calls r1∗r_1^{\ast} an empirically calibrated heuristic, in line with App. B.4. On real Bridge frames it is conservative by a factor of 1.7 to 2.0 (median empirical radius over r1∗r_1^{\ast}), and this factor is stable across two unrelated Action Jets (Table 4).

W2 and Q1. The Bridge setting of Fig. 3 has no simulator, so we measure task success on PushT. The same CEM planner runs with the same proposals, goal cost and budget, and scores its candidates either with the source generator (without Action Jets) or with the Action Jet charted from it. The Action Jet plans 22× faster and succeeds on 84.7% of 300 episodes, against 56.7% for the generator at equal planning time. The generator needs all 300 samples and 22× the time to lead (89.3%). On the same episodes the Action Jet ties LeWM (84.7%). The conclusion now separates the offline selection of Table 2 from closed-loop control.

Q2. Table 3 and App. E now give the configuration.

  • Hardware and timing: one RTX PRO 6000 Blackwell GPU, with IRASim and the Action Jet in fp32 and WorldGym in the fp16 autocast of its own code. Each model scores all K candidates of a state together, batching them in every network call. Times are CUDA-synchronized wall clock, the median over 10 Bridge states after one warm-up call per state, and FLOPs come from PyTorch's FLOP counter. Decoding is excluded for every model because ranking is done in latent space. The timed candidates are the recorded four-step chunk plus Gaussian noise, since their values do not change the cost.
  • Network calls: Table 3 now counts calls rather than sampler steps. IRASim's 25 PNDM steps take 34 calls, because its warm-up spends four calls on each of the first three steps, and WorldGym's classifier-free guidance doubles its 10 × 4 DDIM calls. The counts are therefore 34K and 80K, where the initial submission listed 25K and 40K.
  • What candidates share: the VAE encodes of the current, context and goal frames and the geometry encoder run once per state. In single extraction the correction head also runs once, batched over the candidates' noise draws, and each candidate is then a polynomial evaluation. At K = 16 this is 1715× faster than IRASim and 268× faster than WorldGym, with 1758× and 827× fewer FLOPs. Tables 1 and 2 use single extraction.
  • Composed rollout: 2 calls per step, so 8 for four steps. Steps 2 to 4 run the encoder and the correction head on each candidate's predicted latent, in one batched call per step and without decoding, so its cost grows with K. It is reported for speed comparison only.

Q3. Recorded actions co-vary across axes, so regressing image change on them alone entangles the per-axis fields, and each recorded transition constrains the polynomial along one direction only. Generator probes answer isolated single-axis queries at any state, which the data never contains, so we measure the tangent from the generator and fit its correction and the higher orders to the data (Sec. 3, App. C). A planner mostly queries actions that were not demonstrated, and Bridge has no ground truth for them.

The paper contains one matched comparison of the derivative supervision. The IRASim-basis and WorldGym-basis Action Jets share the architecture, the corrector recipe and the real transitions, and differ only in the probed generator. Fitting the corrector to the same real data does not repair the WorldGym tangent (anchoring +0.249 against −0.011, least-squares gain 0.74 against −0.10, App. Table 17). So the probed tangent, not the data fit, decides whether the Action Jet anchors on real transitions.

The order ablation on Kubric keeps the encoder, latent and decoder fixed (App. Tables 9 and 25). The probed tangent alone, linear in the action and without data fit, reaches 22.80 dB PSNR at gap 4, the 2-jet 25.33 and the 3-jet 26.27, and each added order improves PSNR and LPIPS at every gap. On Bridge the first-order truncation of the same Action Jet, a linear-in-action baseline under matched settings, has higher latent error than the full jet in every bin up to 1.7 r1∗r_1^{\ast} (App. Table 11).

We agree that the same polynomial fit to recorded transitions alone isolates the generator’s contribution most directly, and we will add this comparison on BridgeData V2 to the camera-ready version.

We hope the closed-loop results and the revision address your concerns, and we would be grateful if you would reconsider your score.

Thanks for the clarification and additional experiments. The new results address my concern about closed-loop planning part, so I raised the score. However, the evaluation is still limited to a single simulated task, with no real-robot trials. It remains unclear whether the gains generalize to other tasks or transfer to real robots. I’d welcome a broader evaluation in future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.