acceptodds
Under review as a conference paper at ICLR 2027

PoleWM: Visual World Models with Pole-Based Action Responses

Abstract

Visual planning requires predicting how candidate actions unfold over time. Direct world models avoid recursive state rollout, but direct prediction alone does not specify how action effects should share temporal structure. We introduce PoleWM, which represents a latent trajectory as an initial-condition response plus delayed, action-conditioned responses. Nonlinear networks determine response coefficients from visual history and causal action prefixes; a shared pole basis determines their evolution with elapsed time. This factorization preserves nonlinear action interactions while providing exact initial-state anchoring, non-anticipation, and bounded responses to finite plans. We also characterize the additional basis direction needed to represent re-anchored tails, separating these structural properties from restart consistency. Across four image-goal control environments, PoleWM achieves 90.83% mean success over two training seeds, exceeding the strongest baseline mean by 6.08 percentage points and leading on three tasks. In a first-epoch Cube study with one training seed and no restart penalty, neural and pole response heads improve over a generic direct head by 20.67 and 23.34 points, respectively. Separately, optimized single-frame planning takes 74-87 ms per complete CEM solve on an A100. These results support explicit action-response factorization as an effective inductive bias for direct visual world models, with poles providing an analytically structured realization.

Primary area:
applications to robotics, autonomy, planning
Keywords:
visual world models, model-based planning, temporal representations, action-conditioned prediction, self-supervised learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wZrcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces PoleWM, a direct visual world model that decomposes a learned latent trajectory into an initial-condition response and delayed, action-conditioned responses. Nonlinear action-prefix networks generate response coefficients, while a shared, learned pole basis determines their temporal evolution. The construction enforces initial-state anchoring and non-anticipation and gives bounded responses for a fixed finite plan. The model predicts visual embeddings and is evaluated through CEM image-goal planning on Cube, TwoRoom, PushT, and Reacher, alongside an early prediction-head comparison and a separate planning-latency study.

Critical strengths
  1. The idea of explicitly organizing action effects over time is interesting. Separating response content from elapsed-time evolution provides a concrete inductive bias for direct prediction. The structural analysis also carefully distinguishes anchoring, non-anticipation, and boundedness from the stronger requirement of restart consistency. This makes the proposed architecture and the scope of its guarantees understandable.

  2. The complete-system results provide encouraging evidence of practical value. PoleWM reaches 90.83% mean planning success, compared with 84.75% for the strongest baseline mean, and leads on three of four tasks. A separate timing experiment demonstrates that the analytic predictor can be used at practical latency inside CEM. These results support further investigation of the design.

Critical weaknesses
  1. The empirical justification for the added temporal prior, especially the pole parameterization, remains incomplete. The controlled head comparison is restricted to Cube, one training seed, and the first training epoch. At epoch 1, the direct, neural-response, and pole-response heads reach 56.33%, 77.00%, and 79.67%, respectively. The larger observed gain therefore comes from response organization, while the additional gain from poles is smaller. This is useful early-training evidence, but it does not establish whether either advantage persists after full training or as data and model capacity increase. The classical response analogy also does not by itself establish that a learned visual latent admits this restricted temporal form. Joint representation learning could make the prior appropriate, but the structural propositions characterize the chosen architecture rather than demonstrate its predictive suitability. I would like a more complete head comparison to justify the added structure and clarify the contribution of poles.

  2. The evaluation scope limits the evidence for broader applicability. The four benchmarks involve relatively simple goal-reaching tasks, use a from-scratch ViT-Tiny encoder, and evaluate short control episodes. These are useful settings for testing the idea, but provide limited evidence that the latent response prior remains beneficial under more complex dynamics or visual conditions. The overlapping-window split allows windows from the same episode in both training and validation, and control task pools were reused during development. The paper appropriately acknowledges that this does not establish trajectory-held-out generalization. Nevertheless, the current evaluation leaves my main concern about the applicability of the prior unresolved; an independent held-out evaluation would strengthen the evidence.

Questions for authors
  1. Do the direct, neural-response, and pole-response differences persist after full training and across training seeds or tasks? If such results are available, please report them and clarify model-capacity and training-compute differences. This would materially affect my assessment of the value of response organization and the pole basis beyond early optimization gains.

  2. How are the control evaluation tasks separated from the training episodes and the pools used during development? Are there results on an independent held-out pool or trajectories that clarify whether the temporal prior generalizes beyond the reported protocol?

Reviewer_ZwQcOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PoleWM addresses image-goal visual control through offline world model learning and online model predictive control (MPC). At test time, a CEM planner optimizes action sequences to minimize the predicted embedding distance to a goal image.

The main novelty is an action-response factorization for direct multi-step prediction, rather than recursively rolling out future states or directly predicting outcomes from action prefixes and query times. PoleWM represents a latent trajectory as the sum of a free response determined by the initial observation history and delayed responses triggered by individual action changes. This design explicitly separates action-dependent response content from temporal evolution and enforces initial-state anchoring, non-anticipation, and bounded responses for finite plans.

Experiments on four image-goal control environments (Cube, TwoRoom, PushT, and Reacher) show improved average success over LeWM, Fast-LeWM, and QQWorld. Additional experiments compare generic direct prediction, neural response factorization, and pole-based responses during early training, and evaluate planning latency.

Critical strengths

PoleWM is a structured and interpretable approach to direct world modeling. The paper introduces an explicit action-response factorization that combines nonlinear action-prefix conditioning with a shared analytic pole basis. This provides useful structural properties, including exact initial-state anchoring, non-anticipation, and bounded responses for finite action plans.

Critical weaknesses
  1. The generality of action-innovation-triggered response factorization is insufficiently justified. Important dynamical events, such as collisions and contact transitions, can be more worthy of response decomposition, while frequent action changes may be entirely predictable. Although such events may be implicitly approximated by existing responses, it is unclear why action changes should be privileged as the explicit triggers for new responses.
  2. The experimental validation is weak. The main evaluation covers only four relatively simple image-goal control environments. The main results use only two training seeds, while the response-head ablation uses a single seed and examines only the first training epoch on Cube. Moreover, the ablation does not isolate the benefit of action-innovation triggering itself, and the pole basis provides only a small improvement over the neural response basis in the reported comparison.
Questions for authors
  1. Although nonlinear prefix-conditioned coefficients allow nonlinear action interactions, it remains unclear whether this response organization is an appropriate inductive bias for general nonlinear world models. Why are action changes chosen as the triggers for new responses? Is the proposed action-innovation-triggered response factorization primarily inspired by the step-response superposition principle of linear time-invariant (LTI) systems? What motivates its extension to general nonlinear dynamics?
  2. The main results use only two training seeds on four relatively simple image-goal control environments. The baseline lacks training-seed variance. Could the authors provide wider evaluation? Could the authors clarify the boundary of the introduced inductive bias regarding pole basis?
  3. Could the authors clarify whether the neural response head (A) uses exactly the same action-innovation gating mechanism as the pole response head (C), with the pole basis being the only architectural difference? Please also report the parameter counts and training time of R, A, and C.
  4. Multi-seed ablations across different environments are needed to establish which components actually contribute to the performance gains. Could the authors provide a controlled ablation that removes the action-innovation gate while retaining the temporal basis?
Reviewer_XkmbOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes PoleWM, a direct visual world model that represents latent trajectories as a combination of initial-condition and delayed action-conditioned responses. The model uses nonlinear action-prefix conditioning to generate response coefficients and a learnable pole-based temporal basis to describe their evolution. The paper also analyzes structural properties including initial-state anchoring, non-anticipation, boundedness, and restart closure.

The authors report an average success rate of 90.83% across four image-goal control environments, a 6.08 percentage-point improvement over the strongest baseline in their evaluation. However, the attribution of these improvements and the comparability of the reproduced baselines remain unclear.

Critical strengths
  1. The paper formally characterizes several properties of the proposed response representation, including initial-state anchoring, non-anticipation, and boundedness. The analysis of restart closure also identifies a limitation of the proposed basis and provides a mathematical characterization of the missing direction.
Critical weaknesses
  1. The proposed model combines multiple architectural and training choices, including action-response decomposition, nonlinear prefix conditioning, innovation gating, and a learnable pole basis. The main experiments compare complete systems, making it difficult to attribute the reported improvements specifically to the proposed response factorization. Although Section 5.3 compares different prediction heads, this study is restricted to early training on Cube with a single training seed and differs from the main experimental configuration. Consequently, the contribution of the proposed formulation to the final performance gains remains insufficiently established.

  2. The reported Fast-LeWM and Fast-LeWM + consistency results (83.25% and 83.50%) are noticeably lower than those reported in the original arXiv preprint (90.50% and 92.00%). Although differences in experimental protocols and training variability may explain these discrepancies, the paper does not sufficiently clarify the reproduction procedure and relevant configurations. This raises concerns about the interpretation of the relative performance improvements.

  3. The proposed method primarily introduces a structured action-response parameterization into an existing direct visual world-modeling framework. While the design is reasonable, the methodological contribution appears relatively incremental. Most of the stated structural properties follow directly from the chosen construction, and the paper does not convincingly establish that this particular formulation offers a substantial advance over alternative dynamics representations.

  4. The method introduces several intertwined architectural components and training objectives, making the overall approach difficult to follow. The presentation could better distinguish the central methodological contribution from supporting implementation choices.

Questions for authors
  1. Can the authors isolate the contribution of action-response factorization under the full-training setting, while controlling for other architectural and training differences?

  2. Does the advantage of response-based prediction persist across all four environments after full training?

  3. Were the Fast-LeWM baselines evaluated using official checkpoints or retrained under the unified protocol? What explains the discrepancies from the results reported in the original preprint?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.