acceptodds
Under review as a conference paper at ICLR 2027

Action-Coordinate World Models for Precise Control From Offline Data

Abstract

World models are trained to predict and then used to plan. Where the task demands precision, the two objectives come apart: a model that fits its observations can still leave the planner without a route to the target. World models are trained to predict and then used to plan. Where the task demands precision, the two objectives come apart: a model that fits its observations can still leave the planner without a route to the target. We study tasks in which objects have to be placed precisely to maximize a signal, although their positions are never observed directly and a context both displaces them and distorts every movement. For them, we introduce action-coordinates, latent representations in which actions act linearly and the target sits at the origin, and build a world model architecture on top of them that moves structure from the planner into the representation. Our theory gives conditions under which action-coordinates exist, the structure they impose, and when they can be learned from high-dimensional observations, together with a closed-form policy and conditions under which it is optimal. On challenging benchmarks of high-fidelity, visually observed tasks, action-coordinate world models outperform planner-based and gradient-based world models and confirm the assumptions of our theory.

Primary area:
reinforcement learning
Keywords:
world models, model-based reinforcement learning, reachability analysis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HwwAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies action-coordinate representations for precise positioning from offline visual trajectories. It constrains latent transitions to be linear in the action and anchors the target at the origin, allowing closed-form control instead of online planning search. The paper develops existence, representability, and conditional optimality results, and evaluates the approach on several positioning tasks, including physical systems.

Critical strengths
  1. The representation is closely tied to the control objective. The connection between structural assumptions, observation history, and the resulting controller gives the method a clear conceptual basis.
  2. The evaluation spans different positioning mechanisms, and Appendix F examines important assumptions and design choices.
Critical weaknesses
  1. Some headline comparisons do not isolate the action-coordinate constraint. In Table 2, the image-based Interferobot configuration uses a sharpness encoder and 800 training epochs for AC-WM, versus an autoencoder and 200 epochs for the listed world-model baselines; training duration also differs on LensAlignment. These comparisons establish performance of the configured systems, but do not attribute the entire gap to the proposed representation.
  2. Reward-based anchoring is an important data requirement. Page 30 specifies using the 50 frames closest to the target on random data, and the supplied implementation ranks stored objective or metric values. The relationship between these quantities and the rewards available to each baseline should be explicit for each environment. The limitations when the best available observations remain far from the target also matter.
Questions for authors
  1. Which main comparisons match the encoder, observation history, and training budget simultaneously? Please identify any existing results that separate these factors; a focused comparison on one representative environment would be sufficient if additional evidence is needed.
  2. Which quantities determine anchor ranking, reward regression, and target labels in each environment, and how do they relate to the rewards given to the baselines? Please connect the existing anchor-error analysis to the random-data setting.
4 comments

We thank the reviewer for the constructive feedback. The comments pointed us to some places where the presentation of our paper could be improved.

Q1

We agree: Table 2 lists the best configuration we found per method, and these potentially differ in the observation history kk, the encoder used, and the training epochs.

In the original submissions, the following comparisons already match the factors mentioned by the reviewer precisely:

  • On LightTunnel (image), all methods share k=1k=1, the spectral encoder, and 400400 epochs (now Figure 22).
  • On LensAlignment (vector), they share k=1k=1 and 400400 epochs. Figure 13 shows AC-WM and RG-WM at each k∈{1,3,5}k\in\{1,3,5\} on same encoder and training budget.

For a clearer overview, we provide the following two additional comparisons with the same kk, trainings budget and encoder architectures and hope this will address the reviewers concern appropriately.

LensAlignment (image)

All methods using the spectral encoder and 400400 training epochs:

Methodk=1k=1k=5k=5
AC-WM3.874.64
RG-WM1.982.54
S-WM2.042.45
MPC-WM1.762.29

BeamAlignment (vector)

All methods using 200200 epochs of training budget

Methodk=1k=1k=3k=3k=5k=5k=4k=4, padded
AC-WM13.029.997.89--
RG-WM7.646.657.598.43
S-WM5.255.345.686.15
MPC-WM5.795.375.385.75

Note that k=4k=4 padded repeats the first frame to match k=4k=4, which makes no sense for AC-WM that relies on random actions during its probing phase.

Generally, our search followed one rule across environments: Where a vector version exists, we trained large grid of configurations there and select the best one per method. On images, the encoder adds a further axes to that grid and every run is considerably more expensive, so a full search is out of reach. We therefore start each image environment from the best vector configuration and search only the factors specific to images, foremost the encoder and the window kk. Moreover, by training encoders together with decoders in isolation, we made sure that the encoders are expressive enough to represent the relevant information in the images.

The reviewer's question prompted us to extend our hyperparameter search on the image-based environments to more encoders and more values of kk where computationally feasible and reasonable. On two of them, we found better configurations for the world-model baselines (MPC-WM now better on Interferobot and RG-WM better on BeamAlignment), which does not change the general picture. The revised manuscript now reports these results as well.

Q2

We agree that the manuscript should state this more explicitly, and the revision adds the table below to Appendix E. Every environment returns the reward rtr_t as given in Appendix E.

EnvironmentReward observed (all methods)AC-WM target labels, anchored dataAC-WM target labels, random data
BeamAlignmentrtr_tframes with rt>ln⁡105r_t > \ln 10^5, i.e. weighted beam error below 10−510^{-5} mthe 5050 frames of highest rtr_t
SpheroEnv (sim and real)rtr_tframes whose rtr_t carries the success bonus, i.e. ∥s−sW∥≤15\lVert s-s_W\rVert \le 15 pxas on anchored data
Interferobotrtr_tthe frame at sWs_W each episode is collected fromthe 5050 frames of highest rtr_t
LensAlignmentrtr_tthe frame at sWs_W each episode is collected fromthe 5050 frames of highest rtr_t
LightTunnelrtr_tthe frame at sWs_W each episode is collected fromnot run

All methods, offline or online, model-based or model-free, receive this rtr_t and then use it according to their design. No method receives other ground-truth quantities or evaluation metrics that depend on them (e.g. the beam error in mm or the ball distance in px). Specifically, our anchor ranking reads rtr_t only and no other ground-truth information. On random data, the 5050 frames closest to the target was a wrong formulation, in fact we select the 5050 frames of highest reward in the available dataset, which is not necessarily the same. We apologize if the framing "closest to the target" was creating the impression that ground truth information was used. We have corrected this in the revised manuscript accordingly.

The same holds for the online domain. As AC-WM is designed to be an offline method, to make it working here as well, we mark the frames of its buffer within the goal tolerance and at least the 5050 of highest reward (except on SpheroEnv, see table) for the anchor mechanism. TD-MPC2 observes the same reward at every transition. The two methods differ only in how they use it: TD-MPC2 regresses it through a reward head and a discounted value, while AC-WM additionally uses its ranking to fix the origin of the latent chart. As the controller improves, the 5050 highest-reward frames move closer to the target, so the anchor sharpens with the data.

That said, we extended our anchor-error analysis to more random-data settings, specifically to SpheroEnv to BeamAlignment and Interferobot (both using vector and image observations). The results are in Figure 9 and Figure 32 (Appendix F.4) of the revised manuscript.

Thanks for adding the matched comparisons. That clears up my first concern. In the online experiments, do you use the shared reward alone to decide whether a frame is within the goal tolerance?

Yes, all models receice the same information: the observations, the actions and the reward rtr_t. Whether a frame is within the goal tolerance is decided from this reward alone, by the threshold rt≥Cr_t \ge C. The threshold CC is also set from the rewards in the replay buffer. No additional information enters the anchor loss term.

Reviewer_Md52OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes action-coordinate world models for a more precise control. The method is to first learn a special latent representation, in which actions induce linear transitions while target corresponds to the origin. So that the iterative planning problem can now be converted to a closed-form control procedure. Authors develop theoretical conditions for the existence and learnability of such representations. Then test it among many downstreams like optics, accelerator physics, and robotics. Experiments show that it improve over both model based and model free methods.

Critical strengths
  1. this paper did not see world models and planners as siloed modules, instead it tries to make the planning easier by trying to learn a good representation. This is a novel point.

  2. The authors provide systematic theoretical analysis. Not only did the paper proposed the solution, it also analyzed the existence of action-coordinate representation, learnability and its relationship to optimal control, which builds a strong basis for the method.

  3. The method connects with the downstream task very well, compared with usual world models that tends to mainly care about the prediction precision.

Critical weaknesses
  1. The theoretical formulation could potentially constrain some downstream tasks. Because the theory assumes smooth, driftless, reachable dynamics, symmetric convex action spaces, and episode-constant hidden context. And this might not be true in many realistic robotics seetings that has contact, frictions or changing dynamics etc.

  2. It would be good if the authors could provide more analysis.of the model errors, observation noise, and control degradation. Usually the closed form controller might naturally suffer from the problem of being sensitice to errors. In this case specifically, the errors are in the learned coordinate representation.

Questions for authors

It would be great if the authors could answer the questions brought in the weakness, and address each concern separately. I will consider raising the scores if those get properly addressed.

1 comment

We thank the reviewer for the positive evaluation of our work and for the constructive feedback.

W1

Combining our ideas and transferring them to robotic settings that need precise control and are visually observed is an exciting direction for future work we currently pursue. We agree that the assumptions idealize real systems and our manuscript already contains a critical discussion of them in Appendix D. However, our existing experiments already go beyond them:

  • Physical systems. The physical experiments in the SheroBolt environment contains friction, wheel slip and contact with the arena walls; the physical LightTunnel has discretized focus and gain and hysteresis in its actuators; and every environment clips states and actions at travel limits leading to non-symmetric action spaces particularly on the boundary. Still, AC-WM can solve all of these tasks.
  • Controlled violation. Appendix F.1 uses dynamics that are smooth and fully actuated but admit no action-coordinate system at all. AC-WM copes with moderate violations and fails only for strong ones.

Additionally, the revised manuscripts adds an ablation study when the dynamics changes within an episode (Appendix F.6) and then actuators act in a non-symmetric and non-convex way (like caused by stiction). In all these cases, AC-WM still works.

W2

We added an ablation on observation noise to the revised manuscript (Appendix F.8), where we analyse AC-WM on Interferobot and on BeamAlignment when additional Gaussian pixel noise is added to every image the controller receives, while the reward is computed on the original image. We note that both tasks are highly sensitive to pixel noise by construction. If the reward is recomputed from the noisy image of the optimal state, it drops on Interferobot significantly, making a noisy optimal image indistinguishable from the image of a misaligned system and thus the task unsolvable.

Our new result has two parts. First, AC-WM trained on the original data tolerates noise up to a certain degree, and then degrades. Second, retraining on the same data with partially noised images level mostly removes this sensitivity. Hence, the closed-form controller is sensitive to observation noise outside the training distribution of the encoder, but not inherently: noise augmentation during training is sufficient to make it robust.

Reviewer_XN4fOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies high-precision alignment tasks such as lens alignment and beam steering, where a hidden per-episode context both moves the optimum and changes the effect of actions. Instead of planning in a world model, the authors learn an "action-coordinate" latent space in which actions act as linear translations and the target sits at the origin, so that control reduces to a single closed-form pseudoinverse step. The per-episode gain is either estimated online or pinned to the identity. The theory gives conditions for the existence of such coordinates, step-optimality of the closed-form policy, and the history length the encoder needs. Experiments cover five benchmarks, two of them on real hardware, with comparisons against world models, offline RL, and model-based methods.

Critical strengths
  1. The idea and the theory fit together. By moving structure from the planner into the representation, planning becomes a linear solve, which removes the mismatch between "predicting well" and "planning well" instead of only mitigating it. The appendix ablations break the theoretical assumptions one at a time, and the results broadly follow what the theory predicts. This makes the contribution more than a new set of losses.

  2. It works on hard tasks, including real hardware. On both physical setups, AC-WM reaches the target while the world-model baselines do not, and on beam alignment it is clearly more data-efficient than the offline RL and model-based baselines. For the alignment community, this is the most consequential result.

Critical weaknesses
  1. The main results rely on privileged knowledge of the optimum. In most environments, the anchored data is collected starting from the true per-context optimum, which is computed by the simulator or calibrated on the device; the supplementary code itself calls this step "the privileged calibration assumption". Yet finding the optimum is exactly what an alignment task is about. Experiments with random data anchored only by reward are mostly compared against offline RL; the comparison against world models covers a single environment, where AC-WM is overtaken at larger data sizes.

  2. The world-model baselines are weak. The three world-model baselines are the authors' own implementations, with short planning horizons, and they can adapt to the context only through a short fixed window. There is no comparison with an established offline world model, so the claim of outperforming planner-based world models is not sufficiently supported.

Questions for authors

Questions that could change my assessment:

  1. How would the method be used in realistic settings where the optimum is not available, e.g. production alignment? Could you report reward-only anchoring on more environments, including a comparison with the world-model baselines?
  2. Could you add an established offline world model, or a baseline that can likewise use the full episode history?
  3. How were the best configurations in Table 2 selected? Was a separate validation set used?
  4. Relign is introduced in Sec. 1.2 and Fig. 1, but there are no results for it in the paper and it does not appear in the code. Also, COMBO results are reported, but the supplementary code contains no script that trains COMBO.

Clarifications and suggestions:

  1. Estimating the gain online and then stepping towards the target with a pseudoinverse is closely related to uncalibrated visual servoing. Please discuss the relationship.
  2. On LightTunnel, the best latent dimension does not match the rank predicted by the theory. Is there any further analysis?
  3. Minor: in the world-model literature, "AC" usually stands for action-conditioned (e.g., V-JEPA 2-AC), so "AC-WM" is easily read as "action-conditioned world model". Consider a different name to avoid confusion.
1 comment

We thank the reviewer for the positive evaluation of our work and for the constructive feedback.

Q1

Our methodology strictly addresses realistic and practical settings. For the alignment tasks studied, for instance, there typically exists routines that can explore the position of the optimum or at least a good enough solution, however in a very suboptimal manner in terms of steps needed. For instance, expensive grid scans, structured coordinate walks, or human-led alignments, like for interferometers, can be used to find the optimum.

Nevertheless, we agree that it is interesting to report further results on random data, where the optimum is not available. The main text of the revised manuscript now generalizes the analysis we already did on SpheroBolt to BeamAlignment (vector and image) and Interferobot (vector and image). The results are in Figure 9 and we show that reward-only anchoring suffices to achieve good performance using random data only.

Q2

We emphasize that the world-model baselines used are designed to isolate the planner, the component in which AC-WM departs from established methods. Nevertheless, to address the reviewer's concern, the revised paper now benchmarks against MAPLE [1] in addition to MOPO and COMBO. MAPLE conditions its policy on the full episode history through a recurrent context extractor. The results are in Figure 7 of Section 4.

[1] Chen et al., Offline Model-based Adaptable Policy Learning, NeurIPS 2021

Q3

We did not use a separate validation set. Every configuration is evaluated online in the respective environment. We pool the evaluation episodes of all its seeds, take the median reward at every step, and score the configuration by the area under this median curve per step. Table 2 lists, per method, environment, and observation type, the configuration with the largest area. The area accounts for the speed of convergence, the reward reached, and whether the controller holds that reward: the final reward would ignore the speed of convergence, the best reward along the episode could credit a controller that only touches the optimum and then drifts off. This is now explicitly stated in Appendix E.

Q4

The reviewer is right: our initial submission did not contain experiments in the Relign environment. The reason is that Relign and LensAlignment are closely related: both simulate a three-lens optic imaging the same reference pattern, and both are covered by our problem formulation. They differ in how the image is rendered, and we chose LensAlignment for two practical reasons. First, rendering in Relign is slow, which limits the number of runs we can afford. Second, LensAlignment returns MTF values without rendering an image, giving a fast vector based version of the problem. Thus, experiments in Relign are computationally very expensive. Nevertheless, the revised manuscript now reports a preliminary evaluation of our method in this environment as well (Appendix E.1).

We thank the reviewer for pointing out the missing COMBO code. This was an oversight on our side, and the supplementary code now contains it.

Q5

We thank the reviewer for bringing the connection to visual servoing to our attention. In fact, our test-time controller is closely related to uncalibrated visual servoing in learned coordinates, but there are also key differences. The revised manuscript now holds relevant pointers to related work in the main text (related work and Section 2.2) and we have also added a discussion in the appendix (Section A.7).

Q6

We thank the reviewer for the careful reading. Our original submission already discussed this phenomenon briefly in Appendix E.4. The gap between the configurations lies mainly in the probing phases: All configurations reach the same final reward and, once probing is over, converge at nearly the same speed (Figure 22 in the revised paper). Moreover, we see that, when BB is fitted, l=2l=2 shows better performance than l=4l=4, exactly as our theory predicts. The question is, however, why l=4l=4 with B=IB=I performs equally well.

To answer it, we have conducted an additional analysis on a separately collected dataset that provides a reasonable explanation: We move the actuators by known displacements in three contexts and fit the linear response of the learned embedding. For l=4l=4 with B=IB=I, the two dominant latent directions are the focus and the sensor gain. Hence, the model learns essentially the rank-2 action-coordinate system predicted by our theory although it is not constrained to do so, which explains why it performs as well as l=2l=2. The detailed analysis including a visualization of the latent responses can be found in the revised manuscript (Appendix E.4).

Q7

We thank the reviewer for pointing this out and we consider to take a different name for the final version of the paper, like ACS-WM for action-coordinate system world model.

Comments on the paper

1 comment

Dear Reviewers,

We would again like to thank you for taking the time to review our paper and for your constructive feedback. As the discussion period is coming to an end, we hope that we have addressed all of your concerns, open questions, and comments. We know that the discussion period is intense and time is short. Nevertheless, we would greatly appreciate hearing whether our responses resolve your concerns. If any questions remain, we are happy to clarify them before the deadline. Otherwise, we would appreciate it if you could take our responses into account in your final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.