acceptodds
Under review as a conference paper at ICLR 2027

Latent Action Policies from Suboptimal Data

Abstract

Latent Action Models (LAMs) learn control representations from action-free video, but existing pipelines typically train latent policies with behavioral cloning, without an explicit policy improvement objective. Although offline RL offers a natural approach to policy improvement, standard methods do not transfer directly to learned latent action spaces. We find that the structure of learned latent actions makes naive offline RL unreliable: directly maximizing value over latent actions or fitting a unimodal policy can produce latent actions that cannot be realized by physical execution, while the latent representation can merge physical controls that receive different rewards. To understand when latent-space optimization remains valid, we derive conditions under which policy improvement transfers to decoded execution and under which latent and physical optima align. Building on these insights, we propose LAPIS for goal-conditioned latent policy improvement, grounding value-based policy improvement in observed state–latent transitions and using a conditional flow policy to capture their multimodal structure. Across visual manipulation and navigation tasks, LAPIS consistently improves over latent-space baselines, while retaining strong performance on cube tasks even with only 1% action labels.

Primary area:
reinforcement learning
Keywords:
Latent Action Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_he27OpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

LAPIS learns a policy from videos with few action labels. It selects high-value examples, trains a flow policy to generate latent actions, and decodes them into robot commands. All experiments are in simulation.

Critical strengths

s1. The paper studies whether improvements in latent actions translate into better execution.

s2. The ablations help identify which components improve performance.

Critical weaknesses

w1. There are no real-robot experiments, despite the paper’s focus on whether latent actions can actually be executed.

w2. The theory assumes an exact decoder and actions restricted to the dataset. The actual method does not satisfy these assumptions.

w3. Selecting training examples does not guarantee that the policy’s generated latents are executable. Codebook projection does not guarantee this either.

w4. The method mainly combines existing filtering, flow-policy, and CFG techniques. Its novelty beyond this combination is limited.

Questions for authors

q1. Could you test the method on a real robot to verify its claims about execution?

q2. How often do policy-generated latents fail to produce the intended outcome after decoding?

q3. Does stronger CFG increase these execution errors?

q4. Could you compare LAPIS with a flow-based GCIQL policy using the same latent model, decoder, and training budget? This would help isolate the benefit of your filtering rule.

5 comments
AuthorsOpenReview ↗

Response to Reviewer he27 (1/3): real robots and the theory's assumptions

We thank the reviewer for focusing on the central question of the paper: whether latent actions selected by the policy are actually executed. Changes in the revised PDF are shown in blue.

w1 / q1. Real-robot experiments. We agree that real-robot validation is valuable; Appendix G.2 (Limitations) of the submission states that real-robot deployment is outside the scope of the current evaluation. Our claims are comparative: judging them requires comparing several critics and extraction rules on the same LAM and decoder, with multiple baselines under the same data and evaluation protocol, across tasks and seeds. Such controlled, repeated comparisons are still hard to run on real robots, so we focused on the standard benchmarks of offline goal-conditioned RL (OGBench visual tasks, D4RL Kitchen), and within this setting we verified the execution claim directly (q2/q3 below). Moving this improvement rule onto real robots, building on the imitation results of real-robot latent-action work, is the natural next step.

w2. The theory assumes an exact decoder and actions restricted to the dataset. Thank you for raising this point. Theorems 1 and 2 are indeed stated for the batch-exact decoder and batch-constrained policies, but neither assumption is specific to LAPIS. Existing latent-action pipelines (Schmidt & Jiang, 2024; Ye et al., 2025; Chen et al., 2024; Bruce et al., 2024) train latent policies by behavior cloning and decode latents into actions with a small labeled set; without stating a theory, they rely on the same two conditions to reproduce the data's behavior.

  • Restricted to the dataset. Latent BC outputs latents from the observed qB(z∣s)q_{\mathcal{B}}(z \mid s), and its decoder is trained only on such latents, which keeps decoding within its training range.
  • Exact decoder. With the batch-exact decoder, ∑zΨ∗(a∣s,z),qB(z∣s)=πB(a∣s)\sum_z \Psi^{\ast}(a \mid s,z)\\, q_{\mathcal{B}}(z \mid s) = \pi_{\mathcal{B}}(a \mid s), so decoded latent BC equals BC in action space exactly. An inexact decoder deviates by the mismatch terms of Proposition 2 (App. D.3), which bounds the return gap for any decoder, including the learned Ψ(a∣z)\Psi(a \mid z); App. D.4 shows that the decoder need not reproduce the recorded action, only realize the represented outcome.

What is specific to offline RL is that policy improvement shifts the policy away from qB(z∣s)q_{\mathcal{B}}(z \mid s), and methods that maximize the critic can push latents outside the data, where the decoder receives unseen inputs and execution can break. Theorems 1 and 2 identify when improvement still carries over to execution, and this condition is the design principle of LAPIS: it scores only observed transitions with the critic and trains the policy on the selected ones, which keeps the decoder in the same regime as in BC. We measured how far it departs (q2/q3 below).

AuthorsOpenReview ↗

Response to Reviewer he27 (2/3): execution failures, guidance strength, codebook projection

w3 / q2 / q3. Thank you for these questions, which go to the core of the paper. We agree that selecting training examples and codebook projection do not by themselves guarantee execution; Appendix D.8 says the same. We therefore measured it.

Measurement. A latent zz encodes an intended transition, so its execution fails when the executed transition encodes to a different latent. The submitted paper measures this round-trip error as droundd_{\mathrm{round}} in Fig. 3, averaged over decision steps on antmaze-medium. To report how often execution fails, we turn it into a failure rate on cube-single and scene (4 seeds; 5 goals × 20 episodes per task). At every decision point we save the simulator state, execute the 3-step chunk decoded from the latent zz chosen by the policy, re-encode the start observation and the observation 3 steps later, and record ρ=∥I(ot,o~t+3)−z∥2\rho = \lVert I(o_t, \tilde o_{t+3}) - z \rVert_2. We then restore the saved state and continue the episode unchanged. The reference is the ρ\rho distribution of dataset latents, decoded and executed the same way from dataset states. A latent fails if its ρ\rho exceeds the 95th percentile of this reference, so dataset latents fail 5% of the time by construction; the threshold is set per task and seed, since each seed has its own LAM. We compare LAPIS (w=3w = 3, the reported setting), the same LAPIS checkpoints without codebook snapping, a max-Q method (CRL-style; Eysenbach et al., 2022), and AWR-weighted flow (Peng et al., 2019). The tables are Tables 19–20 in the new App. E.9 of the revision.

Execution failure rate (%), 4 seeds:

settingcube-singlescene
dataset latents (reference)5.05.0
LAPIS (w=3w = 3)5.812.6
AWR-weighted flow7.115.4
max-Q (CRL-style)6.724.0
  • q2. LAPIS policy latents fail 5.8% of the time on cube-single, close to the dataset reference, and 12.6% on scene, the lowest among the compared methods; max-Q fails about twice as often there (24.0%).
  • q3. Fig. 3 already reports this for guidance strength: droundd_{\mathrm{round}} for LAPIS from the BC end (w=0w = 0) to the aggressive end (w=30w = 30) on antmaze-medium, at w∈0,1,3,10,30w \in \\{0, 1, 3, 10, 30\\}. Up to w=3w = 3, the value we use, it stays at the w=0w = 0 level (6.1 at w=0w = 0, 1 and 3; BC 6.0). Much stronger guidance does increase it (6.2 at w=10w = 10, 6.5 at w=30w = 30), and success drops at w=30w = 30 (success panel of Fig. 3).
  • w3. The low failure rate is not produced by codebook projection either. We evaluated the same checkpoints with and without snapping (LAPIS w=3w = 3, 4 seeds):
cube-singlescene
failure rate (%), with / without snap5.8 / 5.812.6 / 12.4
success difference (without − with, points)−2.3 ± 4.0+1.5 ± 6.5

Without snapping, the failure rate is unchanged and the success difference is within seed variation (paired by seed). Both executability and performance come from the latents the policy generates.

AuthorsOpenReview ↗

Response to Reviewer he27 (3/3): novelty and comparison with flow-based GCIQL

w4. The method mainly combines existing filtering, flow-policy and CFG techniques. Thank you for this comment. We agree that the individual components exist: the gate belongs to the family of advantage-filtered regression (CRR; Wang et al., 2020), and flow actors and guidance are existing techniques. The revision adds the missing citations for flow matching and classifier-free guidance (Sec. 6). Our contribution has three parts.

  1. Why these components are needed in a learned latent action space. Which latents can be realized depends on the state. BC only imitates observed latents, so this causes no problem. Offline-RL improvement, however, moves away from the observed distribution and selects latents that are unrealizable at the current state. This difficulty does not exist for physical actions.
  2. An analysis that identifies the condition. We show that latent improvement carries over exactly to execution when the policy stays within observed state–latent transitions (Theorems 1 and 2; Proposition 2 for learned decoders). The gate, which selects only observed transitions, and the flow actor, which represents their distribution, follow from this condition.
  3. Controlled experiments showing what is decisive: changing the extraction rule under the same critic, and changing the critic under the same rule (table below).

With the same extraction rule, replacing the critic lowers performance substantially (first two rows below), so a simple combination of the components does not work.

q4. Comparison with a flow-based GCIQL policy using the same latent model, decoder and training budget. Thank you for the suggestion. We ran this comparison: same LAM, decoder, 5% labels, training budget and evaluation protocol; mean ± std over 4 seeds.

methodantmaze-mediumcube-singlecube-doublescene
LAPIS (TD-InfoNCE critic + gate + flow)49±362±531±629±4
IQL critic + gate + flow (LAPIS actor)32±442±1412±420±3
IQL critic + AWR-weighted flow (Flow-GCIQL)18±455±813±419±4
IQL critic + DDPG+BC flow38±76±45±19±1

The critics are TD-InfoNCE (Zheng et al., 2024) and IQL (Kostrikov et al., 2022); the flow-actor extractions are AWR (Peng et al., 2019) and the one-step DDPG+BC of FQL (Park et al., 2025).

  • LAPIS outperforms the flow-based GCIQL the reviewer suggested (IQL critic + AWR-weighted flow) on all four tasks.
  • With the same flow actor and the same gate, replacing only the critic with IQL lowers performance on all four tasks.
  • The other flow-based GCIQL variant (DDPG+BC flow) is also below LAPIS on all four tasks.
  • The full grid, varying both the critic and the extraction rule, is Table 4(b) in Sec. 7.4 of the revision.

References

  • Bruce et al. (2024). Genie: Generative interactive environments. ICML.
  • Chen et al. (2024). IGOR: Image-GOal representations are the atomic control units for foundation models in embodied AI. arXiv:2411.00785.
  • Eysenbach, Zhang, Levine, Salakhutdinov (2022). Contrastive learning as goal-conditioned reinforcement learning. NeurIPS.
  • Kostrikov, Nair, Levine (2022). Offline reinforcement learning with implicit Q-learning. ICLR.
  • Park, Li, Levine (2025). Flow Q-learning. ICML.
  • Peng, Kumar, Zhang, Levine (2019). Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv:1910.00177.
  • Schmidt, Jiang (2024). Learning to act without actions. ICLR.
  • Wang et al. (2020). Critic regularized regression. NeurIPS.
  • Ye et al. (2025). Latent action pretraining from videos. ICLR.
  • Zheng, Salakhutdinov, Eysenbach (2024). Contrastive difference predictive coding. ICLR.
Reviewer_wuZeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies offline policy improvement in latent action spaces learned from action-free trajectories. It separates two obstacles: a selected latent action may not be faithfully realized through decoding and physical execution, and outcome-based action abstraction may discard reward-relevant distinctions. Using paired physical-action and latent-action batch MDPs, the paper gives conditions under which improvement transfers to decoded execution and optimal policies align within the batch-constrained policy classes.

LAPIS uses a TD-InfoNCE critic to filter observed state–latent transitions and trains a conditional flow policy on both the full and filtered data, with classifier-free guidance at inference. The latent action model and policy use action-free data, while the physical-action decoder is trained with limited action labels. Experiments cover visual OGBench manipulation and navigation, as well as visual Franka Kitchen, and examine label budgets, filtering, policy expressiveness, decoder conditioning, and temporal horizons.

Critical strengths
  1. The distinction between latent optimization and physical execution is valuable. A high-value latent action is useful for control only when decoding realizes the corresponding transition. Theorems 1 and 2 separate execution consistency from the preservation of reward-relevant distinctions, and restrict the optimality claim to the batch-constrained policy classes. Appendix D further discusses the departures introduced by learned representations, decoding, and the practical control interval. This analysis helps identify when improvement measured in latent space can be trusted.

  2. The controlled ablations and scarce-label results provide convincing evidence. Table 4 varies filtering and actor parameterization while holding the LAM, decoder, critic, and training budget fixed. On Cube-single, replacing max-QQ optimization with filtering raises success from 4±2%4\pm2\% to 36±6%36\pm6\% with a deterministic actor; adding the flow actor raises it to 62±5%62\pm5\%. Scene and AntMaze-medium show the same direction of improvement. In Table 3, Cube-single reaches 50±6%50\pm6\% with only 1%1\% action-labeled trajectories, compared with 2±4%2\pm4\% for the best reported baseline. Table 13 also shows that most of the gains on manipulation tasks are already present at w=1w=1, with stronger guidance accounting for only part of the improvement.

Critical weaknesses
  1. The incremental benefit of the particular extraction rule is not fully isolated. The paper attributes some GCIQL failures to a unimodal actor averaging distinct high-value modes, but the flow baseline in Table 4 still uses direct max-QQ optimization. Methods such as IDQL [1] address multimodal policy extraction through samples from a generative behavior model and critic-based reweighting. A comparison in the same latent space, or goal-conditioned IQL with an expressive actor, would help isolate what the LAPIS filtering rule adds. The value-free CFGRL comparison in Table 14 tests the role of guidance itself, but does not resolve this question. This limits my confidence in the algorithmic increment.

  2. The basis for comparing implemented critic scores across actions remains unclear. Appendix C.4 motivates filtering using an ideal binary density-ratio score, whereas Eq. (11) trains TD-InfoNCE with a softmax over candidate goals. At the score-function level, the transformation $ f'(s,z,g)=f(s,z,g)+c(s,z) $ leaves this categorical objective unchanged, while an offset that depends on zz can change action rankings at fixed (s,g)(s,g) and hence the raw-score gate in Eq. (4). The related normalization issue is also discussed in the TD-InfoNCE paper [2]. Appendix D.7 explicitly requires value ordering, but does not establish how practical training obtains it. The normalized bilinear parameterization restricts the score family; whether it provides the required cross-action comparability needs clarification. This leaves a gap between the practical filtering update and its theoretical justification.

[1] Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies. arXiv:2304.10573, 2023.

[2] Chongyi Zheng, Ruslan Salakhutdinov, and Benjamin Eysenbach. Contrastive Difference Predictive Coding. ICLR, 2024.

Questions for authors
  1. With the same frozen LAM, decoder, action-label budget, and evaluation protocol, how does LAPIS compare with goal-conditioned IQL using multimodal policy extraction? A limited comparison on the representative environments in Table 4, using a flow actor or IDQL-style extraction, would affect my assessment of the contribution of the specific filtering rule.

  2. What makes the TD-InfoNCE scores used in Eq. (4) comparable across latent actions? Does the normalized bilinear parameterization resolve the offset issue above, or is there another constraint or normalization in the implementation? If value ordering is primarily an empirical property, can the authors provide evidence for it in the trained critic?

5 comments
AuthorsOpenReview ↗

Response to Reviewer wuZe (1/2): isolating the extraction rule

We thank the reviewer for appreciating the separation between latent optimization and physical execution and the controlled ablations. Changes in the revised PDF are shown in blue.

W1 / Q1. Isolating the contribution of the extraction rule (IDQL-style extraction, goal-conditioned IQL with an expressive actor). Thank you for this suggestion. In Table 4(a), the deterministic and flow actors under the same gate (cube-single 36 vs. 62) reflect the limitation of a unimodal actor, but, as the reviewer notes, this does not isolate the extraction rule. We therefore added a critic × extraction-rule grid in the same latent space. The LAM, decoder, labels and evaluation protocol are shared, and the critic and the extraction rule are varied independently (5% labels, 4 seeds, average over the 300/400/500K checkpoints).

criticextraction ruleantmaze-mediumcube-singlecube-doublescene
TD-InfoNCE (LAPIS)gate + flow (LAPIS)49±362±531±629±4
TD-InfoNCEAWR-weighted flow57±1254±625±420±4
TD-InfoNCEIDQL-style selection (1 of 32)89±433±810±216±3
IQL (expectile 0.9)gate + flow32±442±1412±420±3
IQLAWR-weighted flow (Flow-GCIQL)18±455±813±419±4
IQLIDQL-style selection (1 of 32)66±528±85±25±2
IQLDDPG+BC flow38±76±45±19±1

AWR-weighted flow applies the advantage weights of AWR (Peng et al., 2019) to the flow actor, and DDPG+BC flow follows the one-step scheme of FQL (Park et al., 2025). The last three IQL rows are the goal-conditioned IQL with an expressive actor that the reviewer suggested.

  • (i) Same rule, different critic. Replacing the TD-InfoNCE critic by IQL lowers performance in 10 of the 12 cells and leaves the other two similar.
  • (ii) Same critic, different rule. With the TD-InfoNCE critic, the gate is best on the three manipulation tasks, whereas IDQL-style selection and AWR-flow are higher on antmaze-medium. The gate only reweights observed transitions, whereas selection picks the best of 32 behavior-branch samples at test time, which is less conservative. Long-horizon navigation rewards less conservative extraction: with physical actions and full labels, where realizability is not an issue, CRL, which maximizes the critic directly, also outperforms LAPIS on AntMaze, more so on the larger mazes (Table 1, App. F.1).
  • (iii) Direct Q maximization (DDPG+BC flow) is among the lowest on manipulation, consistent with the max-Q cells of Table 4(a).

Overall, what matters most in the latent space is the critic and an extraction grounded in observed transitions (or behavior-branch samples); between the gate and selection, the better choice depends on the domain. The full table is Table 4(b) in Sec. 7.4 of the revision.

AuthorsOpenReview ↗

Response to Reviewer wuZe (2/2): are TD-InfoNCE scores comparable across ?

W2 / Q2. Thank you for this precise observation; the reviewer is right.

  • Both cross-entropy terms in Eq. (11) are softmaxes over candidate goals for the same row (s,z)(s,z), so the optimum is identified only up to a per-(s,z)(s,z) constant c(s,z)c(s,z). This is the "arbitrary function" c(⋅)c(\cdot) of the Bayes-optimal InfoNCE critic in Sec. 3.1 of Zheng et al. (2024).
  • The normalized bilinear form does not remove it. L2 normalization bounds the scores, but a direction shared by all goal features ψ(g)\psi(g) still contributes a term that depends only on (s,z)(s,z), which acts as c(s,z)c(s,z). Neither the ensemble minimum nor the expectile baseline V^(s,g)\hat V(s,g) used in the gate depends on zz, so neither cancels c(s,z)c(s,z).
  • Order preservation is therefore an empirical property, not one guaranteed by training. The revision states this in App. D.7 and removes from it the sentence saying that the practical gate inherits Corollary 1 and Proposition 1.

What we added:

(a) An offset-free score. Define fˉ(s,z,g)=f(s,z,g)−log⁡Eg′∼pc[exp⁡f(s,z,g′)],\bar f(s,z,g) = f(s,z,g) - \log \mathbb{E}_{g' \sim p_c}\left[\exp f(s,z,g')\right], where pcp_c is the critic's candidate-goal distribution. At the optimum, fˉ⋆(s,z,g)=log⁡Qμ(s,z,g)−log⁡pc(g)\bar f^{\star}(s,z,g) = \log Q^{\mu}(s,z,g) - \log p_c(g), so cc cancels and, for fixed (s,g)(s,g), the order over zz matches QμQ^{\mu}. In the analysis, the log-partition is computed with the in-batch candidate goals, the same quantity as the importance-weight normalization of TD-InfoNCE. With fˉ⋆\bar f^{\star} in place of f⋆f^{\star}, Remark 2, Corollary 1 and Proposition 1 hold as stated (new Remark 3, App. D.7). The practical gate keeps the raw score ff. Gating with fˉ\bar f requires a per-(s,z)(s,z) log-partition estimate, and the in-batch log-sum-exp is a finite-sample estimate with bias and noise. We therefore use fˉ\bar f to state order preservation in the analysis; (b) shows that the raw score largely preserves its order.

(b) Evidence in the trained critic. In trained LAPIS critics (4 seeds), we compared the raw score ff used by the gate with the offset-free score fˉ\bar f over 16 candidate latents per training state (the dataset latent and 15 behavior-branch samples), estimating the log-partition of fˉ\bar f with 16,384 goals.

antmaze-mediumcube-singlescene
Spearman correlation of ff and fˉ\bar f over candidates0.790.860.87
agreement of keep/drop decisions, same threshold rule on ff and fˉ\bar f87%91%91%

Standard deviations across seeds are below 0.01 for the Spearman correlation and below 1% for the agreement. The threshold rule keeps the dataset latent if its score exceeds the 0.9-expectile over the sampled candidates, as a stand-in for V^\hat V. Within a state, the raw score largely preserves the order of the offset-free score, and the gate keeps or drops the same transitions about 90% of the time. For gating, the raw score is thus a practical surrogate for fˉ\bar f that needs no log-partition estimate. This comparison is now Sec. 7.6 and Table 6 of the revision, with the protocol in App. D.7.

References

  • Park, Li, Levine (2025). Flow Q-learning. ICML.
  • Peng, Kumar, Zhang, Levine (2019). Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv:1910.00177.
  • Zheng, Salakhutdinov, Eysenbach (2024). Contrastive difference predictive coding. ICLR.

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

Reviewer_NDNoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers offline policy improvement in a latent action space learned from action-free video. A latent action model relabels transitions with codes z = I(s,s′), and a decoder trained on a small amount of action labels maps these codes back to physical actions. The authors identify three structural properties of such spaces (Section 4.1) and derive two results: that on-support latent improvement transfers to decoded execution under deterministic dynamics (Theorem 1), and that latent and physical optima align when the reward is determined by the transition outcome (Theorem 2). The proposed method, LAPIS, scores observed state-latent transitions with a goal-conditioned contrastive critic, keeps the ones above a learned threshold, fits a two-branch flow policy, and blends the branches with classifier-free guidance at inference. Evaluation is done on the OGBench visual suite and a rendered Franka Kitchen under 100%, 5% and 1% action-label budgets.

Critical strengths

To isolate what makes latent policy improvement work, the authors cross the improvement mechanism with the actor parameterization, while keeping the latent action model, the decoder and the training budget fixed (Table 4). Replacing direct value maximization with filtering over observed transitions moves cube from 4 to 36 and antmaze from 11 to 43, and this already happens with a deterministic actor, before the flow policy is introduced. The same preference is reproduced in two other places. The guidance construction improves physical-action performance on antmaze but not the latent one (Table 14), and the round-trip realization error grows together with the drop in success as the critic is given more weight (Figure 3).

Note that this is a negative result about a pipeline a lot of recent works build upon, and it is supported by three different measurements rather than one. Many latent action model papers default to behavioral cloning for the policy extraction, and the natural alternative – maximizing a learned critic over latents – is shown here to fail in a way that is specific to the learned action space. The 1%-label numbers point in the same direction (Table 3). This is the part of the submission I would like to see published.

Critical weaknesses

The analysis does not cover the system that is evaluated. Definition 1 introduces the realization error that motivates the entire narrative, and it is not used by any result in Appendices C and D. The theory is stated for a one-step latent, while all experiments utilize H=3 with one-step replanning, so the process in Theorem 1 is not the one the controller executes. Appendix D.6 notices this and defines the appropriate one-step object, but then defers to Appendix D.3, where the mismatch is defined against the original quantity. Note, that the properties themselves are also not new: state-conditioned realizability is stated in LAPO (Schmidt & Jiang, 2024), which the submission cites, and the decoder behaviour reported in Table 5 and Appendix F.3 is described in Section 6.2 of the same work. Meanwhile, the title promises suboptimal data, although data quality is kept fixed and only the action-label budget is varied.

If the apparatus is set aside, the empirical finding described above remains, however the comparisons that would secure it are not yet in place. The fully supervised block of Table 1 omits HIQL, which is the best-performing method on these tasks in Park et al. (2025), and whose reported numbers are higher than the best LAPIS number in the paper on 7 of 9 tasks. Moreover, Table 3 is the only comparison against methods that also use action-free data, it is restricted to four cube tasks, and two of its four baselines are not the methods they are named after. The entry called GCIQL is defined without a Q-function (Appendix E.4) and returns zero in all eight cells, and the HIQL entry drops the latent subgoal representation that was introduced in Park et al. (2023) exactly for pixel-based tasks. Therefore, it is not enough to report that the competing methods collapse at 1% labels, since it is unclear how much of the collapse is due to the budget and how much to the adaptation.

Questions for authors
  1. Could the authors add HIQL, GCIVL and QRL to Table 1, and restate the claims depending on that block, including the sentence on full supervision in the introduction?

  2. Does the HIQL entry in Table 3 utilize latent subgoal representations? If not, I would ask to re-run it, as this baseline carries the 1%-label claim together with the GCIQL one.

  3. The GCIQL column in Table 1 matches Park et al. (2025) on all nine means and all nine standard deviations, although only GCBC is reported as taken from there. What is its provenance? Reporting it as imported would prevent the ambiguity.

  4. Is there a version of Theorem 1 for the one-step successor distribution defined in Appendix D.6? As written, neither the exact nor the approximate result applies to the deployed controller.

  5. Why is w=3 used for the reported results, when w=1 is the setting the analysis covers and performs better on antmaze-medium (Table 13)?

Also, probably the naming of the paper is too general and overselling. There were a lot of work on latent action spaces (not in the sense of the LAPO) in the classical offline-rl (not goal-conditioned) which is confusing a lot.

6 comments
AuthorsOpenReview ↗

Response to Reviewer NDNo (1/4): Definition 1 and the deployed controller

We thank the reviewer for a careful and precise review. We agree with the empirical finding the reviewer highlights: in learned latent action spaces, direct value maximization fails, while policy improvement grounded in observed transitions works. Changes in the revised PDF are shown in blue.

We answer in four comments: W1(a)–(c) and Q4 here; W1(d), W1(e), Q2, Q3 and Q5 in (2/4); W2 and Q1 on Table 1 and the claims in (3/4); and W2 on Table 3 and the label budget in (4/4).

W1(a). Definition 1 is not used by any result. Thank you for pointing this out; you are right that no result in Appendices C and D used Definition 1, and we should have made its role clearer. Definition 1 identifies the failure mode: a latent with δ(s,z)>0\delta(s,z) > 0 has no exact physical realization at ss. The theorems characterize the regime that avoids it. They are stated for batch-constrained policies, which select only latents of transitions realized in the data at the same state, so δ(s,z)=0\delta(s,z) = 0 holds by construction, and Theorem 1 shows that latent improvement then transfers exactly to execution.

The revision adds Proposition 3 (new App. D.9), which extends Theorem 1 to policies that are not batch-constrained and bounds their transfer gap in terms of δ\delta. Let MF\mathcal{M}^F be the latent process that moves from (s,z)(s,z) to the predicted successor F(s,z)F(s,z). Under Assumptions 1–2, a forward model accurate up to ϵF\epsilon_F on reachable outcomes, and Lipschitz conditions standard in latent and model-based RL (Asadi et al., 2018; Gelada et al., 2019), for any latent policy πz\pi_z and decoder Ψ^\widehat\Psi, ∣JMF(πz)−JM(Ψ^∘πz)∣≤(Lr+γLV)[LF∑t≥0γt,E[δ(st,zt)+ϵΨ^(st,zt)]+ϵF1−γ],\left| J_{\mathcal{M}^F}(\pi_z) - J_{\mathcal{M}}(\widehat\Psi \circ \pi_z) \right| \le (L_r + \gamma L_V) \left[ L_F \sum_{t \ge 0} \gamma^t\\, \mathbb{E}\left[\delta(s_t,z_t) + \epsilon_{\widehat\Psi}(s_t,z_t)\right] + \frac{\epsilon_F}{1-\gamma} \right], where ϵΨ^≥0\epsilon_{\widehat\Psi} \ge 0 is the decoder's round-trip error in excess of δ\delta. The bound thus grows linearly with the realization error of the selected latents. The same argument extends Proposition 3 to HH-step latents, with the HH-step transition and discount γH\gamma^H (App. D.9).

W1(b)(c) and Q4. The analysis does not cover the deployed controller; is there a version of Theorem 1 for the object of Appendix D.6? Thank you for this suggestion; we agree. Theorem 1 assumes a one-step latent, whereas the controller selects an H=3H = 3 endpoint latent, executes only the first decoded action, and replans (E=1E = 1). The revision adds Theorem 1′ for the object pB,H(1)p^{(1)}_{\mathcal{B},H} of Appendix D.6 (statement and proof in the new App. C.5).

Setting. At each step tt of the logged trajectories, count (st,at,st+1,zt)(s_t, a_t, s_{t+1}, z_t) with zt=I(st,st+H)z_t = I(s_t, s_{t+H}), and let N(s,a,z,s′)N(s,a,z,s') be the counts. The HH-endpoint latent batch MDP MBHz\mathcal{M}_{\mathcal{B}^z_H} has

  • latent actions ZB,H(s)=z:N(s,⋅,z,⋅)>0Z_{\mathcal{B},H}(s) = \\{z : N(s,\cdot,z,\cdot) > 0\\},
  • transitions pB,H(1)(s′∣s,z)=N(s,z,s′)/N(s,z)p^{(1)}_{\mathcal{B},H}(s' \mid s,z) = N(s,z,s') / N(s,z),
  • rewards rB,H(1)(s,z)=∑a,s′N(s,a,z,s′),r(s,a,s′)/N(s,z)r^{(1)}_{\mathcal{B},H}(s,z) = \sum_{a,s'} N(s,a,z,s')\\, r(s,a,s') / N(s,z),

and the first-action batch decoder is ΨH∗(a∣s,z)=N(s,a,z)/N(s,z)\Psi^{\ast}_H(a \mid s,z) = N(s,a,z) / N(s,z).

Theorem 1′. Under deterministic dynamics (Assumption 1), every latent policy πz\pi_z that is batch-constrained in MBHz\mathcal{M}_{\mathcal{B}^z_H} satisfies JMBHz(πz)=JM(ΨH∗∘πz).J_{\mathcal{M}_{\mathcal{B}^z_H}}(\pi_z) = J_{\mathcal{M}}(\Psi^{\ast}_H \circ \pi_z). Supported-outcome identifiability is not needed, since pB,H(1)(⋅∣s,z)p^{(1)}_{\mathcal{B},H}(\cdot \mid s,z) may be stochastic. H=1H = 1 recovers Theorem 1.

Proof sketch. With s′=T(s,a)s' = T(s,a), N(s,a,z,s′)=N(s,a,z),1[s′=T(s,a)]N(s,a,z,s') = N(s,a,z)\\,\mathbf{1}[s' = T(s,a)]. The decoded transition is ∑aΨH∗(a∣s,z),1[T(s,a)=s′]=N(s,z,s′)/N(s,z)=pB,H(1)(s′∣s,z)\sum_a \Psi^{\ast}_H(a \mid s,z)\\,\mathbf{1}[T(s,a) = s'] = N(s,z,s')/N(s,z) = p^{(1)}_{\mathcal{B},H}(s' \mid s,z), and the same computation gives the decoded reward. Batch-constraint ensures N(s,z)>0N(s,z) > 0 at every visited pair, so both processes share transitions, rewards and policy, and hence returns.

Correspondence to LAPIS. The critic bootstraps from (st,zt)(s_t, z_t) to (st+1,zt+1)(s_{t+1}, z_{t+1}) with zt+1=I(st+1,st+4)z_{t+1} = I(s_{t+1}, s_{t+4}), so it evaluates latent policies in exactly this MDP. We also rewrote D.3 with pB,H(1)p^{(1)}_{\mathcal{B},H} as the reference kernel, so Proposition 2, whose proof does not depend on the kernel, covers learned decoders Ψ(a∣z)\Psi(a \mid z); D.6 now points to both results, which therefore apply to the deployed controller.

AuthorsOpenReview ↗

Response to Reviewer NDNo (2/4): novelty with respect to LAPO, naming, and Q2, Q3, Q5

W1(d). The properties are not new (LAPO). Thank you for pointing us to these passages. We agree that LAPO already notes that a latent need not be state-invariant: the same zz can correspond to different actions in different states (Sec. 4.1), so a decoder that does not see the state may fail (Sec. 6.2). We should have cited this in our Sec. 4.1, and the revision does. Our point is that this property is largely harmless in LAPO's setting but becomes the central obstacle once policy improvement is added.

In latent behavior cloning, as in LAPO, it is largely harmless. BC imitates the observed qB(z∣s)q_{\mathcal{B}}(z \mid s), so every latent it outputs was realized at that state. The only cost is decoding accuracy, which a state-conditioned decoder or more labels reduce.

In offline RL, it becomes the central obstacle. Policy improvement shifts the policy away from qB(z∣s)q_{\mathcal{B}}(z \mid s), and common mechanisms (critic maximization, mode averaging, guidance extrapolation) can also leave its support: critics that generalize across states can prefer a latent that was realized elsewhere but is unrealizable at the current state. Physical actions are executable in every state; realizable latents are not, and their set is state-dependent, non-convex and only partially observed, so the usual support constraint of offline RL no longer has a simple form. Table 1 shows the consequence: the same CRL reaches 95 on antmaze-medium with physical actions but 11 in the latent space, and in Fig. 3 its round-trip error grows as its BC weight is removed.

Identifying this failure in offline RL and designing improvement that stays on observed transitions are our contribution.

Table 5 and App. F.3. Our analysis makes precise when this matters. Transfer is exact with the state-conditioned batch decoder Ψ∗(a∣s,z)\Psi^{\ast}(a \mid s,z) (Theorem 1). More generally, transfer needs a decoder that realizes the latent's outcome at each visited state (App. D.4). A latent-only decoder Ψ(a∣z)\Psi(a \mid z) does so only when the action realizing zz does not depend on ss, which is exactly what LAPO's observation questions; otherwise Proposition 2 (App. D.3) bounds the gap by the transition mismatch it induces. LAPIS uses Ψ(a∣z)\Psi(a \mid z), and Table 5 measures this gap, a comparison not reported in LAPO: the state-conditioned decoder is better with 100% labels, as the analysis predicts, while the latent-only decoder is better with 5% and 1% labels, which is why LAPIS uses it. The revision states this and cites LAPO (Sec. 7.5 and App. F.3).

W1(e) and the comment on naming. Thank you; we agree with both comments and will change the title in the final version.

Q2. Does the HIQL entry in Table 3 use latent subgoal representations? Thank you for the question. Yes: all 72 HIQL runs in Table 3 use the learned subgoal representation of the OGBench HIQL implementation, which the high-level actor predicts and the low-level actor conditions on. App. E.4 omitted this description; the revision adds it (App. E.4).

Q3. Provenance of the GCIQL column in Table 1. Thank you for noticing this. In the submission, the nine fully supervised GCIQL entries were the values reported by Park et al. (2025, Table 2); they should have been marked as imported, as GCBC was, and we apologize for the omission. In the revision, we reran them: every entry of Table 1, including the fully supervised GCBC and GCIQL columns, now comes from our own runs with the OGBench implementations and the same evaluation protocol as the other methods (App. E.4).

Q5. Why w=3w = 3, when w=1w = 1 is the setting the analysis covers? Thank you for this question. We agree that w=1w = 1 is the reference point of the analysis (App. D.8). We report w=3w = 3 because, in Table 14, it matches or exceeds w=1w = 1 on 8 of the 9 tasks; the only lower case is antmaze-medium (51 vs. 49), within one standard deviation. Most of the improvement is already present at w=1w = 1, so the method also works in the setting the analysis covers.

task (5% labels)w=0w = 0w=1w = 1w=3w = 3
antmaze-medium30±651±549±3
antmaze-large7±28±210±2
antmaze-giant0±00±00±0
antmaze-teleport14±416±421±3
cube-single13±153±362±5
cube-double13±129±431±6
cube-triple18±120±120±0
cube-quadruple10±413±314±3
scene18±325±429±4

References

  • Asadi, Misra, Littman (2018). Lipschitz continuity in model-based reinforcement learning. ICML.
  • Gelada, Kumar, Buckman, Nachum, Bellemare (2019). DeepMDP: Learning continuous latent space models for representation learning. ICML.
AuthorsOpenReview ↗

Response to Reviewer NDNo (3/4): HIQL, GCIVL and QRL in Table 1 (W2, Q1)

W2 / Q1. HIQL, GCIVL and QRL in Table 1, and the claims. Thank you for this important point; we agree that HIQL in particular should have been included. The revised Table 1 is below, and every entry is from our own runs (four seeds, mean ± std, the same evaluation protocol for every method). For the revision, we newly ran GCIVL, QRL and HIQL in both blocks and LAPIS†, and reran the fully supervised GCBC and GCIQL, which the submission had taken from Park et al. (2025); the other entries are our runs from the submission. Bold marks the best value in each row of each block. Results with 5% labels without the LAM are in App. E.10.

(a) Oracle: 100% labels, physical actions (a reference with full action supervision)

taskGCBCF-GCBCGCIVLGCIQLQRLHIQLCRLLAPIS
antmaze-medium12±08±123±312±20±094±295±248±3
antmaze-large4±21±06±13±11±166±385±314±3
antmaze-giant0±00±01±10±01±18±547±10±0
antmaze-teleport5±07±18±17±110±442±451±123±1
cube-single4±211±258±830±728±1486±618±1045±4
cube-double1±15±310±32±15±138±51±030±6
cube-triple14±219±016±216±216±121±113±322±1
cube-quadruple5±111±30±07±14±111±34±116±1
scene13±220±325±412±214±243±613±345±1

(b) 5% labels, the same LAM for every method (LAPIS† keeps the LAPIS critic but replaces the gate with IDQL-style selection)

taskF-GCBCGCIVLGCIQLQRLHIQLCRLLAPISLAPIS†
antmaze-medium8±126±57±234±489±211±249±389±4
antmaze-large1±06±22±19±349±43±110±258±6
antmaze-giant0±01±10±00±03±20±00±04±2
antmaze-teleport6±16±24±114±734±36±121±339±6
cube-single4±145±35±25±456±64±262±533±8
cube-double4±19±41±11±118±91±031±610±2
cube-triple16±116±112±612±517±211±420±010±4
cube-quadruple10±30±03±14±14±24±214±31±1
scene9±217±26±17±122±76±429±416±3
  • With the same LAM and 5% labels, LAPIS is best on all five manipulation tasks.
  • In the 100%-label oracle, HIQL is the strongest method on cube-single and cube-double, LAPIS on the other three manipulation tasks, and CRL on antmaze. Compared with LAPIS trained on physical actions with 100% labels, LAPIS with the LAM and 5% labels loses little performance on every task except scene.

Why HIQL is stronger on antmaze, and why we added LAPIS†. On antmaze, LAM+HIQL exceeds LAPIS. HIQL is designed for long-horizon goal reaching: its high-level policy proposes intermediate subgoals, and its low-level policy only has to reach them. The LAPIS gate, in contrast, is conservative by design: the policy is trained only on observed state–latent transitions that the critic rates above the baseline. This protects execution in manipulation, but long-horizon navigation favors less conservative extraction. The oracle block shows the same trade-off with physical actions: CRL, which maximizes the critic directly, outperforms LAPIS on all four antmaze tasks (App. F.1).

To check whether the antmaze gap comes from this extraction rule rather than from the LAPIS critic, we ran LAPIS†. It keeps the LAM, decoder and critic of LAPIS, and replaces the gate with IDQL-style selection: it samples 32 latents from the behavior branch of the flow policy and executes the one with the highest critic value, which is less conservative. LAPIS† matches or exceeds LAM+HIQL on all four antmaze tasks, so the gap comes from the extraction rule, not from the critic or the latent action space. LAPIS† is lower than LAPIS on all five manipulation tasks, so the better rule depends on the domain (Table 4(b)), and LAPIS keeps the gate.

Revised claims (Q1). We rewrote the claims to match these results. The abstract now reads: "LAPIS outperforms baselines on most tasks, with the clearest gains in visual manipulation, and remains effective even with only 1% action labels, where competing methods largely collapse." The introduction and the conclusion say the same, and Sec. 7.1 states the antmaze result above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.