acceptodds
Withdrawn from ICLR 2027

Latent Critic, Pixel Policy: Goal-Conditioned Control That Reuses a Frozen World Model Only in Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In offline goal-conditioned reinforcement learning, a learned value weights the imitation of dataset actions, so it must tell which of two nearby frames lies closer to a goal. Pretrained latent world models are released as frozen checkpoints, which current control methods query at test time to plan over predicted futures. We propose LCPP, which instead uses the frozen world-model encoder in a critic, a value on its latents from which advantage-weighted regression extracts a pixel policy that makes no world-model call. An analysis bounds the reliability of advantage weights by value errors at a state and its successor, including a floor from a frozen input map. At 100k updates on five environments, we compare LCPP with seven from-scratch methods and five methods that reuse the same frozen world model. LCPP is best or tied best in all five columns. In place of the latent value, a pixel value lowers Cube and Scene significantly across 20 seeds per arm, while a latent actor lowers them in each of five seeds. Code and demo are available at https://anonymous.4open.science/r/lcpp.

Primary area:
applications to robotics, autonomy, planning
Keywords:
offline goal-conditioned reinforcement learning, world models, frozen pretrained encoders, advantage-weighted regression, visual control, value learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EzVCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes an offline goal-conditioned RL algorithm called LCPP. LCPP involves a pre-trained latent world model. It trains a value function on top of latents and a policy directly from pixels. The authors show that their method outperforms previous approaches on the tasks used in LeWM.

Critical strengths
  • The experimental results seem reasonable.
Critical weaknesses
  • The paper is poorly written and is really difficult to follow. To be honest, the paper (unfortunately) looks entirely AI-generated to me. As a reference, Pangram says Intro is 100% AI-generated.
    • I believe Intro requires a major revision as it lacks a logical structure. For example, what's the point of the first paragraph? "We therefore treat judging progress and producing actions as two jobs that need not share an input, since the value serves training alone." -- What does this mean?
    • "The value answers the question of progress, so it reads the frozen latents rather than pixels (Figure 2, bottom)." -- Why? How are these two clauses logically connected?
  • The novelty of the paper is unclear and not discussed in the paper. How is it different from previous work? What's the contribution of this paper? To me, the method seems a minor variant of existing techniques, without a clear justification or explanation.
  • (Minor) The paper contains a few theoretical results, but the authors do not explain or contextualize them, and it is unclear how they help understand the proposed algorithm.
Questions for authors

Please see the weaknesses section.

Reviewer_UgBZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents an algorithm, LCPP---Latent Critic, Pixel Policy, for offline goal-conditioned, reward-free reinforcement learning (GCRL). By leveraging the pretrained encoder from a frozen, pretrained world model for critic learning, LCPP enables efficient policy learning without forwarding the pretrained world model. Additionally, a formal analysis explains the reliability of the chosen advantage weights for expectile regression value learning and policy extraction, supporting the idea of using the frozen encoder and decoupled learning under the specific assumptions. The authors mainly choose the stable-worldmodel benchmark for evaluating the performance of LCPP across from-scratch and frozen-encoder GCRL baselines.

Critical strengths
  1. Formal results support the design of separating the encoders of the value (critic) and policy (actor) via advantage-weighted expectile regression and a frozen encoder, under Assumption A and environmental assumptions (i.e., fixed goal distance from starting states).
Critical weaknesses
  1. Weak contribution arising from backbone selection and benchmarking results.

LCPP builds upon many prior approaches: expectile regression and advantage-weighted policy extraction (IQL, AWR); world model architecture (LeWM). Hence, the authors provide a formal investigation of extraction reliability, where the critic encoder is decoupled from the actor's encoder by reusing the frozen world model encoder (LeWM) for the critic while training the actor's encoder from scratch. However, this investigation assumes that the evaluation goals lie 50 transition steps from the initial state, which establishes the choice of hyperparameter (e.g., Equation 3, Remarks 2 and 3) used in the analyses, limiting the impact of the formal results on the specialized evaluation suite. Moreover, the benchmarking results from the stable-worldmodel benchmark are unreliable. By referring to Section 4.1 and the anonymous GitHub from the authors, I found that LCPP is trained and evaluated with the stable-worldmodel benchmark. LeWM evaluated performance with the offline dataset used for training to initialize the reachable pairs of the initial state and future goal. While it is common practice in offline GCRL, the stable-worldmodel benchmark does not filter out pairs that have already succeeded. This means that the agent will succeed at whatever it does since the environment is initialized in the successful states. To empirically demonstrate that, I compare the scores of the official LeWM paper, reproduced with the same evaluation setup, and reproduced with a success-filtered evaluation setup below:

Table 1: success rates w/ eval_goals setup (eval_budget=50, goal_offset=50, same episode and start indices)

taskofficial LeWM scorereproducedalready-succeeded pairsreproduced w/ filtering (n: num_envs=num_total - num_filtered)
PushT9632.0032.0 (n=50)
TwoRoom8756.0254.2 (n=48)
Cube7460.01444.4 (n=36)
Reacher8684.0183.7 (n=49)

Table 2: success rates w/ LeWM's setup (eval_budget=50, goal_offset=25, random episodes and starts, seeds 42/43/44)

taskofficial LeWM scorereproducedalready-succeeded pairs (across seeds)reproduced w/ filtering over seeds
PushT9690.00/0/090.0 ± 3.3
TwoRoom8785.33/6/483.9 ± 3.1
Cube7470.017/20/1951.9 ± 12.0
Reacher8686.70/0/086.7 ± 4.1

where I followed the same evaluation setup reported by the authors and LeWM's setup, and I computed averaged success rates over episodes by removing filtered, already-succeeded episodes. As a result, PushT and Reacher tasks do not involve already-succeeded pairs at all, while Cube and TwoRoom tasks include a few pairs. One can find that reproduction of LeWM is not reliable for either the stable-worldmodel protocol or the authors' protocol. Based on the above reproduced results, I feel the experimental results of LCPP are scientifically unreliable and overclaimed, inheriting the limitations from the stable-worldmodel benchmark. Regarding baseline selection, I agree that all references need to be published in peer-reviewed conference/journals in world model research since the research evolves too fast. However, selecting world model baselines that are not peer-reviewed for primary experimental investigations based on unreliable experimental setup (no filtering) can mislead not only future readers but also AI researchers.

  1. Unclear problem formulation and presentation.

Throughout the manuscript, it requires substantial time to understand the problem formulation the authors have tackled and the motivation for decoupling encoders. LCPP enables offline GCRL with zero inference cost of a pretrained world model during test-time by decoupling encoders under an actor-critic framework, which is the most critical benefit of LCPP, I think. There are extensive papers on independently parameterizing the actor and critic encoders in model-free RL [1-4] or removing inference-time cost of world models [5-7]. However, the current manuscript lacks the connection to that literature. Additionally, the manuscript simply lists the prior approaches with their own explanations without logical connections to LCPP, which does not explain the motivation for decoupling the actor and critic encoders for world models. Furthermore, the readability of the manuscript should be improved significantly. I summarized the remaining questions regarding presentation in the question section below.

Reference

[1] Garcin, Samuel, et al. "Studying the interplay between the actor and critic representations in reinforcement learning." ICLR 2025

[2] Ni, Tianwei, Benjamin Eysenbach, and Ruslan Salakhutdinov. "Recurrent model-free RL can be a strong baseline for many POMDPs." ICML 2022

[3] Raileanu, Roberta, and Rob Fergus. "Decoupling value and policy for generalization in reinforcement learning." ICML 2021

[4] Cobbe, Karl W., et al. "Phasic policy gradient." ICML 2021

[5] Nguyen, Hoang, Xiaohao Xu, and Xiaonan Huang. "Latent Geometry Beyond Search: Amortizing Planning in World Models." arXiv 2026

[6] Georgiev, Ignat, et al. "PWM: Policy learning with multi-task world models." ICLR 2025

[7] Zhao, Yi, et al. "Simplified temporal consistency reinforcement learning." ICML 2023

Questions for authors
  1. Experimental setup: How do you obtain the scores of the from-scratch GCRL baseline in Table 1? Why is LeWM not included in the main table, whereas LeWM is widely used across baselines and LCPP?
  2. Baseline comparison: Why did you replace each baseline's encoder with LeWM's frozen encoder for evaluation? If comparing the performance across performant baselines, isn't it fair to compare LCPP with baselines trained with their own training recipe?
  3. Presentation: The current Abstract feels really hard to understand fluently. Each sentence is not connected logically and coherently, listing each item independently. This problem occurs similarly in the Introduction. In Section 3, where are the policy objective and MDP formulation?

Minor requests

  1. Please refrain from abbreviating standard deviation as SD unless it serves a significant role.
  2. Could you define the definition of judging progress and producing actions first, not in the conclusion?
  3. Could you define the definition of progressing step more formally in L265?
Reviewer_8no1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LCPP, a method for offline goal-conditioned control that uses a frozen world-model encoder only during training. LCPP trains a value function on frozen latent representations to compute advantage weights for behavior cloning, while the deployed policy directly maps current and goal images to actions without querying the world model. The paper provides an analysis of how value errors and representation collisions affect advantage ranking, and evaluates the method on five image-based control environments against from-scratch and frozen-encoder baselines. The experiments include input-representation ablations, fixed-teacher weight tests, goal-distance analyses, and additional robustness checks.

Critical strengths
  1. The paper presents a simple and practically motivated design for reusing frozen world models. LCPP separates representation use for value estimation and control: the latent critic exploits a frozen world-model encoder during training, while the deployed pixel actor does not require a world-model call, predictor, or planner.

  2. The empirical analysis is unusually thorough for the central design choice. Beyond the main comparison, the paper includes controlled latent-versus-pixel critic and actor ablations, pretrained-versus-untrained encoder comparisons, fixed-teacher weight tests, goal-distance experiments, a second Cube checkpoint, and value warm-up analyses. These experiments provide useful evidence about when the proposed design helps and identify limitations such as the PushT warm-up effect.

Critical weaknesses
  1. The PDFs lack navigable links for figures, tables, appendices, and citations, and placing the appendices in a separate supplementary document makes it less convenient to verify the referenced details.

  2. The theoretical analysis does not establish a general advantage of the latent critic over a pixel critic. It relies on strong idealized assumptions and shows how value errors or representation collisions can affect advantage ranking, but does not show that LeWM latents have lower error or more reliable rankings. The empirical evidence is also heterogeneous: Figure 5(a) shows that the relative advantage depends on goal distance and environment, and the PushT advantage disappears after warming up the pixel value function.

  3. The generality of the approach remains unclear. The experiments use five relatively controlled offline environments and environment-specific LeWM checkpoints; the comparison with alternative representation or world-model objectives is limited. It is therefore unclear whether the proposed latent-critic design transfers to other world models, tasks, or more diverse goal-conditioned control settings.

Questions for authors
  1. Figure 5(a) suggests that the relative advantage of the latent and pixel critics depends on the goal distance: the pixel critic is often stronger for short-distance goals, while the latent critic can be advantageous at intermediate distances in some environments. Have you considered a distance-dependent or hybrid critic that uses pixel and latent value estimates differently across goal distances? Could such a combination improve performance?

  2. The method is evaluated only with LeWM latents. How sensitive is LCPP to the choice of world model or representation pretraining objective? For example, would latents from other world models, such as DreamerV3, provide similar benefits as critic inputs, or would performance vary substantially across different world models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.