acceptodds
Under review as a conference paper at ICLR 2027

Latent Goal Prediction from Language for Model-Based Planning

Abstract

Joint-Embedding Predictive Architectures (JEPAs) enable agents to plan in latent space by imagining the outcomes of candidate actions, yet task specification remains a bottleneck. Visual targets provide precise local gradients but poor distant guidance, while language is flexible yet limited by noisy cross-modal alignment or dependence on distinct large generative models. We introduce LAGO (Latent Goal Prediction from Language), a hierarchical world model in which a single predictor both forecasts action-conditioned dynamics and grounds language instructions as sequences of intermediate latent subgoals, training both modes with a single regression objective over a shared latent space. At each planning step, LAGO predicts a sequence of latent subgoals from a language instruction and optimizes an action sequence using a soft-minimum alignment cost that rewards subgoal proximity without enforcing a rigid path. Subgoals are repredicted as the agent's state evolves, turning a single long-horizon instruction into a sequence of locally tractable objectives. Across environments spanning navigation and manipulation, LAGO outperforms flat and hierarchical image-goal planners, with the largest gains on long-horizon navigation with curved, hazard-constrained paths, where it more than doubles the success rate of a goal-conditioned behavior cloning policy trained on the same demonstrations. Grounding language in the world model's own latent space also outperforms vision-language reward models, including one that plans with ground-truth dynamics.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, hierarchical planning, subgoal prediction, language-conditioned planning, joint-embedding predictive architecture

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_p22fOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces LAGO, a latent world model for language-conditioned model-based planning. The same predictor is trained in two modes: predicting the next latent state given an action, and predicting a future latent subgoal given the current state, a natural-language instruction, and a continuous completion variable. At inference time, several subgoals corresponding to different values of the continous variable are generated independently from the current observation. CEM then optimizes action sequences using a soft-minimum trajectory-to-subgoal alignment cost, with subgoals regenerated after every MPC. The method is evaluated on TwoRoom, a newly proposed Kingdom navigation domain, OGBench Cube and OGBench Scene. The paper compares against flat and hierarchical latent world models, oracle-subgoal variants, language reward models, and a goal-conditioned behavior-cloning baseline. It additionally presents LAGOSET/LAGO++, showing that language-conditioned subgoal prediction can benefit from action-free video-language data.

Critical strengths

The strongest aspect of the paper is not merely “language goals instead of image goals.” It is that dynamics prediction and task grounding become the same mathematical operation: predict a target latent from the present latent, with the conditioning variable determining whether the prediction represents a one-step transition or a longer-horizon semantic target. Both are optimized using the same latent-regression objective. This is considerably cleaner than attaching a separate reward model or generative visual-goal model.

Critical weaknesses

The paper claims a notable architectural virtue: one predictor, shared parameters, one latent objective. However, I don't think the experiments isolate this contribution. LAGO(O.S.) versus DINO-WM(O.S.) shows that exposing the model to the subgoal-prediction objective improves downstream planning dynamics. That is an interesting result. But it does not tell us whether the benefit comes specifically from sharing one predictor, versus simply learning from additional future-state supervision.

Questions for authors

What happens if dynamics prediction and subgoal prediction use separate predictors with identical supervision and matched capacity? This seems necessary to establish the value of the shared-predictor contribution.

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer,

We would like to thank you for the time and consideration you have given to reviewing our work. As the rebuttal period draws to a close, we would welcome any remaining feedback or questions you may have. Of course, should any issues remain unresolved, we would be happy to provide further explanations or additional experimental results.

Reviewer_rxJzOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LAGO, a JEPA world model in which a single latent predictor over frozen DINOv2 patch tokens runs in two modes: action-conditioned next-state prediction and subgoal prediction conditioned on a language embedding and a completion scalar. Both modes are trained with the same latent MSE, using mostly random interaction data for dynamics and action-free, language-labeled demonstrations for subgoals. At test time, LAGO predicts subgoals, and CEM optimizes 10-step action sequences under a soft-minimum alignment cost, re-predicting subgoals at every MPC step.

Experiments cover locomotion and manipulations, with non-trivial improvements on the goal-conditioned visual planning, language-conditioned policy, and so on.

Critical strengths

The problem of "goal image" is well-defined

Good literature writing, a highly complete paper

Evaluations are diverse and solid

It enables real robot planning with high-fidelity human language. Since humans can provide coordinate instructions, the bridge between humans and the visual world model is absolutely significant.

Critical weaknesses

This paper's key weakness is its motivation and scope:

W1. Unclear problem setting. The paper motivates a new setting, planning from language, because goal images are impractical. So it is unclear whether the claim is (a) a new task setting, which is more reasonable for realistic control and suboptimal/action-free data training, or (b) a better method for goal-conditioned visual planning or a language-conditioned policy?. Each needs different evidence and different storytelling, but from the paper I cannot clearly capture which is the core.

W2. Language is not shown to substitute for goal images. Subgoal prediction is a deterministic MSE regression, so for a one-to-many instruction (which is non-invertible) it learns the conditional mean of goal embeddings rather than a valid goal. The benchmarks largely sidestep this: TwoRoom and Cube instructions contain exact coordinates, and Kingdom goals are unique cells. Where ambiguity exists (Cube, Scene), the arm's final state is unspecified. The claim of "combining the precision of visual goals with the flexibility of language" comes from visual-goal supervision, and also, the flexibility appears only in object position, not in task/skill generalization (maybe "potential" is better?).

W3. Gains are attributed to language but likely come from hierarchy and an expert prior. The long-horizon advantage over goal-image planners comes from subgoals learned from expert demonstrations. This is orthogonal to goal modality, since HWM and FF-JEPA are hierarchical with image goals. At test time, the baselines receive a goal image while LAGO receives language plus a learned demonstration prior, so modality, hierarchy, and prior are confounded.

W4. The most relevant comparison is under-addressed. LAGO has the same access to the inputs and outputs as a language-conditioned policy, so the key question is what a latent-goal-plus-planning pipeline buys over direct language-conditioned behavior cloning. This is only discussion. The single GCBC baseline (15M parameters) appears to be trained only on the 10k demonstrations, while LAGO also uses 50k interaction transitions, so the Kingdom gap mixes extra data with planning. The stated advantages over VLAs, explicit intermediate behavior and scaling with action-free data, are not tested as main claims ====== This matters because, without it, the paper's value relative to the obvious alternative is still unclear.

I will raise the score if the main concerns are resolved

Questions for authors

See Weaknesses.

Reviewer_Cr8gOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces LAGO, a hierarchical world model on frozen DINOv2 features that predicts both the next step environment dynamics and a sequence of independently predicted subgoals conditioned on a language goal. Planning is then conducted by minimizing the distance between all rollout states and each subgoal. Across four environments, they show LAGO outperforms state-of-the-art world models as the task difficulty increases. They also compare with other language-conditioned value functions to show the effectiveness of LAGO’s language goal-conditioning.

Critical strengths
  • The problem of language goal-conditioned planning is quite relevant.
  • The paper is nicely written.
Critical weaknesses
  • Despite the paper being motivated by long-horizon planning and using the idea of hierarchical world models, the planning setup is quite short-horizon. The paper says all methods are trained with “no frameskip and fix the planning horizon to 10 rollout steps”. That is an even shorter planning window than previous world models, which usually use frameskip 5 and a planning horizon of 5 (equivalently 25 environment steps). This artificially makes long-distance goals hard for the flat baselines; notably, GCBC gets 100% on TwoRoom at every difficulty, so the task itself isn’t hard. The paper also contains factual errors. For example, it says DINO-WM’s TwoRoom environment was evaluated using goals randomly sampled within a fixed radius of 25 steps, while that paper actually evaluated on randomly sampled states, ensuring the initial and goal states are on different sides of the wall.
  • The proposed method doesn’t address the core challenges in long-horizon planning. The planning cost compares a single 10-step rollout against all K subgoals at once, even though the subgoals span the whole remaining task. A more effective use of subgoals would be to allow a shorter planning horizon, i.e., only plan towards one subgoal at a time. The proposed method is more like a planning regularizer for a flat world model, where the plan should adhere to a sequence of states generated reactively (like a policy).
  • Since subgoal generation is reactive, it additionally requires a demonstration dataset to supervise subgoal prediction. LAGO therefore needs to see the task distribution during training. The world model baselines also train on the demo transitions, but they don’t use them as task/goal supervision.
  • The proposed method predicts subgoals independently, generating each one conditioned only on the desired distance of that subgoal from the current state. This fundamentally gives no guarantee that the subgoals form a consistent plan, especially when there are multiple valid paths.
  • The subgoal prediction of LAGO is purely reactive, so there’s no guarantee of the quality of the generated subgoals on out-of-distribution goals.
  • Missing baselines on more hierarchical policy and planning methods like HIQL, Hierarchical Diffuser, etc., which use demonstration data in a comparable way.
  • I don’t think this has much to do with language goals. The goals are mostly coordinates written as text or a small fixed set of templates, so it’s more like a “goal-state conditioned” subgoal generation process. The comparisons with VLMs as reward functions in Section 4.3 therefore don’t quite make sense.
Questions for authors
  • Is the GCBC baseline trained using the same dataset as LAGO and the world model baselines, or only demonstration data?
  • What would happen when there are multiple solutions to the same init and goal pairs? How do you enforce the consistency of the subgoals?
  • Can you show generalization of language prompts in terms of planning success? Or run an ablation where you give the goal states directly (i.e., the same info you put in the language prompt)?
  • Why does TwoRoom achieve such low performance for the baselines at larger distances to the goal? In previous papers, TwoRoom performance is mostly saturated when the initial and goal positions are on different sides of the wall, covering the hardest task possible within this environment.
1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer,

We would like to thank you for the time and consideration you have given to reviewing our work. As the rebuttal period draws to a close, we would welcome any remaining feedback or questions you may have. If our responses have adequately addressed your concerns, we would kindly invite you to reconsider your initial score. Of course, should any issues remain unresolved, we would be happy to provide further explanations or additional experimental results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.