acceptodds
Withdrawn from ICLR 2027

Aligning Vision-Language Models for Goal Planning in Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Exploration remains a major challenge for reinforcement learning (RL) agents in Craftax-style environments due to sparse rewards and long horizons. LLM-guided RL is a promising direction, but existing methods still suffer from substantial computational and financial costs as well as training instability due to their reliance on frequent queries to large-scale LLMs or carefully designed task-specific prompts. In this paper, we focus on developing an alternative approach that offers strong performance, stability, and cost-effectiveness. Specifically, we construct CCGP-2M, a multimodal goal-preference dataset for the Craftax-Classic environment, and propose a VLM-guided RL framework that fine-tunes a lightweight VLM to guide the RL agent's exploration. This framework consists of an offline stage, in which a Kullback-Leibler Token-level Direct Preference Optimization (KLTDPO) method is introduced to align the VLM with the goal preferences in CCGP-2M, and an online stage, in which the fine-tuned VLM guides the RL agent's exploration. Experiments on Craftax-Classic show that the proposed approach outperforms state-of-the-art baselines in terms of both score and reward, while achieving the highest success rates on 15 out of 22 achievements and the second-highest success rates on another five.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fCUTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds a dataset, CCGP-2M, from Craftax-Classic trajectories guided by a planner. It selects high-quality data and creates rejected goal sets synthetically. It then trains a Qwen3-VL-8B planner with SFT followed by KLTDPO, which combines a token-level KL correction with reweighting of hard examples. The generated goals are given to a PPO agent. Table 2 compares this system with external planners and with other VLM fine-tuning objectives, and Figure 6 tests the data selection. The KLTDPO planner gets the highest score and reward among the tested methods.

Critical strengths

S1. A concrete dataset, with a relevant selection test: The data collection mixes normal starts with starts that already have an iron pickaxe, to get late-game trajectories. Appendix B explains the event scores, episode quotas and how negatives are built. Figure 6 compares raw and selected datasets of the same size across three fine-tuning methods, which supports the value of the selection.

S2. Useful comparisons inside the VLM family: Table 2 includes the base VLM, SFT, DPO, TDPO and KLDPO. KLTDPO reaches a score of 35.0±0.635.0 \pm 0.6 and a reward of 14.0±0.214.0 \pm 0.2, compared with 33.7±1.033.7 \pm 1.0 and 13.5±0.313.5 \pm 0.3 for KLDPO. These comparisons say more than just comparing a tuned planner with plain PPO, and they make the proposed combination look promising.

Critical weaknesses

W1. The KLTDPO objective is not described the same way in the paper and in the code:

  • Eq. (7) looks wrong as written. In a simple case (zero advantage, uniform reference over two tokens), it reduces to a negative KL term, so minimizing it pushes the policy away from the reference. This does not give the regularized policy that the later TDPO argument needs.
  • Eq. (8) in the main text and Eqs. (15)-(17) in Appendix A.2 use opposite KL directions and opposite chosen/rejected differences.
  • The supplied code implements a different variant (TDPO2) with its own coefficient, and stops the gradient through the chosen-response KL. The example config also mixes in a small SFT loss. So the gradients differ from Eq. (9).
  • In the code, the robust reweighting is always on, for every loss type. So choosing the DPO or TDPO loss does not give their plain, non-robust versions.
  • The example config uses LoRA rank, alpha and beta that differ from Table 4.

TDPO2 plus reweighting may be a sensible method in practice. But the paper needs to say exactly which loss and which settings produced each row of Table 2. These are differences in the released code, and do not prove that the reported baselines used these settings.

W2. The downstream comparisons are not clearly matched, and the cost claim is not measured:

  • Table 2 and Appendix C do not give the number of downstream seeds, the interaction or evaluation budgets, or how checkpoints were chosen.
  • Section 3.2.2 describes planning at every step. But in the supplied code, the local VLM goals are refreshed only every 100 calls, and a text description of the state is added to the goals before encoding. The SGRL baseline gets goal text without this state description. My small tests confirm this behavior. If this code produced the results, the comparison with other planner families also changes what the policy sees and how often. Comparisons inside the VLM family may still be fair if they all share the same setup, and the paper should make this clear.
  • Low cost is a main motivation. But the paper uses 90M offline environment steps (Table 1) and four H100s for fine-tuning (§4.1), and does not report GPU-hours, online planning speed, or total cost compared with the local code planner. Needing fewer API calls does not by itself mean less total compute. The paper admits these costs in §6, but the efficiency claim is still not measured.
Questions for authors

Q1. Which loss did you train? For each preference-training row in Table 2, what exact loss was used: KL direction, TDPO1 or TDPO2, stop-gradients, SFT mixing, and reweighting? Please make Eqs. (7)-(9), Appendix A.2 and the released code agree, and explain how the non-robust baselines turn reweighting off.

Q2. What were the seeds, interaction and evaluation budgets, checkpoint rule, goal refresh interval, and policy inputs? Do you have matched runs, or could you run a focused comparison, showing that the gains remain when the baselines get the same conditioning information?

Q3. What are the separate costs of offline data collection and fine-tuning, and of online planning? Over how many downstream runs are they spread, compared with the local code planner? Or, which narrower efficiency claim do your measurements directly support?

Reviewer_fv69OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds CCGP-2M, a multimodal goal-preference dataset, and proposes KLTDPO to fine-tune a VLM planner. The planner guides PPO exploration in Craftax-Classic, where the method improves achievement scores and rewards over the evaluated baselines.

Critical strengths
  • The multimodal preference dataset is a useful resource for learning goal planners from offline interactions.
  • The downstream RL improvements support the practical value of aligning a local VLM planner.
Critical weaknesses
  • Methodological novelty is incremental. KLTDPO combines TDPO [1] and KLDPO [2], while prioritized goal guidance already appears in SGRL [3]; the main contribution is the dataset and planner-alignment pipeline.
  • Preference reliability is not directly validated. Recorded plans are assumed preferable to synthetic alternatives, but format and trajectory-consistency checks do not establish that altered goals or priorities have lower planning utility.
  • Evaluation is limited to Craftax-Classic and one VLM backbone. Generalization beyond this setting is untested. The claimed cost savings are also not supported by a comparison of total collection, fine-tuning, and planning costs.
  • The objective is inconsistently specified: Eq. (7) minimizes a negative KL term, while Eq. (8) and Appendix Eq. (15) use opposite KL directions. The actual implementation needs clarification.

References:

[1] Token-level Direct Preference Optimization

[2] Robust LLM Alignment via Distributionally Robust Direct Preference Optimization

[3] Goal-Guided Efficient Exploration via Large Language Model in Reinforcement Learning

Questions for authors

please see weaknesses

Reviewer_MMRLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a VLM-guided RL framework for improving exploration in Craftax-Classic. It first constructs CCGP-2M, a 2M-sample multimodal preference dataset from goal-conditioned trajectories generated using LLM-written planning code. The collected data are filtered using a quality-aware selection procedure and converted into chosen/rejected goal-plan pairs. A VLM planner is then fine-tuned using SFT followed by the proposed KLTDPO objective, which combines token-level KL regularization with distributionally robust preference weighting. During online RL, the fine-tuned VLM generates prioritized high-level goals that are provided to a PPO policy. The method achieves the best score and reward among the baselines evaluated in the paper.

Overall, I find the paper well written and the proposed architecture practical, particularly the idea of moving expensive language-model guidance to an offline stage and deploying a smaller local VLM during RL. The KLTDPO objective is also interesting, and the empirical results suggest that it improves downstream performance over the other evaluated VLM fine-tuning methods. However, my concerns are substantial and mainly relate to the necessity of several components, generalization beyond Craftax-Classic, and the fairness and completeness of the experimental evaluation. In particular, the pipeline depends heavily on task-specific LLM-generated planning code and manually designed data-selection rules, while evaluation is limited to a single environment. The proposed method also requires a large offline data-collection budget before RL training, but the paper does not provide a complete comparison of total data and computation costs or downstream RL learning curves.

Critical strengths
  1. The paper is well written and easy to follow.

  2. KLTDPO is an interesting objective. It combines token-level preference optimization with distributionally robust reweighting of preference samples, and the empirical results suggest that this combination improves downstream RL performance.

  3. CCGP-2M is a potentially useful dataset contribution. It contains 2 million multimodal preference samples covering all 22 Craftax-Classic achievements and a diverse set of goals.

The architecture is practically motivated. After offline fine-tuning, a local VLM serves as the high-level planner, avoiding repeated calls to large external LLMs during online RL.

Critical weaknesses
  1. The paper does not establish that an LLM-generated planner is necessary for constructing the exploration curriculum. The same role may potentially be filled by manually designed rules or automatic curriculum-generation methods, and this distinction is not evaluated.

  2. Generalization beyond Craftax-Classic is unclear. Several parts of the pipeline are environment-specific, including the LLM-generated planning code and manually designed quality score. The appendix explicitly states that the quality-score weights and allocation ratios are manually chosen based on trajectory characteristics. A stronger evaluation across environments would be needed to support the generality of the framework.

  3. The empirical comparison does not fully establish the claimed position relative to current Craftax results. Recent strong Craftax-Classic results, such as Kim et al. (ICML 2026), should at least be discussed to better contextualize the reported performance, even if the methods use different training settings.

  4. The total data and compute comparison is unclear. The proposed method requires 90M environment interactions for offline dataset construction in addition to VLM fine-tuning and subsequent RL training. Without comparable end-to-end budget accounting, it is difficult to assess the claimed efficiency relative to other approaches.

  5. The paper does not report downstream RL learning curves. This is particularly important because efficient exploration is one of the paper's main motivations. Final score and reward do not establish better sample efficiency, convergence speed, or stability.

  6. The quality-aware sample-selection procedure is heavily hand engineered. It combines manually weighted signals for goal transitions, achievement proximity, inventory and visibility changes, survival status, rarity, and repetition. Although the paper compares the filtered dataset against random sampling, it does not sufficiently ablate the individual design choices, making it difficult to determine which parts of the selection procedure are necessary.

Questions for authors
  1. Necessity of the LLM-generated planner. The exploration goals used to construct the dataset are produced by planning code generated by large LLMs. Since this appears to be primarily an offline design step, could similar planning logic be provided manually? More importantly, why use an LLM-written planner rather than an automatic curriculum-generation method based on the RL agent's current competence or learning progress?

  2. Generalization to new environments. Since the dataset is produced using environment-specific planning code and manually designed quality signals, would applying the framework to a new environment require writing new planning code, collecting a new large-scale dataset, defining new quality-selection rules, and retraining the VLM? The paper itself notes that the current evaluation is limited to Craftax-Classic because of the cost of data collection, model fine-tuning, and RL training.

  3. Preference optimization versus simpler supervision. In the constructed dataset, the chosen response is the trajectory-consistent goal plan and the rejected response is synthetically modified to be less appropriate for the current state. Could the authors clarify why DPO-style preference optimization is necessary here, rather than a simpler ranking, contrastive, or classification objective?

  4. Role of token-level KL regularization. TDPO was originally motivated by controlling token-level policy divergence in autoregressive generation. Here, however, the planner generates short and highly structured goal sets. Could the authors explain what specific failure mode token-level KL regularization addresses in this setting?

  5. Interpretation of Figure 4. The chosen and rejected implicit rewards for KLTDPO begin at noticeably different absolute values from DPO and KLDPO. Could the authors clarify exactly how these rewards are computed and why KLTDPO exhibits this offset?

  6. Preference metrics versus downstream performance. KLDPO appears to obtain a larger preference reward margin than KLTDPO in Figure 4, while KLTDPO achieves stronger downstream RL performance. This suggests that preference reward margin and accuracy may not correlate strongly with goal-planning quality. Could the authors clarify how these metrics should be interpreted and why they are meaningful diagnostics for this task?

  7. Missing downstream learning curves. Since a major motivation of the method is improved exploration, it would be important to report reward or score versus environment interactions. Final performance alone does not show whether the method improves convergence speed, sample efficiency, or training stability.

  8. Total data and compute budget. The dataset construction stage uses 90M environment interactions before online RL training: 60M under random initialization and another 30M under a manually modified initialization. Could the authors report the total environment-interaction and computational cost of the full pipeline and provide comparable accounting for the baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.