- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper builds a dataset, CCGP-2M, from Craftax-Classic trajectories guided by a planner. It selects high-quality data and creates rejected goal sets synthetically. It then trains a Qwen3-VL-8B planner with SFT followed by KLTDPO, which combines a token-level KL correction with reweighting of hard examples. The generated goals are given to a PPO agent. Table 2 compares this system with external planners and with other VLM fine-tuning objectives, and Figure 6 tests the data selection. The KLTDPO planner gets the highest score and reward among the tested methods.
Critical strengths
S1. A concrete dataset, with a relevant selection test: The data collection mixes normal starts with starts that already have an iron pickaxe, to get late-game trajectories. Appendix B explains the event scores, episode quotas and how negatives are built. Figure 6 compares raw and selected datasets of the same size across three fine-tuning methods, which supports the value of the selection.
S2. Useful comparisons inside the VLM family: Table 2 includes the base VLM, SFT, DPO, TDPO and KLDPO. KLTDPO reaches a score of and a reward of , compared with and for KLDPO. These comparisons say more than just comparing a tuned planner with plain PPO, and they make the proposed combination look promising.
Critical weaknesses
W1. The KLTDPO objective is not described the same way in the paper and in the code:
- Eq. (7) looks wrong as written. In a simple case (zero advantage, uniform reference over two tokens), it reduces to a negative KL term, so minimizing it pushes the policy away from the reference. This does not give the regularized policy that the later TDPO argument needs.
- Eq. (8) in the main text and Eqs. (15)-(17) in Appendix A.2 use opposite KL directions and opposite chosen/rejected differences.
- The supplied code implements a different variant (TDPO2) with its own coefficient, and stops the gradient through the chosen-response KL. The example config also mixes in a small SFT loss. So the gradients differ from Eq. (9).
- In the code, the robust reweighting is always on, for every loss type. So choosing the DPO or TDPO loss does not give their plain, non-robust versions.
- The example config uses LoRA rank, alpha and beta that differ from Table 4.
TDPO2 plus reweighting may be a sensible method in practice. But the paper needs to say exactly which loss and which settings produced each row of Table 2. These are differences in the released code, and do not prove that the reported baselines used these settings.
W2. The downstream comparisons are not clearly matched, and the cost claim is not measured:
- Table 2 and Appendix C do not give the number of downstream seeds, the interaction or evaluation budgets, or how checkpoints were chosen.
- Section 3.2.2 describes planning at every step. But in the supplied code, the local VLM goals are refreshed only every 100 calls, and a text description of the state is added to the goals before encoding. The SGRL baseline gets goal text without this state description. My small tests confirm this behavior. If this code produced the results, the comparison with other planner families also changes what the policy sees and how often. Comparisons inside the VLM family may still be fair if they all share the same setup, and the paper should make this clear.
- Low cost is a main motivation. But the paper uses 90M offline environment steps (Table 1) and four H100s for fine-tuning (§4.1), and does not report GPU-hours, online planning speed, or total cost compared with the local code planner. Needing fewer API calls does not by itself mean less total compute. The paper admits these costs in §6, but the efficiency claim is still not measured.
Questions for authors
Q1. Which loss did you train? For each preference-training row in Table 2, what exact loss was used: KL direction, TDPO1 or TDPO2, stop-gradients, SFT mixing, and reweighting? Please make Eqs. (7)-(9), Appendix A.2 and the released code agree, and explain how the non-robust baselines turn reweighting off.
Q2. What were the seeds, interaction and evaluation budgets, checkpoint rule, goal refresh interval, and policy inputs? Do you have matched runs, or could you run a focused comparison, showing that the gains remain when the baselines get the same conditioning information?
Q3. What are the separate costs of offline data collection and fine-tuning, and of online planning? Over how many downstream runs are they spread, compared with the local code planner? Or, which narrower efficiency claim do your measurements directly support?