acceptodds
Withdrawn from ICLR 2027

IP-GRPO: World-Model-Guided Reinforcement Post-Training for Vision-and-Language Navigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement post-training complements supervised fine-tuning with execution feedback that helps Vision-and-Language Navigation (VLN) policies correct errors after deviating from demonstrations. However, existing RL-based VLN methods often use trajectory-level rewards that provide limited guidance on whether intermediate decisions advance instruction completion. We introduce IP-GRPO, a world-model-guided reinforcement post-training framework that converts imagined futures into progress feedback at the action-chunk level. At each decision state, an action-conditioned world model predicts the short-term visual consequences of independently sampled auxiliary action chunks. An instruction-conditioned progress scorer evaluates current and imagined states using their respective textual and visual contexts. Averaging candidate-future scores yields a prospective state-progress estimate; subtracting the current-state score converts this estimate into a dense auxiliary reward. These auxiliary rewards are accumulated along execution trajectories and combined with task-success and trajectory-alignment rewards for Group Relative Policy Optimization (GRPO). All auxiliary modules operate only during post-training, preserving the policy's original inference procedure without future prediction or candidate search at deployment. Across the R2R-CE and RxR-CE Val-Unseen splits, IP-GRPO improves success rate by 4.83 percentage points on average over the Uni-NaVid and StreamVLN base policies. The post-trained Uni-NaVid also achieves a mean success rate of 76.7% across three zero-shot real-robot tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, Vision-and-language navigation, Embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Zs1VOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

IP-GRPO uses a world model to predict future observations of candidate actions, and uses the difference between the imagined progress and the current progress as an auxiliary reward topost-train existing navigation policies using GRPO. The average success rate across two strategies and two benchmarks improved by 4.83 percentage points; the average success rate across three real-world robotic tasks was 76.7%. The auxiliary module is only used during training.

Critical strengths
  1. IP-GRPO retains the original deployment process, eliminating the need for online future prediction or candidate search, thus possessing practical value.
  2. Improvements are consistent across both policies and benchmarks. The full method also outperforms task-reward-only GRPO, and the real-robot experiments provide preliminary evidence of zero-shot transfer.
Critical weaknesses
  1. Lack of substantial methodological novelty. This paper primarily combines existing navigation strategies, world models, semantic progress reasoning modules, and GRPO. Its main contribution lies in the imagined-progress reward. However, the ablations simultaneously change future observations and reward formulation, failing to isolate the benefit of imagined futures over simple progress differencing.
  2. The mechanism and reliability of rewards remain unclear. Auxiliary candidates and actions are sampled independently, and immediate rewards do not directly evaluate the consequences of actions. The correlation of the scorer on failure trajectories drops to 0.3831; candidate ranking is based on the same scorer. This evaluation does not independently establish that the reward accurately reflects true task progress.
Questions for authors
  1. Could you add two controls: progress differences computed from actual observations before and after execution, and imagined progress scores without subtracting current progress? Please match the number of environment interactions and policy updates, and report results across multiple training seeds.
  2. How does the reward from independently sampled candidates improve policy learning, given the different sampling temperatures (1.0 for candidates and 0.2 for execution)? Please provide the training configuration and the core reward, advantage, and loss implementations.
  3. How are progress labels assigned when the agent leaves the reference route? Please report the evaluation sample sizes and check whether imagined progress differences agree with ground-truth progress changes from actual rollouts.
1 comment
Reviewer_Zs1VOpenReview ↗

Response to Authors

Thank you for the additional experiments and clarifications. While they address some of my concerns, I still find the methodological contribution rather limited.

Prior VLN works have already explored GRPO-based post-training and progress-based supervision (e.g., VLN-R1, Progress-Think, and SACA). The main novelty here lies in constructing rewards from imagined candidate futures, but the benefit of independently sampled candidates remains insufficiently justified, as the reward does not directly evaluate the executed action.

Overall, I do not find the incremental contribution sufficiently convincing to change my original rating.

Reviewer_eWm5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper adds a progress reward to GRPO post-training for vision-and-language navigation. The policy samples auxiliary action chunks independently of the chunk it executes. A world model predicts their future images, and a learned scorer estimates instruction progress. The average future score minus the current score becomes an extra reward. The method improves Uni-NaVid and StreamVLN on R2R-CE and RxR-CE.

Critical strengths
  1. The main new idea is to use predicted futures to build a progress reward. The method combines NWM and Progress-Think with a trained scorer, then uses the reward in GRPO. The extra models are only used during training, so they add no computation at deployment.
  2. SR improves by 4.0 to 5.5 percentage points across the two policies and two benchmarks. The gain over GRPO with only task rewards is 2.5 points. The robot tests also show an improvement: success on the hardest task rises from 20% to 65%, with 20 trials per method.
Critical weaknesses
  1. Removing the world model also changes the reward. The baselines use the current progress score, while the full method uses the average future score minus the current score. It is unclear how much of the 2.2-point gain comes from the predicted futures. A simple baseline is missing: score the actual observations before and after an action chunk and use the difference as the reward.
  2. The scorer performs worse on failed runs. It is trained on successful trajectories, and its Spearman correlation drops from 0.91 on successful runs to 0.38 on failed ones. Appendix A.4 checks whether generated and actual futures receive the same ranking from the scorer. The reward uses the mean future score minus the current score. A correct ranking does not guarantee that this difference has the right magnitude or sign.
Questions for authors
  1. Could you test a reward that subtracts the current score from the actual successor score, using the same scorer and training budget?
  2. For the same auxiliary actions, how close is the imagined reward to the one computed from actual simulator futures? Please report bias and sign agreement on failed runs, and compare with reference-path progress changes.
  3. How do you label progress when the agent is off the reference path, especially near loops or crossings? How many trajectories and states were used for Table 4 and the val-unseen analysis in Appendix A.4, and which splits were they drawn from?
  4. Were training budgets and evaluation settings matched in Table 3? Please give rollout counts, updates, and GPU-hours. How uncertain are the 2.2-2.5-point gains? Please distinguish differences across training seeds from uncertainty over evaluation episodes.
2 comments
Reviewer_eWm5OpenReview ↗

Response to Authors

I am very grateful to the author for the detailed response, which has largely addressed my concerns. Therefore, I maintain my positive view of this paper.

Thank you for your thoughtful review and continued support. We are glad that our responses have largely addressed your concerns, and we will incorporate the clarifications and additional results into the revised manuscript.

Reviewer_6upzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to give VLN policies denser feedback during RL post-training. The proposed method IP-GRPO samples several candidate action chunks from the policy, uses a navigation world model to predict what the agent would see after each one, and scores the current and predicted states with an instruction-conditioned progress scorer. The average predicted progress minus the current progress serves as an auxiliary reward, added to the success and path-alignment rewards in GRPO. The world model and scorer are used only during training, so inference is unchanged. Applied to Uni-NaVid and StreamVLN, IP-GRPO improves success rate on R2R-CE and RxR-CE.

Critical strengths
  • Using the world model and progress scorer only during post-training is a good idea, as it keeps the policy’s inference cost unchanged.
  • Withholding the instruction from the world model, so that its predictions follow the actions rather than the goal, is a good way to reduce success-biased predictions.
  • The gains are consistent across two base policies and two benchmarks. The paper also includes a zero-shot real-robot evaluation and analyses of the scorer and world model, including discussion of where scorer fails.
Critical weaknesses

Main concerns are:

  • Missing simpler dense-reward baselines: All post-training runs in the Habitat simulator, where the real outcome of any candidate action can be rendered. The simulator also gives the agent’s exact progress along the reference path, the same quantity used to train the scorer, and the reference path is already used for the nDTW reward. Yet IP-GRPO is not compared with a dense reward computed from real outcomes, either the scorer applied to real futures or ground-truth progress. Against the real outcomes, the imagined scores agree only weakly so it is unclear what the world model adds. IP-GRPO is also not compared, on the same base policies, with SACA, which likewise gives step-level rewards for VLN post-training.
  • Small gains, single seed: Over GRPO with task rewards only, the imagined-progress reward adds 2.5 points of SR and SPL (Table 3), and this is the only setting ablated (Uni-NaVid on R2R-CE). All results use a single evaluation seed with sampled actions (temperature 0.2), with no variance reported and, as far as I can tell, a single training run. Gains of this size may be within run-to-run variation.
  • The reward is least reliable where it matters most: The paper motivates RL post-training as a way to help policies recover after they deviate from demonstrations. However, the scorer is trained only on successful trajectories, and on failed trajectories its correlation with true progress drops from 0.91 to 0.38 (Table 4). So in off-route states, where feedback is needed most, the reward comes from a scorer that generalizes poorly, applied to imagined frames.
  • Long-horizon and real-world claims are stronger than the evidence: The paper claims larger gains on long-horizon tasks, but in Table 2 the long R2R-CE groups are very small.

Other concerns:

  • Training cost is not reported: At each training decision step, a diffusion world model runs for four candidates. Text descriptions are generated for the current state and each imagined state, and then the scorer runs. The world model must also be fine-tuned separately for each policy’s action space. The paper reports no training time or compute, so the cost relative to task-reward-only GRPO is unclear.
  • Limited novelty: The method combines existing components (Navigation World Models, Progress-Think’s description modules, and GRPO). Using world-model predictions to estimate progress or value for policy improvement has also been explored in robotics as the paper notes.
Questions for authors
  • How does IP-GRPO compare with a dense reward computed from the simulator, such as ground-truth progress along the reference path, or the same scorer applied to the real outcomes of candidate actions?
  • As defined in Eq. 7, the auxiliary reward for a chunk comes from candidates sampled independently of the executed chunk. So the reward at that chunk does not depend on the action actually taken, which is credited only through later states. Why not score the executed chunk instead?
  • The scorer and world model are trained only on R2R-CE data. How reliable are they on RxR-CE, whose instructions and paths are longer?
  • How much training time and compute do the world model and description generation add compared with task-reward-only GRPO?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.