acceptodds
Under review as a conference paper at ICLR 2027

IA-World: Residual VLA Post-Training with Latent World Model

Abstract

Vision-language-action (VLA) models provide strong priors for robotic manipulation, yet adapting them efficiently through online reinforcement learning remains challenging. Existing post-training methods often correct base actions using current observations or policy-derived signals. However, they typically do not explicitly anticipate the consequences of the sampled action plan. Meanwhile, sparse task rewards provide limited feedback on whether a correction actually improves future task progress. In this work, we revisit VLA post-training from the perspective of action-specific foresight and argue that effective residual adaptation should anticipate the consequences of a base plan before modifying it, while learning from how that anticipated future changes after execution. Based on this insight, we propose IA-World, a framework that uses a latent world model for residual VLA post-training. Given an observation, a frozen VLA first proposes a base action plan; we distill its action representations associated with the plan into an Intent-Action Token (IA-Token) and use a lightweight latent world model to predict the multi-step consequences of executing the uncorrected plan. The predicted future directly conditions a residual policy to correct the base action before execution, forming a control loop of observing, proposing, predicting, correcting, and executing. After execution, a foresight scorer conditioned on the goal reevaluates the predicted future from the new state, and the resulting change in task progress provides dense learning signals beyond sparse environment rewards. We evaluate IA-World across LIBERO and ManiSkill with both Isaac GR00T and OpenPI. IA-World consistently improves task success through residual correction and outperforms competing post-training methods based on reinforcement learning. The results demonstrate that modeling the future consequences of a VLA’s own action plan enables a shift from reactive residual correction to post-training guided by foresight.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Latent World Models, Reinforcement Learning, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_y91aOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes IA-World, a residual reinforcement learning framework for adapting frozen vision-language-action models. Action-side hidden representations associated with a base action proposal are compressed into an Intent-Action Token, which conditions a latent world model to predict visual features at multiple offsets within the uncorrected proposal’s execution prefix. These predicted features augment the residual actor–critic context, while a frozen goal-conditioned scorer provides potential-based reward shaping. Bounded residual corrections are optimized with TD3 using real environment transitions. Evaluation covers ManiSkill Peg Insertion with π0.5 and the LIBERO-10 task Soup & Sauce in Basket with GR00T. The paper reports improvements over the frozen policies and evaluated post-training baselines, together with a controlled ablation of predicted-future inputs.

Critical strengths
  1. A well-motivated and technically coherent interface for residual VLA adaptation.

Predicting the consequences of a specific base proposal before applying a residual correction is a sensible way to provide additional context to the residual learner. The paper clearly distinguishes the predicted consequences of the uncorrected proposal from the value of the corrected action, which is learned through real interaction. The implementation also accounts for actual chunk durations, consistent action composition, and terminal handling of the shaping potential (Appendices A.3–A.5). These details make the proposed control and learning procedure reasonably well grounded.

  1. Promising observed gains and a meaningful controlled future-input ablation.

IA-World improves success over the frozen policies from 42% to 62% on Peg Insertion and from 72% to 94% on Soup & Sauce in Basket (Table 1). The future-input ablation is particularly useful: it holds reward shaping, architecture, and optimization fixed while consistently masking the future-feature block in the actor and critics (Table 2 and Appendix B.2). This provides evidence that the additional future context benefits the residual learner in the two evaluated configurations.

Critical weaknesses
  1. The empirical evidence is too narrow and insufficiently characterized to establish robust improvements.

The evaluation contains only two individual tasks, with a different backbone assigned to each task. This provides an initial proof of concept but leaves the breadth of applicability uncertain. The paper also does not report the number of independent training seeds or variability across runs; Tables 1–2 and Figure 3 contain no uncertainty estimates. In particular, the 94% versus 90% comparison with RLT on LIBERO is difficult to assess from 100 evaluation episodes without information about training variability. Evaluation episodes do not substitute for independent training runs. In addition, while the 15,000-step online interaction budget is matched, the amount and collection cost of offline data used to train the auxiliary modules are not reported, and baseline access to comparable data and supervision is insufficiently specified. These omissions limit the strength of both the robustness and efficiency conclusions.

  1. The central prediction-based mechanism and the contributions of its components remain insufficiently validated.

The controlled ablation supports the utility of the future-feature block, but it does not establish whether the gains specifically depend on useful, proposal-sensitive prediction, or quantify the separate contributions of IA-Token conditioning and foresight-based shaping. Appendix C defines horizon-wise prediction error without reporting measured prediction results. There is also no comparison establishing the value of the IA-Token relative to direct action conditioning, and shaping remains enabled in both variants of the main ablation. Consequently, the evidence for the proposed mechanism is less complete than the evidence that the overall feature augmentation helps. Important auxiliary-training details also remain unspecified, including the actual prediction offsets, construction of scorer ranking pairs, and loss/shaping coefficients. These gaps limit interpretation and reproducibility of the main contribution.

Questions for authors
  1. Empirical robustness — clarification and additional evidence.

How many independent training seeds underlie the reported results? Please report variability across independently trained policies and clarify whether evaluation initializations are matched across methods. Do the improvements persist on additional tasks beyond the two current task–backbone configurations? Such evidence would materially affect my assessment of the result’s robustness and scope.

  1. Validation of proposal-specific prediction — additional evidence.

Can the authors report measured horizon-wise prediction errors and provide a targeted control testing the role of proposal conditioning, such as a state-only predictor or direct action-chunk conditioning? It would also be informative to evaluate prediction quality at states visited by the learned residual policy. For such an evaluation, the reference target should be obtained by executing the uncorrected base proposal from the same state, consistent with the paper’s definition of Base Future.

  1. Contribution of reward shaping — additional evidence.

What is the effect of the foresight-based shaping reward independently of the future-feature input? A comparison of shaping enabled versus disabled while retaining future inputs would already be useful; a future-input × shaping ablation would more fully distinguish the two proposed uses of Base Future.

  1. Offline budget and reproducibility — clarification.

Please specify the number of offline trajectories/transitions, their collection cost, and which data, goal information, and reward signals are available to each baseline. Please also provide the actual prediction horizons, scorer ranking-pair construction, and auxiliary loss/shaping coefficients. Is the Residual RL entry in Table 1 the same implementation as the w/o-future variant in Table 2?

Reviewer_6xmYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

IA-World improves a frozen VLA with residual RL guided by a latent world model. A token compressed from the hidden states of the VLA's action branch conditions a small transformer that predicts future DINOv2 features of the uncorrected action chunk. This predictor and a goal-conditioned scorer are trained offline on rollouts of the base policy and then frozen. Online, which is a TD3 residual actor receives the predicted future and outputs a bounded correction, and the scorer adds a potential-based shaping reward. On one ManiSkill task with π0.5 and 50 evaluation episodes, success goes from 42% to 62%; on one LIBERO-10 task with GR00T-N1.7 and 100 episodes, from 72% to 94%.

Critical strengths
  1. Correcting a frozen VLA with a small residual policy, and using the VLA's own action features to predict where the base plan will go, is a practical idea that leaves the VLA untouched. Both settings improve over the frozen policy by 20 to 22 points. In Table 2, removing the predicted future from the inputs of the actor and critics costs 44 and 12 points, while the shaping scorer still uses it, as App. B.2 explains.
Critical weaknesses
  1. The evidence is too narrow for the claims. There are two tasks, and each backbone is tested on only one of them. No seeds or variance are reported. The differences to RLT, the only external baseline, are 31/50 vs 25/50 and 94/100 vs 90/100, which are within evaluation noise: a Fisher exact test gives p≈0.31p \approx 0.31 and 0.440.44. On ManiSkill, the reported 15k-step checkpoint is the highest of the five evaluations in Figure 3, whose first three evaluations equal the frozen policy; there is no learning curve for LIBERO or for RLT. The statement in Sec. 1 that IA-World "consistently improves" across benchmarks and backbones rests on these two pairings.

  2. The ablations do not show which part matters. There is no run without the shaping reward, that is, with λΦ=0\lambda_\Phi = 0, although the dense progress reward is listed as a contribution, and no comparison of the IA-Token with plain action conditioning. The predicted future is computed from the state tokens and the IA-Token, and the actor already receives a pooled version of the state tokens and the IA-Token itself, so the future ablation cannot separate "predicting the future" from "extra features trained offline".

Questions for authors

Questions that could change my score:

  1. How many seeds were run per method? Can you report mean and spread, and was the 15k-step evaluation point chosen before looking at the learning curves?
  2. The "Residual RL" row of Table 1 is identical to "w/o future" in Table 2, 18 and 82. Is it the same run, and which of your components does it still use? Does RLT use the same shaping reward, action bound, warm-up and offline data, and how were its hyperparameters chosen?
  3. Can you add λΦ=0\lambda_\Phi = 0, conditioning on the raw base action instead of the IA-Token, and a version where the future features are replaced by non-future features of the same size?
  4. What is the horizon set, and what are the world-model errors per horizon? EτE_\tau is defined in Appendix C but not reported. How many offline base rollouts are used, and how is PegInsertionSideHardClearance-v1 defined, given that ManiSkill3 provides PegInsertionSide-v1?

Seed-averaged results with a clear gap to RLT, together with the λΦ=0\lambda_\Phi = 0 ablation, would raise my score.

Beyond the rebuttal: results on more tasks, with each backbone on both benchmarks, are what would make the general claim convincing.

Reviewer_JD7hOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes IA-World, a residual RL framework for post-training frozen VLA models. It uses an IA-Token to condition a latent world model that predicts the future consequences of the base VLA's actions. These predictions guide a TD3-based residual policy and provide dense reward shaping signals through a foresight scorer. Experiments on ManiSkill and LIBERO show improvements over frozen VLA policies and existing RL post-training baselines. Overall I feel that the intuition of using "possible future" to guide the policy actor is intriguing, as it could enable more informed action corrections by explicitly considering the potential consequences of the base policy's actions.

Critical strengths
  1. The idea of using predicted future states to guide residual action correction is intuitive and well-motivated
  2. The method achieves consistent improvements over the evaluated baselines on both LIBERO and ManiSkill. In particular, the foresight-conditioning ablation demonstrates substantial performance gains, providing encouraging evidence for the proposed design.
Critical weaknesses
  1. Limited experimental evaluation and statistical evidence. The paper evaluates only two tasks(one in LIBERO and one in ManiSkill) and the author reports no std/min/max range in the results. This is insufficient to establish the generality, robustness, and statistical significance of the reported improvements.
  2. Insufficient validation of the core foresight-conditioned mechanism. The paper lacks quantitative evaluation of world-model prediction accuracy and its relationship to downstream performance(although ablation on foresight conditioning is reported in the paper). An oracle ablation(which I am really eager to see) using ground-truth future observations AgtA_{gt} (which is accessible in simulation) would do help.
  3. The main contribution appears to be future-state conditioning for residual RL, which I find relatively incremental. The remaining components largely combine existing (or even well-established) techniques, e.g. reward shaping follows established formulations (Ng et al., 1999)
  4. Lack of real-world evaluation. All experiments are conducted in simulation, whereas RLT has demonstrated real-world online RL (and from my perspective, RLT is featured for real-world RL). Real-robot validation would strengthen the practical claims.
Questions for authors

see Weaknesses 1,2 and 4

Reviewer_s1fXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes IA-World, which provides forward information for the post-training of the frozen VLA's residual RL through the latent world model. IA-Token compresses the relevant representations of the basic VLA action plans, and the lightweight world model predicts the multi-step Base Future without correcting it; the residual policy combines the current state, the basic actions, and the predicted future generation to generate bounded corrections. The target condition foresight scorer scores the Base Future and uses the score changes before and after the real transfer to construct dense progress feedback. Experiments are conducted on ManiSkill Peg Insertion/π0.5 and LIBERO Soup & Sauce in Basket/Isaac GR00T.

Critical strengths
  1. By explicitly providing "what consequences will result if the base plan is not revised" to the residual strategy, it offers a more explicit modeling of the consequences compared to residual RL that only inputs the current state and base actions.
  2. Freezing the base VLA, visual encoder, and world model, and only optimizing the small residual actor-critic online helps to retain the pre-trained behavioral priors and control the policy drift.
  3. Base Future is used simultaneously for action correction and dense feedback based on the potential function, helping to form a consistent forward representation between the control input and the learning signal.
  4. The method achieves improvements on two VLA backbones, two types of tasks, and two robot learning platforms, demonstrating a certain degree of cross-setting applicability.
Critical weaknesses
  1. Each benchmark only selects one task, and the experimental coverage is insufficient to support broad conclusions regarding general VLA post-training or complex long-term operations.
  2. There is a lack of independent assessment of the quality of Base Future predictions, such as multi-step prediction errors, calibration, and the relationship between prediction errors and residual failures. It is still unclear whether the core world model truly learns reliable consequences.
  3. There are few comparison methods, and only Frozen Base, Residual RL, and RLT are compared under a unified 15,000-step budget; the relationship with more frozen VLA refinement, world-model RL, or direct VLA post-training methods is still insufficient.
  4. The methods rely on task-related goal prototypes, offline future goals, and additional world model pre-training. The overall data and computational costs have not yet been compared end-to-end with the baseline.
Questions for authors

Please refer to Weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.