acceptodds
Withdrawn from ICLR 2027

LIFT-WAM: From Counterfactual Adaptation to Grounded Goal Execution

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In language-guided manipulation, the same scene can support multiple feasible goals, with the current instruction specifying the intended outcome. World-action models (WAMs) may continue executing scene-associated behaviors after an instruction changes, and correct target interaction can still fall short of establishing the requested final relation. We characterize this gap at three levels: alternative-goal responsiveness, goal grounding, and complete goal realization. Our counterfactual post-training framework, LIFT-WAM, learns alternative-goal behavior from native and counterfactual supervision. A Grounded Intent Extractor and an Intent-to-Action Adapter convert scene-grounded entity roles and target relations into explicit action conditions, while replay-verified corrective continuations supervise execution through goal completion. Under same-scene counterfactual evaluation with source initial states held fixed, LIFT-WAM achieves success on LIBERO-10/LIBERO-Object and on RoboTwin Clean/Random, compared with and , respectively, for counterfactual adaptation alone. Controlled Video Expert interventions further show that, after counterfactual adaptation, language-conditioned representations of current observations can redirect closed-loop control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World-Action-Model, Embodied AI, Robotics Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vMfSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper study if WAM follows the current language instruction when the same scene can supports different goals. It evaluates it via counterfactual (CF) instructions while having the scene and initial robot state fixed. The paper- LIFT-WAM builds upon Fast-WAM through CF adaptation, grounded intent extraction & corrective training through expert continuations from states reached while pursuing the wrong goal. The paper evaluate on LIBERO-obj and 10, RoboTwin & a real robot. The results show substantial improvements over the base model, with CF adaptation contributing most of the gain. An extra experiment also examines how language conditioning in the video pathway influences executed actions.

Critical strengths
  • The evaluation addresses an important problem. I liked the use of different feasible instructions for the same scene and initial state. Which provides a useful test of if the policy follows language, beyond what standard task-success evaluations reveal. The distinction between behavioral persistence, grounding errors, and realization failures also helps explain where instruction following can break down.
  • The video-pathway intervention is informative and I like dthat. I found the result that changing only the Video Expert language conditioning can redirect the adapted policy particularly interesting. It shows how language influences control in this architecture and adds insight beyond the overall performance gains.
Critical weaknesses
  • I am not yet convinced that the experiments isolate the contribution of each component. The overall improvement over the base model is substantial, but several gains from grounded intent and full-goal corrective training are small relative to the evaluation size, especially on LIBERO. Without confidence intervals or repeated training runs, it is difficult to judge their reliability. More importantly, plain fine-tuning on the same counterfactual data and a data-matched corrective-training control would help establish whether the gains come from the proposed objectives and supervision strategy. This matters because these components form a central part of the main contribution in the paper.

  • Important methodological details remain difficult to reconstruct. I had to infer how several components work. The counterfactual-adaptation objective should be stated explicitly in the main text, and the completed-clause indicators (m_t), deployment gate, and replay annotations need clearer definitions. For example, it is unclear how completed clauses are detected during execution or whether incorrect completion flags can be revised. The source of the grounding annotations, particularly for the real robot, also needs clarification. These details affect both reproducibility and my assessment of the supervision required.

  • Baseline is missing from the real robot experiment.

  • Hard to read the paper. Many terms are either undefined or defined very late in the paper. What is GIE? What is the purpose of the gain gate? and so on. Even with limited spac main architecture pattern should be clear to the readers.

  • Privileged supervision is not accepted clearly. GIE and routing are trained on replay annotations, (possibly) simulator ground truth for entities, positions, and predicate truth. Baseline dont gets comparable information, and how these labels are obtained for the real robot is unclear.

Questions for authors
  1. Can you state the exact CF-adaptation objective and clarify why the negative-condition error is stop-gradiented in ints equation. Do you have a comparison against plain flow-matching fine-tuning on the same data?

  2. How are (m_t) and the deployment gate computed and updated during execution? What annotations supervise grounding, and how are these obtained for the real robot?

  3. Can you give uncertainty estimates for the component gains and clarify whether corrective-training comparisons use matched data or training budgets? Please also clarify whether the 53.3% diagnostic result and the 26% RoboTwin Clean result use the same checkpoint and evaluation protocol.

  4. Where do LIBERO counterfactual demonstrations come from, given LIBERO has no default counterfactual demos?

  5. Given Section 4.3’s finding that language acts mainly through the video pathway, did you try injecting grounded intent on the video side? Why the action expert?

  6. Can you provide more details about the UNT David A1C robot and its capability and DOF in the paper

Reviewer_zc6zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LIFT-WAM, a post-training framework for improving language-guided manipulation when instructions change but the scene remains fixed. It combines counterfactual adaptation, grounded intent conditioning, and full-goal corrective training. Experiments in simulation and on a physical robot show improved counterfactual goal execution.

Critical strengths
  • The paper isolates instruction following by changing the goal while keeping the scene fixed. Its distinction between behavioral persistence, goal grounding, and incomplete goal realization offers a useful way to analyze failures beyond aggregate success rates.

  • The mechanistic analysis is strong and interesting. Controlled interventions show that changing the video-language condition can substantially improve counterfactual success (from 4/150 to 80/150). Layer masking and residual-replacement experiments further help identify where language influences control, which was a useful findings.

  • The paper evaluates two simulation benchmarks and a physical robot, reporting per-task results, confidence intervals, and limitations. Its component comparisons also help distinguish the main gains from counterfactual adaptation from the smaller gains of additional training components.

Critical weaknesses
  • The statistical evidence is limited for incremental gains. LIBERO evaluations use only 50 episodes per configuration, with substantially overlapping confidence intervals. Some additional gains are very small. For example, full-goal training improves RoboTwin Random success from 27.5% to 28%. Without repeated training seeds or statistical tests, the reliability of these gains remains unclear.

  • The effects are inconsistent across tasks, but the provided explanation is insufficient. For instance, grounded intent reduces stacking-role reversal success on RoboTwin, while full-goal training hurts performance on some slot-reassignment and placement tasks. The paper reports these regressions but does not sufficiently explain when or why the additional components are beneficial.

  • The paper does not clearly establish that evaluation goal pairs and initial states are disjoint from those used in training. It also lacks compute-matched controls to separate the effects of the proposed components from those of additional data and optimization. Evaluations on held-out goal transformations and instruction templates would strengthen the evidence for generalization.

Questions for authors

Please view my comments in the weaknesses section. I'm particularly interested to know about the inconsistent effects across tasks and how that could be explained.

Reviewer_x61iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the problem of improving instruction following capabilities of world action models (WAMs) for robot manipulation tasks. Specifically, it consists of three steps: 1) doing supervised finetuning for a WAM policy with alternative goal instructions, 2) using a small relational network to ground a text instruction into entities and predicate which are used to condition the WAM policy, 3) perform DAgger-style corrective finetuning to further boost performance. Evaluations are conducted in RoboTwin and LIBERO and shown to outperform baselines without these techniques. Evaluations are conducted also in the real world on block manipulation tasks, but it appears that no video results are shown or provided.

Critical strengths
  • Instruction following is a key capability that is often the weakness of VLAs or WAMs, and it is a timely problem to tackle.
  • The three proposed techniques build on top of each other and it appears that they provide clear gains on top of each other.
Critical weaknesses
  • Vague terminology and poor presentation: The paper appears to use more sophisticated terminology than the actual underlying procedure to showcase the “novelty”. However, these invented terms, e.g., counterfactual policy adaptation, grounded intent-to-action, not only makes the paper hard to understand but also hurts the contribution. It further sharpens the problem when the abstract and introduction repeatedly use these undefined terms without bringing new information to the readers. It is perfectly okay to use standard terminology while showcasing the effectiveness of the system while still having solid contributions.
  • The relation graph imposes a possible limitation: The so-called “GIE” component relies on clearly defined entities (e.g., blue and yellow blocks) and a relational predicate (e.g., on the side of). While it works for pick-and-place style tasks, especially block manipulation, and certainly a good way to study instruction following due to their combinatorial property, using this inductive bias as part of the architecture appears to impose significant limitation for tasks where entities and relations can be hard to define. For example, when manipulation a cloth (folding clothes, or folding boxes), it is unclear what even is an entity or what set of relations are needed to clearly specify the task.
  • Limited difficulties of real-world manipulation tasks: Related to the point above, the paper only studies simple block manipulation tasks in the real world. While it’s okay not to extend to tasks where the predicate-based relations are hard to specify, the limited diversity of object geometry and manipulation behaviors limits the contributions of the work.
Questions for authors

See weaknesses section above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.