acceptodds
Withdrawn from ICLR 2027

Correcting WHERE, Preserving HOW: Compositional Generalization for Vision-Language-Action Models via Referential Guidance

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While Vision-Language-Action (VLA) models enable flexible action generation, their generalization across diverse environmental elements, including manipulated objects, destinations, and backgrounds, is limited by the lack of diversity in robotic training data. Trained end-to-end on such data, VLAs tend to exploit visual shortcuts, associating actions with task-irrelevant visual features rather than the intended task semantics. These shortcuts block recomposition of elements already seen by the policy, that is, compositional generalization. Existing approaches mitigate such entanglement through task-relevant perception or targeted data diversification, but offer no explicit mechanism for unseen recomposition and require backbone-specific modifications with retraining. We observe that under such recomposition, VLAs often fail at global grounding while retaining local manipulation skills that recover near the correct target in familiar configurations. Therefore, we propose Referential Guidance (ReGuide), a training-free wrapper that, given object poses from a grounding module, combines semantic and geometric rebinding to guide the end-effector into demonstration-supported configurations of the instructed referent, where the frozen policy can resume execution. Experiments in simulation across multiple VLA backbones as well as on a real robot show that ReGuide improves success rates under compositional shifts by up to 56.8 and 75.0 percentage points, respectively, while preserving standard-task performance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, Vision-Language-Action Models, Robot Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_a7xTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ReGuide, an external module designed to address the compositional generalization challenge in VLAs without modifying model weights. ReGuide first employs a grounding module to identify the correct target region. Subsequently, during inference, ReGuide performs semantic rebinding assessments at each controller step, intervening to steer the robot to the correct location whenever the VLA trajectory deviates.

ReGuide has been validated in both Libero and real-robot settings, demonstrating significant improvements, particularly on real robots.

Critical strengths
  1. The manuscript flows logically, progressing from the formulation of the compositional generalization problem to the observation that "VLAs often fail at global grounding while retaining local manipulation skills that recover near the correct target in familiar configurations," and finally to the ReGuide method, which employs a grounding module to address the limitations in VLA grounding capabilities.
  2. The experimental validation is quite comprehensive. Experiments conducted on Libero and in the real world have both demonstrated the effectiveness of the ReGuide method, particularly the significant improvements observed in real-world settings.

I would consider raising my score if the authors adequately address these two concerns.

Critical weaknesses
  1. ReGuide relies on the unexecuted portions of an action chunk to perform semantic rebinding assessments. This raises a question regarding control modes: is ReGuide limited to synchronous control, or is it compatible with asynchronous control or Real-Time Action Chunking (RTC)? If it is restricted to synchronous control, its applicability would be significantly limited, given the recent prevalence of asynchronous control and RTC in VLA applications. Conversely, if it supports asynchronous control, new challenges arise—such as the potential obsolescence of the action chunk used for assessment, and the handling of actions that have already been submitted to the controller but not yet executed at the moment a correction is triggered.
  2. At its most upstream stage, ReGuide relies on a Grounding Module; further analysis of the module's failure modes—and how these modes impact ReGuide's performance—helps readers understand the system's operational boundaries (for instance, by introducing perturbations of varying magnitudes to the grounding location, orientation, or object dimensions and observing the resulting changes in ReGuide's success rate).
Questions for authors
  1. Does ReGuide support asynchronous control? If so, I would like the author to address the issue regarding asynchronous control raised in Weakness 1.
  2. Performance of the grounding module under various failure modes
Reviewer_UQuAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReGuide is a training-free wrapper that improves compositional generalization of frozen VLA models. It detects incorrect target selection from predicted actions and uses demonstration-derived configurations to guide the robot toward the instructed target. The VLA then resumes execution with additional motion constraints and recovery support. Evaluation includes several VLA backbones in simulation and pick-and-place and pushing tasks on a real robot.

Critical strengths
  • The method improves performance across three VLA backbones without updating their parameters. Improvements also remain substantial with perception-based grounding and on a real robot.

  • The pre-contact initialization experiment provides useful evidence that local grasping skills remain available under compositional shifts: grasp success increases from 64.5% to 95.8% after repositioning, without subsequent assistance.

Critical weaknesses
  • The contribution relative to closely related methods is not sufficiently established. DINOBot already combines demonstration retrieval with geometric alignment, while AFI and Harness VLA use external guidance to support frozen VLA execution. ReGuide introduces specific monitoring and configuration-transfer mechanisms, but lacks convincing comparisons that isolate their contribution. DINOBot is not discussed, AFI is not evaluated, and the Harness comparison uses only one initial state per task with different grounding inputs. The existing naive-transport and action-replacement controls do not substitute for a strong demonstration-based controller using the same information.

  • The evaluation provides limited evidence about robustness beyond the configurations covered by the demonstration bank. Simulation includes 16 single-factor composition settings and only three unseen objects. The real-robot study uses one arm, one VLA, and two unseen objects, whose dimensions are measured beforehand. These are valid compositional tests, but the dependence on reference-bank coverage remains largely untested. Moreover, the wrapper directly assists placement and pushing, making the contribution of the frozen policy less clear for these operations. The authors acknowledge several limitations, but the practical range of the method remains uncertain.

Questions for authors
  • Under the same grounding inputs and demonstration data, how does ReGuide compare with demonstration retrieval followed by geometric alignment and trajectory execution?

  • How does performance change when the reference bank is reduced or the closest matching object type is removed?

  • On the real robot, what performance remains when the wrapper only establishes the pre-contact configuration and leaves subsequent placement or pushing actions to the VLA?

Reviewer_45P7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ReGuide, a training-free wrapper for frozen vision-language-action models. It detects incorrect target commitments or stalled execution, then guides the robot toward demonstration-derived interaction configurations at the instructed object or destination. The VLA resumes execution from the resulting observation, with additional motion constraints and recovery assistance. Experiments show substantial improvements on compositional pick-and-place tasks in LIBERO and on real-world pick-and-place and pushing.

Critical strengths
  1. Useful observation about recoverable local competence. The pre-contact initialization experiment provides evidence that a frozen policy can retain local grasping ability despite failing to approach the correct object under compositional shifts. Disabling subsequent assistance in this diagnostic makes it more informative than overall task success alone.

  2. Large improvements across multiple backbones. ReGuide substantially improves compositional success on three VLA backbones while largely maintaining original-task performance. The perception-based simulation results and physical robot experiments strengthen the practical evidence beyond the main oracle-grounding evaluation.

  3. Detailed experimental analysis. The paper includes component ablations, action-replacement controls, parameter sensitivity, and intervention statistics. These help establish that both configuration transfer and subsequent policy execution contribute to performance.

Critical weaknesses
  1. The evaluation does not establish a sufficiently general manipulation contribution. The evaluated tasks are pick-and-place and pushing, which closely match the wrapper’s explicit approach, grasp, carry, and release structure. The central question is whether the same mechanism generalizes when the interaction configuration and subsequent motion depend on contact constraints, articulation, or the intended manipulation. For insertion, drawer opening, or constrained extraction, correcting the target and reaching a familiar pre-contact pose may be insufficient. Appendix E acknowledges the limited scope, but the paper provides little evidence that extending the method would avoid introducing additional task-specific controllers. Given how closely the current controller matches the evaluated tasks, the large success improvements alone do not establish a broadly applicable contribution.

  2. The most relevant alternative is missing from the comparisons. ReGuide has access to object and destination grounding, demonstrated grasp and release configurations, carrying-height profiles, and execution statistics. A natural alternative is to transfer demonstration keyposes into the current object and destination frames, connect them using motion planning, and execute the resulting manipulation sequence with feedback. The paper does not compare against such a system using equivalent information. Naive transport instead uses a fixed 0.25 m approach height, while the action-replacement controls use limited descend-and-close or hold behaviors. These controls show that VLA actions outperform those particular replacements, but do not establish the advantage of ReGuide over a competent demonstration-composition system.

  3. Suitable demonstration support is assumed rather than validated. Familiar objects use pooled interaction statistics, while unseen objects inherit references from the nearest training type under a simple geometric descriptor. There is no demonstrated mechanism for determining that no suitable reference exists. Similar dimensions do not necessarily imply compatible grasp locations or interaction strategies, and aggregating distinct grasp modes may produce an unsuitable reference. The hand-back conditions verify proximity to the selected configuration, rather than its suitability for the task. The paper acknowledges that grasps absent from the demonstrations are unsupported, but does not characterize performance as reference coverage or geometric similarity decreases.

  4. Real-world perception robustness remains insufficiently characterized. The physical experiments demonstrate that the grounding pipeline works in the tested setup, but use measured object and destination dimensions, including those of unseen objects, together with base- and wrist-camera refinements. The reported 90th-percentile horizontal localization error is 2.6 cm over 14 episodes, and this error floors the horizontal tolerances. This may be adequate for the tested relocation tasks, but does not establish suitability for precise contact-rich manipulation. There is no systematic robustness evaluation under occlusion, clutter, similar-looking instances, or inaccurate dimensions, nor a detailed separation of grounding failures from policy and controller failures.

Questions for authors
  1. Can ReGuide be evaluated on at least one harder manipulation family, such as insertion or articulated-object manipulation, where the backbone already possesses the relevant trained skill? Which parts of the reference representation, intervention detector, and recovery controller would need to change?

  2. How does ReGuide compare with object-relative demonstration keypose transfer, motion planning, and execution feedback using the same grounding and demonstration access? What benefit does VLA execution provide over adapted demonstration replay?

  3. How does performance change when the closest reference object is removed, demonstration coverage is reduced, or demonstrations contain multiple distinct grasp strategies? Can the method identify an unsupported reference and avoid an inappropriate intervention?

  4. How sensitive is the complete system to grounding identity errors, position/orientation errors, and inaccurate object dimensions? Can the authors provide controlled robustness experiments and a real-world failure breakdown?

The results demonstrate an effective system for correcting compositional failures on the evaluated relocation tasks. However, the contribution remains difficult to distinguish from composing demonstration-derived manipulation references with conventional control, and its applicability beyond this task structure is not established. The missing comparison with an equally informed demonstration-composition baseline and the limited evidence of generalization to harder manipulation are central concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.