acceptodds
Withdrawn from ICLR 2027

Teaching Vision-Language Models to Evaluate Actions Leads to Better Manipulation Policies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action models rely on pre-trained vision-language backbones, yet further pre-training with robot-specific descriptive objectives yields little downstream benefit. We therefore investigate whether an evaluative objective that distinguishes viable actions from near-misses and failures transfers more effectively to control. As a first instantiation, we consider object grasping and introduce Action Assessment Pre-Training (AAPT), which trains the backbone to autoregressively generate a physical reasoning trace followed by a scalar viability score for each candidate grasp. After pre-training, the assessment interface is discarded and only the backbone initializes the downstream policy. We separate the objective from both its data and its optimizer. On identical candidates, a predictive control supervised by simulated collision and stability outcomes already improves downstream control. Replacing its binary targets with graded evaluation under matched optimization improves further, and ranking refinement yields on zero-shot LIBERO-Plus and on CALVIN ABCD. Permuting complete trace-and-score targets across candidates causes performance to fall below no pre-training, showing that candidate-quality alignment matters. AAPT's transfer advantage also grows from to points as demonstrations become scarce. Discarded alternatives are therefore useful supervision, and their graded relative ordering is what transfers to manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, robot manipulation, pre-training, representation transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_R8jhOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Action Assessment Pre-Training (AAPT), which trains a VLM to generate physical reasoning traces and graded scores for candidate grasps, followed by ranking-based refinement. The assessment interface is then discarded, and the backbone initializes downstream manipulation policies. Experiments examine transfer across policy architectures, benchmarks, and demonstration budgets.

Critical strengths
  1. The idea of learning from failed and near-successful grasps is interesting. The assessment interface is not needed during downstream control.
  2. The controlled experiments are useful, particularly the paired-seed results and GRPO ablations. The improvements on CALVIN and under limited training data are encouraging.
Critical weaknesses
  1. The role of graded ordering is not clearly isolated. The binary baseline may already contain quality cues in its reasoning traces, while GRPO adds extra training. Shuffling traces and scores together also makes it difficult to tell which part contributes to the improvement.
  2. The gains from the proposed objective are relatively small. On the stronger pipeline, the improvements are only 0.4 points on LIBERO and 1.8 on LIBERO-Plus. The graded SFT comparison shows even smaller gains, and the paper lacks paired-seed results to establish their consistency.
Questions for authors
  1. Can the authors provide paired-seed results comparing graded SFT with collision-and-stability prediction?
  2. Does the ranking reward outperform calibration-only GRPO under the same training budget and base checkpoint?
1 comment
AuthorsOpenReview ↗

Request to check our response

Please check our response and let us know if there any remaining concerns.

Reviewer_9idHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AAPT improves VLM backbones by constructing grasp candidates from successes, perturbed near-misses, and failures, assigning graded targets by distance to the nearest annotated successful grasp. The VLM is trained to generate a physical reasoning trace followed by a score. SFT learns these targets, and GRPO refines candidate ordering. Then the VLM is used to initialize policies, using VLM4VLA and stronger ACoT-VLA pipelines. The experiment is shown in LIBERO-Plus and CALVIN.

Critical strengths

The controlled comparisons are quite rigorous. From finding 1 to finding 9, the authors try to reason why the approach work. Table 3 separates same-data binary outcomes, graded SFT, and ranking refinement, while target permutation tests the importance of candidate-quality alignment. The authors also validate the geometric target using 1,000 simulator executions. Appendix H reports paired results over three training seeds and correctly distinguishes training replication from correlated rollout evaluations.

Critical weaknesses

Depiste the in-depth analysis, there are some concerns in terms of the improvement. The incremental benefit of graded assessment over the matched binary-outcome baseline is quite small (while I understand the authors justify the improvement in Appendix H, Table 10). And its reliability is not established by every paired uncertainty estimates. Although the full method shows some improvements on LIBERO-Plus and CALVIN, there are additional compute/training done in the same env (i.e., in distribution). Then it could very likely suggest that shifting the same compute during VLA training stage will lead to similar improvements.

Questions for authors

How reliable are the gains from graded assessment specifically? Appendix H reports paired results for full AAPT versus the base model. Could you repeat the binary-outcome and graded-SFT variants using the same training seeds and report the improvement for each seed? This would help establish whether the small average gain is reproducible. Given the gains of only 0.6 percentage points on LIBERO-Plus and 0.03 average sequence length on CALVIN, do they remain consistently positive across additional training seeds?

Does ranking refinement outperform an equivalent additional training budget during SFT stage? Could you compare the GRPO stage with continued SFT using matched data and training compute? This would clarify whether the improvement comes from the ranking objective or additional optimization.

1 comment
AuthorsOpenReview ↗

Request to check our response

Please check our response and let us know if there any remaining concerns.

Reviewer_qEySOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Action Assessment Pre-Training (AAPT), which trains a vision-language backbone to generate a physical explanation and a quality score for a candidate grasp before transferring the backbone to a manipulation policy. Candidates are constructed from successful grasps, pose perturbations, and failures, with graded targets based on geometric distance to annotated successful grasps. An offline Qwen3-VL-8B model converts procedural geometric scaffolds into explanation targets. The backbone is trained through supervised fine-tuning followed by GRPO-based ranking refinement. The assessment interface is then discarded, and the backbone initializes downstream policies that are trained on demonstrations. Experiments with VLM4VLA and an adapted ACoT-VLA report improvements on LIBERO, LIBERO-Plus, CALVIN, and SimplerEnv.

Critical strengths
  1. The paper addresses an interesting pre-training question. Learning to distinguish successful actions from near-misses and failures could provide information that is absent from success-only demonstrations. Transferring this supervision through backbone initialization also avoids requiring an additional assessment procedure during policy execution.

  2. The controlled objective comparisons are useful. The same-data collision-and-stability predictor, SFT-only graded assessment, and full AAPT help separate the contributions of failure-aware supervision, graded targets, and ranking refinement. The permuted-target control further shows that preserving the correspondence between candidates and their supervision matters.

  3. The evaluation includes both a minimal policy architecture and a stronger action-reasoning pipeline. The matched-seed comparisons, distribution-shift evaluation, and demonstration-scarcity experiments provide useful evidence that the initialization improves downstream learning under several conditions.

Critical weaknesses
  1. The source of transfer beyond grasping remains insufficiently understood. AAPT evaluates individual grasp poses using proximity to annotated successful grasps; it does not directly assess approach trajectories, placement, or multi-step task outcomes. The authors acknowledge the narrow pre-training scope, but I would like to see more targeted analysis of this central transfer result. How does only the grasp score can improve all different kinds of tasks?

  2. The presentation makes the training pipeline harder to follow than necessary. Some notations are used without introduced. Important distinctions between the offline teacher's privileged inputs, the student's actual inputs, and the explanation supervision are distributed across the main text and appendix. The GRPO description also requires readers to distinguish multiple completions for the same candidate from comparisons across different candidates, while details of the sampling groups and score statistics appear later.

  3. The transfer from action-conditioned assessment to observation-conditioned policy learning needs deeper analysis. During AAPT, the backbone receives a candidate grasp pose, whereas downstream policy learning removes this input and the entire assessment interface. The experiments show that the resulting initialization is useful after end-to-end fine-tuning, but why it is useful when the inputs are different?

  4. Applicability to existing VLA configurations remains only partially established. Both controlled downstream pipelines use Qwen3-VL-4B, and the ACoT-VLA implementation adapts the original backbone and interfaces. This provides evidence across two policy designs, but does not directly establish that AAPT improves standard pretrained VLAs such as pi0 or pi0.5.

Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Request to check our response

Please check our response and let us know if there any remaining concerns.

Thank you for the clarification.

I understand that the grasp score is not reused downstream, and that AAPT provides backbone initialization before policy training on demonstrations. However, the source of transfer beyond grasping remains insufficiently understood to me.

The assessment training only covers grasp poses, while the downstream tasks include pushing, opening drawers, pressing switches, and other operations. These tasks all involve contact, but they require different contact configurations and motion constraints. The explanation that grasp evaluation teaches useful contact geometry is plausible, but the current evidence does not clearly establish how this knowledge helps non-grasp actions.

AuthorsOpenReview ↗

Response to Reviewer qEyS

Thank you for the follow-up. The reviewer is right that grasp assessment does not directly teach the contact configurations of pushing or opening. However, our claim is narrower. AAPT trains the backbone to relate a candidate end-effector configuration to the geometry it sees in the scene, and every downstream action (grasp or anything else) requires that relation. We ran two new analysis to test this directly.

  • First, we tested whether the assessment uses the scene at all. Removing the images and keeping only the candidate grasp lowers ranking accuracy from 86.9 to 62.5. Pairing each candidate with another scene's images lowers it further to 57.2, below the image-free level. Mirroring the image and the candidate together keeps it at 87.4. The model ranks candidates by how they relate to the visible scene, not by grasp or object priors. This grounding is a starting point, not a frozen feature. During VLA training all policy parameters, including the vision encoder, are fine-tuned end to end on the demonstrations, so each task can adapt the representation to its own contact configurations and motion constraints.

  • Second, we tested whether the scene grounding is what transfers. We trained AAPT on the same grasp candidates and targets with the images removed. Initializing ACoT-VLA from this model gives the same performance as the base model. Consequently, learning to evaluate grasps without seeing the scene does not transfer.

  • Third when we breakdown the results, interestingly, the gain is smallest on pick-and-place (LIBERO-Object, +3.0) and largest on tasks with articulated objects and pushing (LIBERO-Goal +5.2, LIBERO-Long +16.4)

Based on these three pointers, we hypothesize that what transfers is the backbone's grounding of end-effector configurations in scene geometry, not grasp-specific knowledge. We agree that a per-action analysis of contact configurations would sharpen this, and we will add that to the final version.

Reviewer_ezn4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks a simple question: instead of just teaching a vision-language model to describe a scene, would teaching it to judge actions make it better at controlling a robot? The authors take successful grasps, create nearby alternatives, and train the model to explain whether each grasp looks physically reasonable and give it a quality score. They then remove the scoring interface and use the trained backbone to initialize a robot policy.

The results suggest that failed and almost-successful actions are useful training data. Learning graded scores works slightly better than predicting collision and stability, and adding ranking training helps further. The full method raises success under unseen perturbations from 87.0% to 88.8%. It also improves five-instruction completion from 77.6% to 84.8%, and its advantage grows when fewer demonstrations are available.

The idea is promising, but the gains over a strong baseline are mostly modest. The study uses 500,000 assessment examples, relies on existing geometry and grasp annotations, and tests everything in simulation. So it makes a useful case for learning from bad alternatives, but it does not yet show that this approach is cheap to scale or reliable on real robots.

Critical strengths

One interesting thing is that it turns failed and almost-successful actions into useful training data. Instead of only showing the model what works, it teaches the model how a good action differs from a slightly bad one.

Another nice point is that the scoring interface is removed afterward. The robot does not need an extra evaluation step every time it acts. The goal is to build that understanding into the backbone during pre-training.

Finally, the method helps more when demonstrations are scarce. Its advantage grows from 1.8 to 6.9 percentage points when only 10% of the demonstrations are used. That makes it potentially useful when collecting robot demonstrations is expensive.

Critical weaknesses

The first concern is that the gain from changing the training objective is actually pretty small. The full method improves success under unseen perturbations from 87.0% to 88.8%, but that combines failure examples, graded scores, and ranking training. When the authors keep supervised training the same and only replace binary collision-and-stability targets with graded scores, the gain is just 0.6 percentage points, plus 0.03 in average instruction-chain length. The jump in five-instruction completion from 77.6% to 84.8% is more convincing, but that comes from the full method. More seeds and paired confidence intervals would help show whether the smaller objective-level gains are reliable.

That also makes the cost question important. The method needs geometry, successful grasp annotations, generated candidates, physical explanations, supervised training, and another stage of ranking optimization. None of that automatically makes it too expensive, especially when the required assets already exist. But without a cost breakdown, it is hard to tell whether the improvement is worth it. The authors could report data-generation and training costs, then compare against alternatives using the same budget, such as more prediction training, longer supervised training, or more robot demonstrations.

The data pipeline may also be much easier to scale for grasping than for other tasks. The main corpus has 500,000 examples, and the largest experiment uses one million, but these are additional examples for an already pre-trained model. They do not replace its large foundation-model corpus. Generating perturbations around known grasps is fairly convenient; getting the geometry and reliable success annotations in the first place is the harder part. It would help to know how many distinct objects, scenes, and successful grasps are behind those examples. Testing with reconstructed geometry or less complete annotations would also show whether the approach works outside well-prepared datasets.

Another concern is what the score actually means. It measures distance to the nearest known successful grasp, rather than directly measuring physical success. The simulator results show a useful overall relationship, but they do not guarantee that nearby candidates are always ordered correctly. An unannotated valid grasp could get a low score, while a tiny change near a successful grasp could suddenly cause a collision. Since those subtle differences are exactly what the method wants to teach, the authors could examine where geometric scores disagree with simulator outcomes, especially among closely ranked candidates.

The paper also leaves some uncertainty about which part of the training really matters. Physical explanations help, but that does not necessarily mean the model is learning a faithful reasoning process; they could simply provide useful extra supervision. Likewise, ranking training adds both a new objective and more optimization. The shuffled-target experiment is useful, but it swaps explanations and scores together, so it only shows that mismatched supervision is harmful. Separately shuffling each component, testing explanation-first against score-first training, and comparing ranking training with an equal amount of continued supervised training would make the argument much clearer.

The broader transfer claim needs some care too. It is encouraging that grasp-assessment pre-training helps downstream placing, stacking, and instruction chains. But that does not show that the same data construction can directly assess those actions. Grasping is unusually convenient because a scene and a candidate pose can provide a lot of the information needed. Pushing or mid-trajectory decisions may require predicting future consequences. A second action class would be a useful next test, followed by a task where judging one pose is not enough.

Finally, everything is tested in simulation. The extra robot embodiments are encouraging, but they do not address real perception errors, friction changes, or unpredictable contact. A small controlled real-robot experiment would make the practical case much stronger. The low-demonstration result also deserves a closer look: the advantage grows to 6.9 percentage points, but that could reflect better representations, faster learning, or both. Learning curves and longer-training baselines would help distinguish those explanations. The idea is interesting; what is still missing is a clearer picture of how much the evaluation objective itself contributes, what it costs, and how well it survives outside this setup.

Questions for authors

see weakness

1 comment
AuthorsOpenReview ↗

Request to check our response

Please check our response and let us know if there any remaining concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Request to check our responses before the discussion closes

Dear Reviewers 9idH, qEyS and ezn4,

As the discussion period ends today, we would greatly appreciate a look at our responses and the revised paper. They add paired-seed uncertainty for every objective-level contrast, compute-matched controls for the ranking stage, longer policy training of the base, and separate trace and score shuffles. Reviewer R8jh found that these resolved their main concerns. If any of your concerns remain, we would be glad to clarify them before the deadline.

Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.