acceptodds
Withdrawn from ICLR 2027

Crucible: Counterfactual RLVR For Dexterous Manipulation Planners

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Combining vision-language planners with control primitives is a promising paradigm for robotic manipulation. However, reliable execution still requires high-level planners to improve from the outcomes of their own actions, particularly when hallucinations or incorrect reasoning lead to failures. Learning from such feedback is challenging because a primitive may appear locally correct but cause failure many steps later, while a task-level failure signal does not reveal which decision should be corrected. This lack of credit assignment makes planner learning inefficient. We present Crucible, a reinforcement learning from verifiable rewards (RLVR) framework with counterfactual credit assignment. Crucible builds on a new vision-language-primitive (VLP) model constructed from robot demonstrations, with each primitive paired with a privileged verifier. For a failed trajectory, our counterfactual RLVR algorithm intervenes on individual primitive decisions from saved states and replays the recorded suffix, identifying decisions whose replacement repairs the observed verifier failure under fixed-suffix replay. Full-episode returns from these alternative executions then define advantages for masked PPO that updates only the identified decision tokens. We evaluate cross-step dependencies in tabletop grasping, where base motions affect the feasibility of a later grasp. A replay probe rescues 49.5% of eligible failures, demonstrating the availability of executable repairs. In closed-loop evaluation, counterfactual PPO achieves 62.38% overall success, compared with 55.45% for call-level episode-return PPO and 56.44% for the shared SFT baseline. Project Page: https://cruciblex2026.github.io/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, Agent, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZGU7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CRUCIBLE trains a high-level manipulation planner by replacing decisions in failed trajectories from saved simulator states and replaying the recorded suffix to identify repairable actions. The resulting counterfactual returns guide masked PPO updates, and the paper reports 62.38% success on tabletop grasping, compared with 56.44% for supervised fine-tuning and 55.45% for episode-return PPO.

Critical strengths
  1. Counterfactual replay provides more localized credit assignment than a single reward for the whole episode.
  2. The controlled comparison shows gains over both the SFT baseline and episode-return PPO.
Critical weaknesses
  1. The experimental design is somewhat confusing. My understanding is that the SFT used in Cosmos3 and RoboBrain 2.5 differs from the SFT proposed in this paper. It is also unclear whether the preceding component is a contribution of this paper.
  2. The proposed reinforcement learning training scheme appears to offer only limited gains over SFT alone, and its scope is quite narrow.
  3. Quantitative evaluation focuses on simulated tabletop grasping, while the hardware deployment is not evaluated with comparable success-rate results.
Questions for authors

I don’t think the main text needs to be packed with so many equations. More space should be devoted to the experiments, as the current experimental content and plans still seem rather limited.

Reviewer_9MT9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper rewinds failed executions in simulation, changes one decision at a time to find a fix, and uses the result to fine-tune the planner with PPO. On simulated movement-and-grasping tasks, success improves. To me, the main contribution is how failures become training signals for specific decisions(to get better training).

Critical strengths
  1. I think the paper tackles a real problem. An earlier decision can cause a later failure.

2.The training does produce a useful gain. With the executor fixed, task success improves by about six percentage points over SFT.

Critical weaknesses

1.It is unclear whether the extra cost is justified(it matters because i want to know if it worth). Backtracking and branch replays require additional computation(may be much more computation time, during which i use PPO or SFT maybe better).

2.The methodological contribution is not clear enough to me(especially exp,). The approach feels like backtracking search combined with PPO(A kind of combinatiion). How much the backward search, fixed continuation, targeted updates add and so on remain to be more scientific.

3.The generalization evidence is limited.No realrobot(i mean quantitative gain evidence, sim2real gap influence these more detailed things, for me, real matters much more than sim);No other tasks;Just simple task,simple env,simple transfer. It's hard to define it as 'methodological'.(like a kindof engineering)

Questions for authors

See Critical Weaknesses

Reviewer_KUBuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Crucible, a method to fine-tune VLM planners that select skills to solve robotic loco-manipulation tasks. The skills themselves are implemented as scripted controllers plus a flow-based model (VLP) that generates two keyframe poses per call. The planner is trained in simulation by capturing the simulator state before each skill call. After a failed episode, the authors restore the saved states, scanning backward from the failing call, and sample alternative skill calls from the policy at each state while replaying the remaining recorded calls unchanged. The first state at which an alternative call makes the episode succeed is treated as the decision to correct. At that state, each sampled call receives a binary return from its replayed branch, and its advantage is that return minus the mean return of the other sampled calls. PPO is then applied only to the tokens of that single call. On a tabletop navigation and grasp benchmark this reaches 62.4% success, versus 56.4% for the SFT initialization and 55.5% for a PPO baseline that credits only the observed failing call.

Critical strengths
  1. Well-motivated direction with a sensible mechanism. Fine-tuning a reasoning module for skill composition with physical feedback is a promising direction with large potential, and the paper correctly identifies that a trajectory-level reward cannot distinguish a bad base move from a bad grasp. Rewinding the simulator and using sibling rollouts in place of a learned value function is a reasonable answer for a large policy trained on few episodes, and the replay probe (49.5% of failures admit a single-call fix) shows the signal exists. The idea is not fully novel (see weaknesses), but it shows a gain in the reported setting.
  2. Strong VLP training and ablations. The VLP ablations are informative, and a lot of effort has clearly gone into training and tuning the executor, including the teacher-student distillation from point clouds to RGB. This is not the main novelty of the paper, but it provides a solid foundation for the experiments.
Critical weaknesses
  1. The central empirical claim is under-powered and under-controlled. The headline comparison rests on 101 paired episodes with no seeds, no confidence intervals, and no repeated runs. The +6.9 pp gain over the PPO baseline amounts to 7 episodes and is within binomial noise. The only RL baseline is a non-standard variant that credits one call per episode; there is no full-trajectory PPO or GRPO baseline, so the paper cannot separate "localizing the step helps" from "group-relative advantages help." The closest prior work, VinePPO (restoring intermediate states and estimating per-step credit from sampled branches), is neither cited nor compared against, so the novelty is overstated. Together these mean the paper does not yet demonstrate that counterfactual credit assignment is responsible for the reported gain.
  2. The evaluated scope is far narrower than the framing suggests, and the hardware claim is unsupported. Closed-loop evaluation and all reasoner fine-tuning use only 3 of the 16 available skills (xy move, yaw rotate, grasp; stated only in Appendix F.3, L2035), while the main text, Figure 2, and the video present a broad prehensile and non-prehensile skill library. This leaves open whether the method scales to a realistic action space. The hardware deployment in Section 4.4 reports no numbers, and the real-robot clips in the supplementary video are footnoted as manually sequenced primitives replayed from simulation, so the trained reasoner, which is the main contribution, is never shown on hardware.
Questions for authors
  1. Why do the real-world demonstrations use human-selected skill sequences instead of the trained reasoner? Are there any quantitative hardware results?
  2. Why was the reasoner restricted to three skills when the other nine VLP primitives are trained and have verifier rules in Table 10? How does the method scale with the number of skills?
  3. Can you report results over several training seeds and a larger evaluation set, with confidence intervals?
  4. How does counterfactual PPO compare to standard full-trajectory PPO or GRPO on the same episodes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.