acceptodds
Withdrawn from ICLR 2027

Beyond Successful Rollouts: Learning from Simulator Near-Failures

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Simulator-verified near-failures expose physical distinctions that successful trajectories leave implicit. We show that supervising these distinctions improves manipulation reasoning beyond the benefit of successful-rollout exposure. Contrastive Embodied Reasoning (CER) rewards intermediate reasoning representations for distinguishing task-matched successful states from plausible physical failures. On CRD-Bench, a 30-task RLBench-derived diagnostic, CER exceeds outcome-only training by 10.9 points on high-complexity tasks after successful-trajectory exposure is matched (95% CI [8.3, 13.1]). Comparisons using the same simulator evidence establish the benefit of process supervision; an independent factorial isolates a further 2.53-point contribution from contrastive normalization (95% CI [0.5, 4.6]). The learned constraint representations affect decisions: removing their subspace lowers accuracy by 8.5 points, versus 1.1 for a rank-matched random subspace. The advantage over strengthened terminal training persists on 20 additional high-complexity classes (+9.6 points), independently constructed distractors, and unseen layouts. In executable RLBench planning, CER raises success from 25.3% to 35.3%, with comparable gains from non-contrastive process supervision. Verified near-failures provide a practical route from outcome verification to process supervision for manipulation reasoning.

Primary area:
reinforcement learning
Keywords:
Embodied reasoning, process supervision, reinforcement learning, vision-language models, contrastive learning, robotic manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KJZpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses embodied reasoning for robotic manipulation, where a vision–language model must infer a physically feasible manipulation plan from a scene observation and an instruction. Existing methods employ outcome-only rewards, which cannot verify whether the reasoning has accounted for the physical constraints behind a feasible plan. The key idea is to use simulator-verified successes and near-failures as process supervision beyond correct-answer rewards. To this end, the paper proposes Contrastive Embodied Reasoning (CER). For each training task, CER builds a fixed library of encoded simulator observations, consisting of a centroid of 32 successful states and 32 individually kept near-failures. A bilinear critic scores every reasoning-token hidden state against this library with an InfoNCE-style log-probability, and the token-averaged score is added to rewards in GRPO training. The evaluation uses Qwen2.5-VL-7B on CRD-Bench, a self-constructed 30-class, four-choice RLBench-derived benchmark which is stratified by physical-constraint count.

Critical strengths
  1. The problem is important and well motivated. In embodied reasoning, outcome-only rewards cannot verify whether a model has checked the physical constraints behind a feasible plan. They are also extremely sparse on multi-constraint tasks. Using simulator successes and near-failures as process supervision is a reasonable direction.
Critical weaknesses
  1. The contrastive design contributes little beyond generic process supervision. The paper’s own ablations show that non-contrastive process rewards using the same library already obtain most of the gain (41.1–42.9% vs. 45.8% for CER). The contrastive-specific gain is only 2.53 points, and it vanishes in executed planning (35.3% vs. 34.3%).

  2. The rationale of the method is not adequately analyzed or validated. The paper does not justify why matching reasoning-token hidden states to fixed visual embeddings could capture physically grounded reasoning. It never validates the score itself. Appendix C acknowledges that token-level scores are not reported, and the probing and intervention analyses examine the trained model rather than the reward. It therefore remains unclear whether the gain stems from the claimed mechanism.

Questions for authors
  1. Do the non-contrastive baselines in Table 1 differ from CER only in joint normalization, and were they tuned comparably? Since the contrastive gain vanishes in execution, what shows that near-failure contrast drives the improvement?

  2. How is the critic trained, and does the process score correlate with answer correctness or rationale validity? Can the authors provide a theoretical analysis of why matching reasoning-token hidden states to a fixed library of visual embeddings should reward physically grounded reasoning

  3. Can the authors report results on newer and larger backbones beyond Qwen2.5-VL-7B? Do the gains persist when stronger base models already face less reward sparsity?

  4. Do the gains hold on an established embodied benchmark not built from the same constraint taxonomy? For EmbodiedBench and SpatialRGPT, which are currently reported in a single sentence, can they provide full settings and uncertainty, and compare against the strengthened terminal baseline and the process-supervision baselines?

  5. How are near-failures constructed and mined, and how is exposure matched in Section 4.1? How many seeds support Table 1 and Figure 3?

Reviewer_fxgtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how simulator feedback can help vision-language models reason about physical tasks. The authors propose Contrastive Embodied Reasoning (CER). The method compares the model's reasoning representations with simulator states that show successful actions and near-failures. These comparisons provide an extra training reward beyond the final answer reward. The authors test CER on manipulation question-answering tasks and executable robot planning. They report better results than terminal-reward training, including an improvement from 33.9% to 44.8% on high-complexity questions. They also show that contrastive scoring gives a smaller but measurable gain over independent binary scoring.

Critical strengths
  1. A useful way to learn from near-failures. The paper studies an important problem: a correct final answer does not show whether the model understands the physical constraints behind that answer. Using successful simulator states and near-failures provides a more detailed training signal. The idea is useful because models can learn from mistakes that are close to success, not only from completely wrong outcomes.

  2. Strong experiments with useful controls. The paper compares CER with stronger terminal-reward training and several other process-supervision methods. The authors also match exposure to correct responses and compare different scoring methods using the same simulator information. This helps separate the value of simulator feedback from the value of contrastive scoring.

Critical weaknesses

The paper uses group-relative reward normalization, but the exact advantage calculation is not fully specified. For example, when responses have the same answer reward but different lengths, the length reward also changes the group-relative advantage. This effect is not fully explained in the analysis. The coupled training condition also allows gradients through the reasoning states, but the complete policy objective and its weight are not given. It is also unclear which experiments use the coupled condition and which use the detached condition.

These missing details make it harder to understand the source of the reported improvements.

Questions for authors
  1. How is the group-relative advantage calculated? How are groups with zero reward variance handled, and how does the length reward affect the advantage?
  2. The paper reports positive-answer rates for different constraint counts, but the training and test tasks are different. On which tasks and with which policy were these rates measured?
Reviewer_hqbsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Contrastive Embodied Reasoning (CER) to improve vision-language models' physical reasoning by contrasting intermediate reasoning representations against simulator-verified successful states and near-failures. Experiments on CRD-Bench show that CER improves high-complexity reasoning accuracy from 33.9% to 44.8%, with an additional 2.53-point gain attributable to contrastive scoring over binary scoring. CER also improves RLBench planning success from 25.3% to 35.3%, although non-contrastive process supervision achieves comparable results. Overall, the findings highlight the benefits of simulator-based process supervision, with a smaller additional advantage from contrastive learning.

Critical strengths

None.

Critical weaknesses
  • The paper does not convincingly establish why the specific combination of existing techniques (InfoNCE-style contrastive scoring, bilinear representation alignment, simulator-generated positive/negative examples, and GRPO-based policy optimization) constitutes a fundamentally new learning principle rather than an application of established contrastive representation learning to embodied reasoning.
  • While the experiments provide reasonably strong evidence that simulator-derived process supervision improves embodied reasoning beyond outcome-only reinforcement learning, much of the reported improvement can also be achieved using conventional process-supervision methods.
Questions for authors
  • Given the competitive performance of simpler process-supervision baselines, is the additional complexity of CER justified by its relatively small incremental improvement?
  • How does CER ensure that individual reasoning tokens learn the appropriate physical constraints rather than simply encoding task-level information that discriminates successes from failures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.