- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper presents an approach to easily recover from states where an error occurred to find a successful trajectory for VLMs. To achieve this, the paper first presents an environment that captures the progress, success, visibility, and penalty of different states within a trajectory. The paper utilizes this environment to collect data to construct the ERR benchmark, which is used to train and evaluate a 2B sized VLM. The experiments show the corresponding trained model has superior recovery rate compared to existing baselines.
Critical strengths
- The paper presents a useful dataset focused on recovery from error states within an environment. The environment and the corresponding benchmark could be useful in other applications than the one presented in the paper.
- The paper shows that training on the proposed dataset is helpful in improving the recovery rate.
Critical weaknesses
-
The paper is not well presented in its current format and several parts within it are difficult to follow. The paper also uses several terminology that is not prevalent in the literature and difficult to understand. Please find the details below:
- Line 162: What is the “pre-defined interface” for task-specific checker.
- Line 163: What does “task-relevant object relations” mean?
- Line 176: What does “language-level semantic-action sequence” mean?
- Eq. 8, what does mean? What is the meaning of “small signed thresholds”.
-
The equations and text around Eq. 7 is very hard to parse. What does rollout group mean? Is a single set or two? If it’s two sets, then how are the mean and std. defined? If it’s a single set, why are we splitting by the action type?
-
The overall contribution of the work is ambiguous. Is the primary contribution of the paper the dataset? Apart from that the turn-level rewards have been already explored in the RL literature. Is the reward computation using the task-specific checker supposed to be a key contribution? Details of the task-specific checker are not specified in the main paper. How is it designed? Is it well calibrated with the ground truth rewards? There should be separately analysis for that.
-
The experiment results are a bit confusing. Many open- and closed-source baselines outperform the proposed method in terms of the success rate metrics. Is the recovery rate improvement the main focus of these results? Could the gains in recovery rate be simply because it was trained on data from the same environment?
-
Many simple baselines are missing from the experimentation section. What are the results when using other popular online RL methods on the VLM like GRPO, PPO, GSPO, etc. without reward shaping? How does the performance change when we only provide outcome-level rewards?
-
The task-specific checker provides dense feedback in this environment, which is not easily available on other environments. This could be another reason why the proposed method shows improvement on this dataset. It would be interesting to see results using baselines like PPO or REINFORCE using these dense rewards.
Questions for authors
Please respond to the weaknesses above.
3 comments
W3. Our primary contribution is a verifiable interaction framework for training and evaluating embodied error recovery, consisting of ERR-Env, ERR-Bench, and AGT-GSPO. ERR-Env provides restorable states and executable verification of task progress and errors, while ERR-Bench enables controlled evaluation across nominal, intermediate, and error states. AGT-GSPO builds on these components to optimize recovery decisions through verified interaction outcomes. We do not claim turn-level rewards or group-relative optimization as independently novel; instead, our methodological contribution is their recovery-oriented integration with action-grouped advantage estimation and semantic advantage shaping.
Regarding checker design, Appendix A.1.2 provides a concrete implementation based on executable simulator-state predicates rather than learned reward estimation. The checker evaluates goal completion, partial progress, erroneous manipulations, and object visibility. Generated tasks undergo automatic execution validation, and randomly sampled task programs are additionally inspected to verify consistency between task instructions, action semantics, and checker logic. The checker thus provides programmatically verifiable task-state feedback rather than subjective reward predictions.
The importance of these signals is further supported by the reward ablations in Table 2. Removing action reward reduces SSR from 20.89% to 10.05%, while removing penalty feedback reduces SSR to 2.80%, despite maintaining relatively high PG. This demonstrates that explicit error feedback is essential for distinguishing meaningful task progress from unreliable execution. Although these results establish the utility of checker-based rewards, they do not constitute an independent quantitative calibration study against human-labeled states. We will clarify the checker design and existing validation procedure in the main text.
W4. Our main objective is to improve recovery capability and reliable task completion, rather than outperform larger general-purpose models across every metric. Although some closed-source models achieve higher overall SR, ERR-VLM (Full) achieves the highest aggregate Recovery Rate of 65.47% and outperforms all evaluated open-source and embodied baselines in overall SR and SSR.
Regarding the effect of in-distribution training, SFT and RL data originate from the same ERR-Env task-generation pipeline and follow the same underlying task distribution, covering nominal, intermediate, and recovery states with disjoint training instances. The SFT policy has already learned from reference recovery corrections, while RL further optimizes the policy through verified outcomes of its own actions. Compared with SFT, full RL improves Recovery Rate from 42.52% to 65.47%, Recovery SSR from 8.58% to 22.16%, and SSR from 10.64% to 20.89%.
Moreover, the recovery-data ablation in Table 2 shows that removing recovery snapshots from RL reduces Recovery Rate to 39.12% and Recovery SSR to 11.57%, while PG remains relatively similar (69.14% vs. 71.08%). This demonstrates that nominal interaction alone is insufficient for effective recovery learning. ERR-Bench also uses exclusively held-out tasks unseen during SFT and RL. We further evaluate zero-shot transfer on RMBench, RoboVoLo, and physical robots without additional training, where the full model consistently improves task progress over its SFT initialization. These results support the effectiveness of recovery-oriented learning beyond memorizing training trajectories.
W5. We clarify that several commonly used online RL methods are already evaluated in Table 2, including Turn-GRPO, Trajectory-GSPO, and Trajectory-GRPO. These baselines differ in their optimization granularity and reward assignment. Specifically, Turn-GRPO applies GRPO with turn-level rewards and turn-level policy updates, replacing the sequence-level importance weighting of our Turn-GSPO formulation with token-level GRPO optimization. In contrast, Trajectory-GSPO and Trajectory-GRPO use only outcome-level rewards, assigning a single reward and advantage to the entire trajectory and applying GSPO or GRPO at the trajectory level. All these comparisons use the same SFT initialization and training budget, allowing us to evaluate the effects of optimization strategy and credit-assignment granularity.
The results demonstrate substantial differences in recovery reliability. Turn-GRPO achieves 8.85% SSR and 11.18% Recovery SSR, compared with 20.89% and 22.16% for our full method. Trajectory-GSPO and Trajectory-GRPO achieve Recovery SSR values of only 5.59% and 4.99%, respectively. These results highlight the importance of assigning feedback to individual interaction turns rather than using a single trajectory-level signal, while also demonstrating the benefits of sequence-level GSPO optimization over token-level GRPO in our setting.
We further provide additional algorithm and reward comparisons in Appendix A.5.3 (Table 11). Specifically, Trajectory-GSPO (Outcome) and Trajectory-GRPO (Outcome) correspond to the outcome-only trajectory baselines reported in Table 2. We additionally evaluate Trajectory-GSPO (Sum-of-Turn) and Trajectory-GSPO (Avg-of-Turn), which aggregate dense turn-level rewards into a single trajectory-level reward through summation and averaging, respectively. The four variants achieve Recovery SSR values of 5.59%, 4.99%, 4.99%, and 7.39%, respectively, compared with 22.16% for our full method. Even when dense rewards are available, aggregating them into a single trajectory-level signal removes their temporal specificity and weakens credit assignment for individual corrective actions.
Together, these experiments systematically compare turn-level GRPO, trajectory-level GRPO/GSPO with outcome-only rewards, and trajectory-level GSPO with aggregated dense rewards. They support the effectiveness of preserving fine-grained turn-level feedback and optimizing each reasoning-and-action response as a coherent sequence. We will make these existing comparisons more prominent in the main paper.
W6. We agree that dense checker feedback is an important factor in recovery learning, and providing such verifiable feedback is itself a central motivation for ERR-Env. Unlike environments that provide only final task success, ERR-Env explicitly evaluates intermediate progress, erroneous state changes, execution validity, and visibility recovery, allowing the policy to receive feedback on individual corrective decisions.
We investigate the contribution of these signals through controlled reward ablations in Table 2. Removing grounding, action, penalty, and outcome feedback reduces Recovery SSR from 22.16% to 19.96%, 12.57%, 7.78%, and 14.37%, respectively. Notably, removing penalty feedback causes a substantial decrease in SSR from 20.89% to 2.80%, indicating that progress-based rewards alone can reinforce actions that advance the task while introducing execution errors.
Furthermore, the outcome-only and dense-reward aggregation comparisons in Appendix A.5.3 (Table 11) provide additional evidence concerning the interaction between reward design and credit assignment. These results show that dense feedback alone is insufficient when its temporal information is collapsed into a single trajectory-level reward. The effectiveness of our framework comes from combining explicit error-aware feedback with turn-level optimization, enabling the model to distinguish corrective actions from error-inducing ones. Although PPO and REINFORCE with identical dense rewards are not included, our existing GRPO/GSPO comparisons and reward ablations provide systematic evidence for the contributions of reward design and optimization granularity.
We thank the reviewer again for the constructive feedback and hope these clarifications better communicate our methodology, experimental comparisons, and contributions.
We thank the reviewer for recognizing the value of ERR-Env/ERR-Bench and the improvements in recovery performance. We appreciate the detailed feedback and provide the following clarifications.
W1. We appreciate the feedback on presentation and clarify the specific terminology and notation below:
Line 162, “pre-defined interface”: This refers to the task-construction/checker API detailed in Appendix A.1.2, which provides operations such as
add_actor,pick_and_place,check_grasp,check_on, andcount_visible_objects. The checker evaluates simulator states and returns success, progress, penalty, and visibility, which are used for reward computation and evaluation but are never exposed to the policy. We will include a concise API example in the main text.Line 163, “task-relevant object relations”: These are executable predicates describing goal-specific object relationships and manipulation states. For example, the checker in Appendix A.1.2 verifies whether a Rubik's cube is placed in a wooden box and a knife in a steamer. Correct intermediate grasps contribute partial progress, while incorrect placements or irrelevant grasps incur penalties. We will clarify this terminology with concrete examples.
Line 176, “language-level semantic-action sequence”: This refers to a symbolic reference plan consisting of named objects and high-level actions, such as
pick object_1; place container_1. It is used during task generation to construct and validate executable trajectories and is never provided to the policy during training rollouts or evaluation.Eq. 8, “small signed thresholds”: As specified in Appendix A.2, the thresholds are μinv=−0.02, μdeg=−0.01, and μimp=0.02. They respectively constrain the advantages of invalid, non-improving, and improving
pick/placeactions, suppressing misleading positive advantages for erroneous actions while ensuring positive feedback for verified improvements. We will make these values and their roles explicit alongside Eq. 8.Eq. 7, rollout group and comparison set: For each training prompt or initial snapshot, six sampled interactive rollouts form a rollout group Oi, while the update batch B contains rollout groups from eight prompts. For each turn t, Gt is a single comparison set selected according to the action type: same-type
pick/placeturns within Oi, or same-typehome/doneturns across B, since the latter occur less frequently. The mean and standard deviation are computed exclusively over the rewards in the selected Gt. We will clarify this piecewise definition and distinguish the rollout group, update batch, and comparison set more explicitly.W2. The motivation behind Eq. 7 is that different high-level actions have distinct reward semantics and scales. Specifically,
pick/placerewards depend on changes in task progress and penalties,homerewards visibility recovery, anddonerewards successful termination. Directly comparing these heterogeneous rewards may produce misleading relative advantages. Our method therefore groups turns by action type, using the same initial snapshot forpick/placeto preserve task-context comparability and the update batch for less frequenthome/doneactions to ensure sufficient comparison samples.We empirically validate this design through the grouping ablations in Table 2. Under the same SFT initialization and RL configuration, our full method achieves 20.89% SSR and 22.16% Recovery SSR, compared with 19.83%/18.16% for Batch (All), 9.25%/9.18% for Sample (All), and 11.11%/8.78% for Batch (Action). These comparisons demonstrate that both action-type grouping and shared initial-task context contribute to reliable recovery. We acknowledge that turns from the same rollout group may still encounter different intermediate states; the grouping is designed to improve contextual comparability rather than guarantee identical states.