- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
ReWorld uses a world model to generate observations of local execution errors and pairs them with corrective trajectories constructed through motion planning and demonstration reuse. These synthetic recovery examples are then used to fine-tune robot policies to improve recovery from execution errors.
Critical strengths
The motivation and central idea are clear, and the overall method is technically reasonable. The experiments support the effectiveness of the proposed approach across different policy backbones on real-world tasks.
Critical weaknesses
-
Clarity of the technical presentation. The method is difficult to follow, with several terms and notations introduced without sufficient explanation of their purpose. For example, although Section 3.4 and Appendix B.3 describe the sampling probabilities and weighting coefficients for competence-aware post-training, they do not clearly assemble these components into an explicit training objective or step-by-step procedure.
-
Limited innovation of competence-aware post-training. This component appears to primarily involve task-dependent sampling and loss weighting based on the base policy’s competence. While the experiments demonstrate its utility, its methodological contribution appears incremental.
-
Insufficient justification of perturbation sampling. The paper does not adequately explain how perturbations are sampled or how their distribution reflects errors encountered during real-world policy execution. Identifying perturbations that the policy cannot recover from does not, by itself, establish their relevance to realistic rollout failures. Further justification of the sampling strategy, along with visualizations and evaluation of the synthetic observations and their consistency with the recovery labels, would strengthen the paper.
Minor Issues
- CuRobo should be cited at its first mention in the main text, around line 252.
- The budget parameters in Eq. (3), around line 213, should be defined when first introduced.
Questions for authors
See weaknesses
3 comments
Response on Competence-Aware Training and Implementation
Response to Reviewer TAhD, Comment 2 of 3
B. Training objective and implementation (W1)
The sampling distribution and loss coefficient form one coherent training objective and control two complementary aspects of corrective learning. The sampling distribution determines which tasks receive corrective examples, while the global loss coefficient determines how strongly corrective examples contribute relative to nominal demonstrations.
For task , let denote the estimated competence of the base policy, calculated from task successes on an independent calibration set using the Laplace smoothed estimate . We define its desired corrective supervision share as
Lower competence produces a higher corrective target within the clipping range. We then define and use different task distributions for nominal and corrective examples,
P_C(i)=\frac{\omega_i}{\sum_{j=1}^K\omega_j}.$$ Nominal supervision remains uniform across tasks, while corrective examples concentrate on weaker tasks. The resulting post-training objective is $$\mathcal L(\theta)=\mathbb E\left[ \sum_{n=1}^{b_N}\ell_\theta(z_n^N) +\lambda_C\sum_{m=1}^{b_C}\ell_\theta(z_m^C) \right],$$ where $b_N$ and $b_C$ denote the nominal and corrective sample counts, and $\ell_\theta$ is the policy backbone's native action learning loss. The corrective coefficient is $$\lambda_C=\frac{b_N}{Kb_C}\sum_{j=1}^K\omega_j.$$ Under the stated task sampling distributions, this coefficient makes the expected **loss coefficient share**, rather than the measured gradient share, of corrective supervision for task $i$ equal to its target $c_i$. The per-example losses are combined with a common normalization across the batch. This preserves the relative weighting used in the derivation, while avoiding separate group-wise averaging that would change the effective nominal and corrective balance. Operationally, each training update samples nominal task identities from $P_N$, corrective task identities from $P_C$, draws examples from the corresponding task caches, and computes their native per-example action losses. The corrective losses are multiplied by $\lambda_C$ before the common batch reduction. For the 50 task $\pi_{0.5}$ experiment, the nominal-to-corrective sample ratio is **3 to 1**, with $K=50$ and $\lambda_C=0.4856$. The competence dependent targets range from **0.0641 to 0.3312**, as determined by the observed task competence scores. These settings implement the derivation directly, and the complete objective and training procedure are now consolidated in Appendix B.3. Thus, competence-aware allocation preserves nominal supervision while targeting weaker tasks. It complements, rather than replaces, the core corrective-data construction mechanism. ## C. Minor presentation details We have added the **CuRobo citation at its first occurrence in Section 3.3**. We have also clarified beside Eq. (3) that **$B_{\min}=48$** and **$\Delta B=96$** are the base and competence-dependent additional probing budgets. Appendix B.3 now presents the complete loss and the corresponding task sampling implementation together.Response on Perturbation Design and Natural Failure Coverage
Response to Reviewer TAhD, Comment 3 of 3
D. Perturbation design and natural-failure coverage (W3)
We appreciate the request to clarify why the perturbations are meaningful for embodied execution. The perturbations represent three common classes of local control deviation, positional misalignment during reaching or placing, rotational misalignment during end-effector orientation control, and premature gripper closure during manipulation. These are action-level deviations with direct consequences for what the robot observes and which corrective action it needs next. They are not arbitrary appearance changes.
To make the perturbation design concrete, translation errors are sampled along the six Cartesian directions, with magnitudes of 3.3, 6.7, and 10 cm. Rotation errors use magnitudes of 5, 10, and 15 degrees. Premature gripper closure represents an action timing error that can interrupt a grasp before the policy reaches its intended manipulation state. Perturbations are introduced at demonstration-aligned moments within approach, pregrasp, transport, and place phases. Their purpose is to create a controlled, reproducible family of local deviations whose physical meaning is clear.
Importantly, sampling a perturbation is not the same as retaining it for training. ReWorld subsequently performs policy-conditioned recovery diagnosis. Each perturbation unit is evaluated with four base policy recovery probes. Position errors are evaluated against a 3 cm local alignment criterion, rotation errors against a 10 degree orientation criterion, and premature gripper closure against successful gripper reopening. A future expert state window allows the probe to recognize realignment with the demonstration without requiring an exact return to one timestamp. A perturbation unit is retained only when the empirical recovery rate is at most 25%, meaning the base policy succeeds in at most one of its four probes.
The perturbation families specify interpretable local errors, while the policy probes determine which instances reveal recovery weaknesses. Retained units proceed to corrective labeling only if feasible recovery and expert continuation are available. The revised supplement reports full 50-task retention statistics by type, magnitude, and phase.
Natural failure evidence and corrective label validity
To assess whether these perturbation families reflect errors encountered during ordinary policy execution, we examined 400 naturally failed base policy rollouts. This study complements controlled perturbation probing by asking which early local deviations actually arise before a natural task failure, and how often those deviations correspond to ReWorld's three structured error families.
| Observation from natural failures | Count | Rate |
|---|---|---|
| Identifiable early local execution deviation | 307 / 400 | 76.8% |
| Early deviation covered by translation, rotation, or premature gripper closure | 249 / 400 | 62.3% |
| Covered among rollouts with identifiable early local deviations | 249 / 307 | 81.1% |
The key result is that 81.1% of identifiable early deviations belong to these three physical error families. Natural failure coverage establishes the relevance of the perturbation design. Policy-conditioned probing then identifies the particular deviations that remain difficult for the current policy.
Finally, the generated observations and corrective labels were independently examined on 500 accepted samples from 10 RoboTwin tasks. Target object mask consistency reached 94.8%, and 472/500 corrective trajectories, or 94.4%, passed the simulator complete task success criterion. These results complement the selection statistics by assessing whether the perturbed observations and associated recovery actions can be used as executable training supervision. Full selection breakdowns and validation details are included in the revised supplementary material.
Together, the natural-failure analysis, policy-conditioned selection, and complete-trajectory validation connect the error families encountered in execution to automatically constructed, executable corrective supervision.
Response on Core Novelty and the Offline Learning Loop
Response to Reviewer TAhD, Comment 1 of 3
A. Central methodological novelty (W2)
Thank you for recognizing the clear motivation, coherent design, and consistent improvements of ReWorld. We appreciate your close examination of novelty because it highlights the distinction between the full learning framework and an auxiliary weighting rule. ReWorld's central contribution is a usable, closed-loop world-model and policy training framework in offline data space, where counterfactual generation, feedback from the policy's recovery behavior, and corrective supervision jointly guide policy improvement. Competence-aware weighting supports this loop but is not its defining novelty.
The challenge is to use a world model for policy-learning feedback, not just plausible images. From a successful demonstration, the framework must expose recovery failures, construct meaningful corrective targets, and learn from them. ReWorld connects these steps through three operations.
1. Action grounded counterfactual generation. Starting from successful demonstrations, the action-conditioned world model predicts the local visual consequences of structured execution deviations. Unlike arbitrary image augmentation, this associates the altered observation with an explicit perturbed action sequence and the corresponding robot state. Because generation is conditioned on the target scene appearance, it also avoids requiring a fully specified scene-specific simulator for every new layout.
2. Policy conditioned recovery diagnosis. A plausible perturbation is not necessarily a useful corrective training example. Some deviations can already be handled by the current policy, while others reveal an actual weakness. ReWorld measures local recovery outcomes using the base policy and selects perturbation units that expose these weaknesses. This makes data generation adaptive to the policy being improved, rather than depending only on a fixed perturbation distribution.
3. Automatic corrective action labeling. Selected counterfactual observations are paired with recovery actions. The planner returns the robot toward a reference point in the successful demonstration, and the corresponding expert continuation completes the corrective action label. Feasibility and validity checks select examples suitable for offline training. This final step is essential because identifying an error alone does not provide the action supervision needed to correct it.
This is how ReWorld forms a world-model and policy feedback loop for offline post-training. The frozen world model generates action-conditioned deviations, the policy identifies which deviations remain difficult, and planning plus demonstration reuse produce corrective labels that improve the policy. The updated policy can be probed again in a subsequent round. Unlike an online world-model controller, the loop operates during data construction and training, and deployment still uses the original policy interface. CUPID primarily addresses training-data selection, while Ctrl-World studies imagined experience. ReWorld combines local imagination with policy-specific recovery diagnosis and executable corrective supervision.
The controlled experiments offer evidence that this integration matters. In Figure 3(a), 1,000 policy selected corrective samples achieve 80.12% task success, approaching the 80.44% obtained using 4,000 unfiltered corrective samples. The result indicates that recovery aware selection can make corrective supervision substantially more data efficient. The competence allocation study in Table 2(b) addresses a different question, how to distribute that selected supervision across tasks. Under the compared corrective budget, adaptive allocation maintains 68% success for the low competence group while improving the high competence group from 82% to 91%.
Across 50 RoboTwin tasks, ReWorld improves π0.5 from 75.64% to 83.32% and SmolVLA from 38.72% to 47.76%. Four real-robot tasks also improve. The deployment interface remains unchanged, with no world model or planner required for inference.
The main takeaway is the complete offline world-model and policy learning loop, not a standalone sampling heuristic. Action-grounded counterfactuals reveal local recovery gaps, policy feedback prioritizes them, and executable labels turn them into corrective learning signals. Competence-aware allocation then balances this supervision across tasks. This is distinct from simply adding synthetic observations or reweighting existing demonstrations.