- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for improving embodied policies through criticality-guided data collection and specialist training. A criticality model is trained on policy rollout outcomes to estimate the risk associated with a given task state. These scores guide data selection from expert demonstrations, while inverse-proposal resampling approximately corrects the resulting sampling bias. The selected data are used to train a specialist policy, which is combined with a frozen nominal policy through a risk-based gate during inference.
Critical strengths
- The paper addresses a practically relevant problem. As policies improve, random collection becomes less effective at discovering remaining failures. The proposed framework provides a clear and implementable approach to targeting these failures.
- The criticality-guided data selection and resampling scheme is novel in the field of Embodied AI, and the overall design is well motivated. The framework recognizes that fine-tuning on a small set of difficult scenarios may compromise previously acquired capabilities. Sampling correction and risk-based composition with a frozen nominal policy offer a reasonable approach to balancing targeted improvement and capability preservation.
- The evaluation covers diverse domains and includes useful comparisons. Experiments span locomotion, manipulation, VLA benchmarks, and a physical robot. Random + gate, specialist-only, and fixed-gate comparisons help clarify the sources of improvement. The ManiSkill results also provide evidence of training gains without selective gating.
Critical weaknesses
- The independent contribution of sampling correction is not established.
Criticality-guided selection and inverse-proposal resampling are evaluated together, making it difficult to determine whether the improvements come from selecting informative scenarios, correcting their sampling distribution, or their combination. The authors should compare: (1) random data selection (which is already included in Random + Gate) , (2) criticality-guided selection without correction, and (3) criticality-guided selection with correction (which is already included in Ours) . The latter two should use exactly the same collected data buffer, with matched optimization settings and a fixed gate. Besides, I think reporting performance on both nominal and high-risk scenarios would help establish whether correction improves difficult-scenario learning or primarily mitigates degradation elsewhere.
- The correspondence between criticality scores and scenario initializations is insufficiently illustrated.
Aggregate numerical results provide limited intuition about which scenarios the model considers difficult. A scatter visualization of candidate scenarios and their criticality scores, annotated with representative scene images and execution outcomes, would make the learned risk estimates more interpretable.
- The BC/VLA evaluation lacks a rejection-sampling fine-tuning baseline using successful rollouts.
In particular, the LIBERO and RoboTwin experiments use policy rollouts to train the criticality model, while policy updates rely primarily on existing expert demonstrations. A natural alternative is to retain all successful base-policy rollouts and directly fine-tune the base policy on these trajectories, i.e., rejection-sampling fine-tuning (RFT). This baseline would help assess whether directly exploiting successful rollout data can provide improvements comparable to criticality-guided selection and resampling within the existing expert dataset. It could also clarify the value of potentially broader trajectory coverage. Comparisons should match rollout and optimization budgets and evaluate both standalone policies and compositions using the same gating mechanism.
- Recursive self-improvement is not sufficiently demonstrated in the BC/VLA settings.
Although the RL experiments include multiple updates, each BC/VLA experiment performs only one update. These results do not establish whether the framework can continually identify remaining failures as the policy changes and obtain further improvements through subsequent rounds. The authors are encouraged to report multiple self-improvement rounds on at least one BC/VLA benchmark. This would show whether improvements accumulate. Otherwise, the claims should be narrowed to distinguish demonstrated single-round policy improvement from multi-round recursive self-improvement.
Questions for authors
-
How necessary is inverse-proposal resampling? Can the authors compare training with and without correction on exactly the same guided data buffer, while fixing optimization settings and the gate? Does correction primarily improve difficult-scenario performance or preserve nominal capabilities?
-
What concrete scenario characteristics do the criticality scores capture? Can the authors provide a scatter visualization linking scenario initializations and risk scores, annotated with representative scene images and actual execution outcomes?
-
How does the method compare with directly learning from successful policy rollouts? Can the authors add an RFT baseline for LIBERO/RoboTwin that fine-tunes the base policy on all successful collected trajectories, with matched rollout and optimization budgets and controlled gating?
3 comments
Thank you for your added experiments and corresponding analyses. This rebuttal has addressed most of my concerns, and I'm raising my rating.
Thank you for your thoughtful feedback and for updating your assessment. Your comments helped us strengthen the experimental evidence and clarify the scope of our claims. We appreciate your time and constructive engagement throughout the review process.
Response to Reviewer DS6t
Response to Reviewer DS6t
We thank the reviewer for the constructive questions about correction, scenario interpretation, RFT, and repeated BC updates.
1. Inverse-proposal correction. Correction addresses the scenario-distribution shift introduced by guided collection. We compared the original Go2 Round1 policy with a correction-disabled counterpart trained on the same archived buffer, using gate 0.8 and 120,000 evaluation episodes per policy. Ordinary importance-weighted failure rates are 0.0358% for baseline, 0.0299% with correction, and 0.0359% without correction. Removing correction increases failure by 0.00600 percentage points (pp), or 20.1% relatively. These newly computed estimates condition on fixed policies and retain importance-weight uncertainty. Appendix Q reports this comparison.
2. What criticality captures. Scores depend on the policy and available context. LIBERO uses a 92-D padded initial-state vector, RoboTwin a 437-D object-level state, Go2 the robot state, heightmap, and candidate terrain perturbation, and ManiSkill the observation and candidate force.
Figure 10 (Appendix S) combines two bump interventions with two Go2 states. At the earlier state, mild/strong interventions score 0.0008/0.0054 and remain failure-free through 9.2/8.4 s. At the later state, scores rise to 0.0075/0.6074. The strong intervention causes rear-thigh self-contact after 0.8 s (498 N), while the mild one remains failure-free through 4.0 s. Both state and intervention affect risk.
To extend this analysis to physical manipulation, Figure 11 links scores to paired baseline/Random/Ours outcomes in 50 importance-sampled physical trials, with gate 0.15. Scene 20 (score 0.633) fails under baseline and Random but succeeds under Ours, while scenes 41 (0.741) and 05 (0.754) fail and succeed under all three, respectively. Scene illustrations are synthetic, while execution outcomes come from physical trials.
We also quantified score–outcome agreement on 2,000 LIBERO training scenes with three baseline rollouts each. Score–failure Spearman correlation is 0.180 pooled and 0.025 after removing task-level differences. This indicates limited within-task discrimination in this diagnostic. To examine whether high risk also predicts specialist benefit, we analyzed paired policy outcomes. The highest-score bin has 36 repairs and 28 regressions across 570 executions, giving a net 1.40 pp benefit if all are switched. The coexistence of repairs and regressions illustrates why nominal risk and specialist advantage are distinct.
3. Rejection-sampling fine-tuning. To test the proposed successful-rollout alternative, we collected 1,000 fresh baseline-policy attempts (25/task), excluding official test starts, retained successful trajectories, and applied BC to the policy's executed actions. Training uses 40,000 updates, batch size 32, learning rate 5e-6, and 2,000 warmup steps. Collection cost is separate from evaluation. Three matched evaluation RNG seeds give:
The comparison uses RFT's own matching baseline, separate from the additional-seed control. To test whether RFT helps on easier scenes, the low-risk rule routes below a threshold selected on independent validation data. The original gate gives a small 0.07 pp reduction, while standalone RFT and low-risk routing have higher mean failure than baseline. Thus these protocols show no consistent RFT improvement. Limited exposure to challenging failures is one possible explanation for successful-only training.
4. Repeated BC versus RL updates. To test whether BC gains accumulate, a separate LIBERO diagnostic continues from its own first-round policies. For Random/Frozen-Ours/Refresh-Ours, standalone failure changes from 1.82/2.13/2.13% to 1.80/2.25/2.15%, while gated failure changes from 1.68/1.40/1.40% to 1.80/1.57/1.53%. Gains do not accumulate consistently in this BC setting.
This BC outcome differs from the separate five-round physical piRL experiment, which yields guided standalone success of 75%/85% at pool sizes 5/10, versus Random's 60%/67.5% and baseline's 60%, over two tests of 20 scenes. These endpoints support improvement through multiple physical RL updates. Section 5.1 clarifies the main BC/VLA results' single-update scope, and Appendix R reports both diagnostics.