acceptodds
Under review as a conference paper at ICLR 2027

Policy-Level Recursive Self-Improvement for Embodied AI with a Criticality World Model

Abstract

Despite advances in policy pretraining, embodied AI systems can plateau during task-specific fine-tuning as uniform scenario collection encounters fewer of the remaining failures. We address this problem with a policy-level recursive self-improvement loop: execution outcomes train a criticality world model, whose risk scores guide scenario collection for specialist training. To account for the resulting shift in scenario frequencies, weighted resampling approximately corrects the collection bias. Because specialized updates can weaken nominal behavior, a risk-based gate selects between the specialist and a frozen nominal policy at inference. We extend this procedure to reinforcement learning and behavior cloning. A sampling analysis characterizes the correction, while a controlled study examines the trade-off between critical and nominal coverage. Across five embodied domains, the composed systems reduce failure rates by 59–67% for locomotion and manipulation and 8–25% for VLA benchmarks relative to their respective baselines.

Primary area:
learning theory
Keywords:
embodied AI, recursive self-improvement, criticality world model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DS6tOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for improving embodied policies through criticality-guided data collection and specialist training. A criticality model is trained on policy rollout outcomes to estimate the risk associated with a given task state. These scores guide data selection from expert demonstrations, while inverse-proposal resampling approximately corrects the resulting sampling bias. The selected data are used to train a specialist policy, which is combined with a frozen nominal policy through a risk-based gate during inference.

Critical strengths
  • The paper addresses a practically relevant problem. As policies improve, random collection becomes less effective at discovering remaining failures. The proposed framework provides a clear and implementable approach to targeting these failures.
  • The criticality-guided data selection and resampling scheme is novel in the field of Embodied AI, and the overall design is well motivated. The framework recognizes that fine-tuning on a small set of difficult scenarios may compromise previously acquired capabilities. Sampling correction and risk-based composition with a frozen nominal policy offer a reasonable approach to balancing targeted improvement and capability preservation.
  • The evaluation covers diverse domains and includes useful comparisons. Experiments span locomotion, manipulation, VLA benchmarks, and a physical robot. Random + gate, specialist-only, and fixed-gate comparisons help clarify the sources of improvement. The ManiSkill results also provide evidence of training gains without selective gating.
Critical weaknesses
  1. The independent contribution of sampling correction is not established.

Criticality-guided selection and inverse-proposal resampling are evaluated together, making it difficult to determine whether the improvements come from selecting informative scenarios, correcting their sampling distribution, or their combination. The authors should compare: (1) random data selection (which is already included in Random + Gate) , (2) criticality-guided selection without correction, and (3) criticality-guided selection with correction (which is already included in Ours) . The latter two should use exactly the same collected data buffer, with matched optimization settings and a fixed gate. Besides, I think reporting performance on both nominal and high-risk scenarios would help establish whether correction improves difficult-scenario learning or primarily mitigates degradation elsewhere.

  1. The correspondence between criticality scores and scenario initializations is insufficiently illustrated.

Aggregate numerical results provide limited intuition about which scenarios the model considers difficult. A scatter visualization of candidate scenarios and their criticality scores, annotated with representative scene images and execution outcomes, would make the learned risk estimates more interpretable.

  1. The BC/VLA evaluation lacks a rejection-sampling fine-tuning baseline using successful rollouts.

In particular, the LIBERO and RoboTwin experiments use policy rollouts to train the criticality model, while policy updates rely primarily on existing expert demonstrations. A natural alternative is to retain all successful base-policy rollouts and directly fine-tune the base policy on these trajectories, i.e., rejection-sampling fine-tuning (RFT). This baseline would help assess whether directly exploiting successful rollout data can provide improvements comparable to criticality-guided selection and resampling within the existing expert dataset. It could also clarify the value of potentially broader trajectory coverage. Comparisons should match rollout and optimization budgets and evaluate both standalone policies and compositions using the same gating mechanism.

  1. Recursive self-improvement is not sufficiently demonstrated in the BC/VLA settings.

Although the RL experiments include multiple updates, each BC/VLA experiment performs only one update. These results do not establish whether the framework can continually identify remaining failures as the policy changes and obtain further improvements through subsequent rounds. The authors are encouraged to report multiple self-improvement rounds on at least one BC/VLA benchmark. This would show whether improvements accumulate. Otherwise, the claims should be narrowed to distinguish demonstrated single-round policy improvement from multi-round recursive self-improvement.

Questions for authors
  1. How necessary is inverse-proposal resampling? Can the authors compare training with and without Pq\frac{P}{q} correction on exactly the same guided data buffer, while fixing optimization settings and the gate? Does correction primarily improve difficult-scenario performance or preserve nominal capabilities?

  2. What concrete scenario characteristics do the criticality scores capture? Can the authors provide a scatter visualization linking scenario initializations and risk scores, annotated with representative scene images and actual execution outcomes?

  3. How does the method compare with directly learning from successful policy rollouts? Can the authors add an RFT baseline for LIBERO/RoboTwin that fine-tunes the base policy on all successful collected trajectories, with matched rollout and optimization budgets and controlled gating?

3 comments
AuthorsOpenReview ↗

Response to Reviewer DS6t

Response to Reviewer DS6t

We thank the reviewer for the constructive questions about correction, scenario interpretation, RFT, and repeated BC updates.

1. Inverse-proposal correction. Correction addresses the scenario-distribution shift introduced by guided collection. We compared the original Go2 Round1 policy with a correction-disabled counterpart trained on the same archived buffer, using gate 0.8 and 120,000 evaluation episodes per policy. Ordinary importance-weighted failure rates are 0.0358% for baseline, 0.0299% with correction, and 0.0359% without correction. Removing correction increases failure by 0.00600 percentage points (pp), or 20.1% relatively. These newly computed estimates condition on fixed policies and retain importance-weight uncertainty. Appendix Q reports this comparison.

2. What criticality captures. Scores depend on the policy and available context. LIBERO uses a 92-D padded initial-state vector, RoboTwin a 437-D object-level state, Go2 the robot state, heightmap, and candidate terrain perturbation, and ManiSkill the observation and candidate force.

Figure 10 (Appendix S) combines two bump interventions with two Go2 states. At the earlier state, mild/strong interventions score 0.0008/0.0054 and remain failure-free through 9.2/8.4 s. At the later state, scores rise to 0.0075/0.6074. The strong intervention causes rear-thigh self-contact after 0.8 s (498 N), while the mild one remains failure-free through 4.0 s. Both state and intervention affect risk.

To extend this analysis to physical manipulation, Figure 11 links scores to paired baseline/Random/Ours outcomes in 50 importance-sampled physical trials, with gate 0.15. Scene 20 (score 0.633) fails under baseline and Random but succeeds under Ours, while scenes 41 (0.741) and 05 (0.754) fail and succeed under all three, respectively. Scene illustrations are synthetic, while execution outcomes come from physical trials.

We also quantified score–outcome agreement on 2,000 LIBERO training scenes with three baseline rollouts each. Score–failure Spearman correlation is 0.180 pooled and 0.025 after removing task-level differences. This indicates limited within-task discrimination in this diagnostic. To examine whether high risk also predicts specialist benefit, we analyzed paired policy outcomes. The highest-score bin has 36 repairs and 28 regressions across 570 executions, giving a net 1.40 pp benefit if all are switched. The coexistence of repairs and regressions illustrates why nominal risk and specialist advantage are distinct.

3. Rejection-sampling fine-tuning. To test the proposed successful-rollout alternative, we collected 1,000 fresh baseline-policy attempts (25/task), excluding official test starts, retained successful trajectories, and applied BC to the policy's executed actions. Training uses 40,000 updates, batch size 32, learning rate 5e-6, and 2,000 warmup steps. Collection cost is separate from evaluation. Three matched evaluation RNG seeds give:

Policy/routingFailures by seedMean failure
Matching baseline36/40/351.85%
RFT standalone48/43/322.05%
Original gated RFT34/40/331.78%
Recalibrated low-risk gate31/45/401.93%

The comparison uses RFT's own matching baseline, separate from the additional-seed control. To test whether RFT helps on easier scenes, the low-risk rule routes below a threshold selected on independent validation data. The original gate gives a small 0.07 pp reduction, while standalone RFT and low-risk routing have higher mean failure than baseline. Thus these protocols show no consistent RFT improvement. Limited exposure to challenging failures is one possible explanation for successful-only training.

4. Repeated BC versus RL updates. To test whether BC gains accumulate, a separate LIBERO diagnostic continues from its own first-round policies. For Random/Frozen-Ours/Refresh-Ours, standalone failure changes from 1.82/2.13/2.13% to 1.80/2.25/2.15%, while gated failure changes from 1.68/1.40/1.40% to 1.80/1.57/1.53%. Gains do not accumulate consistently in this BC setting.

This BC outcome differs from the separate five-round physical piRL experiment, which yields guided standalone success of 75%/85% at pool sizes 5/10, versus Random's 60%/67.5% and baseline's 60%, over two tests of 20 scenes. These endpoints support improvement through multiple physical RL updates. Section 5.1 clarifies the main BC/VLA results' single-update scope, and Appendix R reports both diagnostics.

Thank you for your added experiments and corresponding analyses. This rebuttal has addressed most of my concerns, and I'm raising my rating.

Thank you for your thoughtful feedback and for updating your assessment. Your comments helped us strengthen the experimental evidence and clarify the scope of our claims. We appreciate your time and constructive engagement throughout the review process.

Reviewer_MigxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a policy-level recursive self-improvement procedure for embodied AI. From the execution outcomes of the current policy, the method trains a lightweight criticality world model (hereafter, the criticality model). Its risk scores are used to enrich potential failure cases among a finite set of candidate scenes. The method then collects data using a proposal distribution with a uniform exploration component and applies inverse-proposal probability resampling to approximately recover the original target distribution. Because concentrating training on critical cases can damage normal behavior, deployment retains a frozen nominal policy and uses a risk-based gate to choose between the nominal policy and a specialist policy. The authors instantiate the procedure for reinforcement-learning policies in Go2 and ManiSkill, and for behavior-cloning/VLA policies in LIBERO, RoboTwin, and the AgileX Piper robot. The paper reports lower failure-rate point estimates across five domains, including importance-sampling estimates for Go2 and the physical task: Go2 decreases from 3.58×10−43.58×10^{-4} to 1.47×10−41.47×10^{-4}, ManiSkill reaches 2.08%, and LIBERO, RoboTwin, and the real robot reach 1.40%, 13.30%, and 3.72%, respectively. It also provides an importance-sampling identity, a discussion of finite-buffer bias, a coverage toy study, a fixed-gate Random + gate comparison, and paired uncertainty analyses. At the same time, the authors explicitly acknowledge that the independent effects of multi-round training, criticality-model updates, and resampling have not been separated, and that some results are descriptive best values selected from test curves.

Critical strengths
  1. The criticality model, proposal distribution with positive support, inverse-proposal weights, specialist updates, and risk-based gate form a coherent method. The main text and appendix clearly distinguish the ideal importance-sampling identity from the bias of the self-normalized estimator with a finite buffer, and do not mislabel risk scores as calibrated probabilities.

  2. The study includes RL-pretrained Go2 and ManiSkill, multiple VLA/BC benchmarks, and real Piper tasks. It reports shared-round, specialist-only, Random + gate, and fixed-gate comparisons, demonstrating applicability across different policy interfaces.

  3. The paper reports paired bootstrap intervals, the effective sample size for the physical experiment (ESS = 12.44 over 50 trials), the threshold-selection procedure, and known data quantities. The controlled coverage experiment also shows that increasing coverage of critical cases trades off against nominal coverage, making the limits of the conclusions clearer than a report of average success rates alone.

Critical weaknesses
  1. Criticality-model-driven hard-case sampling, inverse-proposal resampling, specialist/nominal gating, and recursive updating are each close to existing ideas in active data selection, importance weighting, policy composition, and failure-driven self-improvement. Section 2 explicitly positions the work as extending the risk-guided framework of Feng et al. (2026) across RL and BC settings, and Section 3 provides a sampling analysis. The remaining concern is that the added value of the resampling and recursive predictor updates is not sufficiently isolated experimentally; this limits the evidence for the incremental contribution of the extension.

  2. The paper does not provide matched experiments that remove inverse-proposal resampling, freeze the initial criticality model, or update only one of these components over multiple rounds, so the results reflect the complete procedure. The two Go2 methods use different update hyperparameters. ManiSkill uses different final round budgets, but also reports a same-round comparison at Round 12 (2.34% versus 3.36% failure), supporting a training gain at a common round count without isolating resampling and predictor refresh. The table’s “best observed” values are selected from test curves, which can conflate training budget, checkpoint selection, and gating gains.

  3. Gating in LIBERO and RoboTwin requires 92-D/437-D simulator states rather than the policy’s visual inputs; the real-robot experiment relies on a task-specific RGB-coordinate extractor. These input requirements limit the deployment transferability of the claimed cross-embodied-system generality and make reproduction for vision-based end-to-end policies in physical environments more demanding.

Questions for authors
  1. How sensitive is criticality-based scenario ranking to label definitions and predictor errors, and how are persistent ranking errors prevented from biasing later collection rounds?

  2. For the cumulative-weight guard in Appendix F.3, could you report training-buffer ESS, trigger frequency, and sensitivity to the 10−210^{-2} threshold?

  3. For Go2's per-step gate and the initial-scene gates, how robust are routing decisions to risk-score fluctuations, observation errors, and subsequent state changes?

  4. How does the recursive loop detect and correct predictor bias that could be amplified through repeated criticality-guided collection and specialist updates?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Migx

Response to Reviewer Migx

We thank the reviewer for the detailed mechanism questions.

1. Contribution and correction. We implement risk-guided improvement across RL and BC interfaces, building on failure prediction, importance weighting, and policy composition. BC correction operates over scenes/demonstrations, preserving the supervised objective.

With the same archived Go2 Round1 buffer, gate 0.8, and 120,000 evaluation episodes per policy, ordinary importance-weighted failure is 0.0358% for baseline, 0.0299% with correction, and 0.0359% without it. Removing correction increases failure by 0.00600 percentage points (pp), or 20.1% relatively. These fixed-policy estimates retain importance-weight uncertainty (Appendix Q).

2. Refresh and ranking errors. Ranking errors can repeatedly steer collection toward less informative scenarios. At each round, however, the uniform proposal component preserves support, and exact weighting gives E_q[(p/q)g] = E_p[g] for the fixed target integrand, regardless of predictor accuracy. Thus predictor bias affects collection efficiency but leaves the expected gradient direction unchanged under exact correction. Finite buffers and normalized resampling can still produce estimation error across rounds. New rollout labels allow refresh to adapt to policy-dependent risk, while a frozen predictor may remain useful across nearby updates.

To assess the practical value of refresh, ManiSkill checkpoints were selected by mean failure on independent disturbed validation data (200 episodes/task/candidate). Separate tests of 2,000 episodes/task/model give Frozen/Refresh mean failure of 3.10%/3.25%, a +0.15 pp change. We observed no refresh benefit in this comparison (Appendix Q).

3. Labels and predictor sensitivity. To test label-error sensitivity, we flipped 0%, 5%, or 10% of episode labels on fixed collection data from one corrected Round2 policy. Across three predictor-training seeds, features were fixed and validation/test labels remained clean. For predictor seed 17, test average precision was 0.717/0.680/0.734 on Stack and 0.854/0.806/0.756 on Peg. Holding the failure definition fixed isolates failure-ranking sensitivity to label corruption.

4. Cumulative-weight guard. If a proposed perturbation would lower cumulative weight below the threshold, collection falls back to uniform sampling with step weight one, retaining previous accumulated weight and the episode's outcome label.

We measured guard activation in 200 paired episodes/task. At thresholds 0.001/0.01/0.1, counts were 0/0, 0/2, and 8/5 for Stack/Peg. At default 0.01, normalized terminal cumulative-weight ESS was 0.760/0.567 and per-step-weight ESS was 0.983/0.991. Disabling the guard gives cumulative ESS 0.760/0.581, versus 0.859/0.634 at 0.1. In training, ManiSkill uses per-step weights for resampling and unweighted predictor labels.

5. Noisy routing. For a fixed Go2 policy and gate, 100,000 paired episodes at gate-input velocity noise 0/0.01/0.03 give failure 0.033/0.033/0.032% and specialist usage 0.0468/0.0469/0.0476% of steps. Because policy inputs were unchanged, this isolates routing sensitivity. The largest noise level repaired one failure and introduced none, supporting robustness to the tested noise. Go2 routes per step, while the initial-scene gates retain their reset-time decision throughout an episode.

6. Go2 settings. Appendix E.2 corrects the shared advantage temperature, BC coefficient, failure penalty, and margin to 0.5, 0.5, 3.0, and 1.0. Within the first three rounds, best observed failure is 2.48e-4 for Ours and 3.57e-4 for Random. Random's subsequent rates are 3.99e-4 and 3.89e-4. We use the same descriptive best-observed rule and retain full curves. Deployment thresholds are selected separately on validation data.

7. Inputs and transferability. Our interfaces are domain-specific: LIBERO/RoboTwin use simulator state, while physical BC uses visual coordinates. To explore a richer visual interface, we built a prototype with DINOv2, dynamic/environment encoders, auxiliary camera/robot state, and an editor. Figure 12 (Appendix T) shows background and illumination edits with approximate layout retention. Its risk predictor, however, overfits and is sensitive to duration, with separation sometimes emerging after successful traces terminate. Early prediction and closed-loop collection therefore require further evaluation.

We also tested whether iterative improvement extends to physical RL. After five piRL rounds, guided standalone success reaches 75%/85% at pool sizes 5/10, versus Random's 60%/67.5% and baseline's 60%, over two tests of 20 scenes. Within each setting, initialization, rewards, and optimization match, with correction applied to guided collection. These gains support iterative improvement on hardware, while inconsistent accumulation in the LIBERO diagnostic concerns BC. Section 5.1 and Appendix R clarify this distinction.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Reviewer_xwXnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends criticality-guided policy improvement to reinforcement learning and behavior cloning. A risk predictor trained on rollout outcomes guides scenario or demonstration selection. Inverse-proposal resampling approximately corrects the resulting distribution shift during specialist training, while a deployment gate selects between the specialist and a frozen nominal policy. The framework is evaluated on Go2, ManiSkill, LIBERO, RoboTwin, and a physical manipulation task. Go2 and ManiSkill include multiple update rounds; the imitation-learning settings use one specialist update. Component evaluations, Random + gate baselines, fixed-gate comparisons, and paired uncertainty analyses examine the contributions of training and policy composition.

Critical strengths

The cross-system implementations and BC/VLA adaptations have practical value. The paper develops domain-specific perturbation interfaces, risk inputs, failure labels, and policy updates. The BC extension separates rollout-based risk supervision from expert-action supervision. Computing correction ratios over scenes or demonstrations avoids requiring tractable action densities and preserves existing supervised objectives, providing a useful interface for flow- and diffusion-based policies.

The experiments provide meaningful evidence of training gains beyond selective gating. At Round 12 in ManiSkill, failure rates are 2.34% for Ours and 3.36% for Random, with both using \tau = 0 and identical per-round optimizer settings. Random + gate and fixed-gate comparisons further strengthen the evaluation. The paper also carefully distinguishes exact importance-sampling identities from finite-buffer approximations and explicitly acknowledges important experimental limitations.

Critical weaknesses

Comparison with existing demonstration-curation methods is insufficient. The BC/VLA contribution should be positioned against methods such as CUPID, which also use policy rollout outcomes to guide demonstration curation. Please clarify how risk-based scene selection and inverse-proposal resampling differ from their selection objectives and mechanisms. A comparison on a representative BC/VLA benchmark, with matched starting checkpoints, candidate demonstration pools, selection budgets, and fine-tuning settings, would help establish the method’s incremental value. Reporting both standalone and consistently gated performance, alongside rollout and curation costs, would further distinguish data-selection benefits from policy-composition gains.

The incremental value of data selection and the associated training mechanisms remains insufficiently isolated. As acknowledged in Appendix N.4, there is no matched ablation removing inverse-proposal resampling or freezing the initial risk predictor across rounds. The experiments establish the performance of the complete procedure, but leave unclear how much these mechanisms contribute beyond risk-guided collection followed by ordinary fine-tuning. Equation (2) motivates sampling through gradient magnitude, while its relationship to the implemented risk scores is not quantified.

A complementary baseline is particularly relevant for LIBERO. All 2,000 demonstrations are already used during the common 40K-step warmup, after which both continuation arms select approximately 1,000 demonstration slots. Continuing on all 2,000 demonstrations for another 40K steps, from the same checkpoint and with identical continuation settings, would test whether selection adds value over simply continuing training on the available data. This would complement the existing comparison under a fixed subset budget.

The robustness and interpretation of gated improvements over strong controls need further validation. Against Random + gate, LIBERO improves by 0.28 percentage points, RoboTwin improves from 14.00% to 13.30% and the physical task improves by 4.81 percentage points. These estimates are conditional on fixed policies and thresholds; they do not capture variation across training seeds or validation scenes used for threshold selection.

It is also unclear how risk rankings align with the specialist’s relative advantage. A high failure-risk score does not by itself establish that the specialist will outperform the nominal policy. The reported aggregate repair and regression counts are useful; stratifying these paired outcomes by risk score would further show where routing helps or hurts and explain why risk-based thresholding is an effective selection rule.

Questions for authors

Mechanism validation: In one representative domain, can you compare uniform collection, risk-guided collection without correction, and risk-guided collection with correction under matched collection budgets, update settings, and routing? A frozen-predictor variant would clarify the contribution of predictor refresh.

Full-data continuation: For LIBERO, how does the method compare with continuing on all 2,000 demonstrations for another 40K steps from the common warmup checkpoint, using the same optimization settings and gate-selection protocol? This would clarify the practical benefit of subset selection when the full demonstration pool is already available.

Robustness and reproducibility: On at least one principal VLA benchmark, can you report paired comparisons against Random + gate across independent training seeds and different validation subsets used to select \tau? Providing the corresponding per-scene outcomes, proposal weights, and threshold-selection records would support independent verification.

Routing diagnostics: Using the existing paired outcomes, can you report repair rates, regression rates, and net improvement across risk-score bins, together with sample counts and uncertainty? This would clarify whether higher-risk scenes are also those where specialist intervention is most beneficial.

2 comments
AuthorsOpenReview ↗

Response to Reviewer xwXn

Response to Reviewer xwXn

We thank the reviewer for the questions about stronger controls, mechanism evidence, and score interpretation.

1. CUPID comparison. Using the official diffusion curator and influence/TRAK scoring across 40 LIBERO tasks, we selected 25 of 50 demonstrations/task. CUPID used the same seed-43 parent, 40,000 updates, batch size 32, learning rate 5e-6, and 2,000 warmup steps. With 2,000 official starts per RNG seed:

PolicyFailures: RNG 7/11/23Mean
Parent39/36/341.82%
CUPID37/34/401.85%
CUPID + gate37/37/321.77%
Ours + gate28/33/301.52%
Random + gate37/35/311.72%
Full + gate35/38/341.78%

CUPID's gate was selected on 1,000 independent validation episodes and frozen before testing, routing 393/2,000 starts to CUPID. Ours + gate has 0.25 pp lower failure. CUPID's official implementation targets diffusion policies, whereas our flow-matching parent VLA uses different training objectives and action generation and cannot be substituted directly. We therefore use an independently trained curator, whose high-value data may be misaligned with the parent's weaknesses. This offers a possible explanation for the gap. Downstream slots and updates match, while unique demos number 1,000/1,025/2,000 for CUPID/Ours/Random. CUPID incurs additional curator training, rollout, and scoring costs.

2. Correction, refresh, and gradients. With the same Go2 Round1 buffer, gate 0.8, and 120,000 evaluation episodes/policy, ordinary importance-weighted failure is 0.0358% for baseline, 0.0299% with correction, and 0.0359% without it. Removing correction increases failure by 0.00600 percentage points (pp), or 20.1% relatively. These estimates condition on fixed policies and retain importance-weight uncertainty.

To assess refresh separately, we conduct separate tests of 2,000 episodes/task/model, which gives Frozen/Refresh failure of 3.10%/3.25%. Refresh provides no observed benefit here (Appendix Q).

We also tested the gradient motivation. Across 100 guided physical demonstrations, criticality versus BC-gradient norm has Spearman 0.077 and Pearson 0.042. Across 100,000 Go2 trajectories, local score–gradient Spearman correlations are 0.3078 [0.3064, 0.3092] for the joint objective and 0.2841 [0.2827, 0.2855] for the actor, using episode-cluster resampling. Separately, 11,813 trajectories give mean joint gradient norms 858.91 for failures versus 325.67 for non-failures, a difference of 533.24 (95% interval [508.47, 560.10]). Accordingly, Sections 3.1–3.2 now restrict the failure-risk gradient motivation to RL. BC is a theory-inspired extension obtaining informative expert data and broader critical coverage, supported by the toy study. Appendix P reports the diagnostics.

3. Full-data controls and robustness. An additional LIBERO training seed, evaluated with three RNG seeds over 2,000 starts each, gives:

PolicyStandalone failureValidation-frozen gate
Baseline1.82%—
Full-data continuation1.92%1.78%
Random2.03%1.72%
Ours2.00%1.52%

With 40,000 updates for all continuations, Ours + gate improves on Random + gate by 0.20 pp. Ours/Random each use 2,000 slots containing 1,025/2,000 unique demonstrations. These RNG seeds measure evaluation variability (Appendix Q).

We reselected seed-43 thresholds on ten prespecified overlapping validation subsets (781–826/1,000 scenes), minimizing per-suite failures with ties favoring the highest threshold. Dominant choices for Ours/Random were Spatial 0.60/0.65 (9/7 subsets), Object parent-only (10/10), Goal 0.40 (8/9), and LIBERO-10 0.90 (9/7). Test thresholds are frozen after full independent validation. The similar thresholds, also seen in the paper, are consistent with both gates using the same parent's risk scores to identify weaker scenarios. Guided training targets these scenarios, while Random provides smaller gains under comparable routing, as shown by the paper's fixed-gate comparisons.

4. Risk and specialist advantage. To assess the benefit of switching policies at different risk levels, we grouped paired LIBERO outcomes by score:

Score binExecutionsRepairsRegressionsNet benefit (pp)
[0, 0.25)1,53902−0.13
[0.25, 0.50)2,2893141−0.44
[0.50, 0.75)1,6022234−0.75
[0.75, 1]5703628+1.40

Repairs convert nominal failures to specialist successes, while regressions do the reverse. Net benefit is 100 × (repairs − regressions)/executions in pp. The highest-score bin has a positive net benefit, while lower bins have more regressions than repairs. Counts assume switching all scenes in each bin, while deployment uses suite-specific thresholds. Figures 10–11 (Appendix S) provide complementary Go2 and physical scenario/outcome examples.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

General Response

We thank the reviewers for their constructive comments. We have added mechanism tests, stronger controls, and scope clarifications in our manuscript.

1. Correction and predictor refresh (Appendix Q). To isolate correction, we compared policies trained on the same Go2 Round1 buffer. With gate 0.8 and 120,000 evaluation episodes per policy, ordinary importance-weighted failure rates are 0.0358% for baseline, 0.0299% with correction, and 0.0359% without correction. Removing correction increases failure by 0.00600 percentage points (pp), or 20.1% relatively. These newly computed estimates condition on fixed trained policies and retain importance-weight uncertainty. Appendix E.2 also corrects the shared Go2 settings to advantage temperature 0.5, BC coefficient 0.5, failure penalty 3.0, and margin 1.0.

To assess the additional value of refresh, ManiSkill Frozen and Refresh checkpoints were selected on independent disturbed validation data and tested on 2,000 episodes per task per model. Mean failure rates are 3.10% and 3.25%. A fixed predictor remains useful in this experiment, supporting optional refresh.

2. Additional LIBERO controls (Appendix Q). To test robustness beyond the original training seed and compare against full-data continuation, we evaluated an additional independent seed with three evaluation RNG seeds, each using the same 2,000 official task/initial-state pairs:

PolicyStandalone failureValidation-frozen gate
Baseline1.82%—
Full-data continuation1.92%1.78%
Random2.03%1.72%
Ours2.00%1.52%

With 40,000 updates for all continuations, Ours + gate improves on Random + gate by 0.20 pp. Ours and Random each use 2,000 slots, containing 1,025 and 2,000 unique demonstrations, respectively. The three RNG seeds measure evaluation variability within this additional training seed.

3. Successful-rollout fine-tuning. To compare guided collection with learning from successful executions, RFT retains successful trajectories from 1,000 fresh baseline rollouts and uses 40,000 BC updates on executed actions. Across three evaluation RNG seeds, its matching baseline has 1.85% failure, RFT standalone 2.05%, the original gated RFT 1.78%, and independently recalibrated low-risk routing 1.93%. This baseline is separate from the additional-seed comparison above. RFT provides no consistent improvement under these protocols. DS6t-3 gives the paired seed results.

4. Theory and iterative improvement. Sections 3.1–3.2 now restrict the failure-risk gradient motivation to RL. BC is a theory-inspired extension that obtains informative expert data and broader critical coverage, supported by the main-text toy study. This distinction is supported by moderate Go2 score–gradient correlations and weak physical BC correlations (Appendix P).

Section 5.1 states the single-update scope of the main BC/VLA results. Repeated LIBERO BC updates resample a fixed demonstration pool and show inconsistent gains. RL rounds instead collect fresh trajectories: after five physical piRL rounds, guided success is 75%/85% at pool sizes 5/10, versus Random's 60%/67.5% and baseline's 60%, over two tests of 20 scenes. Physical BC also acquires new demonstrations, yielding gated failure of 3.72% versus Random's 8.53% at the same threshold. Together, these results suggest that continued improvement benefits from new data targeted to the policy's remaining weaknesses. Section 5 and Appendix R develop this insight, distinguishing new policy-training data from optional predictor refresh.

5. Scenario interpretation and deployment. To make the scores interpretable through concrete outcomes, Appendix S adds Go2 intervention cases (Figure 10) and paired physical outcomes versus initial-scene criticality (Figure 11). Physical scene illustrations are synthetic, while outcomes come from robot trials. The input interfaces remain domain-specific: LIBERO/RoboTwin use simulator state, while physical BC uses visual coordinates. To explore a richer visual interface, Appendix T presents a visual-latent editing prototype (Figure 12). It uses auxiliary state inputs, with early prediction and downstream collection still requiring evaluation.

6. CUPID comparison. Across all 40 LIBERO tasks, the official curator and influence/TRAK procedure selected 25 demonstrations/task, repeating 1,000 unique demonstrations into 2,000 slots. Training used the same seed-43 parent and 40,000 updates. Over RNG seeds 7/11/23 (2,000 starts each), CUPID had 37/34/40 standalone failures (1.85%) and 37/37/32 gated failures (1.77%). The gate was frozen after independent validation. Ours + gate had 28/33/30 failures (1.52%), improving on CUPID + gate by 0.25 pp. CUPID also incurs curator training, rollout, and scoring costs. The detailed comparison appears in our response to xwXn.

Dear Reviewers,

Thank you for your constructive comments. As the discussion phase is nearing its end, we would like to briefly summarize how our revisions have addressed the shared concerns across three key aspects:

  1. Mechanism evidence and stronger comparisons (DS6t, Migx, xwXn): Removing correction increases Go2 failure from 0.0299% to 0.0359%, while ManiSkill’s Frozen/Refresh results support optional predictor refresh. We added full-data continuation, successful-rollout fine-tuning, an independent LIBERO training seed, and CUPID. Across three evaluation RNG seeds, gated failure rates are 1.52% for Ours, 1.72% for Random, 1.77% for CUPID, and 1.78% for full-data continuation. Further diagnostics examine calibration sensitivity, label corruption, weight safeguards, and noisy routing.

  2. Iterative improvement and informative data (DS6t, Migx): Section 5 and Appendix R distinguish predictor refresh from acquiring new policy-training data. Repeated LIBERO BC updates reuse a fixed demonstration pool and show inconsistent gains, whereas RL acquires fresh trajectories and supports multi-round improvement. Physical BC also collects new demonstrations and shows a larger reduction. Since Random receives new data but achieves smaller gains, these results suggest that continued improvement benefits from new examples targeted to the policy’s remaining weaknesses.

  3. Theory, interpretation, and implementation (DS6t, Migx, xwXn): Sections 3.1–3.2 restrict the gradient motivation to RL and frame BC through informative expert data and critical-scenario coverage, supported by Appendix P and the toy study. Our scene-level correction enables this extension while retaining existing VLA supervised objectives. Figures 10–11 connect criticality scores to concrete interventions and outcomes. We also corrected Go2 settings in Appendix E.2, clarified deployment inputs, and added visual-latent editing examples in Appendix T.

Together, these revisions sharpen the paper’s central insight: iterative embodied learning benefits from acquiring informative data targeted to the current policy’s remaining weaknesses. Our experiments examine how this principle operates across RL and expert-supervised VLA updates, including the contrast between fixed demonstration pools and newly collected training data.

We believe these responses and additional experiments have resolved your initial concerns. Given the limited time remaining, if you have any further questions, please let us know immediately so we can address them.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.