acceptodds
Under review as a conference paper at ICLR 2027

LoopWAM: Closed-Loop Generalist Policy Improvement from Deployment Rollouts

Abstract

Scaling expert demonstrations alone does not enable generalist robot policies to improve from deployment. Real-world rollouts contain successes, suboptimal behavior, and failures: imitating every recorded action can reinforce poor behavior, while filtering out failures discards evidence of physical interactions and their consequences. We introduce LoopWAM, a closed-loop world-action model that learns directly from this full quality spectrum, using rollouts from both itself and other deployed policies. Source-dependent noise sampling unifies behavior learning and action-conditioned world modeling, while quality steering and quality-weighted supervision retain full video supervision across qualities and regulate action imitation. Video and action tokens interact throughout parallel denoising, allowing learned consequences to inform action generation directly. On RoboChallenge Table30-V2, three deployment-and-training rounds raise success from 23.67% to 43.33%, a 19.7-percentage-point gain, with a final score of 56.80. LoopWAM also achieves 94.5% average success on RoboTwin 2.0 and 49.5% overall success on RoboCasa365 without additional embodied pretraining, obtaining the best overall results among the compared methods across all three benchmarks. On challenging shirt folding task, two rounds of deployment-driven improvement after single-task post-training raise success from 13.20% to 55.56%. These results demonstrate closed-loop improvement from mixed-quality experience under both multi-task learning and single-task post-training. https://loopwam.github.io/orangeProject Page

Primary area:
applications to robotics, autonomy, planning
Keywords:
World-Action Models, Self-Improvement, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BUm7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LoopWAM learns from expert demonstrations and mixed-quality deployment rollouts through joint video-action denoising. Source-dependent noise sampling lets relatively clean rollout actions condition consequence prediction, while quality labels condition both streams and weight action imitation. Failed trajectories retain video supervision but receive no direct action-imitation loss. At deployment, the model uses the expert-quality condition. Across separately trained policies for four robot embodiments, three deployment-and-training updates raise Table30-V2 success from 23.67% to 43.33%. Shirt-folding success rises from 13.20% to 55.56% over two updates after task-specific expert training. RoboTwin 2.0 and RoboCasa365 provide additional simulation results. The repeated real-robot gains support the complete pipeline; the individual contributions of failure data, bidirectional interaction, and numerical quality scoring are less clearly established.

Critical strengths
  1. Repeated improvement on substantial real-robot evaluations. Table30-V2 covers 30 tasks across four robot embodiments, with a separate multitask policy for each embodiment. Overall success improves at all three updates, from 23.67% to 28.00%, 37.67%, and 43.33%. The separate shirt-folding study also improves from 13.20% to 55.56% over two updates after task-specific expert training. These results provide substantial evidence that deployment-driven learning is useful across varied manipulation tasks.

  2. Practical evidence for quality-aware training. With rollout action supervision disabled, numerical quality conditioning raises RoboTwin success from 72.3% to 92.7%; simple low/medium/high labels reach 92.8%. Adding weighted rollout action supervision to the numerical scheme raises success to 94.5% (Table 4). Thus quality conditioning and selective imitation are useful parts of the recipe, and simple quality categories already work well in this setting.

Critical weaknesses
  1. Limited methodological novelty. UWM already couples video and action diffusion with independent noise levels; FACT learns failed-action consequences while masking their imitation loss; and π0.7\pi_{0.7} uses quality metadata to learn from mixed-quality data. LoopWAM cites these precedents and adds a combination of source-dependent noise sampling, bidirectional denoising, and quality-aware supervision. The real-robot results support the integrated approach, but the learning advantage provided by this particular combination, especially the noise design and bidirectional interaction, is less directly established.

  2. The distinct value of failure data is unclear. Each deployment update adds successful rollouts as well as failures. Table 4 varies quality conditioning and rollout action supervision, while retaining the same video-supervision objective on expert and rollout data in every configuration. It therefore does not establish what failed outcomes add beyond the useful behavior in successful rollouts. The repeated gains support the complete pipeline, but provide less direct support for the central claim that learning from failure experience itself improves the policy.

  3. The meaning of numerical quality scores needs clarification. All training segments from an episode share one score. Under coarse annotation, successful rollouts from the same policy also share a score. These scores can reflect overall policy performance, but may not capture the quality of each action. For example, useful early actions receive no direct imitation supervision if the episode fails.Table 4 shows that broad labels already work well. Without rollout action supervision, they reach 92.8%, versus 92.7% for numerical scores. With this supervision, broad labels reach 93.6% and numerical scores reach 94.5%.It remains unclear what numerical scores capture beyond these categories. The effect of label errors on conditioning and action weighting is also unclear.

Questions for authors
  1. What learning advantage or new capability does this combination provide beyond existing failure-aware WAM training and quality-conditioned policies? Which design choice is central to that advantage?

  2. What evidence shows that failed trajectories add useful learning signals beyond successful rollouts and quality conditioning?

  3. The paper highlights bidirectional interaction as a way for learned consequences to inform action generation. What evidence supports the added benefit of this design when learning from deployment rollouts?

  4. Appendix A.1 describes completion-based episode ratings and source-policy scores based on success and inspected motion quality. Could you clarify how these criteria translate into numerical scores and illustrate this with a few existing trajectories and short rationales? This would help explain what numerical scores capture beyond the low/medium/high labels in Table 4.

3 comments

Thank you for your careful reading and constructive feedback. We have revised the manuscript PDF by appending Appendix F: Rebuttal Material after the original supplementary material, adding controlled experiments and annotation details. Trajectory visualizations and scoring examples appear in Appendix G.

Weakness 1: Methodological contribution and evidence for the training design

We appreciate this concern. Building on prior WAM and quality-conditioning research, LoopWAM targets closed-loop self-improvement in the physical world from mixed-quality deployment experience. Source-dependent noise sampling keeps recorded rollout actions relatively clean for action-conditioned world modeling; bidirectional video–action attention connects consequence prediction with action generation; and quality steering and quality-aware supervision guide behavior toward the expert reference while retaining predictive learning across qualities.

We test the noise and attention designs on RoboTwin 2.0.

ConfigurationAverage success (%)
Full LoopWAM94.5
Apply the expert noise scheme to all data sources88.9
One-way video-to-action attention89.5

In the first two rows, video and action noise remain independently sampled; adapting their distributions to the data source improves success by 5.6 points. Expert demonstrations support high-quality generation, while relatively clean rollout actions support prediction of their observed consequences. Bidirectional attention lets these action-dependent video representations participate in action denoising, with a 5.0-point benefit over one-way attention (Appendix F.2).

Weakness 2: The distinct contribution of failed trajectories

We agree that adding mixed outcomes alone does not isolate the contribution of failures.

ConfigurationAdditional rollout dataAverage success (%)
Expert demonstrations onlyNone91.4
Expert demonstrations + failed rolloutsFailed92.7
Expert demonstrations + successful rolloutsSuccessful93.3
Expert demonstrations + all rollouts (w/o WM on failed rollouts)Full mixed pool93.1
Expert demonstrations + all rollouts (full LoopWAM)Full mixed pool94.5

The last two rows hold all trajectories, sampling, and optimization updates fixed, changing only failed-rollout WM supervision. Failed-rollout action loss remains zero; expert and successful-rollout supervision and quality conditioning are unchanged. The 1.4-point gain therefore isolates a benefit from learning failed-action consequences beyond successful-rollout supervision, without imitating failed actions.

Adding only failed rollouts also improves over expert-only training (92.7% vs. 91.4%), and the full mixture outperforms successful-only training. The fixed-data WM comparison isolates the contribution of failed-rollout supervision while holding the number of optimization updates and the training-data sampling scheme constant (Appendix F.3).

Weakness 3: Meaning of numerical quality scores and sensitivity to label errors

We agree that a score shared by all segments of a trajectory cannot capture each action's quality. Useful early actions in a failed episode receive no direct imitation supervision, although their transitions remain available for WM learning.

On RoboTwin, low/medium/high labels separate failures, successes, and experts; numerical scores further distinguish source-policy quality within successful rollouts. In Table 4 of the original manuscript, categorical and numerical conditioning achieve 92.8% and 92.7% without rollout action supervision, and 93.6% and 94.5% with quality-weighted supervision. These results show that finer-grained quality annotations can yield additional gains when combined with action weighting. We therefore hypothesize that reliable action-level quality annotations could further improve learning, but their higher annotation cost would require a trade-off between performance and labeling effort.

We randomly corrupt 10% or 30% of rollout labels, uniformly replacing each selected score with another allowed value, fixed before training; expert labels remain unchanged. Modified scores control both conditioning and action weights, without a separate mask based on the original success/failure outcomes. A mislabeled failure can therefore receive nonzero imitation weight.

Quality-label corruptionAverage success (%)
0% (full LoopWAM)94.5
10%92.6
30%85.5

LoopWAM shows some robustness at 10% corruption, but degrades substantially at 30%. Errors disrupt both conditioning and imitation weights, highlighting the importance of quality steering and quality-aware supervision for mixed-quality learning (Appendix F.4).

To be continued in next reply.

Question 1: Learning advantage over existing approaches and the central design choice

The central design connects action-conditioned consequence learning with expert-directed action generation, as tested above. We additionally compare with FACT and RECAP-style conditioning, using the same expert demonstrations as FACT and matching its rollout count (1.3K trajectories) and policy optimization budget:

Method / variantAdditional rollout dataAverage success (%)
LoopWAM (FACT-matched count)1.3K failed92.3
FACT (reported)1.3K failed87.5
LoopWAM (quality steering)1.3K mixed93.8
LoopWAM + RECAP-style conditioningSame 1.3K mixed93.1

FACT: Its result is taken from the paper. We sample failures from our corpus because we could not locate a public release of FACT's specific failure-rollout set. The 4.8-point improvement, at matched failure-data quantity and policy optimization budget, supports the complete LoopWAM training design at this data scale.

RECAP-style conditioning: The mixed rollout samples and all other LoopWAM components are identical. A Gemma-based VLM value function trained on those rollouts supplies binary advantage indicators in place of quality text; inference uses the positive indicator. Quality steering performs 0.7 points better without additional value-model training. That extra training is excluded from the matched policy optimization budget.

Together, these comparisons support the complete design for learning from deployment experience and quality steering within that design. Alongside the component ablations, they support connecting consequence learning with action generation guided toward the expert reference (Appendix F.5).

Question 2: Evidence beyond successful rollouts and quality conditioning

The controlled experiment in Weakness 2 changes only failed-rollout WM loss, retaining successful-rollout supervision and quality conditioning. Its 1.4-point gain directly supports an additional learning signal from failures (Appendix F.3).

Question 3: Evidence for bidirectional video–action attention

The one-way variant lets action tokens attend to video tokens but prevents video tokens from attending to action tokens. With data and optimization unchanged, it achieves 89.5% versus 94.5% for full LoopWAM. This removes the video branch's direct access to action tokens for consequence prediction. Bidirectional attention restores this connection and lets action-dependent video representations participate in action denoising (Appendix F.2).

Question 4: How scores are assigned, with trajectory examples

Thank you for suggesting that we include concrete examples. We have included visualized rollout examples from both Table30-V2 and RoboTwin 2.0 in Appendix G: Qualitative Examples of Trajectory Quality, covering episode-level and coarse-level annotation. The visualizations and accompanying explanations present actual executions, their assigned scores, and the criteria behind those scores.

  • Table30-V2 (episode-level annotation): Trained platform evaluators score task progress and execution quality out of 10. Unfinished steps reduce the score, and each retry incurs a 0.5-point deduction. We retain the platform scores for both successful and failed rollouts; the examples illustrate how completed steps and retries affect the ratings.
  • RoboTwin 2.0 (coarse-level annotation): Expert demonstrations receive q = 5, failures q = 1, and successful rollouts inherit fixed source-policy grades (q = 3 or 4). These grades reflect human judgments of overall policy reliability and inspected motion quality. The examples include trajectory visualizations and action traces illustrating this assessment.

Further discussion of the scoring schemes and their use in training is provided in Appendix F.4.

If you have any further questions, please let us know. Thank you again for your careful review and helpful suggestions!

Dear Reviewer,

Thank you for your detailed and thoughtful review. We have now posted our responses and revised the manuscript with additional experiments and clarifications addressing your concerns. Your feedback has been valuable in strengthening our work, and we would be grateful if you could reconsider your evaluation in light of the new evidence and explanations. We hope these additions have addressed the issues you raised. If any questions or concerns remain, we would welcome the opportunity to discuss them and provide further clarification. Thank you again for the time and care you have devoted to reviewing our manuscript.

Best regards,
The Authors

Reviewer_SYr8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LoopWAM. In a nutshell, the proposed method starts off from a base world-action model (WAM), collects trajectories from roll out, and train the policy objective on good/successful ones (think RFT/DAgger), and the world model prediction objective on all trajectories, including bad ones. Various experiments in real and sim show that this achieves good performance across benchmarks. There are some ablations showing the impact of action and quality label supervision.

Critical strengths
  • The proposed method is a coherent approach to learning from mixed-quality rollouts. While the individual ideas of iterative improvement and WAMs are established, this paper proposes a coherent and valid combination of these ideas.
  • The paper is supported by a decent amount of experimental evidence in real and sim. The proposed method improves across multiple tasks and embodiments. The simulation results provide additional evidence that LoopWAM is useful across various settings.
Critical weaknesses
  • Missing ablations on the benefit of suboptimal/unsuccessful trajectories. While the idea is intuitively sound, it's possible that what's working is simply training on good/successful trajectories, instead of the suboptimal world model rollout. It would be much more convincing if you had an ablation of (a) only training on good/successful trajectories, without video/WM supervision; (b) only training on good/successful trajectories, with video/WM supervision. If this works / doesn't work better than the full LoopWAM, either way it will tell us much more about what exactly works in this setting. Full vs (b) tells us whether data filtering is the key; (b) vs (a) tells us whether world modeling is required.
Questions for authors
  • Ablations: do you have a controlled comparison against using only expert demonstrations + iterative successful/good rollouts? Basically turning the WM objective off/on?
  • Experiment details: how many episodes do you roll out for the real-world success rates? How are the initial conditions controlled across iterations?
3 comments

Thank you for your thoughtful review and for recognizing the coherence of our approach and the evidence across tasks and embodiments. We have revised the manuscript PDF by appending Appendix F: Rebuttal Material after the original supplementary material. Below, we present the requested ablations and clarify the real-robot evaluation protocol.

Weakness 1: Benefits beyond training on successful rollouts

We agree that the proposed comparisons are important for understanding whether the gains come from successful-rollout retraining, world-model supervision, or learning from the broader rollout mixture. We have added the requested ablations on RoboTwin 2.0. All variants retain the same expert demonstrations, and the two successful-only variants use the exact same successful rollout subset. Results are success rates (%); Average combines clean and domain-randomized scenes.

Training settingCleanRand.Average
(a) Successful rollouts, without WM supervision85.683.284.4
(b) Successful rollouts, with WM supervision93.593.193.3
(c) Failed rollouts only92.892.792.7
(d) Expert demonstrations only91.791.291.4
(e) All rollouts, without WM supervision on failed rollouts93.193.293.1
Full LoopWAM, with successful and failed rollouts94.694.494.5

The comparison between (a) and (b) shows that enabling WM supervision improves average success by 8.9 percentage points, indicating that predictive supervision contributes substantially even after successful trajectories have been selected. Full LoopWAM further improves over (b) by 1.2 points, highlighting its ability to learn from trajectories across quality levels: even failed rollouts provide useful supervision for predicting action consequences. Comparing (c) with (d), adding only failed rollouts to the same expert demonstrations raises average success from 91.4% to 92.7%. Since failed-rollout actions receive zero action imitation weight, this result supports learning from their consequences through WM supervision, beyond retraining on successful behavior.

Because filtering changes both dataset size and the number of updates under the five-epoch schedule, we additionally isolate failed-rollout supervision in (e) under unchanged data and optimization budgets, as described in Question 1 below.

Our motivation is to improve policies from mixed-quality deployment experience, thereby enabling closed-loop self-improvement in the physical world. Source-dependent noise sampling and bidirectional video–action attention support learning action consequences, while quality steering and quality-aware supervision regulate action imitation and guide deployment toward expert behavior. This design allows deployment experience to contribute beyond its successful actions. The ablations are detailed in Appendix F.2 and Appendix F.3.

Question 1: Controlled comparison using expert demonstrations and successful rollouts, with WM supervision off/on

Following your suggestion, we have added this controlled comparison. The comparison between (a) and (b) above holds the expert demonstrations, successful rollout samples, action-supervision objective, optimizer settings, and five-epoch training schedule fixed. The only change is whether video/WM supervision is enabled on both expert demonstrations and successful rollouts. Visual observations remain available to the policy in both cases. Thus, the 8.9-point gain measures the contribution of predictive supervision on this fixed dataset.

To determine what failed experience adds, variant (e) retains the full training corpus, sampling scheme, and number of optimization updates, disabling only WM loss on failed rollouts. Average success decreases from 94.5% to 93.1%. Failed trajectories remain in the training samples, and their direct action-loss weight remains zero in both configurations. The 1.4-point improvement from restoring their WM supervision therefore directly supports learning from failed-action consequences beyond successful-rollout supervision and quality conditioning (Appendix F.3).

We further compare 1.3K successful rollouts with 1.3K mixed successful and failed rollouts, sampled from the same corpus, while retaining the same expert demonstrations, LoopWAM training design, and policy optimization budget. Mixed-rollout training achieves 93.8%, versus 93.2% for successful-only retraining. Together with the fixed-data WM control, this result supports learning from a broader range of deployment outcomes and trajectory qualities, beyond increasing the number of training rollouts (Appendix F.5).

To be continued in next reply.

Question 2: Real-world evaluation episodes and control of initial conditions

Thank you for pointing out the need to clarify these details. The two real-robot studies use the following protocols:

Table30-V2. Every checkpoint, including expert-trained LoopWAM and all three updated policies, is evaluated through the official Table30-V2 channel using 10 episodes per task, or 300 episodes across the 30 tasks. These evaluation episodes are separate from the training rollouts collected during each improvement round. The official protocol varies objects and initial placements while applying the same initialization and evaluation procedures to every checkpoint. Trained platform evaluators use unchanged success and scoring criteria across rounds.

The prompting, progress-monitoring assistance, and execution protocol are also fixed across these checkpoints. Four tasks use predefined subtask lists, with Qwen2.5-VL-7B checking completion every five action chunks and advancing to the next instruction upon detecting completion; the other 26 tasks use their original task instructions. No human intervenes in task execution during collection or evaluation. Human operators prepare and reset scenes and score completed rollouts. Thus, the improvements are evaluated under consistent deployment assistance and initialization procedures. Details and switching examples are provided in Appendix F.6.1 and Appendix F.6.2.

Shirt folding. The initial and intermediate success rates, 13.20% and 32.54%, are computed over all deployment rollouts collected by the corresponding checkpoint, with approximately 200 trajectories per checkpoint. After the final update, data collection stops, and a separate evaluation yields five successes in nine trials (55.56%). Before every collection or evaluation episode, a tester scrambles the garment and drops it onto the table to create a challenging initial configuration. This preparation procedure and the predefined success criteria remain unchanged across rounds. The first two rates therefore summarize collection rollouts, whereas the final rate comes from a separate nine-trial evaluation. We now explicitly distinguish these counts and protocols in Appendix F.6.

If you have any further questions, please let us know. Thank you again for your careful review and helpful suggestions!

Dear Reviewer,

Thank you for your detailed and thoughtful review. We have now posted our responses and revised the manuscript with additional experiments and clarifications addressing your concerns. Your feedback has been valuable in strengthening our work, and we would be grateful if you could reconsider your evaluation in light of the new evidence and explanations. We hope these additions have addressed the issues you raised. If any questions or concerns remain, we would welcome the opportunity to discuss them and provide further clarification. Thank you again for the time and care you have devoted to reviewing our manuscript.

Best regards,
The Authors

Reviewer_h9USOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LoopWAM improves world-action policies using deployment rollouts. It combines three components: quality conditioning, source-dependent noise sampling, and quality-weighted action supervision. Together, these let mixed-quality trajectories supervise future prediction while controlling which actions the policy imitates. The paper evaluates repeated real-robot improvement and two simulation benchmarks.

Critical strengths

Learning from failed deployment experience is an important direction. Real-robot success rises from 23.67% to 43.33% across three rounds, which is a substantial gain. The evaluation covers multiple tasks and embodiments, so it provides useful evidence beyond a single-task demonstration.

Critical weaknesses
  1. The experiments do not isolate the main technical contribution. Table 4 varies quality steering and rollout action supervision but keeps failed-rollout video supervision and the source-dependent noise design fixed. As a result, the additional benefit of these two components remains unclear. Comparisons with FACT-style failure-aware training and RECAP-style advantage conditioning, using the same rollout data and training budget, would help establish the additional benefit of the proposed joint training.

  2. The real-robot deployment protocol and its costs are incompletely specified. It is unclear whether the policy receives only the original task instruction throughout execution or also uses subtask prompts or external assistance, and whether these conditions remain fixed across improvement rounds and baselines. The loop also uses approximately 9,000 real rollouts and human quality scores. Without a complete deployment protocol and matched interaction and training budgets, the attribution and efficiency of the improvement remain unclear.


[1] FACT: Failure-Aware Causal Training for World-Action Models
[2] π*0.6: a VLA That Learns From Experience

Questions for authors
  1. At the same rollout and optimization budgets, how much improvement remains over success-filtered or quality-conditioned retraining? What changes when failed-rollout video supervision and the source-dependent noise schedule are removed separately?

  2. Does the policy receive only the original task instruction throughout each rollout? Does deployment use external models for task decomposition or progress monitoring? If so, how frequently are they called, how is instruction switching determined, and is the same procedure used across checkpoints and baselines? Please include any auxiliary model calls and human effort in the cost accounting.

6 comments

Thank you for your detailed assessment and helpful suggestions, particularly on clarifying our technical contribution and strengthening the experimental comparisons. We have revised the manuscript PDF by appending Appendix F: Rebuttal Material after the original supplementary material. Below, we provide the key results and explanations for each concern, with references to Appendix F for additional details and illustrations.

Weakness 1: Isolating the technical contribution and comparison with alternative training schemes

We appreciate this suggestion. Our motivation is to improve policies from mixed-quality deployment experience, thereby enabling closed-loop self-improvement in the physical world. LoopWAM combines source-dependent noise sampling and bidirectional video–action attention to connect consequence learning with action generation, and quality steering and quality-aware supervision to regulate action imitation and guide deployment toward expert behavior.

We agree that the proposed comparisons with FACT and RECAP are particularly valuable for assessing the benefits of our design. We have therefore added RoboTwin 2.0 comparisons with FACT and RECAP-style conditioning, together with a successful-only retraining baseline. All LoopWAM variants retain the same expert demonstrations and use 1.3K rollouts with matched policy optimization budgets. FACT is compared at its reported failure-rollout count; the two mixed-rollout variants use identical samples. The table reports average success rates across clean and domain-randomized settings.

Method / variantAdditional rollout dataAvg. success (%)
LoopWAM (FACT-matched count)1.3K failed92.3
FACT (reported)1.3K failed87.5
LoopWAM (successful-only)1.3K successful93.2
LoopWAM (quality steering)1.3K mixed93.8
LoopWAM + RECAP-style conditioningSame 1.3K mixed93.1

For FACT, we sampled 1.3K failures from our corpus because we could not locate a public release of its specific failure-rollout set. LoopWAM achieves 4.8 percentage points higher success than FACT’s reported result under matched expert data, failure-rollout counts, and policy optimization budgets.

The conditioning comparison uses identical rollout samples, architecture, and policy optimization budgets. Following RECAP, we train a Gemma-based VLM value function on the same rollouts and convert estimated advantages into binary improvement indicators. These replace quality text, with the positive indicator used at inference; all other LoopWAM components remain unchanged. Quality steering achieves 0.7 percentage points higher success. The additional value-model training is excluded from the matched policy budget. Implementation details are provided in Appendix F.5.

Weakness 2: Deployment protocol, attribution, and costs

We agree that the original manuscript did not explain the deployment protocol and costs clearly enough. The expert-trained LoopWAM baseline and all updated checkpoints use the same prompting, auxiliary assistance, and execution protocol (Question 2 below). Each receives 10 official Table30-V2 evaluation episodes per task, or 300 in total, with consistent initialization and scoring procedures. The gains are therefore measured under unchanged deployment assistance.

Each rollout requires approximately 2.5 minutes of execution plus one minute of reset and quality rating. For 9,000 training rollouts, this amounts to approximately 375 cumulative robot-execution hours and 150 reset-and-rating hours, excluding evaluation and training.

For the ARX5 policy on Table30-V2, training takes approximately 30 hours for the expert-trained baseline and 42, 51, and 59 hours for Round 1, Round 2, and Round 3, respectively. Each run uses five epochs on 96 Alibaba PPU-810E accelerators. These are individual training durations for ARX5, providing a concrete example of the resources required for real-robot policy updates. Further details are provided in Appendix F.6.

To be continued in next reply.

Question 1: Gains over alternative training schemes and the separate effects of failed-rollout supervision and noise sampling

To address successful-only retraining, we randomly sampled 1.3K successful rollouts from the same existing corpus as the mixed subset. With the same expert data, full LoopWAM training design, and policy optimization budget, mixed-rollout training achieves 93.8%, compared with 93.2% for successful-only retraining (+0.6 percentage points).

By including both successes and failures, the mixed subset exposes LoopWAM to a broader range of deployment outcomes and trajectory qualities. Together with the controlled ablations below, this result supports our design: source-dependent noise sampling and bidirectional video–action attention help the model learn action consequences across this mixture, while quality steering and quality-aware supervision guide action learning toward higher-quality behavior.

To separately isolate failed-rollout supervision and the denoising design, we also retain the full training data, data-sampling scheme, and number of optimization updates, changing only the stated component:

ConfigurationAvg. success (%)
Full LoopWAM94.5
Disable WM supervision only on failed rollouts93.1
Apply the expert noise scheme to all data sources88.9
One-way video-to-action attention89.5

The world-model (WM) ablation disables only failed-rollout WM loss; failed trajectories remain in sampling with zero action-loss weight. Restoring this supervision improves success by 1.4 points, directly supporting its contribution under a fixed budget. Applying the expert noise distributions to all sources reduces success by 5.6 points. Allowing action tokens to attend to video tokens, while removing attention in the reverse direction, reduces success by 5.0 points. This removes the video branch's direct access to action tokens for consequence prediction; the result supports restoring that connection for policy performance.

These comparisons provide complementary evidence: the 1.3K experiment tests rollout selection at matched training-data quantity and policy budget, while the WM control isolates failed-rollout supervision on unchanged data. Further details are provided in Appendix F.2, Appendix F.3, and Appendix F.5.

Question 2: Instructions, progress monitoring, and human involvement

Four Table30-V2 tasks—button pressing, plant watering, paper shredding, and stamp positioning—use predefined, ordered subtask instructions. Qwen2.5-VL-7B checks current-subtask completion every five action chunks and advances to the next instruction upon detecting completion. It does not generate or revise the subtask list online. The other 26 tasks, both simulation benchmarks, and shirt folding use their original full-task instructions. This procedure is identical for our expert-trained baseline and all updated checkpoints.

Each VLM call takes approximately one second, runs asynchronously, and shares the policy’s PPU. During collection and evaluation, no human intervenes in execution or supplies teleoperated corrections. Human operators prepare and reset scenes and score completed rollouts; reset and rating are included in the costs above. The exact subtask instructions and illustrated switching sequences are provided in Appendix F.6.1 and Appendix F.6.2, respectively.

If you have any further questions, please let us know. Thank you again for your careful review and helpful suggestions!

Thank you for the rebuttal. The deployment protocol and cost accounting are now clear.

My main concern remains how much failed rollouts add beyond simpler alternatives. On RoboTwin, RECAP-style conditioning (93.1) and success-only training (93.2) come within 0.7 points of LoopWAM (93.8). On the real robot, reset and quality rating take about 150 human hours for 9,000 rollouts, and no comparison at matched human effort is provided, such as collecting more expert demonstrations. I suggest discussing this as a limitation. Also, the abstract’s shirt-folding numbers (13.20% → 55.56%) compare collection rollouts with a separate 9-trial evaluation, and this should be clarified.

I maintain my score of 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank You to All Reviewers

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair for overseeing the review process and considering our manuscript. We also thank reviewers h9US, SYr8, and BUm7 for their careful reviews and constructive suggestions, which have helped us strengthen the experimental validation and improve the presentation of our work.

We have appended Appendix F: Rebuttal Material to the revised manuscript PDF, following the original supplementary material, with controlled experiments and deployment details. Appendix G provides visualized trajectories and scoring rationales. Our project page provides real-robot demonstrations across tasks and embodiments, showcasing LoopWAM’s ability to fold a crumpled shirt casually tossed onto a table. We summarize the main evidence and clarifications below.

  1. Motivation and technical contribution. LoopWAM learns from mixed-quality deployment experience to enable closed-loop self-improvement in the physical world. Source-dependent noise sampling supports action-conditioned consequence prediction, and bidirectional video–action attention connects this learning with action generation. Quality steering and quality-aware supervision guide behavior toward the expert reference while preserving predictive learning from successful and failed rollouts.

  2. Controlled evidence for the training design. Holding the full training data, sampling, and update count fixed, disabling only failed-rollout world-model (WM) supervision reduces success from 94.5% to 93.1%. Failed actions receive zero imitation weight in both configurations, isolating the benefit of learning their consequences. Under the same controls, shared expert noise sampling yields 88.9%, and one-way video-to-action attention yields 89.5%. On the identical successful-rollout subset, enabling WM supervision on expert and rollout data raises success from 84.4% to 93.3%. These results support the proposed training choices beyond simply selecting successful behavior (Appendix F.2 and Appendix F.3).

  3. Comparisons with alternative training schemes. With the same expert demonstrations, 1.3K training rollouts, and matched policy optimization budgets, LoopWAM achieves 93.8% with mixed outcomes versus 93.2% with successful rollouts. Replacing only quality steering with RECAP-style conditioning on the identical mixed subset yields 93.1%; its additional value-model training is excluded from the matched policy budget. Matching FACT's failure-rollout count and policy optimization budget, LoopWAM achieves 92.3%, versus FACT's reported 87.5%. We sample failures from our corpus because we could not locate a public release of FACT's specific failure-rollout set (Appendix F.5).

  4. Quality annotations and reliability. Scores describe episode-level or source-policy quality, without action-level credit assignment. With quality-weighted action supervision, numerical labels achieve 94.5%, versus 93.6% for categorical labels. Corrupting 10% or 30% of rollout labels yields 92.6% and 85.5%, respectively, indicating tolerance to limited noise but substantial degradation at higher corruption. Changed scores affect both conditioning and action weights (Appendix F.4). Appendix G illustrates scoring criteria using Table30-V2 and RoboTwin trajectories.

  5. Real-robot improvement, protocol, and costs. On Table30-V2, adding approximately 300 deployment rollouts per task to the same approximately 1,000 expert demonstrations per task raises success from 23.67% to 43.33% across three updates. Each checkpoint receives 10 official evaluation episodes per task under consistent protocols and auxiliary assistance, without human execution-time corrections. The 9,000 training rollouts require approximately 375 robot-execution hours and 150 reset-and-rating hours, with most human effort spent on resets; policy-training resources are also reported (Appendix F.6). We use the fixed official expert dataset and acknowledge the absence of a matched-human-effort comparison with additional expert demonstrations as a limitation to discuss explicitly.

  6. Shirt-folding evaluation. Initial and intermediate success rates use approximately 200 deployment-collection rollouts per checkpoint. Collection stops after two updates; the final model achieves 55.56% in nine separate evaluation trials. We have clarified this distinction in the manuscript.

We sincerely thank the Area Chair for coordinating the review process and the reviewers for their careful evaluations and constructive feedback. We hope the additional experiments and explanations have addressed the concerns raised, and we would be glad to clarify any remaining questions. Thank you again for your time and consideration!

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.