acceptodds
Under review as a conference paper at ICLR 2027

Asynchronous Sim-Real Co-training for On-Policy Robot Learning with Limited Real Hardware

Abstract

Improving an already capable robot policy through online interaction is challenging when physical hardware and interaction opportunities are limited. Simulation can supply additional experience, but differences in collection speed and availability across domains complicate batch composition and data freshness during joint online learning. We present Asynchronous Sim-Real Co-training (ASRC), a framework that uses continually collected simulated and real-world experience for PPO-style policy improvement. ASRC decouples collection from policy optimization, allowing independent interaction in each domain while learning proceeds. To coordinate their heterogeneous experience, an availability-aware scheduler adjusts the real-data fraction within a prescribed range, while latest-first selection and bounded-buffer eviction prioritize recent samples. The framework supports both CNN policy optimization and residual adaptation of . Across three manipulation tasks on a single physical robot, both policy families achieve final real-world success rates above 90% on every task, outperforming the evaluated baselines. System experiments show up to higher update throughput than a synchronous counterpart, while ablations support the importance of balancing domain composition and prioritizing fresh experience.

Primary area:
reinforcement learning
Keywords:
Sim-Real Co-Training, Reinforcement Learning, Robotics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6VGkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an online PPO improvement of an already-capable robot policy when real hardware is scarce. The motivation behind is that sim and real produce experience at very different, time-varying rates — the real branch stalls on resets and operator readiness — so a synchronous collect-then-update loop leaves one branch idle. ASRC decouples them: each domain runs its own env loop and rollout copy of the shared policy into its own buffer, and a single learner pulls from both while collection continues.

It concludes three mechanisms. Availability-aware mixture scheduling (target real fraction α* with an allowed band; each update takes the feasible real count closest to α*B). Latest-first selection with bounded-buffer eviction instead of FIFO. Decoupled PPO to handle batches spanning several behavior-policy versions. Same pipeline for CNN policies and residual adaptation.

Critical strengths

The CNN Peg Insertion result is large, clean, and has no human in the loop. 0% → ~100% real success while real-only PPO, RLPD and every fixed reference stay near zero on the same budget. It's also the one task with fully automated success detection (TCP pose tolerance, A.6), so it's immune to the labeling problem below, and the gap is far too big to be a trial-count artifact. This carries most of my willingness to see the paper developed.

The sim-to-real setup is deliberately crude and they say so. A.2: no system ID, default Panda stiffness/damping, manual camera alignment, composited photo backgrounds. Showing co-training still helps under an uncalibrated simulator is more transferable than a result that needs a digital twin.

Critical weaknesses

The headline throughput number is the best of twelve cells, and their own ablation undercuts it. 2.96× is the α*=0.05, 32-env cell. §5.4 says α*=0.05 "leads to less stable performance"; α*=0.1 learns best. And the speedup decays as the real fraction rises. Looking only at cells where Sync and Async share an identical realized real ratio:

  • α*=0.2, all rows at 16.67%: 1.18× / 1.21× / 1.16×. Saturates by 16 envs.
  • α*=0.1, Async-8 at 11.11% vs Sync-32 at 8.73% — more real data — gives 1.02×.

The 2.96× cell isn't ratio-matched either: 4.76% vs Sync's 5.56%. The table reports this; the text never adjusts for it. Fair read: ~1.2–1.9× at matched real fraction, with the apparent 3× partly from running a leaner mixture.

The experiment that would justify the framework isn't there, and the headline success claim has no table. The motivation is that real hardware is scarce, so wall-clock binds. But Figure 4 plots success against real-world transitions — the one axis where asynchrony can't help by construction. Table 2 reports throughput with no learning outcome attached. Sync never appears in Figure 4; ASRC also never appears on a wall-clock axis. So the system and policy contributions are not connected.

And ">90% on every task" appears in no table. The only checkable numbers are execution times on hand-picked successful trials. Final success has to be read off six small line plots — no trial counts, no CIs, no seeds. At n = 36 success is quantized to ~2.8 pp, and §5.4's α*=0.5 "gain of 2.8 percentage points over the initial 75%" is 27/36 → 28/36. One trial, reported to one decimal place.

Questions for authors
  1. Plot real success against wall-clock, ASRC vs Sync, on at least one task. Does the throughput advantage become faster improvement per hour of robot time?
  2. Add a table of final success rates per method / task / policy family, with trial counts and binomial CIs. How many seeds underlie each Figure 4 curve — is it one?
  3. Add Sync rows at 8 and 16 envs, and give Async-vs-Sync at exactly matched realized real ratio for every α*. What's the speedup attributable to asynchrony alone? And why quote the α*=0.05 cell when §5.4 prefers α*=0.1?
  4. What were α*, α_min, α_max, C_d, B, and segment length for the main results? Table 5 lists learning rates and env counts but not one hyperparameter of either mechanism you claim as a contribution.
  5. On Pick and Place and Pour Marbles success is labeled by an operator at a keyboard, who can also end an episode as a safety failure (A.6). Was the operator blind to the method? If not, the headline metric on two of three tasks comes from a non-blinded human who is also in the training loop.
5 comments

We thank the reviewer for the close examination of the throughput table and evaluation protocol. We agree that the original presentation needed to connect system throughput to learning outcomes, separate composition effects, and make the final success statistics directly inspectable.

W1. The headline throughput number is the best of twelve cells, and their own ablation undercuts it.

We would like to clarify that the system-efficiency analysis and the mixture-ratio ablation address different questions.

The ablation in Section 5.4 examines policy learning on Pick and Place. Its purpose is to show that either too little or too much real-world data can impair learning, rather than to establish 0.1 as a universally optimal target ratio. The appropriate mixture can depend on the task and available data, consistent with prior co-training studies [1–3]. In particular, the best tested ratio on Pick and Place does not establish the optimal ratio on Peg Insertion, the task used for the throughput experiments.

Conversely, Table 2 in Section 5.3 studies system throughput, rather than selecting the mixture that maximizes task success. Its main observations are:

  1. Increasing simulation parallelism can improve throughput, with diminishing benefits as the real-data fraction increases and physical collection becomes the bottleneck.

  2. Asynchronous execution can achieve higher throughput than Sync under the evaluated configurations.

The reported “up to 2.96×” describes the largest observed system speedup, not a recommendation to use α*=0.05 for policy learning. We acknowledge that this cell alone does not isolate asynchrony from differences in realized composition. We therefore add exactly matched-ratio comparisons in Appendix C.2, as detailed in our response to Q3 below.

We have also added success-rate curves against elapsed training time in Section 5.3 and Appendix C.1 to connect these system measurements to policy improvement.

W2, part 1. The experiment that would justify the framework isn't there. The motivation is that real hardware is scarce, so wall-clock binds. But Figure 4 plots success against real-world transitions — the one axis where asynchrony can't help by construction. Table 2 reports throughput with no learning outcome attached. Sync never appears in Figure 4. ASRC also never appears on a wall-clock axis. So the system and policy contributions are not connected.

Q1. Plot real success against wall-clock, ASRC vs Sync, on at least one task. Does the throughput advantage become faster improvement per hour of robot time?

Thank you for identifying this missing comparison. Figure 4 uses real-world transitions to compare policy improvement under a common physical-interaction budget. With the robot operating at a nominal control frequency of 10 Hz, transition counts directly reflect active robot interaction time. They also avoid confounding algorithm comparisons with differences in how quickly an operator performs resets or provides feedback across sessions.

To compare ASRC and Sync in elapsed training time, we have added real-world success versus wall-clock time on Peg Insertion in Section 5.3, Figure 6, using three seeds. Waiting time is included. We linearly interpolate evaluations at their actual checkpoint times to report the mean and standard deviation across runs. ASRC reaches 90% mean success in approximately 55 minutes, compared with 79 minutes for Sync. At approximately 67 minutes, ASRC reaches 97.2%, compared with 82.5% for Sync.

Appendix C.1, Figure 11 additionally includes fully synchronous execution that alternates between collection and optimization. This baseline improves somewhat faster than zero-staleness Sync, but remains slower than ASRC on average in elapsed training time. These comparisons directly connect the execution design to learning outcomes.

W2, part 2. The headline success claim has no table. And “>90% on every task” appears in no table. The only checkable numbers are execution times on hand-picked successful trials. Final success has to be read off six small line plots — no trial counts, no CIs, no seeds. At n=36 success is quantized to approximately 2.8 percentage points, and the reported gain from the initial 75% is 27/36 to 28/36.

Q2. Add a table of final success rates per method / task / policy family, with trial counts and binomial CIs. How many seeds underlie each Figure 4 curve — is it one?

We have expanded the statistical reporting and revised the wording concerning the one-trial difference.

  1. Final success rates and evaluation uncertainty. Each original Figure 4 curve represents one training run, using a common training seed across the experiments. Appendix D.1, Tables 7 and 8 now report final success rates, successful/total trial counts, and 95% Wilson confidence intervals for every method, task, and policy family. Evaluations use 36 trials for Pick and Place and Pour Marbles, and 40 for Peg Insertion. ASRC’s final results are 34/36, 40/40, and 36/36 for CNN policies, and 34/36, 39/40, and 33/36 for π0.5 residual policies, respectively. These tables make the above-90% observed success rates and their uncertainty directly inspectable.

  2. Repeated training runs. Appendix D.2, Figure 12 extends the Peg Insertion experiments to three training seeds for both online methods and static baselines. ASRC achieves final mean success rates of 98.3% for CNN policies and 99.2% for π0.5 residual policies, with sample standard deviations of 2.9 and 1.4 percentage points, respectively. These results show consistently high final performance across the three evaluated seeds.

  3. Interpretation of the 2.8-percentage-point difference. The intended observation was that the high-real-fraction setting provides little improvement over initialization. We have removed the emphasis on this one-trial increase and replaced it with “remaining close to the initial success rate” in Section 5.4.

Q3. Add Sync rows at 8 and 16 envs, and give Async-vs-Sync at exactly matched realized real ratio for every α*. What's the speedup attributable to asynchrony alone? And why quote the α*=0.05 cell when Section 5.4 prefers α*=0.1?

We have added Appendix C.2, Table 6, comparing Async and Sync at exactly fixed real-data fractions of 2/36 and 4/36, with 8, 16, and 32 simulation environments. Every matched pair uses the same hardware resources, simulation parallelism, and domain composition per update. Speedups are computed relative to the corresponding Sync configuration.

Fixed real-data fraction8 simulation environments16 simulation environments32 simulation environments
2/36, approximately 5.56%1.38×2.05×3.04×
4/36, approximately 11.11%1.25×1.27×1.17×

Because each update consumes a fixed number of transitions from each domain, the speedups in real-world data throughput, simulation data throughput, and update throughput are identical within each matched pair. Async remains faster in all six comparisons after eliminating differences in domain composition, although the gains are smaller at the higher real-data fraction.

Regarding α*=0.05 versus α*=0.1, Section 5.3 does not infer that a configuration with higher throughput necessarily produces a better policy. It examines how simulation parallelism and execution mode affect throughput. Section 5.4 separately examines how the mixture affects learning on Pick and Place. Thus, the peak throughput result and the preferred learning configuration in that ablation answer different questions.

Q4. What were α*, α_min, α_max, C_d, B, and segment length for the main results? Table 5 lists learning rates and env counts but not one hyperparameter of either mechanism you claim as a contribution.

We have added these parameters to Table 5 for both policy families and all three tasks. The table now reports:

  • Target real-data fraction α=0.1* and bounds [0.05, 0.15].
  • B=36 segments per update, with 120 transitions per segment.
  • Optimizer batch size 240.
  • Real buffer capacity 37 segments.
  • Simulation buffer capacity 68 segments for CNN policies, and 52, 68, and 52 segments for π0.5 residual policies on Pick and Place, Peg Insertion, and Pour Marbles, respectively.

Q5. On Pick and Place and Pour Marbles success is labeled by an operator at a keyboard, who can also end an episode as a safety failure. Was the operator blind to the method? If not, the headline metric on two of three tasks comes from a non-blinded human who is also in the training loop.

Yes, the operator was blind to the algorithm identity during both training and evaluation. Algorithm development and robot operation were performed by different people. An algorithm developer launched the models, while a separate robot operator labeled outcomes and recorded trial results without knowing which algorithm the deployed model corresponded to.

We have added this clarification to Appendix A.6. The operator followed the same task-specific success criteria across methods.

[1] Maddukuri et al. Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation. 2025.

[2] Wei et al. Empirical Analysis of Sim-and-Real Cotraining of Diffusion Policies for Planar Pushing from Pixels. 2025.

[3] Shi et al. Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Models. 2026.

Reviewer_fRy8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Asynchronous Sim-Real Co-training (ASRC) for improving robot policies through continued interaction with both simulated and real-world environments. The framework combines independent data collection, availability-aware mixture scheduling, and latest-first sample selection with an existing decoupled PPO objective. It supports both CNN policies and residual adaptation of a frozen π₀.₅ model. Experiments on three real-world manipulation tasks report final success rates above 90% for both policy families, while separate system experiments show up to 2.96× higher update throughput than the zero-staleness Sync baseline.

Critical strengths

1.The paper addresses a practical difficulty in online robot learning: simulated and real-world experience arrive at different and variable rates. The framework allows collection and optimization to proceed independently, accommodating interruptions in physical interaction.

2.The evaluation includes three physical manipulation tasks and both CNN and VLA residual policies. The success-rate results and separate throughput measurements provide evidence that the complete framework is useful, rather than demonstrating feasibility only in simulation.

Critical weaknesses

1.The contribution beyond existing online sim-real co-training methods is not sufficiently established. The relevant comparison is with methods in which both simulated and real-world environments continually provide online interaction data for policy updates. Adding online real-world feedback distinguishes ASRC from methods using fixed real datasets, but does not establish its distinction from this closer group of methods. ASRC adopts an existing decoupled PPO objective, while its mixture scheduling and buffer-management rules are relatively straightforward. Its main contribution therefore appears to lie in their integration and empirical benefits. The paper needs to explain more precisely what this integration adds and which limitations of existing online co-training methods it resolves.

2.The experiments do not fully isolate the benefits of some central design choices. Figure 6 (left) varies the target mixture ratio while allowing deviations in every setting, so it does not establish the benefit of availability-aware scheduling over a fixed ratio. Table 2 demonstrates a throughput advantage from relaxing the zero-staleness constraint, but does not report the corresponding success rates. Figure 4 compares the complete framework with other learning methods, rather than with the Sync counterpart. These missing comparisons limit the evidence for the specific scheduling and asynchronous training choices, although the complete framework performs well against the evaluated baselines.

Questions for authors

1.Compared specifically with methods that continually collect online experience from both simulated and real-world environments, what does ASRC add? Which limitations of those methods are addressed by asynchronous collection, mixture scheduling, and freshness management, and which of these designs are new to this work?

2.With the same target real-data fraction, how does availability-aware scheduling compare with keeping the real-data fraction fixed in every training batch? Please report both task success rates and update throughput.

3.Does Sync in Table 2 alternate between collection and optimization, or does it retain concurrent execution while filtering out data from older policy versions? Under the same initialization and comparable real-world interaction budgets, how do its success rates compare with ASRC? Please also specify the mixture settings and actual real-data fractions for this comparison.

4.What GPU models and how many GPUs were used? How were computational resources allocated among simulation, policy inference, and model updates in the throughput experiments?

5.The opening paragraph moves from expert annotation costs to limited physical hardware and interaction opportunities. Could the authors distinguish these constraints more explicitly and clarify which limitation primarily motivates the proposed framework?

2 comments

We thank the reviewer for the detailed assessment and for identifying the need to separate the contribution of online co-training from the specific scheduling and execution choices in ASRC.

Q5. The opening paragraph moves from expert annotation costs to limited physical hardware and interaction opportunities. Could the authors distinguish these constraints more explicitly and clarify which limitation primarily motivates the proposed framework?

Thank you for this suggestion. We have revised the opening three paragraphs of the Introduction to distinguish these two constraints. Here we clarify our motivation and contribution.

Our starting motivation is to improve an existing policy using task-level success or failure feedback, without requiring expert takeovers or corrective action labels. Real-world RL methods such as SERL and HIL-SERL use off-policy learning to reuse experience and improve sample efficiency under limited physical interaction [1, 2]. Related online sim-real methods, including Sim and Real: Better Together and CSAR, also use off-policy learning with experience replay [3, 4]. However, when fine-tuning a pretrained policy, an insufficiently trained action-value function can misguide policy updates. We evaluate this alternative through RLPD in Figure 4 and the additional RLPD co-training baseline in Appendix D.3.

PPO offers a different approach by updating the policy using estimated advantages rather than directly maximizing a learned action-value function [5]. However, its restricted reuse of older experience creates a continuing demand for fresh interaction data. With few physical robots, supplying this data becomes a major practical bottleneck. Our primary contribution is an asynchronous sim-real co-training framework that makes PPO-style online policy improvement practical under limited real hardware.

W1. The contribution beyond existing online sim-real co-training methods is not sufficiently established.

Q1. Compared specifically with methods that continually collect online experience from both simulated and real-world environments, what does ASRC add? Which limitations of those methods are addressed by asynchronous collection, mixture scheduling, and freshness management, and which of these designs are new to this work?

Thank you for identifying this gap in the comparison. We agree that collecting online experience in both domains is not, by itself, sufficient to distinguish ASRC from prior online co-training methods.

The distinction is how ASRC organizes online sim-real interaction for PPO-style policy improvement under unequal and varying collection rates. The related methods discussed above use off-policy updates with experience replay. Our setting instead requires a continuing supply of sufficiently recent policy experience. Asynchronous collection allows simulation and the real robot to proceed independently. Availability-aware mixture scheduling constructs feasible batches near the desired domain composition without always waiting for one exact ratio. Latest-first selection and bounded-buffer eviction prevent updates from continually consuming an accumulated backlog. Our contribution is the joint design of availability-aware batch construction and freshness management within an asynchronous sim-real PPO framework, built on the existing decoupled PPO objective.

We have also added an RLPD-based online sim-real co-training baseline in Appendix D.3, with implementation details in Appendix B.4. Unlike the original RLPD baseline, this variant continually collects new experience in both domains. Figure 13 shows higher mean success at later evaluation points than the original RLPD and real-only PPO baselines, supporting the benefit of online simulated interaction. ASRC nevertheless achieves higher final mean success, reaching 98.3% on Peg Insertion within the evaluated real-world data budget.

W2, part 1. Figure 6 (left) varies the target mixture ratio while allowing deviations in every setting, so it does not establish the benefit of availability-aware scheduling over a fixed ratio.

Q2. With the same target real-data fraction, how does availability-aware scheduling compare with keeping the real-data fraction fixed in every training batch? Please report both task success rates and update throughput.

We have added a direct comparison on Pick and Place in Section 5.4, Figure 7 (middle). Availability-aware scheduling uses a target real-data fraction of 0.1 with bounds [0.05, 0.15]. With 36 segments per update, an exact fraction of 0.1 is infeasible, so the fixed-ratio baseline uses the closest feasible composition, four real-world and 32 simulated segments, giving 4/36 ≈ 0.111.

Availability-aware scheduling achieves higher observed success at all four evaluated checkpoints. At the final point, it reaches 94.4% (34/36), compared with 88.9% (32/36) for the fixed-ratio baseline under the same real-world data budget. This provides evidence that adapting the mixture to data availability can benefit learning beyond selecting an appropriate target fraction.

For update throughput, the Async configurations with 32 simulation environments on Peg Insertion achieve 1.095 rounds/min with availability-aware scheduling at α=0.1 (Table 2), compared with 0.631 rounds/min with the closest feasible fixed ratio of 4/36 (Table 6), a 1.74× increase.

W2, part 2. Table 2 demonstrates a throughput advantage from relaxing the zero-staleness constraint, but does not report the corresponding success rates. Figure 4 compares the complete framework with other learning methods, rather than with the Sync counterpart.

Q3. Does Sync in Table 2 alternate between collection and optimization, or does it retain concurrent execution while filtering out data from older policy versions? Under the same initialization and comparable real-world interaction budgets, how do its success rates compare with ASRC? Please also specify the mixture settings and actual real-data fractions for this comparison.

Sync in Table 2 retains concurrent execution while filtering out samples from older policy versions. This zero-staleness implementation preserves availability-aware mixture scheduling. We have clarified this distinction in Appendix C.1.

We have added learning curves in Section 5.3, Figure 6, and extended the comparison in Appendix C.1, Figure 11 to include fully synchronous execution that alternates between collection and optimization. All three methods use the same CNN BC initialization. ASRC and zero-staleness Sync use a target real-data fraction of 0.1, bounds [0.05, 0.15], and 36 segments per update. Their realized fractions depend on data availability and are not fixed at 0.1. Fully synchronous execution instead collects exactly four real-world and 32 simulated segments per update, giving a realized fraction of 11.11%, the closest feasible value to 0.1. Faster collection branches wait for the remaining data before optimization begins.

Across three seeds, ASRC improves faster on average in elapsed training time. It reaches 90% mean success in approximately 55 minutes, compared with 79 minutes for Sync. Fully synchronous execution improves somewhat faster than zero-staleness Sync, but remains slower than ASRC on average in this comparison.

Q4. What GPU models and how many GPUs were used? How were computational resources allocated among simulation, policy inference, and model updates in the throughput experiments?

We use two RTX 4090 GPUs, each with 24 GB of memory, with identical hardware resources and component placement across execution modes. One GPU hosts policy optimization and real-world policy inference. The other hosts simulation and simulated-policy inference. The robot environment runs on a separate control computer. We have added these details to Appendix B.5.

[1] Luo et al. SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning. 2024.

[2] Luo et al. Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning. 2025.

[3] Di et al. Sim and Real: Better Together. 2021.

[4] Liu et al. Sim-and-Real Reinforcement Learning for Manipulation: A Consensus-based Approach. 2023.

[5] Schulman et al. Proximal Policy Optimization Algorithms. 2017.

Reviewer_DxisOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper propose ASRC as a novel training policy. Simulated and real-world experience are collected independently, then merged for PPO-style policy updates. A scheduler that accounts for data availability decides how much of each batch is real data, and a latest-first rule favors fresher samples. The paper test on three manipulation tasks with one physical robot, and ASRC improves both CNN policies and π0.5-residual policies. On the systems side, they report up to 2.96× higher update throughput than a zero-staleness synchronous baseline.

Critical strengths
  1. This paper solve a critical limitation in the community. Sim and real data rarely arrive at the same pace, and the framework handles that directly instead of just assuming everything is collected in lockstep.
  2. The paper tested on more than one policy setup against several demonstration-based and RL baselines, and reports both real-robot success and execution time, which is nice to see.
  3. The ablations look at the things that matter for the system design, namely the real-data mixture and sample freshness, plus throughput at different levels of simulation parallelism.
Critical weaknesses
  1. The statistical evidence is thin. The real-world evaluation is three tasks on one robot, with only 36 or 40 trials per checkpoint. There are no repeated training runs and no uncertainty intervals.
  2. The sim-to-real scope is pretty narrow. Everything uses paired task environments in ManiSkill, with no system identification and no domain randomization. So readers wouldn’t read these results as showing robustness to bigger simulator or hardware gaps.
  3. It’s not clear where the improvements actually come from. There’s no synchronous sim-real PPO baseline matched on data and update budgets. The throughput comparison tells the system is faster, but it doesn’t show how much the asynchrony itself improves task performance compared with an otherwise identical co-training setup.
Questions for authors
  1. Could you give multi-seed results with uncertainty intervals, using evaluation trials that are independent of checkpoint selection?
  2. Could you compare ASRC against a synchronous sim-real PPO baseline with matched sim and real data and update budgets, and also report wall-clock time to reach the same performance?
3 comments

We thank the reviewer for recognizing the practical importance of unequal sim-real collection rates and for the helpful suggestions on statistical evidence, scope, and synchronized comparisons.

W1. The statistical evidence is thin. The real-world evaluation is three tasks on one robot, with only 36 or 40 trials per checkpoint. There are no repeated training runs and no uncertainty intervals.

Q1. Could you give multi-seed results with uncertainty intervals, using evaluation trials that are independent of checkpoint selection?

  1. Multi-seed results. Appendix D.2 extends the Peg Insertion experiments to three training seeds for both online methods and static baselines, reporting mean success rates and standard deviations. ASRC achieves final mean success rates of 98.3% for CNN policies and 99.2% for π0.5 residual policies.

  2. Uncertainty intervals for the original results. Appendix D.1 now reports final success rates, successful/total trial counts, and 95% Wilson confidence intervals for the original Figure 4 results. These tables make ASRC’s higher observed final success rates and the associated evaluation uncertainty explicit.

  3. Independence from checkpoint selection. Appendix A.7 clarifies that RL evaluation checkpoints are selected using predetermined training intervals and real-data budgets. Evaluation success rates are not used to select models.

W2. The sim-to-real scope is pretty narrow. Everything uses paired task environments in ManiSkill, with no system identification and no domain randomization. So readers wouldn’t read these results as showing robustness to bigger simulator or hardware gaps.

We agree with this scope distinction. Our setup follows the practical approach supported by prior co-training studies. Maddukuri et al. show that simulation data can benefit real-world learning despite substantial domain differences, while identifying camera alignment as an important factor [1]. Wei et al. further show that physics mismatch can matter more than visual fidelity in planar pushing [2]. These findings motivate using approximately aligned, task-relevant simulation rather than requiring a visually exact replica.

Following these findings, our contribution focuses on improving online policy learning with limited real hardware, using approximately aligned simulators without extensive visual reconstruction or system identification.

W3. It’s not clear where the improvements actually come from. There’s no synchronous sim-real PPO baseline matched on data and update budgets. The throughput comparison tells the system is faster, but it doesn’t show how much the asynchrony itself improves task performance compared with an otherwise identical co-training setup.

Q2. Could you compare ASRC against a synchronous sim-real PPO baseline with matched sim and real data and update budgets, and also report wall-clock time to reach the same performance?

We have added success-rate curves against elapsed training time in Section 5.3, Figure 6, comparing Async and Sync over three seeds. The results show that the throughput advantage also translates into faster policy improvement. At approximately 67 minutes, ASRC reaches 97.2% mean success, compared with 82.5% for Sync. To reach the same 90% mean success rate, ASRC takes approximately 55 minutes, whereas Sync takes approximately 79 minutes.

We additionally implement a fully synchronous baseline that alternates between collection and optimization, reported in Appendix C.1, Figure 11. It uses the same BC initialization and collects four real-world and 32 simulated segments before each update, with faster collection branches waiting for the remaining data. This baseline improves somewhat faster than zero-staleness Sync, but remains slower than ASRC on average in elapsed training time.

[1] Maddukuri et al. Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation. 2025.

[2] Wei et al. Empirical Analysis of Sim-and-Real Cotraining of Diffusion Policies for Planar Pushing from Pixels. 2025.

I'm good with the rebuttal so far, I will maintain my score.

Thank you for your time and thoughtful feedback. We are glad to hear that you are satisfied with our rebuttal so far. We would be happy to discuss any further questions or concerns during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General author comment

We thank all three reviewers for their careful reading and constructive feedback. The reviewers recognized the practical importance of learning with limited real hardware and the value of evaluating ASRC across multiple tasks and policy architectures. Their suggestions also helped us clarify the contribution and strengthen the experimental evidence and reporting. We have updated the manuscript and highlighted revisions in blue. The main additions and revisions are as follows:

  1. Clarified motivation (fRy8). We revised the first three paragraphs of the introduction to distinguish expert annotation costs from limited physical interaction, clarifying that ASRC primarily addresses the challenge of on-policy real-world RL with limited hardware.

  2. Success rates against wall-clock training time (Dxis, fRy8, 6VGk). Section 5.3 and Appendix C.1 now compare asynchronous, zero-staleness, and fully synchronous execution over three seeds. These results show that ASRC’s throughput benefits also translate into faster policy improvement on average in elapsed training time.

  3. Mixture-scheduling ablation (fRy8). Section 5.4 now directly compares availability-aware scheduling with a fixed real-data ratio. On Pick and Place, the adaptive setting achieves higher observed success at all four evaluation points, reaching 94.4% compared with 88.9% for the fixed-ratio setting.

  4. Operator blinding and checkpoint selection (Dxis, 6VGk). Appendix A.6 clarifies that operators are unaware of the deployed algorithm during both training and evaluation, reducing the potential for method-dependent labeling bias. Appendix A.7 explains the predetermined RL checkpoint-selection protocol, which does not select models according to evaluation success rates.

  5. Hardware resources and component placement (fRy8). Appendix B.5 now specifies the hardware resources and how simulation, policy inference, and optimization are distributed across the machines, with the same allocation across execution modes.

  6. Previously omitted training parameters (6VGk). Table 5 now reports the target real-data fraction and its bounds, segments per update, segment length, batch size, and buffer capacities for both policy families and all three tasks.

  7. Throughput at exactly matched real-data ratios (6VGk). Appendix C.2 adds Async-versus-Sync comparisons at fixed real-data fractions of 2/36 and 4/36, using 8, 16, and 32 simulation environments. Async achieves higher throughput in all six matched comparisons, removing differences in domain composition as an explanation for these gains.

  8. Final success-rate tables and evaluation uncertainty (Dxis, 6VGk). Appendix D.1 now reports final success rates, 95% Wilson confidence intervals, and successful/total evaluation trial counts for every method, task, and policy family in Figure 4.

  9. Additional training seeds (Dxis, 6VGk). Appendix D.2 adds two training seeds to the Peg Insertion experiments for both online methods and static baselines with CNN and π0.5 policies, reporting means and standard deviations across three runs. ASRC achieves final mean success rates of 98.3% and 99.2%, respectively.

  10. An off-policy online co-training baseline (fRy8). Appendix D.3 adds an RLPD-based baseline that continually collects new experience from both simulation and the real world. This provides a closer comparison with online sim-real co-training than the original RLPD baseline, whose online interaction is restricted to the real world.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.