- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes an online PPO improvement of an already-capable robot policy when real hardware is scarce. The motivation behind is that sim and real produce experience at very different, time-varying rates — the real branch stalls on resets and operator readiness — so a synchronous collect-then-update loop leaves one branch idle. ASRC decouples them: each domain runs its own env loop and rollout copy of the shared policy into its own buffer, and a single learner pulls from both while collection continues.
It concludes three mechanisms. Availability-aware mixture scheduling (target real fraction α* with an allowed band; each update takes the feasible real count closest to α*B). Latest-first selection with bounded-buffer eviction instead of FIFO. Decoupled PPO to handle batches spanning several behavior-policy versions. Same pipeline for CNN policies and residual adaptation.
Critical strengths
The CNN Peg Insertion result is large, clean, and has no human in the loop. 0% → ~100% real success while real-only PPO, RLPD and every fixed reference stay near zero on the same budget. It's also the one task with fully automated success detection (TCP pose tolerance, A.6), so it's immune to the labeling problem below, and the gap is far too big to be a trial-count artifact. This carries most of my willingness to see the paper developed.
The sim-to-real setup is deliberately crude and they say so. A.2: no system ID, default Panda stiffness/damping, manual camera alignment, composited photo backgrounds. Showing co-training still helps under an uncalibrated simulator is more transferable than a result that needs a digital twin.
Critical weaknesses
The headline throughput number is the best of twelve cells, and their own ablation undercuts it. 2.96× is the α*=0.05, 32-env cell. §5.4 says α*=0.05 "leads to less stable performance"; α*=0.1 learns best. And the speedup decays as the real fraction rises. Looking only at cells where Sync and Async share an identical realized real ratio:
- α*=0.2, all rows at 16.67%: 1.18× / 1.21× / 1.16×. Saturates by 16 envs.
- α*=0.1, Async-8 at 11.11% vs Sync-32 at 8.73% — more real data — gives 1.02×.
The 2.96× cell isn't ratio-matched either: 4.76% vs Sync's 5.56%. The table reports this; the text never adjusts for it. Fair read: ~1.2–1.9× at matched real fraction, with the apparent 3× partly from running a leaner mixture.
The experiment that would justify the framework isn't there, and the headline success claim has no table. The motivation is that real hardware is scarce, so wall-clock binds. But Figure 4 plots success against real-world transitions — the one axis where asynchrony can't help by construction. Table 2 reports throughput with no learning outcome attached. Sync never appears in Figure 4; ASRC also never appears on a wall-clock axis. So the system and policy contributions are not connected.
And ">90% on every task" appears in no table. The only checkable numbers are execution times on hand-picked successful trials. Final success has to be read off six small line plots — no trial counts, no CIs, no seeds. At n = 36 success is quantized to ~2.8 pp, and §5.4's α*=0.5 "gain of 2.8 percentage points over the initial 75%" is 27/36 → 28/36. One trial, reported to one decimal place.
Questions for authors
- Plot real success against wall-clock, ASRC vs Sync, on at least one task. Does the throughput advantage become faster improvement per hour of robot time?
- Add a table of final success rates per method / task / policy family, with trial counts and binomial CIs. How many seeds underlie each Figure 4 curve — is it one?
- Add Sync rows at 8 and 16 envs, and give Async-vs-Sync at exactly matched realized real ratio for every α*. What's the speedup attributable to asynchrony alone? And why quote the α*=0.05 cell when §5.4 prefers α*=0.1?
- What were α*, α_min, α_max,
C_d,B, and segment length for the main results? Table 5 lists learning rates and env counts but not one hyperparameter of either mechanism you claim as a contribution. - On Pick and Place and Pour Marbles success is labeled by an operator at a keyboard, who can also end an episode as a safety failure (A.6). Was the operator blind to the method? If not, the headline metric on two of three tasks comes from a non-blinded human who is also in the training loop.
5 comments
W2, part 2. The headline success claim has no table. And “>90% on every task” appears in no table. The only checkable numbers are execution times on hand-picked successful trials. Final success has to be read off six small line plots — no trial counts, no CIs, no seeds. At n=36 success is quantized to approximately 2.8 percentage points, and the reported gain from the initial 75% is 27/36 to 28/36.
Q2. Add a table of final success rates per method / task / policy family, with trial counts and binomial CIs. How many seeds underlie each Figure 4 curve — is it one?
We have expanded the statistical reporting and revised the wording concerning the one-trial difference.
-
Final success rates and evaluation uncertainty. Each original Figure 4 curve represents one training run, using a common training seed across the experiments. Appendix D.1, Tables 7 and 8 now report final success rates, successful/total trial counts, and 95% Wilson confidence intervals for every method, task, and policy family. Evaluations use 36 trials for Pick and Place and Pour Marbles, and 40 for Peg Insertion. ASRC’s final results are 34/36, 40/40, and 36/36 for CNN policies, and 34/36, 39/40, and 33/36 for π0.5 residual policies, respectively. These tables make the above-90% observed success rates and their uncertainty directly inspectable.
-
Repeated training runs. Appendix D.2, Figure 12 extends the Peg Insertion experiments to three training seeds for both online methods and static baselines. ASRC achieves final mean success rates of 98.3% for CNN policies and 99.2% for π0.5 residual policies, with sample standard deviations of 2.9 and 1.4 percentage points, respectively. These results show consistently high final performance across the three evaluated seeds.
-
Interpretation of the 2.8-percentage-point difference. The intended observation was that the high-real-fraction setting provides little improvement over initialization. We have removed the emphasis on this one-trial increase and replaced it with “remaining close to the initial success rate” in Section 5.4.
Q3. Add Sync rows at 8 and 16 envs, and give Async-vs-Sync at exactly matched realized real ratio for every α*. What's the speedup attributable to asynchrony alone? And why quote the α*=0.05 cell when Section 5.4 prefers α*=0.1?
We have added Appendix C.2, Table 6, comparing Async and Sync at exactly fixed real-data fractions of 2/36 and 4/36, with 8, 16, and 32 simulation environments. Every matched pair uses the same hardware resources, simulation parallelism, and domain composition per update. Speedups are computed relative to the corresponding Sync configuration.
| Fixed real-data fraction | 8 simulation environments | 16 simulation environments | 32 simulation environments |
|---|---|---|---|
| 2/36, approximately 5.56% | 1.38× | 2.05× | 3.04× |
| 4/36, approximately 11.11% | 1.25× | 1.27× | 1.17× |
Because each update consumes a fixed number of transitions from each domain, the speedups in real-world data throughput, simulation data throughput, and update throughput are identical within each matched pair. Async remains faster in all six comparisons after eliminating differences in domain composition, although the gains are smaller at the higher real-data fraction.
Regarding α*=0.05 versus α*=0.1, Section 5.3 does not infer that a configuration with higher throughput necessarily produces a better policy. It examines how simulation parallelism and execution mode affect throughput. Section 5.4 separately examines how the mixture affects learning on Pick and Place. Thus, the peak throughput result and the preferred learning configuration in that ablation answer different questions.
Q4. What were α*, α_min, α_max, C_d, B, and segment length for the main results? Table 5 lists learning rates and env counts but not one hyperparameter of either mechanism you claim as a contribution.
We have added these parameters to Table 5 for both policy families and all three tasks. The table now reports:
- Target real-data fraction α=0.1* and bounds [0.05, 0.15].
- B=36 segments per update, with 120 transitions per segment.
- Optimizer batch size 240.
- Real buffer capacity 37 segments.
- Simulation buffer capacity 68 segments for CNN policies, and 52, 68, and 52 segments for π0.5 residual policies on Pick and Place, Peg Insertion, and Pour Marbles, respectively.
Q5. On Pick and Place and Pour Marbles success is labeled by an operator at a keyboard, who can also end an episode as a safety failure. Was the operator blind to the method? If not, the headline metric on two of three tasks comes from a non-blinded human who is also in the training loop.
Yes, the operator was blind to the algorithm identity during both training and evaluation. Algorithm development and robot operation were performed by different people. An algorithm developer launched the models, while a separate robot operator labeled outcomes and recorded trial results without knowing which algorithm the deployed model corresponded to.
We have added this clarification to Appendix A.6. The operator followed the same task-specific success criteria across methods.
[1] Maddukuri et al. Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation. 2025.
[2] Wei et al. Empirical Analysis of Sim-and-Real Cotraining of Diffusion Policies for Planar Pushing from Pixels. 2025.
[3] Shi et al. Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Models. 2026.
We thank the reviewer for the close examination of the throughput table and evaluation protocol. We agree that the original presentation needed to connect system throughput to learning outcomes, separate composition effects, and make the final success statistics directly inspectable.
We would like to clarify that the system-efficiency analysis and the mixture-ratio ablation address different questions.
The ablation in Section 5.4 examines policy learning on Pick and Place. Its purpose is to show that either too little or too much real-world data can impair learning, rather than to establish 0.1 as a universally optimal target ratio. The appropriate mixture can depend on the task and available data, consistent with prior co-training studies [1–3]. In particular, the best tested ratio on Pick and Place does not establish the optimal ratio on Peg Insertion, the task used for the throughput experiments.
Conversely, Table 2 in Section 5.3 studies system throughput, rather than selecting the mixture that maximizes task success. Its main observations are:
Increasing simulation parallelism can improve throughput, with diminishing benefits as the real-data fraction increases and physical collection becomes the bottleneck.
Asynchronous execution can achieve higher throughput than Sync under the evaluated configurations.
The reported “up to 2.96×” describes the largest observed system speedup, not a recommendation to use α*=0.05 for policy learning. We acknowledge that this cell alone does not isolate asynchrony from differences in realized composition. We therefore add exactly matched-ratio comparisons in Appendix C.2, as detailed in our response to Q3 below.
We have also added success-rate curves against elapsed training time in Section 5.3 and Appendix C.1 to connect these system measurements to policy improvement.
Thank you for identifying this missing comparison. Figure 4 uses real-world transitions to compare policy improvement under a common physical-interaction budget. With the robot operating at a nominal control frequency of 10 Hz, transition counts directly reflect active robot interaction time. They also avoid confounding algorithm comparisons with differences in how quickly an operator performs resets or provides feedback across sessions.
To compare ASRC and Sync in elapsed training time, we have added real-world success versus wall-clock time on Peg Insertion in Section 5.3, Figure 6, using three seeds. Waiting time is included. We linearly interpolate evaluations at their actual checkpoint times to report the mean and standard deviation across runs. ASRC reaches 90% mean success in approximately 55 minutes, compared with 79 minutes for Sync. At approximately 67 minutes, ASRC reaches 97.2%, compared with 82.5% for Sync.
Appendix C.1, Figure 11 additionally includes fully synchronous execution that alternates between collection and optimization. This baseline improves somewhat faster than zero-staleness Sync, but remains slower than ASRC on average in elapsed training time. These comparisons directly connect the execution design to learning outcomes.