- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
SIE adapts frozen VLAs through a small latent actor using streaming critic learning and success-triggered Episodic Policy Updates (EPU). Experiments on LIBERO, Meta-World, and simulated ALOHA show improved adaptation.
Critical strengths
- Success-driven actor updates directly complement critic-guided learning under sparse feedback. The matched EPU ablation improves mean success from 66.3% to 72.1%.
- The study includes multiple VLA checkpoints, three simulation benchmarks, and measured two-device latency and memory, supported by a code supplement.
Critical weaknesses
-
Practical streaming adaptation remains insufficiently validated.
- All evaluations are simulated, without real-robot validation under noisy observations, control delays, and limited trials.
- EPU caches the complete current episode's observations and latent actions, then processes them together. Storage and episode-end computation grow with episode length, weakening the low-storage streaming advantage.
- EPU updates only after success. On difficult tasks with few successful trajectories, it rarely activates and cannot itself help obtain the first success.
-
Experimental selection and comparison weaken the evidence. Appendix A.1 explicitly states that LIBERO task subsets were selected after observing results, making aggregate gains vulnerable to selection bias. DSRL-IU and SIE also use different critic scales (0.50 versus 0.10); the matched EPU ablation supports a 5.8-point gain, but does not justify attributing the much larger main-comparison gap to EPU.
Questions for authors
- What evidence supports physical deployment feasibility under noisy observations, control delays, and limited trials?
- What are total CPU/GPU memory and peak episode-end latency as episode length increases, including EPU trajectory storage and processing?
- How often does EPU activate on near-zero-success tasks, and how many trials precede the first success? Does it help under limited interaction budgets?
- What criteria determined the post-hoc LIBERO subsets? Please report all attempted tasks and clarify the separate effects of critic-scale tuning and EPU.
5 comments
Response to Reviewer PJfu (Part 2/2: Questions 3–4)
Question 3: How often does EPU activate on near-zero-success tasks, and how many trials precede the first success? Does it help under limited interaction budgets?
Answer 3:
- On tasks where the frozen checkpoint has a strictly zero success rate and never produces a successful rollout, EPU does not activate (which is why we exclude near-zero-initial-success tasks, also see Appendix A.1). On the hardest evaluated tasks with the lowest initial success rates (~5%–10% at Step 0, such as Tasks 6 and 49 and Task 82), EPU activates in 8.7% (35/402), 7.9% (31/392), and 9.6% (38/394) of completed episodes during the first 50k steps, respectively (rising to 18.7%, 20.4%, and 37.0% over the full 500k steps as the policy improves).
- Across all 45 runs (15 tasks 3 seeds) and 30 runs (10 tasks 3 seeds) on LIBERO-90, the median first success occurs after just 3 episodes (1,083 environment steps) for and 2 episodes (603 environment steps) for , with the median first applied EPU update occurring at 1,932 and 1,926 steps, respectively. Even on the hardest evaluated tasks above ( Tasks 6 and 49, and Task 82), first successes occur within 5/26/24, 7/2/16, and 11/18/7 episodes across the three seeds.
- Yes. Because initial successes occur within just a few episodes, EPU begins reinforcing successful trajectories very early and helps substantially under limited interaction budgets.
This is directly demonstrated by our real-world Piper X experiment in Answer 1: under a strictly limited budget of only 20 online training episodes,
SIEimproves evaluation success rate from 30% to 65% (+35 pp), whereasDSRL-Streamcollapses to 0%.
Question 4: What criteria determined the post-hoc LIBERO subsets? Please report all attempted tasks and clarify the separate effects of critic-scale tuning and EPU.
Answer 4:
- When selecting the LIBERO-90 task subsets, we first evaluated the zero-shot success rate of the frozen checkpoint (See Appendix A.1 for details) across all 90 tasks and then selected tasks with intermediate initial success rates—specifically, approximately 10%–60% for and 5%–70% for —yielding the 15 tasks for and 10 tasks for (25 evaluated task configurations in total). As noted in Section 5.2, this avoids tasks with near-zero initial success rates (where sparse-reward online RL receives almost no learning signal) as well as already saturated tasks (where little room remains for online improvement).
- Regarding the separate effects of critic-scale tuning and EPU,
DSRL-IUuses IU's default , which seems to be overly aggressive for continuous latent-space VLA control; our experiments found that reducing it to (SIE w/o EPU) stabilizes per-step value updates and substantially improves overDSRL-IU(e.g., from 46.31% to 66.27% on –LIBERO and 52.13% to 69.40% on –LIBERO). On top of this matched baseline (SIE w/o EPU), adding EPU (SIE) consistently brings further gains across all four settings: +5.82 pp on –LIBERO (66.27% 72.09%), +4.53 pp on –LIBERO (69.40% 73.93%), +3.20 pp on –Meta-World (59.87% 63.07%), and +2.67 pp on –ALOHA. Furthermore, in a parameter sensitivity analysis in the ALOHA task, we found that simply setting the from 0.05 to 0.025 could gain a 8.7% improvement in success rate (Appendix A.6).
We have clarified the task-selection rule in Appendix A.1 and included the full four-setting ablation and sensitivity results in Appendices A.5 and A.6. We will further clarify the separate contributions of critic-scale tuning and EPU in the main text.
Thanks for the detailed response and new experiments, which address my main concerns and clarify the scope and limitations of the method.
Response to Reviewer PJfu (Part 1/2: Questions 1–2)
We sincerely thank you for the careful reading and constructive feedback. We are glad that you recognized how our EPU updates directly complement critic-guided learning under sparse feedback, and appreciated our evaluation across multiple VLA checkpoints, benchmarks and experiments supported by the code supplement. Below, we show new results and try to address each of your questions in detail.
Question 1: What evidence supports physical deployment feasibility under noisy observations, control delays, and limited trials?
Answer 1:
We conducted a real-world online RL experiment to directly validate the physical deployment feasibility of SIE under real sensor noise, control delays, and a limited interaction budget. Specifically, we compare
SIE,DSRL-Replay, andDSRL-Streamon a Piper X robotic arm for a pick-and-place task ("picking up a blue block and placing it into a yellow bowl"). All three methods share the same π0 backbone and undergo 20 valid online training episodes, after which the final policy is frozen and evaluated over 20 episodes.Experimental settings shared by the three methods:
pi0_basemodel on 20 demonstrations collected in the same workspace for 20k steps. During online RL adaptation, the π0 parameters remain frozen, and only the latent-space actor and critic are updated.Key findings:
SIEandDSRL-Replayadapt effectively within the limited 20-episode online budget—reaching 65% (+35 pp) and 80% (+50 pp) evaluation success rates, respectively—whereasDSRL-Streamcollapses to 0%.SIEachieves substantially lower update latency (40.7 ms vs. ~786.0 ms per action chunk) and less than half the learner GPU memory (1.26 GiB allocated / 2.84 GiB total vs. 3.12 GiB / 6.08 GiB) compared withDSRL-Replay. Specifically, becauseSIE(andDSRL-Stream) performs a lightweight single-sample update that finishes well within the concurrent inter-chunk inference pipeline (data processing, π0 inference, and SAC inference), it can update in real time between action chunks with virtually no extra delay, allowing the actor to immediately use the latest weights at every query. In contrast, becauseDSRL-Replay's heavy minibatch updates would visibly stall robot motion if executed between chunks, its updates must be deferred to the end of each episode, which prevents within-episode policy adaptation.We have incorporated these real-world experimental results into the revised manuscript (Section 6 and Appendix D).
Question 2: What are total CPU/GPU memory and peak episode-end latency as episode length increases, including EPU trajectory storage and processing?
Answer 2:
Our EPU caches only the current episode's observations and latent actions and immediately clears this cache as soon as the episode terminates. Moreover, because the VLA executes action chunks (q=20 environment steps per query on LIBERO; Section 3.1), a full 400-step episode contains at most 20 queries, so the cached data is minimal. We specifically ran profiling experiments to measure both the memory footprint and episode-end latency: at the maximum 20-query episode length, the EPU trajectory cache occupies at most 242.5 KiB (< 0.25 MiB, with peak learner GPU allocated memory at 107.9 MiB and CPU RSS at 1,266 MiB), and the EPU update latency has a median of 31.93 ms (maximum 75.28 ms), confirming that EPU preserves the low-storage and low-latency advantages of streaming RL.