acceptodds
Under review as a conference paper at ICLR 2027

Online VLA Adaptation with Streaming Reinforcement Learning

Abstract

Streaming Reinforcement Learning (RL) learns directly from sequential experience without relying on replay buffers or minibatch updates. It offers an appealing framework for online adaptation of vision-language-action (VLA) models, but this direction remains largely underexplored. In this work, we investigate this problem and develop Streaming with Intentional and Episodic Updates (SIE), a new streaming VLA adaptation framework that combines streaming critic learning with an Episodic Policy Updates (EPU) mechanism, which incorporates episodic success feedback through a lightweight actor update. Experiments with on LIBERO-90 demonstrate that our framework improves mean final success rate from 47.20% to 72.09% compared with the baseline. In a synchronous two-device benchmark, SIE reduces client wall time by 66.5%. Our results show that streaming RL is a promising direction for efficient VLA adaptation.

Primary area:
reinforcement learning
Keywords:
Vision-Language-Action Models, Streaming Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PJfuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

SIE adapts frozen VLAs through a small latent actor using streaming critic learning and success-triggered Episodic Policy Updates (EPU). Experiments on LIBERO, Meta-World, and simulated ALOHA show improved adaptation.

Critical strengths
  1. Success-driven actor updates directly complement critic-guided learning under sparse feedback. The matched EPU ablation improves mean success from 66.3% to 72.1%.
  2. The study includes multiple VLA checkpoints, three simulation benchmarks, and measured two-device latency and memory, supported by a code supplement.
Critical weaknesses
  1. Practical streaming adaptation remains insufficiently validated.

    • All evaluations are simulated, without real-robot validation under noisy observations, control delays, and limited trials.
    • EPU caches the complete current episode's observations and latent actions, then processes them together. Storage and episode-end computation grow with episode length, weakening the low-storage streaming advantage.
    • EPU updates only after success. On difficult tasks with few successful trajectories, it rarely activates and cannot itself help obtain the first success.
  2. Experimental selection and comparison weaken the evidence. Appendix A.1 explicitly states that LIBERO task subsets were selected after observing results, making aggregate gains vulnerable to selection bias. DSRL-IU and SIE also use different critic scales (0.50 versus 0.10); the matched EPU ablation supports a 5.8-point gain, but does not justify attributing the much larger main-comparison gap to EPU.

Questions for authors
  1. What evidence supports physical deployment feasibility under noisy observations, control delays, and limited trials?
  2. What are total CPU/GPU memory and peak episode-end latency as episode length increases, including EPU trajectory storage and processing?
  3. How often does EPU activate on near-zero-success tasks, and how many trials precede the first success? Does it help under limited interaction budgets?
  4. What criteria determined the post-hoc LIBERO subsets? Please report all attempted tasks and clarify the separate effects of critic-scale tuning and EPU.
5 comments
AuthorsOpenReview ↗

Response to Reviewer PJfu (Part 1/2: Questions 1–2)

We sincerely thank you for the careful reading and constructive feedback. We are glad that you recognized how our EPU updates directly complement critic-guided learning under sparse feedback, and appreciated our evaluation across multiple VLA checkpoints, benchmarks and experiments supported by the code supplement. Below, we show new results and try to address each of your questions in detail.

Question 1: What evidence supports physical deployment feasibility under noisy observations, control delays, and limited trials?

Answer 1:

We conducted a real-world online RL experiment to directly validate the physical deployment feasibility of SIE under real sensor noise, control delays, and a limited interaction budget. Specifically, we compare SIE, DSRL-Replay, and DSRL-Stream on a Piper X robotic arm for a pick-and-place task ("picking up a blue block and placing it into a yellow bowl"). All three methods share the same π0\pi_0 backbone and undergo 20 valid online training episodes, after which the final policy is frozen and evaluated over 20 episodes.

Experimental settings shared by the three methods:

  • Backbone: We fine-tune LeRobot's official open-source pi0_base model on 20 demonstrations collected in the same workspace for 20k steps. During online RL adaptation, the π0\pi_0 parameters remain frozen, and only the latent-space actor and critic are updated.
  • Observations: Two RGB views from a fixed external camera and a wrist-mounted camera.
  • Execution: Each query produces a 50-step action chunk, of which the first 25 steps are executed synchronously at 30 Hz. Each episode has a maximum duration of 75 seconds.
  • Hardware: A single local workstation equipped with an Intel Core Ultra 7 270K Plus CPU and one NVIDIA GeForce RTX 5070 Ti GPU (16 GB VRAM).

Key findings:

  1. Performance: Starting from a 30% initial success rate of the frozen π0\pi_0 policy, both SIE and DSRL-Replay adapt effectively within the limited 20-episode online budget—reaching 65% (+35 pp) and 80% (+50 pp) evaluation success rates, respectively—whereas DSRL-Stream collapses to 0%.
  2. Real-time latency and memory: Overall, SIE achieves substantially lower update latency (40.7 ms vs. ~786.0 ms per action chunk) and less than half the learner GPU memory (1.26 GiB allocated / 2.84 GiB total vs. 3.12 GiB / 6.08 GiB) compared with DSRL-Replay. Specifically, because SIE (and DSRL-Stream) performs a lightweight single-sample update that finishes well within the concurrent inter-chunk inference pipeline (data processing, π0\pi_0 inference, and SAC inference), it can update in real time between action chunks with virtually no extra delay, allowing the actor to immediately use the latest weights at every query. In contrast, because DSRL-Replay's heavy minibatch updates would visibly stall robot motion if executed between chunks, its updates must be deferred to the end of each episode, which prevents within-episode policy adaptation.
MetricSIEDSRL-ReplayDSRL-Stream
Successful evaluation episodes13/2016/200/20
Evaluation success rate65%80%0%
Committed online gradient updates7448,540854
Update-to-data ratio (UTD)1101
Batch size12561
Mean inter-chunk command interval256.1 ms257.0 ms258.3 ms
Mean online update time40.7 ms/action chunk33.56 s/episode (~786.0 ms/action chunk)24.6 ms/action chunk
Peak learner CUDA allocated memory1.26 GiB3.12 GiB1.10 GiB
Peak sampled learner GPU memory, including CUDA context and cache2.84 GiB6.08 GiB2.26 GiB

We have incorporated these real-world experimental results into the revised manuscript (Section 6 and Appendix D).

Question 2: What are total CPU/GPU memory and peak episode-end latency as episode length increases, including EPU trajectory storage and processing?

Answer 2:

Our EPU caches only the current episode's observations and latent actions and immediately clears this cache as soon as the episode terminates. Moreover, because the VLA executes action chunks (q=20q = 20 environment steps per query on LIBERO; Section 3.1), a full 400-step episode contains at most 20 queries, so the cached data is minimal. We specifically ran profiling experiments to measure both the memory footprint and episode-end latency: at the maximum 20-query episode length, the EPU trajectory cache occupies at most 242.5 KiB (< 0.25 MiB, with peak learner GPU allocated memory at 107.9 MiB and CPU RSS at 1,266 MiB), and the EPU update latency has a median of 31.93 ms (maximum 75.28 ms), confirming that EPU preserves the low-storage and low-latency advantages of streaming RL.

AuthorsOpenReview ↗

Response to Reviewer PJfu (Part 2/2: Questions 3–4)

Question 3: How often does EPU activate on near-zero-success tasks, and how many trials precede the first success? Does it help under limited interaction budgets?

Answer 3:

  1. On tasks where the frozen checkpoint has a strictly zero success rate and never produces a successful rollout, EPU does not activate (which is why we exclude near-zero-initial-success tasks, also see Appendix A.1). On the hardest evaluated tasks with the lowest initial success rates (~5%–10% at Step 0, such as π0\pi_0 Tasks 6 and 49 and π0.5\pi_{0.5} Task 82), EPU activates in 8.7% (35/402), 7.9% (31/392), and 9.6% (38/394) of completed episodes during the first 50k steps, respectively (rising to 18.7%, 20.4%, and 37.0% over the full 500k steps as the policy improves).
  2. Across all 45 π0\pi_0 runs (15 tasks ×\times 3 seeds) and 30 π0.5\pi_{0.5} runs (10 tasks ×\times 3 seeds) on LIBERO-90, the median first success occurs after just 3 episodes (1,083 environment steps) for π0\pi_0 and 2 episodes (603 environment steps) for π0.5\pi_{0.5}, with the median first applied EPU update occurring at 1,932 and 1,926 steps, respectively. Even on the hardest evaluated tasks above (π0\pi_0 Tasks 6 and 49, and π0.5\pi_{0.5} Task 82), first successes occur within 5/26/24, 7/2/16, and 11/18/7 episodes across the three seeds.
  3. Yes. Because initial successes occur within just a few episodes, EPU begins reinforcing successful trajectories very early and helps substantially under limited interaction budgets. This is directly demonstrated by our real-world Piper X experiment in Answer 1: under a strictly limited budget of only 20 online training episodes, SIE improves evaluation success rate from 30% to 65% (+35 pp), whereas DSRL-Stream collapses to 0%.

Question 4: What criteria determined the post-hoc LIBERO subsets? Please report all attempted tasks and clarify the separate effects of critic-scale tuning and EPU.

Answer 4:

  1. When selecting the LIBERO-90 task subsets, we first evaluated the zero-shot success rate of the frozen checkpoint (See Appendix A.1 for details) across all 90 tasks and then selected tasks with intermediate initial success rates—specifically, approximately 10%–60% for π0\pi_0 and 5%–70% for π0.5\pi_{0.5}—yielding the 15 tasks for π0\pi_0 and 10 tasks for π0.5\pi_{0.5} (25 evaluated task configurations in total). As noted in Section 5.2, this avoids tasks with near-zero initial success rates (where sparse-reward online RL receives almost no learning signal) as well as already saturated tasks (where little room remains for online improvement).
  2. Regarding the separate effects of critic-scale tuning and EPU, DSRL-IU uses IU's default ηQ=0.50\eta_Q = 0.50, which seems to be overly aggressive for continuous latent-space VLA control; our experiments found that reducing it to ηQ=0.10\eta_Q = 0.10 (SIE w/o EPU) stabilizes per-step value updates and substantially improves over DSRL-IU (e.g., from 46.31% to 66.27% on π0\pi_0–LIBERO and 52.13% to 69.40% on π0.5\pi_{0.5}–LIBERO). On top of this matched ηQ=0.10\eta_Q = 0.10 baseline (SIE w/o EPU), adding EPU (SIE) consistently brings further gains across all four settings: +5.82 pp on π0\pi_0–LIBERO (66.27% →\to 72.09%), +4.53 pp on π0.5\pi_{0.5}–LIBERO (69.40% →\to 73.93%), +3.20 pp on π0.5\pi_{0.5}–Meta-World (59.87% →\to 63.07%), and +2.67 pp on π0\pi_0–ALOHA. Furthermore, in a parameter sensitivity analysis in the ALOHA task, we found that simply setting the ηE\eta_E from 0.05 to 0.025 could gain a 8.7% improvement in success rate (Appendix A.6).

We have clarified the task-selection rule in Appendix A.1 and included the full four-setting ablation and sensitivity results in Appendices A.5 and A.6. We will further clarify the separate contributions of critic-scale tuning and EPU in the main text.

Thanks for the detailed response and new experiments, which address my main concerns and clarify the scope and limitations of the method.

Reviewer_zghzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies online adaptation of frozen vision language action (VLA) models under constrained resource using streaming RL. It proposes SIE, combining a streaming SAC critic trained with borrowed Intentional Updates and a new Episodic Policy Update (EPU). EPU raises the actor’s log-likelihood of a latent actions token of a successful episode with an adaptive magnitude. Across four simulated benchmarks SIE outperforms streaming baselines and cuts emulated client wall time by 66.5%.

Critical strengths
  1. The novel component EPU is cleanly stated and verified. The ablation study shows that the performance gain is attribute to EPU and Figure 6 confirms it reinforces successful decisions as designed.
  2. The efficiency is clearly measured. Figure 4 shows a 66.5% cut in client wall time and 70.2% lower learner GPU memory versus replay, with EPU adding negligible overhead.
Critical weaknesses
  1. The method is motivated by efficient deployment on real resource-constrained robots, but every result is in simulation and the efficiency gain is only emulated. However, the base method DSRL and the streaming works it cites (Stream-X, AVG) all validate on real hardware.
  2. The paper itself states (Appendix C.2) EPU is a biased estimate of the success gradient with no improvement guarantee, and since it only reinforces already-successful episodes.
Questions for authors
  1. Can any physical-robot result be provided? The resource-constrained deployment motivation is central but the evidence is simulation-only.
  2. EPU indiscriminately raises the likelihood of every action in a successful episode, regardless of whether that trajectory was optimal. Is there any suboptimality bound, or other theoretical characterization, on how far EPU can drive the policy from optimal?
  3. Since DSRL-Replay consistently beats SIE on accuracy, in what concrete deployment regime does streaming's efficiency make SIE the preferred choice?
  4. DSRL-IU alone does not improve over DSRL-Stream. Is EPU responsible for essentially all of SIE's gain, and if so, is the IU critic integration necessary, or would EPU on top of a vanilla streaming SAC critic suffice?
5 comments
AuthorsOpenReview ↗

Response to Reviewer zghz (Part 1/3: Question 1)

We sincerely thank you for the thoughtful review and constructive comments. We are glad that you found the novel EPU component cleanly stated and verified and appreciated our clear efficiency measurements showing substantial reductions in client wall time and learner GPU memory. Below, we show new results and try to address each of your questions in detail.

Question 1: Can any physical-robot result be provided? The resource-constrained deployment motivation is central but the evidence is simulation-only.

Answer 1:

We conducted a real-world online RL experiment to directly validate the physical deployment feasibility of SIE under real sensor noise, control delays, and a limited interaction budget. Specifically, we compare SIE, DSRL-Replay, and DSRL-Stream on a Piper X robotic arm for a pick-and-place task ("picking up a blue block and placing it into a yellow bowl"). All three methods share the same π0\pi_0 backbone and undergo 20 valid online training episodes, after which the final policy is frozen and evaluated over 20 episodes.

Experimental settings shared by the three methods:

  • Backbone: We fine-tune LeRobot's official open-source pi0_base model on 20 demonstrations collected in the same workspace for 20k steps. During online RL adaptation, the π0\pi_0 parameters remain frozen, and only the latent-space actor and critic are updated.
  • Observations: Two RGB views from a fixed external camera and a wrist-mounted camera.
  • Execution: Each query produces a 50-step action chunk, of which the first 25 steps are executed synchronously at 30 Hz. Each episode has a maximum duration of 75 seconds.
  • Hardware: A single local workstation equipped with an Intel Core Ultra 7 270K Plus CPU and one NVIDIA GeForce RTX 5070 Ti GPU (16 GB VRAM).

Key findings:

  1. Performance: Starting from a 30% initial success rate of the frozen π0\pi_0 policy, both SIE and DSRL-Replay adapt effectively within the limited 20-episode online budget—reaching 65% (+35 pp) and 80% (+50 pp) evaluation success rates, respectively—whereas DSRL-Stream collapses to 0%.
  2. Real-time latency and memory: Overall, SIE achieves substantially lower update latency (40.7 ms vs. ~786.0 ms per action chunk) and less than half the learner GPU memory (1.26 GiB allocated / 2.84 GiB total vs. 3.12 GiB / 6.08 GiB) compared with DSRL-Replay. Specifically, because SIE (and DSRL-Stream) performs a lightweight single-sample update that finishes well within the concurrent inter-chunk inference pipeline (data processing, π0\pi_0 inference, and SAC inference), it can update in real time between action chunks with virtually no extra delay, allowing the actor to immediately use the latest weights at every query. In contrast, because DSRL-Replay's heavy minibatch updates would visibly stall robot motion if executed between chunks, its updates must be deferred to the end of each episode, which prevents within-episode policy adaptation.
MetricSIEDSRL-ReplayDSRL-Stream
Successful evaluation episodes13/2016/200/20
Evaluation success rate65%80%0%
Committed online gradient updates7448,540854
Update-to-data ratio (UTD)1101
Batch size12561
Mean inter-chunk command interval256.1 ms257.0 ms258.3 ms
Mean online update time40.7 ms/action chunk33.56 s/episode (~786.0 ms/action chunk)24.6 ms/action chunk
Peak learner CUDA allocated memory1.26 GiB3.12 GiB1.10 GiB
Peak sampled learner GPU memory, including CUDA context and cache2.84 GiB6.08 GiB2.26 GiB

We have incorporated these real-world experimental results into the revised manuscript (Section 6 and Appendix D).

AuthorsOpenReview ↗

Response to Reviewer zghz (Part 2/3: Question 2)

Question 2: EPU indiscriminately raises the likelihood of every action in a successful episode, regardless of whether that trajectory was optimal. Is there any suboptimality bound, or other theoretical characterization, on how far EPU can drive the policy from optimal?

Answer 2:

We cannot establish a general suboptimality bound for EPU itself, for two reasons inherent to its design (Appendix C.2): (i) EPU updates from a single trajectory collected under intra-episode actor updates with episode-dependent step scaling, making it a biased estimate of the gradient of task success probability JsuccJ_\text{succ} (Eq. 9) without a per-step improvement guarantee; and (ii) EPU relies only on the binary success indicator Sτ∈{0,1}S_\tau \in \lbrace 0, 1 \rbrace (Eq. 7) rather than step-penalized return, so EPU alone cannot distinguish fast from slow successful trajectories. Nevertheless, these two aspects, namely suboptimal actions within a successful episode and suboptimal trajectory length, admit a clear theoretical characterization of how far EPU can drive the policy from optimal:

  1. One might worry that EPU reinforces actions with below-average success probability when they happen to occur within a successful episode, but under the idealized operator it filters them out in expectation, and in practice single-episode log-likelihood drift is capped at the budget ηE\eta_E. Specifically, let Qsuccπ(s,z)Q^\pi_\text{succ}(s, z) denote the probability of eventual success from (s,z)(s, z) under policy π\pi, and let Vsuccπ(s)=Ez∼π(⋅∣s)[Qsuccπ(s,z)]V^\pi_\text{succ}(s) = \mathbb{E}_{z \sim \pi(\cdot \mid s)}[Q^\pi_\text{succ}(s, z)] denote the state's average success probability (Appendix C.2, Eq. 12). Although Sτ=1S_\tau = 1 weights every step of a successful rollout equally, actions with Qsuccπ(s,z)<Vsuccπ(s)Q^\pi_\text{succ}(s, z) < V^\pi_\text{succ}(s) lead to successful episodes less frequently across rollouts. Thus, idealized population success conditioning, π+(z∣s)=π(z∣s)Qsuccπ(s,z)/Vsuccπ(s)\pi_+(z \mid s) = \pi(z \mid s) Q^\pi_\text{succ}(s, z) / V^\pi_\text{succ}(s), solves a χ2\chi^2-divergence trust-region problem (Russo, 2026) that strictly decreases the density of below-average actions and monotonically improves Jsucc(π)J_\text{succ}(\pi), meaning that this idealized operator does not drive the policy away from optimal success rate. Even on a single suboptimal rollout that happens to succeed, our guarded minimum-norm update (Appendix C.1, Eqs. 10 and 11) bounds the realized episode log-likelihood change from pre-update actor θ−\theta^- to post-update actor θ+\theta^+ by 0<ℓτ(θ+)−ℓτ(θ−)≤ηE+ετ0 < \ell_\tau(\theta^+) - \ell_\tau(\theta^-) \le \eta_E + \varepsilon_\tau (for guard tolerance ετ\varepsilon_\tau) and bounds the parameter shift by ∥θ+−θ−∥2≤ηE/∥gτ∥2\lVert \theta^+ - \theta^- \rVert_2 \le \eta_E / \lVert g_\tau \rVert_2, where gτ=∇θℓτ(θ−)g_\tau = \nabla_\theta \ell_\tau(\theta^-) is the episode log-likelihood gradient (Eq. 8).
  2. EPU does not "lock" the policy into slow, detouring trajectories either: the objective gap from ignoring trajectory length has a finite bound, and SIE's critic actively closes this gap. While binary success Sτ∈{0,1}S_\tau \in \lbrace 0, 1 \rbrace ignores step count, our RL objective (Section 3.1, with per-query reward rt=−1r_t = -1 prior to success, 00 upon success, and per-query discount γ\gamma) explicitly prefers faster completion. If an episode allows at most HH queries and requires at least Tmin⁡≥1T_{\min} \ge 1 queries to succeed, then even if running EPU alone favors a policy that maximizes JsuccJ_\text{succ} via the slowest (HH-query) paths rather than shortest (Tmin⁡T_{\min}-query) paths, its worst-case episodic return gap relative to the return-optimal policy π⋆\pi^\star is bounded by γTmin⁡−1−γH−11−γ≤H−Tmin⁡\frac{\gamma^{T_{\min}-1} - \gamma^{H-1}}{1 - \gamma} \le H - T_{\min}. In SIE, moreover, per-query SAC updates optimize the step-penalized critic Qψ(s,z)Q_\psi(s, z) at every transition to continuously shorten trajectories, while EPU's adaptive update magnitude dτd_\tau (Eq. 7) automatically drops to 00 once per-step SAC updates already raise the episode log-likelihood by at least ηE\eta_E (occurring in 30.62% of successful episodes on π0\pi_0 and 28.39% on π0.5\pi_{0.5}; Table 2). Empirically, across all 15 π0\pi_0-LIBERO tasks, mean successful episode length under SIE shrinks from 214.6 environment steps in early training (0 to 100k steps) to 143.5 steps in late training (400k to 500k steps), becoming shorter on 14 of 15 tasks (and shrinking from 206.4 to 173.1 steps on π0.5\pi_{0.5}-LIBERO, shorter on 8 of 10 tasks), confirming that SIE drives trajectories toward faster solutions rather than away from optimal.
AuthorsOpenReview ↗

Response to Reviewer zghz (Part 3/3: Questions 3–4)

Question 3: Since DSRL-Replay consistently beats SIE on accuracy, in what concrete deployment regime does streaming's efficiency make SIE the preferred choice?

Answer 3:

We have actually thought a lot about this exact question ourselves. In practice today, we see two concrete deployment regimes where SIE is the preferred choice:

  1. Latency-critical, continuous operations requiring stall-free real-time updates. As demonstrated in our Piper X physical-robot experiment above (Question 1), SIE completes per-chunk updates in 40.7 ms—fitting entirely within the ~256 ms inter-chunk inference pipeline so the policy adapts within each episode with virtually zero extra motion delay. By contrast, DSRL-Replay (~786 ms/chunk) must either stall the robot mid-episode, halt operations for 33.56 s after every episode to finish minibatch updates, or suffer severe policy lag in asynchronous mode (up to 510 queries in Figure 4c). On continuous industrial assembly or sorting lines with strict cycle-time budgets, pausing for half a minute or more after every trial is impractical, making SIE far more suitable.
  2. Edge devices in industrial and household environments under strict hardware, power, and privacy constraints:
    • On-board compute, memory, and battery budgets: On mobile household robots or low-cost industrial edge boxes, the on-board processor must share limited VRAM/RAM with perception and navigation stacks while operating under tight battery and thermal limits. Eliminating the 100k-transition visual replay buffer and heavy batch=256 updates cuts learner GPU memory by 2.1–3.4$\times$ (2.84 vs. 6.08 GiB on Piper X; Figure 4d) and avoids sustained GPU load.
    • Data privacy and non-stationary drift: In homes, healthcare settings, or confidential manufacturing lines, retaining thousands of raw camera frames in a replay buffer raises privacy and compliance risks; SIE processes frames ephemerally and clears its tiny (< 0.25 MiB) episodic cache immediately upon episode completion. Moreover, when household layouts or industrial fixtures shift over time (Section 5.4), SIE adapts immediately on fresh transitions without being slowed down—or destabilized (Appendix A.3)—by stale replay data.

Question 4: DSRL-IU alone does not improve over DSRL-Stream. Is EPU responsible for essentially all of SIE's gain, and if so, is the IU critic integration necessary, or would EPU on top of a vanilla streaming SAC critic suffice?

Answer 4:

EPU is not solely responsible for SIE's gain: both the IU critic and EPU are necessary, and pairing EPU with a vanilla streaming SAC critic (DSRL-Stream) would not suffice. While off-the-shelf DSRL-IU uses a default critic step scale from prior work that is overly aggressive for continuous latent VLA steering, our matched ablation in Table 1 (SIE w/o EPU) shows that a conservatively scaled IU critic substantially stabilizes streaming learning, upon which EPU provides consistent further improvements.

Mechanistically, EPU only triggers at the end of successful episodes with a small bounded budget, whereas the actor is updated at every transition, including all failed episodes, by the streaming critic. Without IU's gradient-norm normalization, vanilla single-sample critic updates (DSRL-Stream) easily destabilize or collapse (for example, dropping to 0% success on simulated ALOHA in Figure 3f and on our Piper X robot in Question 1), which both prevents the policy from sustaining the successful rollouts needed to activate EPU and overwrites EPU's episode-end updates with noisy per-step gradients. Conversely, while the IU critic stabilizes step-by-step value learning, single-transition TD updates without replay propagate sparse terminal rewards slowly to early decisions in an episode (Figure 6), which is precisely the credit-assignment gap addressed by EPU.

Reviewer_GiCsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adapts frozen flow-based VLAs (π0, π0.5) online with streaming RL, which learns from each transition without replay, minibatches, or target networks. Building on DSRL, where a small SAC actor chooses the VLA's input noise, SIE updates the critic with Intentional Updates (IU) and adds Episodic Policy Updates (EPU): after each successful episode, an IU-style step raises the summed log-likelihood of its latent actions by an adaptive amount capped at a budget ηE (Eqs. 7-8). On 15 LIBERO-90 tasks with π0, SIE reaches 72.09% mean final success, versus 47.20% for DSRL-Stream and 46.31% for DSRL with an IU critic (DSRL-IU). Gains are also reported for π0.5 (LIBERO, Meta-World) and π0 (ALOHA). Against replay-based DSRL, SIE cuts client wall time by 66.5% and peak learner GPU memory by 70.2%.

Critical strengths
  • S1. A practical question, studied broadly. Cheap online adaptation of frozen VLAs matters, and streaming RL for VLAs is little explored (L013-L014). The study spans two VLAs, three benchmarks, a resource benchmark (Sec. 5.3), and task switching (Sec. 5.4).

  • S2. Clear gains over naive streaming at almost no extra cost. SIE beats DSRL-Stream by 24.89 points on π0-LIBERO (L238-L240) and by 11.6 points on π0.5-LIBERO (L304-L305), and EPU adds almost no learner time (73.3 vs. 73.2 ms per query, L363-L364).

  • S3. Transparent reporting. Hyperparameters (Tables 3-4), protocols (App. A), update diagnostics (Table 2), and code (L512-L513) are provided.

Critical weaknesses
  • W1. Most of the headline gain comes from the critic step size, not from EPU. The critic scale ηQ is the only listed setting that differs between DSRL-IU (0.50) and SIE (0.10) (Table 4, L763), and SIE w/o EPU is IU with ηQ = 0.10 (Table 1 caption, L409). That variant reaches 66.3% versus 46.31% for DSRL-IU (and beats it on all 15 tasks in Figure 1). So, unless I misread the setup, about 20 of the 25.8 points SIE gains over DSRL-IU come from the smaller step and 5.8 from EPU. Yet the abstract compares only with DSRL-Stream (L018-L019), Figures 1-3 show IU only at ηQ = 0.50, and the paper does not say how ηQ or the shared learning rates (Table 3) were chosen. SIE w/o EPU is evaluated only on π0-LIBERO, so EPU's effect in the other three settings is unknown.

  • W2. Tasks were selected after the fact, and several gains are weaker than they look. The LIBERO subsets "were selected after observing experimental results" (App. A.1, L672-L673); with three seeds and no stated rule, this can bias the means toward the proposed method. EPU's gain is marginal: it helps on 10 of 15 tasks and hurts on 3 (Task 61: 78.0 to 56.7), and by my calculation a two-sided paired t-test over the 15 per-task differences gives p ≈ 0.08. The "refined subset" (L403-L407) looks stronger, but its trimming rule was also set after seeing the results. SIE leads DSRL-IU by only 4.5 points on Meta-World (L310-L312), and on ALOHA its 46.0% is close to its own success at step 0 (about 40%, Fig. 3f): it avoids the other streaming methods' collapse rather than improving on the frozen policy.

  • W3. The efficiency claim is not yet supported. The case for streaming is cost (L031-L033), but the evidence has gaps. First, the only replay learner tested is the original DSRL configuration, 20 updates per transition at batch size 256 (L230-L231, Table 4); its learner time is roughly 20 times SIE's (Fig. 4b), and no budget-matched replay learner is tested. Second, the 66.5% saving is per 1,000 queries, not per level of success. Replay ends higher in every setting (final-50 means by my reading: 84.5%, 79.5%, 85.2%, and 92.7%, versus SIE's 72.1%, 73.9%, 63.1%, and 46.0%), numbers the text never reports. At Figure 4(a)'s per-query costs (about 2.3 s vs. 0.8 s), 500k SIE steps take as long as about 167k replay steps, and by my reading replay's Figure 1 curve reaches SIE's final success within that budget on 13 of 15 tasks. Third, the memory test covers 1,000 queries and excludes CPU-side storage (L705-L706), so the full 100k-transition buffer, likely streaming's strongest memory argument, is never measured; the GPU gap shown is about 255 MiB. I do not expect streaming to match replay's success, but the abstract's conclusion (L020-L021) needs this trade-off measured directly.

Questions for authors
  • Q1. How were ηQ and the learning rates chosen for each method? Please report SIE w/o EPU as the main streaming baseline in all four settings, with per-seed intervals.

  • Q2. What rule chose the LIBERO-90 subsets, and how many tasks were run in total?

  • Q3. What are DSRL-Replay's final-50 means in each setting, and how long (wall clock and environment steps) does each method, including a budget-matched replay learner, take to reach, say, 50% and 80% success? Since the server already receives every observation, why must the learner run on the edge device?

4 comments
AuthorsOpenReview ↗

Response to Reviewer GiCs (Part 1/3: Questions 1–3)

We sincerely thank you for the detailed and constructive feedback. We are glad that you found our study of streaming RL for online VLA adaptation practical and broadly evaluated across models and benchmarks, recognized SIE's clear gains over naive streaming RL, and appreciated our transparent reporting and code supplement. Below, we show new results and try to address each of your questions in detail.

Question 1: How were ηQ and the learning rates chosen for each method?

Answer 1:

The IU paper (Sharifnassab et al., 2026) recommends ηQ=0.50\eta_Q = 0.50 for most tasks, which we retain for the DSRL-IU baseline. For SIE, we swept ηQ∈{0.10,0.25,0.50}\eta_Q \in \{0.10, 0.25, 0.50\} with EPU on our π0\pi_0 tasks and selected ηQ=0.10\eta_Q = 0.10. All methods share the learning rates for the actor (10−410^{-4}), critic (3×10−43 \times 10^{-4}), and temperature (3×10−43 \times 10^{-4}), which directly follow the default SAC settings in DSRL (Wagenmaker et al., 2025).

Question 2: Please report SIE w/o EPU as the main streaming baseline in all four settings, with per-seed intervals.

Answer 2:

Below we compare SIE w/o EPU (matched at ηQ=0.10\eta_Q = 0.10) with SIE and DSRL-Replay across all four experimental settings:

SettingMethodSeed 0Seed 1Seed 2Mean ± Std
π₀–LIBERODSRL-Replay78.9393.6081.0784.53 ± 7.92
π₀–LIBEROSIE w/o EPU59.8770.5368.4066.27 ± 5.64
π₀–LIBEROSIE68.4074.4073.4772.09 ± 3.23
π₀.₅–LIBERODSRL-Replay77.6080.4080.6079.53 ± 1.68
π₀.₅–LIBEROSIE w/o EPU73.0070.0065.2069.40 ± 3.93
π₀.₅–LIBEROSIE77.4073.0071.4073.93 ± 3.11
π₀.₅–Meta-WorldDSRL-Replay83.6088.4083.6085.20 ± 2.77
π₀.₅–Meta-WorldSIE w/o EPU54.4063.2062.0059.87 ± 4.77
π₀.₅–Meta-WorldSIE65.2056.0068.0063.07 ± 6.28
π₀–ALOHADSRL-Replay96.0078.0094.0089.33 ± 9.87
π₀–ALOHASIE w/o EPU58.0064.0068.0063.33 ± 5.03
π₀–ALOHASIE66.0064.0068.0066.00 ± 2.00

All values are final-50 success rates (%). For each training seed, success rates are averaged equally over the tasks in the corresponding setting. We report the mean ± sample standard deviation across the three training seeds. Across all four settings, EPU consistently improves over the matched SIE w/o EPU baseline (+5.82, +4.53, +3.20, and +2.67 percentage points, respectively). On ALOHA we conduct a rerun of all four methods, with the run of SIE w/o EPU. Furthermore, in a parameter sensitivity analysis in the ALOHA task, we found that simply setting the ηE\eta_E from 0.05 to 0.025 could gain a 8.7% improvement in success rate (Appendix A.6).

We have also incorporated these hyperparameter selection details (Appendix B), the full four-setting SIE w/o EPU and DSRL-Replay comparison table (Appendix A.5), the main-text figure (Figure 3) and the ALOHA sensitivity analysis (Appendix A.6) into the revised PDF manuscript. We will soon update the corresponding ablation code, per-seed evaluation data, and environment specifications in the supplementary material.

Question 3: What rule chose the LIBERO-90 subsets, and how many tasks were run in total?

Answer 3:

When selecting the LIBERO-90 task subsets, we first evaluated the zero-shot success rate of the frozen checkpoint (See Appendix A.1 for details) across all 90 tasks and then selected tasks with intermediate initial success rates—specifically, approximately 10%–60% for π0\pi_0 and 5%–70% for π0.5\pi_{0.5}—yielding the 15 tasks for π0\pi_0 and 10 tasks for π0.5\pi_{0.5} (25 evaluated task configurations in total). As mentioned in Section 5.2, this avoids tasks with near-zero initial success rates, where sparse-reward online RL receives almost no learning signal, as well as already saturated tasks, where little room remains for online improvement.

We have clarified this selection rule in Appendix A.1 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer GiCs (Part 2/3: Question 4)

Question 4: What are DSRL-Replay's final-50 means in each setting, and how long (wall clock and environment steps) does each method, including a budget-matched replay learner, take to reach, say, 50% and 80% success?

Answer 4:

The per-seed and mean final-50 values for DSRL-Replay across all four settings are reported in the summary table under Question 2 above (84.53%, 79.53%, 85.20%, and 89.33%).

Here we report the 50% / 80% success rate's attainment point in thousands of environment steps (Tables 1–2) or wall-clock minutes (Tables 3–4), where attainment is defined as the second of two consecutive evaluations meeting the threshold based on the three-seed mean success rate, for π0\pi_0 (15 tasks) and π0.5\pi_{0.5} (10 tasks) on LIBERO-90. NA indicates that the threshold was not reached within the 500k-step training budget.

Table 1. Environment Steps (×10³): π₀–LIBERO-90

TaskDSRL-ReplayDSRL-StreamDSRL-IUSIE
Task 6110 / 160NA / NANA / NA500 / NA
Task 1050 / 50NA / NA190 / NA350 / NA
Task 2060 / 70240 / 290120 / 370150 / 220
Task 2950 / 210NA / NANA / NA160 / 490
Task 4390 / 240NA / NANA / NA400 / NA
Task 2210 / 6010 / 35040 / NA10 / 250
Task 4980 / 230NA / NANA / NA320 / NA
Task 5550 / 5040 / NA90 / NA190 / NA
Task 5730 / 30190 / 230100 / 120110 / 240
Task 6130 / 30NA / NA200 / NA220 / NA
Task 6770 / 210NA / NA120 / NA140 / NA
Task 4440 / 50200 / 290100 / 400150 / 200
Task 7250 / 120NA / NANA / NANA / NA
Task 4640 / 70180 / NA150 / NA270 / NA
Task 2830 / 4020 / 11040 / 6020 / 170

Table 2. Environment Steps (×10³): π₀.₅–LIBERO-90

TaskDSRL-ReplayDSRL-StreamDSRL-IUSIE
Task 230 / 40130 / 29030 / 22040 / 260
Task 640 / 50180 / NA240 / 300160 / 240
Task 14180 / NANA / NANA / NANA / NA
Task 1550 / 210NA / NANA / NA460 / NA
Task 3010 / 6010 / 25050 / 12010 / 170
Task 4130 / 100170 / 320140 / NA240 / NA
Task 5010 / 8010 / 15010 / 17010 / 240
Task 7710 / 6010 / 25010 / NA10 / NA
Task 7940 / 80220 / 390130 / NA170 / 210
Task 8260 / 110320 / NANA / NA270 / 390

Table 3. Wall-Clock Time (min): π₀–LIBERO-90

TaskDSRL-ReplayDSRL-StreamDSRL-IUSIE
Task 6181.1 / 272.5NA / NANA / NA337.0 / NA
Task 1075.4 / 75.4NA / NA185.2 / NA245.5 / NA
Task 2098.1 / 117.2137.0 / 177.0115.9 / 387.996.7 / 152.2
Task 2974.8 / 378.0NA / NANA / NA111.9 / 372.4
Task 43146.8 / 428.4NA / NANA / NA266.7 / NA
Task 226.2 / 106.46.4 / 239.626.3 / NA6.5 / 174.3
Task 49130.2 / 399.3NA / NANA / NA207.5 / NA
Task 5573.7 / 73.722.6 / NA92.4 / NA128.1 / NA
Task 5741.2 / 41.2106.6 / 133.3116.3 / 138.172.4 / 164.4
Task 6141.4 / 41.4NA / NA213.4 / NA153.9 / NA
Task 67114.4 / 378.5NA / NA144.8 / NA101.2 / NA
Task 4464.2 / 86.9119.0 / 187.262.5 / 294.292.2 / 129.4
Task 7273.7 / 208.3NA / NANA / NANA / NA
Task 4655.5 / 114.9103.3 / NA145.6 / NA177.3 / NA
Task 2864.5 / 96.520.7 / 113.927.7 / 43.613.7 / 123.3

Table 4. Wall-Clock Time (min): π₀.₅–LIBERO-90

TaskDSRL-ReplayDSRL-StreamDSRL-IUSIE
Task 252.8 / 76.189.7 / 212.820.6 / 160.727.4 / 200.1
Task 670.4 / 93.0115.9 / NA157.3 / 199.3114.0 / 171.7
Task 14444.7 / NANA / NANA / NANA / NA
Task 1595.5 / 520.7NA / NANA / NA375.5 / NA
Task 307.2 / 117.87.0 / 185.035.4 / 86.97.1 / 132.4
Task 4147.7 / 197.0107.6 / 211.387.9 / NA161.7 / NA
Task 508.4 / 169.58.3 / 124.88.4 / 141.08.4 / 216.2
Task 776.2 / 112.06.1 / 154.75.9 / NA6.1 / NA
Task 7966.5 / 151.9128.1 / 237.273.9 / NA97.6 / 123.2
Task 82109.9 / 218.0199.7 / NANA / NA186.1 / 267.6

Please note that these runs were executed on A100 servers where sequential LIBERO simulation and frozen VLA inference account for the majority of total runtime; thus, longer attainment times reflect streaming RL's higher sample requirement rather than low update efficiency of SIE (per 10k environment steps, SIE still trains roughly 2 to 2.5$\times$ faster than DSRL-Replay even in these A100 runs).

AuthorsOpenReview ↗

Response to Reviewer GiCs (Part 3/3: Question 5)

Question 5: Since the server already receives every observation, why must the learner run on the edge device?

Answer 5:

Regarding why the learner runs on the edge device in Section 5.3, there are two main reasons:

  1. Placing the learner on an 8 GB laptop GPU allows us to isolate and measure the RL adaptation overhead on resource-constrained local hardware without confounding it with simulator rendering or unquantized π0\pi_0 inference on the A100 server. Notably, DSRL (Wagenmaker et al., 2025, Appendix C.5) also adopted this exact split setup—running frozen π0\pi_0 inference on a remote policy server and DSRL training locally on an 8 GB GPU (RTX 3070)—to demonstrate the compute and memory efficiency of its RL adaptation.
  2. Keeping the learner on the edge allows the central VLA server to remain stateless ((st,zt)↦at(s_t, z_t) \mapsto \mathbf{a}_t) and serve multiple robots concurrently via batched forward passes, whereas any online RL learner—whether replay-based or streaming—is inherently stateful and robot-specific, maintaining mutable actor–critic weights, optimizer/trace states, and task-termination logic. For example, in privacy-sensitive multi-robot warehouse or household deployments, a shared server only performs ephemeral inference on incoming frames and discards them immediately, while each robot samples its own latent actions prior to querying the VLA and updates its private policy locally from on-board task feedback.

Furthermore, even when both the frozen π0\pi_0 backbone and the SAC learner run on the same local GPU, SIE still retains clear latency and memory advantages while maintaining strong task performance: in our new physical-robot experiment on a Piper X arm using a single RTX 5070 Ti GPU, SIE improves success rate from 30% to 65% within 20 online episodes (whereas DSRL-Stream collapses to 0%), updates in real time between action chunks in 40.7 ms with virtually no extra motion delay (compared with a 33.56 s post-episode pause for DSRL-Replay), and uses less than half the learner GPU memory (2.84 GiB vs. 6.08 GiB). We have updated this real-world experiment in Appendix D of the revised PDF manuscript.

Dear Reviewer GiCs,

As the discussion period is ending soon, we would like to gently check if our rebuttal addressed your concerns. Thank you for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.