- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes Dynamic Episode Alignment (DEAL) to improve the sample efficiency of online reinforcement learning for vision-language-action models using terminal outcome rewards. DEAL aligns each episode with the shortest successful reference available for the same scene through dynamic time warping on the policy's own representations. The alignment labels steps as productive or wasted, and RL updates retain only productive steps from successful episodes and wasted steps from failed episodes. The method is applied to GRPO and PPO, with the PPO implementation recomputing GAE over the selected steps. Additional contributions include a recurrent Gaussian action head that provides exact continuous-action likelihoods while reusing VLM features and incorporating fresh proprioception, and a simplified tabular analysis of how step selection affects update directions and the episode count needed to reliably favor productive actions.
Validation is conducted entirely in simulation on LIBERO, RoboTwin 2.0, and the Ego Humanoid Manipulation Benchmark. Policies are initialized through behavior cloning and primarily fine-tuned using PPO with DEAL. The evaluation includes task success comparisons, estimates of training episode requirements, controlled PPO and GRPO comparisons with and without DEAL from identical starting policies on three RoboTwin tasks, numerical experiments in a tabular environment, and action-generation latency comparisons with the native π0.5 flow-matching head.
Critical strengths
S1. A well-motivated approach to credit assignment for GRPO.
In the outcome-based GRPO setting considered in the paper, all actions in an episode share the same advantage, which can reinforce unnecessary steps in successful trajectories and penalize useful progress in failed trajectories. DEAL directly targets this limitation by retaining steps labeled productive from successes and steps labeled wasted from failures. When the alignment labels are reliable, this provides a coherent mechanism for reducing conflicting credit without introducing a learned critic. The selection uses the policy's own representations and successful trajectories, avoiding an additional scoring model or manually annotated subgoals. This is a meaningful conceptual contribution because it enables finer-grained selection while retaining the terminal outcome as the only reward.
Critical weaknesses
W1. The experiments do not substantiate the central claims about sample efficiency and the proposed design choices.
The evaluation does not adequately distinguish the contribution of DEAL from changes in architecture, initialization, and training. The main SimpleVLA-RL comparison uses a different backbone and action representation, while several other baselines use imitation learning without online RL. These comparisons cannot isolate the advantage attributable to DEAL. The new Gaussian head is compared with the native head for inference latency, but not task performance or RL learning efficiency under matched conditions. Relevant RL alternatives that retain flow-based policies are also missing from the experimental comparisons. Figure 4 provides a controlled comparison, but its VLA learning curves report averaged best-so-far success on only three RoboTwin tasks. This provides limited support for the broader sample-efficiency claim without comprehensive success-rate curves against interaction budgets across benchmarks and variability across independent training runs. Moreover, the PPO ablation changes step selection and GAE computation on a compressed timeline together, leaving the source of improvement unresolved. Consequently, neither the advantage of replacing the pretrained head nor the claimed general reduction in interaction cost is convincingly established.
W2. The theoretical analysis provides limited justification for the practical algorithm.
The theory analyzes updates whose advantage signs are determined by episode outcomes and explicitly excludes learned critics, so it does not establish an advantage over credit assignment. PPO may already assign negative advantages to wasted actions in successful episodes or positive advantages to productive actions in failed episodes. DEAL excludes both categories, potentially discarding useful updates rather than merely removing conflicting credit. Furthermore, the method treats a lack of forward progress in reference alignment as evidence that an action is wasted, but this does not establish that the action is unhelpful. Necessary adjustments, recovery actions, or alternative valid subgoal orders may not advance the alignment and could therefore be mislabeled. Recomputing GAE on the selected sequence introduces another substantive change by discounting once between retained steps, regardless of the intervening environment steps. This changes the discounted return being estimated and the critic targets, rather than simply reducing noise in the original advantage estimator. The analysis does not address these potential errors in action selection or justify the altered temporal discounting. Consequently, it offers limited support for the claim that the practical PPO-based algorithm improves credit assignment and sample efficiency over standard PPO.
Questions for authors
Q1. Are per-task success curves against collected episodes or environment steps available across the evaluated benchmarks, including comparisons with PPO/GRPO baselines starting from identical policies?
Q2. Are task-performance and learning-efficiency comparisons available between the new Gaussian head and the pretrained flow head with an existing flow-based RL method?
Q3. What evidence isolates the benefit of DEAL's step selection over standard PPO with GAE?
Q4. What evidence supports interpreting a lack of forward reference alignment as wasted action, particularly for necessary adjustments, recovery actions, or alternative valid execution paths?
6 comments
Part (2/3)
3. Does the gain come from step selection, or from computing GAE over the selected steps? What justifies the changed discounting?
We thank the reviewer for this question. The gain comes from step selection. To show it, we added a third arm to the LIBERO-Long comparison above. The new arm computes GAE over the full episode exactly as standard PPO does, and the alignment then only selects which steps enter the update.
| RL episodes | 0 | 512 | 1,024 | 1,536 | 2,048 |
|---|---|---|---|---|---|
| PPO | 32.4 | ||||
| PPO + DEAL, GAE over the full episode (new arm) | 32.4 | ||||
| PPO + DEAL, GAE over the selected steps | 32.4 |
Step selection alone, with GAE left exactly as in standard PPO, gives the whole gain of about 19 points over PPO. Computing GAE over the selected steps does not add to it. So the improvement comes from which steps are updated, not from the changed discounting. The new arm is reported in Section 4.4, and per task in Table A3.
On the discounting, we compute GAE over the selected steps for a simple reason: in a success, a productive action should not get less credit because the policy wasted steps after it. As the new arm shows, the gain does not depend on this choice, and we now say this in Section 3.3. We thank the reviewer for this suggestion.
4. PPO's critic already gives per-step credit. What does DEAL add to it, and does it discard useful updates?
We thank the reviewer for this point. Conflicting credit is there with or without a critic, because the critic also learns only from the final reward of the same episodes. A critic therefore needs many episodes before it can tell a wasted step from a productive one, while the alignment labels them directly. As shown in question 3 above, adding only the alignment's step selection to PPO improves it by about 19 points.
Our theory (Proposition 1) shows the benefit of step selection for the case without a critic, and it is meant as a motivation. A learned critic is hard to include in such a proof. Its advantages are biased by the critic's own errors [1], and in practice learned critics often fail to fit the true value function [2].
To further address the reviewer's concern, we ran a new experiment that directly compares how well PPO's learned critic and the alignment find wasted steps. We inserted wasted steps into LIBERO-Long demonstrations and into the policy's own episodes. Since we add these steps ourselves, we know exactly which steps are wasted. The inserted steps imitate the waste we see in the policy's own episodes, in four kinds: a pause, a short detour, a missed grasp before the real one, and a rewind, where a carry is undone and repeated. We then measured how many of them PPO's advantage pushes down and how many our alignment labels as wasted (Section 4.4, and Table A4 in Appendix E).
| Inserted waste found (%) | PPO's advantage | Alignment |
|---|---|---|
| Successful demonstrations, pauses and detours | 37.5 | 98.1 |
| Successful demonstrations, missed grasps and rewinds | 47.1 | 100.0 |
| Failed demonstrations, pauses and detours | 39.1 | 97.8 |
| Failed demonstrations, missed grasps and rewinds | 47.3 | 99.0 |
| The policy's own successes | 58.9 | 91.7 |
| The policy's own failures | 54.8 | 91.1 |
PPO's advantage pushes down 38 to 59% of the inserted waste, but also 28 to 38% of the demonstrator's own productive steps, so it barely tells the two apart. Our alignment finds 98 to 100% of the waste inserted into the LIBERO-Long demonstrations and over 91% of the waste inserted into the policy's own episodes, and labels only 3.2 to 9.0% of the demonstrator's own steps as wasted. The supplementary material has videos of the demonstration episodes with the alignment's labels and PPO's advantages.
The new arm in question 3 tests directly whether useful updates are discarded. This arm keeps PPO's own advantages and only removes the steps that the alignment leaves out, and this alone gains about 19 points over PPO. Removing these steps therefore helps, and the policy still learns from productive steps in successes and from wasted steps in failures. Together, these results show that the alignment gives PPO step-level information that its critic does not provide, and that leaving these steps out helps PPO.
References
[1] Schulman et al. High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016.
[2] Ilyas et al. A Closer Look at Deep Policy Gradients. ICLR 2020.
Part (3/3)
5. Is a step that does not advance the alignment really wasted? What about necessary adjustments, recovery actions and other valid paths?
We thank the reviewer for this question. The reference is a success in the same scene, so advancing along it means advancing the task. A step that does not advance the task, such as a pause or a retry that fails, has no later part of the reference to match and is labeled wasted. A retry that succeeds, a recovery or a necessary adjustment does advance the task. Such a step is matched with the part of the reference that completes the same part of the task, and is labeled productive. The test with inserted waste in question 4 supports this: our alignment finds almost all of the inserted waste and rarely labels the demonstrator's own steps as wasted (Table A4).
As we state in Section 3.2 and in the Limitations (Appendix G), the alignment assumes that an episode and its reference complete the subgoals in the same order. When a task allows another valid order, productive steps can be labeled wasted, and we consider an alignment that does not depend on the order of subgoals as future work.
6. How does the recurrent Gaussian action head compare with 's flow-matching head, in task success and in RL sample efficiency?
We thank the reviewer for this question. We trained the two heads under the same conditions: they reach the same success, and ours trains 2.8$\times$ faster. In RL, the flow-based [1] needs about 9.5$\times$ more episodes than ours for a similar success on LIBERO-Long.
For task success, we trained the two heads under the same conditions: the same VLM, random initial weights, the same 40 demonstrations (one per LIBERO task) and the same batch size. The table gives the average success (%) over the four LIBERO suites (Figure A2 in Appendix F). The flow-matching head is below the 77.1% reported in [1] because [1] starts from 's pretrained head, while here both heads start from random weights.
| Training steps | 500 | 1,000 | 2,000 | 3,000 | 4,000 | Time per step (4 GPUs) |
|---|---|---|---|---|---|---|
| Ours | 41.4 | 56.2 | 65.5 | 64.9 | 62.1 | 4.05 s |
| Flow-matching | 0.7 | 23.9 | 54.1 | 61.4 | 62.4 | 11.38 s |
Our head reaches 55% success in 4.3 GPU-hours against 26.6 for the flow-matching head, and also generates actions about 1.4$\times$ faster (Section 4.4).
For RL, flow-based methods inject noise into the denoising steps to get one likelihood for a whole action chunk [1], while our step selection needs the exact likelihood of each action, which the Gaussian head gives. The table compares the published results of [1] with ours on LIBERO-Long with one demonstration per task (Table A5 in Appendix F).
| Before RL | After RL | RL episodes | |
|---|---|---|---|
| (flow-matching head) | 43.9 | 94.0 | 256,000 |
| Ours | 32.4 | 92.6 | 26,880 |
Our policy starts lower and reaches a similar success with about 9.5$\times$ fewer RL episodes. This matters because every RL episode is a complete task attempt by the robot, and the low sample efficiency of online RL is the reason such methods rely on training in simulation [1].
The reviewer's suggestions have made the paper stronger, and we are grateful for them. We hope these answers and the new results address the reviewer's concerns, and we are happy to clarify anything further. In light of the new results, we would greatly appreciate it if the reviewer would reconsider the rating.
References
[1] Chen et al. : Online RL Fine-tuning for Flow-based Vision-Language-Action Models. arXiv:2510.25889, 2025.
Part (1/3)
We thank the reviewer for the careful review and the clear questions. We answer each one below.
1. Sim-to-real evaluation: the gain from RL in simulation transfers to a real robot
Before answering the questions, we are happy to share that we have run sim-to-real experiments similar to those of SimpleVLA-RL [1]: the policies are trained in simulation only and then run on a real Franka arm. The preliminary results on three tasks are in, and we added them to the paper (Section 4.3 and Appendix D, with a video in the supplementary material). We compare the imitation-only policy with the policy after RL with DEAL, with 10 trials per task for each policy. The trials vary the object positions, the objects and the surroundings, and both policies are tested under the same conditions.
The policy after RL with DEAL succeeds in 19 of 30 real trials, against 4 of 30 for imitation only, so the improvement from RL in simulation carries over to the real robot. We are extending this evaluation to more tasks and more trials, and will include them in the camera-ready version.
2. Is the gain from DEAL, and not from the architecture, the initialization or the training? Are there per-task curves from the same starting policy, with the variation across runs?
We would first like to clarify how the contributions of the paper relate to its results. The method we propose has two parts that work together, and both are contributions of the paper. The episode alignment finds the productive and wasted steps of each episode, so that RL updates only the selected steps. The recurrent Gaussian action head makes this possible: the head gives the exact likelihood of each continuous action at every step, without discretizing it. All results in Tables 1 to 4 use both. On RoboTwin 2.0, our success is a relative 21% higher than SimpleVLA-RL, with an estimated 5.6$\times$ fewer RL episodes. On the Ego Humanoid Manipulation Benchmark, our success is a relative 36% higher than EgoVLA. On LIBERO with one demonstration per task, we reach a similar success with at least 8.7$\times$ fewer RL episodes than πRL [2] on every suite, and with an estimated 4$\times$ fewer than SimpleVLA-RL on LIBERO-Long (the only suite with a published learning curve in SimpleVLA-RL).
Since the action head is part of what the paper proposes, we respectfully believe that these results should be read as the gain of our method. The backbone does not explain them either: on the Ego benchmark we use EgoVLA's own VLM, and on RoboTwin 2.0 our policy starts close to SimpleVLA-RL before RL (40.3% vs 38.6%), so the lead comes with RL.
To study the effect of each contribution and further address the reviewer's concerns, we conducted more experiments. The first one shows what the alignment adds alone: it keeps the starting policy, backbone, head and RL algorithm the same and only adds the alignment, as in the comparison the reviewer points to in Figure 4. As the reviewer suggested, we ran it on a second benchmark, LIBERO-Long, with PPO and PPO + DEAL from the same starting policy (32.4%) and three seeds each. The table gives the success (%) at each evaluation, as the mean and standard deviation over the three seeds, and Figure 4(d) in the main text shows the curves in the same form as panels (b, c).
PPO + DEAL is ahead at every evaluation, and at the end it is ahead on every seed, by 16 to 19 points. To address the per-task results, we added them for both comparisons in Appendix C (Tables A2 and A3).
Together, these results show that the alignment alone gives a clear gain from the same starting policy, on two benchmarks and across seeds and tasks. Questions 3 to 5 give more results on the alignment, and question 6 compares the head with π0.5's flow-matching head.
References
[1] Li et al. SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning. ICLR 2026.
[2] Chen et al. πRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models. arXiv:2510.25889, 2025.