acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Episode Alignment: Sample-Efficient Reinforcement Learning of Vision-Language-Action Models

Abstract

Online reinforcement learning (RL) can improve vision-language-action (VLA) models beyond imitation learning, but many methods rely on a final success-or-failure reward that does not reveal which actions helped. RL can therefore reinforce hesitations in successes and discourage productive actions in failures, requiring many task attempts to resolve this conflicting credit. We propose Dynamic Episode Alignment (DEAL), which uses the policy's own successes to identify productive and wasted steps without an extra scoring model. DEAL uses dynamic programming (DP) to align each new episode with the policy's shortest success in the same scene, matching steps by the policy's own representations. RL then updates only productive steps in successes and wasted steps in failures, so successes no longer reinforce hesitations and failures no longer discourage productive actions. In a tabular model under simplifying assumptions, we prove that selecting steps reduces the episodes needed to reliably reinforce productive actions from linear to constant in the number of subgoals. To support DEAL's updates, we introduce a recurrent Gaussian action head that provides exact action likelihoods. Combining DEAL with PPO and the recurrent Gaussian action head yields 97.6% success on LIBERO with one demonstration per task and 80.4% on RoboTwin 2.0, versus SimpleVLA-RL's 96.9% and 66.4%. From the same starting policies on three RoboTwin 2.0 tasks, DEAL improves PPO's sample efficiency and raises mean success by a relative 26% after equal training. On a humanoid with two dexterous hands, online RL with DEAL exceeds EgoVLA's success by a relative 46.8% on unseen backgrounds.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Reinforcement Learning, Vision Language Action models, Robotics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8jkMOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes Dynamic Episode Alignment (DEAL) to improve the sample efficiency of online reinforcement learning for vision-language-action models using terminal outcome rewards. DEAL aligns each episode with the shortest successful reference available for the same scene through dynamic time warping on the policy's own representations. The alignment labels steps as productive or wasted, and RL updates retain only productive steps from successful episodes and wasted steps from failed episodes. The method is applied to GRPO and PPO, with the PPO implementation recomputing GAE over the selected steps. Additional contributions include a recurrent Gaussian action head that provides exact continuous-action likelihoods while reusing VLM features and incorporating fresh proprioception, and a simplified tabular analysis of how step selection affects update directions and the episode count needed to reliably favor productive actions.

Validation is conducted entirely in simulation on LIBERO, RoboTwin 2.0, and the Ego Humanoid Manipulation Benchmark. Policies are initialized through behavior cloning and primarily fine-tuned using PPO with DEAL. The evaluation includes task success comparisons, estimates of training episode requirements, controlled PPO and GRPO comparisons with and without DEAL from identical starting policies on three RoboTwin tasks, numerical experiments in a tabular environment, and action-generation latency comparisons with the native π0.5 flow-matching head.

Critical strengths

S1. A well-motivated approach to credit assignment for GRPO.

In the outcome-based GRPO setting considered in the paper, all actions in an episode share the same advantage, which can reinforce unnecessary steps in successful trajectories and penalize useful progress in failed trajectories. DEAL directly targets this limitation by retaining steps labeled productive from successes and steps labeled wasted from failures. When the alignment labels are reliable, this provides a coherent mechanism for reducing conflicting credit without introducing a learned critic. The selection uses the policy's own representations and successful trajectories, avoiding an additional scoring model or manually annotated subgoals. This is a meaningful conceptual contribution because it enables finer-grained selection while retaining the terminal outcome as the only reward.

Critical weaknesses

W1. The experiments do not substantiate the central claims about sample efficiency and the proposed design choices.

The evaluation does not adequately distinguish the contribution of DEAL from changes in architecture, initialization, and training. The main SimpleVLA-RL comparison uses a different backbone and action representation, while several other baselines use imitation learning without online RL. These comparisons cannot isolate the advantage attributable to DEAL. The new Gaussian head is compared with the native π0.5\pi_{0.5} head for inference latency, but not task performance or RL learning efficiency under matched conditions. Relevant RL alternatives that retain flow-based policies are also missing from the experimental comparisons. Figure 4 provides a controlled comparison, but its VLA learning curves report averaged best-so-far success on only three RoboTwin tasks. This provides limited support for the broader sample-efficiency claim without comprehensive success-rate curves against interaction budgets across benchmarks and variability across independent training runs. Moreover, the PPO ablation changes step selection and GAE computation on a compressed timeline together, leaving the source of improvement unresolved. Consequently, neither the advantage of replacing the pretrained head nor the claimed general reduction in interaction cost is convincingly established.

W2. The theoretical analysis provides limited justification for the practical algorithm.

The theory analyzes updates whose advantage signs are determined by episode outcomes and explicitly excludes learned critics, so it does not establish an advantage over credit assignment. PPO may already assign negative advantages to wasted actions in successful episodes or positive advantages to productive actions in failed episodes. DEAL excludes both categories, potentially discarding useful updates rather than merely removing conflicting credit. Furthermore, the method treats a lack of forward progress in reference alignment as evidence that an action is wasted, but this does not establish that the action is unhelpful. Necessary adjustments, recovery actions, or alternative valid subgoal orders may not advance the alignment and could therefore be mislabeled. Recomputing GAE on the selected sequence introduces another substantive change by discounting once between retained steps, regardless of the intervening environment steps. This changes the discounted return being estimated and the critic targets, rather than simply reducing noise in the original advantage estimator. The analysis does not address these potential errors in action selection or justify the altered temporal discounting. Consequently, it offers limited support for the claim that the practical PPO-based algorithm improves credit assignment and sample efficiency over standard PPO.

Questions for authors

Q1. Are per-task success curves against collected episodes or environment steps available across the evaluated benchmarks, including comparisons with PPO/GRPO baselines starting from identical policies?

Q2. Are task-performance and learning-efficiency comparisons available between the new Gaussian head and the pretrained π0.5\pi_{0.5} flow head with an existing flow-based RL method?

Q3. What evidence isolates the benefit of DEAL's step selection over standard PPO with GAE?

Q4. What evidence supports interpreting a lack of forward reference alignment as wasted action, particularly for necessary adjustments, recovery actions, or alternative valid execution paths?

6 comments

Part (1/3)

We thank the reviewer for the careful review and the clear questions. We answer each one below.

1. Sim-to-real evaluation: the gain from RL in simulation transfers to a real robot

Before answering the questions, we are happy to share that we have run sim-to-real experiments similar to those of SimpleVLA-RL [1]: the policies are trained in simulation only and then run on a real Franka arm. The preliminary results on three tasks are in, and we added them to the paper (Section 4.3 and Appendix D, with a video in the supplementary material). We compare the imitation-only policy with the policy after RL with DEAL, with 10 trials per task for each policy. The trials vary the object positions, the objects and the surroundings, and both policies are tested under the same conditions.

Successes on the real robotStack two bowlsPress staplerPlace the cupAll
Imitation only0/102/102/104/30
After RL with DEAL7/107/105/1019/30

The policy after RL with DEAL succeeds in 19 of 30 real trials, against 4 of 30 for imitation only, so the improvement from RL in simulation carries over to the real robot. We are extending this evaluation to more tasks and more trials, and will include them in the camera-ready version.

2. Is the gain from DEAL, and not from the architecture, the initialization or the training? Are there per-task curves from the same starting policy, with the variation across runs?

We would first like to clarify how the contributions of the paper relate to its results. The method we propose has two parts that work together, and both are contributions of the paper. The episode alignment finds the productive and wasted steps of each episode, so that RL updates only the selected steps. The recurrent Gaussian action head makes this possible: the head gives the exact likelihood of each continuous action at every step, without discretizing it. All results in Tables 1 to 4 use both. On RoboTwin 2.0, our success is a relative 21% higher than SimpleVLA-RL, with an estimated 5.6$\times$ fewer RL episodes. On the Ego Humanoid Manipulation Benchmark, our success is a relative 36% higher than EgoVLA. On LIBERO with one demonstration per task, we reach a similar success with at least 8.7$\times$ fewer RL episodes than πRL\pi_{\mathrm{RL}} [2] on every suite, and with an estimated 4$\times$ fewer than SimpleVLA-RL on LIBERO-Long (the only suite with a published learning curve in SimpleVLA-RL).

Since the action head is part of what the paper proposes, we respectfully believe that these results should be read as the gain of our method. The backbone does not explain them either: on the Ego benchmark we use EgoVLA's own VLM, and on RoboTwin 2.0 our policy starts close to SimpleVLA-RL before RL (40.3% vs 38.6%), so the lead comes with RL.

To study the effect of each contribution and further address the reviewer's concerns, we conducted more experiments. The first one shows what the alignment adds alone: it keeps the starting policy, backbone, head and RL algorithm the same and only adds the alignment, as in the comparison the reviewer points to in Figure 4. As the reviewer suggested, we ran it on a second benchmark, LIBERO-Long, with PPO and PPO + DEAL from the same starting policy (32.4%) and three seeds each. The table gives the success (%) at each evaluation, as the mean and standard deviation over the three seeds, and Figure 4(d) in the main text shows the curves in the same form as panels (b, c).

RL episodes05121,0241,5362,048
PPO32.435.0±5.635.0 \pm 5.639.5±1.039.5 \pm 1.037.0±1.537.0 \pm 1.540.8±2.640.8 \pm 2.6
PPO + DEAL32.443.0±1.743.0 \pm 1.747.2±6.047.2 \pm 6.054.3±1.654.3 \pm 1.658.2±1.758.2 \pm 1.7

PPO + DEAL is ahead at every evaluation, and at the end it is ahead on every seed, by 16 to 19 points. To address the per-task results, we added them for both comparisons in Appendix C (Tables A2 and A3).

Together, these results show that the alignment alone gives a clear gain from the same starting policy, on two benchmarks and across seeds and tasks. Questions 3 to 5 give more results on the alignment, and question 6 compares the head with π0.5\pi_{0.5}'s flow-matching head.

References

[1] Li et al. SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning. ICLR 2026.

[2] Chen et al. πRL\pi_{\mathrm{RL}}: Online RL Fine-tuning for Flow-based Vision-Language-Action Models. arXiv:2510.25889, 2025.

Part (2/3)

3. Does the gain come from step selection, or from computing GAE over the selected steps? What justifies the changed discounting?

We thank the reviewer for this question. The gain comes from step selection. To show it, we added a third arm to the LIBERO-Long comparison above. The new arm computes GAE over the full episode exactly as standard PPO does, and the alignment then only selects which steps enter the update.

RL episodes05121,0241,5362,048
PPO32.435.0±5.635.0 \pm 5.639.5±1.039.5 \pm 1.037.0±1.537.0 \pm 1.540.8±2.640.8 \pm 2.6
PPO + DEAL, GAE over the full episode (new arm)32.446.2±2.846.2 \pm 2.850.8±4.650.8 \pm 4.650.7±7.350.7 \pm 7.359.8±6.159.8 \pm 6.1
PPO + DEAL, GAE over the selected steps32.443.0±1.743.0 \pm 1.747.2±6.047.2 \pm 6.054.3±1.654.3 \pm 1.658.2±1.758.2 \pm 1.7

Step selection alone, with GAE left exactly as in standard PPO, gives the whole gain of about 19 points over PPO. Computing GAE over the selected steps does not add to it. So the improvement comes from which steps are updated, not from the changed discounting. The new arm is reported in Section 4.4, and per task in Table A3.

On the discounting, we compute GAE over the selected steps for a simple reason: in a success, a productive action should not get less credit because the policy wasted steps after it. As the new arm shows, the gain does not depend on this choice, and we now say this in Section 3.3. We thank the reviewer for this suggestion.

4. PPO's critic already gives per-step credit. What does DEAL add to it, and does it discard useful updates?

We thank the reviewer for this point. Conflicting credit is there with or without a critic, because the critic also learns only from the final reward of the same episodes. A critic therefore needs many episodes before it can tell a wasted step from a productive one, while the alignment labels them directly. As shown in question 3 above, adding only the alignment's step selection to PPO improves it by about 19 points.

Our theory (Proposition 1) shows the benefit of step selection for the case without a critic, and it is meant as a motivation. A learned critic is hard to include in such a proof. Its advantages are biased by the critic's own errors [1], and in practice learned critics often fail to fit the true value function [2].

To further address the reviewer's concern, we ran a new experiment that directly compares how well PPO's learned critic and the alignment find wasted steps. We inserted wasted steps into LIBERO-Long demonstrations and into the policy's own episodes. Since we add these steps ourselves, we know exactly which steps are wasted. The inserted steps imitate the waste we see in the policy's own episodes, in four kinds: a pause, a short detour, a missed grasp before the real one, and a rewind, where a carry is undone and repeated. We then measured how many of them PPO's advantage pushes down and how many our alignment labels as wasted (Section 4.4, and Table A4 in Appendix E).

Inserted waste found (%)PPO's advantageAlignment
Successful demonstrations, pauses and detours37.598.1
Successful demonstrations, missed grasps and rewinds47.1100.0
Failed demonstrations, pauses and detours39.197.8
Failed demonstrations, missed grasps and rewinds47.399.0
The policy's own successes58.991.7
The policy's own failures54.891.1

PPO's advantage pushes down 38 to 59% of the inserted waste, but also 28 to 38% of the demonstrator's own productive steps, so it barely tells the two apart. Our alignment finds 98 to 100% of the waste inserted into the LIBERO-Long demonstrations and over 91% of the waste inserted into the policy's own episodes, and labels only 3.2 to 9.0% of the demonstrator's own steps as wasted. The supplementary material has videos of the demonstration episodes with the alignment's labels and PPO's advantages.

The new arm in question 3 tests directly whether useful updates are discarded. This arm keeps PPO's own advantages and only removes the steps that the alignment leaves out, and this alone gains about 19 points over PPO. Removing these steps therefore helps, and the policy still learns from productive steps in successes and from wasted steps in failures. Together, these results show that the alignment gives PPO step-level information that its critic does not provide, and that leaving these steps out helps PPO.

References

[1] Schulman et al. High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016.

[2] Ilyas et al. A Closer Look at Deep Policy Gradients. ICLR 2020.

Part (3/3)

5. Is a step that does not advance the alignment really wasted? What about necessary adjustments, recovery actions and other valid paths?

We thank the reviewer for this question. The reference is a success in the same scene, so advancing along it means advancing the task. A step that does not advance the task, such as a pause or a retry that fails, has no later part of the reference to match and is labeled wasted. A retry that succeeds, a recovery or a necessary adjustment does advance the task. Such a step is matched with the part of the reference that completes the same part of the task, and is labeled productive. The test with inserted waste in question 4 supports this: our alignment finds almost all of the inserted waste and rarely labels the demonstrator's own steps as wasted (Table A4).

As we state in Section 3.2 and in the Limitations (Appendix G), the alignment assumes that an episode and its reference complete the subgoals in the same order. When a task allows another valid order, productive steps can be labeled wasted, and we consider an alignment that does not depend on the order of subgoals as future work.

6. How does the recurrent Gaussian action head compare with π0.5\pi_{0.5}'s flow-matching head, in task success and in RL sample efficiency?

We thank the reviewer for this question. We trained the two heads under the same conditions: they reach the same success, and ours trains 2.8$\times$ faster. In RL, the flow-based πRL\pi_{\mathrm{RL}} [1] needs about 9.5$\times$ more episodes than ours for a similar success on LIBERO-Long.

For task success, we trained the two heads under the same conditions: the same π0.5\pi_{0.5} VLM, random initial weights, the same 40 demonstrations (one per LIBERO task) and the same batch size. The table gives the average success (%) over the four LIBERO suites (Figure A2 in Appendix F). The flow-matching head is below the 77.1% reported in [1] because [1] starts from π0.5\pi_{0.5}'s pretrained head, while here both heads start from random weights.

Training steps5001,0002,0003,0004,000Time per step (4 GPUs)
Ours41.456.265.564.962.14.05 s
Flow-matching0.723.954.161.462.411.38 s

Our head reaches 55% success in 4.3 GPU-hours against 26.6 for the flow-matching head, and also generates actions about 1.4$\times$ faster (Section 4.4).

For RL, flow-based methods inject noise into the denoising steps to get one likelihood for a whole action chunk [1], while our step selection needs the exact likelihood of each action, which the Gaussian head gives. The table compares the published results of πRL\pi_{\mathrm{RL}} [1] with ours on LIBERO-Long with one demonstration per task (Table A5 in Appendix F).

Before RLAfter RLRL episodes
πRL\pi_{\mathrm{RL}} (flow-matching head)43.994.0256,000
Ours32.492.626,880

Our policy starts lower and reaches a similar success with about 9.5$\times$ fewer RL episodes. This matters because every RL episode is a complete task attempt by the robot, and the low sample efficiency of online RL is the reason such methods rely on training in simulation [1].

The reviewer's suggestions have made the paper stronger, and we are grateful for them. We hope these answers and the new results address the reviewer's concerns, and we are happy to clarify anything further. In light of the new results, we would greatly appreciate it if the reviewer would reconsider the rating.

References

[1] Chen et al. πRL\pi_{\mathrm{RL}}: Online RL Fine-tuning for Flow-based Vision-Language-Action Models. arXiv:2510.25889, 2025.

Reviewer_ZnMUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DEAL, a VLA RL framework that aligns each rollout with the shortest successful trajectory from the same scene. Specifically, DEAL reinforces only useufl steps in the successful rollouts and penalizes only wasted or bad steps in the failures. Experiments include LIBERO, RoboTwin 2.0 and simulated humanoid manipulation.

Critical strengths
  1. This paper presents a practical approach to credit assignment during RL using the policy's own representations and successful rollouts, without need for additional scoring model and manual designs.

  2. Comprehensive experiments support the effectiveness of the proposed method: on three RoboTwin tasks, DEAL reaches PPO’s and GRPO’s peak performance with approximately 2.5× and 1.7× fewer episodes, respectively.

Critical weaknesses
  1. Step-label reliability is insufficiently validated. Repeated DTW matches do not necessarily indicate wasted actions; necessary waiting, grasp maintenance, or recovery may also be mislabeled. The paper provides qualitative examples but no quantitative assessment of label accuracy.

  2. Alignment assumes a fixed subgoal order. Valid alternative execution orders may cause productive actions to be labeled as wasted, potentially discouraging useful exploration. It would be nice to have discussion on this point in the paper.

Questions for authors
  1. How are training scenes revisited and references initialized on each benchmark? What fraction of rollouts have a usable same-scene reference throughout training?

  2. Can the step labels be quantitatively validated, with sensitivity analyses for reference quality and feature choice?

4 comments

Part (1/2)

We thank the reviewer for the constructive review and for the positive assessment of the approach and its experiments. We answer each point below.

1. Sim-to-real evaluation: the gain from RL in simulation transfers to a real robot

Before answering the questions, we are happy to share that we have run sim-to-real experiments similar to those of SimpleVLA-RL [1]: the policies are trained in simulation only and then run on a real Franka arm. The preliminary results on three tasks are in, and we added them to the paper (Section 4.3 and Appendix D, with a video in the supplementary material). We compare the imitation-only policy with the policy after RL with DEAL, with 10 trials per task for each policy. The trials vary the object positions, the objects and the surroundings, and both policies are tested under the same conditions.

Successes on the real robotStack two bowlsPress staplerPlace the cupAll
Imitation only0/102/102/104/30
After RL with DEAL7/107/105/1019/30

The policy after RL with DEAL succeeds in 19 of 30 real trials, against 4 of 30 for imitation only, so the improvement from RL in simulation carries over to the real robot. We are extending this evaluation to more tasks and more trials, and will include them in the camera-ready version.

2. Can the step labels be validated quantitatively? How sensitive are they to the reference quality and the feature choice?

We thank the reviewer for this suggestion. To address it, we ran a new experiment that measures the label accuracy. We inserted wasted steps (pauses, detours, missed grasps and rewinds) into LIBERO-Long demonstrations and into the policy's own episodes, so we know exactly which steps are wasted (Section 4.4, and Table A4 in Appendix E).

Inserted waste found (%)AlignmentPPO's advantage
Successful demonstrations, pauses and detours98.137.5
Successful demonstrations, missed grasps and rewinds100.047.1
Failed demonstrations, pauses and detours97.839.1
Failed demonstrations, missed grasps and rewinds99.047.3
The policy's own successes91.758.9
The policy's own failures91.154.8

Our alignment finds 98 to 100% of the waste inserted into the LIBERO-Long demonstrations and over 91% of the waste inserted into the policy's own episodes, where PPO's advantage finds only 38 to 59%. Necessary steps are rarely labeled wasted: the demonstrator's own steps include the waiting and the holding of the grasp that the task needs, and our alignment labels only 3.2 to 9.0% of them as wasted. For recovery, the rows with missed grasps and rewinds show that the failed attempt is labeled wasted, while the retry that succeeds is matched with the reference.

The test also covers the reference quality: the labels stay accurate both with a clean demonstration as the reference (first four rows) and with the policy's own shortest success, as in RL (last two rows). On the feature choice, the alignment uses the average of the VLM's output tokens and the action head's embedding of the proprioceptive state (Section 3.2). Matching a step to the right step of the reference needs a representation of what is happening in the scene, such as which object is grasped and where it is, and the VLM's features provide this. The test uses these same features, and its results show that they give accurate labels for step selection. Overall, the labels are accurate for wasted steps and for necessary steps, with the references and the features used in RL.

References

[1] Li et al. SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning. ICLR 2026.

Part (2/2)

3. How are training scenes revisited and references initialized on each benchmark? What fraction of episodes have a usable reference?

We thank the reviewer for this question. On all three benchmarks, RL trains on a fixed set of scenes (a scene is a task with its initial object arrangement), so every scene is revisited across rounds. Each scene keeps its own reference: its first success becomes the reference, and a shorter success later replaces it. We added this to Appendix A, together with the share of episodes that have a reference.

The fraction of episodes with a reference grows during training, as the policy succeeds in more scenes. The table gives this fraction on LIBERO-Long, in training that starts from 32.4% success, with no reference for any scene at the start.

RL rounds1 to 89 to 1617 to 2425 to 32
Episodes with a reference (%)41576673

An episode without a reference is not lost: all of its steps enter the update, as in ordinary RL. A missing reference therefore never hurts, and the alignment helps as soon as the scene has a success. The fraction also depends on the starting success: the higher the success when RL starts, the sooner each scene has a reference. So the alignment applies to a growing share of the episodes during training, and the remaining episodes are trained with ordinary RL.

4. The alignment assumes a fixed subgoal order. Other valid orders may cause productive actions to be labeled wasted and discourage useful exploration.

We thank the reviewer for this point, and we extended the discussion of it in the Limitations (Appendix G). As the paper states in Section 3.2, the alignment assumes that an episode and its reference complete the subgoals in the same order, so when a task allows another order, productive steps can be labeled wasted. The effect on exploration is limited. In a success, a step labeled wasted is only left out of the update and is never discouraged. In a failure, such a step is discouraged no more than in ordinary RL. The reference is also the policy's own shortest success, so when the policy finds a faster way to complete the task, that success becomes the new reference. We consider an alignment that does not depend on the order of subgoals as future work.

Beyond these points and the sim-to-real evaluation, the revision adds three results. The comparison of Figure 4, which the reviewer notes as a strength, now also holds on LIBERO-Long: from the same starting policy and over three seeds, PPO reaches 40.8±2.640.8 \pm 2.6% and PPO + DEAL 58.2±1.758.2 \pm 1.7% (Section 4.4 and Figure 4(d) in the main text). A new arm that computes GAE over the full episode, as in standard PPO, and only adds the step selection reaches 59.8±6.159.8 \pm 6.1%, which verifies that the gain comes from the step selection. We also trained our action head and π0.5\pi_{0.5}'s flow-matching head under the same conditions: the two heads reach the same success, and ours trains 2.8$\times$ faster (Figure A2 in Appendix F). On LIBERO with one demonstration per task, we reach a similar success to πRL\pi_{\mathrm{RL}} [1], which keeps the flow-matching head, with at least 8.7$\times$ fewer RL episodes on every suite (Table A5 in Appendix F).

The reviewer's suggestions have made the paper stronger, and we are grateful for them. We hope these answers and the new results address the reviewer's concerns, and we are happy to clarify anything further. In light of the new results, we would greatly appreciate it if the reviewer would consider raising the score.

References

[1] Chen et al. πRL\pi_{\mathrm{RL}}: Online RL Fine-tuning for Flow-based Vision-Language-Action Models. arXiv:2510.25889, 2025.

We thank the reviewer again for the suggestions, which have made the paper stronger. We would like to kindly ask whether our response and the new results address the reviewer's concerns, and we are happy to clarify anything further. If no concern remains, we would greatly appreciate it if the reviewer would consider raising the score.

Thank you for the careful response to my review, which has resolved my concerns. I remain supporting the acceptance of this paper after reading the author's response and other reviewers' reviews.

Reviewer_NAm9OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DEAL (Dynamic Episode Alignment) a credit-assignment approach for robot policies from Vision-Language-Action models trained with RL. DEAL selects actions for fine-grained attribution: In successful episodes it updates productive steps and in failed episodes "wasted" steps. For each scene it aligns new episodes with the policy's shortest success so far which is considered a reference scene. For the approach a recurrent Gaussian head is used and introduced because flow-matching or token-based heads either generate action-chunks without action likelihoods or bin actions while log-probabilities for each action are needed.

Critical strengths

Aligning robot trajectories with dynamic programming alone seems like a strong contribution with potentially large impact. Addressing credit assignment with sparse rewards for robot policies is a worthwhile direction.

The results are tested in simulation and on a real-data robot dataset. Baseline results on the Ego Humanoid Manipulation Benchmark are taken from existing work without compromise from potential re-training. Deal shows impressive advantages over the presented baseline.

On Libero long the approach needs approximately 4-times fewer episodes to reach competitive performance with SimpleVLA-RL.

The comparisons in Figure 4 show impressive sample efficiency while improving results. The experiments are able to isolate DEAL's contribution and show a satisfactory performance.

Critical weaknesses

The assumptions underlying the theoretical explanations are strong. Reference episodes are assumed to have the same subgoal order as a new episode. This should limit exploration and potentially finding global optimums.

It is unclear how much the Gaussian head alone contributes. [Kim et al. 2025] found that continuous actions improve Libero success by 5% so potentially quite a bit. It may be difficult to use another head with DEAL but with BC it is possible. Therefore, an ablation on the role of the Gaussian head would have been useful to judge its contribution.

Outside of the RoboTwin 2.0 experiments it is hard to attribute and isolate gains to DEAL because of comparison against different base models, action heads and training pipelines.

Questions for authors

Could the authors elaborate on potential failure cases? Some movements may benefit from slower execution. Pulling fragile plates from a dishwasher near the success border where the robot is just not too fast so they fly around and break may not be the global optimum. Taking length of an episode seems like a useful proxy for success but it would be interesting to hear thoughts about conceptual limitations from the authors regarding tasks where the optimum lies in a slower execution even though a faster execution could be considered binary success. (Minor point in terms of rating)

The video examples are very impressive but how are those "wasted" steps annotated. Are they attributed through the approach or are they human-labeled in hindsight and the videos only serve to show how the behavior was improved?

It could be a bit clearer that Ego benchmark refers to the Ego Humanoid Manipulation Benchmark.

Is it possible to know how accurate the steps were labeled? Could there be another way for specific tasks to find wasted steps automatically and then calculate how confident these can be found with the approach?

4 comments

Part (1/3)

We thank the reviewer for the encouraging review and the thoughtful questions. We answer each point below.

1. Sim-to-real evaluation: the gain from RL in simulation transfers to a real robot

Before answering the questions, we are happy to share that we have run sim-to-real experiments similar to those of SimpleVLA-RL [1]: the policies are trained in simulation only and then run on a real Franka arm. The preliminary results on three tasks are in, and we added them to the paper (Section 4.3 and Appendix D, with a video in the supplementary material). We compare the imitation-only policy with the policy after RL with DEAL, with 10 trials per task for each policy. The trials vary the object positions, the objects and the surroundings, and both policies are tested under the same conditions.

Successes on the real robotStack two bowlsPress staplerPlace the cupAll
Imitation only0/102/102/104/30
After RL with DEAL7/107/105/1019/30

The policy after RL with DEAL succeeds in 19 of 30 real trials, against 4 of 30 for imitation only, so the improvement from RL in simulation carries over to the real robot. We are extending this evaluation to more tasks and more trials, and will include them in the camera-ready version.

2. How much does the Gaussian head alone contribute?

We thank the reviewer for this suggestion. To address it, we ran the ablation with BC that the reviewer proposes. We trained our head and π0.5\pi_{0.5}'s flow-matching head under the same conditions: the same π0.5\pi_{0.5} VLM, random initial weights, the same 40 demonstrations (one per LIBERO task) and the same batch size. The table gives the average success (%) over the four LIBERO suites (Figure A2 in Appendix F).

Training steps5001,0002,0003,0004,000Time per step (4 GPUs)
Ours41.456.265.564.962.14.05 s
Flow-matching0.723.954.161.462.411.38 s

Our head learns faster, and the two heads end at the same success, so the head alone does not raise the final success. The flow-matching head is below the 77.1% reported in [2] because [2] starts from π0.5\pi_{0.5}'s pretrained head, while here both heads start from random weights. The 5 points reported by Kim et al. [3] compare continuous actions with discrete tokens, and both heads here are continuous.

What our head adds is the exact likelihood of each continuous action, which the step selection needs, and speed: its training steps are 2.8$\times$ faster and its action generation is about 1.4$\times$ faster (Section 4.4). In RL, our method with this head is also more sample efficient than a method that keeps the flow-matching head: on LIBERO with one demonstration per task, we reach a similar success to πRL\pi_{\mathrm{RL}} [2] with at least 8.7$\times$ fewer RL episodes on every suite (Table A5 in Appendix F). This matters because every RL episode is a complete task attempt by the robot, and the low sample efficiency of online RL is the reason such methods rely on training in simulation [2].

References

[1] Li et al. SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning. ICLR 2026.

[2] Chen et al. πRL\pi_{\mathrm{RL}}: Online RL Fine-tuning for Flow-based Vision-Language-Action Models. arXiv:2510.25889, 2025.

[3] Kim et al. Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success. arXiv:2502.19645, 2025.

Part (2/3)

3. Outside RoboTwin 2.0, it is hard to attribute the gains to DEAL, because the comparisons use different base models, action heads and training pipelines.

We thank the reviewer for this point. The action head is part of what the paper proposes, so the comparisons with the baselines show the alignment and the head together. To show what the alignment adds alone on a second benchmark, we repeated the PPO comparison of Figure 4 on LIBERO-Long: PPO and PPO + DEAL from the same starting policy (32.4%), with three seeds each. The table gives the success (%) at each evaluation, as the mean and standard deviation over the three seeds (200 scenes during training, 500 at the end) (Section 4.4 and Figure 4(d) in the main text).

RL episodes05121,0241,5362,048
PPO32.435.0±5.635.0 \pm 5.639.5±1.039.5 \pm 1.037.0±1.537.0 \pm 1.540.8±2.640.8 \pm 2.6
PPO + DEAL32.443.0±1.743.0 \pm 1.747.2±6.047.2 \pm 6.054.3±1.654.3 \pm 1.658.2±1.758.2 \pm 1.7

PPO + DEAL is ahead at every evaluation, and at the end it is ahead on every seed, by 16 to 19 points. The base model does not explain the other results either: on the Ego Humanoid Manipulation Benchmark we use EgoVLA's own VLM, and on RoboTwin 2.0 our policy starts close to SimpleVLA-RL before RL (40.3% vs 38.6%), so the lead comes with RL. Together, these results show that the gains come from the alignment and the action head that we propose, and not from a different base model or training pipeline.

4. How are the wasted steps in the videos annotated, and how accurate are the labels?

We thank the reviewer for these questions. Every label in the videos and figures comes from our dynamic programming alignment, and no human labels any step. The labels are computed automatically, with the same features and the same alignment as in RL. The alignment is also fast: it is a single dynamic program per episode and needs no extra model. We now state this in the caption of Figure 2 and in Appendix B.

To measure how accurate the labels are, we ran a new experiment of the kind the reviewer suggests, in which the wasted steps are known automatically. We inserted wasted steps (pauses, detours, missed grasps and rewinds) into LIBERO-Long demonstrations and into the policy's own episodes, and measured how many of them our alignment labels as wasted (Section 4.4, and Table A4 in Appendix E).

Inserted waste found (%)AlignmentPPO's advantage
Successful demonstrations, pauses and detours98.137.5
Successful demonstrations, missed grasps and rewinds100.047.1
Failed demonstrations, pauses and detours97.839.1
Failed demonstrations, missed grasps and rewinds99.047.3
The policy's own successes91.758.9
The policy's own failures91.154.8

Our alignment finds 98 to 100% of the waste inserted into the LIBERO-Long demonstrations and over 91% of the waste inserted into the policy's own episodes, where PPO's advantage finds only 38 to 59%. Necessary steps are rarely labeled wasted: our alignment labels only 3.2 to 9.0% of the demonstrator's own steps as wasted. Together, these results show that the automatic labels of the alignment are reliable. The supplementary material has new videos of the demonstration episodes with their labels.

5. The assumptions of the theory are strong, and the same subgoal order may limit exploration and finding global optima.

We thank the reviewer for this point. Our theory is meant as a motivation: it explains, in a simple setting that can be analyzed, why selecting steps makes RL more sample efficient, and Appendix H.6 lists where its assumptions may not hold in practice. Our claims rest on the experiments with VLAs on three benchmarks, which do not need these assumptions.

On the subgoal order, the alignment assumes that an episode and its reference complete the subgoals in the same order, and we state this as a limitation in Section 3.2 and in the Limitations (Appendix G). Its effect on exploration is limited. In a success, a step labeled wasted is only left out of the update and is never discouraged, and in a failure such a step is discouraged no more than in ordinary RL. The reference is also the policy's own shortest success, so when the policy finds a faster way to complete the task, that success becomes the new reference, and the policy is not tied to its first way of solving the task. We extended this discussion in Appendix G, and we consider an alignment that does not depend on the order of subgoals as future work.

Part (3/3)

6. Failure cases: tasks where a slower execution is better, although a faster one also counts as a success.

We thank the reviewer for this thoughtful question. The alignment does not reward speed. The alignment selects the steps that advance the task, relative to a success in the same scene, and the reward stays the binary outcome. If a scene requires a slow and careful execution, a motion that is too fast fails, for example by breaking the plate, and cannot become the reference. The reference is then a slow success, and the careful steps are matched as productive. The remaining case is when a faster execution also counts as a success although a slower one is preferred. The binary outcome cannot tell these two apart, for our method or for any RL method that learns from it, so the preference has to be part of the success signal. We added this discussion to the Limitations (Appendix G).

7. It could be clearer that the Ego benchmark refers to the Ego Humanoid Manipulation Benchmark.

We thank the reviewer for pointing this out. We now write the full name in the caption of Table 1 and define the short name once, at the start of Section 4.

Beyond these points and the sim-to-real evaluation, the revision adds two more results. A new arm on LIBERO-Long computes GAE over the full episode, as in standard PPO, and only adds the step selection. This arm reaches 59.8±6.159.8 \pm 6.1%, which verifies that the gain comes from the step selection itself. We also report the comparisons per task in the appendix: PPO + DEAL is ahead of PPO on 9 of the 10 LIBERO-Long tasks, and it is the best of the four methods of Figure 4 on each of the three RoboTwin 2.0 tasks (Tables A2 and A3 in Appendix C).

The reviewer's suggestions have made the paper stronger, and we are grateful for them. We hope these answers and the new results address the reviewer's concerns, and we are happy to clarify anything further. In light of the new results, we would greatly appreciate it if the reviewer would consider raising the score.

I thank the authors for the extensive additional investigations and their careful evaluation over several seeds. My questions have been answered well and I raised my rating to clear accept.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

General response to all reviewers

We appreciate the time and effort taken by all the reviewers to provide constructive feedback on our paper, and we are pleased that all three reviewers see value in the proposed approach. In particular, the reviewers remarked that DEAL is "a meaningful conceptual contribution" (8jkM), "a practical approach to credit assignment" backed by "comprehensive experiments" (ZnMU), and "a strong contribution with potentially large impact" with "impressive sample efficiency" (NAm9).

Across three benchmarks, our method reaches higher success with far fewer RL episodes than prior work: 97.6% on LIBERO with one demonstration per task, 80.4% against SimpleVLA-RL's 66.4% on RoboTwin 2.0 with an estimated 5.6$\times$ fewer RL episodes, and 72.4% against EgoVLA's 53.2% on the Ego Humanoid Manipulation Benchmark.

We are also happy to share new sim-to-real results. We have run experiments similar to those of SimpleVLA-RL [1]: the policies are trained in simulation only and then run on a real Franka arm. On three tasks, the policy after RL with DEAL succeeds in 19 of 30 trials, against 4 of 30 for imitation only (Section 4.3 and Appendix D, with a video in the supplementary material). We are extending this evaluation to more tasks and more trials, and will include them in the camera-ready version.

Below we answer the questions shared by several reviewers. The new experiments are in the revised paper, and each individual reply says where to find them.

Q1. Is the gain from the alignment itself? (8jkM, NAm9)

Yes. On LIBERO-Long, with the same starting policy, backbone, head and RL algorithm, adding the alignment raises PPO from 40.8±2.640.8 \pm 2.6% to 58.2±1.758.2 \pm 1.7% over three seeds (Section 4.4 and Figure 4(d) in the main text).

Q2. Does the gain come from selecting the steps? (8jkM)

Yes. A new variant that estimates advantages exactly as PPO does, and uses DEAL only to select the steps, reaches 59.8±6.159.8 \pm 6.1% (Section 4.4).

Q3. Are the step labels accurate? (8jkM, ZnMU, NAm9)

Yes. In a new test we insert wasted steps, so we know exactly which steps are wasted. Our alignment finds 98 to 100% of the waste inserted into the LIBERO-Long demonstrations and over 91% in the policy's own episodes, where PPO's advantage finds only 38 to 59% (Appendix E).

Q4. Does our head cost success against π0.5\pi_{0.5}'s flow-matching head? (8jkM, NAm9)

No. We trained the two heads under the same conditions: they reach the same success (62.1% and 62.4%), and ours trains 2.8$\times$ faster. In RL, we need at least 8.7$\times$ fewer episodes than πRL\pi_{\mathrm{RL}} [2], which keeps the flow-matching head (Appendix F).

Q5. Are necessary adjustments, recoveries and successful retries labeled wasted? (8jkM, ZnMU, NAm9)

No. These steps advance the task, so they match the reference and are labeled productive. Our alignment labels only 3.2 to 9.0% of the demonstrator's own steps as wasted (Appendix E).

References

[1] Li et al. SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning. ICLR 2026.

[2] Chen et al. πRL\pi_{\mathrm{RL}}: Online RL Fine-tuning for Flow-based Vision-Language-Action Models. arXiv:2510.25889, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.