- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work investigates the out-of-distribution (OOD) generalization of Vision-Language-Action (VLA) models fine-tuned via reinforcement learning (RL). The authors find that standard RL fine-tuning (RFT) actively diversifies the subset of successful trajectories, covering a broader portion of the task-valid solution manifold with fewer rollouts. To address this, the authors propose DRIVE (Diversity-driven RL fIne-tuning for VLA gEneralization), which clusters rollouts under matched environmental conditions and computes a success-conditioned relative diversity bonus. Extensive experiments and real-world evaluation demonstrate that DRIVE improves the OOD success rate.
Critical strengths
-
This paper offers a fresh and valuable perspective for robot learning by identifying that RL diversifies successful solution modes while contracting failure modes.
-
The paper evaluates two distinct VLA backbones across multiple benchmark domains. Crucially, the physical dual-arm experiments on the AgileX PiPER-X platform provide compelling evidence that the diversity-driven generalization gains translate directly to physical hardware.
Critical weaknesses
-
Intuitively, penalizing trajectory consistency and incentivizing behavioral divergence may inadvertently encourage unnecessary redundant actions, thereby reducing execution efficiency. The paper lacks an analysis of this issue.
-
The relative diversity reward in DRIVE depends on pairwise comparisons within matched condition groups. In early training phases or challenging exploration tasks where task success is sparse, the diversity bonus degenerates to zero or exhibits extreme variance, making the relative diversity reward difficult to compute.
Questions for authors
-
Did the authors observe any degradation in execution efficiency (e.g., increased trajectory length) when DRIVE incentivizes trajectory diversity?
-
How does DRIVE handle rollout groups in which the success count is zero or one?
5 comments
Author Response (Part 2/2)
W2 & Q2: Handling sparse successes and stabilizing diversity rewards.
A2. Thank you for raising this concern. We clarify how DRIVE handles insufficient successful samples and controls the diversity reward scale.
Groups with 0-2 successes prioritize task success without diversity shaping. We set for every trajectory in such groups and retain the original environment rewards, avoiding normalization over an insufficient successful set. This fallback is intentional: when successful samples are scarce, optimization focuses on learning to succeed. Once a sampled group contains at least three successful trajectories, DRIVE additionally rewards diversity among successful executions to encourage broader coverage of feasible solutions. This rule is applied independently to each matched-condition group at every training iteration.
SFT initialization makes success-conditioned diversity shaping practical. DRIVE is a post-training method that starts from a policy with existing task-solving ability. In our ManiSkill3 analysis, the SFT policies already achieve success rates of 41% for and 52% for (Fig. 2). Successful rollouts are therefore available from the outset, providing a practical basis for comparing successful behaviors and rewarding their diversity during RFT.
A normalization floor limits reward amplification, while clipping keeps diversity scores bounded. When diversity shaping is active, we normalize successful trajectories’ diversity scores using the denominator , where . This prevents negligible score differences from being amplified to the full bonus range when the observed spread is small. Clipping the normalized scores to further controls the reward scale.
We have clarified these implementation details and their motivation in Sec. 4.3 and Appendix E of the revised manuscript.
Author Summary Response
Thank you for your constructive review and for recognizing the value of our perspective on successful-behavior diversity and our physical robot evaluation. Our additional execution-length statistics and EEF visualizations support comparable overall execution lengths, with aggregate changes below 0.6% relative to Vanilla RFT (W1, Q1). We also clarify the feasibility provided by SFT initialization, the zero-bonus fallback for groups with 0–2 successes, and normalization safeguards that control bonus scale (W2, Q2). We would appreciate your reconsideration in light of these results and clarifications.
Author Response (Part 1/2)
W1 & Q1: Analysis of execution efficiency (e.g., increased trajectory length, unnecessary redundant actions).A1. Thank you for raising this concern. We have added quantitative execution-length statistics and qualitative trajectory comparisons to examine whether diversity incentives introduce redundant actions.
Aggregate execution lengths remain comparable to Vanilla RFT, with changes below 0.6%. We measure the number of executed environment steps per successful evaluation trajectory for all methods in Table 1, covering both backbones and all benchmarks under IND and OOD conditions. For π₀, the reported average lengths change from 122.70 to 122.52 (IND) and from 128.07 to 127.70 (OOD). For π₀.₅, they change from 105.53 to 105.95 (IND) and from 111.76 to 112.40 (OOD). Thus, each aggregate change is less than 0.6% in magnitude. Individual benchmarks exhibit both increases and decreases, but the aggregate results show no systematic length inflation relative to Vanilla RFT.
EEF visualizations provide complementary evidence of different successful execution paths. We compare Vanilla RFT and DRIVE on four ManiSkill tasks, using 16 rollouts per method and checkpoint from the same task-specific initial state. The representative successful trajectories illustrate alternative feasible paths to task completion. These spatial comparisons complement the execution-length measurements by showing that DRIVE encourages different feasible task-solving paths without a systematic increase in execution length or apparent redundant motions. (Appendix C.2, Fig. 5).
DRIVE targets diverse successful strategies while reducing sensitivity to superficial timing differences. These observations are also consistent with the design of our trajectory diversity measure. Specifically, GAK performs monotonic temporal alignment between VLM feature sequences, making trajectory similarity less sensitive to differences in execution rates and local temporal misalignment. This helps avoid treating similar execution patterns as behaviorally distinct merely because they progress at different speeds. Moreover, our success-conditioned reward prevents unsuccessful behaviors from receiving diversity bonuses.
Together, these results suggest that DRIVE promotes diverse successful execution strategies without systematically increasing execution length or compromising execution efficiency. We have included the additional execution-length statistics and representative trajectory visualizations in the revised manuscript (Appendix C.2).