acceptodds
Under review as a conference paper at ICLR 2027

Many Ways to Succeed: Diversity-Driven RL Fine-Tuning for VLA Generalization

Abstract

Reinforcement learning (RL) fine-tuning improves vision-language-action (VLA) policies through closed-loop experience, yet generalization beyond the fine-tuning distribution remains limited. Our analysis reveals a selective reshaping of exploration: RL contracts behavior globally, yet diversifies successful trajectories, elicits success with fewer rollouts, and covers more of the latent task-valid solution space than supervised fine-tuning. Broader successful-mode coverage may provide alternative strategies under distribution shifts. Inspired by this, we introduce **DRIVE** (**D**iversity-driven **R**L f**I**ne-tuning for **V**LA g**E**neralization), which turns successful-behavior diversity into an explicit RL objective. DRIVE groups rollouts under matched task conditions, compares their trajectories with temporal alignment, and derives a success-conditioned intrinsic reward from relative behavioral diversity. This design encourages broader coverage of feasible solutions without rewarding diverse failures or superficial timing differences. Across LIBERO-Plus, ManiSkill3, and RoboTwin 2.0, DRIVE improves the average out-of-domain (OOD) performance over vanilla RL fine-tuning by 5.3 points on and 2.0 points on . On a dual-arm AgileX PiPER-X platform, DRIVE further increases average OOD success from 64.1% to 73.3% (+9.2 points), demonstrating gains that persist under physical deployment.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, generalization, diversity

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_AysROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates the out-of-distribution (OOD) generalization of Vision-Language-Action (VLA) models fine-tuned via reinforcement learning (RL). The authors find that standard RL fine-tuning (RFT) actively diversifies the subset of successful trajectories, covering a broader portion of the task-valid solution manifold with fewer rollouts. To address this, the authors propose DRIVE (Diversity-driven RL fIne-tuning for VLA gEneralization), which clusters rollouts under matched environmental conditions and computes a success-conditioned relative diversity bonus. Extensive experiments and real-world evaluation demonstrate that DRIVE improves the OOD success rate.

Critical strengths
  1. This paper offers a fresh and valuable perspective for robot learning by identifying that RL diversifies successful solution modes while contracting failure modes.

  2. The paper evaluates two distinct VLA backbones across multiple benchmark domains. Crucially, the physical dual-arm experiments on the AgileX PiPER-X platform provide compelling evidence that the diversity-driven generalization gains translate directly to physical hardware.

Critical weaknesses
  1. Intuitively, penalizing trajectory consistency and incentivizing behavioral divergence may inadvertently encourage unnecessary redundant actions, thereby reducing execution efficiency. The paper lacks an analysis of this issue.

  2. The relative diversity reward in DRIVE depends on pairwise comparisons within matched condition groups. In early training phases or challenging exploration tasks where task success is sparse, the diversity bonus degenerates to zero or exhibits extreme variance, making the relative diversity reward difficult to compute.

Questions for authors
  1. Did the authors observe any degradation in execution efficiency (e.g., increased trajectory length) when DRIVE incentivizes trajectory diversity?

  2. How does DRIVE handle rollout groups in which the success count is zero or one?

5 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)


W1 & Q1: Analysis of execution efficiency (e.g., increased trajectory length, unnecessary redundant actions).

A1. Thank you for raising this concern. We have added quantitative execution-length statistics and qualitative trajectory comparisons to examine whether diversity incentives introduce redundant actions.

Aggregate execution lengths remain comparable to Vanilla RFT, with changes below 0.6%. We measure the number of executed environment steps per successful evaluation trajectory for all methods in Table 1, covering both backbones and all benchmarks under IND and OOD conditions. For π₀, the reported average lengths change from 122.70 to 122.52 (IND) and from 128.07 to 127.70 (OOD). For π₀.₅, they change from 105.53 to 105.95 (IND) and from 111.76 to 112.40 (OOD). Thus, each aggregate change is less than 0.6% in magnitude. Individual benchmarks exhibit both increases and decreases, but the aggregate results show no systematic length inflation relative to Vanilla RFT.

BackboneMethodSpatial INDSpatial OODObject INDObject OODGoal INDGoal OODLong INDLong OODManiSkill INDManiSkill OODRoboTwin INDRoboTwin OODAvg. INDAvg. OOD
π₀SFT123.43 ± 53.21123.71 ± 56.50152.36 ± 39.63154.86 ± 33.09103.80 ± 31.95129.54 ± 57.51258.43 ± 63.25267.45 ± 65.4928.28 ± 9.9533.48 ± 16.53101.08 ± 33.8096.61 ± 38.83127.90134.27
π₀Vanilla RFT105.31 ± 24.06106.51 ± 26.84132.18 ± 15.87134.91 ± 17.33120.90 ± 40.45129.33 ± 44.09257.57 ± 68.18267.52 ± 73.3832.50 ± 14.4642.09 ± 18.5687.76 ± 36.9388.08 ± 38.62122.70128.07
π₀DRIVE104.88 ± 21.79108.93 ± 26.51134.81 ± 19.48136.98 ± 20.78122.82 ± 37.11132.99 ± 48.96258.58 ± 70.35264.53 ± 67.5036.06 ± 17.1041.92 ± 18.0577.97 ± 32.1780.88 ± 36.19122.52127.70
π₀.₅SFT111.15 ± 45.61111.75 ± 41.50141.15 ± 41.48145.32 ± 48.03101.59 ± 29.82109.81 ± 46.58262.78 ± 53.06283.61 ± 58.5229.10 ± 11.7036.60 ± 16.7292.48 ± 28.2391.53 ± 27.64123.04129.77
π₀.₅Vanilla RFT93.69 ± 20.2497.17 ± 26.50123.29 ± 19.52123.91 ± 21.80107.14 ± 28.83110.69 ± 35.01228.34 ± 57.34240.58 ± 63.2527.30 ± 10.4535.76 ± 17.0353.41 ± 20.3762.42 ± 26.71105.53111.76
π₀.₅Higher Noise88.57 ± 17.8496.13 ± 30.55119.64 ± 21.25120.53 ± 21.35101.15 ± 29.18101.75 ± 29.79223.29 ± 54.82231.11 ± 60.4826.55 ± 8.7336.04 ± 17.1767.25 ± 32.5266.66 ± 31.45104.41108.70
π₀.₅Clip-Higher95.17 ± 17.7099.60 ± 26.73126.20 ± 19.61128.29 ± 21.73106.96 ± 28.38108.64 ± 31.64226.48 ± 55.10238.98 ± 64.0026.68 ± 11.3735.60 ± 17.1752.36 ± 22.2152.13 ± 22.34105.64110.54
π₀.₅KL Reg.100.49 ± 19.26104.00 ± 25.03132.57 ± 21.34133.52 ± 21.49103.50 ± 21.48107.80 ± 31.80243.11 ± 61.20252.14 ± 61.1426.59 ± 9.9935.72 ± 16.6863.67 ± 25.1763.31 ± 24.77111.65116.08
π₀.₅DRIVE93.02 ± 19.3996.42 ± 26.53132.39 ± 18.40134.22 ± 21.59102.58 ± 23.92107.89 ± 32.36231.64 ± 62.51245.37 ± 65.1524.37 ± 8.8633.82 ± 16.9951.72 ± 26.9756.70 ± 27.86105.95112.40

EEF visualizations provide complementary evidence of different successful execution paths. We compare Vanilla RFT and DRIVE on four ManiSkill tasks, using 16 rollouts per method and checkpoint from the same task-specific initial state. The representative successful trajectories illustrate alternative feasible paths to task completion. These spatial comparisons complement the execution-length measurements by showing that DRIVE encourages different feasible task-solving paths without a systematic increase in execution length or apparent redundant motions. (Appendix C.2, Fig. 5).

DRIVE targets diverse successful strategies while reducing sensitivity to superficial timing differences. These observations are also consistent with the design of our trajectory diversity measure. Specifically, GAK performs monotonic temporal alignment between VLM feature sequences, making trajectory similarity less sensitive to differences in execution rates and local temporal misalignment. This helps avoid treating similar execution patterns as behaviorally distinct merely because they progress at different speeds. Moreover, our success-conditioned reward prevents unsuccessful behaviors from receiving diversity bonuses.

Together, these results suggest that DRIVE promotes diverse successful execution strategies without systematically increasing execution length or compromising execution efficiency. We have included the additional execution-length statistics and representative trajectory visualizations in the revised manuscript (Appendix C.2).


AuthorsOpenReview ↗

Author Response (Part 2/2)


W2 & Q2: Handling sparse successes and stabilizing diversity rewards.

A2. Thank you for raising this concern. We clarify how DRIVE handles insufficient successful samples and controls the diversity reward scale.

Groups with 0-2 successes prioritize task success without diversity shaping. We set ridiv=0r_i^{\mathrm{div}}=0 for every trajectory in such groups and retain the original environment rewards, avoiding normalization over an insufficient successful set. This fallback is intentional: when successful samples are scarce, optimization focuses on learning to succeed. Once a sampled group contains at least three successful trajectories, DRIVE additionally rewards diversity among successful executions to encourage broader coverage of feasible solutions. This rule is applied independently to each matched-condition group at every training iteration.

SFT initialization makes success-conditioned diversity shaping practical. DRIVE is a post-training method that starts from a policy with existing task-solving ability. In our ManiSkill3 analysis, the SFT policies already achieve success rates of 41% for π0\pi_0 and 52% for π0.5\pi_{0.5} (Fig. 2). Successful rollouts are therefore available from the outset, providing a practical basis for comparing successful behaviors and rewarding their diversity during RFT.

A normalization floor limits reward amplification, while clipping keeps diversity scores bounded. When diversity shaping is active, we normalize successful trajectories’ diversity scores using the denominator max⁡(dmax⁡−dmin⁡,δ)\max(d_{\max}-d_{\min},\delta), where δ>0\delta>0. This prevents negligible score differences from being amplified to the full bonus range when the observed spread is small. Clipping the normalized scores to [0,1][0,1] further controls the reward scale.

We have clarified these implementation details and their motivation in Sec. 4.3 and Appendix E of the revised manuscript.


AuthorsOpenReview ↗

Author Summary Response


Thank you for your constructive review and for recognizing the value of our perspective on successful-behavior diversity and our physical robot evaluation. Our additional execution-length statistics and EEF visualizations support comparable overall execution lengths, with aggregate changes below 0.6% relative to Vanilla RFT (W1, Q1). We also clarify the feasibility provided by SFT initialization, the zero-bonus fallback for groups with 0–2 successes, and normalization safeguards that control bonus scale (W2, Q2). We would appreciate your reconsideration in light of these results and clarifications.

Reviewer_2A4dOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how reinforcement fine-tuning changes the behavioral diversity of vision-language-action policies and proposes DRIVE to improve generalization by rewarding diverse successful trajectories. DRIVE collects rollouts under matched task conditions, compares sequences of VLM features using a normalized global alignment kernel, and adds a success-conditioned diversity bonus to PPO training. Experiments cover two policy backbones, three simulation benchmark families, and two tasks on a physical dual-arm robot. The reported OOD macro-average improves from 48.1% to 53.4% for π₀ and from 71.1% to 73.1% for π₀.₅, with additional gains in physical deployment.

Critical strengths
  1. Focusing on diverse successful behavior is a useful idea. Matching task conditions and rewarding only successful executions gives the objective a clear connection to task performance. The separate analyses of all trajectories and successful trajectories also provide an interesting view of how RFT changes policy behavior.

  2. The evaluation covers several relevant settings. Table 1 reports improvements over vanilla RFT in 11 of 12 IND/OOD entries for each backbone, with additional exploration and regularization baselines for π₀.₅. The physical experiments improve seven of eight task-condition combinations, providing useful evidence beyond simulation.

Critical weaknesses
  1. The policy-invariance proof does not cover the implemented reward. Eq. (10) retains a trajectory-dependent terminal bonus that can change preferences among equally successful behaviors. The potential also depends on eventual success and other sampled rollouts, whereas Appendix A assumes a fixed state potential. These differences need to be resolved before the guarantee can be claimed.

  2. The generalization analysis leaves the role of success conditioning unclear. Figure 5 reports this ablation only on IND performance, although OOD generalization is the main motivation. The OOD result would be more informative here, particularly since the π₀.₅ macro-average gain over Clip-Higher is only 0.5 percentage points in Table 1.

  3. Several presentation details need correction. References and figure/table citations in the PDF are not clickable, and captions sit too close to the tables, particularly Tables 1, 7, and 8. Table 1 also bolds DRIVE's 56.8% for π₀.₅ Goal OOD despite higher competing values. These are minor issues, but they make the results harder to navigate and read.

Questions for authors
  1. How does the terminal reward in Appendix D satisfy the conditions of Theorem 1, given the retained boundary term in Eq. (10) and the dependence on eventual success and the rollout group?

  2. Can you provide the OOD results for the success-conditioning ablation in Figure 5 to show whether success gating also helps generalization?

  3. What behavioral differences account for the lower GAK similarity in Figure 4? Can representative successful trajectories show that these reflect distinct task-solving strategies rather than execution speed or incidental motion?

  4. What is the exact Coverage@K matching rule? Given RFT's higher success rate, does its coverage advantage remain when both policies contribute the same number of successful trajectories?

11 comments
AuthorsOpenReview ↗

Author Response (Part 1/5)


W1 & Q1. Applicability of policy invariance to the implemented diversity reward.

A1. The critical requirement for potential-based reward shaping is that the potential function Φ(s)\Phi(s) is independent of actions, not of eventual success or other sampled rollouts. This ensures the shaping reward does not alter the optimal action selection as

= \arg\max_{a} \left[ Q_{M}^{\ast}(s,a) - \alpha\Phi(s) \right] = \arg\max_{a} Q_{M}^{\ast}(s,a)$$ Our diversity-based potential $d(s)$ satisfies this requirement. The other samples $j\neq i$ are produced by independent rollouts and are fully determined before computing the diversity score, i.e., they are entirely independent of actions taken during the generation of sample $i$. Hence, $d(s)$ **remains action-independent and is a strictly valid potential function under Theorem 1.** We would like to clarify the distinction between “dependence on other samples” and “dependence on actions”: the batch context $j\neq i$ is external to the decision process of sample $i$ and does not violate the action-independence requirement. **A trajectory-dependent terminal bonus need not permanently change preferences among equally successful behaviors.** Consider two equally successful behaviors represented by trajectories $\tau_a$ and $\tau_b$. A higher diversity bonus for $\tau_a$ can temporarily favor it over $\tau_b$ within a particular group. However, DRIVE recomputes these bonuses from the evolving rollout distribution. As the behavior represented by $\tau_a$ becomes more common, its relative novelty can decrease, while the less frequent behavior represented by $\tau_b$ can become more distinctive and receive a larger bonus. This feedback can balance exploration across successful modes. Thus, a change in preferences under one sampled group does not by itself imply that the final policy will permanently favor the same successful behavior. *** **`W2(a) & Q2: OOD evaluation of the success-conditioning ablation.`** A2(a). Thank you for this suggestion. We have extended the success-conditioning ablation in Fig. 6 to OOD evaluation to examine its contribution beyond the fine-tuning distribution. **A matched IND/OOD ablation directly examines the role of success conditioning.** We compare full DRIVE with a variant that also assigns diversity bonuses to unsuccessful trajectories on ManiSkill3 with $\pi_{0.5}$. Both variants share identical SFT initialization, trajectory similarity measure, rollout grouping, diversity coefficient, training budget, and evaluation protocol, with success conditioning being the only difference. **Success conditioning improves OOD success by 2.0 percentage points.** DRIVE achieves 86.8% IND and 51.9% OOD success rates, compared with 81.2% and 49.9% without success conditioning, corresponding to improvements of **5.6 and 2.0 percentage points**, respectively. The results are summarized below. | **Method** | **IND (%)** | **OOD Avg. (%)** | | --- | --- | --- | | DRIVE | **86.8** | **51.9** | | DRIVE w/o Success Conditioning | 81.2 | 49.9 | **These results further support the importance of directing diversity incentives toward successful executions.** The comparison suggests that encouraging diversity among task-successful behaviors is more effective than rewarding behavioral variation regardless of task completion. The improvements under both IND and OOD conditions are consistent with DRIVE's motivation to promote diverse, task-valid solutions for better generalization. We have incorporated the additional OOD results and corresponding analysis into **Sec. 5.2** of the revised manuscript. ***
AuthorsOpenReview ↗

Author Response (Part 2/5)


W2(b): Comparison and complementarity with Clip-Higher.

A2(b). Clip-Higher is a strong exploration baseline with demonstrated empirical benefits. We agree that Clip-Higher is a competitive comparator. Introduced in DAPO [1], it was designed to mitigate entropy collapse and improve exploration during RL training. Its competitiveness is also evident in our π0.5\pi_{0.5} results: Clip-Higher achieves 72.6% OOD macro-average success, compared with 73.1% for DRIVE. We therefore interpret the 0.5-percentage-point advantage as a modest gain over a strong baseline. Clip-Higher's competitive performance also highlights the importance of effective exploration in VLA reinforcement fine-tuning, consistent with the broader motivation of our work.

Clip-Higher and DRIVE improve exploration through different mechanisms. Clip-Higher raises the upper importance-ratio clipping threshold, allowing positively advantaged actions to receive larger probability increases before the PPO surrogate objective is clipped. DRIVE operates at the reward level: it explicitly rewards behaviorally distinct successful executions under matched task conditions to promote task-valid exploration and generalization. Thus, Clip-Higher relaxes an optimization constraint, while DRIVE supplies an explicit incentive for successful-behavior diversity.

Combining DRIVE with Clip-Higher demonstrates the additional value of diversity incentives. We further evaluate the combination of Clip-Higher and DRIVE on ManiSkill3 with π0.5\pi_{0.5}, pairing DRIVE's success-conditioned diversity reward with Clip-Higher's relaxed clipping mechanism. Compared with Clip-Higher alone, the combined method improves IND and OOD success rates by 5.3 and 1.9 percentage points, respectively. It also achieves 2.9-point higher OOD success than DRIVE alone. These results support the complementarity of the two mechanisms, suggesting that explicit diversity incentives can provide additional benefits even when the optimization already encourages exploration through relaxed clipping.

MethodManiSkill3 IND (%)ManiSkill3 OOD Avg. (%)
Clip-Higher80.052.9
DRIVE86.851.9
Clip-Higher + DRIVE85.354.8

[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale, NeurIPS 2025.


AuthorsOpenReview ↗

Author Response (Part 3/5)


Q3. Interpretation of trajectory diversity in Fig. 4.

A3. Thank you for this question. We provide additional end-effector (EEF) trajectory visualizations and execution-length statistics, together with an explanation of GAK's temporal alignment, to clarify whether the lower similarity in Fig. 4 reflects meaningful behavioral differences rather than execution-speed variations or incidental motions.

Matched successful rollouts make the execution-path differences observable. We compare successful EEF trajectories from Vanilla RFT and DRIVE on ManiSkill3 with π0.5\pi_{0.5} under matched task conditions and initial states, as illustrated in Fig. 5 of the revised manuscript. The visualizations reveal differences in spatial routes and path geometries among successful executions, rather than merely showing identical paths traversed at different speeds. These observations provide a direct behavioral interpretation of the feature-space differences in Fig. 4(d), illustrating distinct feasible execution behaviors under the same task conditions. More extensive trajectory visualizations across training checkpoints are provided in the Appendix.

Execution-length statistics check whether increased diversity is accompanied by longer rollouts. For the corresponding ManiSkill3 setting, we analyze the number of executed environment steps per successful trajectory for Vanilla RFT and DRIVE with α=0.1\alpha=0.1 and α=0.2\alpha=0.2, under both IND and OOD conditions. As summarized below, the execution lengths remain broadly comparable across the three settings, with no systematic increase as the diversity incentive is introduced or strengthened. This suggests that the observed behavioral diversity is not simply driven by longer executions or additional redundant motions. We further extend this analysis across both VLA backbones and all six benchmark settings, where the aggregate execution-length differences between DRIVE and Vanilla RFT remain below 0.6% across the four backbone–evaluation combinations. Together, these results suggest that DRIVE's successful-behavior diversity is not achieved at the cost of systematically longer executions or reduced execution efficiency. Detailed cross-benchmark statistics are provided in the revised Appendix.

MethodIND (Steps)OOD (Steps)
Vanilla RFT27.30 ± 10.4535.76 ± 17.03
DRIVE (α=0.1\alpha=0.1)24.37 ± 8.8633.82 ± 16.99
DRIVE (α=0.2\alpha=0.2)26.43 ± 10.6336.72 ± 17.55

GAK reduces sensitivity to execution speed through temporal alignment. GAK compares VLM feature sequences using monotonic temporal alignment, allowing similar behavioral patterns at different temporal positions to be matched while preserving their order. This reduces the influence of execution-rate differences and local temporal misalignment on trajectory similarity. The EEF visualizations and execution-length statistics provide complementary evidence for interpreting these feature-space differences. Specifically, the trajectory comparisons reveal spatial differences among successful executions, while the execution-length analysis shows that these differences are not accompanied by systematic increases in execution length. Taken together, these observations support our interpretation that DRIVE encourages diverse, task-valid execution behaviors rather than merely longer or temporally misaligned trajectories.

We have incorporated representative EEF trajectory visualizations and execution-length analyses into the revised manuscript, with additional experimental results and details provided in Appendix C.2.


Reviewer_fuYuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DRIVE, a reinforcement fine-tuning method that encourages diverse successful behaviors in VLA policies. It compares matched-condition rollouts using a temporally aligned similarity measure over VLM features and assigns a diversity bonus only to successful trajectories. Experiments on two backbones, three simulation benchmark families, and a physical dual-arm platform report improvements in out-of-distribution generalization.

Critical strengths
  1. A useful and practical exploration objective. Distinguishing successful-behavior diversity from overall behavioral variation provides a compelling motivation for targeted exploration. Matched-condition grouping, temporal alignment, and success conditioning form a coherent approach that integrates directly into existing VLA reinforcement fine-tuning.

  2. Consistent improvements across models and evaluation settings. DRIVE improves OOD performance over vanilla RFT in all six reported settings for π0\pi_0 and five of six for π0.5\pi_{0.5}, increasing the benchmark-family OOD averages by 5.3 and 2.0 points, respectively. The physical experiments provide additional evidence that the benefits extend beyond simulation. This breadth is the primary reason for my positive recommendation.

Critical weaknesses
  1. The policy-invariance theorem does not apply to the implemented reward. Equation (10) retains a trajectory-dependent terminal bonus, which can distinguish equally rewarded, equal-length successful trajectories and therefore change the optimal-policy set. Appendix A does not address this terminal boundary, and the potential additionally depends on eventual success and the sampled rollout group. The paper should remove or substantially qualify this guarantee and present DRIVE as an auxiliary diversity objective.

  2. The mechanism and statistical reliability need stronger validation. Constant-success-bonus and shuffled-diversity controls would help establish whether gains specifically arise from behavioral diversity. The success-conditioning ablation reports only IND results, while generalization is the central claim. Training-seed variability and uncertainty estimates are also missing, which matters for the 0.5-point OOD average advantage over Clip-Higher on π0.5\pi_{0.5} and the physical evaluation with 20 trials per task-condition pair.

Questions for authors
  1. Can you clarify the exact terminal reward passed to GAE and revise Theorem 1 to address the trajectory-dependent terminal potential? Would you agree that the implemented method changes the training objective rather than preserving all optimal policies?

  2. Do the OOD improvements persist against constant and shuffled success bonuses matched in reward scale? Does the success-conditioning ablation also improve OOD performance?

  3. How many independent training runs underlie Table 1, and what is the variability across runs? Please clarify checkpoint and hyperparameter selection, including whether OOD evaluations influenced these choices.

6 comments
AuthorsOpenReview ↗

Author Response (Part 1/4)


W1 & Q1. Applicability of policy invariance to the implemented diversity reward.

A1. The critical requirement for potential-based reward shaping is that the potential function Φ(s)\Phi(s) is independent of actions, not of eventual success or other sampled rollouts. This ensures the shaping reward does not alter the optimal action selection as

= \arg\max_{a} \left[ Q_{M}^{\ast}(s,a) - \alpha\Phi(s) \right] = \arg\max_{a} Q_{M}^{\ast}(s,a)$$ Our diversity-based potential $d(s)$ satisfies this requirement. The other samples $j\neq i$ are produced by independent rollouts and are fully determined before computing the diversity score, i.e., they are entirely independent of actions taken during the generation of sample $i$. Hence, $d(s)$ **remains action-independent and is a strictly valid potential function under Theorem 1.** We would like to clarify the distinction between “dependence on other samples” and “dependence on actions”: the batch context $j\neq i$ is external to the decision process of sample $i$ and does not violate the action-independence requirement. **A trajectory-dependent terminal bonus need not permanently change preferences among equally successful behaviors.** Consider two equally successful behaviors represented by trajectories $\tau_a$ and $\tau_b$. A higher diversity bonus for $\tau_a$ can temporarily favor it over $\tau_b$ within a particular group. However, DRIVE recomputes these bonuses from the evolving rollout distribution. As the behavior represented by $\tau_a$ becomes more common, its relative novelty can decrease, while the less frequent behavior represented by $\tau_b$ can become more distinctive and receive a larger bonus. This feedback can balance exploration across successful modes. Thus, a change in preferences under one sampled group does not by itself imply that the final policy will permanently favor the same successful behavior. *** **`W2(a) & Q2: diversity ablation, success-conditioning ablation, and OOD evaluation.`** **A2(a). We add constant and shuffled success-bonus ablations to test whether OOD gains specifically arise from behavioral diversity.** Following your suggestion, we introduce **Constant Success Bonus**, which assigns an identical bonus to every successful trajectory using the same coefficient as DRIVE. We also introduce **Shuffled Diversity Bonus**, which randomly permutes diversity bonuses among successful trajectories within each matched-condition group, preserving the bonus values while breaking their correspondence with the underlying behaviors. **We extend the success-conditioning ablation to OOD evaluation to test whether success gating also improves generalization.** Specifically, we compare full DRIVE with a variant that applies diversity shaping to both successful and unsuccessful trajectories, while keeping all other settings unchanged. The results under both IND and OOD evaluations are summarized below. | **Method** | **IND (%)** | **OOD (%)** | | --- | --- | --- | | Vanilla RFT | 81.5 | 48.6 | | Constant Success Bonus | 82.5 | 49.8 | | Shuffled Diversity | 78.4 | 49.8 | | DRIVE w/o Success Conditioning | 81.2 | 49.9 | | **DRIVE** | 86.8 | 51.9 | The results show that **DRIVE achieves the highest success rates under both IND and OOD evaluations**. Compared with the constant and shuffled bonuses, DRIVE improves IND success rates by **4.3 and 8.4 percentage points**, respectively. Across all three ablated variants, DRIVE also maintains an **OOD advantage of approximately 2 percentage points**. These results demonstrate that **DRIVE's improvements arise not merely from providing additional rewards to successful trajectories, but from explicitly rewarding their behavioral diversity**. The performance degradation under shuffled diversity bonuses further highlights the importance of assigning diversity rewards according to the actual behavioral differences among successful trajectories. The comparison without success conditioning further highlights the importance of focusing diversity incentives on successful executions. Rather than encouraging behavioral variation indiscriminately, DRIVE directs diversity shaping toward task-successful behaviors, yielding more favorable overall IND and OOD performance. Together, these findings support DRIVE's central design principle of **promoting behaviorally meaningful diversity among successful trajectories**. We have incorporated these additional ablation results and the corresponding analysis into **Sec. 5.2** of the revised manuscript.
AuthorsOpenReview ↗

Author Response (Part 2/4)


W2(b): training-seed variability, uncertainty estimates, and statistical reliability.

A2(b). Thank you for highlighting the need for uncertainty estimates and independent training runs. We extend statistical reporting to both the main experiments and the new ablations in A2(a), while distinguishing evaluation uncertainty from training-seed variability.

We report uncertainty estimates and evaluation scales for the main experiments. We supplement the IND and OOD success rates in Table 1 with standard deviations and clarify the corresponding evaluation scales. For evaluation scale, each LIBERO-Plus task definition uses one fixed initial state, with 250 IND task definitions per suite. ManiSkill3 uses 320 episodes per setting, totaling 320 IND episodes and 4,160 OOD episodes across 13 OOD settings. RoboTwin 2.0 evaluates 350 IND and 500 OOD configurations per task. These evaluation counts describe the test coverage, rather than the number of independent training runs.

We add uncertainty estimates for the new diversity and success-conditioning ablations. All five variants in A2(a) are evaluated on ManiSkill3 with π0.5\pi_{0.5}, using the same SFT initialization, training budget, and IND-based checkpoint-selection protocol; the shaping variants use the same diversity coefficient. We supplement the IND and OOD comparisons with corresponding uncertainty estimates, further supporting the analysis of the contributions of behavioral diversity and success conditioning.

We conduct three independent training runs to assess training-seed variability. On ManiSkill3 with π0.5\pi_{0.5}, we independently train Vanilla RFT and DRIVE using three random seeds under matched training and evaluation configurations. Both methods start from the same SFT checkpoint, and each trained policy is evaluated under IND and OOD conditions. The table reports mean success rates and standard deviations across the three training runs.

Across three independent training runs, DRIVE achieves higher average IND and OOD success rates than Vanilla RFT, with improvements of 3.88 and 4.15 percentage points, respectively. DRIVE also exhibits lower standard deviations in both settings, particularly under OOD evaluation. These results provide further evidence that DRIVE's improvements are consistent across different training seeds, rather than being specific to a particular random seed.

MethodIND (%)OOD (%)
Vanilla RFT82.87 ± 3.1848.51 ± 3.12
DRIVE86.75 ± 1.7252.66 ± 0.66

We further examine the modest 0.5-point OOD advantage over Clip-Higher. On π0.5\pi_{0.5}, DRIVE outperforms Clip-Higher by 2.5 points on LIBERO-Plus and 0.2 points on RoboTwin 2.0, while trailing by 1.0 point on ManiSkill3. Thus, the overall advantage is primarily driven by LIBERO-Plus rather than uniform improvements across benchmarks. Given this small margin and the evaluation variability reported in Table 1, we do not claim statistically established superiority over Clip-Higher. Instead, we regard it as a strong exploration baseline and further investigate whether DRIVE can complement its optimization mechanism.

We have included the statistical reporting for Table 1 and the corresponding evaluation protocols in Appendix E. The expanded physical evaluation has been addressed separately in A2(d).


AuthorsOpenReview ↗

Author Response (Part 3/4)


W2(c): Comparison and complementarity with Clip-Higher.

A2(c). Clip-Higher is a strong exploration baseline with demonstrated empirical benefits. We agree that Clip-Higher is a competitive comparator. Introduced in DAPO [1], it was designed to mitigate entropy collapse and sustain exploration during RL training. Its competitiveness is also evident in our π0.5\pi_{0.5} results: Clip-Higher achieves 72.6% OOD macro-average success, compared with 73.1% for DRIVE. We therefore interpret the 0.5-percentage-point advantage as a modest gain over a strong baseline. Clip-Higher's competitive performance also highlights the importance of effective exploration in VLA reinforcement fine-tuning, consistent with the broader motivation of our work.

Clip-Higher and DRIVE improve exploration through different mechanisms. Clip-Higher raises the upper importance-ratio clipping threshold, allowing positively advantaged actions to receive larger probability increases before the PPO surrogate objective is clipped. In contrast, DRIVE operates at the reward level, explicitly rewarding behaviorally distinct successful executions under matched task conditions to promote task-valid exploration and generalization. Thus, Clip-Higher relaxes an optimization constraint, while DRIVE supplies an explicit incentive for successful-behavior diversity.

Combining DRIVE with Clip-Higher demonstrates the additional value of diversity incentives. We further evaluate the combination of Clip-Higher and DRIVE on ManiSkill3 with π0.5\pi_{0.5}, pairing DRIVE's success-conditioned diversity reward with Clip-Higher's relaxed clipping mechanism. Compared with Clip-Higher alone, the combined method improves IND and OOD success rates by 5.3 and 1.9 percentage points, respectively. It also achieves 2.9-point higher OOD success than DRIVE alone. These results support the complementarity of the two mechanisms, suggesting that explicit diversity incentives can provide additional benefits even when the optimization already encourages exploration through relaxed clipping.

MethodManiSkill3 IND (%)ManiSkill3 OOD Avg. (%)
Clip-Higher80.052.9
DRIVE86.851.9
Clip-Higher + DRIVE85.354.8

[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale, NeurIPS 2025.


W2(d): Trial budget and statistical reliability of physical evaluation.

A2(d). We expand physical evaluation from 20 to 50 trials per task–condition pair. Thank you for highlighting the limited physical-trial budget. Our original 20-trial budget followed the evaluation scale used in prior VLA studies [2–4]. To provide stronger empirical evidence, we expand the evaluation to 50 trials per task–condition pair on the dual-arm AgileX PiPER-X platform. We evaluate Vanilla RFT and DRIVE on Click Bell and Press Stapler under Clean (IND), Distractor, Background, and Lighting (OOD) conditions, resulting in 400 physical trials per method, including 100 IND and 300 OOD trials. Both methods follow the same evaluation protocol, with a 2.5-fold increase in the trial budget.

The expanded evaluation shows consistent OOD improvements under a larger trial budget. In our original 20-trial evaluation, DRIVE achieved an average OOD success rate of 73.3%, compared with 64.1% for Vanilla RFT, representing a 9.2-percentage-point improvement. With the expanded 50-trial evaluation, DRIVE maintains this advantage, achieving 75.0% OOD success compared with 64.0% for Vanilla RFT, corresponding to an 11.0-point improvement. Moreover, DRIVE outperforms Vanilla RFT across all six OOD task–condition pairs. The consistency between the original and expanded evaluations provides stronger evidence that DRIVE's generalization improvements persist under a substantially larger physical evaluation budget. We have updated Table 2 and the corresponding discussion in Sec. 5.3 of the revised manuscript to reflect these results.

TaskMethodClean (IND)Distractor (OOD)Background (OOD)Lighting (OOD)OOD Avg.
Click BellVanilla RFT78%60%68%64%64.0%
DRIVE84%76%76%66%72.7%
Press StaplerVanilla RFT76%64%68%60%64.0%
DRIVE80%74%82%76%77.3%

[2] Policy Contrastive Decoding for Robotic Foundation Models, ICLR 2026.

[3] Human-assisted Robotic Policy Refinement via Action Preference Optimization, NeurIPS 2025.

[4] Object-Centric Residual RL for Zero-Shot Sim-to-Real VLA Enhancement, CoRL 2026.


Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary and Revisions

Dear Area Chairs, Senior Area Chairs, and Reviewers,

We sincerely thank all reviewers for their thoughtful evaluations, constructive feedback, and continued engagement throughout the discussion. We appreciate the reviewers' recognition of our motivation to promote successful-behavior diversity and the breadth of our simulation and real-world evaluations. Your valuable suggestions have helped us strengthen the experimental evidence, clarify our findings, and improve the manuscript.

Rebuttal Highlights.

  1. Diversity Reward and Success Conditioning. Building on the original success-conditioning ablation, we added constant-success-bonus and shuffled-diversity controls to examine whether DRIVE's improvements arise from meaningful behavioral diversity rather than generic success bonuses or shuffled reward assignments. We further extended these ablations to OOD evaluation and clarified the handling of sparse-success groups and reward normalization.
  2. Behavioral Diversity and Coverage. To clarify whether diversity reflects distinct task-solving behaviors rather than redundant actions, we added EEF visualizations and execution-length analyses. We also clarified the Coverage@K definition and matching rule, and conducted equal-success-count comparisons using reference coverage and Vendi Scores to distinguish successful-behavior accessibility, reference-space coverage, and within-success diversity.
  3. Experimental Validation and Reliability. To address concerns about statistical reliability, we reported evaluation standard deviations and conducted three independent training runs. We also expanded physical evaluation to 50 trials per task–condition pair, confirming an 11.0-percentage-point OOD improvement (75.0% vs. 64.0%). Additionally, we evaluated DRIVE combined with Clip-Higher to examine the complementarity of their exploration mechanisms.
  4. Theoretical Formulation. We further discussed the terminal-boundary concern and the assumptions underlying potential-based reward shaping, alongside the adaptive nature of group-relative diversity incentives during on-policy optimization.

Summary of Manuscript Revisions.

  • Main Text: Added diversity-reward controls and OOD success-conditioning ablations (Sec. 5.2, Fig. 6), EEF visualizations and execution-length analyses (Fig. 5), expanded physical evaluation (Sec. 5.3), and evaluation standard deviations (Table 1).
  • Appendix C.1: Added the formal Coverage@K definition and matching rule, together with detailed fixed-budget and equal-success Coverage/Vendi analyses and sampling-cost comparisons.
  • Appendix C.2: Added detailed execution-length statistics and EEF trajectory visualizations across training checkpoints.
  • Appendix E: Clarified independent training-run counts, the interpretation of evaluation variability, and checkpoint and hyperparameter selection protocols.
  • Presentation: Improved citation navigation and refined table formatting and caption spacing.

Overall, we believe these discussions and revisions have strengthened the empirical foundation of DRIVE and clarified the scope and interpretation of our findings. We hope that our responses and revised manuscript provide a more comprehensive basis for evaluating the contributions of this work.

We sincerely appreciate your time, constructive feedback, and careful consideration.

Best regards,

Authors of Submission 33188

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.