acceptodds
Under review as a conference paper at ICLR 2027

Stochastic Self-Distillation for Vision-Language-Action Models

Abstract

Flow-based vision–language–action (VLA) models achieve strong performance in robotics through imitation learning, yet fine-tuning them with on-policy reinforcement learning (RL) is limited. In group-based RL methods like Group Relative Policy Optimization (GRPO), every rollout step requires multi-step ODE integration, and the sampler's limited stochasticity leads to rollout groups with low variance. We propose VLA-SSD, which addresses both high inference latency and limited stochasticity. In particular, our method self-distills a pretrained flow-matching action head into a stochastic one-step policy by conditioning a single-jump generator on a Brownian path. We derive a conditional action density via our stochastic one-step policy, enabling efficient post-training via likelihood-ratio RL algorithms. We show that distilling VLA-SSD from the reduces the action head's forward-pass time by times, with a relative decrease in LIBERO success rate. Leveraging the increased sample diversity of VLA-SSD during group generation, we show that post-training with GRPO from a few-shot checkpoint raises LIBERO success from 77.1% to 93.0%, achieving a relative improvement over four-step GRPO.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Flowmatching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_a8v2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes VLA-SSD, a stochastic self-distillation method for flow-matching VLA models. Specifically, it employs the Itô mapping to distill multi-step flow-matching action heads into a single-step stochastic policy conditioned on Brownian paths. This method not only reduces the number of forward passes required for each action but also handle the issue where the stochasticity comes only from the initial noise of the existing sampler. The overall training can be divided into two stages: self-distillation and GRPO. Experiments show that VLA-SSD significantly reduces the time required for reasoning and achieves competitive success rates.

Critical strengths
  1. The paper proposes VLA-SSD, a one-step generation method that substantially reduces the forward-pass time required for action generation in flow-matching VLAs. It only requires 7.6 ms compared with 24.5 ms for the SOTA method.

  2. The authors fully considers the characteristics of flow-match VLA and build the two-stage training pipeline of self-distillation and GRPO on these characteristics.

Critical weaknesses
  1. One of the major issues is the performance gap. Although VLA-SSD significantly reduces inference time, its success rate on LIBERO (single step, 93.0%) is still significantly lower than the current best result of 98.3% for four step πRL\pi_{RL} PPO.

  2. Further ablation experiments are needed to verify the effectiveness of the two-stage training. Please refer to Questions For Authors,

Questions for authors

VLA-SSD adopts two-stage training pipeline: starting from the $\pi$0.5 SFT checkpoint for self-distillation, and then performing GRPO. However, Table 1 shows that the success rate of the Itô student on LIBERO after self-distillation (68.6%) is significantly lower than the success rate of $\pi$0.5 SFT (77.1%). Could the author provide a reasonable explanation for this? This phenomenon also raises the question of whether two-stage training is necessary: if we skip self-distillation and use VLA-SSD's RL strategy directly on the $\pi$0.5 SFT checkpoint for training, can we achieve better results? I hope the author can provide the results of this experiment to demonstrate the necessity of self-distillation before RL training.

1 comment
AuthorsOpenReview ↗

Response to Reviewer a8v2

We are grateful to the reviewer for the time and effort they devoted to the paper. We particularly appreciate their observation that the two-stage training design “fully considers the characteristics of flow-match VLA” by pairing self-distillation with GRPO. We also thank them for noting that VLA-SSD “substantially reduces the forward-pass time required for action generation” while maintaining competitive success rates. Given the character limit, we first address the trade-off between LIBERO success and inference latency (W1), then discuss the role of self-distillation before GRPO (W2, Q1).

(W1) Performance and Inference Efficiency

We thank the reviewer for highlighting the performance gap on LIBERO. We agree that four-step πRL + PPO achieves the highest success rate in Table 1 (98.3%, compared with 93.0% for VLA-SSD). The two policies differ in both sampling depth and RL algorithm, so the 5.3-point gap alone cannot tell us how much of the difference is due to one-step generation.

To keep the RL algorithm fixed, we also compare VLA-SSD with four-step πRL trained with GRPO (Section 4.2, Table 1). The πRL success rates are from the published baselines, and the action-head timings are per chunk on one A100.

MethodStepsLIBERO success (%)Action-head time (ms)
πRL + PPO498.324.5
πRL + GRPO491.524.5
VLA-SSD + GRPO193.07.6

VLA-SSD has the lowest action-head latency of the three methods, taking 7.6 ms per chunk on one A100 versus 24.5 ms for both four-step πRL baselines. This gives a 3.2× action-head speedup while achieving 93.0% success on LIBERO.

(W2, Q1) Effect of Self-Distillation

We agree that it is important to isolate the contribution of self-distillation to one-step policy performance. Following the reviewer’s suggestion, we ran a one-step LIBERO ablation from the same π0.5 SFT checkpoint. We applied self-distillation before GRPO in one run and skipped it in the other, keeping the GRPO settings fixed. Both policies used one-step sampling with exploration noise turned off at evaluation.

Training pipelineLIBERO success before RL (%)LIBERO success at step 100 (%)
Self-distillation → one-step Itô + GRPO68.673.5
Direct one-step Itô + GRPO without distillation40.5552.05

The self-distilled policy started GRPO with a higher one-step success rate and retained a 21.45-percentage-point lead on LIBERO at the evaluation checkpoint. We have added the self-distillation ablation to Appendix E and summarize its main result in Section 4.4.

Concluding remarks

We thank the reviewer for drawing attention to the trade-off between LIBERO success and action-generation time and for suggesting the self-distillation ablation. The one-step policy achieves 93.0% success with the lowest action-head latency in the comparison. The new ablation in Appendix E shows that self-distillation gives the policy a stronger starting point and an advantage early in GRPO training. We hope these results address the reviewer’s concerns and would be happy to discuss any remaining questions. If the reviewer finds the new evidence convincing, we would be grateful if they would consider raising their score.

Reviewer_4YATOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper distills a flow-based VLA into a one-step stochastic Itô map and fine-tunes it with GRPO. The stochastic map retains variation across samples, while an auxiliary conditional-density formulation provides a policy score for learning. Experiments on LIBERO and MetaWorld evaluate distillation, rollout diversity, and reinforcement learning.

Critical strengths
  1. In the separate full-demonstration π0.5\pi_{0.5} experiment, the one-step student retains 96.55% success against the teacher’s 96.85% on LIBERO (Section 4.4). Table 1 reports action-head times of 24.5 ms for four steps and 7.6 ms for one step. These results support the feasibility of retaining task performance with one-step action generation.

  2. The diversity analysis helps explain which samplers provide useful GRPO groups. Table 2 includes deterministic, isotropic-noise, and Flow-SDE controls, and measures the fraction of rollout groups that provide a nonzero GRPO learning signal. The distinction between the exact auxiliary conditional density and the approximate score used in practice is also stated explicitly.

Critical weaknesses
  1. My main concern is the accuracy of the policy score during RL. Proposition 1 assumes that the drift Jacobian has spectral norm below one, while the implementation uses one inverse iteration and a first-order, single-probe log-determinant estimate (Appendix C). Appendix B reports small corrections at the start of RL and states that the Brownian pathway norm was not logged during RL. A ratio of one before the first policy update does not test the approximation after updates. The reported diagnostics therefore do not establish that the score remains accurate where GRPO uses it.

  2. The learning comparison does not isolate the benefit of the Itô map from that of using a one-step policy with added noise. Table 2 includes an isotropic-noise diversity control, but the paper does not report a matched GRPO training comparison. The πRL\pi_{\mathrm{RL}} results in Table 1 are taken from prior work. Appendix D gives settings per RL iteration, but not the total number of RL iterations or variation across training seeds. The 3.2×3.2\times timing improvement concerns action-head inference; end-to-end RL time, including policy scoring, is not reported.

Questions for authors
  1. At early, middle, and late RL checkpoints, how do the approximate likelihood ratios compare with a converged inverse and an explicitly computed Jacobian on small batches? Reporting Jacobian norms on those samples would also test whether the contraction assumption is plausible during training.

  2. Can the authors compare GRPO with the deterministic one-step map plus isotropic output noise under the same starting checkpoint and interaction budget? Reporting the total updates, training-seed variation, and end-to-end time, including policy scoring, would clarify the learning and efficiency gains.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4YAT

We are grateful to the reviewer for their careful reading of the paper and the effort they put into their feedback. We appreciate their observation that the results “support the feasibility of retaining task performance with one-step action generation” and that the diversity analysis “helps explain which samplers provide useful GRPO groups.” Given the character limit, we group the related weaknesses and questions. We first address policy scoring during RL (W1, Q1), then the output-noise control, training budget, and end-to-end timing (W2, Q2).

(W1, Q1) Accuracy of Policy Scoring During RL

We thank the reviewer for highlighting the importance of checking scoring accuracy after the policy changes. We agree that a likelihood ratio of one before the first update does not test this.

Following the reviewer’s suggestion, we tested the scoring approximation at early, middle, and late stages of GRPO training. For each pair of pre- and post-update policies, we replayed eight LIBERO action chunks and compared the approximate log-ratios recomputed in FP32 with reference values obtained using a converged inverse and an explicit Jacobian.

RL checkpointMean absolute log-ratio error (FP32)
Early1.81 × 10⁻⁴
Middle7.0 × 10⁻⁵
Late6.1 × 10⁻⁵

The mean absolute log-ratio error was below 0.0002 at all three checkpoints when the scores were recomputed in FP32. We have added this FP32 scoring accuracy analysis to Appendix C.

(W2, Q2) Comparison with Isotropic Output Noise

We thank the reviewer for raising this point. To incorporate this suggestion, we evaluated whether the Itô map offers any advantage during GRPO over simply adding isotropic Gaussian noise to the output of a one-step policy. Specifically, we trained the Itô policy and an output-noise control from the same distilled checkpoint using identical GRPO settings. In the control, we disabled the Brownian-path input and increment and added independent isotropic Gaussian noise to the output. We evaluated both policies with exploration noise turned off.

BenchmarkMethodSuccess at step 100 (%)
LIBEROItô + GRPO73.5
LIBEROOne-step map + isotropic noise + GRPO71.50
MetaWorld MT50Itô + GRPO59.40
MetaWorld MT50One-step map + isotropic noise + GRPO59.00

The Itô policy’s success rate exceeded that of the isotropic-noise control by 2.0 percentage points on LIBERO and 0.4 points on MT50. We have added this comparison to Section 4.3 of the revised manuscript.

(W2, Q2) Training Details and End-to-End Efficiency

We thank the reviewer for pointing out that we omitted the total number of RL updates and the end-to-end training time, including policy scoring.

Following the reviewer’s suggestion, we supplement the per-iteration settings in Appendix D with the full training budget and wall-clock time for the original GRPO runs. The first table lists the total iterations, optimizer updates, and rollouts collected on each benchmark.

BenchmarkTotal RL iterationsOptimizer updatesEnvironment interactions
LIBERO60015,00038,400 rollouts
MetaWorld MT503007,50076,800 rollouts

The table below reports end-to-end RL training times for the Itô runs summarized above and for the isotropic-noise control on both benchmarks.

End-to-end RL timeLIBEROMetaWorld MT50
Itô + GRPO33.3 h49.3 h
Isotropic noise + GRPO74.8 h72.1 h

Itô + GRPO took 41.5 fewer hours on LIBERO and 22.8 fewer hours on MT50 than the isotropic-noise control. We have added the training budgets and end-to-end timings to Appendix D of the revised manuscript.

Concluding remarks

We thank the reviewer for asking how accurately the policy is scored and what the Itô map contributes during training. The revised manuscript includes a check against reference log-ratios at three RL checkpoints and a direct comparison with isotropic output noise. The FP32 scoring check yielded a mean absolute error below 0.0002 at each checkpoint. The training comparison showed a larger advantage on LIBERO than on MT50. We also report the rollout and update counts alongside end-to-end training times. We welcome further questions and would appreciate the reviewer reconsidering their score in light of these results.

AuthorsOpenReview ↗

Follow-Up to Reviewer 4YAT

Dear Reviewer 4YAT,

We would greatly value hearing whether our response resolved the concerns you raised as the discussion deadline approaches. In brief, we added an FP32 scoring-accuracy check at early, middle, and late GRPO checkpoints against a converged inverse and explicit Jacobian, a matched isotropic output-noise GRPO control, and full training budgets plus end-to-end timings including policy scoring. Thank you for highlighting the importance of these diagnostics and for noting that the one-step results support the feasibility of retaining task performance with faster action generation.

Reviewer_MyXNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper targets on-policy RL fine-tuning of flow-based VLA models, whose multi-step ODE action heads are slow to sample and offer little rollout diversity for group-based RL. The goal is a policy that produces sufficiently stochastic actions in a single action-expert denoise per chunk.

  • Concretely, it first self-distills the flow-matching head into a one-step stochastic policy built on Itô flow maps, whose sampler conditions on a Brownian path. Second, it derives a conditional action density for this one-step stochastic policy (via an auxiliary-variable decomposition), which makes likelihood-ratio algorithms such as GRPO applicable for post-training.

  • On two standard simulation benchmarks (LIBERO and MetaWorld MT50), the authors report that one-step GRPO reaches 93.0% and 72.0% success, surpassing the multi-step πRL baselines (91.5% GRPO / 70.7% PPO), with a 7.7× faster action head and 3.2× faster training rollouts.

Critical strengths

S1. One-step RL fine-tuning for flow-based VLAs at little performance cost.

The paper formulates on-policy RL for flow-matching VLA models around a single action-expert denoise per chunk. This affects both training and deployment: rollouts are 3.2× faster during GRPO, and the trained policy generates action chunks with a 7.7× faster action head (58.4 → 7.6 ms). The reported cost is small: distillation from the full-data teacher loses 0.3 points (96.85 → 96.55), and after GRPO the one-step policy is comparable to the four-step πRL baselines (93.0 vs. 91.5 with GRPO on LIBERO; 72.0 vs. 70.7 with PPO on MetaWorld).

The recipe is self-contained: stochasticity is introduced internally via Brownian-path-conditioned Itô maps, one-step generation via diagonal (Lagrangian) self-distillation, and a derived conditional action density makes GRPO applicable — without an external noise network, a multi-step SDE chain, or a critic.

Critical weaknesses

W1 — The experimental validation is narrow in both baselines and settings.

The RL comparison rests almost entirely on the πRL work: the main-table baselines are πRL's published GRPO/PPO numbers, and no other RL fine-tuning method for flow-based VLAs is compared. Yet several recent approaches in this space (e.g., DPPO, ReinFlow, SimpleVLA-RL, FORCE) report strong results on LIBERO or MetaWorld; without comparisons against them, it is difficult to locate this method's advantage within the existing landscape.

In addition, validation is limited to two simulation benchmarks. Real-robot experiments — or at least additional environments (e.g., real-to-sim-to-real transfer settings) — would be needed to test whether the approach still works beyond these two simulators.

Questions for authors

Q1. Effect of the noise scale.

The main experiments use a fixed $ \sigma=0.1$. How sensitive are both task success and GRPO performance to $ \sigma $ ? In particular, is there a clear trade-off between exploration diversity and action-quality degradation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer MyXN

We are grateful to the reviewer for the time and effort they put into reviewing the paper. We appreciate their observation that VLA-SSD enables one-step RL fine-tuning “at little performance cost” and that the “self-contained” training recipe dispenses with an external noise network. Given the character limit, we address baseline comparisons and evaluation scope under W1, then report how the choice of noise scale affects GRPO under Q1.

(W1) Comparisons with Related RL Methods

We thank the reviewer for suggesting additional baselines. We agree that a comparison beyond πRL would strengthen the evaluation. We therefore added a four-step ReinFlow-style PPO baseline to compare one-step and multi-step flow policies. DPPO targets diffusion policies, SimpleVLA-RL uses a different action head, and FORCE uses a critic to guide policy updates. These differences make the three methods less directly comparable for this particular question, so we did not include them in the new experiment.

On each benchmark, VLA-SSD and the ReinFlow-style baseline started from the same π0.5 SFT checkpoint before method-specific training. We evaluated both policies on the same initial states.

BenchmarkMethodSampling steps per chunkSuccess Rate after GRPO (%)Action-head time (ms)
LIBEROVLA-SSD + GRPO173.57.6
LIBEROReinFlow-style PPO440.5568.5
MetaWorld MT50VLA-SSD + GRPO159.407.6
MetaWorld MT50ReinFlow-style PPO455.4080.0

VLA-SSD led the ReinFlow-style baseline by 32.95 percentage points on LIBERO and 4.0 points on MT50. Its action head took 7.6 ms per chunk on both benchmarks. We have added the four-step ReinFlow-style PPO baseline to Section 4.1 and report the comparison in Appendix F.

(W1) Evaluation Scope

We thank the reviewer for raising this point. We agree that testing on physical hardware or in a new environment would give a stronger test of transferability. We unfortunately do not have access to a physical robot during the rebuttal, Nevertheless, the evaluation spans 40 LIBERO tasks across four suites and 50 MetaWorld MT50 tasks on a second simulated robot embodiment. Evaluating VLA-SSD on physical robots remains an exciting direction for future work, as discussed in Section 6.

(Q1) Sensitivity to the Noise Scale

We thank the reviewer for raising this question and agree that it is important to test how sensitive GRPO performance is to the choice of noise scale. At σ=0.10\sigma = 0.10, the Itô policy produced fewer zero-variance GRPO groups on LIBERO than the deterministic map (35.4% vs. 56.2%, Table 2). On MT50, turning on exploration reduced the pre-RL student’s success from 49.0% to 36.8% (Section 4.3). Following the reviewer’s suggestion, we ran a sensitivity analysis in which we varied only the rollout noise scale during GRPO (σ∈{0.05,0.10,0.20}\sigma \in \{0.05, 0.10, 0.20\}). Each run on a given benchmark started from the same distilled checkpoint. We evaluated task success with exploration noise turned off.

Training σ\sigmaLIBERO success at step 100 (%)MetaWorld MT50 success at step 100 (%)
0.0569.1058.80
0.1073.559.40
0.2065.1555.20

Among the three noise scales tested, σ=0.10\sigma = 0.10 yielded the highest success rate on both LIBERO and MT50 after RL post-training using GRPO, supporting the choice of σ=0.10\sigma = 0.10 as the default. We have added this noise-scale sensitivity analysis to Section 4.3 of the revised manuscript.

Concluding remarks

We thank the reviewer for encouraging us to broaden the evaluation. In the revised manuscript, we added a four-step ReinFlow-style PPO baseline and tested three rollout noise scales. The default scale gave the highest success rate on both benchmarks after RL post-training using GRPO. We also clarified the scope of the simulation results and identified real-robot evaluation as a next step. We welcome any further questions and would be grateful if the reviewer would consider raising their score if these additions address their concerns.

AuthorsOpenReview ↗

Follow-Up to Reviewer MyXN

Dear Reviewer MyXN,

We would greatly value hearing whether our response and revised manuscript addressed your concerns as the discussion deadline approaches. In brief, we added a four-step ReinFlow-style PPO baseline starting from the same SFT checkpoint, a rollout-noise sensitivity study over three noise scales, and clarified the scope of the simulation evaluation while identifying real-robot validation as future work. Thank you for recognizing the value of one-step RL fine-tuning for flow-based VLAs and the self-contained nature of the VLA-SSD training recipe.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to the Area Chair and All Reviewers

We thank the reviewers (MyXN, 4YAT, a8v2) for their valuable feedback and the Area Chair for overseeing this process. As the discussion deadline approaches, we were wondering whether any further clarification would be helpful. We summarize the additions to our revised manuscript below.

1. Additional Experiments

  • Additional baseline (MyXN W1). We added a four-step ReinFlow-style PPO baseline starting from the same SFT checkpoint. Our one-step policy led by 32.95 percentage points on LIBERO and 4.0 points on MT50 (Appendix F).

  • Noise-scale sensitivity (MyXN Q1). We evaluated three rollout noise scales during GRPO, starting from the same distilled checkpoint on each benchmark. The default scale of 0.10 yielded the highest success on both benchmarks among the scales tested (Section 4.3).

  • Isotropic output-noise control (4YAT W2/Q2). We trained a one-step isotropic output-noise control from the same distilled checkpoint under identical GRPO settings. Our Itô policy outperformed this control by 2.0 percentage points on LIBERO and 0.4 points on MT50 (Section 4.3).

  • Policy-scoring accuracy during RL (4YAT W1/Q1). At early, middle, and late GRPO checkpoints, we compared approximate log-ratios against a converged inverse and an explicit Jacobian on eight LIBERO action chunks per checkpoint. The mean absolute error was below 0.0002 at each checkpoint when recomputed in FP32 (Appendix C).

  • Self-distillation ablation (a8v2 W2/Q1). We compared GRPO with and without prior self-distillation under matched settings. Self-distillation raised pre-RL one-step LIBERO success from 40.55% to 68.6%, and the distilled policy retained a 21.45-point lead (Appendix E).

2. Clarifications

  • Training budgets and end-to-end time (4YAT W2/Q2). We added total rollout counts, RL iterations, optimizer updates, and training times including policy scoring. Estimated end-to-end GRPO times for the Itô policy were 33.3 hours on LIBERO and 49.3 hours on MT50, compared with 74.8 and 72.1 hours for the isotropic output-noise control (Appendix D).

  • Role of self-distillation (a8v2 Q1). We expanded Appendix E to explain how self-distillation improves one-step action generation and provides a stronger starting point for GRPO, alongside the efficiency gains from one-step sampling.

  • Evaluation scope (MyXN W1). We clarified the scope of the two simulation benchmarks and acknowledged real-robot validation as future work (Section 6).

We hope our responses and the revised manuscript help clarify the reviewers’ concerns. We welcome any further comments or questions during the remaining days of the discussion period. Thank you again for your time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.