acceptodds
Withdrawn from ICLR 2027

StructRL: Structured Action-Space Exploration for Flow-Based VLAs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Flow-based Vision-Language-Action (VLA) models are now widely used for continuous robotic manipulation, and online reinforcement learning (RL) is emerging as a key technique for adapting them to new tasks. Existing RL methods typically inject stochasticity inside the denoising chain, often through isotropic or temporally independent noise. However, effective robot exploration calls for structured noise: temporally smooth and scaled differently across action groups. We show that simply switching the in-chain noise to a structured form does not suffice: noise added at an intermediate flow time can be weakened by the remaining denoising steps before execution, a phenomenon we call Structured Noise Dilution. We propose StructRL, which avoids dilution by relocating policy stochasticity to the action space via three coupled choices: (i) a deterministic ODE decoder, (ii) structured noise injected directly in the action space, and (iii) terminal replay, which avoids assigning likelihoods to intermediate denoising states while shortening the policy-gradient path. This keeps structured exploration tied to the executed action while providing a tractable training signal for the flow decoder. Across three flow-based VLA models on multiple simulated manipulation benchmarks and two real-world tasks, StructRL improves exploration efficiency and OOD performance over prior in-chain baselines, demonstrating the effectiveness of structured action-space exploration for adapting flow-based VLA with RL.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Flow-based VLA, Reinforcement Learning, Structured Exploration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qPCjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

StructRL proposes an RL fine-tuning method for flow-based vision-language-action models. Its central observation is that noise injected during denoising may lose its intended temporal and action-group structure before the action is executed. StructRL instead runs a deterministic ODE decoder, adds temporally correlated noise with state-dependent scales to the completed action chunk, and uses terminal replay to update the policy through only the final denoising steps. The paper evaluates three backbones on LIBERO and ManiSkill, reports long-horizon results on CALVIN, and tests two tasks on a Franka robot. The results suggest that moving exploration to the action output accounts for most of the OOD gain, while structuring that noise chiefly improves learning speed; final success-rate gains are less consistent.

Critical strengths
  1. The paper shows that structured noise inserted within the denoising chain can have little effect on the executed action’s statistics. This motivates a simple change in where exploration is applied and makes the work relevant beyond its particular noise parameterization.
  2. Results span three VLA backbones, several simulation benchmarks, and two robot tasks. Comparing StructRL with output-end Gaussian noise helps distinguish the benefit of moving exploration to the action output from the added benefit of structuring the noise.
Critical weaknesses
  1. The cached intermediate state was produced by the old policy and depends on its parameters. Holding it fixed when evaluating the new policy yields a local surrogate, so the resulting PPO ratio and clipping need not describe changes in the full action policy. This affects the interpretation of the central optimization method.
  2. Gradients from the executed action and reward pass through only the final n steps, although earlier steps influence that action. Because parameters are shared across steps, updates can also change the omitted trajectory. The method provides no criterion for when this missing gradient is negligible.
  3. Independent per-dimension AR(1) processes with learned scales cannot represent coordinated perturbations across action dimensions, and correlation is specified only within a chunk. This limits the method for coupled position–rotation control and for continuity across chunk boundaries.
Questions for authors
  1. Which structured-noise component produces the learning-speed gain? Can the authors separate the effects of AR(1) correlation, learned scales, group-specific bounds, and scale regularization under matched noise budgets?
  2. Given the mixed final success rates, can the authors report matched-budget learning-curve summaries across backbones and tasks, with uncertainty, rather than relying chiefly on the LIBERO-Long ablation?
  3. Figure 3(c) appears to show faster learning for a matched n=2 setting. What determines the best n, and what precisely changes in the corresponding “last-step” versus “second-last-step” inference settings?
  4. The robot implementation clips commands before execution, whereas the derived likelihood is for additive continuous noise before those constraints. Once clipping occurs, the distribution of executed actions changes. The paper does not explain how this is handled in actor optimization, which matters when exploratory actions reach the bounds.
Reviewer_3cVwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StructRL for online RL adaptation of flow-based VLAs. The method uses a deterministic ODE decoder, injects temporally correlated and action-group-scaled noise directly into the decoded action chunk, and applies terminal replay to construct a truncated likelihood-based PPO/AWAC objective.

Critical strengths
  • The paper addresses an important exploration-design problem for robot action chunks. Temporally independent, uniformly scaled noise is poorly matched to smooth trajectories and to action groups with different ranges and safety sensitivities.
  • The full latent-conditioned policy construction is a meaningful technical contribution: when the initial latent comes from a parameter-independent prior, its density cancels in the joint policy ratio.
Critical weaknesses
  • The central Structured Noise Dilution claim is not measured directly. The reported final-action correlations and standard deviations mix the clean action, latent-induced variability, and injected noise. A paired effective perturbation F(x+Delta)-F(x) would isolate whether the remaining flow preserves the intended covariance. Endpoint noise retaining its own structure is largely true by construction, injection time is under-specified, and no Jacobian/covariance analysis shows that a general flow must dilute structured noise.
  • Terminal replay is not an exact likelihood ratio for the complete current policy when the replay depth is shorter than the denoising chain. The cached terminal state was produced by the old, parameter-dependent policy, upstream gradient paths are dropped, and the hidden representation is detached before scale prediction.
  • The empirical evidence lacks the uncertainty and ablations needed to isolate the proposed mechanism. Main simulation tables and curves omit training seeds, confidence intervals, and error bands; the real-robot result is based on limited trials.
Questions for authors

See above

Reviewer_UyPNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work presents StructRL, a structured exploration framework designed for flow-based vision-language-action models. Rather than introducing stochasticity into the intermediate denoising trajectory, the proposed approach applies structured perturbations directly to the final action output, aiming to mitigate the issue referred to as "Structured Noise Dilution." The proposed method is evaluated across three VLA backbones, multiple simulation benchmarks, and two real-world manipulation settings.

Critical strengths
  • The proposed approach offers a relatively straightforward way to introduce exploration into pretrained flow-based policies. In particular, it avoids modifying the underlying flow architecture or introducing stochasticity at every denoising step, which could make the method easier to integrate into existing VLA pipelines.

  • The authors evaluate StructRL across multiple VLA architectures and simulation environments, and further demonstrate its applicability through real-world robotic manipulation experiments. This provides a reasonable level of empirical coverage.

Critical weaknesses
  • The central mechanism of StructRL, namely perturbing the output of a pretrained policy and subsequently optimizing the perturbed policy with RL, bears considerable similarity to existing residual policy learning and action-space exploration methods. More importantly, NCDPO (arXiv:2505.10482) has already investigated a closely related formulation, treating a generative policy as a noise-conditioned deterministic policy and performing likelihood evaluation directly in the final action space. Given this substantial conceptual overlap, the paper needs to provide a more rigorous discussion of how StructRL differs from these existing approaches. In particular, it remains unclear what fundamental methodological insights or new capabilities are introduced beyond adapting structured action-space perturbations to flow-based VLAs.

  • Restricting policy updates to the final denoising stages is conceptually related to prior methods such as DPPO, which also investigates fine-tuning only a subset of the denoising process. Although the specific implementation may differ, the paper does not sufficiently explain why Terminal Replay is particularly necessary in the proposed framework or what advantages it offers over existing partial denoising optimization strategies. Furthermore, the approximation introduced by replaying only terminal steps is not thoroughly examined. It would be valuable to compare this strategy against full ODE backpropagation and systematically investigate the impact of different replay depths. Such experiments would help establish both the validity of the approximation and the practical benefits of the proposed design.

Questions for authors
  • How does StructRL ensure that perturbations applied directly to the final action output remain consistent with the pretrained policy’s learned behavior and the task’s physical constraints? Could these perturbations lead to unstable execution or physically implausible trajectories, particularly in precision-sensitive manipulation tasks?

  • Can the authors provide empirical evidence quantifying the effects of perturbation magnitude on action feasibility and execution reliability, and clarify how these effects compare with exploration introduced through an SDE formulation of the original ODE dynamics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.