- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
StructRL proposes an RL fine-tuning method for flow-based vision-language-action models. Its central observation is that noise injected during denoising may lose its intended temporal and action-group structure before the action is executed. StructRL instead runs a deterministic ODE decoder, adds temporally correlated noise with state-dependent scales to the completed action chunk, and uses terminal replay to update the policy through only the final denoising steps. The paper evaluates three backbones on LIBERO and ManiSkill, reports long-horizon results on CALVIN, and tests two tasks on a Franka robot. The results suggest that moving exploration to the action output accounts for most of the OOD gain, while structuring that noise chiefly improves learning speed; final success-rate gains are less consistent.
Critical strengths
- The paper shows that structured noise inserted within the denoising chain can have little effect on the executed action’s statistics. This motivates a simple change in where exploration is applied and makes the work relevant beyond its particular noise parameterization.
- Results span three VLA backbones, several simulation benchmarks, and two robot tasks. Comparing StructRL with output-end Gaussian noise helps distinguish the benefit of moving exploration to the action output from the added benefit of structuring the noise.
Critical weaknesses
- The cached intermediate state was produced by the old policy and depends on its parameters. Holding it fixed when evaluating the new policy yields a local surrogate, so the resulting PPO ratio and clipping need not describe changes in the full action policy. This affects the interpretation of the central optimization method.
- Gradients from the executed action and reward pass through only the final n steps, although earlier steps influence that action. Because parameters are shared across steps, updates can also change the omitted trajectory. The method provides no criterion for when this missing gradient is negligible.
- Independent per-dimension AR(1) processes with learned scales cannot represent coordinated perturbations across action dimensions, and correlation is specified only within a chunk. This limits the method for coupled position–rotation control and for continuity across chunk boundaries.
Questions for authors
- Which structured-noise component produces the learning-speed gain? Can the authors separate the effects of AR(1) correlation, learned scales, group-specific bounds, and scale regularization under matched noise budgets?
- Given the mixed final success rates, can the authors report matched-budget learning-curve summaries across backbones and tasks, with uncertainty, rather than relying chiefly on the LIBERO-Long ablation?
- Figure 3(c) appears to show faster learning for a matched n=2 setting. What determines the best n, and what precisely changes in the corresponding “last-step” versus “second-last-step” inference settings?
- The robot implementation clips commands before execution, whereas the derived likelihood is for additive continuous noise before those constraints. Once clipping occurs, the distribution of executed actions changes. The paper does not explain how this is handled in actor optimization, which matters when exploratory actions reach the bounds.