- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper analyzes how RL post-training modifies flow-based VLA action experts, identifying low-rank updates in timestep-conditioning modules. Through module replacement and representation analyses, it finds that these updates preserve most RL gains in π₀.₅ and that shift-update directions encode success-related information. Low-rank updates are observed across multiple VLA models, and activation steering further improves success rates in π₀.₅ by 2~4 percentage points.
Critical strengths
-
The paper identifies low-rank RL updates in timestep-conditioning modules across multiple VLA models. This provides a concrete parameter-level characterization of VLA post-training and suggests opportunities for more parameter-efficient adaptation.
-
In π₀.₅, module-replacement experiments preserve most RL gains using only timestep-module updates, while shift-based activation steering further improves success rates.
Critical weaknesses
-
The central claim about where RL gains reside is overgeneralized from π₀.₅. In GR00T N1.5, retaining only timestep-module updates reduces success from 90.3% to 61.3%. In GR00T N1.6, evaluated only on LIBERO-Spatial, retaining Attention+FFN updates performs almost identically to retaining timestep-module updates. These results do not establish the claimed dominance of timestep modules across VLA architectures. The paper does not adequately explain this discrepancy or identify the conditions under which its central conclusion holds.
-
The proposed explanation for low-rank updates lacks a direct intervention in RL. The paper attributes the low-rank structure primarily to training on fixed, discrete denoising timesteps, but supports this explanation by restricting BC to discrete timesteps rather than varying the timestep schedule during RL. A controlled RL comparison using randomized denoising schedules is missing, and the corresponding mechanism analysis is largely limited to π₀.₅. Consequently, the evidence does not adequately distinguish a general property of VLA RL from an effect of the particular denoising schedule, limiting the explanatory scope of the paper.
Questions for authors
-
In GR00T N1.5, retaining only timestep-module updates preserves approximately 19% of the RL improvement. In GR00T N1.6, retaining only Attention+FFN updates achieves nearly identical performance to retaining only timestep-module updates. Do these results suggest that the paper overgeneralizes findings from π₀.₅? Please also clarify why the GR00T N1.6 evaluation is restricted to LIBERO-Spatial.
-
Have you conducted RL experiments with denoising timesteps randomized across rollouts while keeping the number of denoising steps and training budget comparable? If not, beyond showing that discrete-timestep BC also produces low-rank updates, what evidence establishes fixed-timestep training as the primary cause of low-rank RL updates?
-
For each analyzed checkpoint, was the VLM backbone frozen during RL, or were all VLA parameters trained? Does “full-parameter RL” refer to the entire action expert or the entire VLA?
-
Are the 2~4 percentage improvements from steering reproducible across independent runs, and what are their confidence intervals? Was the steering target, the mean projection value of successful episodes, computed exclusively from rollouts on training tasks and then held fixed during testing?
7 comments
Official Rebuttal (2/4)
Implementation notes: We trained GR00T PPO in-house to limit numerical rounding effects in the update-rank analysis. The π0.5 checkpoints we analyzed maintain Timestep Module weights in FP32, whereas the public RLinf GR00T PPO configuration [1] we inspected updates them directly in BF16. Entrywise rounding can disrupt the linear dependencies within an update matrix and inflate its measured rank, making it difficult to distinguish learned structure from numerical artifacts. We therefore maintained and saved the action-head weights in FP32, with FP32 optimizer states, without imposing any low-rank constraint.
Q1. ...Please also clarify why the GR00T N1.6 evaluation is restricted to LIBERO-Spatial.
W2. … and the corresponding mechanism analysis is largely limited to π₀.₅….
Scope of our analyses.
The publicly released GR00T N1.6 BC/RL checkpoint pair covers only LIBERO-Spatial (Appendix B.1, Table 5). Going beyond this pair requires training GR00T ourselves, including RL on the remaining suites and, for the mechanism analysis, BC, discrete-timestep BC, and RL variants from the same initialization. The released checkpoints were trained for 400 to 500 RL steps per suite. On our setup of four NVIDIA H100 80GB GPUs with an Intel Xeon Platinum 8480+ (Appendix B), even 150 steps take about 200 GPU-hours per suite, so matching the released budget on the two remaining suites alone would take roughly two weeks of continuous training. This was not feasible within the rebuttal period, so we relied on the released checkpoint pair and conducted the mechanism analysis mainly on π0.5. Accordingly, we softened our claims in the revision to avoid over-generalizing from π0.5, mainly in Section 3.2.
Official Rebuttal (3/4)
W2: The proposed explanation for low-rank updates lacks a direct intervention in RL. The paper attributes the low-rank structure primarily to training on fixed, discrete denoising timesteps, but supports this explanation by restricting BC to discrete timesteps rather than varying the timestep schedule during RL. A controlled RL comparison using randomized denoising schedules is missing, and the corresponding mechanism analysis is largely limited to π₀.₅. Consequently, the evidence does not adequately distinguish a general property of VLA RL from an effect of the particular denoising schedule, limiting the explanatory scope of the paper.
Q2: Have you conducted RL experiments with denoising timesteps randomized across rollouts while keeping the number of denoising steps and training budget comparable? If not, beyond showing that discrete-timestep BC also produces low-rank updates, what evidence establishes fixed-timestep training as the primary cause of low-rank RL updates?
Randomized-schedule RL.
Following the reviewer's suggestion, we train π0.5 with PPO on LIBERO-Spatial under a randomized denoising schedule, keeping all other settings identical to our fixed-schedule run. Since on-policy RL requires actual rollouts, we still generate them with the default K=5 schedule, but at every PPO step we shift each of the interior training timesteps uniformly within ±0.1 of its default value. For example, instead of the fixed timesteps {0, 0.2, 0.4, 0.6, 0.8}, one PPO step can be trained on {0.09, 0.12, 0.32, 0.62, 0.71}. Because the default timesteps are spaced 0.2 apart, these ±0.1 windows are adjacent without overlap, so training covers a continuous range of timesteps across PPO steps, similar to BC. As in BC, evaluation uses the default K=5 schedule.
| Base | Fixed RL | Randomized RL | |
|---|---|---|---|
| Accuracy | 0.850 | 0.966 | 0.926 |
Table R2. Accuracy on LIBERO-Spatial.
| Component | BC | Fixed RL | Randomized RL |
|---|---|---|---|
| Time MLP | 629 | 6.0 | 139 |
| AdaRMS | 656 | 30.1 | 295.2 |
| Attention | 1,907 | 1,810 | 1,834.2 |
| MLP | 2,539 | 2,510 | 2,528.6 |
Table R3. Effective rank of parameter updates.
Randomizing the training timesteps raises the effective rank of the Timestep Module updates, consistent with our discrete-timestep BC result (Section 4.1), suggesting that discrete-timestep training contributes to the low-rank structure. Nevertheless, the rank remains well below that of BC, suggesting that discretization is not the only factor and that the on-policy nature of RL may also contribute. One possible explanation, also discussed in the LLM RL literature, is that scalar reward feedback on self-generated samples provides more limited supervision than expert demonstrations, potentially allowing lower-capacity updates to suffice [1].
Importantly, flow-based VLAs typically generate actions with a small number of Euler steps on a fixed timestep grid, which keeps action generation consistent at deployment. On-policy RL trains on self-generated rollouts using these same timesteps, so the policy learns under the conditions in which it will be used. In contrast, the training timesteps in BC need not correspond to the denoising timesteps used at inference, because it learns from fixed demonstrations and constructs flow-matching training examples by interpolating between demonstrated actions and sampled noise at any timestep. Furthermore, our controlled comparison supports keeping the training schedule consistent with the rollout schedule: randomizing the training timesteps increases the effective rank while decreasing the success rate (Table R2 above). The low-rank structure can therefore be understood as a property of VLA RL under a practically motivated and empirically supported training setup, rather than an arbitrary scheduling choice. We added these results to Section 4.1. and Appendix C.8
[1] John Schulman et al, LoRA Without Regret 2025
Official Rebuttal (1/4)
We thank the reviewer for finding our characterization of low-rank Timestep Module updates across multiple VLAs concrete, and for noting its potential for parameter-efficient adaptation.
We added RL with randomized denoising schedules, a closer analysis of GR00T, confidence intervals for steering over independent runs, and clarifications of the trained parameters and evaluation split. We revised the manuscript accordingly, with changes highlighted in blue.
Generalization to GR00T.
As the reviewer points out, transferring only the Timestep Module updates to the base policy is less effective in GR00T than in π0.5. To first explain why this happens in GR00T, this difference may primarily reflect how the two architectures process timestep information, rather than indicate that low-rank Timestep Module updates are unimportant in GR00T. In π0.5, timestep information enters the action expert only through the Timestep Modules. In GR00T, timestep embeddings are also concatenated with action token embeddings, allowing subsequent Attention and MLP layers to process timestep information (Appendix A.2). The Timestep Module updates may therefore need to work together with RL updates to these layers to produce the full gain. We therefore qualify our original claim: while Timestep Modules are not necessarily the dominant contributors to RL gains in GR00T, a small number of low-rank components in their updates still make a critical contribution when combined with RL updates to other modules.
To support this more specific claim, we conducted an additional experiment that tests whether low-rank Timestep Module updates remain important for GR00T’s RL gains when RL updates to the other components are preserved. On GR00T N1.5 in LIBERO-Spatial, we started from an in-house PPO policy (see Implementation notes below). We either retained only the leading rank-1 component of each Timestep Module weight update or removed that component while retaining the remaining update. We also evaluated removing all Timestep Module updates. All other parameters remained at their RL-trained values.
Table R1. Effect of low-rank Timestep Module updates on GR00T N1.5.
With all other RL updates preserved, retaining only these rank-1 components achieves 77.8% success, compared with 69.8% for full PPO. Removing these rank-1 components reduces success to 33.8%, the same success rate as removing all Timestep Module updates. Both conditions perform worse than the base policy (37.0%). This supports our finding that low-rank Timestep Module updates contribute substantially to RL improvement beyond π0.5, while their full effect may depend on updates to other components that process timestep information.
We thank the reviewer for pointing this out and agree that our wording about which components account for most of the RL gain should be more carefully qualified. We have incorporated this architectural interpretation into the revised manuscript (Section 3.2, Appendix A.2), and revised our wording to avoid implying that Timestep Modules are the dominant contributors across all architectures (Introduction, Section 3.2).