acceptodds
Under review as a conference paper at ICLR 2027

The Low-Rank Structure of VLA Reinforcement Learning

Abstract

Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models, yet how RL reshapes these policies remains poorly understood. We find that RL across widely used flow-based VLA models, including and GR00T N1.5/N1.6, on LIBERO, ManiSkill, MetaWorld, and CALVIN induces substantially lower-rank parameter updates that are highly concentrated in the action expert's Timestep Modules, a small and previously overlooked component. Through systematic module-replacement experiments, we further show that these modules capture a disproportionate share of the performance gains from RL: in , keeping only their RL updates yields performance within 1.6% of the full RL policy despite comprising only 27.6% of the action expert's parameters. We then characterize what is encoded in these Timestep Modules. First, we show that RL specializes them to the discrete denoising timesteps used during rollouts, and that this discrete-timestep training underlies the low-rank updates. Second, we find that among their outputs, the shift vector changes most distinctly under RL, and through probing, we show that shift update directions strongly predict task success (ROC-AUC up to 99.6%). Third, we find that the geometry of shift updates reflects task relationships, as their pairwise similarity correlates with cross-task transfer patterns. Building on these findings, we show that steering along shift update directions further improves RL-trained policies without additional RL training. Overall, we provide a systematic understanding of how RL reshapes VLA policies by studying how learned signals are encoded in parameter space, offering insights into more efficient and interpretable VLA post-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language-action models, reinforcement learning, interpretability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VYAgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes how RL post-training modifies flow-based VLA action experts, identifying low-rank updates in timestep-conditioning modules. Through module replacement and representation analyses, it finds that these updates preserve most RL gains in π₀.₅ and that shift-update directions encode success-related information. Low-rank updates are observed across multiple VLA models, and activation steering further improves success rates in π₀.₅ by 2~4 percentage points.

Critical strengths
  • The paper identifies low-rank RL updates in timestep-conditioning modules across multiple VLA models. This provides a concrete parameter-level characterization of VLA post-training and suggests opportunities for more parameter-efficient adaptation.

  • In π₀.₅, module-replacement experiments preserve most RL gains using only timestep-module updates, while shift-based activation steering further improves success rates.

Critical weaknesses
  • The central claim about where RL gains reside is overgeneralized from π₀.₅. In GR00T N1.5, retaining only timestep-module updates reduces success from 90.3% to 61.3%. In GR00T N1.6, evaluated only on LIBERO-Spatial, retaining Attention+FFN updates performs almost identically to retaining timestep-module updates. These results do not establish the claimed dominance of timestep modules across VLA architectures. The paper does not adequately explain this discrepancy or identify the conditions under which its central conclusion holds.

  • The proposed explanation for low-rank updates lacks a direct intervention in RL. The paper attributes the low-rank structure primarily to training on fixed, discrete denoising timesteps, but supports this explanation by restricting BC to discrete timesteps rather than varying the timestep schedule during RL. A controlled RL comparison using randomized denoising schedules is missing, and the corresponding mechanism analysis is largely limited to π₀.₅. Consequently, the evidence does not adequately distinguish a general property of VLA RL from an effect of the particular denoising schedule, limiting the explanatory scope of the paper.

Questions for authors
  1. In GR00T N1.5, retaining only timestep-module updates preserves approximately 19% of the RL improvement. In GR00T N1.6, retaining only Attention+FFN updates achieves nearly identical performance to retaining only timestep-module updates. Do these results suggest that the paper overgeneralizes findings from π₀.₅? Please also clarify why the GR00T N1.6 evaluation is restricted to LIBERO-Spatial.

  2. Have you conducted RL experiments with denoising timesteps randomized across rollouts while keeping the number of denoising steps and training budget comparable? If not, beyond showing that discrete-timestep BC also produces low-rank updates, what evidence establishes fixed-timestep training as the primary cause of low-rank RL updates?

  3. For each analyzed checkpoint, was the VLM backbone frozen during RL, or were all VLA parameters trained? Does “full-parameter RL” refer to the entire action expert or the entire VLA?

  4. Are the 2~4 percentage improvements from steering reproducible across independent runs, and what are their confidence intervals? Was the steering target, the mean projection value of successful episodes, computed exclusively from rollouts on training tasks and then held fixed during testing?

7 comments
AuthorsOpenReview ↗

Official Rebuttal (1/4)

We thank the reviewer for finding our characterization of low-rank Timestep Module updates across multiple VLAs concrete, and for noting its potential for parameter-efficient adaptation.

We added RL with randomized denoising schedules, a closer analysis of GR00T, confidence intervals for steering over independent runs, and clarifications of the trained parameters and evaluation split. We revised the manuscript accordingly, with changes highlighted in blue.


W1. The central claim about where RL gains reside is overgeneralized from π₀.₅. In GR00T N1.5, retaining only timestep-module updates reduces success from 90.3% to 61.3%. In GR00T N1.6, evaluated only on LIBERO-Spatial, retaining Attention+FFN updates performs almost identically to retaining timestep-module updates. These results do not establish the claimed dominance of timestep modules across VLA architectures. The paper does not adequately explain this discrepancy or identify the conditions under which its central conclusion holds.

Q1: In GR00T N1.5, retaining only timestep-module updates preserves approximately 19% of the RL improvement. In GR00T N1.6, retaining only Attention+FFN updates achieves nearly identical performance to retaining only timestep-module updates. Do these results suggest that the paper overgeneralizes findings from π₀.₅? Please also clarify why the GR00T N1.6 evaluation is restricted to LIBERO-Spatial.

Generalization to GR00T.

As the reviewer points out, transferring only the Timestep Module updates to the base policy is less effective in GR00T than in π0.5. To first explain why this happens in GR00T, this difference may primarily reflect how the two architectures process timestep information, rather than indicate that low-rank Timestep Module updates are unimportant in GR00T. In π0.5, timestep information enters the action expert only through the Timestep Modules. In GR00T, timestep embeddings are also concatenated with action token embeddings, allowing subsequent Attention and MLP layers to process timestep information (Appendix A.2). The Timestep Module updates may therefore need to work together with RL updates to these layers to produce the full gain. We therefore qualify our original claim: while Timestep Modules are not necessarily the dominant contributors to RL gains in GR00T, a small number of low-rank components in their updates still make a critical contribution when combined with RL updates to other modules.

To support this more specific claim, we conducted an additional experiment that tests whether low-rank Timestep Module updates remain important for GR00T’s RL gains when RL updates to the other components are preserved. On GR00T N1.5 in LIBERO-Spatial, we started from an in-house PPO policy (see Implementation notes below). We either retained only the leading rank-1 component of each Timestep Module weight update or removed that component while retaining the remaining update. We also evaluated removing all Timestep Module updates. All other parameters remained at their RL-trained values.

ConditionSuccess (%)
Base37.0
Full PPO69.8
PPO: Time top-1 retained77.8
PPO: Time top-1 removed33.8
PPO: all Time updates removed (Attn+FFN only)33.8

Table R1. Effect of low-rank Timestep Module updates on GR00T N1.5.

With all other RL updates preserved, retaining only these rank-1 components achieves 77.8% success, compared with 69.8% for full PPO. Removing these rank-1 components reduces success to 33.8%, the same success rate as removing all Timestep Module updates. Both conditions perform worse than the base policy (37.0%). This supports our finding that low-rank Timestep Module updates contribute substantially to RL improvement beyond π0.5, while their full effect may depend on updates to other components that process timestep information.

We thank the reviewer for pointing this out and agree that our wording about which components account for most of the RL gain should be more carefully qualified. We have incorporated this architectural interpretation into the revised manuscript (Section 3.2, Appendix A.2), and revised our wording to avoid implying that Timestep Modules are the dominant contributors across all architectures (Introduction, Section 3.2).

AuthorsOpenReview ↗

Official Rebuttal (2/4)

Implementation notes: We trained GR00T PPO in-house to limit numerical rounding effects in the update-rank analysis. The π0.5 checkpoints we analyzed maintain Timestep Module weights in FP32, whereas the public RLinf GR00T PPO configuration [1] we inspected updates them directly in BF16. Entrywise rounding can disrupt the linear dependencies within an update matrix and inflate its measured rank, making it difficult to distinguish learned structure from numerical artifacts. We therefore maintained and saved the action-head weights in FP32, with FP32 optimizer states, without imposing any low-rank constraint.

[1] https://github.com/RLinf/RLinf/blob/df0303eed32f1ea456df5e792a9d5309dd62fb93/examples/embodiment/config/libero_spatial_ppo_gr00t.yaml


Q1. ...Please also clarify why the GR00T N1.6 evaluation is restricted to LIBERO-Spatial.

W2. … and the corresponding mechanism analysis is largely limited to π₀.₅….

Scope of our analyses.

The publicly released GR00T N1.6 BC/RL checkpoint pair covers only LIBERO-Spatial (Appendix B.1, Table 5). Going beyond this pair requires training GR00T ourselves, including RL on the remaining suites and, for the mechanism analysis, BC, discrete-timestep BC, and RL variants from the same initialization. The released checkpoints were trained for 400 to 500 RL steps per suite. On our setup of four NVIDIA H100 80GB GPUs with an Intel Xeon Platinum 8480+ (Appendix B), even 150 steps take about 200 GPU-hours per suite, so matching the released budget on the two remaining suites alone would take roughly two weeks of continuous training. This was not feasible within the rebuttal period, so we relied on the released checkpoint pair and conducted the mechanism analysis mainly on π0.5. Accordingly, we softened our claims in the revision to avoid over-generalizing from π0.5, mainly in Section 3.2.

AuthorsOpenReview ↗

Official Rebuttal (3/4)

W2: The proposed explanation for low-rank updates lacks a direct intervention in RL. The paper attributes the low-rank structure primarily to training on fixed, discrete denoising timesteps, but supports this explanation by restricting BC to discrete timesteps rather than varying the timestep schedule during RL. A controlled RL comparison using randomized denoising schedules is missing, and the corresponding mechanism analysis is largely limited to π₀.₅. Consequently, the evidence does not adequately distinguish a general property of VLA RL from an effect of the particular denoising schedule, limiting the explanatory scope of the paper.

Q2: Have you conducted RL experiments with denoising timesteps randomized across rollouts while keeping the number of denoising steps and training budget comparable? If not, beyond showing that discrete-timestep BC also produces low-rank updates, what evidence establishes fixed-timestep training as the primary cause of low-rank RL updates?

Randomized-schedule RL.

Following the reviewer's suggestion, we train π0.5 with PPO on LIBERO-Spatial under a randomized denoising schedule, keeping all other settings identical to our fixed-schedule run. Since on-policy RL requires actual rollouts, we still generate them with the default K=5 schedule, but at every PPO step we shift each of the interior training timesteps uniformly within ±0.1 of its default value. For example, instead of the fixed timesteps {0, 0.2, 0.4, 0.6, 0.8}, one PPO step can be trained on {0.09, 0.12, 0.32, 0.62, 0.71}. Because the default timesteps are spaced 0.2 apart, these ±0.1 windows are adjacent without overlap, so training covers a continuous range of timesteps across PPO steps, similar to BC. As in BC, evaluation uses the default K=5 schedule.

BaseFixed RLRandomized RL
Accuracy0.8500.9660.926

Table R2. Accuracy on LIBERO-Spatial.

ComponentBCFixed RLRandomized RL
Time MLP6296.0139
AdaRMS65630.1295.2
Attention1,9071,8101,834.2
MLP2,5392,5102,528.6

Table R3. Effective rank of parameter updates.

Randomizing the training timesteps raises the effective rank of the Timestep Module updates, consistent with our discrete-timestep BC result (Section 4.1), suggesting that discrete-timestep training contributes to the low-rank structure. Nevertheless, the rank remains well below that of BC, suggesting that discretization is not the only factor and that the on-policy nature of RL may also contribute. One possible explanation, also discussed in the LLM RL literature, is that scalar reward feedback on self-generated samples provides more limited supervision than expert demonstrations, potentially allowing lower-capacity updates to suffice [1].

Importantly, flow-based VLAs typically generate actions with a small number of Euler steps on a fixed timestep grid, which keeps action generation consistent at deployment. On-policy RL trains on self-generated rollouts using these same timesteps, so the policy learns under the conditions in which it will be used. In contrast, the training timesteps in BC need not correspond to the denoising timesteps used at inference, because it learns from fixed demonstrations and constructs flow-matching training examples by interpolating between demonstrated actions and sampled noise at any timestep. Furthermore, our controlled comparison supports keeping the training schedule consistent with the rollout schedule: randomizing the training timesteps increases the effective rank while decreasing the success rate (Table R2 above). The low-rank structure can therefore be understood as a property of VLA RL under a practically motivated and empirically supported training setup, rather than an arbitrary scheduling choice. We added these results to Section 4.1. and Appendix C.8

[1] John Schulman et al, LoRA Without Regret 2025

Reviewer_ueZeOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how reinforcement learning (RL) reshapes the parameter space of flow-based Vision-Language-Action (VLA) models. The authors find that RL induces low-rank parameter updates concentrated in the Timestep Modules, which capture a substantial portion of the performance gains. Further analysis reveals that these updates are closely related to discrete denoising timesteps and encode task-relevant information. Based on these findings, the authors demonstrate that representation steering along the learned shift directions can further improve policy performance without additional RL training.

Critical strengths
  1. Novel and Interesting Findings. The paper reveals interesting properties of RL post-training in VLA models, particularly the observation that low-rank updates in Timestep Modules account for a substantial portion of the RL performance gains. These findings provide new insights into how RL improves VLA policies.

  2. Solid Experimental Validation. The paper provides comprehensive evaluations across multiple benchmarks and representative VLA architectures. The extensive analyses, including module replacement, low-rank reconstruction, probing, and representation steering, provide convincing empirical support for the main findings.

Critical weaknesses
  1. Generalization to Recent Flow-Based VLAs. While the experiments cover diverse benchmarks, the evaluation could be further extended to more recent flow-based VLA architectures, such as VLANeXt. This would help verify whether the observed findings generalize across different VLA designs.

  2. Extension to World Action Models. It would be interesting to investigate whether similar findings hold for World Action Models (WAMs), such as Fast-WAM. This could further demonstrate the generality of the findings beyond conventional VLA architectures.

  3. Further Understanding of Timestep Modules. While the paper demonstrates that Timestep Modules capture a substantial portion of the RL performance gains, a more intuitive explanation of this phenomenon would be valuable. One possible hypothesis is that pretrained VLAs already possess substantial task-relevant capabilities, and RL primarily adjusts or activates these existing capabilities rather than learning entirely new ones. Further investigation into this hypothesis could provide deeper insights and make the findings more compelling.

Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Official Rebuttal (1/3)

We thank the reviewer for describing our findings as novel and interesting, and for calling our empirical support convincing. We extend our analysis to additional VLA and WAM models (VLANeXt, MolmoAct2, FastWAM) and further discuss the reviewer’s hypothesis that RL largely sharpens capabilities already present in the base policy, and revised our manuscript accordingly with updates in blue.


W1: Generalization to Recent Flow-Based VLAs. While the experiments cover diverse benchmarks, the evaluation could be further extended to more recent flow-based VLA architectures, such as VLANeXt. This would help verify whether the observed findings generalize across different VLA designs.

Additional VLA

Thank you for the suggestion. We added two more recent flow-based VLAs, VLANeXt [1] and MolmoAct2 [2], using the same pipeline as for π0.5, a short BC phase followed by PPO on LIBERO-Spatial.

VLANeXtMolmoAct2
SFT93.278.8
RL97.693.6
Timestep rank-196.894.2

Table R1. Success rates (%) on LIBERO-Spatial.

ModelComponentRank95 (BC)Rank95 (RL)Relative ΔW, BC (%)Relative ΔW, RL (%)
VLANeXtTime MLP14563.410.87
AdaLN72.04.538.673.94
Attention422.8335.27.590.79
MLP314.2140.66.250.96
MolmoAct2Time MLP733.210.06
AdaLN2.12.032.801.29
Attention455.1254.16.100.17
MLP566.2365.68.240.21

Table R2. Effective rank (Rank95) and relative magnitude of parameter updates under BC and RL.

Both models exhibit a low-rank structure similar to that of π0.5 and GR00T. RL updates to the Timestep Modules are low-rank, with AdaLN receiving the largest relative change, while attention and MLP updates remain high-rank. These updates also encode a disproportionate share of the RL gain, as keeping only the top singular direction of each Timestep Module update recovers most of it, following the same protocol as our GR00T analysis (see our response to Reviewer dF8p, W2). We added these results to the revised manuscript in Section 3 and Appendix C.2, C.5.

[1] Xiao-Ming Wu et al., VlaNeXt: Recipes for building strong vla models, 2026

[2] Haoquan Fang et al., Molmoact2: Action reasoning models for real-world deployment, 2026

AuthorsOpenReview ↗

Official Rebuttal (2/3)


W2: Extension to World Action Models. It would be interesting to investigate whether similar findings hold for World Action Models (WAMs), such as Fast-WAM. This could further demonstrate the generality of the findings beyond conventional VLA architectures.

Additional WAM

Thank you for the suggestion. Due to limited compute, we analyzed publicly released RL checkpoints of two WAMs, Fast-WAM [3] trained with GRPO on LIBERO-10, and Cosmos3-Edge [4] fine-tuned on a real-robot bottle-righting task and then trained with GRPO inside a learned world model. For Fast-WAM, the pre-RL SFT checkpoint is unavailable, so we measure the RL update from the released base checkpoint, which also includes the few-shot SFT preceding RL. As a BC counterpart, we additionally train SFT from the same base checkpoint for longer and compare the two updates. For Cosmos3-Edge, the base, SFT, and GRPO checkpoints are all released, so we compare the BC update (base to SFT) with the RL update (SFT to GRPO).

ModelComponentBCRL
Fast-WAMTime MLP in59.428.5
Time MLP out36.324.9
Time projection60.43.7
Attention71.328.8
MLP76.740.6
Cosmos3-EdgeTime MLP in16.835.2
Time MLP out8.20.3
Attention41.559.3
MLP62.778.9

Table R3. Effective rank of BC and RL updates in WAMs, as a percentage of full rank.

Low-rank structure also emerges in both WAMs. As in VLAs, the RL update is most strongly low-rank in the timestep-conditioning layer (the time projection in Fast-WAM and the Time MLP output in Cosmos3-Edge), where it is far lower-rank than the BC update, while attention and MLP updates remain high-rank. In Fast-WAM, this holds even though the RL update also includes a few-shot SFT, whereas longer SFT alone yields a high-rank update. Together with Appendix C.4, where RL-trained image and video DiTs lack this structure, this suggests that it arises from RL on action generation.

We could not conduct module-replacement evaluations because the pre-RL checkpoint of Fast-WAM and the world and reward models used to train Cosmos3-Edge are not publicly released. We therefore focus on the update structure. For the final version, we will attempt to obtain a pre- and post-RL pair for Fast-WAM by running RL from the released LIBERO checkpoint, and to evaluate module replacement on it. We added these results to the revised manuscript in Section 3.1 and Appendix C.3.

[3] Tianyuan Yuan et al., Fast-WAM: Do World Action Models Need Test-time Future Imagination?, 2026.

[4] NVIDIA, Cosmos 3: Omnimodal World Models for Physical AI, 2026.

AuthorsOpenReview ↗

Official Rebuttal (3/3)

W3: Further Understanding of Timestep Modules. While the paper demonstrates that Timestep Modules capture a substantial portion of the RL performance gains, a more intuitive explanation of this phenomenon would be valuable. One possible hypothesis is that pretrained VLAs already possess substantial task-relevant capabilities, and RL primarily adjusts or activates these existing capabilities rather than learning entirely new ones. Further investigation into this hypothesis could provide deeper insights and make the findings more compelling.

Further Analysis of Timestep Modules

We thank the reviewer for this insightful suggestion and consider it a plausible interpretation : the base policy’s representation space already contains directions that predict task success, although the model does not fully exploit these directions, while RL may learn how to steer the hidden states along these directions to improve performance. We provide three pieces of evidence supporting this hypothesis.

First, RL-induced shift directions predict success and failure even in the base policy (Section 5.1.1, Table 2), suggesting that its representation space already contains directions along which task success can be predicted.

Second, we show that adding only the RL-induced shift difference to the base policy's hidden activations improves success across all five benchmarks (Appendix C.11, Table 18), meaning that the same shift directions used for probing can also causally improve the base policy’s performance when used for steering.

BenchmarkBase (%)PPO (%)Base + shift (%)
LIBERO Spatial70.086.781.3
LIBERO Object92.794.797.3
LIBERO Goal90.790.794.7
ManiSkill34.090.055.0
MetaWorld33.366.752.0

Table R4. Success rates of base policy steering in π0.5.

Third, we compare PPO-induced shifts with BC-induced shifts to examine whether steering effectiveness is specific to RL. Starting from the same base policy, we continued training with BC for an additional 3,000 steps on LIBERO-Spatial and extracted the resulting BC-induced shift directions. We evaluated the probing and steering effectiveness of both BC- and PPO-induced directions on the base policy.

BC-induced directions also strongly predict success and failure, achieving 99.07% F1 and 99.25% ROC-AUC. This confirms that success-predictive information is not exclusive to RL. However, more importantly, BC-induced directions are substantially less effective for steering than PPO-induced directions. We evaluated both across multiple steering strengths, which scale each direction before it is added to the representation.

DirectionBase0.51.01.52.0Full BC/PPO
BC-induced70.073.370.773.370.089.3
PPO-induced70.074.781.380.784.786.7

Table R5. Comparison of BC- and PPO-induced steering: Success rates (%) on evaluation tasks (7:3 task split). Columns 0.5--2.0 indicate steering strengths.

BC-induced directions yield only small and inconsistent changes (+0.0 to +3.3%), whereas PPO-induced directions consistently improve success by up to +14.7%. These findings support our claim that RL learns to steer hidden states along success-relevant directions that more directly contribute to performance improvement, whereas BC-induced directions have much weaker effects on policy behavior.

Together, these results suggest that RL learns to better exploit task-relevant directions already present in the base policy. Since the Timestep Modules provide input-independent modulation that can steer hidden states along these directions, this may explain why updating them alone recovers much of the RL gain. This interpretation is also consistent with prior findings in robotics and LLMs. In robotics, frozen flow policies can be substantially improved without updating their weights, by steering their input noise via RL [5] or flow reversal [6], indicating that successful behaviors often already lie within the base policy's distribution. In LLMs, reinforcement learning with verifiable rewards (RLVR) improves performance by favoring successful reasoning paths already present in the base model [7]. We added this discussion and the additional results to the revised manuscript.

[5] Wagenmaker et al., Steering Your Diffusion Policy with Latent Space Reinforcement Learning, CoRL 2025

[6] Tang et al., Improving Robotic Generalist Policies via Flow Reversal Steering, 2026

[7] Yue et al., Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? NeurIPS 2025.

Thanks for the detailed reply. All of my concerns are solved, I would like to raise my score.

Reviewer_dF8pOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides an interesting perspective on VLA RL post-training through parameter-space analysis. The authors find that RL induces low-rank updates concentrated in Timestep Modules and investigate the role of discrete denoising timesteps in producing this structure. They further analyze how shift updates relate to task success and cross-task transfer, and explore their use for parameter-efficient adaptation and test-time steering. Clarifying these points would help readers better understand the significance and generalizability of the findings.

Critical strengths
  1. The paper provides an interesting perspective on understanding VLA reinforcement learning through parameter-space analysis. The experiments span multiple architectures and benchmarks, and the module-replacement studies provide useful evidence beyond simple parameter statistics.
  2. The findings have potential practical value. The targeted LoRA and test-time steering experiments demonstrate how the observed parameter structure might be exploited for more efficient adaptation.
Critical weaknesses
  1. Limited generalization and unclear RL-specificity. The observed low-rank structure appears closely tied to training with fixed discrete denoising timesteps. Since discrete-timestep BC produces similar updates, it is unclear whether this is a general property of VLA reinforcement learning or mainly a consequence of the timestep schedule. The paper does not examine RL with randomized or adaptive denoising schedules, which limits the generality of its central claim.
  2. The interpretation of the findings is not fully supported. The importance of Timestep Modules varies considerably across architectures. For GR00T N1.5, keeping only these updates reduces success from 90.3% to 61.3%, substantially weakening the claimed dominance of these modules. and shift-direction probes are highly predictive even for base policies, making it difficult to establish whether these directions encode information specifically learned through RL rather than pre-existing task-relevant representations.
Questions for authors
  1. Would the low-rank structure still hold with randomized denoising timesteps or different schedules? How would these settings affect the effective rank and success rate?

  2. Why does retaining only Timestep Module updates lead to such a large performance drop for GR00T N1.5? What explains this difference across architectures?

  3. Given the strong probing results on base policies, how can we tell whether the shift directions capture RL-specific information? I think a comparison with BC-induced directions would help clarification.

5 comments
AuthorsOpenReview ↗

Official Rebuttal (1/4)

We thank the reviewer for finding our parameter-space perspective on VLA RL interesting, and for recognizing the scope of our experiments across multiple architectures and benchmarks as well as the practical value of the targeted LoRA and steering results.

We add RL with randomized denoising schedules, an analysis of why GR00T N1.5 differs from π0.5 together with rank-1 reconstruction analysis of GR00T, and a comparison against BC-induced shift directions, and revised our manuscript accordingly with updates in blue.


W1: Limited generalization and unclear RL-specificity. The observed low-rank structure appears closely tied to training with fixed discrete denoising timesteps. Since discrete-timestep BC produces similar updates, it is unclear whether this is a general property of VLA reinforcement learning or mainly a consequence of the timestep schedule. The paper does not examine RL with randomized or adaptive denoising schedules, which limits the generality of its central claim.

Q1: Would the low-rank structure still hold with randomized denoising timesteps or different schedules? How would these settings affect the effective rank and success rate?

Randomized-schedule RL.

Following the reviewer's suggestion, we train π0.5 with PPO on LIBERO-Spatial under a randomized denoising schedule, keeping all other settings identical to our fixed-schedule run. At every PPO step, we keep K=5 denoising steps but shift each of the four interior timesteps uniformly within ±0.1 of its default value, with the initial timestep fixed at 0. For example, instead of the fixed timesteps {0, 0.2, 0.4, 0.6, 0.8}, one PPO step can use {0, 0.12, 0.32, 0.62, 0.71}. All rollouts in that step are generated on this randomized grid, and the same grid is used for the corresponding PPO update, so rollout generation and training remain consistent. Because the default timesteps are spaced 0.2 apart, these ±0.1 windows are adjacent without overlap, so training covers a continuous range of timesteps across PPO steps, similar to BC. As in BC, evaluation uses the default K=5 schedule.

BaseFixed RLRandomized RL
Accuracy0.8500.9660.926

Table R1. Accuracy on LIBERO-Spatial.

ComponentBCFixed RLRandomized RL
Time MLP6296.0139
AdaRMS65630.1295.2
Attention1,9071,8101,834.2
MLP2,5392,5102,528.6

Table R2. Effective rank of parameter updates.

Randomizing the denoising schedule raises the effective rank of the Timestep Module updates, consistent with our discrete-timestep BC result (Section 4.1), suggesting that discrete-timestep training contributes to the low-rank structure. Nevertheless, the rank remains well below that of BC, suggesting that discretization is not the only factor and that the on-policy nature of RL may also contribute. One possible explanation, also discussed in the LLM RL literature, is that scalar reward feedback on self-generated samples provides more limited supervision than expert demonstrations, potentially allowing lower-capacity updates to suffice [1].

Importantly, flow-based VLAs typically generate actions with a small number of Euler steps on a fixed timestep grid, which keeps action generation consistent at deployment. On-policy RL trains on self-generated rollouts using these same timesteps, so the policy learns under the conditions in which it will be used. In contrast, the training timesteps in BC need not correspond to the denoising timesteps used at inference, because it learns from fixed demonstrations and constructs flow-matching training examples by interpolating between demonstrated actions and sampled noise at any timestep. Furthermore, our controlled comparison supports training on the same schedule used at deployment, as randomizing the schedule during RL increases the effective rank while decreasing the success rate under the default evaluation schedule (Table R1). The low-rank structure can therefore be understood as a property of VLA RL under a practically motivated and empirically supported training setup, rather than an arbitrary scheduling choice. We added these results to Section 4.1 and Appendix C.8.

[1] John Schulman et al, LoRA Without Regret 2025

*Edited Nov 14: corrected the rollout-generation description (see our reply to Reviewer VYAg).

AuthorsOpenReview ↗

Official Rebuttal (2/4)

W2: The importance of Timestep Modules varies considerably across architectures. For GR00T N1.5, keeping only these updates reduces success from 90.3% to 61.3%, substantially weakening the claimed dominance of these modules.

Q2: Why does retaining only Timestep Module updates lead to such a large performance drop for GR00T N1.5? What explains this difference across architectures?

Generalization to GR00T.

As the reviewer points out, transferring only the Timestep Module updates to the base policy is less effective in GR00T than in π0.5. To first explain why this happens in GR00T, this difference may primarily reflect how the two architectures process timestep information, rather than indicate that low-rank Timestep Module updates are unimportant in GR00T. In π0.5, timestep information enters the action expert only through the Timestep Modules. In GR00T, timestep embeddings are also concatenated with action token embeddings, allowing subsequent Attention and MLP layers to process timestep information (Appendix A.2). The Timestep Module updates may therefore need to work together with RL updates to these layers to produce the full gain. We therefore qualify our original claim: while Timestep Modules are not necessarily the dominant contributors to RL gains in GR00T, a small number of low-rank components in their updates still make a critical contribution when combined with RL updates to other modules.

To support this more specific claim, we conducted an additional experiment that tests whether low-rank Timestep Module updates remain important for GR00T’s RL gains when RL updates to the other components are preserved. On GR00T N1.5 in LIBERO-Spatial, we started from an in-house PPO policy (see Implementation notes below). We either retained only the leading rank-1 component of each Timestep Module weight update or removed that component while retaining the remaining update. We also evaluated removing all Timestep Module updates. All other parameters remained at their RL-trained values.

ConditionSuccess (%)
Base37.0
Full PPO69.8
PPO: Time top-1 retained77.8
PPO: Time top-1 removed33.8
PPO: all Time updates removed (Attn+FFN only)33.8

Table R3. Effect of low-rank Timestep Module updates on GR00T N1.5.

With all other RL updates preserved, retaining only these rank-1 components achieves 77.8% success, compared with 69.8% for full PPO. Removing these rank-1 components reduces success to 33.8%, the same success rate as removing all Timestep Module updates. Both conditions perform worse than the base policy (37.0%). This supports our finding that low-rank Timestep Module updates contribute substantially to RL improvement beyond π0.5, while their full effect may depend on updates to other components that process timestep information.

We thank the reviewer for pointing this out and agree that our wording about which components account for most of the RL gain should be more carefully qualified. We have incorporated this architectural interpretation into the revised manuscript (Section 3.2, Appendix A.2), and revised our wording to avoid implying that Timestep Modules are the dominant contributors across all architectures (Section 1, Section 3.2).

Implementation notes: We trained GR00T PPO in-house to limit numerical rounding effects in the update-rank analysis. The π0.5 checkpoints we analyzed maintain Timestep Module weights in FP32, whereas the public RLinf GR00T PPO configuration [2] we inspected updates them directly in BF16. Entrywise rounding can disrupt the linear dependencies within an update matrix and inflate its measured rank, making it difficult to distinguish learned structure from numerical artifacts. We therefore maintained and saved the action-head weights in FP32, with FP32 optimizer states, without imposing any low-rank constraint.

[2]https://github.com/RLinf/RLinf/blob/df0303eed32f1ea456df5e792a9d5309dd62fb93/examples/embodiment/config/libero_spatial_ppo_gr00t.yaml

AuthorsOpenReview ↗

Official Rebuttal (3/4)

W2: The interpretation of the findings is not fully supported. shift-direction probes are highly predictive even for base policies, making it difficult to establish whether these directions encode information specifically learned through RL rather than pre-existing task-relevant representations.

Q3: Given the strong probing results on base policies, how can we tell whether the shift directions capture RL-specific information? I think a comparison with BC-induced directions would help clarification.

Interpretation of the shift direction.

Thank you for pointing this out. Our interpretation is that the base policy’s representation space already contains directions that predict task success, although the model does not fully exploit these directions, while RL may learn how to steer the hidden states along these directions to improve performance. We provide two pieces of evidence for this interpretation.

First, RL-induced shift directions predict success and failure even in the base policy (Section 5.1.1, Table 2), suggesting that its representation space already contains directions along which task success can be predicted.

Second, we show that adding only the RL-induced shift difference to the base policy improves success, meaning that the same shift directions used for probing also have a causal effect on the base policy’s performance (Appendix C.11, Table 18). We added only the RL-induced shift difference to the π0.5 base policy at all sublayer–timestep pairs, without replacing the Timestep Modules.

BenchmarkBase (%)PPO (%)Base + shift (%)
LIBERO Spatial70.086.781.3
LIBERO Object92.794.797.3
LIBERO Goal90.790.794.7
ManiSkill34.090.055.0
MetaWorld33.366.752.0

Table R4. Success rates of base policy steering in π0.5.

Adding only the shift difference improves success across all five benchmarks, recovering a substantial portion of the PPO improvement and demonstrating a causal effect on the base policy's behavior. Together with the probing results, this supports our interpretation that RL improves the use of pre-existing task-relevant representations. This interpretation is also consistent with findings in LLMs that reinforcement learning with verifiable rewards (RLVR) improves performance by favoring successful reasoning paths already present in the base model [3].

[3] Yue et al., Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? NeurIPS 2025.


Comparison to BC-induced shift directions

Following the reviewer’s suggestion, we further compared PPO-induced shift directions with BC-induced directions. We first trained the same base policy with BC for 3,000 steps on LIBERO-Spatial and extracted shift directions from the base-to-BC updates. We then evaluated their probing and steering effectiveness on the base policy.

BC-induced directions also strongly predict success and failure, achieving 99.07% F1 and 99.25% ROC-AUC. This confirms that success-predictive information is not exclusive to RL. However, BC-induced directions are substantially less effective for steering than PPO-induced directions. We evaluated both across multiple steering strengths, which scale each direction before it is added to the representation.

DirectionBase0.51.01.52.0Full BC/PPO
BC-induced70.073.370.773.370.089.3
PPO-induced70.074.781.380.784.786.7

Table R5. Comparison of BC- and PPO-induced steering. Success rates (%) on evaluation tasks (7:3 task split). Columns 0.5--2.0 indicate steering strengths.

BC-induced directions yield only small and inconsistent changes (+0.0 to +3.3%), whereas PPO-induced directions consistently improve success by up to +14.7%. These findings support our claim that RL learns to steer hidden states along success-relevant directions that more directly contribute to performance improvement, whereas BC-induced directions have much weaker effects on policy behavior.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

Dear AC and Reviewers,

We thank all reviewers again for the constructive feedback. Following the rebuttal, Reviewers dF8p and ueZe raised their scores, and we have responded to Reviewer VYAg's follow-up questions. Below, we summarize the main concerns and our responses, with all new results update in the manuscript (changes in blue).

1. Discrete timesteps' contribution to low-rank updates (dF8p W1, VYAg W2).

  • Concerns: Reviewers asked for a direct RL intervention, beyond our discrete-timestep BC result, to clarify the relationship between the low-rank structure in VLA RL and the timestep schedule.
  • Response:
    • We trained π0.5 on LIBERO-Spatial under a randomized timestep schedule, applied to both rollouts and updates. This raises the effective rank of Timestep Module updates by an order of magnitude (AdaRMS 30.1 → 295.2), confirming that discrete-timestep training contributes to the low-rank structure. The rank still stays below BC (656), suggesting that the on-policy, scalar-reward nature of RL also contributes.
    • Discrete-timestep training is also the natural setting for VLA RL, as on-policy RL trains on the same fixed timestep grid the policy uses at deployment. Randomizing the schedule indeed lowers success (96.6 → 92.6). The low-rank structure is thus a property of VLA RL in practice (Sec. 4.1, App. C.8).

2. Interpretation of the GR00T results (dF8p W2, VYAg W1).

  • Concerns: Reviewers questioned the claimed dominance of Timestep Modules across architectures, as performance dropped when retaining only their updates in GR00T N1.5.
  • Response: An additional experiment shows that low-rank Timestep Module updates also remain critical in GR00T N1.5: with all other RL updates preserved, retaining only their leading rank-1 components yields 77.8% success (full RL: 69.8%), while removing them drops success to 33.8% (base: 37.0%). We also qualified the claim that Timestep Module updates alone recover most RL gains, while supporting their critical contribution beyond π0.5 (Sec. 1, 3.2, App. C.5).

3. Generalization to additional VLAs and WAMs (ueZe W1, W2).

  • Concerns: Additional validation on other VLAs and world action models (WAMs) was requested.
  • Response:
    • Our findings generalize to additional VLAs (VLANeXt and MolmoAct2), where (1) RL Time MLP updates need only 6 and 3 directions (BC: 145 and 7), respectively, and (2) a single direction per Timestep Module matrix preserves RL performance. -The low-rank structure also appears in WAMs such as Fast-WAM and Cosmos3-Edge, whose timestep-conditioning layers use only 3.7% and 0.3% of full rank under RL (BC: 60.4% and 8.2%) (Sec. 3, App. C.2, C.3, C.5).

4. Interpretation of shift directions (dF8p Q3, ueZe W3).

  • Concerns: Reviewers asked whether shift directions capture RL-specific information or pre-existing success-relevant representations.
  • Response:
    • RL-induced shift directions predict success even in the base policy, and steering along them improves base performance on all five benchmarks (e.g., ManiSkill: 34.0% → 55.0%).
    • Following the reviewer's suggestion, we compared BC- and RL-induced directions. Both are similarly predictive, but BC-induced directions are less effective for steering (+0.0 to +3.3% vs. up to +14.7% for RL).
    • This suggests that RL learns to exploit pre-existing success-relevant directions by steering hidden states along them through Timestep Modules, explaining their substantial contribution to RL gains (Sec. 5.1.1, 7, App. C.11).

5. Clarifications on experiment protocol and reproducibility (VYAg Q3, Q4).

  • Concerns: The reviewer asked about trainable parameters, train–test separation in steering, and reproducibility of the reported gains.
  • Response:
    • We clarified that the VLM is frozen and only the action expert is trained in RL (Sec. 2.2).
    • We confirmed strict task-level train–test separation (7:3 split): steering directions and targets come only from training-task rollouts and are fixed during evaluation (Sec. 5.2).
    • We showed reproducible steering gains across five evaluation seeds, with strictly positive paired 95% CIs on all three LIBERO suites (App. C.12).

Strengths noted by reviewers

  • Novel perspective: "an interesting perspective on understanding VLA reinforcement learning through parameter-space analysis" (dF8p); "new insights into how RL improves VLA policies" (ueZe); "a concrete parameter-level characterization of VLA post-training" (VYAg).
  • Extensive validation: "the module-replacement studies provide useful evidence beyond simple parameter statistics" (dF8p); the analyses "provide convincing empirical support for the main findings" (ueZe).
  • Practical value: "suggests opportunities for more parameter-efficient adaptation" (VYAg); targeted LoRA and steering show "how the observed parameter structure might be exploited for more efficient adaptation" (dF8p).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.