acceptodds
Under review as a conference paper at ICLR 2027

Stabilizing RL Fine-Tuning of Diffusion VLAs with Sparse MoE Action Heads: Advantage- and Noise- Aware Load Balancing and Geometric-Mean Ratio

Abstract

Sparse mixture-of-experts (MoE) is an effective architecture for scaling vision- language-action (VLA) policies, but imitation learning alone is limited by demon- stration coverage and cannot fully exploit MoE capacity under distribution shift. Reinforcement learning (RL) fine-tuning can leverage task feedback, yet directly applying RL to diffusion MoE-VLAs is unstable because continuous denoising dynamics interact with discrete expert routing. We diagnose two failure modes: (1) advantage-signed routing updates sharpen experts for positive-advantage sam- ples and flatten them for negative-advantage samples, while our measurements show larger routing changes at late low-noise steps; and (2) expert switching along the denoising chain causes step-wise likelihood-ratio deviations to accumulate, making the resulting trajectory ratios heavy-tailed and frequently clipped. To ad- dress these issues, we propose two minimal interventions. First, as a loss term, Advantage- and Noise-Aware Load Balancing (ANA-LB) applies noise-scaled load balancing only to trajectories with negative group-relative advantage, sta- bilizing relatively worse samples while preserving positive-advantage expert spe- cialization. Second, for ratio computation, Geometric-Mean Ratio (GMR) uses a geometric-mean surrogate for the accumulated step-wise log-ratios, trading a small amount of exact importance-weighting fidelity for reduced ratio variance. Experiments on LIBERO, CALVIN, RoboTwin 2.0, and real-robot tasks show competitive overall performance, with the clearest gains in several long-horizon settings, while reducing routing collapse, excessive ratio clipping, and training oscillation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Embodied Intelligence, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oX2AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies two key challenges in reinforcement learning for sparse MoE-based diffusion VLA models: unstable expert routing and exploding likelihood ratios during denoising. The authors first analyze how advantage signals and diffusion noise affect routing behavior, then propose Advantage- and Noise-Aware Load Balancing (ANA-LB) to encourage more stable expert selection. They also introduce Geometric Mean Ratio (GMR) to reduce the variance of trajectory-level importance ratios. Experiments on simulated benchmarks and real-world robotic tasks show consistent improvements in training stability and task performance. Overall, the paper provides useful insights into MoE-RL dynamics and offers two lightweight, practical solutions.

Critical strengths
  1. The paper points out two important problems in MoE-based diffusion RL, and the analysis of how advantage affects expert routing is quite interesting.

  2. The proposed methods are simple and lightweight, and the experiments on both simulation and real robots show solid improvements.

Critical weaknesses
  1. GMR changes the original importance ratio and introduces bias. The reduced clipping may simply come from a looser trust region rather than genuinely more stable policy updates.

  2. The novelty of GMR is somewhat limited, as it is essentially a form of ratio tempering. It is unclear whether the proposed method offers meaningful advantages over simpler alternatives.

  3. The connection between expert switching and ratio instability is not fully convincing, since similar issues can also occur in dense diffusion policies.

  4. The theoretical analysis of ANA-LB relies on soft routing, while the actual model uses hard Top-1 routing. Also, the advantage-aware gating shows only a small improvement in the ablation study.

  5. Some figures have very small fonts, especially in the histograms, while others appear blurry or distorted, making them harder to read.

Questions for authors
  1. GMR introduces bias by pushing the importance ratio toward 1. Could the authors clarify whether the improved stability comes at the cost of larger policy shifts?

  2. Given that GMR is essentially a form of ratio tempering, could the authors compare with simpler methods like (r^\alpha), or different clipping ranges?

  3. Could the authors clarify how much of the observed instability is actually caused by MoE expert switching?

  4. Given the gap between the soft-routing analysis and the actual hard Top-1 implementation, how well does the theory explain the routing behavior in practice? Would the conclusions still hold under Top-k or soft routing?

Reviewer_4YYJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reinforcement learning fine-tuning of diffusion-based VLAs with sparse MoE action heads, focusing on the interaction between iterative denoising and discrete expert routing. This paper identifies two sources of instability: advantage-dependent changes in router behavior across denoising steps and the accumulation of step-wise likelihood-ratio deviations caused by expert switching. To address these issues, they propose Advantage- and Noise-Aware Load Balancing (ANA-LB), which selectively regularizes negative-advantage trajectories with noise-dependent weighting, and Geometric-Mean Ratio (GMR), which replaces multiplicative trajectory-ratio aggregation with geometric averaging to reduce ratio variance and clipping. The validation is conducted on LIBERO, CALVIN, RoboTwin 2.0, and real-world manipulation tasks.

Critical strengths

S1. The connection from the diagnosed failure modes to the proposed methods is intuitive and well-targeted.

The paper identifies two concrete issues—advantage-dependent routing behavior and likelihood-ratio instability caused by expert switching—and introduces ANA-LB and GMR to address these two issues separately. In particular, the design choices of applying load balancing only to negative-advantage samples and using geometric averaging for accumulated ratios follow naturally from the corresponding observations.

Critical weaknesses

W1. The theoretical routing analysis does not account for the multi-layer structure of the implemented MoE.

Theorems 3.1 and 3.2, including their Appendix derivations, analyze a single MoE routing site, while the actual DiT replaces FFNs across multiple layers with Shared-MoE modules. The analysis therefore does not model how an expert choice in an earlier layer changes the hidden representation and consequently the routing decisions of downstream MoE layers. It remains unclear whether the derived local router dynamics extend to the coupled routing behavior of the full multi-layer MoE policy.

W2. The paper does not empirically verify the claimed task/skill specialization of the sparse MoE.

A key motivation for introducing sparse MoE is that heterogeneous manipulation behaviors can be routed to specialized experts, thereby reducing interference across skills, objects, and task contexts. However, the paper does not analyze whether the learned router actually selects different experts for different tasks, objects, or behavioral primitives; the reported routing entropy only measures concentration, not semantic specialization. A task- or skill-conditioned analysis of expert utilization would substantially strengthen the architectural motivation.

Questions for authors

Q1. Multi-layer routing analysis

The theoretical analysis in Theorems 3.1 and 3.2 is formulated for a single MoE routing site, whereas the implemented DiT contains Shared-MoE modules across multiple layers. How do the derived local routing dynamics extend to the multi-layer setting, where an upstream expert choice can change the hidden representation and thus affect downstream routing decisions?

Q2. Isolating the benefit of sparse MoE

Could the authors provide an analysis of whether different manipulation tasks, objects, or behavioral primitives are actually routed to different experts? Since task/skill specialization is a central motivation for using sparse MoE, it would be helpful to show task-conditioned expert usage or routing distributions to verify that the learned experts specialize according to the heterogeneous embodied behaviors claimed in the motivation.

Reviewer_WFVWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies unstable RL fine-tuning of flow-based VLA policies with sparse MoE action heads. It proposes ANA-LB, which applies noise-weighted load balancing to negative-advantage trajectories, and GMR, which geometrically averages likelihood ratios across denoising steps and action chunks. Evaluation covers LIBERO, CALVIN, RoboTwin 2.0, and 24 real-robot tasks. The strongest evidence comes from the matched MoE ablations and real-robot comparisons.

Critical strengths
  1. The method improves on a matched MoE RL baseline. In Table 3, CALVIN average chain length increases from 4.42 to 4.78, and success rates rises from 77.6% to 89.5%. The component ablations make a useful case that both routing regularization and ratio aggregation contribute.
  2. The results provide evidence beyond simulation. Under the matched MoE setup in Table 4, average success improves from 71.8% to 78.3% over vanilla RL. The larger gains on precision and long-horizon tasks make the practical contribution credible.
Critical weaknesses
  1. The added value of the particular ANA-LB design is not fully isolated. Table 3 includes standard load balancing and GMR separately, but omits standard load balancing combined with GMR. ANA-LB alone improves average chain length from 4.63 to 4.70 over standard balancing. Appendix I.2 reports three seeds, but no variability is given. The missing combined control and uncertainty estimates leave open whether advantage gating and noise weighting provide a reliable benefit beyond the simpler combination.
  2. The clipping-probability bounds in Appendix H.4 need correction. Equations (102) and (104) apply centered tail bounds without requiring the mean log-ratio to lie inside the log-clipping interval. If the mean lies outside it, concentration can increase clipping probability toward one. The rates in Eqs. (103) and (105) also require a margin from the interval boundaries. This does not invalidate the centered-variance result or the empirical gains, but the stated clipping bounds do not follow under the current assumptions.
Questions for authors
  1. How does standard load balancing combined with GMR compare with the full method under the same CALVIN setup? Please report the individual seed results or their dispersion for these comparisons.
  2. Can you correct the mean and margin conditions in Appendix H.4 and check whether they hold for the measured log-ratios? This would clarify which clipping claims are supported by the theory and which rest on empirical observations.
  3. Figure 3(c) compares timesteps within a fixed ten-step sampler, whereas Section 4.4 discusses increasing the denoising horizon. Could you vary the total number of denoising steps, for example T = 5, 10, and 20, on CALVIN and report chain length and clipping frequency for the baseline and full method under the same interaction budget?
  4. With ANA-LB held fixed, could you compare GMR against a product-ratio baseline whose learning rate is adjusted to achieve a similar old-to-new policy KL per update? This would help distinguish the benefit of ratio aggregation from changes in the effective update size. Learning curves and final task performance would be useful here.
  5. Could you repeat the component ablation on a representative subset of RoboTwin 2.0 Hard, keeping the MoE backbone and training budget fixed? Comparing vanilla GRPO, ANA-LB alone, GMR alone, and the full method would test whether the complementary gains observed on CALVIN also hold for bimanual manipulation.

Comments on the paper

1 comment

Thanks for your response. The content is sufficient and already addresses the five problems I listed. It's a good work, and I'll maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.