- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies two key challenges in reinforcement learning for sparse MoE-based diffusion VLA models: unstable expert routing and exploding likelihood ratios during denoising. The authors first analyze how advantage signals and diffusion noise affect routing behavior, then propose Advantage- and Noise-Aware Load Balancing (ANA-LB) to encourage more stable expert selection. They also introduce Geometric Mean Ratio (GMR) to reduce the variance of trajectory-level importance ratios. Experiments on simulated benchmarks and real-world robotic tasks show consistent improvements in training stability and task performance. Overall, the paper provides useful insights into MoE-RL dynamics and offers two lightweight, practical solutions.
Critical strengths
-
The paper points out two important problems in MoE-based diffusion RL, and the analysis of how advantage affects expert routing is quite interesting.
-
The proposed methods are simple and lightweight, and the experiments on both simulation and real robots show solid improvements.
Critical weaknesses
-
GMR changes the original importance ratio and introduces bias. The reduced clipping may simply come from a looser trust region rather than genuinely more stable policy updates.
-
The novelty of GMR is somewhat limited, as it is essentially a form of ratio tempering. It is unclear whether the proposed method offers meaningful advantages over simpler alternatives.
-
The connection between expert switching and ratio instability is not fully convincing, since similar issues can also occur in dense diffusion policies.
-
The theoretical analysis of ANA-LB relies on soft routing, while the actual model uses hard Top-1 routing. Also, the advantage-aware gating shows only a small improvement in the ablation study.
-
Some figures have very small fonts, especially in the histograms, while others appear blurry or distorted, making them harder to read.
Questions for authors
-
GMR introduces bias by pushing the importance ratio toward 1. Could the authors clarify whether the improved stability comes at the cost of larger policy shifts?
-
Given that GMR is essentially a form of ratio tempering, could the authors compare with simpler methods like (r^\alpha), or different clipping ranges?
-
Could the authors clarify how much of the observed instability is actually caused by MoE expert switching?
-
Given the gap between the soft-routing analysis and the actual hard Top-1 implementation, how well does the theory explain the routing behavior in practice? Would the conclusions still hold under Top-k or soft routing?
Thanks for your response. The content is sufficient and already addresses the five problems I listed. It's a good work, and I'll maintain my score.