- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RLRD, a method that routes token-level expert feedback during multimodal reinforcement learning. Image-perturbation sensitivity selects perception-related tokens, while predictive entropy selects reasoning-related tokens. Two frozen experts score the corresponding positions, and their likelihood gaps modify GRPO advantage magnitudes through the sign-preserving reweighting rule. Experiments with Qwen2.5-VL-3B and Qwen2.5-VL-7B across six benchmarks show improvements over RLVR and distillation baselines.
Critical strengths
-
The paper is generally clear and easy to follow.
-
The main results show consistent improvements across both student scales on six benchmarks.
-
Ablations and analysis are provided to validate individual design choices of the routing branches and expert assignments.
Critical weaknesses
-
The technical contribution over prior work is insufficiently established. RLRD combines visual-sensitivity and entropy signals with the reweighting rule adopted from RLSD [1]. Given the use of visual degradation in prior multimodal post-training methods [2–4], the paper should better justify the value of this combination and clarify how its use of image perturbations for feedback routing differs from these approaches.
[1] Yang, Chenxu, et al. "Self-distilled rlvr." arXiv preprint arXiv:2604.03128 (2026).
[2] Liu, Chenxi, et al. "Modality-balancing preference optimization of large multimodal models by adversarial negative mining." arXiv preprint arXiv:2506.08022 (2025).
[3] Wang, Tianfu, et al. "All-Weather VLM: Enhancing Vision Language Models' Robustness Under Adverse Imaging Conditions." Third Conference on Language Modeling.
[4] Liu, Xiangyan, et al. "Noisyrollout: Reinforcing visual reasoning with data augmentation." Advances in Neural Information Processing Systems, 2025.
-
Computational comparability remains unclear. RLRD introduces a perturbed-image pass and two full-response teacher-scoring streams. Tables 8–9 report hardware allocation rather than measured GPU-hours or throughput. It would be more convincing to report actual training costs and comparisons under matched total compute budgets.
-
The mechanism analysis is restricted to successful trajectories. Both role-localization and expert-likelihood analyses do not establish whether routing remains reliable or the assigned experts provide useful corrective feedback on failed trajectories.
-
Crucial implementation details for the perception expert are missing from Appendix A.2, including its training objective and hyperparameters.
Questions for authors
Could the authors provide qualitative comparisons between RLRD and GRPO on perception- and reasoning-oriented questions?
3 comments
Thanks the authors for their detailed respone. I have increased my score to 3.
Thank you for your positive feedback and for raising your score. We sincerely appreciate your time and constructive comments.
Dear Reviewer nuCb,
With the discussion period nearing its end, we would like to follow up on our rebuttal and see whether it has adequately addressed your concerns. We hope our responses and additional analyses have provided useful clarification. If you have any remaining questions or suggestions, we would be happy to address them before the discussion closes.
Thank you again for your valuable feedback, time, and consideration.
Best regards,
The Authors