acceptodds
Under review as a conference paper at ICLR 2027

Multimodal RLVR With Routed Distillation

20 views

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves multimodal reasoning, yet a shared outcome-derived advantage cannot distinguish errors in image perception from errors in reasoning over the image. These functions often recur within the same response, making both the location and the source of expert feedback consequential. We introduce RLRD, short for RLVR with Routed Distillation, which allocates token-level feedback from frozen perception and reasoning experts with different post-training backgrounds. Image-perturbation sensitivity and predictive entropy serve as routing proxies, selecting positions along each on-policy response for the corresponding expert. The routed teacher-student log-probability gaps are incorporated into an existing sign-preserving advantage reweighting rule, adjusting local update magnitudes while retaining the update direction set by the verifier reward. Across six vision-language benchmarks and Qwen2.5-VL-3B/7B students, RLRD achieves the highest perception and reasoning averages among the evaluated methods, improving over the strongest non-RLRD baseline in each group by 1.39-2.02 percentage points. Controlled ablations show gains over random routing at the same teacher-token budget and over swapped or averaged expert assignments. Together, these results support selecting both feedback positions and expert sources to improve learning from interleaved multimodal responses.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
reinforcement learning with verifiable rewards, multimodal reasoning, routed distillation, vision-language models, token-level feedback

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nuCbOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RLRD, a method that routes token-level expert feedback during multimodal reinforcement learning. Image-perturbation sensitivity selects perception-related tokens, while predictive entropy selects reasoning-related tokens. Two frozen experts score the corresponding positions, and their likelihood gaps modify GRPO advantage magnitudes through the sign-preserving reweighting rule. Experiments with Qwen2.5-VL-3B and Qwen2.5-VL-7B across six benchmarks show improvements over RLVR and distillation baselines.

Critical strengths
  • The paper is generally clear and easy to follow.

  • The main results show consistent improvements across both student scales on six benchmarks.

  • Ablations and analysis are provided to validate individual design choices of the routing branches and expert assignments.

Critical weaknesses
  • The technical contribution over prior work is insufficiently established. RLRD combines visual-sensitivity and entropy signals with the reweighting rule adopted from RLSD [1]. Given the use of visual degradation in prior multimodal post-training methods [2–4], the paper should better justify the value of this combination and clarify how its use of image perturbations for feedback routing differs from these approaches.

    [1] Yang, Chenxu, et al. "Self-distilled rlvr." arXiv preprint arXiv:2604.03128 (2026).

    [2] Liu, Chenxi, et al. "Modality-balancing preference optimization of large multimodal models by adversarial negative mining." arXiv preprint arXiv:2506.08022 (2025).

    [3] Wang, Tianfu, et al. "All-Weather VLM: Enhancing Vision Language Models' Robustness Under Adverse Imaging Conditions." Third Conference on Language Modeling.

    [4] Liu, Xiangyan, et al. "Noisyrollout: Reinforcing visual reasoning with data augmentation." Advances in Neural Information Processing Systems, 2025.

  • Computational comparability remains unclear. RLRD introduces a perturbed-image pass and two full-response teacher-scoring streams. Tables 8–9 report hardware allocation rather than measured GPU-hours or throughput. It would be more convincing to report actual training costs and comparisons under matched total compute budgets.

  • The mechanism analysis is restricted to successful trajectories. Both role-localization and expert-likelihood analyses do not establish whether routing remains reliable or the assigned experts provide useful corrective feedback on failed trajectories.

  • Crucial implementation details for the perception expert are missing from Appendix A.2, including its training objective and hyperparameters.

Questions for authors

Could the authors provide qualitative comparisons between RLRD and GRPO on perception- and reasoning-oriented questions?

3 comments

Dear Reviewer nuCb,

With the discussion period nearing its end, we would like to follow up on our rebuttal and see whether it has adequately addressed your concerns. We hope our responses and additional analyses have provided useful clarification. If you have any remaining questions or suggestions, we would be happy to address them before the discussion closes.

Thank you again for your valuable feedback, time, and consideration.

Best regards,

The Authors

Thanks the authors for their detailed respone. I have increased my score to 3.

Thank you for your positive feedback and for raising your score. We sincerely appreciate your time and constructive comments.

Reviewer_faeMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper propose a new RL algorithm for training multimodal reasoning models. The idea is to assign tokens to two different "experts", one perception, one reasoning; the way we route the tokens based on image perturbation and entropy; the experts will provide a weighting based on a delta similar to OPD/OPSD KL divergence; then the weighting will be finally applied to the GRPO style loss.

Critical strengths
  • The key idea is to explicitly set two experts to guide the post-training; which is interesting and aligns with recent work on perception-aware multimodal post-training and credit-assignment related task such as the 80/20 rule
  • I can see this paper mixed a diverse range of recent RL/OPD related methods and tricks; showing that the insights can be combined to get a better performance; the motivation of using each method is clear and reasonable.
Critical weaknesses
  • The method is quite complex, containing various design choices, this makes the question "is this necessary?" hard to justify.

    • it is hard to do comprehensive ablation for all combinations: for example, what if we directly use the expert's delta to update the model using a OPD/OPSD style loss and removes the GRPO part
  • Is the small performance gain worth the complicatedness? how about computational overhead which the main text did not seem to discuss in detail?

  • The assumption to get pretrained perception-specific and reasoning-specific experts is also a strong assumption; and potentially a not fair comparison with main baslines such as GRPO

Questions for authors

N/A

1 comment

Thank you for highlighting the need for explicit computational measurements. We report the total additional training cost starting from publicly released checkpoints, including teacher preparation where applicable and the complete training run.

Configuration. We use Qwen2.5-VL-3B-Instruct, ViRL39K, one epoch (284 rollout iterations), a batch of 128 prompts, and 8 responses per prompt, on 8 NVIDIA A800 GPUs. Maximum prompt/response lengths are 1,024/2,048 tokens. OPD uses a validation-selected single 7B expert; RLRD uses complementary 7B perception and reasoning experts.

MethodReasoning Avg. ↑ (%)Perception Avg. ↑ (%)Total A800 GPU-hours ↓
GRPO46.35 ± 0.4959.45 ± 0.4357.86
OPD (teacher + student)47.22 ± 0.4060.23 ± 0.3681.44
RLRD (two teachers + student)51.13 ± 0.3062.90 ± 0.29127.78

Accuracy values are the Avg@8 results from Table 1, reported as mean ± standard deviation across three training seeds. Costs cover one complete training run, including all allocated GPUs for teacher scoring and student optimization, plus applicable teacher preparation counted once without amortization. Upstream training costs of publicly released checkpoints are excluded.

Compared with OPD, RLRD requires 56.9% more total compute, while improving reasoning and perception accuracy by 3.91 and 2.67 percentage points, respectively. The corresponding improvements over GRPO are 4.78 and 3.45 points, at 2.21× the total cost. These measurements make the computational trade-off explicit and complement our ablations demonstrating the benefits of the proposed capability integration.

We hope this quantitative comparison, together with our earlier clarifications, addresses your concerns and supports reconsideration of your assessment.

Reviewer_QyhROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RLRD, a multimodal RLVR method that routes token-level feedback from frozen perception and reasoning experts. The method uses changes in predictions after image masking and the model's uncertainty to decide which tokens each teacher should guide. Teacher feedback then adjusts the strength of each token's update, while keeping the direction. Experiments show improvements across several benchmarks.

Critical strengths

-The paper addresses an intersting problem, that task-level rewards cannot distinguish perception errors from reasoning errors.

-The method is straightforward and introduces no additional trainable routing modules.

-The main results are consistent across two models.

Critical weaknesses

-The novelty of this paper is incremental. The advantage reweighting is similar to RLSD, while VPPO already uses image masking and token-level distribution changes to select visually dependent positions. The main contribution is the combination with role-specific external experts, rather than a new optimization mechanism.

-Using different teachers for different tokens is not new, H-OPD already adjusts each teacher's influence throughout a response. RLRD uses signals from the student model to choose which tokens receive feedback and from which teacher. However, it does not directly compare this strategy with confidence-based approach of H-OPD.

-The paper only compares role-based teacher assignment with averaging the teacher logits. This averaging may weaken a useful prediction when one teacher supports it but the other does not. Comparing with probability averaging or confidence-based teacher weighting is essential.

Questions for authors

See the weaknesses.

4 comments

W1. Novelty beyond RLSD and VPPO

We thank the reviewer for this comment. We acknowledge that RLRD adopts the sign-preserving reweighting rule from RLSD and builds upon visual sensitivity signals explored in VPPO. Our key contribution is not these individual components, but the role-aware allocation of external expert feedback within multimodal RLVR: determining both where expert guidance is needed and which specialized expert should provide it.

Importantly, our ablations validate this design. Under matched teacher-token budgets, RLRD outperforms random routing by 3.11/2.82 points (P-Avg/R-Avg). Swapping the expert assignments leads to 3.56/4.01-point drops, while matched assignment also outperforms routed logit averaging (Table 2). These results support the value of student-guided, role-specific feedback allocation beyond simply introducing expert supervision.

We will clarify this distinction from prior work in the revision.

Dear Reviewer QyhR,

With the discussion period nearing its end, we would like to follow up on our rebuttal and see whether it has adequately addressed your concerns. We hope our responses and additional analyses have provided useful clarification. If you have any remaining questions or suggestions, we would be happy to address them before the discussion closes.

Thank you again for your valuable feedback, time, and consideration.

Best regards,

The Authors

Dear Reviewer,

We hope this message finds you well. As the discussion period draws to a close, with about six hours remaining for author replies, we wanted to follow up on our rebuttal and the additional experiments we shared. We hope these materials address the concerns raised in your review. If there are any points you would like us to clarify or any further feedback you would like us to consider, please let us know. We would be happy to respond before the window closes.

Thank you for your time and effort in reviewing our paper.

Dear Reviewer QyhR,

Thank you for suggesting the comparisons with probability averaging and confidence-weighted fusion. As the discussion draws to a close, we would like to highlight two results relevant to your review.

  1. Contribution beyond RLSD and VPPO (W1). RLRD builds on RLSD’s advantage reweighting and the visual-sensitivity signal explored in VPPO. Our contribution is to use student-side signals to decide which tokens receive feedback and which expert provides it. At the same teacher-token budget, RLRD improved perception and reasoning averages by 3.11 and 2.82 percentage points over random routing. Swapping the two experts reduced those averages by 3.56 and 4.01 points, respectively.

  2. Comparison with alternative teacher assignments (W2, W3). Following your suggestion, we held the experts, selected tokens, training budget, and RL objective fixed and varied only how teacher feedback was combined or assigned. On the 3B student, role-specific assignment reached 62.90% average perception accuracy, compared with 61.14% for probability averaging and 60.33% for H-OPD-style confidence-weighted fusion. Reasoning accuracy was comparable across the three methods.

We hope these comparisons clarify the contribution of role-specific routing. If they address your concerns, we would be grateful if you would reconsider your overall assessment in light of these results.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.