- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies dynamic coordination between visual information acquisition and textual reasoning in multimodal reinforcement learning. It proposes DyCo-RL, a token-level advantage-reweighting method for RLVR. The method uses Fisher–Rao geodesic distance between consecutive within-modality attention distributions to assign visual, textual, or neutral roles, and adjusts token advantages according to the alignment between the assigned role and actual attention allocation. Extensive experiments using 11K examples for RL training demonstrate that the algorithm-agnostic DyCo-RL can improve the average performance of all four evaluated RLVR algorithms across seven benchmarks.
Critical strengths
- The empirical improvements are consistent across all configurations. Table 1 reports average improvements across all 12 combinations of three backbone configurations and four RL algorithms. The three-seed GRPO experiments in Table 18 improve from 47.4 to 49.2 on 3B and from 55.8 to 58.1on 7B. Table 20 also reports gains across six additional benchmarks.
- The ablations provide meaningful evidence for role-conditioned reweighting. Random or reversed role assignments and fixed visual-only or text-only scoring underperform the full method as shown in table3 and 4.
Critical weaknesses
- The gap remains between the automatic role estimator and the proposed causal interpretation. Table 11 offers useful annotation-based validation, but its 74.0% accuracy applies only to the 26.3% of annotated tokens receiving non-neutral predictions, using erroneous trajectories from the 3B model. Moreover, Fisher–Rao’s balanced accuracy of 72.8% does not exceed KL’s 73.5%. These limitations reduce confidence in the semantic interpretation of the assignments and the specific advantage attributed to Fisher–Rao.
- The incremental value over closely related methods and under matched computational budgets remains insufficiently established. The experiments establish benefits at equal training-data exposure, but do not determine whether those benefits remain competitive when the baseline receives the same computational budget.
Questions for authors
The authors manually annotated 200 incorrect responses to evaluate the accuracy of automatic token-role assignment. Was the annotated diagnostic set used to select the role-assignment metric or threshold? Could you report coverage and role accuracy separately for correct and erroneous spans, and clarify whether role reliability has been evaluated on fully correct trajectories or larger backbones?
4 comments
Response to uWCS (Part 2/3): Model Scale and Automatic-Role Interventions
Point 1: W1 & Q1 (continued)
(4) Larger backbones and trajectory correctness (Q1). While fully correct trajectories were not annotated as a separate set, our 3B diagnostic set includes locally correct reasoning spans. Within this group, Fisher–Rao achieves 75.3% accuracy on non-neutral predictions (Table U1). To test larger backbones, we newly annotated 100 rollouts (4,370 annotated tokens) from Qwen2.5-VL-7B following Appendix F:
Table U4. Fisher–Rao () role agreement across model scales.
| Backbone | Coverage (%) | Accuracy (%) | Balanced Accuracy (%) |
|---|---|---|---|
| Qwen2.5-VL-3B (Table 11) | 26.3 | 74.0 | 72.8 |
| Qwen2.5-VL-7B | 34.7 | 70.9 | 66.2 |
On 7B, Fisher–Rao achieves 70.9% accuracy (again the highest among all five evaluated measurements at , ahead of KL's 68.1%, Attention's 62.4%, Cosine's 59.2%, and Entropy's 59.1%), supporting transferability to larger models.
(5) Bridging the automatic estimator and causal interpretation (W1). To directly test whether automatic predictions carry the causal utility of human role labels, we replace human roles with step-wise Fisher–Rao predictions () at the exact same annotated error spans, keeping intervention strengths , replay budgets, and decoding settings fixed (Appendix H). Neutral steps skip enhancement, and all 200 responses remain in the denominator.
Table U5. Answer recovery at visually-oriented erroneous spans (%).
| Human roles | Our predicted roles | No-label | Random location | Wrong-Role | |
|---|---|---|---|---|---|
| 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| 0.1 | 7.0 | 4.5 | 4.0 | 5.5 | 4.0 |
| 0.2 | 15.5 | 14.0 | 16.0 | 11.5 | 7.5 |
| 0.3 | 13.0 | 15.5 | 14.5 | 10.5 | 9.5 |
| 0.4 | 21.5 | 18.5 | 13.0 | 15.0 | 8.0 |
| 0.5 | 32.5 | 25.5 | 11.5 | 20.0 | 6.5 |
| 0.6 | 23.5 | 27.0 | 7.0 | 14.5 | 6.5 |
| 0.7 | 28.0 | 21.5 | 10.5 | 14.5 | 0.5 |
| 0.8 | 11.0 | 10.5 | 2.0 | 8.5 | 0.5 |
| Avg. () | 19.0 | 17.1 | 9.8 | 12.5 | 5.4 |
Table U6. Answer recovery at text-oriented erroneous spans (%).
| Human roles | Our predicted roles | No-label | Random location | Wrong-Role | |
|---|---|---|---|---|---|
| 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| 0.1 | 14.5 | 13.0 | 11.0 | 9.0 | 7.5 |
| 0.2 | 19.0 | 21.5 | 15.0 | 12.5 | 4.0 |
| 0.3 | 26.0 | 22.0 | 21.5 | 19.5 | 8.5 |
| 0.4 | 23.0 | 22.5 | 13.5 | 18.0 | 9.0 |
| 0.5 | 21.5 | 19.5 | 17.0 | 19.0 | 7.5 |
| 0.6 | 26.5 | 21.5 | 14.0 | 18.5 | 4.0 |
| 0.7 | 24.5 | 20.0 | 7.5 | 17.0 | 2.5 |
| 0.8 | 16.0 | 12.5 | 1.5 | 14.0 | 1.0 |
| Avg. () | 21.4 | 19.1 | 12.6 | 15.9 | 5.5 |
(Bold under "Our predicted roles" marks strengths outperforming all three non-oracle controls.)
Automatic-role interventions beat all three controls at 6 of 8 active strengths for visual errors and 7 of 8 for textual errors, retaining ~90% of the average recovery rate of human annotations (within 1.9 and 2.3 points) and providing complementary functional evidence for the automatic role assignments.
Response to uWCS (Part 3/3): Training Cost and Matched Budget
Point 2: W2 — Incremental Value Over Related Methods Under Matched Compute Budgets
(1) Comparable compute cost to VPPO with nearly double the gain over GRPO. Measuring training cost on the same 8×A100 server (11,031 samples) shows that VPPO incurs nearly the same rollout/processing cost as DyCo-RL:
Table U7. Measured 1-epoch training cost on Qwen2.5-VL-3B (8×A100 GPUs).
| Method | Throughput (samples/s) | Wall-clock time (hours) | GPU-hours | Avg. Score |
|---|---|---|---|---|
| GRPO | 0.447 | 6.85 | 54.8 | 47.4 |
| VPPO | 0.337 | 9.09 | 72.7 | 48.4 |
| GRPO + DyCo-RL | 0.327 | 9.37 | 75.0 | 49.3 |
DyCo-RL requires 9.37 hours versus 9.09 hours for VPPO—only a 3.1% increase (+0.28 h, +2.24 GPU-hours), with zero extra parameters or runtime cost at inference. Over the shared GRPO baseline (47.4), where PAPO-G gains +0.5 (47.9) and VPPO gains +1.0 (48.4), DyCo-RL gains +1.9 points (49.3, outperforming PAPO-G on all 7 benchmarks and VPPO on 6 of 7) at virtually the same training budget.
(2) Incremental gains persist when baselines receive the full compute budget. We independently retrain GRPO and VPPO from the same initial checkpoint for the full 9.37 hours (75.0 GPU-hours) under identical hardware and evaluation settings:
Table U8. Performance at equal data exposure (1 epoch) vs. matched compute budget (9.37 hours).
| Method | Time (h) | WeMath | MathVision | MathVerse | MMBench | LogicVista | HallusionBench | MME | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| GRPO (1 epoch) | 6.85 | 23.0 | 21.4 | 34.4 | 72.8 | 39.8 | 60.8 | 79.9 | 47.4 |
| VPPO (1 epoch) | 9.09 | 26.9 | 22.0 | 35.8 | 75.1 | 39.6 | 61.2 | 78.5 | 48.4 |
| GRPO (Matched Budget) | 9.37 | 25.2 | 23.4 | 35.1 | 69.3 | 38.7 | 61.8 | 79.4 | 47.6 |
| VPPO (Matched Budget) | 9.37 | 27.3 | 22.0 | 36.4 | 72.4 | 37.4 | 61.0 | 79.1 | 47.9 |
| GRPO + DyCo-RL (1 epoch) | 9.37 | 25.9 | 22.4 | 36.0 | 76.2 | 41.2 | 62.4 | 80.7 | 49.3 |
At matched training compute (9.37 h), DyCo-RL remains ahead of GRPO and VPPO by +1.7 and +1.4 average points, respectively (and +0.9 points above VPPO's 1-epoch score of 48.4), supporting the value of dynamic cross-modal coordination under the matched compute budget.
Dear Reviewer uWCS,
We hope this message finds you well.
We have carefully addressed your questions and concerns, including conducting additional experiments on role reliability across different settings, matched-coverage evaluations, automatic-role interventions, and matched-compute comparisons, and have provided detailed responses in our rebuttal.
As the discussion deadline is approaching, we would sincerely appreciate it if you could share your updated thoughts based on our rebuttal. If you have any additional questions or concerns, please do not hesitate to let us know, and we will respond promptly.
Thank you again for your thoughtful review and valuable feedback, which have helped us improve the quality of our work.
Kind regards,
Authors of Submission 18709
Response to uWCS (Part 1/3): Role Diagnostics and Matched Coverage
Due to the comment length limit, our response is provided in three parts.
We sincerely thank the reviewer for the careful and constructive assessment. We appreciate your recognition of the consistent improvements across configurations, the three-seed results and out-of-domain evaluations, and the evidence from role-conditioned ablations. Below, we clarify the diagnostic protocol and provide additional role validation and matched-budget comparisons to address your concerns.
Point 1: W1 & Q1 — Role Reliability Across Span Correctness, Matched Coverage, Scale, and Causal Interventions
Clarification on metric and threshold selection (Q1). No—the 200-sample human-annotated set was strictly used for post-hoc diagnostic evaluation and was never used to choose the distance metric or threshold τ. Fisher–Rao was motivated a priori as a symmetric, bounded ([0,π]) geodesic metric on the probability simplex, with τ=0.05 set as a small stability margin to filter near-tie transitions. Crucially, this single configuration was held strictly fixed without retuning across all 12 backbone–optimizer combinations (Table 1) and 6 held-out out-of-domain benchmarks (Table 20), with post-hoc ablations (Tables 3 and 13) confirming robustness across every tested τ∈{0,0.05,0.1}.
(1) Separate evaluation on locally correct vs. erroneous spans (Q1 & W1). Splitting the 6,875 annotated tokens in Table 7 into locally correct spans (Pr∪Rr, 5,157 tokens) and locally erroneous spans (Pe∪Re, 1,718 tokens) at τ=0.05 reveals why Fisher–Rao outperforms KL:
Table U1. Role agreement (%) on locally correct vs. erroneous spans (3B, τ=0.05).
Fisher–Rao achieves the highest accuracy in both groups (75.3% on correct spans; 68.6% on erroneous spans). Notably, while KL is comparable on correct spans, its reliability drops sharply on erroneous spans, where Fisher–Rao leads KL in both accuracy (+4.1 points) and balanced accuracy (+3.1 points: 69.5% vs. 66.4%). Furthermore, on tokens misclassified by the Attention-magnitude baseline, Fisher–Rao (τ=0.05) covers 57.9% of tokens with 65.8% accuracy, directly correcting 38.1% of Attention's role errors.
(2) Superior accuracy and balanced accuracy under matched coverage (W1). In Table 11, a fixed τ=0.05 yielded slightly different coverages across differently scaled metrics (e.g., 28.2% for KL vs. 26.3% for Fisher–Rao). Matching non-neutral coverage at 25%, 50%, and 75% via per-response top-k% ranking removes scale differences:
Table U2. Role Accuracy / Balanced Accuracy (%) at matched non-neutral coverage (3B).
Under matched selectivity, Fisher–Rao achieves both the highest accuracy and the highest balanced accuracy at every coverage level.
(3) Reliability beyond the 26.3% subset (W1). Varying τ across {0,0.025,0.05,0.1} confirms that agreement with human roles is not limited to a narrow subset. At τ=0, the observed non-neutral coverage is 100.0% on this diagnostic set:
Table U3. Fisher–Rao role agreement across thresholds τ (3B).
At 52.9% coverage (τ=0.025, over 2× the default coverage), Fisher–Rao retains 70.2% accuracy.