acceptodds
Under review as a conference paper at ICLR 2027

Alliance of Stochastic Routing and Reinforcement Fusion for Multi-Model Reasoning

Abstract

This paper studies dynamic model routing under uncertainty in multimodal settings, where a system must select and potentially compose multiple models to answer a query. Standard reinforcement learning approaches for routing often suffer from premature convergence to suboptimal models due to noisy rewards and insufficient exploration. We propose a unified vision-language Stochastic routing framework (VL-Scout) for efficient multi-Modal and multi-step reasoning with turn-based reinforcement fusion with three unique features. First, VL-Scout combines Bayesian priors, policy gradient optimization, and sequential decision-making to enable structured exploration and adaptive inference. A context-dependent Dirichlet belief model allows for epistemic uncertainty over model quality and is updated online based on observed rewards and context-specific evidence, enabling Thompson-style exploration while serving as a prior for regularization. Second, a routing policy is trained via a KL-regularized REINFORCE objective by maximizing a reward-weighted evidence lower bound, where the policy acts as an amortized posterior, and the Dirichlet belief serves as a prior. Third, we introduce a fusion policy trained with Group Relative Policy Optimization (GRPO) or Reinforcement Learning with Verifiable Rewards (RLVR), which operates over intermediate model rationales and selects between two actions: CONTINUE, invoking another model via the router, or ANSWER, terminating with a generated output. This enables adaptive computation depth and selective model composition conditioned on reasoning quality. We evaluate VL-Scout on four representative multi-modal benchmarks (MMMU, MMMU-Pro, OKVQA, and RealWorldQA), compared against 11 representative routing and policy optimization baselines. Our method provides consistent performance improvements, while reducing premature policy collapse. These results highlight the importance of integrating uncertainty-aware priors with both amortized policy learning and sequential fusion for robust and efficient model selection in discrete action spaces. Code is available at anonymous.4open.science/r/vl-scout-8EB2/

Primary area:
reinforcement learning
Keywords:
Routing, Reinforcement Learning, VLMs, Multi-armed bandits

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_D4XHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces a unified vision-language Stochastic routing framework (VL-Scout) for efficient multi-Modal and multi-step reasoning with turn-based reinforcement fusion with three unique features. After GRPO, the proposed model enables adaptive computation depth and selective model composition conditioned on reasoning quality. Evaluated on four representative multi-modal benchmarks (MMMU, MMMU-Pro, OKVQA, and RealWorldQA), and compared against 11 representative routing and policy optimization baselines demonstrate the superiority of VL-Scout.

Critical strengths
  1. The approach is novel, the paper is motivated to minimize the loss by finding the parameters θand ϕthat parametrize the routing and fusion function.
  2. After GRPO, the proposed model enables adaptive computation depth and selective model composition conditioned on reasoning quality is very interesting.
  3. Evaluated on four representative multi-modal benchmarks (MMMU, MMMU-Pro, OKVQA, and RealWorldQA), and compared against 11 representative routing and policy optimization baselines demonstrate the superiority of VL-Scout.
Critical weaknesses
  1. The writing is very unclear and hard to understand the paper.
  2. The evaluated benchmarks are limited, only 4 benchmarks are demonstrated.
  3. The performance gap to oracle is still huge.
  4. The tested models are very old, does this method generalization to new and bigger models?
Questions for authors

see above.

2 comments

Thank you for the review. We answer each point with new experiments.

First, a clarification about the "superiority" you noted. Table 1 reports GRPO as pass@4, so a query counts as solved if any of several sampled rollouts is correct. We re-ran all experiments scoring one greedy answer given before the label is seen. Under that scoring, VL-Scout is 0.3–0.8 points above the best single model on MMMU, MMMU-Pro and RealWorldQA, and 1.2 below it on A-OKVQA, using fewer calls than voting (details in our reply to qAyi). We revise the claim accordingly.

W4: newer and larger models. We added Qwen3-VL-32B and InternVL3.5-38B (4–5× larger than the original models) and re-ran routing on the resulting 9-model pool (online, top-1, 10 seeds):

MMMUMMMU-ProA-OKVQARWQA
Best original model57.7651.8291.9774.90
Qwen3-VL-32B68.5766.2789.4376.99
TS-MA65.13 ± 1.5662.76 ± 1.0388.95 ± 0.8672.52 ± 1.16
REINFORCE64.39 ± 5.1361.66 ± 4.5487.35 ± 2.1571.37 ± 5.36
VL-Scout67.25 ± 1.0963.96 ± 0.5588.59 ± 0.3773.79 ± 1.14

The router exploits the new models (+11 points on MMMU over the old pool). It beats TS-MA on three of four benchmarks and has far lower variance than REINFORCE. It needs an up-to-date operator prior, though. A stale prior that ignores the new models anchors exploration (MMMU 57.13), and we now state this limitation.

Fusion also benefits. We use the frozen fusion policy, with the three candidates chosen on the cross-domain source benchmark, so no target labels are used:

FusionBest single (hindsight)Calls
MMMU72.0569.942.11
A-OKVQA92.6691.971.75
RWQA78.3076.991.94

Pooled over 2,715 queries, fusion is +1.3 points above the best single model chosen in hindsight (McNemar p = 0.036).

W2: more benchmarks. We added HallusionBench (951 yes/no questions on charts, tables, OCR, maps and visual illusions), using a 6-model open-weight pool:

HallusionBenchAcc.Calls
Best single model (Qwen3-VL-32B)76.341
Vote over [Qwen3-VL-8B, InternVL3-8B, LLaVA-13B]68.773
VL-Scout fusion, same 3 models75.811.56
VL-Scout fusion, best 3 models77.181.50

Fusion beats voting by +7.0 points with half the calls (McNemar p < 0.0001), and with good candidates it matches the best single model (+0.84, not significant).

W3: gap to the oracle. The oracle is a theoretical upper bound. It queries all 7 models on every input and then picks a correct answer whenever one exists. The gap is large because many queries are solved by only one or two models (% of queries by number of correct models):

Correct models (of 7)01234567
MMMU11.413.415.318.014.511.88.76.8
MMMU-Pro14.114.618.817.213.38.96.96.2
A-OKVQA2.22.42.72.95.88.213.462.4
RWQA4.67.613.112.313.316.517.115.6

On MMMU and MMMU-Pro, 29% and 33% of queries are solved by at most two models. On A-OKVQA only 5% are, and 62% are solved by all seven, which is why its gap is small. A system can recover such answers only if it happens to query the one or two models that hold them. That is possible only by calling every model, which defeats the purpose of routing. VL-Scout instead samples at most kk (lets say k=3k=3) models per query. When a single model solves a query, a random set of 3 contains it only 3/7 = 43% of the time. The bound that matches our budget is therefore any of the 3 consulted models (3 seeds):

Upper boundMMMUMMMU-ProA-OKVQARWQA
All 7 models queried (7 calls)88.5785.8797.8295.42
3 random models74.4969.4995.0886.58
3 models routed by VL-Scout77.0674.3794.9186.19
Best single model57.7651.8291.9774.90

Limiting calls to 3 lowers the bound by 9–12 points on MMMU, MMMU-Pro and RWQA. The router's selections contain a correct model more often than random ones on MMMU and MMMU-Pro (+2.6 / +4.9). We keep the any-of-7 oracle as the theoretical bound and add the budget-matched bound next to it. Within the budget, the remaining headroom is ordering the candidates: with the fusion policy fixed, the oracle order is 5–15 points above random order.

W1: writing. We restructure Sections 3–4 around an algorithm box for the full online loop, a protocol box (predict → score → update), a notation table and an early figure. The figure separates routing (which models, in what order) from fusion (when to stop, what to answer).

Thank the authors for their feedback. Adding one more HallusionBench evaluation is good, but I don't think one more enough is for this general multimodal reasoning approach as multimodal reasoning benchmarks are very abundant. Therefore I maintain the score.

Reviewer_7hFWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VL-Scout unifies model routing and output fusion for vision-language reasoning under one reinforcement-learning framework. Routing is a contextual bandit: a category-conditioned Dirichlet belief over model quality acts as a Thompson-sampled prior, and a neural policy is trained with a KL-regularized REINFORCE objective that maximizes a reward-weighted ELBO (with a Gibbs-posterior interpretation in Proposition 1). Fusion is a second policy, GRPO/RLVR-trained on Qwen3-VL-4B with LoRA, that reads the routed models' rationales and emits CONTINUE (query the next model) or ANSWER (stop), giving adaptive computation depth. On MMMU, MMMU-Pro, A-OKVQA, and RealWorldQA, VL-Scout (GRPO) reaches 60.37/52.01/90.13/76.47 online, beating all 11 pooled VLMs and 5 baselines, at ~8× lower inference cost than DART. Appendix B builds the router up step by step (TS -> LinUCB -> TS+LinUCB -> REINFORCE -> REINFORCE+Prior).

Critical strengths
  1. Routing and fusion are genuinely optimized together. Nearly all prior work does one or the other; closing this gap is the paper's real contribution and the main reason I'm positive on the idea.
  2. The Bayesian prior is principled, not a bolt-on. The Dirichlet belief with Thompson-style exploration regularizing KL-REINFORCE, plus the clean Gibbs-posterior result in Proposition 1, give the "avoid premature convergence" claim actual substance.
  3. The bottom-up ablation (TS -> LinUCB -> TS+LinUCB -> REINFORCE -> +Prior) and the adaptive CONTINUE/ANSWER depth are honest and effective: they explain why each component is needed and deliver real efficiency (4.09s vs DART's 32.75s per query).
Critical weaknesses
  1. No statistical significance anywhere. Every result is a single accuracy number, no multiple seeds, no standard deviation, no confidence intervals. For online RL with noisy rewards and a stochastic policy, run-to-run variance can exceed the headline gaps, so I can't tell whether the reported wins are real or noise. This is the main thing that keeps me from accepting.
  2. The DART comparison looks off. "DART (Best Model)" scores 65.3 on A-OKVQA, far below individual pretrained models (llava-v1.6-7b 82.62, InternVL3-8B 88.47), while VL-Scout gets 90.13. A 25-point gap between two multi-model systems is implausible and makes me suspect you compared against DART's single best model rather than its full multi-agent framework.
  3. "Up to 5.36%" doesn't match the paper's own tables. From the numbers I can compute, the improvement over the best base model is 5.10 (MMMU) or 6.37 (RealWorldQA), no comparison yields 5.36. A headline claim that can't be reproduced from the paper's own results is a credibility problem.
  4. The "8× lower cost" ignores training cost. VL-Scout requires online RL training of the router plus LoRA fine-tuning of the fusion model (Qwen3-VL-4B), whereas DART is pure inference over pretrained agents. Once training compute and latency are counted, the 8× advantage shrinks substantially, and the "online without training data" framing is misleading, the fusion model is explicitly trained.
  5. The "epistemic uncertainty" framing is loose. Eq. 4's Dirichlet update is a pseudo-count preference whose concentrations can go negative and are reset to αmin\alpha_{min}, not a true posterior (the paper concedes this), yet the abstract's Bayesian-uncertainty language overstates it.
Questions for authors
  • Is "DART (Best Model)" in Table 2 the full multi-agent DART framework or only its single best model? Its 65.3 on A-OKVQA is below individual pretrained models (82–88); please clarify the protocol.
  • Can you report multiple seeds or standard deviations for the headline VL-Scout (GRPO) results? Online RL training is stochastic, and the single-number results don't let me judge whether the gains are significant.
1 comment

Thank you. Your W1 and W3 led us to re-run every experiment with seeds under an explicit predict-then-update protocol. The results follow.

W1 / Q2: seeds and significance. Every result now reports mean ± std (95% CIs in the appendix), with 10 seeds for routing and 3 for fusion. Seeds fix stream order, initialisation and sampling noise, and are shared across methods so that tests are paired. Each query is answered greedily before its label is used for any update.

RLVR fusion (3 seeds; revised fusion setup, see the revised appendix):

MMMUMMMU-ProA-OKVQARWQA
Online58.01 ± 0.2551.61 ± 0.2487.57 ± 0.1872.07 ± 0.40
Cross-domain58.51 ± 0.2552.07 ± 0.2390.74 ± 0.5275.38 ± 0.72
Best single model57.7651.8291.9774.90

For the headline GRPO results, note that Table 1 reports GRPO as pass@4: a query counts as solved if any sampled rollout is correct. We now report the greedy answer, with pass@1–4 for reference (10 rollouts, cross-domain, 1 seed):

GRPOGreedypass@1pass@2pass@3pass@4Calls
MMMU58.1457.9359.3059.9660.432.42
MMMU-Pro52.4552.3754.0354.8755.442.47
A-OKVQA92.7592.5992.7292.7792.791.00
RWQA75.8275.7876.5777.0077.321.63

What the seeded results support:

  • Fusion. Cross-domain RLVR is 0.3–0.8 points above the best single model on MMMU, MMMU-Pro and RealWorldQA, with 1.2–2.5 calls instead of 3. Greedy GRPO is 0.2–2.2 points below RLVR.
  • Routing. The prior stabilises routing. On MMMU (top-1, 10 seeds), REINFORCE gets 51.25 ± 6.65, REINFORCE + Dirichlet prior 56.02 ± 0.98 (Δ = +4.77, Wilcoxon p = 0.0098) and TS-MA 54.00 ± 1.09 (prior vs TS-MA: p = 0.002).

W2 / Q1: DART. The "DART (Best Model)" row was taken from the DART paper, which uses its own agent pool and protocol.

W3: "up to 5.36%". The figure was 76.47 − 71.11: VL-Scout (GRPO) against Qwen3-VL-8B on RealWorldQA. 76.47 is the pass@4 figure discussed above. Table 1 also used strict string matching for base models, under which Kimi-K2.5 scored 64.97 on RealWorldQA instead of 74.90. We replace the claim with the margins above (+0.3 to +0.8 over the best single model). With two larger pool models (Qwen3-VL-32B, InternVL3.5-38B), routing + fusion is +1.3 points above the best single model chosen in hindsight, pooled over 2,715 queries (McNemar p = 0.036).

W4: training cost. We now report it separately. One online RLVR pass costs 0.24 / 0.45 / 0.17 / 0.17 GPU-hours (MMMU / MMMU-Pro / A-OKVQA / RealWorldQA, one H100/H200), and cross-domain warm-up adds one pass over the source benchmark. GRPO with 10 rollouts costs about 1.7 GPU-hours on A-OKVQA or RealWorldQA and 2.4 on MMMU or MMMU-Pro, including warm-up. The router costs milliseconds per query on a CPU. We agree that "online without training data" was misleading and rename the setting. Fusion inference adds 0.4–0.8 s per query on one H100/H200, so total latency is calls × base-VLM latency + fusion overhead. On average, we reach 4.02s per query in latency, compared to DART (32.75s) it 8x times much faster.

W5: "epistemic uncertainty". Agreed. We now call Eq. 4 a pseudo-count preference belief and remove "posterior" and "epistemic" from the abstract and introduction. Proposition 1 is unaffected, because it characterises the maximiser of the KL-regularised objective given the sampled belief (a Gibbs form). It does not claim that the belief is a Bayesian posterior.

Reviewer_qAyiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VL-Scout, a framework for adaptive multi-model visual reasoning. VL-Scout first uses a context-aware router, regularized by category-conditioned Dirichlet beliefs, to select models from a heterogeneous VLM pool. A separate fusion model then sequentially decides whether to query another routed model or generate the final answer, and is trained with RLVR or GRPO under a model-call budget. Experiments on MMMU, MMMU-Pro, A-OKVQA, and RealWorldQA show that the GRPO-based variant outperforms individual VLMs, routing-only baselines, and several voting or multi-agent aggregation methods.

Critical strengths
  1. VL-Scout provides a practical way to combine adaptive model selection with sequential response fusion. Rather than committing to one model or querying a fixed number of models for every example, the fusion model evaluates the responses collected so far and decides whether further evidence is needed. This design offers a flexible approach to trading off answer quality against the cost of consulting multiple heterogeneous VLMs.
  2. The experiments cover several benchmarks and consistently show that the complete routing-and-fusion pipeline outperforms individual VLMs, routing-only variants, and conventional voting methods. The comparison between RLVR and GRPO, along with the analyses of routing behavior and prior initialization, also supports the benefit of learned sequential fusion over fixed aggregation strategies.
Critical weaknesses
  1. The online training and evaluation protocol is insufficiently specified. VL-Scout uses correctness feedback to update the router, category-conditioned belief, and potentially the fusion policy. However, it is unclear whether each example is evaluated before its label is revealed, where the GRPO/RLVR fusion model is trained in the “online without training data” setting, and whether model states are reset between benchmarks. If evaluation labels are used for online adaptation, the reported accuracy is not directly comparable to that of frozen VLMs unless a clear predict-before-update protocol is followed. This ambiguity affects the interpretation of the main results in Table 1.
  2. The contribution of stochastic routing is not sufficiently isolated. The routing-only variant does not outperform the strongest individual VLM on any of the four benchmarks, while the main improvements appear after adding GRPO fusion and allowing multiple model calls. The current results therefore demonstrate the effectiveness of multi-model fusion, but do not establish whether the proposed Dirichlet-regularized router provides an independent benefit. Comparisons with random routing, fixed Top-K routing, routing without the prior, and oracle routing under the same fusion model and call budget would clarify this contribution.
Questions for authors
  1. Could the authors clarify the complete training and evaluation procedure for both settings in Table 1? In particular, where is the GRPO/RLVR fusion model trained in the “online without training data” setting, is every example evaluated before its label is used for updating, are the router and fusion model reset between benchmarks, and how sensitive are the results to sample order and random seeds?
  2. How much of the final improvement comes specifically from the proposed stochastic router? It would be helpful to compare random Top-K, fixed best Top-K, REINFORCE without the Dirichlet prior, and VL-Scout routing while keeping the fusion model, training procedure, and model-call budget identical.
  3. Could the authors report the average number of VLM calls and clarify whether the inference times in Table 2 include all base-model inference and fusion costs? This would make the accuracy–efficiency trade-off easier to interpret.
1 comment

Thank you for the precise questions. We re-ran every experiment under an explicit predict-then-update protocol and answer each question below.

Q1 / W1: protocol.

  • For each query the system sees only (image, question) and records one greedy answer before the label is revealed. Only then are the router, the Dirichlet belief and the fusion LoRA updated.
  • "Online without training data" means label feedback only, with no offline data. The fusion LoRA is trained on the same stream (prequential), and we rename the setting accordingly.
  • Cross-domain: one online pass over the paired source benchmark (MMMU↔MMMU-Pro, A-OKVQA↔RealWorldQA), then the target.
  • All state is reset between benchmarks.
  • A seed fixes stream order, initialisation and sampling noise. Seeds are shared across methods (paired tests): 10 for routing, 3 for fusion.

Results under this protocol (RLVR fusion, 3 seeds; revised fusion setup, see the revised appendix). We also extract the option letter for every model ("A." → A). This raises some base models, e.g. Kimi-K2.5 on A-OKVQA: 91.97, against 80.96 in Table 1.

MMMUMMMU-ProA-OKVQARWQA
RLVR, online58.01 ± 0.2551.61 ± 0.2487.57 ± 0.1872.07 ± 0.40
RLVR, cross-domain58.51 ± 0.2552.07 ± 0.2390.74 ± 0.5275.38 ± 0.72
Calls (online / cross-domain)2.53 / 2.442.58 / 2.521.23 / 1.211.56 / 1.28
Best single model57.7651.8291.9774.90

Cross-domain RLVR exceeds the best single model chosen in hindsight on MMMU, MMMU-Pro and RWQA, by 0.3–0.8 points, and it uses 1.2–2.5 calls instead of 3. We revise the headline claims to match these numbers.

Table 1 reports GRPO as pass@4: a query counts as solved if any sampled rollout is correct. A deployed system returns one answer, so we now report the greedy answer, with pass@1–4 for reference (estimated from 10 rollouts; cross-domain; 1 seed):

GRPOGreedypass@1pass@2pass@3pass@4Calls
MMMU58.1457.9359.3059.9660.432.42
MMMU-Pro52.4552.3754.0354.8755.442.47
A-OKVQA92.7592.5992.7292.7792.791.00
RWQA75.8275.7876.5777.0077.321.63

Greedy GRPO is 0.2–2.2 points below RLVR on all four benchmarks, so we no longer claim GRPO > RLVR. On seed sensitivity, the fusion std is 0.2–0.7. Plain REINFORCE routing varies by 3.4–6.7 points across seeds, against 0.6–1.0 for our prior-regularised router.

Q2 / W2: isolating the router. The fusion policy is frozen and shared and the budget is ≤3 calls; only the candidate order changes (3 seeds). The difference from random order is in parentheses:

OrderMMMUMMMU-ProA-OKVQARWQA
Random50.48 ± 1.0044.35 ± 0.8689.14 ± 0.2069.85 ± 0.92
REINFORCE, no prior51.64 ± 1.77 (+1.2)44.14 ± 2.57 (−0.2)89.37 ± 1.57 (+0.2)67.89 ± 2.80 (−2.0)
VL-Scout router53.42 ± 0.97 (+2.9)46.08 ± 1.86 (+1.7)90.74 ± 0.57 (+2.60)75.38 ± 2.43 (+5.53)
Oracle order62.94 (+12.5)56.68 (+12.3)94.00 (+4.9)84.53 (+14.7)
  • MMMU / MMMU-Pro. Our router gives the best learned order (+2.9 / +1.7 over random). Without the prior, REINFORCE stays at the random level with higher variance. In routing alone on MMMU, REINFORCE gets 51.25 ± 6.65 and our router 56.02 ± 0.98 (Wilcoxon p ≈ 0.01, 10 seeds).
  • A-OKVQA / RWQA. No learned order beats random beyond seed noise. Our operator prior points away from Kimi-K2.5, which is the strongest model there.
  • Headroom. The oracle order is 5–15 points above random. Router-in-the-loop training is our next step.

Q3: calls and time. Calls per query are in the first table: 1.2–2.6, against 3 for voting. Fusion inference adds 0.4–0.8 s per query on one H100/H200, so total latency is calls × base-VLM latency + fusion overhead. On average, we reach 4.02s per query in latency, compared to DART (32.75s) it 8x times much faster.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.