acceptodds
Desk rejected at ICLR 2027

RouteLedger: Routing-Aware Answer Selection with Task-Gated Visual Verification

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Mixture-of-experts vision-language models expose input-dependent routing sig- nals, but directly intervening on expert routes is unreliable and can replace correct answers with plausible errors. We propose RouteLedger, a certificate-based resid- ual selection framework that uses routing evidence conservatively. RouteLedger records expert assignments under text-only, visual, and full multimodal views, and converts their differences into route-aware candidate proposals. A beam-first selec- tor accepts a route residual only when it receives sufficient independent support and passes explicit fidelity constraints. We further introduce SafeSwitch-Margin, a task-family-gated CLIP-L/336 verifier that evaluates a fixed non-route candi- date pool and preserves the Beam-based answer when no residual is certified. On 45,857 examples from 12 VQA-style datasets, RouteLedger obtains 23,419 cor- rect answers, improving Beam by 277 predictions and significantly outperforming the verifier-free selector. A matched Beam+Anchor system with the same veri- fier reaches 23,397 correct answers, leaving a small and non-significant residual for routing; visual verification therefore explains most of the gain. These results support MoE routing as supplementary evidence within a conservative candidate recipe.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Mixture-of-Experts, Inference-Time Routing, Multimodal Hallucination, Visual Verification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jZa1OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RouteLedger, an inference-time framework for reducing hallucinations in MoE vision-language models. It uses expert routing differences to generate alternative answers and applies certificate-based checks to select reliable corrections. Beam Search serves as the fallback. The framework also introduces SafeSwitch-Margin, a task-gated CLIP-L/336 verifier for selecting non-route candidates. Experiments on 45,857 examples across 12 VQA datasets show an accuracy increase from 50.47% to 51.07%. However, my main concern is that the paper does not provide convincing evidence for its central routing-based contribution. Most of the improvement can be attributed to the CLIP verifier, while the routing-specific gain over the strongest matched baseline is small and not statistically significant. Moreover, the routing ablations do not clearly support the proposed mechanism.

Critical strengths
  1. The problem is interesting and relevant. Directly modifying expert routing can sometimes correct wrong predictions, but it may also damage correct ones. I think treating routing interventions as candidate proposals rather than directly replacing the original answer is a reasonable approach.
  2. The paper includes relatively comprehensive experiments. The authors evaluate the method on 12 datasets with over 45K examples. The ablation studies provide some useful insights. The authors investigate routing alignment, route divergence, and candidate dependency.
Critical weaknesses
  1. The contribution of routing seems rather limited. My main concern is that most of the performance gain comes from the CLIP verifier instead of the proposed routing mechanism. In Table 3, RouteLedger only improves over Beam+Anchor+Verifier by 22 correct predictions out of 45,857 examples, and the difference is not statistically significant (p=0.0903). This makes it difficult to judge how much the routing component actually contributes.
  2. The effectiveness of the routing information is not sufficiently demonstrated. In Appendix A.8, shuffling the aligned routing units only reduces accuracy by 0.08 percentage points. Moreover, Figure 5 shows that larger routing divergence does not lead to larger improvements. These results make me question whether the proposed routing alignment is really important for the final answer selection.
  3. The generalization and efficiency of the method remain unclear. The main results are based on a single MoE-LLaVA-Qwen backbone, while the additional cross-backbone routing experiments show inconsistent improvements. The method also requires multiple decoding and routing-tracing passes, but no actual inference latency is reported. Given the relatively small routing-specific gain, the additional computational cost should be better justified.
Questions for authors
  1. Given that Beam+Anchor+Verifier already achieves nearly the same performance, can the authors further clarify the unique contribution of routing? A comparison with a routing-free candidate generation method under a similar inference budget would be helpful.

  2. Why does shuffling the aligned routing units have such a small impact on accuracy? Can the authors provide additional analysis showing which routing signals are actually responsible for the improvements?

  3. Can the authors provide the inference latency and memory overhead compared with Beam+Anchor+Verifier? It would also be useful to see whether the complete method achieves consistent improvements on another MoE backbone.

4 comments
AuthorsOpenReview ↗

Part 1 of 2: routing increment and the extra-sampling control

Thank you for identifying the distinction between proposing an alternative through routing and deciding whether to trust it. That distinction is central to RouteLedger. We answer the three questions through the strongest matched control, an ordinary-sampling control, the exact acceptance conditions, and measured costs.

Q1. What does routing add beyond Beam+Anchor+Verifier, and how does it compare with extra non-route sampling?

On the submitted 45,857-example evaluation, Beam+Anchor+VT obtains 23,397 correct answers and RouteLedger+VT obtains 23,419. The routing-associated increment is +22 correct answers, or +0.048 percentage points, with paired 95% CI [-0.004, +0.102] percentage points and p=0.0903p=0.0903. The complete system's +277 over Beam measures the combined decision policy; the matched comparison above measures the routing-associated increment.

The specific capability is an additional route to a corroborated selection decision. Under Eqs. (6)–(9), there are two logical route entries. Since their support contribution is at most two and acceptance requires at least three votes, route agreement must be accompanied by support from a non-route entry. The candidate must also agree with the gate consensus and pass fidelity checks. Equation (12) then excludes route candidates from the visual-verification pool. These conditions prevent the route branch from approving an answer through its two entries alone.

Figure 6(a) makes this concrete. For “What is the woman on?”, Beam answers “boat,” and Beam+Anchor+VT selects “table under umbrella.” RouteLedger selects the correct “chair,” a corroborated answer that the matched verifier-based selector misses. Figure 6(c) illustrates the complementary decision of retaining the correct Beam answer.

For the requested generation-budget control, we replaced nine route-generated answers with nine independent non-route samples at temperature 0.2, preserving the frozen selector and non-route verifier pool. Each complete branch uses 16 generations; the routing branch additionally uses two routing captures.

Backbone, 600 questions eachFull+global VTExtra non-route sampling+global VTRouting disabled+global VT
Qwen395395395
Phi2436436436

The routing-disabled condition preserves candidate slots, decoding settings, seeds, selector, pool, and margin while zeroing intervention biases. It produces the same normalized final answers as the full branch on these questions, so routing introduces zero regressions under frozen transfer. This control matches the acceptance policy, while the sampling control matches generation counts. Together they separate the source of the proposals from the rule that admits them.

AuthorsOpenReview ↗

Part 2 of 2: shuffle effect, latency, memory and backbone transfer

Q2. Why is the shuffle effect small, and which routing signals affect the decision?

The distinction between a routing diagnostic and an acceptance variable is central. Equation (3) measures how frequently paired expert assignments differ. Equation (9) accepts an answer according to gate consensus, corroboration, and fidelity. A larger divergence is not itself a stronger acceptance score.

Figure 5 reports the following verifier-free comparison with Beam+Anchor:

Divergence quartileMean divergenceAdditional correct answers
Q10.1207+9
Q20.1393+7
Q30.1603+8
Q40.2219+9

The measured effect is therefore spread across the quartiles, consistent with an acceptance rule that does not use divergence as a confidence score.

The aligned-unit shuffle audit changes selector accuracy from 50.88% to 50.80%, with route-selection frequency changing from 17.5% to 17.1%. The base answer, other candidates, acceptance policy, and fallback remain fixed in that comparison. The small final-output effect follows from the acceptance rule: a perturbed proposal still has to win non-route corroboration, gate agreement, and fidelity checks before it can change the answer.

For reproducibility, the expanded repository audit uses fixed expert/layer boost pairs and frozen gate-bias configurations; experts are not selected per example by maximizing divergence. The variables directly controlling acceptance are the resulting normalized-answer consensus, support from the prescribed candidate sources, and fidelity predicate. This connects the mechanism to the decision actually made, rather than interpreting routing distance as an uncalibrated confidence measure.

These conditions make each selection decision inspectable: a changed answer has an identifiable proposal, corroborating candidate, and acceptance path. The matched controls then measure whether those decisions improve task accuracy.

Q3. What are latency and memory costs, and does the complete pipeline transfer to another backbone?

Complete-request measurements for the expanded repository audit are:

BackboneMethodMean secondsp95 secondsSerial requests/sPeak allocated MiB
QwenBeam+Anchor+global VT1.4593.6480.68610,202
QwenFull+global VT3.3708.2040.29710,202
Phi2Beam+Anchor+global VT1.7962.1970.55715,691
Phi2Full+global VT3.9604.3930.25315,691

The added mean latency is 1.911 seconds on Qwen and 2.165 seconds on Phi2, giving ratios of 2.31× and 2.21× relative to the matched non-route system. Sequential candidate execution gives the same measured peak allocated memory for the matched and full branches.

Configuration scope: this profile contains 16 generations and two shared routing captures, versus six generations for the matched non-route branch. It measures the expanded audit implementation, rather than the eight-decode/three-trace configuration listed in Section 4.1/Table 20.

The hardware is an RTX 4090, with batch size 1 and a 32-token output cap. The VLM and CLIP remain resident for both branches; loading is excluded. Three repetitions yield 30 timed observations over ten unique inputs per backbone on an otherwise idle GPU.

For transfer, we executed the complete pipeline on the same 300 POPE and 300 AI2D questions using MoE-LLaVA-Qwen and MoE-LLaVA-Phi2. All routing parameters, the selector, verifier, prompt, and margin were frozen; Phi2 receives the same fixed gate biases as Qwen.

Backbone, 600 questions eachBeamMatched, default gateFull, default gateMatched, global VTFull, global VT
Qwen399395395395395
Phi2442437437436436

The default task gate is off on POPE and AI2D. Global VT is a separate policy that removes dataset identity from the verification decision. On the second backbone, the route-enabled pipeline matches the non-route policy exactly under frozen transfer, so the corroboration rule keeps the route branch free of regressions without any re-tuning.

The full pipeline adds a corroborated correction path whose accuracy and computational effects are measured against the same non-route decision policy.

Thank you for the detailed response and additional experiments. I appreciate the effort the authors have put into addressing my concerns. The additional analyses have helped improve my overall assessment of the work. I raise my rating from 1 to 2.

Thank you for the reconsideration. Two measurements completed after our reply and bear on points in your review, so we add them briefly.

On cost, we re-timed the method under the pass accounting of Table 20, with every decode, capture, and CLIP call executed afresh over 100 unique inputs × 3 fresh-process repetitions per backbone. Relative to the matched non-route selector, Full takes 1.257× the mean time on MoE-LLaVA-Qwen and 1.236× on MoE-LLaVA-Phi2, with 0 MiB additional peak CUDA allocation. The 2.31× / 2.21× ratios in our earlier reply belong to the 16-generation expanded audit configuration.

On the design you highlighted, treating routing interventions as proposals rather than replacements, a frozen held-out cohort of 2,000 new VQAv2 questions per backbone quantifies the difference. Adopting the routed answers directly regresses 54–61 correct answers per backbone, while certificate admission regresses 1 on Qwen and 0 on Phi2.

Reviewer_fqrXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to use modality-dependent expert routing in mixture-of-experts vision-language models (MoE-VLMs) to improve answer selection. The main problem is that a model may give a wrong answer even when useful visual or textual information is available. The paper proposes RouteLedger, which compares routing patterns under different input settings, generates candidate answers based on routing information, and uses a visual-text verifier to decide whether to change the original answer.

Existing work has studied multimodal expert routing, test-time rerouting, and answer verification. The specific combination of routing-aware candidate selection and task-gated visual verification may be new, but the paper does not clearly show how much of the improvement comes from its routing mechanism itself.

The experiments cover 12 VQA datasets, with a large number of test examples and several useful ablation studies. The improvement over Beam + Anchor + verifier is small and not statistically significant (p=0.0903p=0.0903). The results suggest that the visual-text verifier provides most of the gain, while the additional contribution from routing-aware selection is limited.

The proposed method involves several components, routing comparisons, candidate-selection rules, and verifier mechanisms.Some of these details are technically dense, and the contribution of each component is not always easy to follow. The many ablation studies and statistical comparisons also require effort to understand the main takeaway.

Critical strengths

Strength 1: Careful experimental analysis. The paper includes several ablation studies and statistical tests, rather than reporting only the best result. It also analyzes where the verifier helps and where it changes correct answers into incorrect ones. This makes the experimental results more transparent.

Strength 2: Conservative answer selection and failure analysis. The method tries to avoid unnecessary changes to answers that may already be correct. The paper also reports the cases where verification causes regressions. This is a useful direction for reducing errors in vision-language models.

Critical weaknesses

Weakness 1: The routing-specific contribution is not clearly demonstrated. The full method improves accuracy by only 0.05 percentage points over Beam + Anchor + verifier, and this difference is not statistically significant. In addition, shuffling the routing units changes accuracy by only 0.08 percentage points, while the gains across routing-divergence groups are almost constant. These results make it difficult to conclude that the proposed routing information is the main reason for the improvement.

Weakness 2: The novelty and literature gap need to be better established. Several recent studies have investigated multimodal expert routing, test-time rerouting, and verification-based correction. The paper should more clearly explain how RouteLedger differs from these methods and what new research insight it provides. In particular, Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts (ACL 2026) studies routing differences between text and visual inputs and uses routing-guided intervention to improve model performance. Also, R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts (ICML 2025) directly optimizes routing weights at test time to improve multimodal task performance. These two papers are highly relevant and should be discussed in detail. The authors should explain more clearly what is different about RouteLedger and what new insight it provides beyond existing routing methods. More importantly, I suggest comparing RouteLedger with these two methods under the same backbone and benchmark settings, if technically feasible. Without this discussion and comparison, it is difficult to judge whether RouteLedger provides a meaningful new contribution or mainly combines existing routing and verification techniques.

Weakness 3: Evaluation on different models is limited. Most experiments use one main backbone. The additional tests on three other backbones are relatively small and show inconsistent results. Therefore, it is still unclear whether the method can work reliably across different MoE-VLM architectures.

Weakness 4: Evaluation independence and practical value need further validation. The appendix reports an earlier CLIP-B/32 candidate-pool audit of 19,319 examples, which exactly matches the total size of the four verifier-enabled datasets. The paper should clarify whether these examples overlap with the final evaluation set and whether any design decisions were influenced by test results. This does not by itself prove data leakage, but the evaluation protocol needs to be explained clearly. In addition, the paper does not report actual inference latency, memory use, or throughput, making it difficult to judge whether the method is practical.

Questions for authors
  1. Can the authors better isolate the contribution of routing information? Please provide a controlled comparison in which the routing information is the only changed component. Report confidence intervals, correction rates, regression rates, and additional inference costs.

  2. How does RouteLedger differ from recent routing and verification methods? Please discuss the closest related work, especially routing-guided interventions and test-time rerouting, and explain what new insight or capability RouteLedger provides beyond combining existing techniques.

  3. Does the 19,319-example CLIP-B/32 audit overlap with the final test set? Please clarify how this audit was used during method development. If the final evaluation was influenced by test-set results, an additional evaluation on a fresh held-out set would strengthen the paper.

  4. Can the authors provide stronger evidence of generalization? Please evaluate the method on more MoE-VLM backbones and, if possible, additional hallucination-focused benchmarks. Metrics such as correction precision, regression rate, and coverage-risk trade-offs would also help show when the method is useful.

  5. What is the actual inference cost of RouteLedger? Please report end-to-end latency, memory use, and throughput, compared with the baseline methods. More details about implementation and reproducibility would also help readers assess the practical value of the approach.

5 comments
AuthorsOpenReview ↗

Part 1 of 3: matched routing controls, correction and regression counts

Thank you for the detailed questions. RouteLedger addresses a specific decision problem: when should a route-derived answer be allowed to replace a strong existing answer? Its acceptance rule tracks candidate origin, requires corroboration and fidelity checks, and keeps route proposals outside the visual-verification pool. Below we separate the evidence for that decision rule, the complete system, and transfer to additional settings.

Q1. Can routing be isolated with matched controls, uncertainty, correction/regression counts, and costs?

The strongest submitted control is Beam+Anchor with the same visual verifier, candidate pool, task gate, and margin. It obtains 23,397 correct answers, compared with 23,419 for RouteLedger+VT on 45,857 examples: +22 correct answers, +0.048 percentage points, paired 95% CI [-0.004, +0.102] percentage points, p=0.0903p=0.0903. This is the comparison relevant to the incremental contribution of routing.

The complete system obtains +277 correct answers over Beam, with 797 recoveries and 520 regressions, paired 95% CI [+0.449, +0.759] percentage points and p=2.29×10−14p=2.29\times10^{-14}. These system and component comparisons retain their respective references.

We also performed a controlled intervention on 600 additional questions per backbone. The routing-disabled condition zeros intervention biases while preserving candidate slots, decoding settings, seeds, the selector, verifier pool, and margin. A second control replaces the nine route-generated answers with nine independent ordinary samples at temperature 0.2, keeping the selector and paper-only verification pool fixed.

BackboneFull+global VTRouting disabledNine extra non-route samplesFull vs routing-disabled: recovered / regressedPaired image-cluster 95% CI, pp
Qwen395/600395/600395/6000 / 0[0.000, 0.000]
Phi2436/600436/600436/6000 / 0[0.000, 0.000]

The full and routing-disabled conditions produce identical normalized final answers on these questions, so the routing branch introduces zero regressions on fresh data under frozen parameters. The extra-sampling comparison matches the number of generations, with two additional routing captures in the full branch. Actual request costs are given under Q5.

The acceptance mechanism explains what routing is permitted to do. Under Eqs. (6)–(9), the two logical route entries contribute at most two votes, whereas acceptance requires at least three support votes, agreement with the gate consensus, and passage through fidelity checks. A route answer therefore needs corroboration from at least one non-route entry. Equation (12) independently excludes route candidates from visual verification. These are explicit constraints on admission, with effectiveness evaluated by the matched outcome comparisons above.

AuthorsOpenReview ↗

Part 2 of 3: Seeing but Not Thinking, R2-T2 and evaluation independence

Q2. What distinguishes RouteLedger from Seeing but Not Thinking and R2-T2?

Both papers are directly relevant. The distinction concerns the object being controlled and the decision made after an intervention:

MethodIntervention and evidenceOutput decision
Seeing but Not ThinkingIdentifies domain-related experts from routing-frequency differences and applies soft routing guidanceUses the prediction produced under guided routing
R2-T2Optimizes routing weights at test time using correctly predicted reference neighbors in its multimodal expert architectureUses the prediction under the optimized routing
RouteLedgerProduces alternatives through routing interventions and compares normalized answers with a fixed candidate portfolioAdmits a route alternative after corroboration and fidelity checks; otherwise follows the verifier/base decision

The corresponding primary sources are Seeing but Not Thinking and R2-T2.

RouteLedger makes the accept/reject decision after routing explicit. It can reject an intervention's answer, select an answer already available from a non-route decoder when the strongest matched selector misses it, or preserve Beam. Figure 6(a) provides the second case: Beam predicts “boat,” Beam+Anchor+VT selects “table under umbrella,” and the route-corroborated decision selects the correct “chair.”

For a same-backbone comparison, we implemented a paper-based adaptation of Seeing but Not Thinking's expert-frequency criterion and standard-deviation-scaled soft bias. Each domain uses 20 input-only reference examples, with 20 general-text inputs; reference and evaluation images are disjoint. The frequency threshold 0.3 and bias coefficient 0.5 are fixed, and the random-expert control matches expert counts per layer.

This criterion selects zero experts on POPE and AI2D for both tested MoE-LLaVA backbones, so the resulting intervention is the identity operation and its outputs equal Beam. On these backbones, a frequency-based expert criterion finds no target, while RouteLedger's fixed proposals and acceptance rule remain applicable.

The official R2-T2 implementation optimizes routing over MoAI's vision-expert interfaces using reference neighbors. MoE-LLaVA routes inside the language decoder, so applying R2-T2 there would change its intervention target. The two methods therefore control different objects, and RouteLedger's contribution lies in the decision made after any such intervention.

The research insight is that changing expert activation and deciding whether to trust the resulting answer are separate operations. RouteLedger specifies the latter operation through candidate origin, corroboration, protected-content checks, and a verifier pool that excludes route proposals.

Q3. Did the 19,319-example audit overlap with evaluation, and what was frozen for the additional evaluation?

A fresh evaluation, frozen before any of its questions were scored, confirms the decision rule outside the original suite. It uses 300 POPE questions, balanced across adversarial/popular/random subsets and yes/no labels, plus 300 AI2D questions; AI2D is outside the original twelve-dataset suite. The intervention parameters, candidate recipe, selector, CLIP-L/336 verifier, prompt, and margin were all frozen beforehand, and the same 600 questions are evaluated on both backbones. Results are reported in full under Q4.

The CLIP-B/32 candidate-pool audit covers the 19,319 examples of the four verifier-enabled datasets. Its role was to compare candidate-pool structures, and it led to the rule in Eq. (12) that excludes route proposals from verification. The verifier threshold was selected separately: the shared margin τ = 1.0 was chosen on the 1,024-example training split, so no per-dataset or test-set accuracy tuning determines the margin. POPE draws its images from COCO; the additional questions themselves are new to every stage of development.

AuthorsOpenReview ↗

Part 3 of 3: second backbone, POPE, coverage metrics and inference cost

Q4. What happens on a complete second backbone, a hallucination benchmark, and correction/coverage metrics?

The additional experiments execute the complete candidate-generation and selection pipeline on MoE-LLaVA-Qwen-1.8B-4E and MoE-LLaVA-Phi2-2.7B-4E. POPE supplies the requested hallucination-focused evaluation; AI2D tests diagram questions. Phi2 uses the same fixed routing biases as Qwen, so this measures frozen parameter transfer.

BackboneTaskBeamMatched, default gateFull, default gateMatched, global VTFull, global VT
QwenPOPE, 300264261261261261
QwenAI2D, 300135134134134134
Phi2POPE, 300273270270270270
Phi2AI2D, 300169167167166166

“Matched” denotes the non-routing Beam+Anchor selector with the same verification policy. The default task gate is off on both tasks; global VT is a separate policy that does not use dataset identity. Routing changes no final normalized answers relative to the routing-disabled control on either backbone, so the frozen route branch transfers to a second backbone and two new tasks without introducing regressions. The small differences from Beam are shared exactly by the matched non-route selector and therefore arise in the anchor and verification stages on these yes/no and diagram tasks, independently of routing.

To make the acceptance trade-off explicit, the following diagnostic compares the frozen margin 1.0 with margin 0. The primary margin remains 1.0.

BackboneMarginNormalized answer changes / 600Recovered Beam errorsRegressed correct Beam answersCorrect / 600
Qwen0151 (25.17%)2428395
Qwen197 (16.17%)1115395
Phi2058 (9.67%)1226428
Phi2119 (3.17%)39436

This comparison shows the role of a positive margin: it reduces answer-changing coverage, and on Phi2 avoids 17 regressions while forgoing nine recoveries, yielding eight additional correct answers relative to margin 0.

For the primary global-VT configuration, correction precision, defined as recovered errors divided by all normalized answer changes, is 11/97 = 11.34% for Qwen and 3/19 = 15.79% for Phi2. Regression rates, using initially correct Beam answers as the denominator, are 15/399 = 3.76% and 9/442 = 2.04%, respectively. Correct-to-correct paraphrases are changes but not recovered errors. These metrics use Beam as their reference.

The thresholds compared here are the frozen margin and its zero-margin counterpart; no new threshold is selected from the additional evaluation.

Q5. What are actual latency, memory, throughput, and reproducibility details?

We measured complete requests on an RTX 4090, with batch size 1, a 32-token output cap, and sequential candidate execution:

BackboneMethodMean secondsp95 secondsSerial requests/sPeak allocated MiB
QwenBeam0.2530.4673.9539,544
QwenBeam+Anchor+global VT1.4593.6480.68610,202
QwenFull+global VT3.3708.2040.29710,202
Phi2Beam0.3180.6363.14113,941
Phi2Beam+Anchor+global VT1.7962.1970.55715,691
Phi2Full+global VT3.9604.3930.25315,691

Configuration scope: this is the expanded repository audit: six non-route generations, one unmodified greedy generation, six boost generations, three gate-corrector generations, and two shared routing captures. The full branch therefore has 16 generations, versus six for the matched non-route branch. This differs from the eight-decode/three-trace configuration listed in Section 4.1/Table 20; the table above measures the expanded audit.

The incremental mean latency is 1.911 seconds on Qwen and 2.165 seconds on Phi2, or 2.31× and 2.21× the matched baseline latency. The matched and full branches have the same peak allocated memory within each backbone. All rows retain the same resident VLM and CLIP models, including the Beam row; loading is excluded. Throughput is serial request throughput.

Each backbone has three repetitions of a fixed subset, producing 30 timed observations over ten unique inputs on an otherwise idle GPU. Repeated inputs reproduced 36/36 normalized final decisions and 900/900 complete candidate texts per backbone, so every decision is deterministic and repeatable.

Per-question candidates, decisions, and timing logs are retained, which makes the central decision inspectable: accept a corroborated route correction, apply the scoped verifier, or retain the base answer.

Reviewer_PikoOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RouteLedger, which uses expert-routing differences in a mixture-of-experts vision-language model to propose alternative answers. It compares routing under text-only, visual-prefix, and full multimodal inputs, and then applies agreement and fidelity checks before accepting a route-derived answer. When no route correction is accepted, a task-gated CLIP verifier selects among non-route candidates or preserves the base answer. Across 45,857 examples from twelve datasets, RouteLedger improves accuracy over Beam by approximately 0.60 percentage points. Ablation studies show that visual verification contributes most of the gain, but adding routing to Beam+Anchor with the same verifier yields only 22 additional correct answers. The paper therefore supports the overall selection recipe more strongly than the usefulness of aligned routing evidence.

Critical strengths
  1. The matched-verifier baseline is an important controlled experiment. Comparing RouteLedger with Beam+Anchor with the same verifier, candidate pool, task gate, and margin helps identify what routing actually contributes. The authors explicitly acknowledge that this comparison is not statistically significant, avoiding attributing of the total improvement to routing.
  2. The experimental results covers a substantial number of examples and diverse tasks. The twelve datasets include natural-image QA, OCR, documents, charts, and multimodal exams. This can show where the method helps and where it produces little benefit.
  3. The design of the conservative fallback is practical and makes sense. Treating route interventions as proposals and retaining a strong default answer can reduce unnecessary changes. Restricting the verifier to non-route candidates also prevents route proposals from supplying their own verification evidence.
Critical weaknesses
  1. The benefit of the routing mechanism is not very convincing. RouteLedger improves over the strongest matched non-routing control by only 22/45,857 examples. The small shuffled-unit effect and absence of a divergence–benefit relationship further weaken the case for the routing ledger.
  2. Some details of the route-proposal mechanism are not provided. Equation 4 introduces an intervention bias but does not explain in detail how experts, tokens, bias signs, or magnitudes are selected from the ledger. The construction of the two route candidates and the fidelity function also needs further explanation.
  3. The task-family gate limits general applicability. Verification is enabled for only four named dataset families and disabled for the rest. A fixed gate avoids test-time threshold selection, but still makes deployment depend on known task metadata. The paper should show whether the rule transfers to unseen datasets.
Questions for authors
  1. How is the intervention bias computed? Please specify the expert-selection rule, affected token positions, intervention strength, and construction of each route candidate.
  2. How are correlations and duplicated evidence across route candidates, decoding methods, and anchor votes handled?
  3. Does the task-family gate work on unseen datasets or without dataset identity?
  4. What is the incremental cost of routing? Please compare latency, memory, and accuracy against the strongest matched non-routing baseline.
4 comments
AuthorsOpenReview ↗

Part 1 of 2: intervention recipe and correlated candidates

Thank you for recognizing the matched-verifier control and the value of retaining a strong default answer. The central design choice is that a change in routing proposes an answer; replacing the default requires a separate acceptance decision. We clarify the intervention recipe, the evidence counted by that decision, and its measured accuracy and cost below.

Q1. How are experts, token positions, intervention strengths, route candidates, and fidelity checks specified?

Equation (4) adds a bias to router logits before Top-K selection:

g~ℓ,t,e=gℓ,t,e+αℓ,t,e.\widetilde g_{\ell,t,e}=g_{\ell,t,e}+\alpha_{\ell,t,e}.

The concrete recipe in the frozen repository implementation used for our supplementary audit is as follows:

Expert and layer selection. Six boost proposals use the fixed pairs (expert 0, layer 12) and (expert 2, layer 14), each at strengths 2.5, 5, and 10. These intervention settings are fixed in advance for every example.

Token positions and signs. The boost is positive and applies at all router positions during prefill and decoding; all other bias entries are zero.

Two logical route entries. The direct route answer uses expert 2 at layer 14 with strength 2.5. The route-vote answer aggregates the normalized answers of the six boost proposals.

Gate consensus. A separate ensemble uses the full gate-bias configuration below together with two variants that remove layer 8 or layer 14. At least two of its normalized answers must agree.

LayerExpert 0Expert 1Expert 2Expert 3
80-2.52.5-2.5
10-2.52.50-2.5
12-2.5-2.52.5-2.5
142.5-2.52.50

This distinguishes the intervention operator in Eq. (4), the fixed proposal recipe, and the subsequent acceptance rule. The paired ledger aligns the shared text suffix across modality views, excluding modality-only prefix positions.

The fidelity predicate checks protected answer content and form, including numbers, dates/units, quoted OCR text, visual attributes, spatial expressions, answer type, and inappropriate shortening or detail changes, according to the applicable branch. “Certificate” denotes passage through this deterministic acceptance predicate. Image–text compatibility is assessed separately by the verifier.

Q2. How are correlated candidates and duplicated votes handled?

The candidates share a backbone, and anchor/vote entries reuse their parent outputs. We treat their agreement as a selection criterion, rather than as independent statistical observations.

Two structural constraints are useful here. First, under Eqs. (6)–(9), there are two logical route entries, so their combined contribution satisfies Ert(a)≤2E_{\mathrm{rt}}(a)\leq 2. Acceptance requires

Egate(a)=1,Enr(a)+Ert(a)≥3,R(a,abase)=0,E_{\mathrm{gate}}(a)=1,\qquad E_{\mathrm{nr}}(a)+E_{\mathrm{rt}}(a)\geq 3,\qquad R(a,a_{\mathrm{base}})=0,

as well as a normalized answer different from the base. Agreement confined to the two route entries cannot approve a correction: at least one non-route entry must corroborate it.

Second, Eq. (12) restricts visual verification to the non-route pool and deduplicates normalized answer strings. A route proposal cannot enter that pool as an additional candidate to be visually verified. The verifier is consulted when the guarded selector still retains Beam, and a replacement must exceed Beam's image–text score by the fixed margin.

Statistical significance is assessed from paired question outcomes, not from the number of agreeing candidates.

Figure 6(a) illustrates the resulting decision: Beam predicts “boat,” Beam+Anchor+VT selects “table under umbrella,” and RouteLedger selects the correct “chair,” a corroborated answer that the matched selector misses. Figure 6(c) illustrates retaining the correct Beam answer when the matched selector changes it.

AuthorsOpenReview ↗

Part 2 of 2: gate transfer, routing increment and cost

Q3. Does the gate transfer to unseen datasets or operate without dataset identity?

The submitted gate encodes compatibility with the visual verifier: it enables the same CLIP model, prompt, and margin for the TextVQA, DocVQA, VQAv2, and VizWiz families, where answers are expected to be visually grounded, and leaves arithmetic, chart reasoning, and external-knowledge questions to the base decision.

We evaluated the frozen pipeline on 600 additional questions: 300 POPE and 300 AI2D questions, using both MoE-LLaVA-Qwen and MoE-LLaVA-Phi2. We also evaluated a separate identity-free policy that enables the verifier globally, keeping the candidate pool, prompt, and margin fixed.

Backbone, 600 questions eachBeamMatched non-route, default gateFull, default gateMatched non-route, global VTFull, global VT
Qwen399395395395395
Phi2442437437436436

Entries are correct-answer counts. The default gate is off on these two tasks. With the verifier enabled globally, the full and matched systems remain identical on both backbones, and accuracy stays within one answer of the default gate (395 vs 395 on Qwen, 436 vs 437 on Phi2). The pipeline therefore operates without dataset identity. The task gate specifies where the verifier is used, while the margin specifies when a candidate can replace Beam within that scope.

The intervention parameters, selector, verifier, and margin were frozen before evaluating these questions. Phi2 uses the same fixed gate biases as Qwen, making this a test of frozen parameter transfer.

Q4. What is the routing increment in accuracy and computational cost?

For the submitted 45,857-example evaluation, the strongest matched non-route baseline is Beam+Anchor+VT with 23,397 correct answers; RouteLedger+VT obtains 23,419. The routing-associated increment is therefore +22 correct answers, or +0.048 percentage points, with paired 95% CI [-0.004, +0.102] percentage points and p=0.0903p=0.0903. The complete system's +277 over Beam is a different comparison and is reported as a system result.

The divergence and shuffle audits address different quantities. Equation (3) is diagnostic; acceptance uses answer agreement and fidelity in Eq. (9). Consequently, the selector does not predict monotonically increasing benefit with routing divergence. Shuffling aligned units changes the reported selector accuracy from 50.88% to 50.80%, which measures end-to-end sensitivity of the selector in that audit.

We additionally measured complete requests for the expanded repository implementation:

BackboneMethodMean secondsp95 secondsSerial requests/sPeak allocated MiB
QwenBeam+Anchor+global VT1.4593.6480.68610,202
QwenFull+global VT3.3708.2040.29710,202
Phi2Beam+Anchor+global VT1.7962.1970.55715,691
Phi2Full+global VT3.9604.3930.25315,691

Configuration scope: these timings measure the expanded audit with 16 generations and two routing captures, compared with six generations for the matched non-route branch. They are not a timing measurement of the eight-decode/three-trace configuration listed in Section 4.1/Table 20.

Measurements use an RTX 4090, batch size 1, a 32-token output cap, sequential candidate execution, and three repetitions of a fixed subset, yielding 30 timed observations over ten unique inputs per backbone after exclusions. Model loading is excluded; the same VLM and CLIP models remain resident in both branches. GPU-process sampling observed no competing compute process during accepted runs.

In this measured configuration, routing adds 1.911 seconds per request on Qwen and 2.165 seconds on Phi2, corresponding to 2.31× and 2.21× the matched baseline latency. Peak allocated memory is unchanged between the matched and full branches. Thus the additional work is visible in decoding and tracing time, with sequential execution reusing resident model memory.

The contribution is an explicit way to admit corroborated route corrections while controlling their interaction with verification and fallback. The matched controls quantify that contribution separately from the complete system's accuracy gain.

AuthorsOpenReview ↗

Follow-up: reference-recipe cost and held-out admission behavior on two backbones

Two measurements have completed since our first reply. They answer Q4 with the paper's reference recipe rather than the expanded audit configuration timed earlier, and they extend Q2 and Q3 to new questions, new images, and a second backbone with all settings unchanged.

Q4, cost against the strongest matched non-routing baseline. The timed recipe follows the pass accounting of Table 20: six non-route decodes (original, VCD, PAI, PAI-CFG, Beam, OPERA), three captures, up to two nonempty route decodes, and one CLIP-L/336 call. Every pass executes afresh. Measurements use 100 unique inputs × 3 fresh-process repetitions per backbone, with two complete warmups excluded per repetition, on an RTX 4090 with VLM and CLIP resident and loading excluded.

Backbone / procedureMean sp95 sSerial requests/sPeak CUDA allocated MiB
Qwen / matched non-route+VT5.6959.5810.17610,067.2
Qwen / Full7.15611.8750.14010,067.2
Phi2 / matched non-route+VT8.38013.5370.11915,354.2
Phi2 / Full10.36116.7250.09715,354.2

Routing adds 1.461 / 1.980 s, or 1.257× / 1.236× the matched mean, with 0 MiB additional peak allocation; peak reserved memory is also equal at 10,504 / 17,484 MiB. The repetitions reproduce 4,785/4,785 candidate texts and 1,200/1,200 final decision records. The earlier 2.31× / 2.21× ratios belong to the 16-generation expanded audit over ten inputs.

Q2, admission behavior on held-out data. The cohort contains 2,000 VQAv2 questions on 1,918 images, shared by MoE-LLaVA-Qwen and MoE-LLaVA-Phi2. It excludes the reconstructed historical first-5,000 question IDs, their 883 image IDs, and decoded-RGB overlap with the earlier 600-question pool; labels are joined offline after decisions are stored. The table applies two policies to the same saved route proposals before SafeSwitch, scored by repository matching against the same calibrated base.

Policy before SafeSwitchQwen: rewrites / recovered / regressedPhi2: rewrites / recovered / regressed
Direct T→VT1,023 / 53 / 591,053 / 41 / 55
Direct V→VT1,029 / 53 / 611,021 / 42 / 54
Certificate admission12 / 0 / 111 / 0 / 0

Direct adoption regresses 54–61 correct answers per backbone. Corroboration and fidelity reduce this to 1 and 0, preserving 1,448/1,449 and 1,458/1,458 matched-correct base answers. Because two route entries supply at most two of the three required supports, every admitted residual carries non-route corroboration, and the verifier pool separately excludes route entries.

Q3, transfer with the gate unchanged. VQAv2 is a verifier-enabled family, so the cohort tests example and backbone transfer within an enabled family with the same prompt and margin 1. Unknown or unsupported families keep the verifier-disabled base/route path by design.

Reviewer_PikoOpenReview ↗

Responses for Authors

Thanks the authors for their detailed responses. My concern over the marginal improvements of the RouteLedger (Table 1 and Table 3) is not fully resolved. This is critical to demonstrate the effectiveness of the proposed RouteLedger.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.