- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RouteLedger, an inference-time framework for reducing hallucinations in MoE vision-language models. It uses expert routing differences to generate alternative answers and applies certificate-based checks to select reliable corrections. Beam Search serves as the fallback. The framework also introduces SafeSwitch-Margin, a task-gated CLIP-L/336 verifier for selecting non-route candidates. Experiments on 45,857 examples across 12 VQA datasets show an accuracy increase from 50.47% to 51.07%. However, my main concern is that the paper does not provide convincing evidence for its central routing-based contribution. Most of the improvement can be attributed to the CLIP verifier, while the routing-specific gain over the strongest matched baseline is small and not statistically significant. Moreover, the routing ablations do not clearly support the proposed mechanism.
Critical strengths
- The problem is interesting and relevant. Directly modifying expert routing can sometimes correct wrong predictions, but it may also damage correct ones. I think treating routing interventions as candidate proposals rather than directly replacing the original answer is a reasonable approach.
- The paper includes relatively comprehensive experiments. The authors evaluate the method on 12 datasets with over 45K examples. The ablation studies provide some useful insights. The authors investigate routing alignment, route divergence, and candidate dependency.
Critical weaknesses
- The contribution of routing seems rather limited. My main concern is that most of the performance gain comes from the CLIP verifier instead of the proposed routing mechanism. In Table 3, RouteLedger only improves over Beam+Anchor+Verifier by 22 correct predictions out of 45,857 examples, and the difference is not statistically significant (p=0.0903). This makes it difficult to judge how much the routing component actually contributes.
- The effectiveness of the routing information is not sufficiently demonstrated. In Appendix A.8, shuffling the aligned routing units only reduces accuracy by 0.08 percentage points. Moreover, Figure 5 shows that larger routing divergence does not lead to larger improvements. These results make me question whether the proposed routing alignment is really important for the final answer selection.
- The generalization and efficiency of the method remain unclear. The main results are based on a single MoE-LLaVA-Qwen backbone, while the additional cross-backbone routing experiments show inconsistent improvements. The method also requires multiple decoding and routing-tracing passes, but no actual inference latency is reported. Given the relatively small routing-specific gain, the additional computational cost should be better justified.
Questions for authors
-
Given that Beam+Anchor+Verifier already achieves nearly the same performance, can the authors further clarify the unique contribution of routing? A comparison with a routing-free candidate generation method under a similar inference budget would be helpful.
-
Why does shuffling the aligned routing units have such a small impact on accuracy? Can the authors provide additional analysis showing which routing signals are actually responsible for the improvements?
-
Can the authors provide the inference latency and memory overhead compared with Beam+Anchor+Verifier? It would also be useful to see whether the complete method achieves consistent improvements on another MoE backbone.
4 comments
Part 2 of 2: shuffle effect, latency, memory and backbone transfer
Q2. Why is the shuffle effect small, and which routing signals affect the decision?
The distinction between a routing diagnostic and an acceptance variable is central. Equation (3) measures how frequently paired expert assignments differ. Equation (9) accepts an answer according to gate consensus, corroboration, and fidelity. A larger divergence is not itself a stronger acceptance score.
Figure 5 reports the following verifier-free comparison with Beam+Anchor:
| Divergence quartile | Mean divergence | Additional correct answers |
|---|---|---|
| Q1 | 0.1207 | +9 |
| Q2 | 0.1393 | +7 |
| Q3 | 0.1603 | +8 |
| Q4 | 0.2219 | +9 |
The measured effect is therefore spread across the quartiles, consistent with an acceptance rule that does not use divergence as a confidence score.
The aligned-unit shuffle audit changes selector accuracy from 50.88% to 50.80%, with route-selection frequency changing from 17.5% to 17.1%. The base answer, other candidates, acceptance policy, and fallback remain fixed in that comparison. The small final-output effect follows from the acceptance rule: a perturbed proposal still has to win non-route corroboration, gate agreement, and fidelity checks before it can change the answer.
For reproducibility, the expanded repository audit uses fixed expert/layer boost pairs and frozen gate-bias configurations; experts are not selected per example by maximizing divergence. The variables directly controlling acceptance are the resulting normalized-answer consensus, support from the prescribed candidate sources, and fidelity predicate. This connects the mechanism to the decision actually made, rather than interpreting routing distance as an uncalibrated confidence measure.
These conditions make each selection decision inspectable: a changed answer has an identifiable proposal, corroborating candidate, and acceptance path. The matched controls then measure whether those decisions improve task accuracy.
Q3. What are latency and memory costs, and does the complete pipeline transfer to another backbone?
Complete-request measurements for the expanded repository audit are:
| Backbone | Method | Mean seconds | p95 seconds | Serial requests/s | Peak allocated MiB |
|---|---|---|---|---|---|
| Qwen | Beam+Anchor+global VT | 1.459 | 3.648 | 0.686 | 10,202 |
| Qwen | Full+global VT | 3.370 | 8.204 | 0.297 | 10,202 |
| Phi2 | Beam+Anchor+global VT | 1.796 | 2.197 | 0.557 | 15,691 |
| Phi2 | Full+global VT | 3.960 | 4.393 | 0.253 | 15,691 |
The added mean latency is 1.911 seconds on Qwen and 2.165 seconds on Phi2, giving ratios of 2.31× and 2.21× relative to the matched non-route system. Sequential candidate execution gives the same measured peak allocated memory for the matched and full branches.
Configuration scope: this profile contains 16 generations and two shared routing captures, versus six generations for the matched non-route branch. It measures the expanded audit implementation, rather than the eight-decode/three-trace configuration listed in Section 4.1/Table 20.
The hardware is an RTX 4090, with batch size 1 and a 32-token output cap. The VLM and CLIP remain resident for both branches; loading is excluded. Three repetitions yield 30 timed observations over ten unique inputs per backbone on an otherwise idle GPU.
For transfer, we executed the complete pipeline on the same 300 POPE and 300 AI2D questions using MoE-LLaVA-Qwen and MoE-LLaVA-Phi2. All routing parameters, the selector, verifier, prompt, and margin were frozen; Phi2 receives the same fixed gate biases as Qwen.
| Backbone, 600 questions each | Beam | Matched, default gate | Full, default gate | Matched, global VT | Full, global VT |
|---|---|---|---|---|---|
| Qwen | 399 | 395 | 395 | 395 | 395 |
| Phi2 | 442 | 437 | 437 | 436 | 436 |
The default task gate is off on POPE and AI2D. Global VT is a separate policy that removes dataset identity from the verification decision. On the second backbone, the route-enabled pipeline matches the non-route policy exactly under frozen transfer, so the corroboration rule keeps the route branch free of regressions without any re-tuning.
The full pipeline adds a corroborated correction path whose accuracy and computational effects are measured against the same non-route decision policy.
Thank you for the detailed response and additional experiments. I appreciate the effort the authors have put into addressing my concerns. The additional analyses have helped improve my overall assessment of the work. I raise my rating from 1 to 2.
Thank you for the reconsideration. Two measurements completed after our reply and bear on points in your review, so we add them briefly.
On cost, we re-timed the method under the pass accounting of Table 20, with every decode, capture, and CLIP call executed afresh over 100 unique inputs × 3 fresh-process repetitions per backbone. Relative to the matched non-route selector, Full takes 1.257× the mean time on MoE-LLaVA-Qwen and 1.236× on MoE-LLaVA-Phi2, with 0 MiB additional peak CUDA allocation. The 2.31× / 2.21× ratios in our earlier reply belong to the 16-generation expanded audit configuration.
On the design you highlighted, treating routing interventions as proposals rather than replacements, a frozen held-out cohort of 2,000 new VQAv2 questions per backbone quantifies the difference. Adopting the routed answers directly regresses 54–61 correct answers per backbone, while certificate admission regresses 1 on Qwen and 0 on Phi2.
Part 1 of 2: routing increment and the extra-sampling control
Thank you for identifying the distinction between proposing an alternative through routing and deciding whether to trust it. That distinction is central to RouteLedger. We answer the three questions through the strongest matched control, an ordinary-sampling control, the exact acceptance conditions, and measured costs.
Q1. What does routing add beyond Beam+Anchor+Verifier, and how does it compare with extra non-route sampling?
On the submitted 45,857-example evaluation, Beam+Anchor+VT obtains 23,397 correct answers and RouteLedger+VT obtains 23,419. The routing-associated increment is +22 correct answers, or +0.048 percentage points, with paired 95% CI [-0.004, +0.102] percentage points and p=0.0903. The complete system's +277 over Beam measures the combined decision policy; the matched comparison above measures the routing-associated increment.
The specific capability is an additional route to a corroborated selection decision. Under Eqs. (6)–(9), there are two logical route entries. Since their support contribution is at most two and acceptance requires at least three votes, route agreement must be accompanied by support from a non-route entry. The candidate must also agree with the gate consensus and pass fidelity checks. Equation (12) then excludes route candidates from the visual-verification pool. These conditions prevent the route branch from approving an answer through its two entries alone.
Figure 6(a) makes this concrete. For “What is the woman on?”, Beam answers “boat,” and Beam+Anchor+VT selects “table under umbrella.” RouteLedger selects the correct “chair,” a corroborated answer that the matched verifier-based selector misses. Figure 6(c) illustrates the complementary decision of retaining the correct Beam answer.
For the requested generation-budget control, we replaced nine route-generated answers with nine independent non-route samples at temperature 0.2, preserving the frozen selector and non-route verifier pool. Each complete branch uses 16 generations; the routing branch additionally uses two routing captures.
The routing-disabled condition preserves candidate slots, decoding settings, seeds, selector, pool, and margin while zeroing intervention biases. It produces the same normalized final answers as the full branch on these questions, so routing introduces zero regressions under frozen transfer. This control matches the acceptance policy, while the sampling control matches generation counts. Together they separate the source of the proposals from the rule that admits them.