- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies hidden-state probes as routers in a two-model hybrid system. A small MLLM either keeps a query or escalates it to a large model (GPT-4.1), based on a predicted score for whether the small model is correct. Extensive experiments on five multimodal benchmarks and three backbones show that the proposed ReLope, which trains a LoRA copy of one layer with a KL-regularized stochastic bottleneck, achieves the best correctness prediction AUC in all 15 settings and improves the accuracy versus cost trade-off at about 1.8% extra inference time.
Critical strengths
-
Broad and consistent gains at low additional inference cost. ReLope achieves the best mean correctness-prediction AUC in all 15 dataset–backbone combinations, with results reported over three seeds. Its improvements over the standard Probe are often substantial: +8.13 AUC points on ScienceQA with Qwen, +10.68 on A-OKVQA with Gemma, and +8.98 on MMMU with Phi. A parameter-matched widened MLP still trails ReLope by 2.14–5.65 AUC points in Table 5, suggesting that the improvements cannot be explained by parameter count alone. In the reported H200 benchmark, ReLope adds approximately 0.31 ms per example (1.79%) and 0.11 GiB of peak memory relative to the standard Probe. The combination of sizeable gains across models and tasks with modest measured local overhead is a major strength.
-
A simple design motivated by an interesting empirical observation. ReLope adds a separately adapted copy of a single Transformer layer with LoRA, leaving the small model's answer-generation path unchanged. The motivation is supported by an intriguing finding: replacing images with captions lowers the small model's answer accuracy but makes correctness easier to predict from its hidden states. The authors investigate this with a trained Probe and two classifier-free dependence measures (HSIC and label CKA), and report the same overall image-versus-caption pattern on three benchmarks. Table 3 also connects the proposed methods to this observation: ReLope reduces the image–caption AUC gap from 3.31 points for the standard Probe to 0.93 points.
Critical weaknesses
-
The supervised training data and split construction are unclear for MMMU and MathVision. Appendix A.1 states that each router is trained on a 90/10 split of the "official training pool" and evaluated on the "official validation split." However, MMMU provides dev (150), validation (900), and test (10,500) splits, but no designated training split. Likewise, MathVision provides test (3,040) and testmini (304).
-
The experiments do not establish which part of the improvement is specifically multimodal. Attention pooling over token states, adaptation of a copied Transformer layer with LoRA, and a Gaussian bottleneck on the last-token representation are all applicable to text-only LLMs. This does not invalidate the approach for MLLMs, but the paper does not isolate whether ReLope addresses the particular multimodal difficulty identified in Section 3 or provides a generally stronger correctness Probe. Indeed, Table 3 shows that ReLope also improves the caption-only condition over the standard Probe (87.17 versus 85.25 AUC).
-
The paper's two claims about the KL bottleneck, preserving useful multimodal information and improving transfer, are not verified. Section 4.2 argues that the bottleneck keeps multimodal information useful for correctness while discarding unrelated visual details. However, the KL penalty is applied to a single last-token vector and only limits the total amount of information about the input. Nothing in the loss tells it which information to keep or discard, and the paper gives no analysis of what the bottleneck actually removes. The ablations also suggest a more ordinary role. On frozen states, the bottleneck adds only about 0.7 AUC points on average (VIB + Probe), while removing KL from ReLope reduces A-OKVQA AUC by 2.16 points (Table 7). This pattern is also consistent with KL acting as a regularizer for the adapted layer rather than specifically filtering irrelevant visual information. The transfer evidence is also weaker than the paper's interpretation. On Qwen, ReLope's mean AUC advantage over Probe falls from 4.90 in-domain (Table 2) to 2.13 on held-out datasets (Table 8), and ReLope's AUC drops more than Probe's on four of five targets.
Questions for authors
-
Please clarify the training data source for MMMU and MathVision. Which data were the routers trained and developed on for these two benchmarks, and how large were the splits?
-
Could you report results on some text benchmarks and discuss W2 on that basis? Running Probe, Attention Probe, and ReLope on a few text benchmarks such as MMLU, TriviaQA, or GSM8K. If ReLope's advantage turns out to be larger for multimodal inputs, it would help to show which part of the design is responsible.
-
Can you show what the KL bottleneck keeps and discards, and whether it actually helps transfer?
2 comments
Dear Reviewer QZ9m,
We have revised the manuscript and added clarifications and controls addressing your questions. If time permits, we would appreciate your reviewing our response and sharing any remaining concerns.
Thank you again for your thoughtful feedback.
Response to Reviewer QZ9m
Thank you for the questions about data, multimodal specificity, and bottleneck interpretation. We address each below.
W1 & Q1. MMMU and MathVision data
For MMMU, the 150-example official
devsplit is stratified by correctness label into 135 training and 15 development examples; development AUC selects the checkpoint, and all 900 officialvalidationexamples form the test pool. For MathVision, the 2,736testexamples excludingtestminiform the training pool, split 90/10 into training and development, while the disjoint 304-exampletestminiis the test pool. This 90/10 protocol and development-AUC selection apply to all datasets. We clarify these sources rather than calling them all official training splits.W2 & Q2. What is multimodal-specific?
Thank you for asking us to define the method's scope more precisely. The components are not exclusive to MLLMs, but their organization addresses the weak probe performance observed in multimodal states. The key question is whether gains are larger for image-conditioned states. Table 3 shows that ReLope improves over Probe by 4.30 points with images versus 1.92 with captions.
We also apply component controls to identical ScienceQA questions under original-image and caption-only inputs.
The much larger multimodal gain supports ReLope as a response to this difficulty without claiming that its components apply only to images.
As a scope check, we apply the same routers to three text-only benchmarks.
The small text-only gains support our multimodal focus rather than universal improvement.
W3 & Q3. Bottleneck interpretation and transfer
KL does not identify irrelevant visual details. Formally, the expected KL to the prior upper-bounds I(H;Z) and constrains information transmitted from the adapted state to the decision representation. Joint optimization with correctness loss encourages this representation to retain routing-relevant information. We therefore replace “discards irrelevant visual details” with “regularizes input-specific information while preserving correctness-related information.”
We report held-out HSIC and label CKA before and after the bottleneck under both conditions.
The posterior mean depends more strongly on correctness in both conditions, supporting preservation and concentration of correctness-related information without implying that KL assigns semantic relevance to visual details.
We do not claim that KL independently improves transfer. Table 8 evaluates the complete ReLope branch and shows higher absolute cross-dataset AUC than Probe on all five targets; a larger drop from a higher in-domain value does not imply lower held-out performance. We additionally vary only the KL coefficient:
KL improves transfer to some targets, especially MMMU at larger values.