acceptodds
Under review as a conference paper at ICLR 2027

ReLope: From Hidden-State Probing to a Decision Module for Multimodal LLM Routing

Abstract

Routing balances performance and cost in hybrid systems by escalating selected queries from a lightweight model to a powerful but expensive one. Hidden-state probes provide an effective routing signal by predicting the small model's correctness from hidden states it already computes. Although effective in text-only settings, their behavior on multimodal LLMs (MLLMs) is less understood. We find that correctness is harder to predict from hidden states when questions are paired with images rather than with captions of those images. Two training-free dependence measures, HSIC and label CKA, show the same pattern. We first investigate whether changing hidden states used by the probe improves correctness prediction for MLLMs. We propose Attention Probe, which learns an attention query to pool frozen token states into a routing feature and improves over the standard probe. However, pooling only recombines hidden states computed for answer generation, not for deciding whether the answer should be trusted. We therefore build a decision layer inside the existing MLLM and train it for the routing decision. The resulting router, ReLope (KL-Regularized LoRA Probe), adapts a separate copy of one MLLM layer with LoRA and regularizes it with a KL-penalized stochastic bottleneck. This decision branch shares the frozen lower layers and leaves the model's answer-generation path unchanged. ReLope attains the highest correctness prediction AUC on all five benchmarks and three backbones we evaluate and improves the accuracy versus cost trade-off at negligible overhead. These results suggest a new perspective and a practical technical path on routing: rather than reading decisions off states built for generation, routing decisions can be learned by a dedicated decision layer inside the model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal LLM, Route

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QZ9mOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies hidden-state probes as routers in a two-model hybrid system. A small MLLM either keeps a query or escalates it to a large model (GPT-4.1), based on a predicted score for whether the small model is correct. Extensive experiments on five multimodal benchmarks and three backbones show that the proposed ReLope, which trains a LoRA copy of one layer with a KL-regularized stochastic bottleneck, achieves the best correctness prediction AUC in all 15 settings and improves the accuracy versus cost trade-off at about 1.8% extra inference time.

Critical strengths
  1. Broad and consistent gains at low additional inference cost. ReLope achieves the best mean correctness-prediction AUC in all 15 dataset–backbone combinations, with results reported over three seeds. Its improvements over the standard Probe are often substantial: +8.13 AUC points on ScienceQA with Qwen, +10.68 on A-OKVQA with Gemma, and +8.98 on MMMU with Phi. A parameter-matched widened MLP still trails ReLope by 2.14–5.65 AUC points in Table 5, suggesting that the improvements cannot be explained by parameter count alone. In the reported H200 benchmark, ReLope adds approximately 0.31 ms per example (1.79%) and 0.11 GiB of peak memory relative to the standard Probe. The combination of sizeable gains across models and tasks with modest measured local overhead is a major strength.

  2. A simple design motivated by an interesting empirical observation. ReLope adds a separately adapted copy of a single Transformer layer with LoRA, leaving the small model's answer-generation path unchanged. The motivation is supported by an intriguing finding: replacing images with captions lowers the small model's answer accuracy but makes correctness easier to predict from its hidden states. The authors investigate this with a trained Probe and two classifier-free dependence measures (HSIC and label CKA), and report the same overall image-versus-caption pattern on three benchmarks. Table 3 also connects the proposed methods to this observation: ReLope reduces the image–caption AUC gap from 3.31 points for the standard Probe to 0.93 points.

Critical weaknesses
  1. The supervised training data and split construction are unclear for MMMU and MathVision. Appendix A.1 states that each router is trained on a 90/10 split of the "official training pool" and evaluated on the "official validation split." However, MMMU provides dev (150), validation (900), and test (10,500) splits, but no designated training split. Likewise, MathVision provides test (3,040) and testmini (304).

  2. The experiments do not establish which part of the improvement is specifically multimodal. Attention pooling over token states, adaptation of a copied Transformer layer with LoRA, and a Gaussian bottleneck on the last-token representation are all applicable to text-only LLMs. This does not invalidate the approach for MLLMs, but the paper does not isolate whether ReLope addresses the particular multimodal difficulty identified in Section 3 or provides a generally stronger correctness Probe. Indeed, Table 3 shows that ReLope also improves the caption-only condition over the standard Probe (87.17 versus 85.25 AUC).

  3. The paper's two claims about the KL bottleneck, preserving useful multimodal information and improving transfer, are not verified. Section 4.2 argues that the bottleneck keeps multimodal information useful for correctness while discarding unrelated visual details. However, the KL penalty is applied to a single last-token vector and only limits the total amount of information about the input. Nothing in the loss tells it which information to keep or discard, and the paper gives no analysis of what the bottleneck actually removes. The ablations also suggest a more ordinary role. On frozen states, the bottleneck adds only about 0.7 AUC points on average (VIB + Probe), while removing KL from ReLope reduces A-OKVQA AUC by 2.16 points (Table 7). This pattern is also consistent with KL acting as a regularizer for the adapted layer rather than specifically filtering irrelevant visual information. The transfer evidence is also weaker than the paper's interpretation. On Qwen, ReLope's mean AUC advantage over Probe falls from 4.90 in-domain (Table 2) to 2.13 on held-out datasets (Table 8), and ReLope's AUC drops more than Probe's on four of five targets.

Questions for authors
  1. Please clarify the training data source for MMMU and MathVision. Which data were the routers trained and developed on for these two benchmarks, and how large were the splits?

  2. Could you report results on some text benchmarks and discuss W2 on that basis? Running Probe, Attention Probe, and ReLope on a few text benchmarks such as MMLU, TriviaQA, or GSM8K. If ReLope's advantage turns out to be larger for multimodal inputs, it would help to show which part of the design is responsible.

  3. Can you show what the KL bottleneck keeps and discards, and whether it actually helps transfer?

2 comments
AuthorsOpenReview ↗

Response to Reviewer QZ9m

Thank you for the questions about data, multimodal specificity, and bottleneck interpretation. We address each below.

W1 & Q1. MMMU and MathVision data

For MMMU, the 150-example official dev split is stratified by correctness label into 135 training and 15 development examples; development AUC selects the checkpoint, and all 900 official validation examples form the test pool. For MathVision, the 2,736 test examples excluding testmini form the training pool, split 90/10 into training and development, while the disjoint 304-example testmini is the test pool. This 90/10 protocol and development-AUC selection apply to all datasets. We clarify these sources rather than calling them all official training splits.

W2 & Q2. What is multimodal-specific?

Thank you for asking us to define the method's scope more precisely. The components are not exclusive to MLLMs, but their organization addresses the weak probe performance observed in multimodal states. The key question is whether gains are larger for image-conditioned states. Table 3 shows that ReLope improves over Probe by 4.30 points with images versus 1.92 with captions.

We also apply component controls to identical ScienceQA questions under original-image and caption-only inputs.

MethodOriginal AUCCaption AUCGain over Probe, original / caption
Probe83.6588.42—
LoRA + Probe85.2588.58+1.60 / +0.16
Widened MLP86.1388.04+2.48 / $-$0.38
ReLope91.7889.03+8.13 / +0.61

The much larger multimodal gain supports ReLope as a response to this difficulty without claiming that its components apply only to images.

As a scope check, we apply the same routers to three text-only benchmarks.

Text benchmarkProbeReLopeReLope −- Probe
MMLU79.22 ±\pm 0.4178.83 ±\pm 0.71$-$0.39
GSM8K76.09 ±\pm 0.3976.24 ±\pm 0.34+0.15
TriviaQA80.61 ±\pm 0.2081.24 ±\pm 0.18+0.63

The small text-only gains support our multimodal focus rather than universal improvement.

W3 & Q3. Bottleneck interpretation and transfer

KL does not identify irrelevant visual details. Formally, the expected KL to the prior upper-bounds I(H;Z)I(H;Z) and constrains information transmitted from the adapted state to the decision representation. Joint optimization with correctness loss encourages this representation to retain routing-relevant information. We therefore replace “discards irrelevant visual details” with “regularizes input-specific information while preserving correctness-related information.”

We report held-out HSIC and label CKA before and after the bottleneck under both conditions.

Input conditionRepresentationHSIC with correctnessLabel CKA
Image–textAdapted state before bottleneck3.06×10−33.06\times10^{-3}0.152
Image–textPosterior mean after bottleneck12.45×10−312.45\times10^{-3}0.264
CaptionAdapted state before bottleneck6.42×10−36.42\times10^{-3}0.223
CaptionPosterior mean after bottleneck27.47×10−327.47\times10^{-3}0.372

The posterior mean depends more strongly on correctness in both conditions, supporting preservation and concentration of correctness-related information without implying that KL assigns semantic relevance to visual details.

We do not claim that KL independently improves transfer. Table 8 evaluates the complete ReLope branch and shows higher absolute cross-dataset AUC than Probe on all five targets; a larger drop from a higher in-domain value does not imply lower held-out performance. We additionally vary only the KL coefficient:

β\betaIn-domain ScienceQA→\rightarrow A-OKVQA→\rightarrow MMMU→\rightarrow ChartQA→\rightarrow MathVisionMean transfer
191.7982.1467.8767.9042.9065.20
0.189.9181.6762.5768.1543.3063.92
088.1679.8958.0171.1143.5863.15

KL improves transfer to some targets, especially MMMU at larger values.

Dear Reviewer QZ9m,

We have revised the manuscript and added clarifications and controls addressing your questions. If time permits, we would appreciate your reviewing our response and sharing any remaining concerns.

Thank you again for your thoughtful feedback.

Reviewer_d7VaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies prefill-hidden-state correctness prediction for routing between a small multimodal language model and a larger model. It reports a descriptive image–caption gap, introduces Attention Probe to pool frozen token states, and proposes ReLope: a separate copy of one transformer layer adapted with LoRA and a KL-regularized Gaussian bottleneck. This branch shares the frozen lower layers without changing answer generation. In relation to prior work on adapting features for correctness estimation and information-bottleneck probes for multimodal reliability, I view the main contribution as this constrained decision-branch design and its evaluation for multimodal routing. ReLope achieves the highest mean correctness-prediction AUC across five benchmarks and three backbones. Component controls, hybrid-system curves, calibration, overhead measurements, and cross-dataset transfer provide additional evidence, although the scope of the comparison with existing routing strategies requires clarification.

Critical strengths
  1. A practically useful architecture for adapted correctness prediction. Whereas Kapoor et al. (NeurIPS 2024, Large Language Models Must Be Taught to Know What They Don't Know) adapt model features for uncertainty estimation, ReLope restricts adaptation to a separate single-layer branch that shares lower-layer computation and leaves answer generation unchanged. Unlike the generation-time attention-head signals used by VIB-Probe, its score is derived from prefill states. This makes the constrained architecture a useful contribution to investigate for multimodal routing; its strongest appeal is the potential prediction-quality versus computation trade-off.

  2. Broad predictive evidence and meaningful component controls. The highest mean AUC across all 15 dataset–backbone combinations is supported by comparisons against frozen-state probes and Attention Probe. The widened-MLP, LoRA-only, VIB-only, and KL controls in Tables 5–7 address several straightforward alternative explanations. These results support the effectiveness of combining adaptation and regularization in this architecture, even though the underlying ideas have precedents. Hybrid-system, calibration, overhead, and transfer experiments further help characterize practical behavior and limitations.

Critical weaknesses
  1. The contribution beyond the closest precedents is not sufficiently isolated. Kapoor et al. (NeurIPS 2024) already compare frozen probes with LoRA-based correctness estimators and establish the benefit of adapting features for uncertainty prediction. VIB-Probe already applies a Gaussian bottleneck with BCE and KL regularization to internal VLM signals for hallucination detection. The submission appropriately cites both, so this is not a missing-citation concern. Rather, these precedents narrow the main methodological contribution to the particular prefill, shared-backbone, single-layer decision branch. Tables 5–7 establish that its components help, but do not fully characterize why this constrained design offers a better quality–cost trade-off than alternative adapted correctness estimators. A budget-controlled adaptation comparison, or an analysis of adaptation depth and cost, would make the architectural contribution more convincing. Such comparisons must account for the different information available to answer-conditioned and prefill estimators; I am not requesting an unqualified comparison across incompatible settings.

  2. The comparison with prior routing methods does not yet establish the same strength of conclusion as the correctness-prediction results. RouteLLM predicts relative model preference, BEST-Route jointly chooses a model and a best-of-n sampling budget, and MMR-Bench includes per-model utility/cost prediction. The submission explicitly adapts MMR baselines to small-model correctness prediction, and the preservation of the other methods' original objectives and decision mechanisms needs more detail. A common correctness task is useful for comparing representations, but is not equivalent to comparing the original routing policies. Under binary correctness and equal escalation costs, the expected accuracy benefit of escalation is P(large correct | x) minus P(small correct | x), which need not rank queries like small-model error probability. The hybrid-system curves are relevant evidence, but the paper should establish whether its advantage persists against a baseline retaining a relative-performance objective under matched budgets. Table 11 further illustrates the distinction: gains over Attention Probe at 20%, 40%, and 60% escalation are 0, 0.18, and 0.03 percentage points despite the AUC improvement. These concerns leave the incremental routing contribution insufficiently established for me, motivating a weak rejection at present. I would reconsider this assessment if the baseline protocol and architectural advantage can be substantiated.

Questions for authors
  1. Closest-prior-work comparison. Beyond combining LoRA adaptation and a variational bottleneck, what empirical evidence isolates the benefit of the separate single-layer decision branch relative to adapted correctness estimation such as Kapoor et al. and VIB-based multimodal probing? Can you provide a quality–cost comparison controlling input information and training budget, or an adaptation-depth analysis that explains the single-layer choice?

  2. Baseline fidelity and routing objective. For RouteLLM, BEST-Route, and MMR-MLP/CMR, please specify the training labels, input features, decision rule, and retained or removed mechanisms. In particular, does BEST-Route retain best-of-n selection? If a method is reduced to a correctness scorer, please distinguish that adapted variant from the original policy. Can you compare with at least one baseline retaining a relative-performance or per-model utility objective on the same model pair and data split, reporting hybrid quality at matched budgets and the supervision costs?

  3. Material system-level gains. Can you quantify the paired uncertainty of the hybrid-accuracy gains, especially the small differences from Attention Probe in Table 11? A summary across the five benchmarks at matched budgets would help determine whether the improvement is practically consistent. Please clarify which costs include small-model prefill/generation, the decision branch, and any additional sampling, rather than only large-model API charges.

  4. Protocol clarification. Please provide source splits and train/development/evaluation counts for each dataset, and explain how the readout layer, rank, and KL coefficient were selected without using final evaluation labels, including in the held-out-dataset experiments. This is a reproducibility question, not an allegation of leakage.

3 comments
AuthorsOpenReview ↗

Response to Reviewer d7Va

Thank you for the thoughtful questions. We respond below.

W1 & Q1. Closest work and the single-layer choice

Kapoor et al. score completed text answers, while VIB-Probe detects hallucinations from generated-token attention; both are answer-conditioned. ReLope instead predicts MLLM correctness at prefill from only the image and question, using frozen shared layers, a copied LoRA decision layer that preserves generation, and a regularized representation. For matched prefill variants, VIB-Probe applies its bottleneck to frozen states, while Kapoor adapts all 28 layers with LoRA r=5r=5, matching our LoRA budget; neither sees answer tokens.

DatasetReLopePrefill VIB-ProbePrefill Kapoor
MMMU80.79 ±\pm 0.4076.82 ±\pm 0.8773.41 ±\pm 0.49
A-OKVQA86.24 ±\pm 0.6383.05 ±\pm 0.7586.07 ±\pm 1.20
ScienceQA91.78 ±\pm 0.5284.44 ±\pm 1.0590.19 ±\pm 0.08

ReLope has the highest mean AUC on all three datasets. We also vary adaptation depth while fixing the bottleneck and classifier:

Dataset/cost1 layer2 layers4 layers
Trainable params29.54M32.42M38.19M
LoRA params2.88M5.77M11.53M
MMMU80.79 ±\pm 0.40Δ\Delta+0.41 ±\pm 0.17Δ\Delta+0.14 ±\pm 0.79
ScienceQA91.78 ±\pm 0.52Δ\Delta+0.85 ±\pm 0.06Δ\Delta+0.67 ±\pm 0.86
A-OKVQA86.24 ±\pm 0.63Δ\Delta+0.64 ±\pm 0.02Δ\Delta−0.01 ±\pm 1.51

Two layers gain at most 0.85 points; four layers are inconsistent despite more parameters. We therefore use one layer as a fixed efficient design, not because depth can never help.

W2 & Q2. Baseline fidelity and routing objective

Table 2 retains MMR's OpenCLIP architecture, RouteLLM's routing machinery, Post-Hoc Embed's decoder state, and MoT's five-sample agreement, but adapts their targets to small-model correctness; we now state this explicitly. Our correctness cascade needs no strong-model labels and remains reusable when the cloud model changes. We additionally compare the original relative-performance and utility objectives on the same model pair and budgets:

DatasetEscalationReLopeRouteLLM BERTMMR-CMR utility
A-OKVQA20%85.5982.1081.95
A-OKVQA40%89.3584.5484.60
A-OKVQA60%89.8786.5286.78
MMMU20%53.3350.0749.11
MMMU40%55.6753.5952.74
MMMU60%61.3356.2256.19

ReLope leads at every budget without the one GPT-4.1 training label per example required by these baselines.

W2 & Q3. System gains and uncertainty

Hybrid accuracy at 60% escalation is:

DatasetProbeAttention ProbeReLope
MMMU60.1160.1161.33
A-OKVQA89.5289.8789.87
ScienceQA84.3184.6985.36
ChartQA77.4577.2478.13
MathVision47.3742.4350.99

Using the same per-question predictions, we run 10,000 paired bootstrap resamples. Cells are ReLope−Attention Probe accuracy points with 95% CIs; bold excludes zero. Table 11 separately reports the three-seed A-OKVQA mean, explaining its small difference here.

Dataset20%40%60%
MMMU+1.00 [−0.44,2.44]+0.11 [−1.22,1.44]+1.22 [−0.33,2.78]
A-OKVQA0.00 [−0.64,0.90]+0.17 [−0.26,0.70]0.00 [−0.17,0.59]
ScienceQA+0.43 [0.10,0.81]+0.76 [0.05,1.53]+0.67 [0.16,1.37]
ChartQA−0.47 [−1.25,0.31]−0.16 [−0.53,0.18]+0.89 [0.33,1.47]
MathVision+3.95 [1.91,6.75]+11.18 [6.89,15.80]+8.55 [5.21,11.93]

ReLope's mean gain is positive in 11/15 comparisons; CIs exclude zero for ScienceQA and MathVision at all budgets and ChartQA at 60%.

The small model/router run locally without per-query API charges. Pricing measured H200 latency at $10/GPU-hour, the shared prefill costs 5.02% of one GPT-4.1 call; Probe, Attention Probe, and ReLope branches add 0.016%, 0.014%, and 0.106%, for total local costs of 5.04%, 5.03%, and 5.13%. ReLope adds only 0.32 ms/sample over Attention Probe. Revised Table 11 reports this local cost beside GPT-4.1 API cost.

Q4. Data and selection

For every dataset, we stratify the designated training pool by correctness label into 90% train/10% development, select the checkpoint by development AUC, and evaluate on a separate test pool. A-OKVQA, ScienceQA-image, and ChartQA use official train/validation as training/test pools. MMMU uses official dev/validation. MathVision uses 2,736 test examples excluding testmini for training/development and the disjoint 304-example testmini for testing. Hyperparameters and the three-epoch schedule are fixed in advance.

Thank you for the clarification. I have adjusted my rating accordingly.

Dear Reviewer d7Va,

Thank you for revisiting our response and adjusting your rating. We sincerely appreciate your constructive feedback, which helped us strengthen the manuscript.

Reviewer_HpYdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies routing in hybrid multimodal LLM (MLLM) systems, specifically whether correctness of a small model's answers can be predicted from its frozen hidden states. The authors find that replacing input images with generated captions lowers the small model's accuracy but raises the hidden-state probe's correctness-prediction AUC, indicating that visual input makes correctness harder to read out. They propose Attention Probe, which learns attention pooling over frozen token hidden states, and ReLope, which inserts a LoRA-adapted copy of a layer with a KL-regularized stochastic bottleneck trained for correctness classification while keeping the answer-generating backbone frozen. Across five multimodal benchmarks, three backbones, and eight baselines, ReLope achieves the highest mean correctness-prediction AUC in all 15 dataset-backbone pairs and improves the hybrid accuracy-cost tradeoff at low overhead.

Critical strengths
  • The finding that images make correctness harder to probe is corroborated by three dependence measures (probe AUC, HSIC, label CKA) and replicated on three benchmarks, providing a solid empirical motivation for multimodal routing research.
  • ReLope's combination of a LoRA-adapted layer with a VIB-style KL bottleneck, trained on a separate routing branch without disturbing answer generation, is a clean and principled design that consistently outperforms frozen probes and Attention Probe across 15 settings.
  • The evaluation is unusually thorough: five benchmarks, three backbones, eight baselines, ablations decomposing pooling/adaptation/bottleneck/KL/rank contributions, image-perturbation robustness, cross-dataset transfer, and detailed GPU/CPU overhead measurements.
Critical weaknesses
  • he image-vs-caption comparison is descriptive rather than causal. Replacing images with generated captions changes both task difficulty and information form; the random-image control rules out token-count effects but cannot disentangle whether the AUC gap stems from visual-token redundancy, shifted correct/incorrect sample distributions, or caption text being more readable. The introduction frames this finding more strongly than Appendix C.2 acknowledges.
  • ReLope's practical gain over Attention Probe is small in several settings: correctness AUC improves by only 1–3 points, and at 40% escalation on A-OKVQA the hybrid accuracy advantage is just 0.18 points (89.32 vs 89.14), while trainable parameters grow ~30× (0.96M to 29.54M). Whether this warrants the parameter and training-data cost is not fully discussed.
  • Cross-dataset transfer partially undermines deployability: in Table 8, ReLope wins on only 1 of 5 held-out datasets, and on MMMU Attention Probe (71.22) actually beats ReLope (70.66), suggesting ReLope's advantage relies heavily on in-distribution correctness labels—a premise often unavailable in real routing scenarios.
Questions for authors
  • Add a causal decomposition of the image-vs-caption gap, for example by matching subsets with similar accuracy distributions across conditions or probing frozen visual-encoder features, and temper the introduction's framing accordingly.
  • Report cost savings at matched accuracy (rather than fixed escalation rate) and quantify whether ReLope's gains at deployment scale justify the ~30× parameter increase over Attention Probe.
  • Add lightweight uncertainty baselines (output entropy, logprob margin) and discuss when hidden-state routing is worth its implementation complexity.
3 comments
AuthorsOpenReview ↗

Response to Reviewer HpYd

Thank you for the thoughtful assessment of our motivation, methods, and deployment evidence. We address each concern below.

W1 & Q1. Image–caption comparison

Thank you for this insightful comment. Changing the input modality can alter several factors, including visual-token redundancy and the greater readability of captions. These factors may explain why correctness signals are harder to probe from image-conditioned hidden states, so they do not conflict with our motivation. Table 1 supports this interpretation beyond probe training because both information-correlation measures, HSIC and label CKA, are lower under image input.

To control for shifted correctness-label prevalence, we repeatedly sampled subsets with equal numbers of correct and incorrect examples (243 / 243) from A-OKVQA in both conditions. We use the same four-layer MLP Probe architecture and training setup as in the main experiments. The table reports the mean and standard deviation over 20 balanced resamples.

InputProbe AUCHSIC (×10−3\times10^{-3})Label CKA
Original input79.22 ±\pm 1.964.90 ±\pm 0.440.178 ±\pm 0.014
Caption only83.10 ±\pm 2.2013.53 ±\pm 1.510.244 ±\pm 0.027

We add the complete protocol and results to the revised paper. We also temper the Introduction by describing the observation as reduced accessibility of correctness information rather than intrinsic information loss or a universal causal effect.

W2 & Q2. Practical gains and cost

Thank you for highlighting the need to connect routing AUC with deployment value. Attention Probe is our simple remedy for distributed routing information. ReLope improves further. With Qwen2.5-VL-7B-Instruct, it exceeds Attention Probe by 5.67 AUC points on ScienceQA and 4.40 on MathVision. It also wins on all five datasets with Gemma3-12B and Phi-4.

We directly compare the escalation needed at matched hybrid accuracy. Recovering f%f\% of the gap means reaching As+f(Al−As)A_s+f(A_l-A_s), where AsA_s and AlA_l are small-only and large-only accuracy. Each cell below is the minimum escalation rate for Attention Probe / ReLope, obtained by inverting the submitted curves.

Dataset75% gap recovered90% gap recovered
MMMU.56 / .51.65 / .69
A-OKVQA.29 / .29.41 / .40
ScienceQA.26 / .21.34 / .29
ChartQA.34 / .33.63 / .44
MathVision.65 / .36.76 / .55

At the 75%/90% targets, ReLope reduces Attention Probe's strong-model calls by 18.1%/13.4% on ScienceQA, 1.8%/29.1% on ChartQA, and 44.0%/28.7% on MathVision.

Although ReLope has substantially more trainable parameters than Attention Probe, its 29.54M parameters are about 0.4% of a 7B backbone. The answer model remains frozen. On an H200, ReLope adds 1.79% latency and 0.11 GiB of memory over Probe. This local overhead is small relative to the MLLM and can offset recurring API calls at favorable operating points.

W3. Cross-dataset transfer

Thank you for raising the transfer question. In the leave-one-dataset-out evaluation in Table 8, ReLope exceeds Attention Probe on every target and is the best evaluated method on four of five targets.

Held-out targetAttention ProbeReLope
MMMU73.2276.66
A-OKVQA81.8783.68
ScienceQA81.1481.58
ChartQA73.9875.57
MathVision66.4266.83

These results show that ReLope improves not only in-distribution performance but also the absolute held-out AUC of the stronger frozen-state baseline.

Q3. Lightweight uncertainty baselines

Thank you for suggesting these inexpensive comparisons. We evaluated answer-token maximum probability, entropy, top-two margin, and P(True)P(\mathrm{True}) on the same test questions.

DatasetMax probabilityEntropyMarginP(True)P(\mathrm{True})ReLope
MMMU69.5773.1160.6467.6680.79
MathVision57.8064.9456.8862.6393.56
ScienceQA83.4482.8983.4369.3691.78
A-OKVQA83.8383.5783.5069.2886.24
ChartQA81.1181.3080.0373.2385.94

ReLope exceeds the strongest lightweight signal by 2.41–28.62 points. This shows that a learned hidden-state probe can capture correctness information beyond simple output confidence. The revised appendix reports every signal and its computation.

Dear Reviewer HpYd,

We have revised the manuscript and added experiments addressing your main concerns. If time permits, we would appreciate your reviewing our response and letting us know whether any concerns remain.

Thank you again for your constructive feedback.

The authors have added substantial new experiments to address my concerns. I maintain my positive score of 3 and ask the authors to incorporate these new experiments into the next revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.