acceptodds
Under review as a conference paper at ICLR 2027

Entropy Shortcut in LLM Correctness Probes

Abstract

Large language models (LLMs) achieve strong performance across textual, visual, and other multimodal tasks, yet they frequently hallucinate, producing outputs that are incorrect and unsupported. Preventing such failures is difficult, hence recent works have increasingly focused on detecting them. Two promising approaches that do not modify the model parameters are: (a) sampling the model at high temperature and measuring the entropy of the sampled responses, and (b) training a linear probe on hidden states to estimate the probability of a correct response. Entropy-based detection rests on the hypothesis that sampling entropy and correctness are inversely associated. In this work, we show that a sizable fraction of mispredictions exhibit low sampling entropy. More importantly, standard linear correctness probes fail to distinguish these low-entropy mispredictions from high-entropy correct predictions. We trace this failure to an entropy shortcut- the probes exploit the entropy–correctness association rather than learning a correctness signal that remains stable across entropy regimes. To address this, we propose linear detectors that predict correctness with reduced entropy shortcut. Experiments with four LLMs and five multimodal LLMs across eight datasets, including safety-critical domains, establish the entropy shortcut as a systematic failure mode for LLM probes and demonstrate substantially more robust correctness detection across both entropy-aligned and entropy-misaligned cases.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Uncertainty Estimation, Abstention, Large Language Model, Multimodal Language Models, Confidence Estimation, LLM Probe, Hallucination Detection, Semantic Entropy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NzWSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper shows that correctness probes can exploit the association between low response entropy and correctness, causing poor discrimination between confidently wrong and uncertain-but-correct answers. It diagnoses this behavior through entropy-conditioned evaluation, matched training, and probe geometry, then proposes group-robust and routed probes to improve performance on the conflicting groups.

Critical strengths

Clear scientific claim: probes trained directly on correctness labels can inherit the same entropy dependence.

Broad empirical coverage: Text, visual QA, and medical-image tasks support the relevance of the phenomenon

Simple remedies.

Critical weaknesses
  1. The novelty of the finding needs to be clearer. The existence of low-entropy incorrect answers and high-entropy correct answers is already recognized in prior work. The potentially new contribution is showing that correctness probes learn to rely on this association, rather than simply documenting these cases again. The paper should distinguish these two claims more clearly.

  2. The remedies have limited novelty and incomplete theoretical justification. Group reweighting, group DRO, and routing are established techniques. For the routed method, removing band-specific means and rescaling scores does not generally guarantee that predictions become independent of entropy given correctness. The paper should clarify the assumptions behind this interpretation and distinguish empirical improvements in subgroup metrics from a rigorous guarantee of shortcut removal.

  3. Suppressing entropy information is not clearly justified as a general detection objective. Entropy can provide useful information about correctness, even though it is imperfect. The reported reduction in pooled AUROC suggests a trade-off between subgroup robustness and overall discrimination. Calibration, selective prediction, or distribution-shift experiments are needed to show when reducing entropy dependence improves practical detection.

Questions for authors

See weaknesses

5 comments
AuthorsOpenReview ↗

Addressing the concerns

We thank the reviewer NzWS for reviewing the draft and providing valuable comments and suggestions.

  1. Clear scientific claim: probes trained directly on correctness can inherit entropy dependence, Broad empirical coverage: Text, visual QA, and medical-image tasks, Simple remedies: Thank you for appreciating our problem setup and the findings.

  2. The novelty needs to be clearer. The existence of low-entropy incorrect answers and high-entropy correct answers is already recognized. New contribution is showing that correctness probes learn to rely on this association, rather than simply documenting these cases again: We emphasize the existence of low entropy mispredictions and benchmarking, while showing that the LLM correctness probes use entropy shortcuts. The additional contributions are in ways that allow mitigation of this problem.

Title (Page 1): NOVEL Contributions - ENTROPY SHORTCUT IN LLM CORRECTNESS PROBES

Abstract (Page 1): In contrast to prior works, we show that a sizable fraction of mispredictions exhibit low sampling entropy. More importantly, standard linear correctness probes fail to distinguish these low-entropy mispredictions from high-entropy correct predictions.

Prior Works (Page 9): .. Recent works point out the existence of low-entropy mispredictions and that consistency is not equivalent to correctness. They show that multiple detector families deteriorate on such errors Simhi et al. (2025); Gekhman et al. (2025); Tan et al. (2025). These studies establish this as a failure mode, but neither formalize it nor examine whether correctness probes themselves exploit entropy as a shortcut.

Regarding the novelty and importance of benchmarking the entropy-correctness subgroups (Section 4): Benchmarking (Section 4) in our paper adds additional findings. The two papers we find closest to this are:

a. Trust Me, I'm Wrong: High-Certainty Hallucinations in LLMs. EMNLP 2025: the prevalence of low-entropy mispredictions is acknowledged. However,  it is limited to text-only tasks (Trivia-QA, NQ) and three LLMs. Further, no mitigation strategies are suggested. 

b. Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs. EMNLP 2025:  Only benchmarks low-entropy and high-entropy mispredictions and not all the sub-groups. 

We did not find any prior work on the four entropy-correctness association subgroups which are benchmarked on both text-only and multimodal tasks. Our work benchmarks both text-only (Trivia-QA, NQ, SciQ) and multimodal datasets (OKVQA, VQAv2, AdVQA, RSNA, Camelyon) totaling ~165k samples across four LLMs and five multimodal LLMs.

In terms of novelty, the paper's central contribution is the finding that LLM probes use entropy hacking and take a shortcut to predict correctness, which is undesirable. The paper uses (a) Joint learning using DRO and (b) routed learning as shortcut mitigation approaches. The paper clearly mentions DRO as a prior work developed for shortcut mitigation, and uses it as is for our problem. Routing-based networks are common; however, they have not been used for entropy-shortcut mitigation, to the best of our knowledge.

AuthorsOpenReview ↗

Continuation

  1. The remedies have limited novelty and incomplete theoretical justification. Group reweighting, group DRO, and routing are established techniques. For the routed method, removing band-specific means and rescaling scores does not guarantee predictions become independent of entropy given correctness. The paper should clarify the assumptions and distinguish empirical improvements: Our draft submission explicitly acknowledges this: Section 6.1 attributes the worst-group optimization procedure to Sagawa et al. (2020), and Section 8 connects the approach to established shortcut-learning methods. We employ these techniques as targeted interventions on the four correctness-entropy groups, rather than claiming a new general-purpose reweighting or DRO algorithm. Their improvements on entropy-misaligned comparisons, together with the controlled entropy-matching and probe-direction analyses, provide converging evidence supporting our diagnosis of learned entropy shortcut.

We also acknowledge that expert routing is an established technique for various applications, for example, mixture-of-experts approaches for spurious correlation [4] and robustness [5]. Our contribution is the entropy-specific post-hoc construction for correctness detection for LLM probes: routing examples to high- or low-sampling-entropy correctness probes and normalizing their score streams before pooling. This directly targets the cross-band ranking asymmetry identified by our analysis. We did not find any paper that uses a routing method for the same. We will clarify that the methodological contribution lies in this problem-specific design and its evaluation, while the central contribution is the systematic diagnosis of reliance on entropy in correctness-supervised probes.

The theoretical result relates to the cancellation of an additive band-dependent term under the assumptions of Proposition A.3, rather than a general guarantee of conditional independence. Proposition A.3 (Appendix A.6.1) assumes that the class-conditional correctness-evidence likelihood ratio is linear and shared across entropy bands. Under this assumption, band-specific correctness prevalence appears as an additive logit offset, which cancels under the centering operation in Eq. 45.

  1. Suppressing entropy information is not clearly justified. Entropy can provide useful information about correctness, even though it is imperfect. The reported reduction in pooled AUROC suggests a trade-off. Calibration, selective prediction, or distribution-shift experiments to show reducing entropy dependence improves practical detection: We agree that entropy can provide useful information about correctness in certain scenarios, and suppressing it indiscriminately is not a desirable general objective. However, when an LLM probe is trained to predict correctness, it does not serve the purpose when the usual entropy-correctness association reverses. We highlight that this creates a failure mode for the probe due to entropy-shortcut. Our paper unveils this phenomenon, which gets concealed in overall AUC. We argue the seriousness of the shortcut and attempt to mitigate it. Ideally a probe should be devoid of entropy-shortcut and perform uniformly across all subgroup.

Moreover, to understand in what situation the shortcut leads to degradation in the overall CvI AUC, we conducted an additional test-time entropy-composition-shift experiment. We rearrange test samples while preserving overall correctness prevalence and within-group score distributions. Specifically, u=P(H∣C)u=P(H\mid C) and v=P(L∣I)v=P(L\mid I) control the fractions of high-entropy correct and low-entropy incorrect predictions, respectively.

Table 7: Correctness-detection performance under natural and shifted distributions.

DatasetMethodNatural CvIES-xAUCShifted CvI (80% conflict; u=v=0.8u=v=0.8)
Text QAStandard probe0.8650.6220.697
Routed Learning0.7660.7370.802
Image QAStandard probe0.8190.5520.639
Routed Learning0.7490.7150.742

The standard probe's pooled AUROC declines substantially as the entropy–correctness association changes, while Routed Learning remains comparatively stable. In many practical applications, entropy-correctness correlation may not be controllable at deployment scenarios, and hence this may make the probes vulnerable.

[4] Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networks. ICML 2023.

[5] Not Eliminate but Aggregate: Post-Hoc Control over Mixture-of-Experts to Address Shortcut Shifts in Natural Language Understanding. TACL 2024.

AuthorsOpenReview ↗

Gentle reminder

Dear reviewer NzWS,

We have attempted to address the valuable suggestions you pointed out. This is a gentle reminder in case you have any further questions, and also to reconsider your scores.

Thanks,

The authors

Reviewer_mZT4OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether correctness probes rely on the association between low sampling entropy and correct model responses. The experiments are conducted with 4 text-only LLMs, and 5 multimodal LLMs, spanning text QA, visual QA, and medical classification. It splits samples into four correctness-entropy groups and evaluates all four correct-versus-incorrect group comparisons. They propose ES-xAUC metric, which captures the weakest AUROC among the 4 subsets, while ES-δAUC measures their disparity.

On the proposed data, standard correctness probes achieve substantially weaker discrimination between high-entropy correctness and low-entropy incorrect answers than their pooled AUROC suggests. Entropy-matched training and probe-direction analysis provide evidence that the training distribution's entropy vs. correctness association influences probe behavior. They finally propose group-weighted joint learning and entropy-routed experts with bandwise score normalization; these improve the reported subgroup metrics, although routed learning substantially reduces pooled AUROC in several settings.

Critical strengths
  1. The evaluation reveals a substantial limitation of pooled AUROC. In particular, the standard correctness probe obtains a pooled AUROC of 0.905 on TriviaQA but only 0.630 for high-entropy correct versus low-entropy incorrect examples. The paper also shows that this comparison is consistently problematic across all 21 general-QA model-dataset pairs (the entropy-shortcut). This is a useful diagnostic contribution for evaluating correctness detectors.

  2. The paper goes beyond documenting a simple correlation, but proposes an effective way to improve the weakest atomic AUROC using an entropy matching training strategy. They show that probe behaviour depends on the training distribution's entropy-correctness association.

Critical weaknesses
  1. There are some inconsistencies between the reported experimental protocol and the results.
  • The paper says that the data are divided into 70% training and 30% testing, and that the entropy matching is performed only on the training subset. However, these numbers appear inconsistent with some data in the tables: table 6 reports 631 high-entropy correct examples for Gemma on Natural Questions across the entire dataset, and table 11 reports 629 (for the same setup) samples used as the training dataset (> 70% of 631); similarly for Gemma on SciQ. These inconsistencies might reveal data leakage.
  • In the medical results, the paper defines high and low-entropy groups by applying a threshold to the entropy score; if the same score is then used to predict correctness, every low-entropy example must receive a higher correctness score than every high-entropy example. Therefore, in the comparison between high-entropy correct answers vs. low-entropy incorrect answers, the entropy baseline must rank every pair incorrectly, giving an AUROC of exactly zero. Table 18 reports instead values such as 0.027 and 0.986.
  1. Routed learning combines several changes together, and the proposed experiments do not show which component produces the improvement. This is particularly important because table 10 shows that the ordinary probe performs better than the specialized probes in 37 of 42 within-group comparisons. Also the theoretical explanation requires more care; the paper argues that normalization removes the score offsets associated with entropy and therefore reduces the shortcut. However, each group's average score depends partially on how many correct answers it contains. A group with more correct answers can reasonably have a higher average correctness score, and subtracting that average changes both unwanted offsets and differences arising from the composition of the group. So placing both groups on a similar score scale does not establish that predictions have become independent of entropy. The reported improvements might be useful, but the theory does not fully explain them. There is also a substantial performance and computation tradeoff. The router uses hidden states from ten sampled responses, whereas the standard correctness probe uses features from the greedy response: despite this additional information and computation, routed learning reduces pooled AUROC on TriviaQA (from 0.905 to 0.764); this reduction is acknowledged by the paper, but they don't demonstrate a concrete setting in which the subgroup improvements compensate for it.

Finally, the additional worst group improvement over the simpler joint-learning method averages only 1.6% on text QA. Without confidence intervals or repeated-split results, it is difficult to judge how reliable that incremental advantage is.

Questions for authors
  1. Could you clarify what the numbers in table 11 represent? Please clarify the order of splitting and matching, and confirm that the natural and matched probes were evaluated on the same held-out examples.

  2. Why does the semantic-entropy baseline give cross-band AUROCs different from zero and one on RSNA? These comparisons should be completely determined by the threshold.

  3. How much does training separate experts contribute beyond normalizing the scores? Table 10 shows that the generalist usually performs better within each band. Did you try to keep that generalist and apply the same routing and normalization? I would also appreciate clarification of the theoretical argument: equation 45 still uses band-specific means and variances, so what ensures that normalization removes the conditional dependence on entropy?

  4. In what setting would routed learning's subgroup gains outweigh its lower overall performance and additional sampling cost?

6 comments
AuthorsOpenReview ↗

Clarifying the questions and suggestions

We thank the reviewer mZT4 for reviewing the draft and providing valuable comments and suggestions.

  1. reveals a substantial limitation of pooled AUROC, standard correctness probe obtains a pooled AUROC of 0.905 on TriviaQA but only 0.630 for high-entropy correct versus low-entropy incorrect examples, consistent across all 21 general-QA model-dataset pairs, useful diagnostic contribution, proposes an effective way to improve the weakest atomic AUROC using an entropy matching: Thank you for appreciating our paper and the seriousness of the problem of entropy shortcut occurring in LLM probes consistently across 21 model-dataset pairs studied here.

  2. table 6 reports 631 high-entropy correct examples for Gemma on Natural Questions, and table 11 reports 629 (for the same setup) as training dataset (> 70% of 631); similarly for Gemma on SciQ, might reveal data leakage: We thank the reviewer for identifying this inconsistency. We acknowledge an error in the caption of Table 11.

Previous Caption: Table 11: Detailed dataset statistics and atomic correctness–entropy subsets across text and image QA benchmarks, under the entropy-MATCHED training split.

Corrected Caption: Table 11: Detailed dataset statistics and atomic correctness–entropy subsets across text and image QA benchmarks, under the entropy-MATCHED entire dataset.

Table 11 represents the statistics of the full entropy-MATCHED dataset, not the training split only. Further, the sample difference of 631 vs 629 between natural and matched datasets comes from the entropy matching process on the training split, which is a subsampling-based algorithm leading to a reduction of 2 samples (training data) after subsampling in this case. The same applies to Gemma on SciQ. We have re-verified the data splits again to confirm that there is no data leakage.

Corrections: We will correct Table 11's caption.

  1. if the same entropy score is used to predict correctness, in the comparison between high-entropy correct answers vs. low-entropy incorrect answers the entropy baseline must give an AUROC of exactly zero. Table 18 reports instead values such as 0.027 and 0.986:

We thank the reviewer for raising this point. We acknowledge that this methodological distinction was not sufficiently explicit in the manuscript. In the medical results, the LLMs are fine-tuned to act as three (RSNA) or two (Camelyon17) class classifiers. Unlike open QA settings like TriviaQA, the possible predictions are only two or three, leading to low entropy, in general. Hence, we set the threshold criterion that a sample is low-entropy if at least 8 out of 10 high-temperature sampling responses are identical. This is based on sample consistency-based thresholding, often used as an alternative to entropy [1]. This leads to the discrepancy with the entropy-based band creations and hence, AUCs 0.027 (>0) and 0.986 (<1.0) are obtained.

We recomputed the cross-band entropy AUROCs under both the definitions, and using entropy threshold recovers the AUCs 0.0 and 1.0 as shown in below Table 3.

Table 3: CHC_H vs ILI_L AUC under entropy threshold and model agreement based threshold.

DatasetBand definitionCHC_H vs ILI_LCLC_L vs IHI_H
Camelyon17Entropy threshold0.0001.000
Camelyon17Modal agreement ≥0.80.0001.000
RSNAEntropy threshold0.0001.000
RSNAModal agreement ≥0.8 (reported)0.0250.985

Corrections: We will clarify this consistency-based threshold definition for medical results in the paper.

[1] Lyu et. al. Calibrating Large Language Models with Sample Consistency. AAAI 2025.

AuthorsOpenReview ↗

Continuation

  1. Routed learning combines several changes, and the proposed experiments do not show which component produces the improvement. Table 10 shows that the ordinary probe performs better than the specialized probes in 37 of 42 within-group comparisons: Thank you for raising this.

Firstly, Table 10 reports the AUC(CH, IH) and AUC(CL, IL). The within-group (within high and low entropy bands) correctness AUC performances of the standard probe are indeed marginally better than the within-group performances of the specialized probe. However, the shortcut leads to collapse in the cross-group performance (high-entropy correct vs low-entropy incorrect) as shown in Tables 4 and 14, which is not shown in Table 10. The paper focuses on the entropy shortcut and hence highlights the improvements in the inverse association of AUC(CH, IL), along with ES-xAUC, and ES-δ\deltaAUC which are robust to entropy shortcut. Hence, the within band performances of the standard probe and the specialized probe in Table 10 serves as an ablation analysis and do not violate the shortcut mitigation achieved by routed learning.

Secondly, to understand the contributions of several components in routed learning, we present the table below. We observe that entropy band-wise normalization significantly helps. The routing with separate probes for entropy bands and then pooling the scores helps enhance the AUC but normalizing the standard probe scores in each band and then pooling them achieves similar AUC performance. We will add this in the revised draft.

Table 4: Ablation of routing, separate experts, and score normalization. Results are averaged across models within each dataset, followed by dataset-level averaging.

Text QA (12 model–dataset pairs)

MethodRoutingExpertsRescaleCHC_H vs ILI_L ↑ES-xAUC ↑ES-δAUC ↓Pooled CvI ↑
Standard probe———0.6220.6220.3120.865
Standard probe + bandwise normalization + pooling✓—✓0.8170.7410.0790.771
Routing and pooling without normalization✓✓—0.7150.7150.1550.836
Routing and pooling with normalization (paper)✓✓✓0.8160.7370.0810.766

Image QA (9 model–dataset pairs)

MethodRoutingExpertsRescaleCHC_H vs ILI_L ↑ES-xAUC ↑ES-δAUC ↓Pooled CvI ↑
Standard probe———0.5520.5520.3650.819
Standard probe + bandwise normalization + pooling✓—✓0.7330.7210.0570.761
Routing and pooling without normalization✓✓—0.6750.6750.1490.783
Routing and pooling with normalization (paper)✓✓✓0.7430.7150.0550.749
  1. Each group's average score depends partially on how many correct answers it contains. A group with more correct answers have a higher average correctness score, and subtracting that changes both unwanted offsets and differences arising from the composition. The improvements might be useful, but the theory does not fully explain them: We thank the reviewer for this observation. The composition of correct and incorrect samples is indeed different for high and low entropy bands. The z-normalization performed on the specialized probe scores does remove the entropy bias and also affects the correctness scores. However, we observe that the mean score from correct and incorrect groups still differ significantly after normalization. We see that the performance improvements with only intercept removed stays the same as ones achieved by performing z-normalization on the entire score as shown in below Table. Hence, we conclude that performing z-normalization on the entire score does not adversely affect the mean score difference arising from the correctness composition of entropy bands.

Table 5: AUC performance metrics with intercept removed from probe scores vs z-normalization (reported in paper) in routed learning.

MethodText CHC_H vs ILI_LText ES-xAUCImage CHC_H vs ILI_LImage ES-xAUC
Routed learning, raw scores0.7150.7150.6750.675
Routed learning, intercept removed0.8190.7290.7460.713
Routed learning, full normalization0.8160.7370.7430.715
AuthorsOpenReview ↗

Continuation

  1. The router uses hidden states from ten sampled responses: despite this additional information and computation, routed learning reduces pooled AUROC on TriviaQA (from 0.905 to 0.764) acknowledged by the paper, but don't demonstrate a setting in which the subgroup improvements compensate for it: The paper finds out the entropy shortcut used by the LLM probes and proposes to mitigate it through maximization of shortcut independent metrics. Hence, the overall AUC of 0.905 is an overestimation of the probe performance obtained by entropy hacking. Once the short-cut effect is corrected, we see that the AUC across sub-groups are far from desirable.

Coming to the computational requirements for the router, it does take additional 10 sampling passes to predict entropy. Below, we show the analysis of the router performance and overall performance metrics using 10 or lesser sampling passes.

Table 6: Sampling cost curves for text and image QA.

Text QA

GenerationsRouter AUROC ↑CHC_H vs ILI_L AUROC ↑ES-xAUC ↑Pooled CvI ↑
20.7800.7560.7350.775
30.8200.7760.7380.770
50.8550.7990.7410.767
100.8840.8160.7370.766
Standard probe (0)—0.6220.6220.865
Joint DRO (0)—0.7170.7170.836

Image QA

GenerationsRouter AUROC ↑CHC_H vs ILI_L AUROC ↑ES-xAUC ↑Pooled CvI ↑
20.7460.6910.6910.755
30.7830.7120.7060.753
50.8080.7310.7120.752
100.8280.7420.7150.749
Standard probe (0)—0.5520.5520.819
Joint DRO (0)—0.6800.6800.779

The purpose of shortcut mitigation tools such as DRO and routed learning is to make the correctness-detection performance as invariant as possible across entropy-correctness association subgroups. The pooled CvI AUC can simply be maximized by entropy-hacking, and that is exactly what the standard probe does. Such CvI becomes an overestimation. A CvI AUC should be acceptable when the performance holds steady across entropy subgroups. Here are some precedents [2,3] in the literature that emphasize that shortcut mitigation as a diagnostic tool need not improve the overall performance.

Moreover, to understand in what situation the shortcut leads to degradation in the overall CvI AUC, we conducted an additional test-time entropy-composition-shift experiment. We rearrange test samples while preserving overall correctness prevalence and within-group score distributions. Specifically, u=P(H∣C)u=P(H\mid C) and v=P(L∣I)v=P(L\mid I) control the fractions of high-entropy correct and low-entropy incorrect predictions, respectively.

Table 7: Correctness-detection performance under natural and shifted distributions.

DatasetMethodNatural CvIES-xAUCShifted CvI (80% conflict; u=v=0.8u=v=0.8)
Text QAStandard probe0.8650.6220.697
Routed Learning0.7660.7370.802
Image QAStandard probe0.8190.5520.639
Routed Learning0.7490.7150.742

The standard probe's pooled AUROC declines substantially as the entropy–correctness association changes, while Routed Learning remains comparatively stable. In many practical applications, entropy-correctness correlation may not be controllable at deployment scenarios, and hence this may make the probes vulnerable.

  1. The additional worst group improvement over the simpler joint-learning method averages only 1.6% on text QA. report confidence intervals or repeated-split results: The joint learning using DRO and routed learning are two independent methods that attempt to mitigate the entropy shortcut in the probes. Routed learning performs better than DRO in most cases. We report the statistical test metrics below, which confirm that routed learning performs statistically significantly better (p<0.05).

Table 8: Statistical tests between routed learning and DRO.

ComparisonDatasetES-xAUC improvementp-value
Routed – Joint Group DROText QA+0.01690.010
Routed – Joint Group DROImage QA+0.02660.005

[2] Just Train Twice: Improving Group Robustness without Training Group Information. ICML 2021.

[3] Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. ICLR 2020.

Reviewer_mLY8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores an interesting phenomenon for mechanistic interpretability probes. They look into the entropy shortcut whereby linear correctness probes fail to distinguish low-entropy mispredictions from high-entropy correct predictions. The authors proceed with a systematic empirical study of how prevalent a failure mode this entropy shortcut is.

Critical strengths
  1. The entropy shortcut phenomenon is important for the mechanistic interpretability community to be aware of
  2. The empirical study proposes routed learning that remains robust across different entropy groups
Critical weaknesses
  1. While the problem of entropy shortcuts is very interesting, this is not the first paper to expose it, so the novelty seems mostly limited to the proposed routed learning, which is fairly straightforward
  2. A formal definition of entropy shortcut seems to be missing in the main body. The closest I found was the following statement: "We formally establish that these metrics help quantify entropy-shortcut in Appendix A.2". While I intuitively see the gist of what an entropy shortcut is, it would be good to have a formal mathematical definition
Questions for authors

I found both the DRO and routed learning ideas interesting. Can they be combined? If so, what is the probability distribution and uncertainty set that the combined idea should be robust to?

5 comments
AuthorsOpenReview ↗

Addresing the suggestions and questions

We thank the reviewer mLY8 for reviewing the draft and providing valuable comments and suggestions.

  1. Interesting phenomenon for mechanistic interpretability probes, a systematic empirical study, entropy shortcut phenomenon is important, proposes routed learning that remains robust across different entropy groups: Thank you for appreciating the importance of the problem statement, systematic experimental setup and the shortcut mitigation results.

  2. the problem of entropy shortcuts is very interesting, this is not the first paper to expose it, so the novelty seems mostly limited to the proposed routed learning: The paper is centered on the finding of entropy shortcut in LLM correctness probes- a widely used tool for uncertainty estimation and inference-time correctness prediction for LLM predictions. To the best of our knowledge, the finding of entropy shortcuts in LLM probes remains unknown and unexplored in prior literature. We have conducted an extensive survey of prior works and the below table shows the closest four papers to our work. The distinction between them and the proposed work are mentioned too. We were unable to find any other relevant works.

Table 1: Comparison of prior literature with the proposed work.

Prior worksCited in the submissionSummaryContrast with our work
Inside-Out: Hidden Factual Knowledge in LLMs. COLM 2025.yesLLMs can make prediction errors which remain consistently the same during sampling. LLM probes can detect them better than sampling entropy.Does not explore anything about entropy-shortcut in LLM probes.
Trust Me, I'm Wrong: High-Certainty Hallucinations in LLMs. EMNLP 2025.yesLLMs make confident mispredictions.Our work highlights that existing probes for correctness have limitations in identifying confident mis-predictions. It also frames a strategy to partially mitigate this.
Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs. EMNLP 2025.yesLLMs make consistent mispredictions and they are difficult to detect using error detection methods including LLM probes.Does not provide directions for detection of consistent errors.
LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations. ICLR 2025.yesSizable LLM mispredictions can be recovered from its hidden states using LLM probes.Does not highlight the link between mis-predictions and entropy
  1. A formal definition of entropy shortcut seems to be missing in the main body: We do have a formal notion of entropy-shortcut in Section 2 (page 3, Equation (3)) in the paper, which introduces our formal criterion for a correctness score to be free of the entropy shortcut. We quote from the paper:

An entropy shortcut arises when the probe exploits the association between BB and ZZ and fails when that association changes Geirhos et al. (2020).

Following equalized odds Hardt et al. (2016), a correctness probe without an entropy shortcut should satisfy

However, we will make the definition more explicit in the revised draft with additional clarifications as below:

An entropy shortcut arises when a correctness probe exploits the usual association between low entropy and correctness, achieving stronger discrimination on entropy-aligned pairs (CL,IH)(C_L, I_H). For the cross-band gap

Δcross:=∣AUC⁡(CL,IH)−AUC⁡(CH,IL)∣\Delta_{\mathrm{cross}}:=\left|\operatorname{AUC}(C_L,I_H)-\operatorname{AUC}(C_H,I_L)\right|

Here, Δcross≫0\Delta_{\mathrm{cross}} \gg 0 signifies a strong entropy-shortcut.

In an ideal setting, we require Z^⊥B∣Z\widehat{Z} \perp B \mid Z, indicating that, within each true correctness class, the predicted correctness-score distribution is invariant across entropy bands. This also implies Δcross=0\Delta_{\mathrm{cross}}=0.

AuthorsOpenReview ↗

Continuation

  1. the DRO and routed learning ideas interesting. Can they be combined? If so, what is the probability distribution and uncertainty set that the combined idea should be robust to?:

Thank you for the suggestion. In the paper, they have been studied independently for entropy-shortcut mitigation. However, combining them is an interesting ablation. Below are the three additional approaches we have evaluated during this rebuttal period

DRO: Entropy subgroup weighting during probe training (reported in the paper).

Routed learning: Separate linear probes for high and low entropy bands followed by bandwise normalization.

Routing+DRO: Separate linear probes and bandwise normalization. Similar to Routed learning, but DRO loss used to train the linear probes instead of BCE loss.

DRO+Routing: DRO linear probe is used, but the scores are separately normalized bandwise and then pooled. Samples are classified in entropy bands using the same router used in Routed learning in paper. The results are listed below in Table 2.

Table 2: Combining DRO and routed learning for entropy-shortcut mitigation.

Text QA (12 model–dataset pairs)

MethodCHC_H v IHI_HCLC_L v ILI_LCHC_H v ILI_LCLC_L v IHI_Hpooled CvIES-xAUC
Standard probe0.8060.8320.6220.9340.8650.622
DRO0.7960.8130.7170.8730.8360.717
Routed learning0.7820.7810.8160.7380.7660.737
Routing+DRO0.7750.7760.8170.7240.7570.724
DRO+Routing0.7880.7860.8210.7460.7710.744

Image QA (9 model–dataset pairs)

MethodCHC_H v IHI_HCLC_L v ILI_LCHC_H v ILI_LCLC_L v IHI_Hpooled CvIES-xAUC
Standard probe0.7620.7980.5520.9160.8190.552
DRO0.7370.7740.6800.8180.7790.680
Routed learning0.7250.7590.7420.7430.7490.715
Routing+DRO0.7180.7450.7500.7130.7310.697
DRO+Routing0.7280.7620.7530.7370.7480.715

The results for both text QA and image QA (averaged over models) are shown in the table above. Routing+DRO performs marginally better (0.817) than Routing itself (0.816) on CHvIL, and degrades on ES-xAUC (0.724 from 0.737). DRO+Routing improves over DRO itself by a large margin (CHvIL: 0.821 from 0.717, ES-xAUC: 0.744 from 0.717). Overall, DRO+Routing performs best on Text QA and matches Routed Learning on Image QA in ES-xAUC. Also, the robustness of the uncertainty subsets is preserved.

AuthorsOpenReview ↗

Gentle reminder

Dear reviewer mLY8,

We have attempted to address the important points and questions you raised. This is a gentle reminder in case you have any further questions and also to reconsider your scores.

Thanks,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.