acceptodds
Under review as a conference paper at ICLR 2027

Beyond Confidence? Auditing Routing Entropy in Attention-Residual Transformers

Abstract

Internal routing traces may correlate with errors without improving uncertainty estimates beyond what a model's output already reveals. We test this increment for routing entropy in Attention-Residual (AR) variants of Swin-Tiny and DeiT-Small trained on CIFAR-10 and CIFAR-100 with a soft-binned calibration auxiliary loss. Three checks ask whether a routing signal appears at fixed confidence, whether it replicates across training seeds, and whether a held-out predictor can exploit it against output-only and shuffled-trace controls; a sensitivity audit injects effects of known size to measure how much of them each probe recovers. None of 30 binned diagnostic tests survives multiplicity correction, and neither a nominal hit nor a borderline result recurs in its sibling seeds. Across 24 paired runs, a scalar routing probe gives no pooled improvement in routing-stratified calibration. An entropy-profile probe predicts correctness better than the same probe given shuffled profiles, yet worse than a confidence-only predictor in both binary log-loss and Brier score: its gain over shuffled traces does not become a gain over the output. Conditioning on the complete logit vector leaves the corresponding comparison unresolved. The audit shows how far these non-detections can be read: at an injected effect of 0.010 nats, the profile probe recovers 24–59% of the oracle gain, and a reference-preserving correction probe recovers 8% and 23% in the two CIFAR-100 settings, below the threshold we set for applying it to real labels. The results establish control-dependent gains and incomplete estimator recovery, not the absence of conditional routing information.

Primary area:
interpretability and explainable AI
Keywords:
Attention residual calibration, calibration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BpFsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Summary

This paper asks the question of whether the internal routing of an Attention-Residual (AR) transformer carries information about when its predictions are wrong, beyond what the model's confidence already shows. AR networks replace the fixed residual sum with input-dependent attention over earlier layers' outputs. The authors summarise each layer's routing weights as an entropy value and ask whether, among predictions of equal confidence, routing entropy helps separate correct from incorrect ones.

Critical strengths

Strengths

  • I think the entire question the paper is looking to answer is an interesting one.
  • Reproducibility is available, and the appendix includes more details to try and replicate the large code base.
Critical weaknesses

Weaknesses

  • The abstract and introduction dive straight into technical jargon/details without any thorough motivation or introduction of terms used throughout the paper. I feel like this is typically necessary, and thus, without them, the paper will read strange/be difficult to read to someone not exactly familiar to the field.
  • Following on from the above point, the paper's presentation is poor. There is undefined notation throughout, dense text with no explanations provided (as stated above), any explanations are hidden in dense appendix sections (which is fine in parts but if a reader has to read 30 pages of appendices to grasp any form of results/motivation, then it is poorly constructed). IMO, the paper needs a large overhaul on the presentation to be conference-ready.
  • Whilst it is a stylistic choice, I do not like introducing some motivation via notation is the introduction. I feel like this is a reason for the authors to not include textual motivation (which I mentioned above) which hurts presentation, and does not fill me with confidence that they can explain the motivation of their own paper.
  • The null result that the paper concludes is honest, but not very convincing for the venue. Whilst a null result is fine, analysis is weak and I feel like I did not gain additional knowledge from the paper.
  • Only running experiments on small models and the CIFAR datasets is small-scale and I feel like is not thorough enough for the venue.
  • If uncertainty information is a key theme of the paper, why have the authors not ran any tests in distributionally shifted data, such as out-of-distribution data/adversarially attacked data which is meant to induce uncertainty?
Questions for authors

Can you add a short, plain-language motivation to the abstract and introduction, defining AR, routing entropy, nats, and oracle gain before they are used?

Why did you not consider OOD/Adversarially attacked data in your experiments alongside the current evaluation?

Reviewer_yShPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether the routing entropy in Attention-Residual Transformer models contains information about prediction accuracy that goes beyond what is already captured by the model's outputs. The authors employ several diagnostic methods: permutation tests at fixed confidence levels, reproducibility checks across random seeds, comparisons between output-only models and models using shuffled routing traces, and sensitivity audits involving the injection of known routing effects. The results indicate that, while routing profiles outperform a shuffled baseline, they do not perform better than a predictor based solely on confidence; furthermore, the findings remain inconclusive when conditioned on the full set of model logits. Simulation experiments further demonstrate that a probe can only recover a portion of a known signal; consequently, the failure to detect an incremental gain cannot be interpreted as evidence that routing information does not exist.

Critical strengths

1.The question of "whether internal signals provide incremental information" is defined rigorously; the study identifies the limitations of using a "shuffled control" and introduces an "output-only baseline" instead. 2.The approach to handling negative results is thorough: the use of cross-seed replication, corrections for multiple testing, proper scoring rules, and an "injected-signal audit" significantly enhances the credibility of the findings.

Critical weaknesses

The empirical scope is too narrow, covering only small-scale, trained-from-scratch autoregressive (AR) models—augmented with auxiliary calibration losses—evaluated on the CIFAR dataset; these findings cannot be generalized to typical dynamic routing models. Furthermore, the core results concerning "complete-output" remain inconclusive, and the signal recovery rate from the probes is low. Consequently, the paper neither identifies a viable new uncertainty signal nor provides a robust proof that such a signal does not exist.

Questions for authors

1 If the experiments were repeated using a standard autoregressive (AR) model with cross-entropy loss (i.e., without the "soft-binned calibration loss"), would the conclusions still hold? 2 Why focus exclusively on the entropy profile? Would it be possible to employ a probe for the raw routing weights that accounts for permutation invariance or sequence structure? 3 Could this be extended to at least one non-AR routing mechanism—such as Mixture-of-Experts (MoE), token pruning, or adaptive depth?

Reviewer_PdP2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies if routing entropy in attention-residual models contain uncertainty information that is not already in a model's output. The authors perform three checks and a sensitivity audit, the results are generally not significant. Specifically, the paper presents simple baselines that fail to reveal additional information from a routing that isn't present in the final prediction, but also can't prove that there is no additional information from the routing choices.

Critical strengths

I appreciate the authors' attention to detail in their experiment design. The experiment protocols are sound, there are careful controls in each experiment to repeat and ablate findings, and the paper is very honest about the null results.

Critical weaknesses

This appears to be a negative result paper. However, my main concern is that there isn't a clear conclusion, and the intuition for why routing could contain additional information on predictive uncertainty itself is poorly-motivated. The paper does not motivate or give intuition for why routing might contain uncertainty calibration information, nor does it present sufficient background on how existing works have utilized information in the routing choices. This is the big motivation that could potentially make the findings in this paper interesting and surprising, but right now this paper reads more as an uncertainty quantification idea that turned out to not work in practice.

Questions for authors

What is the intuition for thinking attention residual routing can contain uncertainty information that will not be present in the final prediction, that motivated your research? Since your simulation experiment shows that the probes you fitted can't recover uncertainty inormation within the routing even when they are manually injected, do you think using a deeper probe or more data could reveal uncertainty from routings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.