acceptodds
Under review as a conference paper at ICLR 2027

WHEN ROUTING PROBES IMPROVE WITHOUT NEW INFORMATION: AUDITING UNCERTAINTY BEYOND MODEL OUTPUTS

Abstract

Improved error prediction from routing traces does not by itself establish information beyond a model's outputs. We examine this inference with real output–routing pairs from vision transformers and correctness labels redrawn from a fixed output-only function fitted on disjoint data, so that routing adds no information by construction. A width-matched MLP comparison nevertheless reports a routing gain in 308 of 600 confidence-only evaluations, whereas a linear comparison reports none. On a six-model panel, holding each training trajectory fixed and choosing the checkpoint by validation log loss instead of validation accuracy reduces detections from 50/120 to 0/120, and an independently implemented MLP shows the same contrast (83/120 to 0/120). Log-loss selection removes all 528 raw detections among the 1,920 MLP null evaluations but does not make the raw comparison sensitive: in two matched synthetic settings it detects a signal of about 0.005 nats in 0/20 replicates each, while a conditional permutation test based on an estimated routing law detects it in 11/20 and 10/20 and has low measured rejection rates on the null benchmark. On real correctness labels, the primary conditional analysis finds model-relative evidence of conditional-assignment dependence in five DeiT attention-residual families; in four of them the evidence persists under two specified variants of the conditional model, and no family passes an additional noise criterion. Improving a probe's fit and testing for incremental information are separate problems that need separate validation.

Primary area:
interpretability and explainable AI
Keywords:
Uncertainty, RIA, attention residual, routing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VGJYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors of this paper investigate whether using both model outputs and internal routing information improves a probe's ability to predict whether a model is correct. They argue that although routing information can improve performance, this does not necessarily mean it provides any new information beyond what is already available from the outputs. Instead, they say that these improvements might come from how the probes are trained or how easily they can learn from the additional features. The authors investigate this using a benchmark where routing is known to provide no additional information, and evaluate how checkpoint selection and different testing approaches affect the results.

The authors find that the choice of checkpoint selection can produce misleading improvements, which disappear when selecting checkpoints using validation log loss rather than accuracy. They also compare this against conditional permutation testing, which is better at detecting small additional signals in their experiments. Finally, they evaluate real correctness labels across several vision transformer models, finding some evidence of conditional dependence in DeiT models, although none passes their additional noise criterion.

Critical strengths
  1. The paper raised a useful point about how improvements in probe performance can be incorrectly interpreted as evidence of additional information. Although this is not a new observation, investigating how it affects routing-based uncertainty probes is worthwhile.

  2. The experimental setup is reasonable, particularly the benchmark where routing is known to provide no additional information.

Critical weaknesses
  1. First, I find the contribution somewhat limited. The distinction between improved probe performance and additional information has already been talked about in previous work, and the authors reuse conditional permutation testing. The empirical audit, particularly the controlled checkpoint selection experiments, is useful, but I am not convinced that the resulting insights are anything more .

  2. The conditional testing results are not particularly conclusive. The tests depend on an estimated conditional distribution of routing given outputs, and four of the twelve positive R1 comparisons don't remain positive under the alternative models. Although the authors investigate the power of individual tests, they do not establish the sensitivity of the complete R1/R2 decision rules. This makes it difficult to interpret the real-label findings, particularly the lack of R2 detections, without knowing how sensitive these rules actually are.

  3. I also found the presentation of the paper unnecessarily difficult to follow in places, with several important experimental results placed in the appendices rather than the main paper, and the subsections titles are often overly long and awkward.

Questions for authors
  1. (Relating to W1): Given that the distinction between probe performance and additional information is already established, what do the authors consider the main novel insight of this work?

  2. (Relating to W2): Could the authors evaluate the power of the complete R1/R2 decision rules at confident can we be in the reported findings without establishing the sensitivity of these rules?

Reviewer_5ekeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper challenges the inference that if a probe that utilises a model’s routing predicts the model’s errors better than one reading only its outputs, the routing must carry information about errors beyond those outputs. It argues that these higher-performing probes can be explained for reasons that do not require additional routing information. Namely,

  • Checkpoint selection: the MLP probe’s checkpoint is chosen by validation accuracy but evaluated using log loss, which can leave the output-only baseline probe underfit.
  • Baseline capacity: a restricted probe class cannot represent the target from the outputs alone, so a redundant feature can still improve its fit (Proposition 1).
  • Unfair comparison: global shuffling cannot directly establish whether routing adds conditional information, since it breaks routing’s dependence on the outputs. They test this by constructing a known-answer null where they keep the real outputs and routing but redraw correctness labels from confidence alone, meaning routing provably adds nothing beyond the outputs, and measuring how often each comparison reports a gain despite this. The authors provide experiments that demonstrate the first reason and address the third using a conditional permutation test based on an estimated routing distribution (the second is argued via Proposition 1). On real labels, they cannot conclude whether the conditional-assignment evidence reflects additional routing information or misspecification of the conditional model.
Critical strengths
  • The central claim is sound. The authors justify why a probe may perform better with routing without the routing containing extra information beyond the outputs, and identify three issues that can lead to this inference; the checkpoint-selection route in particular is demonstrated empirically.
  • The experiments are extensive, with some nice experimental design, particularly the checkpoint-selection demonstration and the construction of the label null to guarantee T⊥R∣OT \perp R \mid O by construction, conditional on the fixed generator.
  • The results are statistically careful, e.g., the distinction between the exact label null and the assumptions needed for the conditional permutation test of Berrett et al. (2020).
Critical weaknesses
  • My primary issue with this work is its presentation, and therefore its communication. The core message is capable of being explained simply, but the exposition makes it very hard to grasp. I break this into a few issues:
    • Notation and terminology are often used before they are properly explained (e.g. OO, RR in the introduction; I(T;R∣O)I(T;R\mid O) should be explicitly identified as conditional mutual information; "flag" vs "detection"). The sheer quantity of notation seems unnecessary—a single, consistent set introduced once would have served better. Section 3 also introduces many marking criteria (Type 0–III, R1/R2) without making their relationships clear enough.
    • The abstract and introduction do a poor job of setting up the basic motivation and problem, instead focusing heavily on exact results that are stated before the material needed to interpret them. Figure 1 is heavily overloaded and would be better split so that each component is introduced where it is used.
    • There are many results, but presenting them in-line in the main text makes them nearly impossible to read and understand. Figure 5 (in the appendix) is a much clearer presentation of the results, yet it is relegated to the appendix. The main portion of the paper should be legible without the appendix.
  • The contribution is narrow. After accounting for correctly cited prior work, the main contribution is therefore measuring probe gains under the label null and diagnosing the effect of checkpoint selection. While these experiments are convincing, I remain unsure how they change our understanding of existing probe results. How commonly is the problematic methodology used, and does it affect those results? This leaves the impact on the wider literature unclear.
  • Finally, the real-label results remain inconclusive. The synthetic results show that the corrected comparison can miss small signals, while the power of the complete screen is not established. Consequently, “no family passes” provides limited evidence about whether routing carries additional information in the evaluated models.
Questions for authors
  • The strongest result (the checkpoint-selection effect) seems agnostic to the routing setup, as mentioned in Section 4.2. Could the authors connect this more directly to the wider use of probes in the literature?
  • What do these results imply for the motivating uses of routing uncertainty in Pavlitska et al. and Lee et al.? Should the interpretation of these results change?
  • Could the authors characterise the complete screen’s power across different implanted-signal strengths, including the required contrasts, both scores, and seed aggregation? Section 4.5 is hard to interpret without it.
Reviewer_qgeEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors investigate whether routing traces contain information about prediction errors beyond a model’s outputs. To this end, they construct an output-only label generator and compare different probing and conditional permutation procedures. The controlled null experiments and comparisons along fixed training trajectories are useful, but I have major concerns regarding the significance of the findings and the readability of the manuscript.

Critical strengths

see above

Critical weaknesses

First, the main empirical finding largely amounts to a mismatch between checkpoint selection and evaluation: scikit-learn’s MLP selects checkpoints using validation accuracy when early stopping is enabled, whereas the authors evaluate probability predictions using log loss and Brier score. The accuracy-based validation criterion is documented behaviour. Once checkpoints are selected by validation log loss, all 528 raw detections in the null benchmark disappear. This is a useful implementation warning, but I do not think quantifying this particular mismatch constitutes a sufficiently significant contribution. Reproducing the effect in PyTorch confirms that it is not specific to scikit-learn, but still comes down to the same metric mismatch.

Second, I do not see substantial novelty in the conceptual argument. Proposition 1 illustrates that adding a deterministic nonlinear transformation can improve a restricted probe without adding Shannon information. This is a straightforward example of the established distinction between usable information and information content, as discussed by Xu et al. and Hewitt et al., which the authors themselves acknowledge. The conditional permutation test is likewise an existing method. The contribution therefore rests primarily on the empirical audit, whose main finding is the well-known checkpoint-selection issue above.

Most importantly, I am not convinced that the real-label experiments establish a clear scientific result. All three conditional models have the same rank-normal Gaussian framework, and the authors report concerning model-fit diagnostics in some positive settings. Furthermore, the power of the final decision rules is not analysed, so the non-detections are also difficult to interpret.

Finally, the manuscript is unnecessarily difficult to read. The extensive discussion of internal task identifiers, git hashes, protocol filenames, superseded analyses and even SLURM and login-node execution details often reads like an internal experiment log. The repeated qualifications and references to historical corrections make it particularly difficult to determine what the paper actually want to establish.

Questions for authors

see above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.