acceptodds
Under review as a conference paper at ICLR 2027

FSR: Assessing Reliability in Language Model Interpretability

Abstract

Affine lenses translate the hidden states of large language models into intermediate predictions. But when should a researcher trust those predictions to draw a conclusion about the full model? We introduce Fisher–Schur Reliability (FSR), which uses readout errors on reference prompts to assess reliability for the specific quantity being measured. For token distributions, FSR measures how readout errors align with distinctions between likely tokens. This information reduces error-ranking loss (EAURC) by 4.1–27.3% beyond confidence, input atypicality, and token frequency across five models, helps richer predictors, and remains useful on a disjoint cohort with the score frozen. For intervention effects, FSR compares each estimate with its residual error scale to select directions to report. With the readout fixed, it improves lens and attribution-patching reporting. A small model of each reading's error scale further improves selection on a fresh 512-prompt cohort, and both forms of reliability scoring replicate on Gemma beyond the two Qwen models used for development. FSR thus complements effect estimation: it assesses which readings support an interpretation and which warrant exact checking.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, affine lenses, language models, uncertainty quantification, Fisher information, intervention analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8q8AOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors tackle the problem of estimating how much an intermediate token readout (e.g. logit lens) differs from the final layer readout. They introduce a method called FSR that collects (intermediate readout, final readout) data on a corpus and learns to predict the intermediate-final deviation using 4 main features: Mahalanobis distance of the intermediate hidden state, entropy of the intermediate readout token distribution, lexical frequency, and anisotropic alignment. They show better results than previous methods.

Critical strengths
  • Designs several features for predicting the deviation between intermediate and final token readouts. The results show the features’ effectiveness.
Critical weaknesses
  • The exposition is incomplete and hard to follow, making it difficult to evaluate the contributions and soundness. Most importantly, the precise impact of the work is unclear. The experiments solely evaluate the ability to predict how close an intermediate readout (at fixed layers of 75% and 87.5% depth) matches the final readout. The paper frames FSR as an interpretability tool, but this framing is complicated by FSR’s highly expressive feature design, which calls into question how faithful FSR’s predictions are to the model’s computation (i.e. what it’s “thinking” at the intermediate layer). On the applicability side, no downstream insights are obtained, such as how this can improve safety with interventions, improve efficiency with early stopping, or improve trustworthiness with confidence of answer correctness.
Questions for authors
  • How are interventions related to the main problem of estimating the accuracy of an intermediate readout? Are interventions just to construct more features for FSR?
  • What is the motivation for choosing the layers at 75% and 87.5% depth for all experiments?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 8q8A

Thank you for the review. The abstract and introduction now open with the two questions the paper asks and a concrete case, each definition follows a plain-language statement of what it measures, and Figure 1 is redrawn; the general response lists the changes. Numbers below refer to the revised PDF. Brackets are paired 95% intervals over prompts.

W1. What the paper evaluates

FSR does not produce a reading of the model. It estimates how far an existing reading can be trusted. The paper applies it to two kinds of reading:

  1. Intermediate token distributions (Sections 4.1–4.3). How far is a lens distribution from the model's final distribution? This is the part the review summarizes.
  2. Estimated intervention effects (Sections 4.4–4.5). When a lens or attribution patching (AtP) estimates that ablating a component raises token A relative to token B, does the real ablation do so? These experiments cover three models and two kinds of ablation, each scored against the exact intervention.

Q1. How do interventions relate to readout accuracy?

They are a second application, not extra features for the first. In both, FSR measures a reading's errors on reference prompts and asks how those errors affect the conclusion being drawn: the likely tokens of a distribution, or the token comparison whose change is reported. Exact intervention results are used only for fitting and evaluation, never as inputs when a new estimate is scored.

W1. Downstream use

Deciding which estimated effects to report. When about 10% of estimates are reported, the share that point in the wrong direction (%; Table 4, where FSR is the per-reading score and Gemma uses pre-softcap contrasts):

EstimatesLargest effects firstFSR
Lens, Qwen13.7811.50
AtP, Qwen1.760.07
Lens, Gemma-2-2B25.0921.35
AtP, Gemma-2-2B6.853.91

Deciding which estimates to verify exactly (Table 6, new). Every report is kept, and a fixed number of exact reruns is spent on the reports a score ranks least reliable. Wrong AtP directions that remain among 4,096 reports (%):

Checks chosen by25% checked50% checked
No checks8.868.86
Effect magnitude3.221.39
FSR2.250.39

At the 25% budget FSR leaves 40 fewer wrong reports than effect magnitude (0.98 points [0.56, 1.39]). Table 6 gives the 25% budget; the 50% column and the no-check rate are secondary results reported in this reply. The comparison uses the 512 Qwen prompts of the reporting study. For lens estimates, and for AtP at a 10% budget, the two choices are comparable.

Early stopping (Appendix H, Table 19, new). A rule lets each position stop at the first of eight depths where the score predicts the lens distribution is close enough to the final one; thresholds are set on separate prompts. With accepted readings held to a mean KL of 0.5, it stops 66% of positions early and avoids 11.8% of decoder blocks, averaged over the five models, against 10.2% for an entropy rule (1.62 points [1.46, 1.79]) and 11.0% for the same score without the covariance statistic (0.80 points [0.74, 0.86]). FSR is ahead of both at all five targets. These are block counts on fixed text, not a measured generation speedup.

Answer correctness. Both applications measure agreement with the model's own behaviour, not whether its answer is factually correct (Section 5).

W1. Is the score faithful to the model's computation?

FSR makes no claim about what the model is "thinking" at a layer. It predicts quantities that are measured on the model itself: the distance between the lens distribution and the actual final distribution, and the direction of the exact intervention effect. Every result checks these predictions on prompts that were not used for fitting.

The predictor is small, with four inputs, and the gain from the covariance statistic persists in matched-predictor comparisons, in all ten model–layer settings each time:

  • added to the other three inputs of the learned score: 4.1–27.3% lower error-ranking loss (Table 2), and 4.2–25.8% on separate prompts with nothing refitted (Table 3);
  • with simple additive weighting: 1.5–12.5% (Section 4.2);
  • added to a predictor with eleven other diagnostics: 3.1% on average (Table 1).

Q2. Why 75% and 87.5% depth?

One rule was fixed in advance for all five models: read after 75% and 87.5% of the blocks. These are late readings that still leave about an eighth to a quarter of the network to run, and a shared rule avoids picking the best layer for each model (Section 4.1). The result is not specific to these depths: at about 25% and 50% depth the additive score improves error ranking in all ten settings, by 0.7–23.6% (Table 16). The intervention experiments edit the same two layers in the Qwen models.

Thank you for your response and clarifications. I will keep my score for the following reasons.

  • The writing remains very difficult to read and is not fit for human readers.
  • Evaluation metrics with limited practical relevance. For deciding which estimated effects to report, I don't see the utility in predicting such a small proportion (10%) of the estimates, since the motivated problem is to assess the reliability of any given reading (intermediate token distribution or estimated intervention effect). For deciding which estimates to verify exactly, a very large proportion (25% or 50%) is verified exactly, which severely limits the efficiency benefits compared to just exactly verifying 100% of the time. Also, for both experiments, only counting the cases with a wrong direction seems to be too coarse of an evaluation, since there can be large errors with the same sign.
  • For early stopping, I would expect that the extra computation of FSR would eliminate any gains that stand to be gained from early stopping.
AuthorsOpenReview ↗

Response to Reviewer 8q8A

Thank you for the follow-up. Points 2–4 are answered with new measurements on the prompts of the paper's reporting study (Qwen, 512 prompts, eight settings, attribution patching). "FSR" is the per-reading score, "magnitude" ranks estimates by the size of the estimated effect, and brackets are paired 95% intervals over prompts.

1. Writing

We made a further readability revision. The abstract explains what each score is used for, and the introduction works through a red-versus-blue example. Section 3 now begins with how FSR is fitted and used. Each application presents its actual score before the derivation; the general error formula follows the concrete cases. Supporting variants are introduced where they are compared, and fitting settings are in the appendix.

2. Reporting 10% of the estimates

FSR scores every reading; 10% is one point on that ranking, and the paper also gives 25% and 50% (Table 13). The advantage holds at 10%, 25%, 50% and 75%; at 100% both rankings report every estimate. Wrong directions, and estimates that miss the exact effect by more than half its size (relative error = |estimate − exact| / |exact|), in %:

Share of estimates reported10%25%50%75%All
Wrong direction, magnitude1.761.952.664.078.78
Wrong direction, FSR0.070.170.712.568.78
Relative error > 0.5, magnitude13.7013.9916.2320.4730.34
Relative error > 0.5, FSR2.765.119.8817.2730.34

For any given reading, with no selection at all: over all 457,912 readings, the rank correlation between the score and the realised relative error is 0.572 for FSR against 0.370 for magnitude (+0.202 [0.199, 0.205]). Averaged over every share from 1% to 100%, the wrong-direction rate is 1.79% against 3.30%. Sections 4.1–4.3 likewise score every intermediate token distribution, with no selection.

3. Exact checks at 25% and 50%

The budget is the user's choice. Table 6 reports the 25% budget and Appendix C.4 the 50% budget; both are points on a curve. Share of all wrong directions removed (%) among the 4,096 reports:

Reports checked exactly10%25%50%75%
Chosen by FSR41.974.795.699.4
Chosen by magnitude38.063.684.394.2
Chosen at random10.225.050.075.0

A quarter of the exact runs removes three quarters of the wrong directions, and half of them removes 95.6%. On these reports, leaving at most 1% with a wrong direction takes exact runs on 36.9% of them [31.6, 43.2] under FSR, 58.4% under magnitude and 88.9% under random checking. To match what FSR reaches with 25%, magnitude needs 34.6% [30.5, 39.8] and random checking 74.8%. These figures count exact runs: as in Section 4.5, the comparison is of how a fixed number of checks is allocated, not of total runtime.

4. Large errors with the same sign

FSR also orders estimates by the size of their error. Among the 10% of estimates it reports (point 2):

  • 2.76% miss by more than half the exact effect, against 13.70% under magnitude (−10.95 points [−11.40, −10.51]), and 0.82% against 6.50% miss by more than the whole effect;
  • large errors with the correct sign make up 2.68% of the reported estimates, against 11.94% (−9.26 [−9.62, −8.87]);
  • the median relative error is 0.109 against 0.191.

The same holds on 683 further Gemma-2-2B prompts: 23.42% against 32.20% miss by more than half (−8.78 [−9.27, −8.29]). The error scale that FSR predicts for a reading has rank correlation 0.579 with its realised absolute error; the estimated effect itself has 0.386. Lens estimates of effects are coarse in size under any ranking (median relative error above 1), so for them the direction is the quantity to report, as in Table 4.

The exact checks of point 3 remove large errors whatever their sign. Reports with relative error above 0.5 fall from 30.93% to 13.67% after 25% of checks (magnitude 15.62%, random 23.20%) and to 5.27% after 50% (8.18%, 15.47%).

5. Early stopping

We measured this. Once the cost of computing the score is counted, the stopping rule gives no net saving on the three models we timed; the block counts in our first reply did not include that cost. The stopping study was not in the submission; the first revision added it as Appendix H, and we have now removed it. None of the paper's claims rests on it, and the paper's practical uses are the two in points 2–4: deciding which estimates to report and which to verify.

Reviewer_jjC4OpenReview ↗
Rating
3
Confidence
2(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Fisher–Schur Reliability (FSR), a framework for assessing the reliability of intermediate predictions and intervention effect estimates in language models. The method uses residual statistics to estimate measurement-specific errors. Experiments across multiple language models demonstrate improvements in prediction error ranking and the reliability of reported intervention directions.

Critical strengths

The paper is well-written and technically sound, with clear motivation and detailed mathematical derivations. The experimental evaluation is also thorough, covering comparisons with existing methods, extensive ablation studies, and validation across multiple models.

Critical weaknesses

The incremental benefit of AFS over the Rich Baseline is relatively modest (approximately 3.1% improvement in macro EAURC). It remains unclear whether this improvement justifies the additional computational and storage costs associated with covariance estimation and evaluation.

Questions for authors

See Critical Weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer jjC4

Thank you for the review. The revision now measures both sides of this question: what the covariance statistic AFSA_{\rm FS} adds, and what it costs. Numbers below refer to the revised PDF.

W1. How much does it add?

The 3.1% is the gain over our strongest control, Rich GBM, which already combines eleven other diagnostics. Added to a smaller predictor, the same statistic gives more (reduction in the error-ranking loss, EAURC):

AFSA_{\rm FS} added toReductionModel–layer settings improved
Three inputs: Mahalanobis distance of the hidden state, entropy, token frequency (Table 2)4.1–27.3%10 of 10
The same, on 128 separate prompts with nothing refitted (Table 3)4.2–25.8%10 of 10
Rich GBM, eleven inputs (Table 1)3.1% on average10 of 10

Every paired interval excludes zero, except in one setting of the last row.

W1. What does it cost?

No extra model passes. The covariance comes from the same reference states that fit the lens. Building it takes 0.234 seconds, once.

Time (Table 7, new). Scoring time on top of the model's forward pass, for Qwen1.5-0.5B at L18:

Readings per batchRich GBM (ms)Rich FSR (ms)Increase
326.7007.83717.0%
969.18510.24911.6%

The statistic adds about 1.1 ms per batch; it never forms a vocabulary-sized covariance matrix (Appendix F.1).

Memory. The statistic itself stores 9 MiB. The larger cost is a double-precision copy of the output matrix, 1.16 GiB, kept for numerical accuracy. Peak GPU memory rises from 2.62 to 3.79 GiB at batch 32.

W1. Is the gain worth the cost?

In the timed setting, about 1.1 ms per batch and 1.2 GiB give a ranking loss 1.6% below Rich GBM. Across all ten model–layer settings the gain over Rich GBM averages 3.1%. The statistic needs no extra model passes, so the gain comes at a fixed cost per batch.

Reviewer_jjC4OpenReview ↗

Response to the Authors Comments

Thank you for your response. It addresses my concerns. Since I have already given the paper a positive rating, I will maintain my current rating. However, I will increase my confidence score from 1 to 2.

Reviewer_WoDeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Fisher-Schur Reliability (FSR) to quantify whether intermediate readouts or estimated causal effects in LLMs faithfully reflect the final model output. FSR uses statistics estimated from reference queries to model errors between an affine lens and the final model state, then projects the error covariance onto the quantity being measured. It uses Fisher-weighted covariance for token distributions and projected error scales for intervention effects.

Critical strengths
  1. The mathematical formulation and the evaluation for Fisher alignment is clean and computationally practical. Proposition 1 separates the anisotropic residual covariance alignment from the isotropic spread. The factorization in equation 8 avoids the V \times V covariance matrix by caching V diagonal entries. This is consequential to acceptance because it makes vocabulary-scale covariance propagation tractable for large-vocabulary LLMs up to 27B parameters.

  2. The paper demonstrates sound statistical analysis and transparent ablation. The authors strictly separate fitting, calibration, test, and pre-frozen replication cohorts and candidly report negative controls in the appendix. This is consequential because it reduces the risk of selecting thresholds on evaluation outcomes. Adding the covariance feature improves distributional error ranking in all ten primary model–layer cells, with the compact model’s improvement also holding under frozen-score validation.

Critical weaknesses

The abstract and main text highlight that conditioning the error scale on each reading reduces AtP wrong-direction reports from 1.45%1.45\% to 0.07%0.07\% on 512 fresh Qwen prompts. The retrospective control in Appendix C.3 (Table 9, line 980) reports that Response norm alone (∥Δy^∥=∥JcΔc∥2\Vert{}\hat{\Delta y}\Vert{} = \Vert{}J_c \Delta c\Vert{}_2) achieves a 0.063%0.063\% wrong-direction rate and Response + change norms achieves 0.060%0.060\%. As the authors acknowledge, the norm-only controls achieve comparable AtP performance without using projected residual risk. Thus, these results do not establish an additional benefit from covariance geometry for per-reading AtP.

Questions for authors

What is the wrong-direction rate of per-reading AtP if restricted strictly to features available from a single scalar backward pass (excluding the full response-vector norm) and how does the wall-clock runtime of double-backward per-reading AtP compare against exact forward patching from layer ll as the number of candidate interventions grows?

2 comments
AuthorsOpenReview ↗

Response to Reviewer WoDe

Thank you for the careful reading. The revision separates the sources of the gain for attribution patching (AtP), and the new results show what FSR adds beyond response size: a score that uses no response vector keeps about two thirds of the improvement with one backward pass (Q1a), and the full score chooses exact checks better than the response norm alone (W1). Numbers below refer to the revised PDF, where Table 9 of the submission is Table 12. Brackets are paired 95% intervals over prompts.

W1. What covariance contributes

For per-reading AtP, most of the gain comes from the size of the predicted state change: the response norm alone gives 0.063% wrong-direction reports and the full score 0.074%. The abstract, introduction and Section 4.4 now state this. Covariance contributes in two places:

  • Lens estimates. Adding the covariance-based error scale to the two norms lowers wrong-direction reports by 2.35 points [2.09, 2.60] (Table 12).
  • Global FSR, which uses the covariance-based error scale and no norm, improves on effect magnitude for both estimators: 13.78% → 12.31% (lens) and 1.76% → 1.45% (AtP) on fresh Qwen prompts, and by 2.19 [1.51, 2.84] and 1.09 [0.77, 1.42] points on Gemma-2-2B (pre-softcap contrasts; Table 4).

Beyond response size, the full AtP score also helps when it chooses which reports to verify exactly: with 25% of reports checked it leaves 2.246% wrong directions, against 2.710% for the response norm alone (0.464 points [0.122, 0.732]; Table 6, new). The full score adds the error scale, the edit size and the lens gap to the norm, so this is their combined contribution.

Q1a. Inputs from one scalar backward pass

The wrong-direction rate is 0.545% (Table 5, new). This score uses the fitted error scale for the token comparison, the size of the edit, and the clean lens gap between the two tokens. It uses neither the predicted state change nor its norm. It is newly fitted on the original fitting and calibration prompts.

ScoreWrong-direction reports (%)Readings reported (%)
Global FSR1.44710.161
Without the response-vector norm0.54510.457
Full per-reading FSR0.07410.038

It recovers about two thirds of the full score's improvement over global FSR (0.902 points [0.804, 1.013]). When it chooses the exact checks (Table 6), it leaves 2.368% wrong directions, against 3.223% for effect magnitude (0.854 points [0.391, 1.172]) and 2.246% for the full score.

Q1b. Runtime as the number of candidate edits grows

Table 18 (new, Appendix G.2) times one token comparison at the final position, for edits of the L21 MLP output in Qwen1.5-0.5B, where three blocks remain to run. Times are the median added time after the shared clean forward pass, excluding FSR's own scoring:

Candidate editsScalar AtP, one backward pass (ms)Double-backward AtP, response + norm (ms)Exact forward patching (ms)
15.979.752.05
45.9619.836.44
166.0459.7423.59
645.97218.8092.33

Double-backward AtP is slower than exact patching at every count: 4.8× at one edit and 2.4× at 64. There is no crossover over the tested 1–64 sequential candidates; the times rise by about 3.3 ms and 1.4 ms per added edit. Scalar AtP stays near 6 ms because one gradient serves every edit, and it is faster than exact patching from four edits upward.

For many candidate edits of one token comparison at a late layer, the scalar-input score is therefore the economical form of FSR: one backward pass serves every edit, and it keeps about two thirds of the full score's improvement (Q1a). The timed response covers one output position. The reporting study uses the sequence-wide response of a single ablation, which was not timed, so the error rates and these timings describe different workloads.

The added comparisons in Tables 5 and 6 use the 512 Qwen prompts evaluated in the submission.

Thank you for addressing the weaknesses raised here. After going through the response to all the reviewers, I'll maintain my initial rating of 3 (weak acceptance).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We thank the reviewers for their careful reading. A revised PDF is uploaded, and the table and section numbers in our replies refer to it. The title, method, fitted models and primary result tables are unchanged. The changes, by concern:

1. Clarity and purpose (8q8A, and the presentation scores). The abstract and introduction now open with the two questions the paper asks: how far an intermediate token distribution is from the model's final distribution, and whether an estimated intervention effect has the correct direction. Each definition follows a plain-language statement, a short example explains why paired errors can cancel, Figure 1 is redrawn, and the limitations are stated in one paragraph (Section 5). A further readability revision puts how FSR is fitted and used at the start of Section 3 and presents each score before its derivation.

2. Source of the attribution-patching gain (WoDe). The abstract, introduction and Section 4.4 now state that most of the per-reading gain for attribution patching comes from the size of the predicted state change, and that covariance adds information for lens estimates. New Table 5 reports the score restricted to inputs available from one scalar backward pass. The controls that WoDe cites (Table 9 of the submission) are now Table 12.

3. Concrete uses (8q8A). New Table 6 and Section 4.5: every report is kept and a fixed number of exact intervention checks is spent on the reports a score ranks least reliable. Choosing the checks with FSR leaves 2.25% wrong attribution-patching directions, against 3.22% when choosing by effect magnitude (0.98 points; paired 95% interval [0.56, 1.39]).

4. Cost (jjC4, WoDe). New Table 7 measures the time and memory that the covariance statistic adds to distributional scoring. New Table 18 (Appendix G.2) compares scalar attribution patching, full-response attribution patching and exact checks as the number of candidate edits grows.

The added comparisons in Tables 5 and 6 use the 512 Qwen prompts evaluated in the submission. The scalar-input control in Table 5 is newly fitted on the original fitting and calibration prompts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.