- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors tackle the problem of estimating how much an intermediate token readout (e.g. logit lens) differs from the final layer readout. They introduce a method called FSR that collects (intermediate readout, final readout) data on a corpus and learns to predict the intermediate-final deviation using 4 main features: Mahalanobis distance of the intermediate hidden state, entropy of the intermediate readout token distribution, lexical frequency, and anisotropic alignment. They show better results than previous methods.
Critical strengths
- Designs several features for predicting the deviation between intermediate and final token readouts. The results show the features’ effectiveness.
Critical weaknesses
- The exposition is incomplete and hard to follow, making it difficult to evaluate the contributions and soundness. Most importantly, the precise impact of the work is unclear. The experiments solely evaluate the ability to predict how close an intermediate readout (at fixed layers of 75% and 87.5% depth) matches the final readout. The paper frames FSR as an interpretability tool, but this framing is complicated by FSR’s highly expressive feature design, which calls into question how faithful FSR’s predictions are to the model’s computation (i.e. what it’s “thinking” at the intermediate layer). On the applicability side, no downstream insights are obtained, such as how this can improve safety with interventions, improve efficiency with early stopping, or improve trustworthiness with confidence of answer correctness.
Questions for authors
- How are interventions related to the main problem of estimating the accuracy of an intermediate readout? Are interventions just to construct more features for FSR?
- What is the motivation for choosing the layers at 75% and 87.5% depth for all experiments?
6 comments
Thank you for your response and clarifications. I will keep my score for the following reasons.
- The writing remains very difficult to read and is not fit for human readers.
- Evaluation metrics with limited practical relevance. For deciding which estimated effects to report, I don't see the utility in predicting such a small proportion (10%) of the estimates, since the motivated problem is to assess the reliability of any given reading (intermediate token distribution or estimated intervention effect). For deciding which estimates to verify exactly, a very large proportion (25% or 50%) is verified exactly, which severely limits the efficiency benefits compared to just exactly verifying 100% of the time. Also, for both experiments, only counting the cases with a wrong direction seems to be too coarse of an evaluation, since there can be large errors with the same sign.
- For early stopping, I would expect that the extra computation of FSR would eliminate any gains that stand to be gained from early stopping.
Response to Reviewer 8q8A
Thank you for the follow-up. Points 2–4 are answered with new measurements on the prompts of the paper's reporting study (Qwen, 512 prompts, eight settings, attribution patching). "FSR" is the per-reading score, "magnitude" ranks estimates by the size of the estimated effect, and brackets are paired 95% intervals over prompts.
1. Writing
We made a further readability revision. The abstract explains what each score is used for, and the introduction works through a red-versus-blue example. Section 3 now begins with how FSR is fitted and used. Each application presents its actual score before the derivation; the general error formula follows the concrete cases. Supporting variants are introduced where they are compared, and fitting settings are in the appendix.
2. Reporting 10% of the estimates
FSR scores every reading; 10% is one point on that ranking, and the paper also gives 25% and 50% (Table 13). The advantage holds at 10%, 25%, 50% and 75%; at 100% both rankings report every estimate. Wrong directions, and estimates that miss the exact effect by more than half its size (relative error = |estimate − exact| / |exact|), in %:
| Share of estimates reported | 10% | 25% | 50% | 75% | All |
|---|---|---|---|---|---|
| Wrong direction, magnitude | 1.76 | 1.95 | 2.66 | 4.07 | 8.78 |
| Wrong direction, FSR | 0.07 | 0.17 | 0.71 | 2.56 | 8.78 |
| Relative error > 0.5, magnitude | 13.70 | 13.99 | 16.23 | 20.47 | 30.34 |
| Relative error > 0.5, FSR | 2.76 | 5.11 | 9.88 | 17.27 | 30.34 |
For any given reading, with no selection at all: over all 457,912 readings, the rank correlation between the score and the realised relative error is 0.572 for FSR against 0.370 for magnitude (+0.202 [0.199, 0.205]). Averaged over every share from 1% to 100%, the wrong-direction rate is 1.79% against 3.30%. Sections 4.1–4.3 likewise score every intermediate token distribution, with no selection.
3. Exact checks at 25% and 50%
The budget is the user's choice. Table 6 reports the 25% budget and Appendix C.4 the 50% budget; both are points on a curve. Share of all wrong directions removed (%) among the 4,096 reports:
| Reports checked exactly | 10% | 25% | 50% | 75% |
|---|---|---|---|---|
| Chosen by FSR | 41.9 | 74.7 | 95.6 | 99.4 |
| Chosen by magnitude | 38.0 | 63.6 | 84.3 | 94.2 |
| Chosen at random | 10.2 | 25.0 | 50.0 | 75.0 |
A quarter of the exact runs removes three quarters of the wrong directions, and half of them removes 95.6%. On these reports, leaving at most 1% with a wrong direction takes exact runs on 36.9% of them [31.6, 43.2] under FSR, 58.4% under magnitude and 88.9% under random checking. To match what FSR reaches with 25%, magnitude needs 34.6% [30.5, 39.8] and random checking 74.8%. These figures count exact runs: as in Section 4.5, the comparison is of how a fixed number of checks is allocated, not of total runtime.
4. Large errors with the same sign
FSR also orders estimates by the size of their error. Among the 10% of estimates it reports (point 2):
- 2.76% miss by more than half the exact effect, against 13.70% under magnitude (−10.95 points [−11.40, −10.51]), and 0.82% against 6.50% miss by more than the whole effect;
- large errors with the correct sign make up 2.68% of the reported estimates, against 11.94% (−9.26 [−9.62, −8.87]);
- the median relative error is 0.109 against 0.191.
The same holds on 683 further Gemma-2-2B prompts: 23.42% against 32.20% miss by more than half (−8.78 [−9.27, −8.29]). The error scale that FSR predicts for a reading has rank correlation 0.579 with its realised absolute error; the estimated effect itself has 0.386. Lens estimates of effects are coarse in size under any ranking (median relative error above 1), so for them the direction is the quantity to report, as in Table 4.
The exact checks of point 3 remove large errors whatever their sign. Reports with relative error above 0.5 fall from 30.93% to 13.67% after 25% of checks (magnitude 15.62%, random 23.20%) and to 5.27% after 50% (8.18%, 15.47%).
5. Early stopping
We measured this. Once the cost of computing the score is counted, the stopping rule gives no net saving on the three models we timed; the block counts in our first reply did not include that cost. The stopping study was not in the submission; the first revision added it as Appendix H, and we have now removed it. None of the paper's claims rests on it, and the paper's practical uses are the two in points 2–4: deciding which estimates to report and which to verify.
Response to Reviewer 8q8A
Thank you for the review. The abstract and introduction now open with the two questions the paper asks and a concrete case, each definition follows a plain-language statement of what it measures, and Figure 1 is redrawn; the general response lists the changes. Numbers below refer to the revised PDF. Brackets are paired 95% intervals over prompts.
W1. What the paper evaluates
FSR does not produce a reading of the model. It estimates how far an existing reading can be trusted. The paper applies it to two kinds of reading:
Q1. How do interventions relate to readout accuracy?
They are a second application, not extra features for the first. In both, FSR measures a reading's errors on reference prompts and asks how those errors affect the conclusion being drawn: the likely tokens of a distribution, or the token comparison whose change is reported. Exact intervention results are used only for fitting and evaluation, never as inputs when a new estimate is scored.
W1. Downstream use
Deciding which estimated effects to report. When about 10% of estimates are reported, the share that point in the wrong direction (%; Table 4, where FSR is the per-reading score and Gemma uses pre-softcap contrasts):
Deciding which estimates to verify exactly (Table 6, new). Every report is kept, and a fixed number of exact reruns is spent on the reports a score ranks least reliable. Wrong AtP directions that remain among 4,096 reports (%):
At the 25% budget FSR leaves 40 fewer wrong reports than effect magnitude (0.98 points [0.56, 1.39]). Table 6 gives the 25% budget; the 50% column and the no-check rate are secondary results reported in this reply. The comparison uses the 512 Qwen prompts of the reporting study. For lens estimates, and for AtP at a 10% budget, the two choices are comparable.
Early stopping (Appendix H, Table 19, new). A rule lets each position stop at the first of eight depths where the score predicts the lens distribution is close enough to the final one; thresholds are set on separate prompts. With accepted readings held to a mean KL of 0.5, it stops 66% of positions early and avoids 11.8% of decoder blocks, averaged over the five models, against 10.2% for an entropy rule (1.62 points [1.46, 1.79]) and 11.0% for the same score without the covariance statistic (0.80 points [0.74, 0.86]). FSR is ahead of both at all five targets. These are block counts on fixed text, not a measured generation speedup.
Answer correctness. Both applications measure agreement with the model's own behaviour, not whether its answer is factually correct (Section 5).
W1. Is the score faithful to the model's computation?
FSR makes no claim about what the model is "thinking" at a layer. It predicts quantities that are measured on the model itself: the distance between the lens distribution and the actual final distribution, and the direction of the exact intervention effect. Every result checks these predictions on prompts that were not used for fitting.
The predictor is small, with four inputs, and the gain from the covariance statistic persists in matched-predictor comparisons, in all ten model–layer settings each time:
Q2. Why 75% and 87.5% depth?
One rule was fixed in advance for all five models: read after 75% and 87.5% of the blocks. These are late readings that still leave about an eighth to a quarter of the network to run, and a shared rule avoids picking the best layer for each model (Section 4.1). The result is not specific to these depths: at about 25% and 50% depth the additive score improves error ranking in all ten settings, by 0.7–23.6% (Table 16). The intervention experiments edit the same two layers in the Qwen models.