acceptodds
Under review as a conference paper at ICLR 2027

Surrogate Fidelity: When Can Open LLMs Explain Closed Ones?

Abstract

Mechanistic interpretability (MI) requires full access to model internals, yet the APIs for most widely deployed language models at best expose log-probabilities over output tokens. This creates a surrogate problem: when do measurements made on open models allow us to make claims about a closed model? We evaluate *surrogate fidelity* at the prediction, attribution, and representation levels. For binary classification tasks, log-odds provide an API-compatible scalar readout of the model's representation space, and leave-one-out attributions provide insight into model behavior. Across eleven models spanning four families (Llama, Qwen, GPT, and Gemini), we find that prediction fidelity substantially overstates attribution fidelity: models that agree on what the answer is often disagree on why. We document an access-validity inversion: white-box signals like attention patterns and perturbation magnitudes are highly stable across models but only weakly predictive of causal attributions, which black-box input ablations capture by design. Mechanistic insight does not automatically transfer to closed targets, and prediction-level agreement is insufficient to warrant such transfer. Code and results are available at https://anonymous.4open.science/r/surrogate-40B5.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Benchmarking Interpretability, Surrogate Models, Attribution, Probing, Representational Similarity

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gjfnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

How would we even go about falsifying the common mech interp belief that "if we understand open source LLMs, that will help explain the behavior of closed LLMs too"? This paper notes that so long as a closed API gives access to log-odds, there are two API-compatible ways to operationalize "behavior": 1. The log odds themselves (referred to as "prediction-level") and 2. given some intervention, the projection of the difference between representations (which authors show is equal to the difference between log-odds: called "attribution-level").

The authors then study whether correlation between "mechanistic-level" metrics (e.g. attention patterns etc) of different models means that their "prediction-level" and "attribution-level" behavior also match. They claim that in general they do not: hence, ç

The validation is necessarily restricted to API-compatible interventions, so the authors focus on leave-one-out sentence ablations. These interventions and the corresponding cross-metric relationships are studied across a variety of open source and closed models (although notably not the latest GPT-5 or Claude models since they do not release log-probabilities).

Critical strengths
  1. I particularly appreciated the contribution of the "atttribution level": noting that the difference in log odds from two inputs is equal to the a projection of the difference-vector between the representations themselves. Section 2 and Figure 2 were very clear explainers of this. I was surprised that any representational information could be derived from the difference of log odds (even if it is far short of the information content of the representations themselves).
  2. While I have some hesitations about the negative "mechanistic-to-causal" results, I want to highlight a conclusion that holds regardless, paraphrased from line 428: Most mechanistic-level prediction of attributions are outperformed by API-compatible attribution metrics alone. In my own words: If the point of mech-interp is to understand model behavior, there are better ways to do so that require less model access! Which seems like a potentially large boon for public auditing of closed models.
Critical weaknesses
  1. Attention Fidelity is incorrectly treated as equivalent to Representational Fidelity.

Attention mechanisms alone are linear, and can't encode the nonlinear complexity of the MLP layers in transformers. Given this fact, it's surprising that the negative conclusions in this paper rely so heavily on the over-alignment of attention patterns in Tables 1 and 2! If they instead had chosen to study just the residual stream representations (F_mag and F_align in those tables) which encode the information from both the linear attention heads and the nonlinear MLPs, then the conclusion would not be "mechanistic info doesn't predict attributions" but just the unsurprising "only directional info predicts attributions" (unsurprising due to the magnitude rescaling of LayerNorm).

In short, the more appropriate thesis seems like "We can use surrogate fidelity to assess which mechanistic info is actually predictive of attributions": F_align is the only one which doesn't report an artificially high agreement. Not that "Most representation-level properties do not predict attributions well" (line 428).

Questions for authors
  1. Could you please clarify the so-what takeaways for Figure 3b and Figure 4? Even on reread, it's still quite unclear. These captions may benefit from a one-sentence takeway summary, to guide readers.
3 comments

Thank you for your thoughtful comments. Your alternative thesis is helpful framing, and it's fairly close to where the analyses in the new appendices F and G ended up.

Attribution fidelity is not representation fidelity: We agree that attention pattern agreement does not mean agreement of residual stream representations, and by that metric only FmagF_\text{mag} and FalignF_\text{align} are true representation-level metrics. In our paper, we use "representation level" to mean "requires access to internals," as a distinct rung on the access hierarchy of Figure 1. We do not believe we treat any of our "representation-level" quantities as equivalent and only seek to test whether they (a) transfer across models, or (b) predict attributions.

We are open to suggestions for a more neutral term than "representation" to convey "internals-based."

Captions: We have expanded and clarified the captions for figures 3 and 4 in the revised draft.

Presentation: We have simplified the notation in Section 2, which now uses hh for the residual stream and zz for the normalized last-layer representation, and corrected Equation 2 accordingly.

Thank you for the edits to the paper, they look like a suitable improvement. I'm still a bit confused by Figure 4 but the caption seems better this time around.

How about "Mechanistic"? That would align nicely with your abstract and main-body prose. Or "Attention"?

This is a good suggestion. We have updated the paper to use "mechanistic." Thank you!

Reviewer_1Gq5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests whether open models can explain closed models by comparing their responses to matched input ablations. Across eleven models and four classification benchmarks, prediction agreement generally exceeds attribution agreement. The analysis links this gap to readout alignment and examines whether attention provides useful evidence of explanation transfer.

Critical strengths
  • The paper provides a useful empirical basis for selecting explanatory surrogates. Evidence across model families makes the proposed fidelity measurements relevant to interpreting closed models through open ones.
  • The system-prompt experiments strengthen the central finding by reproducing it under a complementary intervention. This increases confidence in the result and extends its relevance to prompt-dependent behavior.
Critical weaknesses
  • Nonnegative attention may track effect magnitude yet correlate poorly with signed Δℓ\Delta_\ell. Table 1 uses this signed comparison, so low correlation alone cannot rule out useful importance information. A comparison with ∣Δℓ∣|\Delta_\ell | should account for segment length and within-prompt variation.
  • Attribution fidelity is pooled across prompts and segments, leaving its practical meaning unclear. The paper does not show how often models disagree on the most influential evidence for an individual prediction. Given the confidence dependence in Appendix F, agreement on decisive segments would help assess the severity of the reported explanation gap.
Questions for authors
  • Does the prediction–attribution gap persist on prompts that both models answer correctly?
  • On BoolQ, does the gap persist when only passage sentences are ablated, leaving the question and task instructions intact?
3 comments

We thank the reviewer for their thoughtful feedback. Both weaknesses pointed at comparisons we had not made, and which improved our paper and changed howe we read some of our results. Our complete new results can be found in Appendices F and G, and are summarized in Section 5.4.

Attention against unsigned attribution (W1): We compared attention to unsigned attribution in Appendix F.7 and the last 3 rows of Table 4. We found that:

  • Over all segments, mean, max, and rollout attention explains 0.20, 0.12, and 0.04 of ∣ℓ∣|\ell|, respectively.
  • Most of this is because question segments have highest attention scores and ∣ℓ∣|\ell| across the board. On passage sentences only, the fidelities are 0.14, 0.12, and 0.02.
  • Further residualizing passage attention on log segment length gives 0.08, 0.06, and 0.02.

Thus, we conclude that attention carries basically no information about the direction (sign) of a segment's effect, and a very modest amount of information about its magnitude. We have also removed mention of perturbation magnitude from our discussion of the access-validity inversion, as an open model's ∥Δz∥\|\Delta z \| explains 0.43 of another model's ∣Δℓ∣|\Delta \ell| on passage sentences. The Table 1 value (0.11) is depressed by system prompt segments, whose deletion changes the residual stream but not the answer.

Agreement on most influential evidence (W2): Across all pairs, two models choose the same most influential segment 62% of the time, compared to a 14% random baseline. Stratified by confidence, this goes from 34% (bottom quintile) to 78% (top quintile). Oftentimes this is the question segment, which is the target's top segment 61% of the time. Restricting to passage sentences gives us 56% agreement over 27% by chance, nearly flat across confidence quintiles (52% vs 61%).

Our case study in Appendix G takes this further, attempting to replace confidence stratification with a support-based metric instead (how strongly the surrogate's most influential passage favors the surrogate's own answer). This helps us get a lot further towards a nuanced and actionable picture of when closed models can explain open ones, at a saliency level.

Gap when both models answer correctly (Q1): If we restrict our study to both models answering correctly, the gap actually widens: FpredF_\text{pred} is 0.88 against FattrF_\text{attr} of 0.59, with FpredF_\text{pred} higher in all pairs. However, if we further stratify into True and False, the gap actually reverses: FpredF_\text{pred} drops to 0.21 (True) and 0.20 (False), while FattrF_\text{attr} stays at 0.56 (True) and 0.51 (False). Pooling mostly measures agreement on the answer, not something more granular. We have updated our Discussion to be explicit about this.

Gap when only passage sentences are ablated (Q2): Yes, median FattrF_\text{attr} goes from 0.45 to 0.40 over all pairs, and FpredF_\text{pred} is unchanged (no ablations). Passage-only ablations are interesting in their own right, as they avoid the aforementioned "large change in representations, small change in answer" phenomenon inherent to system prompt ablations; they are discussed in detail in Section F.6.

Thanks for the careful follow-up. I found the breakdown by answer label especially helpful. Among the examples both models got right, prediction fidelity is higher in the pooled analysis, but the gap reverses when True and False are analyzed separately. That suggests the pooled score partly reflects how well both models separate the two labels. I’d make this distinction explicit in the main takeaway, since it changes how readers should interpret the gap. I’m keeping my rating at 3 (Weak acceptance).

Thank you for your endorsement. We agree that the paper is improved by the new results. After our initial response, we further revised the paper to integrate the pooled / conditioned fidelity findings more tightly into the abstract, results, and discussion sections. Our latest reply to kVgb contains detailed examples of key edits.

Reviewer_kVgbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper asks when interpretability findings from open models carry over to closed models that only expose log-probabilities. It proposes correlation-based "surrogate fidelity" metrics at increasing depth: prediction (agreement in answer log-odds), attribution (agreement in log-odds changes when a sentence or word is deleted), and, for open models, representation and attention agreement. Across 11 models from four families (0.5B-70B) and four classification benchmarks, models agree on answers more than on which inputs drive them, and attention agrees strongly across models but barely predicts attributions. The authors conclude that surrogate validity must be measured, not assumed.

Critical strengths
  1. Simple, practical, works on closed models. Subtracting log-probs cancels the hidden normaliser, so prediction and attribution agreement can be measured through standard APIs. This gives practitioners a cheap check before using an open model as a stand-in.

  2. Broad, consistent evaluation. 11 models from four families (two closed), 0.5B-70B, four benchmarks. The main pattern holds across all of them (Table 2).

Critical weaknesses
  1. It shows whether models differ, not what differs, and claims more than that. The motivation that MI "implicitly assumes" transfer (lines 38-39) is uncited and doesn't match the field, which is generally open that findings don't generalise: SAEs are trained per model, and even per dataset, because features don't carry over (Sharkey et al., 2025). The metrics measure input saliency and coarse internal signals, not mechanisms, yet the abstract concludes "mechanistic insight does not automatically transfer". Most internal metrics need both models open, so they can't inform open-to-closed transfer (Table 1). The paper never shows an open model explaining a closed one, despite the title. Several findings are also well established (same predictions, different reasons: Fisher et al., 2019; attention is not mechanism: Jain & Wallace, 2019; Elhage et al., 2021). Reframing as an API-compatible screen, and adding a worked case study on the best-matching pair, would help.

  2. Prediction and attribution fidelity aren't directly comparable. Prediction agreement is inflated by both models getting the label right, which cancels out in attribution. So Fpred > Fattr is partly expected. There is no noise ceiling, so it's unclear whether Fattr = 0.40 is low. Please report Fpred within each label class and a noise ceiling for Fattr.

Questions for authors
  1. What is the noise ceiling for Fattr, and Fpred within each label class?
  2. How much do API log-probs vary across repeated calls, and does this lower closed-model Fattr?
  3. Does magnitude agreement hold after controlling for segment length?
  4. How are label variants and labels missing from the top-K handled?
  5. Can Table 2 be broken down by pair type (same-family, cross-family) to show when fidelity is high?

Minor feedback;

  • Llama 3 and Qwen 2.5 use RMSNorm, not LayerNorm. Equation 2 is exact only after the final norm, not in the residual stream.
  • The claim that MI "implicitly assumes" transfer is uncited.
5 comments

We thank you for your thoughtful review. We have run all of the requested analyses, included the results in the new Appendices F and G, and added a summary of the results in Section 5.4. We respond to specific concerns inline:

Noise ceiling (W2 / Q1):

Noise ceiling: To measure the noise ceiling, we re-ran all 11 models 5 times on BoolQ with sentence-level ablations. For open models, all metrics reproduced exactly; this was expected, as there is no sampling in our prefill + single-token approach. For closed models, there was some drift: test-retest Pearson rr over FattrF_\text{attr} ranged from 1.000 (Gemini) to 0.953 (GPT-4o). Full results are available in Section F.3.

Noise floor: We additionally measured a noise floor by shuffling one model's attributions within (role, position, answer) groups, measuring 0.017 on passage sentences and 0.175 on all segments.

Within-label stratification: You are correct that FpredF_\text{pred} and FattrF_\text{attr} are not on a common scale. Conditioning on model answers does a lot: with one gold label, median FpredF_\text{pred} drops from 0.72 to 0.50 (True) and 0.41 (False), while FattrF_\text{attr} stays at 0.43. If we further restrict to prompts both models answer correctly + one label, FpredF_\text{pred} drops further to 0.21 (True) and 0.20 (False), while $F_\text{attr} actually rises to 0.56 (True) and 0.51 (False). Conditioned this way, Fpred>FattrF_\text{pred} > F_\text{attr} in only 5 of 110 (surrogate, target, label) pairs.

We are inclined to keep the pooled comparison as the headline number for FpredF_\text{pred} and FattrF_\text{attr}, since agreement on the answer is part of what prediction fidelity is meant to capture; however, this nuance has been very helpful in formulating our Qwen-GPT case study. We have also updated our Discussion to prominently state that FpredF_\text{pred} agreement is largely driven by answer agreement and call out this ordering-reversal when conditioning on labels.

What the metrics show (W1):

Claim scoping: We agree that our metrics measure agreement in input attributions and coarse internal signals, not mechanistic agreement. We believe we state this explicitly in the paper, e.g.:

151-152: Concretely, attribution measures the causal effect of changing the input on ℓ\ell; it is not a complete internal mechanism.

486-487: leave-one-out ablations are useful for claims about input evidence or saliency, but may not be appropriate for circuit-level claims

Accordingly, we have scoped our case study to input saliency as well. Following this feedback and the new robustness analyses in Appendices F and G, we have softened the language in our discussion of the access-validity inversion, and we are open to further dialogue on how to appropriately scope our discussion of model internals.

Closed models:* You are correct that open-to-closed transfer is not possible in general; we attempt to measure this using the mechanistic-to-causal rows of Table 1, with representation-level transfer as an open-to-open baseline.

An open model explaining a closed one:* In Appendix G, we have added a case study identifying the situations in which Qwen 2.5-14B is able to explain GPT-4o's behavior. In Qwen, we measure how strongly a passage supports Qwen's baseline answer; based on this, we are able to identify subsets of prompts where Qwen is a good surrogate for GPT-4o ablations. This rule transfers to other target models and surrogate models. Interestingly, we reveal that, despite Qwen being a good surrogate for GPT-4o, it is a much weaker explainer on False answers; unsurprisingly, Qwen cannot explain GPT-4o's errors unless it also makes them.

Implicit transfer assumption (W1): We agree that your formulation is more accurate. We have replaced two sections to reflect this: In the introduction, we replace

This assumption that mechanistic insights transfer across models is implicit, pervasive, and under-examined in the literature.

with "Whether mechanistic insights transfer across models is a known open problem (Sharkey 2025) , but it is rarely measured directly. In the contributions, we replace

We argue that the MI community’s implicit assumption that mechanistic findings transfer across models deserves direct scrutiny, and propose surrogate fidelity as a conceptual framework for this dialogue.

with "We argue that the question of whether mechanistic findings transfer across models deserves direct measurement, and propose surrogate fidelity as a conceptual framework for it."

We believe our paper is improved by this narrower, more accurate formulation.

Prior work: we believe our expanded related work section makes it clearer that our goal is not to relitigate established findings, only to treat the rediscovery of these findings by our metric as a sign of life. Please let us know if you think further reformulation is needed.

API call variability (Q2): This is a part of our noise ceiling investigation and can be found in Table 6: prediction log-odds have Pearson r≥0.992r \geq 0.992.

Magnitude agreement and segment length (Q3): Our conclusions hold across both: residualizing on log segment length changes FmagF_\text{mag} from 0.843 --> 0.845, and doesn't move any representation- or cross-level medians by more than 0.02 (see Table 4 and Section F.4).

Interestingly, segment role does matter: shuffling segments with the same role and position takes us down to 0.68 of 0.84, because system prompt ablations tend to have large effects. On passage sentences, FmagF_\text{mag} is 0.60, with a role floor of 0.05 (Section F.3, F.6).

Label variants, missing labels (Q4): Appendix A, sections "log-odds scoring" and "missing values" describe our approach: we aggregate case and prefix variants of each token using log-add-exp, and prompts only get scored when the log-odds are defined for both models. Coverage is generally good (median of 0.95 for open-closed pairs, with the exception of GPT-4.1 which has 0.75). Section F.2 shows our missing value robustness results.

Table 2 by pair type (Q5): We have added this as Table 8 in Appendix F.8.

RMSNorm (minor): We thank you for snapping us out of our bad habit of referring to any normalization generically as "LayerNorm." We have removed all mentions of LayerNorm, opting for either "normalization" (generic) or "RMSNorm" (specific to the models tested in this paper).

If you have any further questions or concerns, please feel free to share them. If we have addressed your concerns, please consider raising your score. Thank you!

Thank you for the substantial revisions and for engaging with my review feedback! The noise ceiling and floor, the API variability and segment-length checks, and the rewording of the transfer assumption address a lot of my concerns. I also think the Qwen 2.5-14B / GPT-4o case study is a particularly good addition that supports the title of the work.

One issue remains, and I'd like to get a more detailed view from the authors. Pooled prediction fidelity is high largely because models give the same answer, and that shared answer cancels out in attribution fidelity. So the pooled comparison doesn't show how differently the models actually behave. In fact, the label-conditioned analysis (where the answer is fixed), where attribution fidelity > prediction fidelity, shows a reversal of the paper's core claimed trend. The pooled view says models agree on the answer but not on why. The answer-controlled view says the opposite, when two models give the same answer, they agree more on why than on how confident they are. Since the two lenses give opposite answers, the paper needs to account for both, or at the very least explain why one view is the better lens for the community to adopt. I would be inclined to update my score if this is addressed in the main text and reflected in the abstract.

Nit: "Mechanistic insight does not automatically transfer" (abstract) should be scoped to input attribution, in line with your response.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General changes to the paper

We thank the reviewers and ACs for their attention. We ran every analysis requested by the reviewers and have updated the manuscript to reflect the new results. This comment lists out the edits we have made to the paper.

New material:

  • Appendix F: Robustness. This covers the statistic and label readout (F.1), missing values (F.2), noise floor and ceiling (F.3), segment length normalization (F.4), results after conditioning on labels and correctness (F.5), passage-only ablations (F.6), the effect of unsigned attribution (F.7), and fidelity by pair type (same family, cross-family, open-closed) (F.8).
  • Apendix G: Case Study. We study the agreement between Qwen 2.5-14B and GPT-4o on a more granular level to understand precisely when Qwen 2.5-14B is a good surrogate for 4o.

Appendix changes:

  • Appendix A, related work --> Section 7 of the new paper
  • Appendix B, additional mathematical details --> inlined into section 2
  • Appendix C, experimental details --> Appendix A
  • Appendix D, additional results --> Appendix B
  • Appendix D.4, metric calibration --> Rolled into F.1
  • Appendix D.6, floor sensitivity --> Rolled into F.2
  • Appendix E, per-layer trajectories --> Appendix C
  • Appendix F, confidence stratification --> Rolled into G
  • Appendix G, representation fidelity decomposition --> Rolled into F.6
  • Appendix H, system prompt perturbation --> D
  • Appendix I, connection to representational similarity --> E
  • Appendix J, statement on AI use --> Moved to correct location (unnumbered subsection before references)

Other edits:

  • Added a discussion of robustness and case study appendices to Section 5.4
  • Restrict access-validity inversion in abstract/intro to attention; softened claims about FattrF_\text{attr} being driven by FalignF_\text{align}
  • Updated notation to use hh for hidden representations and zz for normalized representations

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.