acceptodds
Under review as a conference paper at ICLR 2027

When Are Local Linear Models Faithful to Large Language Model Behavior?

Abstract

Local surrogate explanations are used to summarize how black-box models respond to interpretable changes in their inputs. Their fidelity is usually assessed by how well a fitted surrogate predicts the observed responses. Prediction error alone does not distinguish parameter mismatch from model-class limits. We give exact limits for local affine explanations and separate parameter mismatch from the minimum error on the same measured responses. We change one aspect of a request at a time in two opposite ways while keeping the others fixed. For each pair, an affine model can fit the difference between the two responses using its slope, but the average of the two predictions is fixed by the same shared intercept. Therefore, if different pairs have different observed averages, no choice of affine parameters can fit them all exactly. We show that the variance of these averages is the minimum mean-squared error over all affine fits, while their smallest enclosing radius is the minimum error required to fit every measured response. The remaining mean-squared error of a fitted model separates exactly into intercept and slope mismatch. Two response pairs also give a four-response lower bound on the minimum worst-case error. Across 22 language models and eight task-specific neighborhoods, every evaluable setting exceeds the specified affine tolerance under two scorers, including most cases with full-rank directional responses. Across 155 evaluations, the four-response bound reaches a median 96.3% of the exact minimum.

Primary area:
interpretability and explainable AI
Keywords:
large language models, interpretability, local surrogate models, explanation fidelity, approximation error, error decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_suo3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the fidelity of local affine surrogates, y^(u)=b+Bu\hat y(u) = b + Bu, as explanations of LLM behavior. The design measures responses at symmetric endpoints ±hkek\pm h_k e_k along dd independent intervention directions. Empirically, the authors build eight benign task neighborhoods adapted from XSTest. Each has five prompt-level factors (authorization, consequence severity, ambiguity, evidence requirements, instruction conflict) at five levels. They collect responses from 22 open-weight LLMs and score caution and epistemic uncertainty with two fine-tuned reward-model scorers. Anchored and OLS affine surrogates underperform a constant training-mean baseline.

Critical strengths
  1. Clean separation of error sources. Distinguishing model-class limits from parameter mismatch is a useful framing for evaluating surrogate explanations. The exact three-term decomposition (Eq. 3) is easy to interpret.
  2. Concrete case study. The baseball certificate in Section 4.6 and Appendix C.5 includes full prompts and responses.
  3. Four-point certificate gives a human-inspectable witness of infidelity. Corollary 2 makes a useful practical point: a well-conditioned slope matrix does not certify fidelity.
Critical weaknesses
  1. The experiments cannot separate model nonlinearity from measurement artifacts. The headline result is that every setting fails, but factors unrelated to LLM nonlinearity can inflate midpoint dispersion: a. No noise floor. Each prompt is generated once (in B.2 "Most models use greedy decoding"), and the paper explicitly does not estimate variability over repeated generations. Paraphrase sensitivity is also not measured. b. No positive control. The certificate is never shown to accept affine fidelity under realistic noise. A test that rejects in 155/155 cases needs evidence that it can pass. Examples would be simulated exactly affine responses with empirically estimated noise, or near-trivial interventions in real models.

  2. Coordinates are assigned, not measured, and ± symmetry is not verified. The coordinate values are labels attached to discrete textual edits, and the paper states they do not imply a continuous semantic metric. The entire theory depends on the + and − edits being equidistant from the center in effect. If one edit is perceptually stronger than its opposite, a perfectly linear responder still shows a midpoint shift, which the paper would attribute to the affine class. No manipulation check verifies effect symmetry, for example through human ratings or a perception probe.

  3. Lack of verification that target models registered the interventions. The rubric scores expressed caution and ignored correctness, so score shifts may reflect misreading or non-compliance rather than sensitivity to the intended factor. Compliance rates per model would strengthen the interpretation.

  4. Explanation methods used in practice are not evaluated. The paper positions itself relative to LIME, MAPLE, and especially LAMP (Chen et al., 2026). LAMP fits local linear surrogates on LLM-stated factors and reports reasonably good linear fits. However, the experiments evaluate only the authors' own anchored and OLS surrogates on a custom symmetric design.

  5. The title question is only partly answered. The paper gives a necessary condition on measured endpoints (R⋆≤ϵR^\star \le \epsilon) and a failure certificate. Since no setting passes, there is no positive characterization of when local linear models are faithful. The conclusion also depends heavily on the tolerance (The value 0.250.25).

Questions for authors

What is R⋆R^\star under (a) repeated sampling at temperature > 0, (b) paraphrased center prompts used as placebo ± directions, and (c) repeated or alternative scoring? How does this noise floor compare with 0.25s0.25s?

Can you show a positive control, with synthetic affine responses plus empirically estimated noise, or a real low-perturbation setting, where the certificate accepts affine fidelity?

1 comment

Thank you for the detailed review. We added repeated-generation and rewording controls, compatibility checks, and an application to LIME to examine your central concern: what an exclusion tells us about model behavior. The anonymous repo has also been updated.

Noise floor. (a) All 22 models answered every r4r_4 endpoint and base request in eight rounds at temperature 0.7 (Appendix C.23). All 201 complete, evaluable single-round tables exceed 0.25s0.25s; they are repeated measurements sharing models and prompts, not independent trials. As a no-edit control, ten responses to the unchanged base request placed at the endpoint coordinates also exceed 0.25s0.25s in all 27 evaluable settings (0.46s0.46s–1.29s1.29s). A single sampled table therefore cannot separate the edits from sampling variation. (b) Two raters judged that ten rewordings of the baseball base request keep its requirements (60/60). On greedy responses of all 22 models, any coordinate-only predictor has RMS error at least 0.72s0.72s on one of them (Appendix C.25); used as placebo ±\pm directions for four models, they exceed the tolerance in every group (Appendix C.24). (c) Two scoring runs agree exactly (Appendix C.20); changing only the scorer's prompt moves the rewording radii by at most 0.033s40.033s_4 (Appendix C.24). Even greedy responses changed across new engine processes for three of four models. Averaged over 32 sampled rounds in new processes, four models' endpoint mean tables are within tolerance for uncertainty and above it for caution. The four prespecified e-value tests leave open whether the mean response admits an affine fit (Appendix C.24). We therefore report the exclusions as exact limits of the recorded tables for this coordinate representation, not as evidence of a nonlinear response to the intended edits.

Positive control. By Theorem 1, an affine fit within ϵ\epsilon exists exactly when R⋆≤ϵR^\star \le \epsilon, and the proof gives the fit, so the criterion can accept. Constructed tables with known affine signals and perturbations up to half the tolerance are accepted, while opposed midpoint shifts of twice the tolerance are rejected (Appendix C.18). Measured tables are also accepted: both 32-round uncertainty endpoint mean tables (Appendix C.24), the eight-round mean table for Witcher Nemotron uncertainty (0.242s0.242s; Appendix C.23), 10 of 16 word-deletion slice tables at tolerance 0.05 (Appendix C.26), and 25 of 31 r4r_4 settings at 0.75s0.75s (Table 3). These are finite observed cases, not calibrated noise controls. In a further control, adding observed differences between sampled response tables to a known affine signal exceeds the tolerance in all eight cases (Appendix C.22).

Symmetry. We agree that a perceived asymmetry can create a midpoint shift. Raters found no reversed direction and no change outside the edited aspect in nine r4r_4 pairs, but equal strength in only five and six (Appendix C.16). If each pair's magnitude ratio is shared by the 22 models, exclusion holds in 155/155 settings for ratios in [1/2, 2] and 151/155 for any ratio (§4.5, Appendix C.14). If each model may choose its own ratio, far fewer remain excluded (55/176 Nemotron caution combinations; Appendix C.17). The baseball bound persists in all 16 original and replacement tables, although raters rated some replacements stronger (Appendix C.21).

Registration of interventions. The prompt-preservation review assessed the edits. For generated answers, a literal audit of the instruction-conflict endpoints finds prohibited list markers in 136 of 176 positive-side responses (20 models) and in none of the 176 negative-side responses (Appendix C.19). Most models thus follow the lower-priority list request, so score changes can reflect non-compliance as you note; the audit does not assess semantic correctness or the other four factors.

Explanation methods. We applied the official LIME text explainer at 256 samples (Appendix C.26). M13 has complete fits: held-out MSE exceeds a constant's in three of four channels, and on two fixed slices LIME's largest error exceeds R⋆R^\star in all eight evaluations. M10 has no fit because some scores are missing, so this covers one model and one prompt. Our bound covers every affine map of these coordinates and scores; LAMP uses other coordinates and targets, and we did not replicate LAMP or MAPLE.

Title question and tolerance. On measured endpoints the answer is exact: an affine explanation within ϵ\epsilon exists if and only if R⋆≤ϵR^\star \le \epsilon, so R⋆R^\star is the smallest attainable tolerance. Table 3 varies the tolerance from 0.15s0.15s to 0.75s0.75s, and §5 now explains which failures refitting can fix and which it cannot.

If these revisions address your concerns, we would be grateful if you would consider revising your rating upward.

Reviewer_3ys1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper tries to check why and when a simple linear explanation does not match LLM behavior: is it because the fitted parameters do not work well, or because the linear form itself is too limited? The authors edit prompts and assign numbers to changes such as ambiguity or conflicting instructions. Other trained models then read the answers and give caution and uncertainty scores. The linear rule predicts these scores, not the actual answer text or whether the answer is correct. The main contribution is calculating the smallest error any affine fit could have on the measured scores and separating this unavoidable error from the error caused by the fitted parameters. The evaluation uses 22 models across eight main task context setups and two scorers. The authors fit on smaller edit levels and test on a larger one. Around 80% of the main predictor’s squared error is parameter mismatch, but they also find unavoidable error above their chosen limit at the smaller levels, so the result is not only due to larger test edits

Critical strengths

A useful aspect is that the authors do not simply fit one bad predictor and conclude that linear explanations do not work. They separate error from the fitted parameters from error that no affine fit can remove on the measured scores. This seems to be the main useful contribution because it distinguishes a problem with the particular fitted predictor from a limitation that remains even with better parameters.

The four-response certificate is useful because it provides compact evidence of the limitation. However, it does not mean only four model calls: the authors select four vectors after inspecting ten, and each vector contains scores for 22 models.

Critical weaknesses

The prompt-edit numbers are assigned rather than obtained by measuring how ambiguous the text actually is. Calling two changes +0.8-0.8 does not establish that they are equally opposite in meaning. I therefore remain concerned about how much the result depends on the wording and pairing of the prompts. The authors acknowledge this, but it still matters for interpreting the results.

The scorers learn from model-generated labels rather than human gold labels, and the final number comes from averaging predicted rubric levels. How much of the measured difference reflects the intended behavior, and how much reflects the scorer? Using two scorers is a positive aspect, but I would like to know whether people checking the same answers would agree with their judgments. Repeating the same score does not establish that it is correct. These issues matter because the conclusions about behavior depend on both the prompt representation and the scores assigned to the answers. Would have perfererd even a small eveluation to see how much aligntment there is between the human annotators and the generated ones.

I do not fully understand why 0.25 of the training scale is the appropriate error limit. The authors already examine other limits, so this analysis is not missing. My concern is what amount of score error makes an explanation actually unhelpful: some error does not automatically mean that the explanation is useless.

Also i think illustrative figure on how the data was genertated\tested would have help alot.

Questions for authors
  1. How much do the results depend on the wording and pairing of the prompts? What supports treating the assigned positive and negative changes as equally opposite in meaning?

  2. What evidence shows that people assessing these answers would agree with the caution and uncertainty scores? How should we distinguish differences in the intended behavior from differences introduced by the scorers?

  3. Why is 0.25 of the training scale the appropriate limit for a useful explanation? After a test fails, what should someone change the fitting procedure, the prompt representation, or the explanation model? A concrete example would help me assess the practical contribution.

2 comments

Thank you for the clear summary and for identifying the separation of parameter error from the affine class limit as the main contribution.

Wording and pairing. We agree that assigning +0.8+0.8 and −0.8-0.8 does not show that two edits are equally strong. Two human raters compared nine r4r_4 prompt pairs with the base request (§4.1, Appendix C.16). Neither judged a side opposite to its assigned direction or reported a change in another aspect, but they gave both sides equal coarse strength in only five and six of the nine pairs. The ratings support the directions, not equal magnitudes. We therefore relaxed this assumption: if the two magnitudes of each pair may differ by a ratio shared by all 22 models, the exclusion still holds in all 155 evaluable settings for ratios between 1/2 and 2, and in 151 for any positive ratio (§4.5, Appendix C.14). For wording, we fixed one replacement for each of the four baseball witness prompts before collecting responses. The bound stays above 0.25s0.25s in all 16 original and replacement tables (§4.6, Appendix C.21), although the raters rated some replacements as stronger (three of eight ratings rose), so wording and perceived strength changed together. Wording matters even at a single coordinate: for ten rewordings of the base request that both raters judged to keep its requirements, any predictor that uses only the coordinates has RMS error at least 0.72s0.72s on one of them (Appendix C.25). The revised §4.6 therefore states that a certificate on one wording per prompt does not by itself show a nonlinear response to the intended change.

Agreement with human readers. We added the small evaluation you asked for (§4.1, Appendix C.15). Two raters who were not authors assigned rubric levels to 48 r4r_4 responses without seeing model identities or machine scores. The result is mixed. For uncertainty, the raters agreed on 30 of 32 routine responses, and both scorers are closer to each rater than a constant score of zero. For caution, the raters agreed on only 20 of 32, all at level zero, and against one rater a constant zero is closer than either scorer (MAE 0.070 vs. 0.096 and 0.073). We therefore treat the caution results as findings about the scorers' caution scores. Repeated scoring gives identical results (Appendix C.20), which shows repeatability, not correctness, and recoding the rubric levels by squares or square roots keeps every evaluable r4r_4 exclusion (Appendix C.14). With the responses fixed, changing only the prompt shown to the scorer moves scores by RMS 0.10s0.10s–0.24s0.24s; also replacing the responses moves them by 0.87s0.87s–1.09s1.09s (Appendix C.21). These sizes do not divide a difference into model and scorer causes.

The tolerance and what to change after a failure. We fixed 0.25s0.25s in the collection protocol before analysis; it is not a universal threshold of usefulness. Table 3 reports counts from 0.15s0.15s to 0.75s0.75s, and R⋆R^\star is the smallest tolerance any affine explanation can meet on these endpoints, so readers can apply their own requirement. A new guide in §5 ("Using the diagnostics") links outcomes to actions. If responses at the same coordinates, such as rewordings that keep the request, differ by more than twice the tolerance, no function of the coordinates fits them, so the input representation or the explained quantity must change. If R⋆R^\star exceeds the tolerance, no affine refit can succeed. Otherwise the excess is parameter mismatch, which refitting over the full affine class can reduce on these endpoints. We illustrate with the official LIME text explainer applied to one model's caution score with 256 samples (Appendix C.26). On a fixed word-deletion slice, chosen because it shows all three outcomes, R⋆=0.0333R^\star = 0.0333 and LIME's largest error is 0.07450.0745: at tolerance 0.025 no affine fit exists, at 0.05 one exists but LIME's fit misses it, and at 0.10 LIME's fit meets it. The class optimum uses all 152 word features; LIME selects ten.

Four responses, not four calls. The four vectors are selected after measuring all ten endpoints, each holding 22 models' scores. §5 states that the certificate's size does not specify discovery cost.

Readability. The worked example in §4.6 and Figure 5 follows one certificate from the four baseball prompts through the 22 models' midpoint profiles to the bound; Appendix C.5 gives the complete prompts and responses. The revised organization in §4.3–§4.6 links each check to its diagnostic question.

The anonymous repo has also been updated.

If these revisions address your concerns, we would be grateful if you would consider revising your rating upward.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the robustness analysis of prompt-edit magnitudes, the human evaluation, and the practical LIME example.

These additions improve the paper and clarify its contribution. However, my main concern remains the validity of the behavioral measurements. The limited human agreement on caution scores, including cases where a constant predictor outperforms the learned scorers, raises questions about how well the results reflect actual LLM behavior rather than scorer artifacts. The dependence on the chosen prompt representation also remains a limitation.

While I acknowledge the improvements, I do not think they fully resolve my concerns about the interpretation and practical significance of the findings. I therefore maintain my original score

Reviewer_5Mk8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the fidelity of local affine explanations for LLM behavior. It derives exact minimum affine errors from symmetric response pairs, separates model-class error from intercept and slope mismatch, and proposes a four-response certificate for detecting when no affine surrogate can satisfy a given tolerance.

Critical strengths
  1. The paper provides a clean theoretical analysis that separates irreducible affine error from parameter mismatch.
  2. The empirical study is relatively comprehensive, covering 22 language models, multiple intervention directions and levels, and two behavioral scorers.
Critical weaknesses
  1. The behavioral scores are produced by Nemotron and Skywork, while their reference annotations are model-generated rather than human labels. This makes the quality of the scores unclear and may affect the measured affine error.
  2. The analysis relies on manually assigned symmetric coordinates for semantic changes such as authorization and ambiguity. Since these are discrete text edits rather than a natural continuous scale, the conclusions may depend on how the coordinates are defined.
  3. The main fidelity results aggregate errors across 22 models. It remains unclear how many individual LLMs actually fail the fidelity criterion, since the appendix shows substantial variation in per-model errors.
Questions for authors

See the weaknesses above.

1 comment

Thank you for your review. Your three concerns each identified a check that the submission did not cover. This revision adds them. The anonymous repo has also been updated.

Scorers trained on model-generated labels. The submission stated that the reference labels are model-generated (Appendix B.5) but did not compare scores with human readings. Two human raters assigned rubric levels to 48 recorded r4r_4 responses without seeing model identities or machine scores (§4.1, Appendix C.15). The result differs by attribute. For uncertainty, the raters gave the same level to 30 of 32 routine responses, and both scorers have lower mean absolute error (MAE) than a constant score of zero against each rater. For caution, the raters agreed on only 20 of 32, all at level zero, and against one rater a constant zero (MAE 0.070) is closer than either scorer (0.096 and 0.073). We therefore read the caution results as statements about the scorers' caution scores, not as human-validated measurements (§5). Two further checks: both scoring runs give identical scores and geometry (Appendix C.20), which shows repeatability rather than correctness, and recoding the ordinal levels by squares or square roots leaves every evaluable r4r_4 setting excluded (Appendix C.14).

On how score error affects R⋆R^\star: if a score error is at most η\eta in RMS at every endpoint, R⋆R^\star moves by at most η\eta (Appendix A.5). The smallest margin among the 155 exclusions is about 0.008 in raw RMS units (§4.5). The human comparison does not provide a uniform bound of this kind, so it cannot certify a class limit in human units.

Assigned coordinates for discrete edits. The analysis uses only the measured endpoints and needs no continuous semantic scale between them (§1). Any invertible linear recoding of the coordinates leaves the affine class and its limits unchanged (§2). What the theorem requires is that the two edits of a pair have equal and opposite coordinates, and the revision examines this in three ways. First, two raters compared nine r4r_4 prompt pairs with the base request (§4.1, Appendix C.16): neither judged a side opposite to its assigned direction or reported a change in another aspect, but they gave both sides equal coarse strength in only five and six of the nine pairs. The ratings support the directions, not equal magnitudes. Second, if the two magnitudes may differ by a ratio shared across the 22 models, the exclusion holds in all 155 evaluable settings for ratios between 1/2 and 2, and in 151 for any positive ratio (§4.5, Appendix C.14). Third, we examine distinctions the coordinates do not retain. Two raters judged that ten rewordings of the baseball base request keep its task, facts, and constraints (60 of 60 judgments). All share the coordinate u=0u = 0, so any coordinate-only predictor must give them one prediction; on the 22 models' responses, its RMS error on at least one rewording is at least 0.72s0.72s (§4.6, Appendix C.25). The revised §4.6 therefore states that a certificate on one wording per prompt does not by itself show a nonlinear response to the intended change.

How many individual models fail. Each model's own minimum worst-case error is now reported for all 176 model–neighborhood combinations (§4.3, Appendix C.17, Table 33). Under Nemotron at the neighborhood tolerance, caution exceeds 0.25s0.25s in 112 of 176 combinations covering 21 of 22 models; uncertainty in 111 of 176, covering all 22. Under each model's own training scale, the counts are 99 of 101 and 63 of 66 evaluable combinations; those below the scale threshold are reported separately. For the baseball example, the five-direction optimum excludes 20 of 22 models. Individual failures occur across nearly all models, with variation across neighborhoods.

If these revisions address your concerns, we would be grateful if you would consider revising your rating upward.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.