acceptodds
Under review as a conference paper at ICLR 2027

Interventional Commutativity: Effect Preservation and Reference Selection Under Weight-Only Quantisation

Abstract

Can an intervention discovered before quantisation be reused after deployment? This question has two targets: preserving its original effect and matching an intervention re-discovered on the deployed model. We formalise the latter as interventional commutativity and audit both on behaviourally matched inputs. A round-to-nearest construction proves that exact functional equivalence can coexist with complete source-effect loss; an effect-mass bound separates mean sign agreement from magnitude retention. Empirically, a corrected 504-cell FP16/INT8 audit finds median native-relative retention 0.999 across 245 informative INT8 cells, with eight severe gaps in four configurations. However, the 243 cells whose source-effect magnitudes exceed the same descriptive guard have source-fidelity point estimates between 0.792 and 1.186. Crossed execution explains the distinction: on 117 matched template-held-out Qwen prompts, 4.481 logits of a 4.497-logit native-reference gap already exist before quantisation; only 0.016 comes from different responses to deployment. Thus a large native gap can diagnose reference selection while the transported effect's point estimate stays near its source value. Control-calibrated floors, paired gaps and cluster uncertainty make the comparison measurable; sequence checks show that its verdict remains endpoint-specific. The resulting audit quantifies preservation against the chosen reference and clarifies the target of repair.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, post-training quantization, model compression, causal interventions, activation patching, steering vectors, circuit ablation, intervention transfer, evaluation methodology, reliability of explanations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CsYuOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether interpretability interventions (circuit ablation, activation patching, steering vectors have been mentioned in the paper) discovered on a development model remain valid when reused unchanged on a weight-quantised version of that model. It distinguishes two targets: source fidelity, phi = u/d, the transported intervention's effect on the quantised model relative to its original effect; and native parity, rho = u/v, the transported effect relative to that of an intervention re-discovered on the quantised model, which the authors call "interventional commutativity". Both are measured only on inputs where the two models' unintervened outputs agree. To diagnose disagreements, the paper proposes crossed execution: running the re-discovered intervention back on the development model, which splits the gap into a part that already exists before quantisation and a part caused by the two interventions responding differently to quantisation. On the theory side, a round-to-nearest construction shows that exactly equivalent behaviour can coexist with complete loss of a transported intervention's effect, and an effect-mass bound separates preservation of the mean effect's sign from preservation of its magnitude.

Validation centres on an FP16/INT8 audit that crosses seven small base models (up to 1.5B parameters), four tasks, three intervention types and three seeds, using effect floors calibrated on FP16 controls and prompt-clustered uncertainty. Native parity is close to one in the vast majority of cells, with a small number of severe gaps, mostly in an assistant-prefix ablation stress task, while source fidelity remains largely preserved. A held-out study on one Qwen configuration uses crossed execution to show that nearly all of its native gap already exists before quantisation. Further extensions cover additional quantisation and pruning schemes, instruction-tuned models, official GPTQ/AWQ checkpoints, sequence-level refusal checks, disjoint discovery/evaluation splits on GPT-2, and a small gate-based repair pilot.

Critical strengths

The main strength is the distinction between keeping an intervention's original effect and matching what re-discovery finds on the quantised model. The two sound similar but they are not the same, and crossed execution (running the re-discovered intervention back on the original model) is a simple, clean way to tell them apart. I think the lesson reaches beyond quantisation: whenever people compare interventions across model versions, a large gap to a re-discovered intervention does not necessarily mean the original one broke.

The paper's evaluation framework is also rigorous. The authors calibrate against FP16 controls, report uncertainty at the prompt level, re-ran the whole main audit after finding a bug, and released the raw records and code needed to regenerate the main results. Because of this I trust the numbers. My concerns are about what the numbers mean, not whether they are right.

Critical weaknesses

As mentioned in the Summary part, the main weakness is that the theory, the experiments and the conclusions point in different directions. Proposition 1 and the title are about quantisation breaking interventions, but the INT8 experiments show the opposite: the original effects are mostly preserved. The handful of cases with very low rho happen because the re-discovered intervention is very different from the original one, usually much stronger, not because the original effect was lost. In the case the authors look at closely, their own crossed execution shows that the gap was already there before quantisation. So what the paper really finds is that the discovery procedure is unstable. Switching the compute backend, or fixing a single implementation detail, is enough to pick a quite different intervention. That is interesting in its own right, but the paper does not try to explain it. In the end, the experiments do not answer the question the paper asks, and the question they do raise is left open. This is the main reason for my recommendation.

A second weakness is that the evidence is thinnest exactly where problems are most likely. INT8 with group-wise rounding is close to lossless, so it is not surprising that the original effects survive. The settings practitioners actually worry about are 4-bit quantisation and larger models, and there the paper has only a few results, some of them from before the bug fix. Even accepting the paper's framing, it cannot tell a practitioner whether their interventions will still work in the settings they actually deploy.

Questions for authors

1, How do the experiments answer the question the paper poses? As I mentioned that the biggest problem is the misalignment between the theory, experiments, and conclusions. The experiments mostly show that the original effects are preserved, and in the one configuration analysed with crossed execution, the gap to the re-discovered intervention already exists before quantisation. Could the authors state explicitly what rho is meant to measure: the effect of quantisation on an intervention, or the stability of the discovery procedure? If it is the former, could they apply crossed execution to all of the severe cases, not only the one examined in detail, and report how much of each gap is caused by quantisation rather than already present on the development model? If a substantial part of these gaps turns out to come from quantisation, the paper's framing would be much better supported and I would reconsider my assessment. If not, the paper's main message is about discovery stability rather than quantisation, and the framing should change accordingly.

2, This one should be considered as a kind of suggestion for future work. The results suggest that the instability of the discovery procedure may be a more fundamental finding than the quantisation question itself. Have the authors considered developing it as a study in its own right? Possible directions include: treating quantisation as one of several small perturbations (compute backend, numerical precision, batching, small weight noise) and measuring how often the selected sites change under each; relating this instability to how close the competing scores are, and to whether exact or approximate scoring is used; explaining why swapping a single site can change the joint effect so much, for example through redundancy or non-additive interactions between components; and testing whether discovery procedures that aggregate over such perturbations yield more reliable interventions. I would be interested to hear whether the authors plan to pursue this direction, either by reframing the current paper around it or as separate work.

2 comments

We agree with your central distinction: a low ratio to re-discovery is not itself evidence that quantisation broke the original intervention. Rho measures mean-effect parity between the reused and native interventions on the compressed model; it is neither a pure quantisation-effect measure nor a calibrated measure of discovery instability. Source fidelity measures the original-effect comparison, and crossed execution separates reference selection from execution changes (Section 3, pp. 3–4; Equation 4 (p. 3)).

1. Alignment between the question, experiments and conclusion. We follow the branch you specify: “If not, the paper's main message is about discovery stability rather than quantisation, and the framing should change accordingly.” The revised framing is an audit of effect preservation and reference selection, not an attribution of the eight low-native-parity cells to quantisation-induced source loss.

The additional T4 crossing covers retained site choices in seven of the eight historical severe seed keys, across three of four configurations. It refits each intervention's means on its own model and recomputes matching before fixing both operators. The signed decomposition is:

Configuration (runs)Total gap v-uReference term b-dSource change u-dNative change v-bInteraction
Qwen-0.5B prefix (3)+4.502+4.482-0.043-0.023+0.020
Qwen-1.5B prefix (3)+4.406+4.453+0.516+0.469-0.047
Qwen-1.5B Greater-Than (1)+0.765+0.770+0.047+0.042-0.005

All terms are in logits. These original-model terms account for 99.6–101.1% of the signed reference gaps; values above 100% reflect cancellation by the execution interaction. We also retain the individual changes: in Qwen-1.5B/prefix they are +0.516 for the source intervention and +0.469 for the native intervention, despite an interaction of -0.047. A small interaction therefore does not show that either effect was unchanged. Source means in two T4 configurations fall below the historical magnitude guard; the Llama steering configuration has no completed additional crossing. These are crossings of retained sites with refitted payloads, not replays of all historical operators (Table 4 (p. 7); Table 42 (p. 41); Appendix N.1 (p. 40)).

The paired NF4-grid capture answers a different part of the question: legacy scoring gives a resolved negative source mean and a resolved positive signed shift toward zero. Equal-example scoring changes the selected interventions, sharply narrows their reference gap and leaves the new source denominator unresolved. The paper reports both findings, rather than treating the low legacy point or high equal-example point as protocol-independent preservation evidence (Section 7.1 (p. 9); Table 45 (p. 44); Table 46 (p. 45)). The valid dense-7B comparison and matched-protocol discovery-null calibration remain unmeasured.

2. Discovery stability as a research direction. The revision brings the existing controlled diagnostics into the argument. It documents exact ties at the selection boundary, distinguishes exact single-site scoring from first-order scoring, and tests the joint interventions rather than assuming additivity. In the targeted held-out Qwen experiment, replacing site 22 with 21 changes the joint effect from 0.426 to 4.388 logits; the two-swap interaction is 0.787 [0.770, 0.802] logits. The equal-example follow-ups isolate partial-minibatch gradient weighting from effect averaging. These observations identify concrete selection sensitivities (Table 25 (p. 27); Appendix L.2 (p. 35); Table 38 (p. 37); Table 45 (p. 44)).

They do not yet establish a perturbation-aggregated discovery method, a factor-separated null for all flagged configurations or a broad repair advantage. The gate pilot retains its negative source-target result and its dominant mean-offset component. We treat systematic perturbation-robust discovery as the future direction you suggest, while the present contribution is the explicit effect reference, four-execution diagnosis and evidence-bound audit protocol (Table 35 (p. 36); Table 37 (p. 37); Section 8, p. 10).

Thank you for the additional experiments and clarifications. But I regret to say that I will maintain my original assessment. The additional evidence further reinforces my concern that the paper's experimental findings and contributions do not adequately support its original claims of significance. In my view, the current work is better considered as an exploratory or diagnostic study, rather than a paper claiming substantial methodological or theoretical contributions. Therefore I will hold the same judgement, and I believe this work might be more suitable for another venue with a different scope and expectations.

Best and good luck.

Reviewer_idScOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work have formulated interventional commutativity and audit both on behaviourally matched inputs.

Critical strengths

(Still trying to find..)

Critical weaknesses

Honestly, I found it difficult to follow the paper. I actually spent about an hour going through this paper, but I cannot understand what this paper aims to solve or what the motivation is, or what the takeaway.. (Also, when I carefully read this paper, I feel that many of the sentences are AI-written.. which is looks fancy at first glance but just cannot understand deeply)

The opening sentence of the abstract is quite confusing. Does this paper focus on language models? What is meant by "intervention"? Furthermore, why is this question important? While I am not an expert in this specific area, the paper is generally difficult to read. I encourage the authors to clearly articulate the following points for a broader audience:

  • Problem statement: What is the core problem being addressed? (one sentence)
  • Motivation: Why is this problem important? (one sentence)
  • Prior work: What prior approaches have attempted to tackle this issue? (one sentence)
  • Proposed solution & novelty: What is the proposed solution, and what makes it novel? (one sentence)
  • Takeaway / Punchline: What is the main message or contribution to the community? (one sentence)
Questions for authors

Please check the weakness.

1 comment

Thank you for specifying the information needed to make the paper understandable. We revised the opening to define an intervention as a change to a language model's hidden activations, then explain why its effect needs checking separately from ordinary predictions. Here are the five requested sentences, matching the Section 1 overview (p. 1, with the bibliography citation omitted here):

Problem statement: How can we check whether an intervention that changes a language model's hidden states retains its intended effect after model compression?

Motivation: Explanations and steering controls depend on hidden-state intervention effects, which ordinary prediction accuracy does not measure.

Prior work: Circuit discovery, activation patching and steering construct interventions, while representation alignment compares model internals.

Proposed solution and novelty: We compare the reused intervention's effect with its original effect and a re-discovered reference, then execute both fixed interventions on both models to separate reference selection from execution changes.

Takeaway: A large gap to re-discovery can reflect a different selected reference rather than loss of the original effect, so an audit must specify which effect it aims to preserve.

The next paragraph defines ablation, patching and steering. The worked example then distinguishes an estimated 89% of the original mean effect from 17% of the re-discovered reference effect (Section 1, pp. 1–2). Figure 1 (p. 2) summarises the two comparisons; Equation 4 (p. 3) gives the four-execution matrix before the formal diagnosis. The same distinction now organises the results and conclusion. The AI-use disclosure remains in the paper.

Reviewer_bwJtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author asks whether an interpretability intervention found before quantisation (a component ablation, an activation patch, or a steering vector) can be reused on the quantised model. Source fidelity asks whether the transported intervention keeps its original effect. Native parity, called commutativity, asks whether it matches an intervention re-discovered on the quantised model. The authors give an almost-near construction where exact functional equivalence still destroys a transported ablation, and a bound relating mean sign agreement to effect magnitude. Their main experiment is a 504-cell FP16/INT8 audit on seven small models (0.1B to 1.5B). It finds median native-relative retention of 0.999 and 8 severe cells in 4 configurations, while source fidelity stays between about 0.79 and 1.19. The crossed-execution study the author ran on Qwen-0.5B shows that a 4.481-of-4.497 logit gap already exists on the development model before quantisation. Further experiments cover instruct models, official GPTQ/AWQ checkpoints, 48-token refusal outcomes, and a small repair pilot.

Critical strengths
  • Separating source fidelity from native parity, plus the crossed-execution decomposition, is simple and useful. It shows that a low native-relative ratio can come entirely from which intervention the explainer picks, not from the transported intervention changing. This is a good lesson for anyone validating circuits or steering vectors throughout checkpoints.
  • The empirical work is very careful and honest. They report a bug fix which reduced the number of severe cells from 14 to 8. They use control based noise floors, prompt clustered intervals and leave out calibration checks and they make it clear what each of the results does not establish. The negative and rather unflattering results are all in the paper (failed alignment maps, failed upstream predictor, repair pilot which only fixes an offset).
Critical weaknesses
  • The paper’s main finding is about the reference intervention, not about quantisation, but the title and framing read like a deployment result. The crossed-execution study shows that almost the whole gap (4.481 of 4.497 logits) is already there on the development model, before any quantisation. The authors also do not measure how much the native reference itself varies. They never repeat the discovery on the same unquantized model with a different seed, batch order or tie-breaking and see how far two native runs differ. The FP16 control on Qwen-0.5B at k = 12 already gives a gap of 1.555 logits, and 3 of the 7 severe attribution cells have exact ties at the top-k boundary. Because of this, we cannot tell how much of the 8/245 severe tail is really due to quantisation and how much is just normal variation in discovery.
  • The evidence is also narrow for the question. Six of the 8 severe cells come from two prefix-ablation configurations that use the same eight prompts across three seeds. The main audit is only on models up to 1.5B with simulated INT8. The INT4, NF4 and pruning results are exploratory, have low coverage after matching, and were run before the bug fix that changed the INT8 severe count from 14 to 8. Source fidelity near one is only a point estimate, and for Qwen-0.5B the interval goes down to about 0.49. Proposition 1 only shows that such a failure is possible, and the authors do not give a verifiable prediction from it for the failures actually seen.
Questions for authors
  1. If you do native re-discovery on the unquantised model with different seeds, different batch orders and different tie-breaks, how big is the gap between two native runs? How many of the 8 severe cells are actually larger than this gap?
  2. Is there any cell where the transported intervention loses its source effect because of quantisation, that is, low source fidelity with a clearly resolved denominator?
  3. Does the result hold for a 7B or larger model with a real 4-bit pipeline and more than eight prompts?
2 comments

Thank you for separating reference-selection variability from preservation of the source effect. We address your three questions directly.

  1. How large is unchanged-model discovery variation, and how many of the eight severe cells exceed it?

The existing frozen-model diagnostic makes six discoveries per Qwen configuration and executes each fitted intervention on the same 128 evaluation rows. The six-run spans are 0.861 logits for Qwen-0.5B/Greater-Than, 0.492 for Qwen-0.5B/prefix, 0.790 for Qwen-1.5B/Greater-Than and 4.621 for Qwen-1.5B/prefix. Identical-input discovery repeats give exactly equal effect arrays in all four configurations. However, the perturbed discoveries jointly vary overlapping 64-row subsets and exact-tie rules; prefix fitting and evaluation share eight clean prompts. This is combined protocol sensitivity, not the factor-separated, historical-protocol null in your question. We do not have a calibrated exceedance count for the eight historical cells. The revision states this gap rather than using the observed spans as a null threshold (Table 5 (p. 8); Appendix N.2 (p. 41)).

The eight flags remain within-grid observations, not a deployment failure rate: six are repeated seed cells in two prefix configurations, and two are Greater-Than cases. Source-fidelity point estimates in the 243 INT8 cells also passing the source guard remain 0.792–1.186, but the Qwen-0.5B prefix interval is approximately [0.487, 0.985]. We keep point estimates, denominator guards and confidence intervals distinct (Table 2 (p. 6); Appendix K (p. 33)).

  1. Is there source-effect loss with a clearly resolved denominator?

Under the captured legacy-scoring NF4 numerical-grid protocol, the original mean is resolved negative and the paired source-effect shift toward zero is resolved positive. The original-model Qwen-1.5B/Greater-Than capture uses 107 matched rows and 91 clean-identity clusters, with 2,000 paired cluster draws retaining repeated rows:

Original effect: d = -0.6370, 95% interval [-1.1722, -0.0976]. Reused effect: u = -0.0662, interval [-0.5338, 0.3969]. Paired change: u-d = +0.5708, interval [0.4347, 0.7218] logits. Source-fidelity point: 0.1039, with broad signed interval [-2.7369, 0.4853]. Thus the source denominator interval excludes zero, but 16 of the 2,000 denominator draws are positive and the ratio remains broad; we do not convert the point into a precise 90% attenuation claim. The old 107-row scalar record still has no retrospective paired interval. The new intervals belong to the captured protocol (Section 7.1 (p. 9); Table 45 (p. 44); Table 46 (p. 45)).

The controlled equal-example scoring setting retains the same support and canonical mean payloads but changes one source site and one native site. Its original mean is +0.4033 [-0.1347, 0.9140], so its 2.2252 source-fidelity point is unstable. Its native-reference gap is 0.2114 rather than 1.1847 logits, with paired reduction 0.9733 [0.5561, 1.3930], mainly through the original-model reference term. This distinguishes a signed execution change under legacy scoring from sensitivity to which intervention is discovered. Both settings fit and evaluate on the same support, retain disabled matching-projection checks for the predeclared anchor, and use quantise–dequantise NF4-grid weights rather than packed NF4 inference (Appendix P (p. 44)).

  1. Does the result extend to more than eight prompts and a real 7B 4-bit pipeline?

Fresh discovery on the 96-prompt authored pool gives native parity 1.155 and 0.805 for the two Qwen models, with 89/96 and 26/96 matched prompts. It changes the fitting pool, backend, batch size and discovery weights; it is not a prompt-count-only test or a replay of the eight-prompt operators (Table 3 (p. 6); Appendix N.3 (p. 41)). Separately, frozen historical site choices with CPU-refitted means retain low native parity on 88 new-instance and 117 new-template prompts; these held-out results evaluate the fixed site choices (Section 6.1 (p. 6); Table 33 (p. 35)).

A valid dense-7B-to-packed-4-bit effect comparison has not completed. We exclude the NF4-initialised 7B engineering ratios because the generic adapter treated packed integer storage as ordinary weights. Official 0.5B/1.5B GPTQ/AWQ checkpoints remain a separate, completed comparison, not evidence at 7B (Appendix N.5 (p. 42); Table 18 (p. 24)). The revision also reports the mask-corrected 60-cell Qwen low-bit subset separately from the earlier breadth screen (Table 6 (p. 10); Appendix N.4 (p. 42)).

Finally, Proposition 1 is a possibility result, not a predictor of neural failure sites or their prevalence. Appendix O (p. 43) supplies a worked instance and 24 tolerance-specified numerical settings; the language-model diagnosis comes from measured crossed effects.

Thanks very much to the authors for the detailed rebuttal and the revisions. The revised framing and the additional analyses really do improve the paper and help to clarify the difference between reference selection and source-effect preservation.

I'm still not convinced about the absence of a calibrated discovery-variability baseline and the lack of validation under realistic low-bit quantisation conditions. That's important for the wider empirical significance of the work. I do see the improvements but am holding to my original score of 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for identifying the mismatch between a gap to re-discovery and loss of the original intervention's effect. We agree that a low native-parity ratio alone cannot establish quantisation-induced source-effect loss. The revised paper, Interventional Commutativity: Effect Preservation and Reference Selection, centres on an explicit audit target: preserving the original effect is different from matching the effect of a newly selected reference.

1. What the comparisons measure (bwJt, CsYu). Source fidelity compares the reused effect with its original effect; native parity compares it with re-discovery on the compressed model. Executing both fixed interventions on both models identifies the reference gap already present on the original model and the difference between their execution changes. In the 117-prompt template-held-out Qwen study, the original-model term is 4.481 of the 4.497-logit gap, while the source intervention's own change is -0.045 [-0.077, -0.014] logits. The paper retains that nonzero change and the unresolved source-fidelity interval. Paper change: the opening explanation and worked example in Section 1 (pp. 1–2), Equation 4 (p. 3), Table 34 (p. 36) and Appendix L.1 (p. 35).

2. A resolved signed source-effect change under a specified low-bit protocol (bwJt, CsYu). The paired original-model Qwen-1.5B Greater-Than NF4-grid capture supplies two completed scoring settings and four actual discoveries on the same 107 rows, clustered by 91 clean identities. Under legacy scoring, the source mean is -0.6370 [-1.1722, -0.0976], and the paired execution change is +0.5708 [0.4347, 0.7218] logits, toward zero. The reused-effect interval crosses zero, and the source-fidelity interval is broad: [-2.7369, 0.4853]. Equal-example scoring changes one of 12 selected sites on each side and reduces the native-reference gap by 0.9733 [0.5561, 1.3930] logits; its source denominator crosses zero, so its 2.2252 ratio is not stable amplification evidence. These are conditional fixed-discovery intervals, not an unchanged-model null or packed NF4 results. Both captures retain the predeclared disabled matching-projection checks. Paper change: Section 7.1 (p. 9), Appendix P (p. 44), Table 45 (p. 44) and Table 46 (p. 45).

3. Discovery sensitivity, corrected coverage and the remaining gaps (bwJt, CsYu). The primary audit remains 252 FP16 controls and 252 INT8 cells: 245 native-guarded INT8 cells, 243 also passing the source guard, and eight low-native-parity cells in four configurations. Six flags come from two eight-prompt prefix-ablation settings. The additional T4 crossing covers seven historical seed keys in three configurations, retaining sites but refitting means. Frozen-model discovery perturbations span 0.492 and 4.621 logits in the two prefix settings, but combine overlapping discovery subsets with tie rules; they do not calibrate an exceedance count for the eight flags. Fresh 96-prompt discovery fits different operators and retains 89/96 and 26/96 prompts. The corrected low-bit subset has 24 support-passing, 17 native-guarded, 14 source-guarded and 12 both-guarded cells, rather than replacing the full exploratory breadth screen. A matched-protocol discovery null, the additional Llama crossing and a valid dense-7B-to-packed-4-bit comparison remain unmeasured. Paper change: Table 2 (p. 6), Table 3 (p. 6), Table 4 (p. 7), Table 5 (p. 8), Table 6 (p. 10) and Appendix M (p. 39).

4. Accessibility (idSc). The abstract now defines a language-model intervention in its first sentence. Section 1 provides the requested five sentences, followed by the concrete Qwen example; the framework introduces the four executions before the formal results. We also revised the main-body and appendix prose to give each experiment a clear evidential role. Paper change: abstract and Section 1 (pp. 1–2), Figure 1 (p. 2) and Section 3 (pp. 3–4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.