acceptodds
Under review as a conference paper at ICLR 2027

Programmable Semantic Interventions: Learning Causal Carriers from Behavioral Programs

Abstract

Causal abstraction explains a frozen network by aligning its activations with the variables of a simpler, interpretable model, and checking that interventions on the two agree. Finding such an alignment currently requires a counterfactual dataset: for every pair of prompts, the answer the network ought to give once the variable has been changed. Producing those labels means committing to an executable algorithm for the behavior before studying the model supposed to implement it. However, the criterion these methods optimize only leverages the answer an intervention should produce. Here, we developed an approach that lets the investigator define that relationship directly: which answers the variable is about, and how they should move when it changes. We call this interface a *semantic program* and introduce *Programmable Semantic Interventions* (PSI, ): a low-rank activation subspace learned so that interchanging it moves the model's answer the way the program declares. On the MIB benchmark matches or surpasses the quality of supervised alignment, while requiring minimal supervision. The declaration also fixes a coordinate system with known semantics, enabling donor-free steering: jointly training for both localization and control realizes % of held-out requests for answers the investigator chooses, without the need for donor prompts.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, mechanistic interpretability, causal variable localisation, causal abstraction, activation steering, distributed alignment search, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nRKjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Programmable Semantic Interventions (PSI), which replaces the executable high-level causal model and counterfactual labels used in prior causal abstraction methods with a weaker “semantic program”: an outcome library and a transport rule over model-derived output distributions. PSI learns a low-rank activation subspace whose interchange intervention follows this declared transport rule, and further uses the same semantic coordinate system for donor-free control. Experiments on MIB tasks show that PSI is generally competitive with DAS, while requiring less supervision. An ablation further suggests that curated base-source pairing contributes little compared with having access to a larger donor pool.

Critical strengths
  1. The semantic-program interface is a useful simplification.Requiring only an outcome library and transport rule is substantially lighter than specifying a full executable high-level causal model and per-pair counterfactual targets.
  2. The supervision ablation is useful.The result that larger donor coverage can recover most of the fully supervised performance suggests that some of the supervision assumed by prior methods may be unnecessary.
Critical weaknesses
  1. It is unclear whether the weaker specification also weakens the explanatory claim. By replacing a full high-level causal model with an output-level transport rule, PSI may move from identifying a variable within a high-level causal computation toward identifying a subspace that is simply effective at controlling the declared outputs. Since training and evaluation are both defined largely in terms of whether the output follows the declared transport, it is difficult to distinguish a true representation of the intended causal variable from a downstream output-control subspace.

  2. The evaluation mainly uses variables with small and explicit outcome spaces. MCQA answer position, carry, and entity attributes are cases where the semantic program is relatively easy to specify. It is therefore unclear how much specification burden is actually saved for more abstract variables, where defining a clean outcome library and transport rule may itself be difficult.

Questions for authors
  1. What stronger explanatory claim does PSI certify beyond finding a low-rank subspace whose intervention satisfies the declared output transport rule? In particular, how should PSI be distinguished from a semantically conditioned activation-steering method once the full high-level causal model is removed?

  2. Can the learned carrier transfer across different realizations of the same semantic variable without retraining? For example, could an MCQA position carrier learned with A/B/C/D still work with 1/2/3/4 or another answer encoding? Such a result would make the causal-variable interpretation substantially more convincing.

  3. How much of PSI’s advantage over DAS remains when both methods are restricted to exactly the same training observations and donor pool? The ablation suggests that donor-pool size explains a substantial part of the gain. A controlled comparison would help separate the benefit of the semantic-program formulation from the benefit of having access to more donors.

3 comments

We thank the reviewer for the careful review and for three precise questions, each of which we ran as an experiment. The test the reviewer proposed in Q2, transfer across answer encodings, also gives the most direct answer to the main concern (W1), so we discuss the two together.

W1, Q1 and Q2: A causal variable, or a subspace that controls the output?

Q2 gives a direct way to tell the two apart. A subspace that controls the declared outputs would write the donor's answer, while a subspace that carries the variable writes the donor's position in the recipient's own symbols. We froze the MCQA subspaces (Gemma-2 2B, layer 17, rank 16), trained with the benchmark's uppercase option labels, and re-encoded the options of donor and recipient independently as lowercase letters, digits or per-prompt permuted digits, with no retraining (16 combinations, 150 pairs each; §3.1, Appendix A.10, Fig. 6).

The learned subspace writes the position. With an uppercase recipient, it transfers the position from lowercase donors in 100% of pairs and from digit-labelled donors in 92.4%, at least as well as between two prompts with the original labels (86.2%). It never outputs the donor's own symbol: with a lowercase donor and an uppercase recipient, this happens in 0 of 150 pairs. In the 44 test pairs where both the position and the symbols change, the edited model gives the recipient's symbol at the donor's position in 66.7% of pairs, and the donor's literal symbol in 9.1%. Copying the full activation at the same site does carry the token (with a letter donor and a digit recipient, it outputs the donor's letter in 44% of pairs), so the subspace selects the position from an activation that also holds the answer.

The one encoding that does not transfer well is a digit-labelled recipient: there the edit moves the answer in 45–49% of pairs, against 32% with no edit, although the model answers digit-labelled questions correctly 95.0% of the time. Our reading is that digit-labelled prompts do not rely on this subspace alone to produce their answer. DAS and one-hot targets at the same budget show the same pattern, so it belongs to the model at this site, and all the methods we measured agree on it.

On the explanatory claim, removing the executable high-level model does not change what is certified about the tested variable. Interchange accuracy scores a high-level model only through its outputs on the tested pairs, so for the variable under test PSI and DAS certify the same kind of claim, and the transfer results above hold for both. What a full high-level model adds is hypotheses about other variables, and in PSI each of these can be tested with its own program.

On the difference from steering: in an interchange the experimenter chooses nothing, since the value written is whatever the donor prompt's activation holds, and a subspace that could only push the output in a fixed direction would fail the test. The controller of §2.3 is steering by design. What the paper adds is where it writes: inside a subspace that has passed the interchange test, in coordinates the program defines. This matters in practice: the same controller realizes 94.8% of held-out requests when trained jointly with PSI's subspace, against 83.9% in the top principal subspace of the same rank and 34.2% in a random one (§3.3, Fig. 4).

Q3: How much of the advantage over DAS remains with the same observations and donor pool?

The reviewer's hypothesis is right. We retrained the official DAS code with the same training examples and number of updates as PSI (Gemma-2 2B, MCQA, layer 17, rank 16, 768 updates, last checkpoint, three seeds). DAS reaches 91.70 ± 0.93 and PSI 90.24 ± 0.52 (92.00 ± 1.40 with the reference variant of the loss). With 122 test pairs these differences amount to one or two pairs, so at matched data and budget the two perform on par. The margins in Table 1 therefore are mainly driven from the increase data availability, and we have adapted the language in our manuscript to reflect this fact (§3.1, Appendix A.8). Importantly, PSI reaches this parity with random pairs, and without any gold label when the correctness filter is removed (−0.9 points; Appendix A.7).

The benefit of the formulation itself can be separated from the donor pool by holding prompts and pairs fixed and changing only the training target:

Training targetMCQA: MIB scoreMCQA: KL to donor ↓ARC: KL to donor ↓ARC: same answer as a wrong donor
model's distribution (PSI)90.24 ± 0.520.1030.02085.9 ± 1.0%
one-hot on the model's top option88.55 ± 0.490.2770.11779.3 ± 3.9%
one-hot gold labelas above*as above*0.09072.1 ± 5.6%

*On MCQA's filtered pool the gold label and the model's top option coincide. ARC: 2241 unfiltered natural questions, 589 held-out pairs, 111 of them with a donor the model answers wrongly (§3.2, Fig. 3).

With the donors fixed, reading the target from the model brings the edited model closer to the donor on both tasks. On ARC, where the model answers about a fifth of the questions wrongly, the subspace trained on gold labels transfers a wrong donor's answer in 72.1% of pairs, against 85.9% for PSI.

W2: Small, explicit outcome spaces

The answer list describes the outputs of the task, and the computation behind them never enters it, so how hard it is to write depends on the task's answer space more than on how abstract the variable is. The natural ARC questions of the new §3.2 show this: an executable high-level model for them would have to answer science questions, while the answer list is four symbols, and PSI localizes the variable there without any gold label. Many interpretability targets share this shape, such as multiple-choice, classification, yes/no questions and factual recall. Where the outputs are open-ended, naming a clean answer list is harder, and the savings depend on the variable, as we state in the Limitations (§4).

We hope that the transfer test the reviewer proposed, which shows that the subspace carries the position independently of the donor's symbols, together with the comparison of targets at fixed donors, addresses the concern about the explanatory claim and gives reason to reconsider the score. All changes are marked in blue in the revised manuscript.

Thanks for the detailed rebuttal. The additional experiments address most of my concerns, particularly the cross-encoding transfer and fair comparison with DAS. While the causal interpretation and generalizability remain somewhat limited, I appreciate the contribution of achieving comparable performance with weaker supervision. I have raised my score from 2 to 3.

Reviewer_TLeWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The work proposes a framework PSI that does not explicitly rely on a hypothesis H for the underlying causal mechanism. Since the interchange intervention criterion does not really rely on the details of H but only on the output under intervention, the method uses the model's own output distribution as a "reading" of the relevant causal variable without making any other assumptions on how the variable influences the output. Experiments on MIB show improvements over DAS in various settings.

Critical strengths
  1. Carefully controlled results on MIB with honest reporting on failed cases with a finding that MIB’s curated pair lists underuse the available data
Critical weaknesses
  1. The information in a semantic program (variable values plus the value-to-output map) is the same information needed to label randomly sampled pairs for DAS. Where the source’s value isn’t input-computable, it can be obtained by querying the model, as PSI itself does on RAVEL. It is unclear what setting, if any, PSI enables that DAS with random pairs cannot. The empirical advantage on MIB may be chalked down to the fact that the curated pair list may be overly restrictive.

  2. In general, I struggled with AI-typical phrasings such as "only asks the investigator to declare what the criterion consumes", "semantic program names the answers a variable is about", "moves the answer the way the program says" which are packed very densely in the introduction and are cognitively demanding to parse. Please rewrite the introduction in your own words which may be less efficient or punchy but easier to read.

Questions for authors

Please identify experimental settings where the proposed advantage of PSI is obvious.

2 comments

We thank the reviewer for the review, and for finding our results carefully controlled and sound. We agree with the observation on information content, and it made clear to us that our framing could be improved. PSI belongs to the DAS lineage, and the revised paper presents it as a change in where the training labels come from. The two methods are complementary: where an executable high-level model is available, they perform on par at matched data and budget, and where none is, PSI still applies, because it needs no labels. Its targets are read from the frozen model and its pairs are random, so no ground-truth label enters at any stage, and on questions the model answers wrongly its subspace follows the model more closely than one trained on gold labels. We ran new experiments to test each of these points, and we report them below.

W1: The same information as DAS with random pairs

We agree. A semantic program carries the information needed to label random pairs for DAS, and the source's value can be read from the model. With labels read from the model and a loss on the output distribution, the two methods optimize the same objective, and we make no claim of a localization setting that only PSI can handle. We removed the claim that PSI surpasses DAS (§3.1).

The published DAS baselines on MIB, like the other alignment methods we cite, obtain their labels by running an executable high-level model on a curated list of pairs. Training on random pairs with targets read from the model drops both requirements, and we tested the effect of each choice separately.

Curated pairs are not needed. We retrained the official DAS code at matched data and training budget (Gemma-2 2B, MCQA, layer 17, rank 16, 768 updates, three seeds, last checkpoint; §3.1, Appendix A.8):

Training targetPairsMIB score (%)KL to the donor's distribution ↓
PSI, model's distributionrandom90.24 ± 0.520.103
PSI, model's distribution (reference loss)random92.00 ± 1.400.102
one-hot on the model's top optionrandom88.55 ± 0.490.277
one-hot on the model's top option (reference loss)random89.71 ± 0.950.391
DAS, official implementationcurated91.70 ± 0.93n/a

With 122 test pairs, the differences between PSI and DAS amount to one or two pairs, so random pairs with targets read from the model perform on par with the official DAS. This is in line with the reviewer's observation that MIB's curated pair lists underuse the available data.

The model's distribution is a better target than a label read from the model. Training on random pairs with a one-hot label on the model's preferred option, as DAS would with labels read from the model, scores below PSI's target in every condition we ran (§3.1, Appendix A.7):

MIB score (%)Gemma-2 2B, filteredGemma-2 2B, unfilteredQwen-0.5B, filteredQwen-0.5B, unfiltered
model's distribution (PSI)90.24 ± 0.5289.34 ± 0.5299.57 ± 0.7497.78 ± 2.96
one-hot on the model's preferred option88.55 ± 0.4986.44 ± 0.5196.65 ± 0.6996.99 ± 2.82
one-hot gold label88.55 ± 0.49*87.59 ± 1.4396.65 ± 0.69*97.08 ± 1.60

*On the filtered pool the gold label and the model's preferred option coincide. On unfiltered Qwen-0.5B the differences are within the seed variation.

The gap is larger on the quantity an interchange is meant to reproduce: the KL from the donor's distribution to the edited recipient's is 0.103 with PSI's target and 0.277 with the one-hot one on MCQA, and 0.020 against 0.117 on ARC (below). A one-hot label, even one read from the model, discards the model's confidence, and the subspace it trains reproduces the donor's state less faithfully. The distribution this target is computed over is the one the program's answer list defines.

No gold labels are needed. The unfiltered columns above use no gold label at any stage: not to select prompts, not as targets and not to choose checkpoints. Removing the filter costs PSI 0.9 points on Gemma-2 2B.

Q: Settings where the advantage of PSI is obvious

Questions the model answers wrongly. We took the 2241 ARC (Easy) training questions without filtering (Gemma-2 2B answers 79.7% of them correctly), used other, unrelated questions as donors, and tested on 589 pairs of held-out questions (§3.2, Fig. 3, Appendix A.11). The answer list is again four option symbols, while an executable high-level model would have to answer science questions, so in practice the counterfactual labels come from the answer key.

Training targetKL to the donor's distribution ↓Same answer as the donor, all 589 pairsSame answer as the donor, 111 wrong donors
model's distribution (PSI)0.02094.8 ± 0.2%85.9 ± 1.0%
top option and confidence only0.02895.2 ± 0.4%84.7 ± 1.8%
one-hot on the model's top option0.11792.9 ± 1.1%79.3 ± 3.9%
one-hot gold label0.09092.4 ± 1.1%72.1 ± 5.6%
no intervention1.54825.6%30.6%

When the donor's answer is wrong, the subspace trained on gold labels transfers it in 72.1% of pairs, against 85.9% for PSI. A gold label on such a question describes the task, while an interchange asks about the model's state, so the gold label is the weaker target where the model and the task disagree, which are often the cases an interpretability study cares about. Most of the difference is already recovered by keeping only the model's confidence, so what matters is that the target is graded and read from the model. On the benchmark's official score, which rewards the correct counterfactual answer, the gold-trained subspace is ahead (71.45 against 70.03), as expected.

Control without a donor. The second setting is already in the submission (§3.3, Fig. 4). Because the answer list is fixed, PSI can train the controller jointly with the subspace, while for a DAS subspace the same controller has to be attached afterwards. Joint training realizes 94.8% of held-out requests, against 81.7% for the controller on a frozen DAS subspace and 86.6% on our own frozen subspace.

W2: Writing

We thank the reviewer for clarifying which part of our manuscript needed improvements. We leveraged the extended page limit to rewrite the quoted sentences and other parts in the abstract and introduction. The Methods section now uses a concrete multiple-choice question as its running example. We marked all changes to the manuscript in blue in the revised PDF.

Summary

The new experiments add three results to the paper:

  • At matched data and budget, random pairs with targets read from the frozen model perform on par with the official DAS on MCQA, without an executable high-level model or a curated pair list.
  • The model's distribution over a declared answer list is a better training target than a label, whether the label is read from the model or from the answer key; with gold labels, the subspace tracks the model less well where the model is wrong.
  • Localization works with no ground-truth label at any stage, including on natural questions for which no short algorithm exists.

Even if PSI and a suitably modified DAS optimize the same objective, these results show which modifications matter and what each one contributes. We believe a second route to the identification problem, complementary to DAS and needing no labels, is useful to the field, and we hope these results give the reviewer reason to reconsider the score.

Reviewer_k24LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper starts from the observation that previous methods require a high-level algorithm and curated prompt pairs, specifying what the model should answer after an intervention on a high-level variable. It replaces these with a “semantic program”: a list of the answer strings that each value of the variable would produce, and a rule for how the model’s probabilities over that list should change under an intervention. The training targets come from the frozen model’s own probabilities on randomly paired prompts. As in previous methods, a low-rank subspace is trained so that swapping it reproduces these targets. Since the list has a known meaning, the same subspace can also be steered towards any requested answer distribution without a donor prompt. On the MIB benchmark, the method matches or beats DAS in most cells, with mixed results on RAVEL and IOI.

Critical strengths

The core observation, that IIA only uses the high-level model’s output, so only that output needs to be specified, is well argued.

The way the declared library is reused as an interface for steering is elegant.

The writing is good to follow and the paper is well structured.

Critical weaknesses

The claimed reduction in supervision is small in the experiments: A DAS training target requires the variable’s value on the source and the map from that value to the base’s answer. The proposed method still needs the second part (the outcome library) and drops the first using the model’s own probabilities on the source instead of the ground turth. But training is restricted to prompts the model answers correctly, so on those prompts the model’s probabilities peak on the ground truth and the target is then nearly the same as DAS’s. Hence the supervision from ground truth has not actually been removed.

The method is presented as a simpler alternative to specifying the high-level algorithm, which may be overstated: constructing the libraries O(x) can require as much information, or nearly as much, as the algorithm itself (e.g. in the arithmetic task, writing the library requires computing both sums, which already reveals the carry).

Relatedly, the setting where the method should help most is not tested: variables whose value is hard to compute but whose library O(x) is easy to specify, without ground-truth labels available.

Questions for authors

Did the authors run experiments without the ground-truth filter? If so, how does performance change?

2 comments

We thank the reviewer for the thoughtful review and for having raised three important points, which we have addressed in this rebuttal with two new experiments and a revision of the method section.

W1: The supervision saving is small

We agree with the reviewer that the presence of a correctness filter limits the claim of a reduction in the supervision. However, even on correctly answered prompts, the model's answer is far from a one-hot distribution: on the MCQA task, for example, Gemma-2 2B gives the correct option a mean probability of 63.6% (78.8% for Qwen-0.5B), while on the natural ARC questions only 28.7% of all questions have a top probability exceeding 0.9 (new Fig. 3a). Even when the model is right, 21% (Qwen-0.5B) to 36% (Gemma-2 2B) of the probability mass sits on other options on average, so the PSI and DAS targets differ in practice.

To fully address the general point on the supervision signal provided by the correctness filter, we ran a new experiment with the filter removed.

We used the same MCQA site as in the paper (layer 17, last token, rank 16), three seeds, exactly 768 updates and the last checkpoint, so that no gold label enters through checkpoint selection either. Each model was trained either on the filtered pool (prompts the model answers correctly) or on all prompts, with three kinds of target: the model's own distribution over the options (PSI), a one-hot label on the option the model prefers, and a one-hot gold label. The unfiltered runs with the first two targets thus use no gold label at any stage: not to select prompts, not as targets and not to choose checkpoints.

MIB score (%)Gemma-2 2B, filteredGemma-2 2B, unfilteredQwen-0.5B, filteredQwen-0.5B, unfiltered
model's distribution (PSI)90.24 ± 0.5289.34 ± 0.5299.57 ± 0.7497.78 ± 2.96
one-hot on the model's preferred option88.55 ± 0.4986.44 ± 0.5196.65 ± 0.6996.99 ± 2.82
one-hot gold label88.55 ± 0.49*87.59 ± 1.4396.65 ± 0.69*97.08 ± 1.60

Mean ± standard deviation over three seeds. *On the filtered pool the gold label and the model's preferred option coincide, so these are the same runs.

In general, removing the filter affects performance only mildly (−0.9 points for PSI on Gemma-2 2B, −2.1 with one-hot labels on the model's preferred option; on Qwen-0.5B all changes are within the seed variation), and PSI's targets give the highest score in all four conditions (§3.1; Appendix A.7, Table 3).

W2: The answer list requires as much as the algorithm

The reviewer is right in saying that, in the case of the arithmetic task, the requirement of an answer library and the algorithm are equivalent: the saving we argued for in the paper is conditional on the task. In the case of a multiple-choice task, for example, the list is the four option symbols, with no algorithmic requirement, and the saving is real: on the natural ARC questions of W3, an executable algorithm would have to answer science questions, while the answer list is still the four symbols.

We have addressed this point in the method section of the manuscript, which now uses a multiple-choice question as its running example (§2.1), with the caveat for the arithmetic task stated explicitly in the same section and in the Limitations (§4).

W3: A setting where the method helps most

We thank the reviewer for this important suggestion: we have now included a new experiment to illustrate this point precisely (new §3.2, Fig. 3; Appendix A.11). We considered the ARC science questions task, where the four option symbols fully define the answer list, while answering itself needs science knowledge.

We took the 2241 ARC training questions as they come, without filtering (Gemma-2 2B answers 79.7% of them correctly), at the same site and rank as above, and used other, unrelated questions as donors. We compared four ways of building the training target, three seeds each, with 768 updates and the last checkpoint: the model's full distribution over the four options (PSI); a control that keeps only the model's top option and its probability, spreading the rest evenly over the other three; a one-hot label on the model's top option; and a one-hot gold label. Only the last of these sees gold labels. We tested on 589 pairs of held-out questions, 111 of which have a donor question the model answers wrongly, and measured how closely the intervened model reproduces the donor's own distribution and answer.

Training targetKL to the donor's distribution ↓Same answer as the donor, all 589 pairsSame answer as the donor, 111 wrong donors
model's distribution (PSI)0.02094.8 ± 0.2%85.9 ± 1.0%
top option and confidence only0.02895.2 ± 0.4%84.7 ± 1.8%
one-hot on the model's top option0.11792.9 ± 1.1%79.3 ± 3.9%
one-hot gold label0.09092.4 ± 1.1%72.1 ± 5.6%
no intervention1.54825.6%30.6%

Mean ± standard deviation over three seeds; the standard deviations of the KL values are below 0.004.

This experiment shows that, without gold labels, the subspace can be learned on natural questions: swapping it makes the recipient answer like the donor 95% of the time, and still 86% of the time when the donor's answer is wrong. The subspace trained on gold labels is the worst in exactly that case (72%).

Additionally, on the benchmark's own score, which compares the output with the gold counterfactual answer, the gold-trained subspace is slightly ahead (71.45 vs 70.03 for PSI), as expected for a score that rewards the correct answer.

Q: Did the authors run experiments without the ground-truth filter?

Yes, in two settings: on MCQA, where removing the filter costs about one point (W1), and on natural ARC, where every arm is trained without the filter (W3).

In summary, we argue that these new experiments illustrate how the training labels can come from the frozen model (no filtering) and the pairs can be random. At matched data and budget, this performs on par with DAS on its curated pairs (general response; §3.1, Appendix A.8), and we revised the claims of the paper accordingly. We see this as the main practical gain of PSI: localization no longer requires ground-truth labels, which makes it applicable to tasks where none exist. All changes are marked in blue in the revised PDF, and we hope these results address the reviewer's concerns and give reason to reconsider the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: the reviewers’ experiments and manuscript improvements

We thank the reviewers for their careful reviews and for important questions, which we have addressed with new experiments. The answers have made the paper considerably stronger: on MCQA, PSI performs on par with the official DAS at matched data and budget while using random pairs and no executable high-level model; it localizes the variable with no ground-truth label at any stage; and when the donor is a natural science question that the model answers wrongly, its subspace transfers the model's own answer substantially more often than a subspace trained on gold labels (85.9% against 72.1%).

Parity with DAS, with random pairs and no high-level model (Reviewers nRKj and TLeW; §3.1, Appendix A.8). We retrained the official DAS implementation on MCQA with the same data and number of updates as PSI. DAS reaches 91.70 ± 0.93 and PSI 90.24 ± 0.52, or 92.00 ± 1.40 with the reference variant of the loss, a difference of one or two of the 122 test pairs. PSI uses random pairs and labels read from the model, so the executable high-level model and the curated pair list can be dropped on this task without a measurable loss.

No ground-truth label at any stage (Reviewer k24L; §3.1, Appendix A.7). With the correctness filter removed, no gold label enters prompt selection, training targets or checkpoint choice. PSI loses 0.9 points on Gemma-2 2B, and on Qwen-0.5B the change is within seed variation. On both models, with and without the filter, the model's own distribution scores above one-hot labels, whether read from the model or from the answer key.

Following the model where it is wrong (Reviewers k24L and TLeW; new §3.2, Fig. 3, Appendix A.11). Natural ARC science questions are the setting both reviewers asked for: the answer list is four option symbols, while an executable high-level model would have to answer science questions. Trained on 2241 unfiltered questions with gold labels withheld, the PSI subspace makes the recipient give the donor's answer in 95% of held-out pairs. When the donor is a question the model answers wrongly, it still does so in 85.9% of pairs, against 72.1% for a subspace trained on gold labels. A gold label describes the task, while an interchange asks about the model's state, so where the two disagree (often the cases an interpretability study cares most about), the gold-trained subspace is the weaker account of the model.

The subspace carries the variable across answer encodings (Reviewer nRKj; §3.1, Appendix A.10, Fig. 6). As suggested, we froze the subspace trained with the labels A/B/C/D and swapped it between prompts labelled differently. A subspace that only controls the output would write the donor's symbol; one that carries the variable writes the donor's position in the recipient's own symbols. The PSI subspace writes the position: with a lowercase donor and an uppercase recipient, it outputs the donor's symbol in 0 of 150 pairs, and when both position and symbols change (44 pairs), it gives the recipient's symbol at the donor's position in 66.7% of pairs, against 9.1% for the donor's own symbol. Copying the full activation at the same site, by contrast, does carry the token.

Requests without a donor (as in the submission; §3.3, Fig. 4). Because the answer list is fixed, the investigator can request an answer directly in the subspace: trained jointly, it realizes 94.8% of held-out requests, against 81.7% for the same controller on a frozen DAS subspace.

Together, these results open interchange-based localization to tasks where no executable high-level model can be written and no labels exist.

What changed in the PDF (in blue)

  • Abstract and introduction: claims revised as above, and the sentences Reviewer TLeW quoted rewritten in plainer terms.
  • §2: a multiple-choice question as the running example, the cost of the arithmetic library, and training without ground-truth labels.
  • §3.1: matched-budget comparison with DAS, correctness filter removed, transfer across answer encodings. New §3.2 and Fig. 3: natural ARC questions.
  • Appendix A.7 to A.11: filter and budget tables, per-family scores, transfer across encodings, ARC details.
  • Limitations updated.
  • RAVEL (Appendix A.12, Limitations), a correction we found ourselves: the submission scored PSI and the random subspace with a different protocol from the published baselines. Under the baselines' protocol, PSI is 3 to 11 points below published DAS in all six cells and above the other baselines in all but one, so we withdraw the claim that PSI ties or leads every published method on RAVEL, and the remark that a random subspace beats them. A controlled comparison on Gemma-2 2B traces most of the gap on continent, and about half on language, to the training targets: the model's reading, spread over a long answer list shared by three attributes, is a weaker target there than the gold label.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.