acceptodds
Under review as a conference paper at ICLR 2027

PLOT: Progressive Localization via Optimal Transport in Neural Causal Abstraction

Abstract

Causal abstraction offers a principled framework for mechanistic interpretability, aligning a high-level causal model with low-level neural computation through interchange intervention analysis. Finding such an alignment, however, often requires fitting and evaluating separate learned mappings across many candidate neural locations. We introduce PLOT, a gradient-free approach for joint correspondence discovery via a global matching of intervention effects. PLOT represents abstract and neural interventions by geometric signatures of their effects on the shared task output and fits an optimal transport coupling between the two collections. The coupling can be calibrated directly into an executable intervention handle or used progressively to select coarse parent sites for finer localization, reducing the cost of searching broad collections of neural sites. Experiments on hierarchical equality, binary addition, and multiple-choice question answering (MCQA) demonstrate fast and accurate direct handles without learning intervention rotations and show that progressive localization can improve accuracy under intervention-size constraints while reducing computational cost. PLOT can also guide gradient-based subspace learning, with PLOT-guided distributed alignment search (DAS) attaining accuracy comparable to full DAS at approximately and lower serial runtime on binary addition and MCQA, respectively. PLOT thus separates the search for candidate correspondences from optional subspace learning and provides a common framework for constructing and testing neural intervention handles for a given causal model.

Primary area:
interpretability and explainable AI
Keywords:
causal abstractions, optimal transport, mechanistic interpretability, intervention interchange, activation patching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TLBPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PLOT, which is a gradient-free approach for discovering joint effects using optimal transport. They evaluate the approach on increasingly difficult tasks (hierarchical equality, binary addition, MCQA) and on increasingly complex models (an MLP, an RNN, and a Transformer). In general, PLOT is seen as a supplement to methods like Distributed Alignment Search (DAS), which tries to identify neural subspaces that implement one (or more) abstract variables. Assuming a correctly specified causal model, PLOT tries to discover its implementation in the neural network without using subspace analysis or gradients. My understanding of how PLOT works is:

  1. They require 3 datasets: one for fitting, one for calibration, and one for testing
  2. Measure abstraction intervention effects by swapping values (in the causal model) between two examples and recording the output change
  3. Measure neural intervention effects by swapping the base coordinates of a candidate neural site (for instance a PCA direction or a neuron) by the ones corresponding to the activations for the alternatively value in step 2
  4. Fitting an optimal transport coupling between the two effects
  5. Use the fitted coupling and the calibration dataset to calibrate intervention strengths (maximizing for largest logit agreement between intervented token and abstract target)
  6. Evaluating on a held out dataset.

PLOT additionally becomes hierarchical by considering its step 1 and 2 (2, 3, and 4 in my list above) applied first to coarser sites and then fitting new couplings within them. A concrete example from the paper is first considering token-wise plot, then layer-wise PLOT and then within-layer PLOT. This lets the method potentially be applicable also to large models by gradually honing on the correct neural sites that implement the abstract model.

They validate the method on 3 experimental configurations that increase in complexity:

  1. A ReLU MLP with 3 layers on Hierarchical Equality
  2. A GruCELL based RNN on Binary Addition
  3. Gemma-2-2B on MCQA

Across all experimental configurations, PLOT is usually worse than DAS, but only barely and in a few configurations (such as the majority of AT rows for MCQA) it is consistently better. It is always much faster, and for the latter 2 experiments, the authors also propose a variant, PLOT-DAS, which first uses PLOT for coarse localization and then trains DAS only within selected states. The biggest performance gap is in the MCQA setting, where PLOT-DAS ranks 0.02 below full DAS but runs 20 times as fast.

The authors conclude the paper by stating that sharper localization rules may improve joint correspondence. They also note that unlike DAS, which must operate in the natural neural bases' of the network being analyzed, PLOT can work on interpretable feature constructions such as cross-layer transcoders or, for instance, models derived from parameter decomposition.

Critical strengths

Formalization of causal localization as a problem of optimal transport: Highly novel and interesting. PLOT and PLOT-DAS for joint correspondence: The described algorithm performs in general as well as DAS on the experiments, is much faster, and has more flexibility in being applicable also to things like transcoders. This gives much more utility than DAS itself and opens the door to usage in much larger models.

Critical weaknesses

Justification for optimal transport: While the authors do make good arguments for OT, there is still some unresolved tension. Simpler independent matching methods are quite competitive with OT, and it is not clear to me that optimal transport is the main source of the increase in performance. It would be nice to compare PLOT-DAS against DAS guided by one of the simpler localization strategies. Experimental breadth: This is a bit of a nitpick, but it would be nice to see a few more experiments on a task slightly more complex than MCQA. Note that I do not actually consider this necessary for acceptance, I just think it would slightly improve the paper.

Questions for authors

Could you elaborate on our justification for optimal transport? In particular, I'm interested in:

  • Given the performance of the non-OT methods, how much of the performance can we really attribute to OT?
  • Why do the authors expect that the causal correspondence problem has the constraints imposed by optimal transport? For instance, an abstract variable might be distributed over multiple sites.
4 comments

Thank you very much for the extensive comments and feedback. We appreciate the time you took to make explicit your concerns about our paper. We address each of your concerns individually below.

Justification for optimal transport: While the authors do make good arguments for OT, there is still some unresolved tension. Simpler independent matching methods are quite competitive with OT, and it is not clear to me that optimal transport is the main source of the increase in performance. It would be nice to compare PLOT-DAS against DAS guided by one of the simpler localization strategies.

Thank you for the positive outlook on our paper and the constructive feedback. We agree that the current comparisons do not isolate how much of PLOT-DAS’s benefit comes from OT itself, rather than from a matching-based screening of locations before subspace learning. We focus the paper around OT because it provides a principled joint matching formulation and performs well empirically. However, the matching rule is a flexible component of the approach, as noted in Remark 3.1, and we will gladly add experiments to the appendix comparing PLOT-DAS with DAS guided by non-OT localization, such as those considered in Appendix B.1. In particular, we will keep the candidate sites, data splits, and downstream DAS procedure fixed so that the comparison isolates the choice of localization strategy. If we obtain these results during the discussion period, we will update the reviewer here with the findings and resulting insights.

Experimental breadth: This is a bit of a nitpick, but it would be nice to see a few more experiments on a task slightly more complex than MCQA. Note that I do not actually consider this necessary for acceptance, I just think it would slightly improve the paper.

Thank you for the suggestion. We agree that additional experiments will further strengthen the paper. After addressing the comments concerning our current experiments, we will work to add an experiment on ARC-Challenge from MIB, which involves answering grade-school multiple-choice science questions requiring knowledge beyond the information provided in the prompt (MIB, Section 2.1). Another option we will consider is the RAVEL task from MIB, which tests whether neural interventions can selectively change a model’s answers about one attribute of a city, such as its country, while preserving its answers about other attributes. These results will likely be included in the appendix.

Given the performance of the non-OT methods, how much of the performance can we really attribute to OT?

Thanks again for this excellent question. Please see our response to the first Critical Weakness above. We believe that the additional comparison with DAS guided by non-OT localization will help distinguish the benefit of screening itself from any additional benefit provided specifically by OT, and plan to discuss the resulting insights in the revision.

Why do the authors expect that the causal correspondence problem has the constraints imposed by optimal transport? For instance, an abstract variable might be distributed over multiple sites.

We hope that we understand the reviewer’s point correctly, but please let us know if this is not the case. In general, an abstract variable being distributed across multiple neural sites is fully compatible with PLOT. Each row of the coupling can assign mass to several sites, and the resulting handle combines interventions at those sites, potentially across layers (based on calibration). Different abstract variables can also assign mass to the same site. OT does not require a one-to-one correspondence. For example, in 1-stage PLOT PCA for binary addition, both C2C_2 and C3C_3 carry variables were matched to the same PCA component in the same timestep.

The use of uniform reference weights is a modeling choice. Uniform weights avoid an a priori preference for particular variables or sites. However, enforcing a uniform neural marginal also constrains how total matching mass is distributed across sites. This couples the localization decisions across variables, but can be restrictive when sites differ in relevance or are shared unevenly among variables. This motivates our use of one-sided UOT in MCQA, which retains the abstract marginal while allowing nonuniform total mass on neural sites.

I thank the authors for their response! Especially the last point; it seems like I partly misunderstood part of the algorithm. Thank you for the clarification.

Overall, depending on the result of the extra experiments and the insights, I could justify going to a Clear Accept. For now I retain my score, but I also want to make it clear to the ACs and PCs (and authors), that I do believe this paper warrants publication in its current form.

We are glad that our response helped clarify this point, and very much appreciate your positive outlook and support for our paper. Since posting our response, have begun work on the immediate extensions of the experiments already in the paper: DAS guided by non-OT methods, Boundless DAS comparisons, and fuller runtime accounting. After that, we plan to work on an additional task, such as ARC-Challenge or RAVEL. We hope to share follow-up results soon during the discussion period and, if progress and time permit, results from an additional task as well.

Thank you again for your constructive feedback and support.

AuthorsOpenReview ↗

Follow-up: New screening and Boundless DAS results

Following up on our previous update, we can now share the completed non-OT-guided DAS and full Boundless DAS comparisons on MCQA, together with their runtime results.

The first five methods in the table below were run on NVIDIA H100 80GB GPUs using Gemma-2-2B, five seeds, and identical MCQA train/calibration/test pair banks. Measured values are mean ± sample SD, with IIA averaging answer pointer (AP) and answer token (AT) within each seed. Layers are zero-indexed.

MethodTest IIA (%)Serial runtime (min/seed)Seed 0 AP: layer / dimSeed 0 AT: layer / dim
PLOT-DAS87.45 ± 3.199.83 ± 0.81L17 / 1152L24 / 128
Cosine matching + DAS86.45 ± 3.419.45 ± 0.73L19 / 256L24 / 128
Lowest-cost + DAS86.45 ± 3.419.66 ± 1.24L19 / 256L24 / 128
IIA matching + DAS87.40 ± 3.8110.09 ± 1.24L18 / 512L24 / 128
Full Boundless DAS88.35 ± 1.8871.18 ± 3.83L17 / 334L22 / 420
Full DAS (paper; time estimated)89.60 ± 2.13198 (est.)L17 / 32L20 / 512

Matching formulation and alternatives.

As discussed in Appendix B.1 (Non-OT Matching Alternatives), we adopt OT because it provides a principled global constrained matching formulation, accommodates marginal relaxation through UOT, and draws on well-developed statistical and computational foundations. OT provides a coherent correspondence formulation, but non-OT alternatives can also be competitive in practice. We extend the appendix’s cosine and IIA matching to layer screening before DAS, and add lowest-cost screening using squared distances without transport, as proposed by Reviewer AvSU. Given the complementary roles of screening and subspace learning, the choice of matching is a flexible component of our framework.

The table shows that the four screening methods achieve similar mean test IIA, within 1.00 percentage point of each other, with PLOT-DAS having the highest mean at 87.45%. Their serial runtimes are also close, ranging from 9.45 to 10.09 min/seed, a difference of about 38 seconds. These results support the flexibility of the matching framework.

Boundless DAS (bDAS).

Runtime follows Appendix A.2; the MCQA procedures are in Appendix A.5 (Algorithms 11–12). Full bDAS adapts Algorithm 12 to learn the subspace dimension instead of searching over fixed sizes like standard/full DAS. In these experiments, standard DAS stops training when improvement plateaus. bDAS instead trains for 50 epochs while gradually sharpening a soft selection of directions. At evaluation, each direction is either included in the intervention or excluded, yielding the eventual subspace.

The first five rows are measured on H100s, while the full-DAS accuracy and handles come from the paper’s A40 runs. Its H100 time is estimated by 509.60×9.83/25.27≈198509.60\times9.83/25.27\approx198 min, scaling by the observed PLOT-DAS hardware speedup. We provide this estimate for a timely comparison, but emphasize that it is not a measurement of a full H100 sweep. We plan to rerun full DAS on H100s for the final version and add method-by-method operation counts and a computational-complexity discussion.

Relative to the estimated full-DAS runtime, full bDAS (without screening) is approximately 2.8×2.8\times faster, taking 71.18 min per seed. In comparison, the guided-DAS methods take 9.45–10.09 min per seed, making them 7.17.1–7.5×7.5\times faster than full bDAS. For PLOT-DAS specifically, this saves about an hour per seed, with mean IIA less than one percentage point lower: 87.45% versus 88.35%. Across all guided-DAS variants, the gap is 0.90–1.90 percentage points. Screening therefore preserves strong intervention accuracy while sharply reducing serial runtime, even when considering full bDAS.

Combining the strengths of screening and bDAS.

As proposed in our initial response to Reviewer rbT5, a natural next step is to use matching to select the layer, then let bDAS learn the dimension. This would avoid both the full layer sweep in bDAS and the ten-dimension search in PLOT-DAS. We plan to evaluate PLOT-guided and non-OT-guided bDAS in the final version, and expect the combination to further improve the accuracy–runtime tradeoff.

We are currently working on extending evaluation to additional tasks such as ARC-Challenge or RAVEL, as outlined in our previous update, and preparing a consolidated comparison on common hardware for the final version, although these results will likely be available only after the discussion period concludes. We will also revise the presentation as outlined in our initial response, clarifying the matching formulation and the complementary roles of screening and subspace learning. We sincerely thank you for your time and valuable feedback.

We hope these additional experimental results and the insights they provide address your remaining questions and support the stronger assessment you mentioned in your previous response.

Reviewer_rbT5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper contributes to solving the search problem implicit in many intervention-based interpretability methods: we want to find the most causally impactful sites, but this can require open-ended search over all possible representations/activations. There have been a number of efforts to search in smarter ways here, and this paper contributes a new, smart way of searching based on ideas from optimal transport theory. The paper benchmarks the method on some analytic tasks and on parts of the Mechanistic Interpretability Benchmark (MIB), and it finds that the method is comparable to Distributed Alignment Search (DAS) in terms of performance, but at a fraction of the cost in terms of runtime. The method also works well in combination with DAS and other methods as the feature finding method.

Critical strengths

I really like this paper. It is always great to see ideas productively imported from other fields. The ideas from OT used here seem expertly applied to very good effect. I also found the paper very clear and easy to read; it includes necessary background from causal interpretability and OT while still extending those methods technically and offering a solid amount of experimental evidence.

Critical weaknesses

As I said, I am a supporter of the paper. It contains an appropriate amount of new material and evidence for an ICLR paper.

I do think the experimental comparisons could be richer. In particular, it seems like the competitors are not DAS per se, but rather HyperDAS and maybe also (in a more limited way) Boundless DAS. These are cited appropriately in section 1.2 but not really brought into the framework of the paper. HyperDAS in particular seeks to reduce exactly the search costs the current paper is addressed to.

One might worry a bit about the dependent variable for the timing experiments. The feature finding methods are highly parallelizable, and so runtime alone is not especially informative (as the authors do concede in places).

Questions for authors

Could the authors report on systematic comparisons with other methods oriented toward this same goal, or else explain why such comparisons are not needed for the current contribution.

3 comments

Thank you very much for the extensive comments and feedback. We appreciate the time you took to make explicit your concerns about our paper. We address each of your concerns individually below.

I do think the experimental comparisons could be richer. In particular, it seems like the competitors are not DAS per se, but rather HyperDAS and maybe also (in a more limited way) Boundless DAS. These are cited appropriately in section 1.2 but not really brought into the framework of the paper. HyperDAS in particular seeks to reduce exactly the search costs the current paper is addressed to.

Thank you for the positive outlook on our paper and your support. We agree that HyperDAS and Boundless DAS deserve a more explicit discussion in relation to our framework.

BoundlessDAS is a valuable additional comparison, and we will gladly include a comparison to it for the MCQA experiment in the revised paper. We will also consider PLOT-guided Boundless DAS and compare it in terms of accuracy and runtime. These results will be added to Table 1.

HyperDAS automates token-position selection and constructs input-dependent intervention subspaces within a chosen layer. While PLOT can localize across token positions, following the MIB setting, our MCQA experiment fixes the final prompt-token position and study localization across layers and within-layer subspaces. Thus, token-position search, which is the central capability of HyperDAS, is not part of the evaluated problem. We will clarify this scope and the relationship to HyperDAS in the revision.

Two final points we would like to make in this regard are as follows. First, while HyperDAS and BoundlessDAS automate token-position and subspace-size selection, potentially reducing runtime, neither removes the cost which comes from optimization across all layers. PLOT eliminates this sweep for all DAS variants. Second, our submitted experiments also include DBM in canonical, PCA, and SAE representations. We selected DAS and DBM based on their performance in MIB (Mueller et al., 2025), which identifies DAS as the strongest overall method and DBM as a strong baseline for causal-variable localization.

One might worry a bit about the dependent variable for the timing experiments. The feature finding methods are highly parallelizable, and so runtime alone is not especially informative (as the authors do concede in places).

This is an excellent point. Parallelization can substantially change elapsed runtime for both methods. Our full-DAS searches were split into separate jobs across candidate layers, illustrating how parallel execution can reduce waiting time without reducing the total search workload. We report serial method runtimes under the stated hardware configuration, but agree that these should be supplemented with counts of training and forward-only intervention evaluations, as also requested by Reviewer AvSU. Across the five reported MCQA seeds, full DAS trained 520 candidates per seed, compared with 20 for PLOT-DAS. Accounting for early stopping, these searches averaged 1,096,640 and 53,120 training-pair evaluations involving backpropagation per seed, respectively. PLOT-DAS additionally has a coarse-screening grid of 43,600 forward-only intervention-pair evaluations before reuse and 16 transport solves of size 2×262 \times 26. We will include separate accounting for downstream calibration, testing, shared preprocessing, and reuse, and distinguish these workload counts from FLOPs.

More broadly, as discussed in our response to Reviewer AvSU, we will revise the presentation to emphasize the computational benefit of separating site discovery from subspace learning. We will contextualize the runtime comparisons in terms of the candidate searches, implementations, and hardware evaluated, and distinguish reductions in aggregate computational work from reductions in elapsed time through parallelization.

Could the authors report on systematic comparisons with other methods oriented toward this same goal, or else explain why such comparisons are not needed for the current contribution.

Indeed, we plan to add a further comparison to BoundlessDAS as well as its PLOT-guided version, as noted above. Thank you for this suggestion.

Reviewer_rbT5OpenReview ↗

Excellent plans

I appreciate the author's comments, and I think their plan for future work is excellent. Since this work is not incorporated into the review version, I will stick with my current (quite positive) score.

AuthorsOpenReview ↗

Follow-up: New screening and Boundless DAS results

Thank you for your encouraging follow-up. We can now report completed experiments for the revision: the full Boundless DAS comparison promised in our initial response, together with three non-OT screening-plus-DAS baselines on MCQA.

The first five methods in the table below were run on NVIDIA H100 80GB GPUs using Gemma-2-2B, five seeds, and identical MCQA train/calibration/test pair banks. Measured values are mean ± sample SD, with IIA averaging answer pointer (AP) and answer token (AT) within each seed. Layers are zero-indexed.

MethodTest IIA (%)Serial runtime (min/seed)Seed 0 AP: layer / dimSeed 0 AT: layer / dim
PLOT-DAS87.45 ± 3.199.83 ± 0.81L17 / 1152L24 / 128
Cosine matching + DAS86.45 ± 3.419.45 ± 0.73L19 / 256L24 / 128
Lowest-cost + DAS86.45 ± 3.419.66 ± 1.24L19 / 256L24 / 128
IIA matching + DAS87.40 ± 3.8110.09 ± 1.24L18 / 512L24 / 128
Full Boundless DAS88.35 ± 1.8871.18 ± 3.83L17 / 334L22 / 420
Full DAS (paper; time estimated)89.60 ± 2.13198 (est.)L17 / 32L20 / 512

Matching formulation and alternatives.

As discussed in Appendix B.1 (Non-OT Matching Alternatives), we adopt OT because it provides a principled global constrained matching formulation, accommodates marginal relaxation through UOT, and draws on well-developed statistical and computational foundations. OT provides a coherent correspondence formulation, but non-OT alternatives can also be competitive in practice. We extend the appendix’s cosine and IIA matching to layer screening before DAS, and add lowest-cost screening using squared distances without transport, as proposed by Reviewer AvSU. Given the complementary roles of screening and subspace learning, the choice of matching is a flexible component of our framework.

The table shows that the four screening methods achieve similar mean test IIA, within 1.00 percentage point of each other, with PLOT-DAS having the highest mean at 87.45%. Their serial runtimes are also close, ranging from 9.45 to 10.09 min/seed, a difference of about 38 seconds. These results support the flexibility of the matching framework.

Boundless DAS (bDAS).

Runtime follows Appendix A.2; the MCQA procedures are in Appendix A.5 (Algorithms 11–12). Full bDAS adapts Algorithm 12 to learn the subspace dimension instead of searching over fixed sizes like standard/full DAS. In these experiments, standard DAS stops training when improvement plateaus. bDAS instead trains for 50 epochs while gradually sharpening a soft selection of directions. At evaluation, each direction is either included in the intervention or excluded, yielding the eventual subspace.

The first five rows are measured on H100s, while the full-DAS accuracy and handles come from the paper’s A40 runs. Its H100 time is estimated by 509.60×9.83/25.27≈198509.60\times9.83/25.27\approx198 min, scaling by the observed PLOT-DAS hardware speedup. We provide this estimate for a timely comparison, but emphasize that it is not a measurement of a full H100 sweep. We plan to rerun full DAS on H100s for the final version and add method-by-method operation counts and a computational-complexity discussion.

Relative to the estimated full-DAS runtime, full bDAS (without screening) is approximately 2.8×2.8\times faster, taking 71.18 min per seed. In comparison, the guided-DAS methods take 9.45–10.09 min per seed, making them 7.17.1–7.5×7.5\times faster than full bDAS. For PLOT-DAS specifically, this saves about an hour per seed, with mean IIA less than one percentage point lower: 87.45% versus 88.35%. Across all guided-DAS variants, the gap is 0.90–1.90 percentage points. Screening therefore preserves strong intervention accuracy while sharply reducing serial runtime, even when considering full bDAS.

Combining the strengths of screening and bDAS.

As proposed in our initial response, a natural next step is to use matching to select the layer, then let bDAS learn the dimension. This would avoid both the full layer sweep in bDAS and the ten-dimension search in PLOT-DAS. We plan to evaluate PLOT-guided and non-OT-guided bDAS in the final version, and expect the combination to further improve the accuracy–runtime tradeoff.

We are currently working on extending evaluation to additional tasks and preparing a consolidated comparison on common hardware for the final version, although these results will likely be available only after the discussion period concludes. We will also revise the presentation as outlined in our initial response, clarifying the HyperDAS comparison, fixed-token MCQA scope and computational accounting. We hope these additional comparisons address your concerns and clarify the benefits of separating site discovery from subspace learning. We sincerely thank you for your time and valuable feedback.

Reviewer_AvSUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PLOT, which is a causal abstraction method powered by optimal transport. Since, the neural sites that correspond to a causal variable might live not only in a single subspace, authors design the method to consider transporting different neural sites (e.g. layers) to causal variables at once, while keeping away some unmatched neural cells by introducing unbalanced variant of optimal transport. PLOT is also able to be done with hierarchical neural sites rather regarding all neurons at the same level, by warm-starting a finer family of neural sites utilizing the transport plan from the coarse one. The results show that PLOT can find a compatible correspondence as much as DAS does while having much more time efficiency and also can serve as a sub-algorithm of DAS providing neural sites candidates rather than sweeping all layers.

Critical strengths
  • The writing is accessible even to readers unfamiliar with complex concepts like causal abstraction and optimal transport, especially Sec 2.2..

  • Authors provide algorithm blocks, which are helpful to follow runtime analysis and the algorithms themselves.

Critical weaknesses

Limitations of prior work, according motivation and the commensurability between DAS and PLOT are not clear.

While authors explain the proposed method clearly, but it is ambiguous how proposed work is addressing the limitations and whether DAS and the proposed method aim to solve the same causal abstraction task or not. I articulate my concerns as following:

  1. DAS can set neural sites across layers. Authors point out that discovering a correspondence might require considering many neural sites as both a motivation and a limitation of DAS. Even though it is true that the original DAS paper [1] demonstrates its method by sweeping single layers, it is unlikely that DAS cannot express cross-layer correspondence by design. We can conduct DAS after defining neural sites as different layers of neurons, or moreover, we can select a sufficiently late layer, conduct the DAS, then distribute each row direction of the rotation matrix to all earlier layers in a post-hoc manner, utilizing the residual structure (as formulas below). Also, following works after DAS have been proposed and some of them can choose or train to choose different hierarchies of neural sites, such as token positions [2]. Considering this, it would be helpful if authors further clarify the limitations and also the motivation of proposing PLOT .
    hL=h0+∑ℓ<Lwℓh_L = h_0 + \sum_{\ell < L} w_\ell

ΔL:=W⊤(hL(s)−hL(b))=W⊤Δh0+∑ℓ<LW⊤Δwℓ\Delta_L := W^\top\bigl(h_L(s) - h_L(b)\bigr) = W^\top \Delta h_0 + \sum_{\ell < L} W^\top \Delta w_\ell

= \frac{\mathbb{E}\bigl[\langle \Delta_L,\, W^\top \Delta w_\ell \rangle\bigr]} {\mathbb{E}\bigl[\lVert \Delta_L \rVert_2^2\bigr]}, \qquad m_0 + \sum_{\ell < L} m_\ell = 1$$ 2. **DAS does hard assignment on a learned subspace but optimal transport used in PLOT does soft assignment so that it is not sure whether they can be compared.** In causal abstraction [3], there exist some kind of properties (e.g., intervention algebra, please refer to Sec. 2.2 in [3]) to ensure that interventions are modular, but it is not clear that to what extent the OT-based approach satisfies. Furthermore, DAS does interchange each direction exclusively, while PLOT chooses top-k from a transport plan besides the global intervention strength lambda. To sum up, it is obscure whether those two methods solve the same class of causal abstraction problem. It would be better to clarify whether the causal abstraction task is sufficient to find a correspondence that maximizes IIA given a structural causal model and a neural model, or whether additional constraints or properties must be satisfied. **Other concerns** It seems to be exaggerated to criticize the runtime of DAS and justify the time efficiency of PLOT. The main bottleneck of DAS is pointed out as a sweep process of layers, while PLOT can serve as a candidate search limiting layers. As mentioned before, a learned subspace of residual stream can be unfolded to preceding layers in post-hoc manner. Besides, if the purpose of PLOT is limiting candidate states, what if we choose bottom-k of j whose costs with respect to causal variables (i) are small or use other simple calibration methods? Besides, I believe that it is better to provide either asymptotic time-complexity analysis or operation counts. There is no Limitations section. Please add this section to the main text, or include it in the Appendix and reference it in the main body. Also please specify them in comments for dicussion. There is only DAS for a correspondence method baseline besides basis selections. Please justify in detail or add different baselines both in literature review section and experiments. Possible candidates include general correspondence algorithms and the methods from causal abstraction literature. [1] A. Geiger, et al., Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations, 2023 [2] J. Sun, et al., HyperDAS: Towards Automating Mechanistic Interpretability with Hypernetworks, 2025 [3] A. Geiger, et al., Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability, 2023
Questions for authors
  1. Why does PLOT need the intervention coefficient lambda, while UOT itself can express non-uniform weights? What different roles do they take?
  2. Could you please specify the rationale of how each design component (single vs. Multi-stage, optimal transport vs. other correspondence algorithms, etc.) contributes to solving casual abstraction task, and how it can improve DAS in terms of what aspect?
  3. What is the choice of beta? Could you explain whether the role is the same and/or different from Y_0 in DAS?
  4. In terms of basis, DAS learn a subspace while a PLOT utilizes a given basis. There exists a case that a given basis is not aligned with the causal variables (demonstrated in [1]). DAS addresses this problem by learning a subspace while PLOT could avoid this by allowing soft assignments. Considering this, how do you think each design could lead to better or different interpretations?
6 comments

Thank you very much for the extensive comments and feedback. We appreciate the time you took to make explicit your concerns about our paper. We address each of your concerns individually below. Due to space restrictions, we continue our response in follow-up comments.

While authors explain the proposed method clearly, but it is ambiguous how proposed work is addressing the limitations and whether DAS and the proposed method aim to solve the same causal abstraction task or not. I articulate my concerns as following

DAS and PLOT address the same overarching problem of neural causal abstraction: identifying executable neural intervention handles that reproduce the counterfactual effects of interventions on the corresponding high-level variables. Both assess proposed correspondences through interchange intervention accuracy. This localization objective is also pursued by approaches based on ranking, exhaustive activation-patching sweeps, and differential binary masking (DBM), and is the focus of the causal-variable localization track of the Mechanistic Interpretability Benchmark (Mueller et al., 2025).

The roles of PLOT and DAS within this problem, however, are different and complementary. In the standard DAS formulation used in our experiments, a neural location is specified and DAS learns a rotated subspace within it. PLOT instead approaches correspondence discovery as a global matching problem between abstract and neural intervention effects. The resulting coupling can directly define an intervention handle or progressively narrow the search to promising locations for finer localization, including subsequent DAS refinement. PLOT-DAS illustrates this complementarity: PLOT proposes the location, and DAS learns the intervention subspace within it.

More broadly, we view this matching-based perspective as a central contribution of the work. It separates correspondence discovery from optional subspace learning and opens the problem to tools from optimal transport and related matching frameworks. Importantly, we do not propose PLOT as a replacement for DAS, which we view as the best method for extracting a high-IIA handle from a given location. We recognize that some passages in the submitted manuscript may overemphasize the contrast between the methods, as opposed to emphasizing this relationship. We will revise those passages to clarify their complementary roles, remove the numerical speedup factors from the abstract, and properly contextualize the runtime comparisons in the experiments.

DAS can set neural sites across layers.

We agree that DAS is not intrinsically restricted to a single layer, and acknowledge that extensions such as HyperDAS automate additional aspects of localization, including token-position selection and subspace construction within a specified layer (as discussed in Section 1.2). Our motivation is not to overcome an inherent deficiency of DAS.

Rather, PLOT introduces a matching-based perspective on neural causal abstraction. As mentioned, we view this formulation as a central contribution since (i) it provides a natural framework for the task, (ii) performs well in practice as demonstrated by our experiments, and (iii) importantly, opens the problem to tools from optimal transport and other matching methods.

Within this framework, site discovery and subspace learning play complementary roles. PLOT can propose promising locations without fitting a separate intervention rotation at each candidate site. DAS can learn the intervention subspace within a selected location(s). PLOT-DAS illustrates this combination. We will revise the motivation to emphasize this perspective and complementarity, while distinguishing the layer-by-layer DAS implementation used in our experiments from the broader capabilities of DAS and its extensions. We will also clarify that multilayer correspondence is not unique to PLOT, since DAS can operate on representations constructed from multiple locations.

The proposed decomposition offers another way that DAS could inform localization across layers. In particular, it attributes the source–base difference within a late-layer DAS subspace to contributions from earlier residual updates, potentially identifying candidate sites for intervention. These attributions would still need to be translated into intervention handles and evaluated. Indeed, the decomposition uses the original source and base computations, whereas intervening earlier generally changes subsequent residual updates. We will acknowledge this possibility alongside the broader capabilities of DAS discussed above, and clarify that PLOT provides a complementary matching-based approach to discovering candidate handles.

DAS does hard assignment on a learned subspace but optimal transport used in PLOT does soft assignment so that it is not sure whether they can be compared.

We think that the reviewer is making an excellent point that high IIA alone is insufficient to certify that a neural network implements a proposed causal model. Establishing this claim requires additional consistency conditions, e.g., that the neural handles reproduce the causal relationships between high-level variables, rather than just output IIA. Recent work introducing Causal Abstraction Error (CAE) reinforces this distinction, explicitly identifying limitations of output-only evaluation and incorporating additional consistency and faithfulness tests (Méloux et al., 2026). In fact, we are currently working on an algorithm for boosting the internal faithfulness of causal abstractions algorithms, such as DAS and PLOT.

That said, the present paper addresses the narrower task of discovering neural intervention handles that reproduce the output effects of high-level interventions, evaluated through held-out IIA. This follows an established evaluation practice, including the causal-variable localization track of MIB (Mueller et al., 2025). As stated in Section 2.1, “High empirical IIA supports the proposed correspondence on the evaluated pairs, but does not by itself establish an exact or constructive abstraction.” Accordingly, our comparison of PLOT and DAS concerns their effectiveness under this common output-level criterion, not a claim that PLOT’s weighted interventions and DAS’s hard subspace swaps satisfy identical algebraic constraints.

We acknowledge that this distinction could be more prominent in the manuscript. We will clarify the task definition and scope of our claims, explicitly distinguishing high-IIA handle discovery from certification of a faithful causal abstraction.

It seems to be exaggerated to criticize the runtime of DAS and justify the time efficiency of PLOT...

We acknowledge that the current presentation overemphasizes the runtime comparison between PLOT and DAS, and we will change this in the revision. The main goal of these experiments is to examine the benefit of separating site discovery from subspace learning. We therefore compare a full DAS sweep, PLOT-only methods without learned subspace refinement, and their combination in PLOT-DAS. The combined approach matches full-DAS accuracy on binary addition and retains most of its accuracy on MCQA while requiring substantially less computation. This complementarity is the main message we intend to convey, rather than a claim that PLOT replaces DAS. We thank the reviewer for pointing out where the presentation obscures it.

Regarding the proposed “lowest-cost” screening, we interpret the idea as selecting sites with the smallest distances between abstract and neural effect signatures, using the cost matrix directly without solving OT. This is a reasonable alternative to OT-based screening. Appendix B.1 already compares OT with cosine similarity and direct fit-bank IIA scoring for handle extraction. We will extend these comparisons to DAS guided by non-OT localization methods, including the proposed lowest-cost screening. We will hold the candidate sites, data splits, screening budget, and downstream DAS procedure fixed to distinguish the benefit of screening itself from any additional benefit attributable to OT. If the reviewer intended a different notion of cost, we would appreciate clarification.

We agree that serial runtime alone provides an incomplete picture of computational cost, and thank the reviewer for suggesting operation counts. Our full-DAS searches were split into separate jobs across candidate layers, illustrating how parallel execution can reduce waiting time without reducing the total search workload. Across the five reported MCQA seeds, full DAS trained 520 candidates per seed, compared with 20 for PLOT-DAS. Accounting for early stopping, these searches averaged 1,096,640 and 53,120 training-pair evaluations involving backpropagation per seed, respectively. PLOT-DAS additionally has a coarse-screening grid of 43,600 forward-only intervention-pair evaluations before reuse and 16 transport solves of size 2×262 \times 26. These counts illustrate the computational benefit of screening candidate locations before training DAS. We will include separate accounting for downstream calibration, testing, shared preprocessing, and reuse, and distinguish these workload counts from FLOPs.

There is no Limitations section. Please add this section to the main text, or include it in the Appendix and reference it in the main body. Also please specify them in comments for dicussion.

Thank you for pointing this out. We will add a dedicated Limitations section and reference it from the main text. In particular, we will discuss that high held-out IIA does not certify internal faithfulness or a formal causal abstraction, and that PLOT performance and runtime depends on the candidate sites, effect signatures, calibration grids considered, and implementation details. We will also discuss the computational cost of collecting intervention effects for large candidate families and the scope of the tasks and models evaluated.

There is only DAS for a correspondence method baseline besides basis selections. Please justify in detail or add different baselines both in literature review section and experiments. Possible candidates include general correspondence algorithms and the methods from causal abstraction literature.

In addition to DAS, our submitted experiments include DBM in canonical, PCA, and SAE representations (Table 1). Appendix B.1 further compares OT with cosine similarity and direct fit-bank IIA matching. We selected DAS and DBM based on their performance in the causal-variable localization experiments of MIB (Mueller et al., 2025), where DAS generally performs best and DBM with canonical or PCA features provides a competitive baseline. Following the reviewer’s suggestion, we will also add Boundless DAS as a baseline, and would welcome additional suggestions on other methods they consider important to include.

Why does PLOT need the intervention coefficient lambda, while UOT itself can express non-uniform weights? What different roles do they take?

UOT and λ\lambda serve different purposes. UOT relaxes the neural marginal, allowing different neural sites to receive different amounts of total matching mass. In particular, sites that are irrelevant to all high-level variables can receive negligible mass, rather than being assigned a prescribed share as in balanced OT. Its role is therefore to identify relevant correspondences without forcing every candidate site to participate.

By contrast, λ\lambda calibrates the intensity of the resulting intervention. Once the sites and their relative weights are determined, λ\lambda scales the entire intervention profile to find the strength that best reproduces the abstract counterfactual effects on calibration data. Thus, UOT governs correspondence, while λ\lambda governs intervention strength.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.