acceptodds
Under review as a conference paper at ICLR 2027

Interpreting Model Reasoning through Behavioral Counterfactuals

Abstract

Faithfully understanding the internal process that occurs as models form their chain of thought is an important problem with the rise in model capabilities. However, this remains challenging due to the variance in sampled responses, making it difficult to isolate the effects of interventions and compare model reasoning systematically. We propose a method for localizing and analyzing how a given intervention affects the model's reasoning through behavioral counterfactuals: given a fixed response, how does introducing an intervention to the prompt change the next-token distributions? We train a module to edit the residual stream such that it minimizes the KL divergence between the output distributions for a fixed response under a source prompt and its counterfactual target prompt. We find that matching the output distributions recovers the counterfactual behavior more effectively than standard activation reconstruction objectives and validate that attribution of the trained module outputs is effective for identifying causally important locations for counterfactual recovery. The attribution maps and visualizations of the downstream effects of model edits provide an interface for generating hypotheses about the original model's computation, which can then be tested with activation patching. In a case study on Gemma-4-E4B-it and DeepSeek-R1-0528-Qwen3-8B, we investigate how a suggestion influences model reasoning even when it is not verbally acknowledged. Our analysis identifies representations at the question statement and filler tokens that mediate a substantial part of the suggestion's influence, with downstream effects such as changing the framing of the justification or even the final answer.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, chain of thought, AI safety

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BSSPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how intervention affects model output by counterfactuals. The authors propose to insert an edit moule at the residual stream and minimize the KL div between the source and target prompt output distribution.

Critical strengths
  • The reconstruction caomparison is insightful.
  • Holding response fixed is reasonably clean.
  • THe analysis and diagnostics and thorough.
Critical weaknesses
  • As most findings are for the midpoint layer (e.g., Section 3.1), the position where the edit module is applied needs further investigation, e.g., with layer sweep or sensitivity analysis. At least one more dataset should be included, performing experiments only on Scruple is not sufficient.
  • In my opinion, the localization baseline is missing. Since the response is fixed and the token positions are alreayd well-aligned, why don't you just compute the position-wise activation difference and use this as a baseline? Furthermore, I was wondering whether fixing the response will lead to any off-policy issue, as the model may not generate such a trajectory.
Questions for authors
3 comments

We thank the reviewer for finding the reconstruction comparison insightful, found the setup clean, and the analysis thorough. We also appreciate the thoughtful feedback which we address below.


[W1-1] Layer sweep/Sensitivity Analysis

We agree that experiments on different layers would be helpful. We used the midpoint as a common intervention site, since there are several downstream layers available to propagate the edit. Moreover, prior work on activation steering, such as [1], found that interventions at intermediate layers can effectively steer a range of behaviors.

As suggested, we further tested how sensitive our findings are to this choice of layer. We trained edit modules at layers with index 3m3m with m=0,…,13m = 0, \dots, 13 and measure the KL recovery for each of the models and have included these results in the revision. We found that recovery is highest at intermediate layers, and we see a steady decrease in recovery from layer 15 to 39, suggesting that results are relatively stable across layers. We will include these in a revised manuscript.


[W1-2] More datasets

We agree that evaluating on another dataset is important. As suggested, we added an evaluation on WMDP [2] using the sandbagging scenario from [3]. The intervention is a statement that the model will not be deployed if it demonstrates advanced knowledge in biology, chemistry, or cybersecurity and is provided with a possible incorrect answer. We provide results for Gemma below.

ModelActivation RecoveryBehavioral Recovery
Gemma + Behavior48.8%92.6%
Gemma + Activation37.5%-0.2%

Our method recovered substantially more counterfactual behavior than activation reconstruction, consistent with the main finding on Scruples. We also computed attribution maps using the edits and Integrated Gradients and found that within the intervention, the area most relevant to the question out of biology, chemistry, and cybersecurity has high attribution scores compared to others. As in Scruples, whitespace and punctuation tokens received high attribution in the generated reasoning. This demonstrates a shared structure across datasets. The results have been added to the revised version.


[W2-1] Activation Difference Baseline

Thank you for the great suggestion. This is a useful baseline to have, and we have run experiments using the activation differences as edit vectors. We found that using the activation differences directly covers 64.7% of counterfactual behavior for DeepSeek and 70.6% for Gemma. Thus, exactly matching the counterfactual activations at the edit layer recovers only part of the counterfactual output behavior. Importantly, our learned edit module adds two advantages:

  1. Attribution to prompt and intervention tokens. The raw differences define edits at aligned response positions. When treated as fixed vectors, they provide no attribution path from those edits back to the prompt tokens that produced them. Our learned module provides that path. In the sandbagging experiment, for example, the domain named in the intervention that matched the question received higher attribution than the other domains.

  2. Compact modeling of the edit trajectory. The baseline computes each difference by running both prompt conditions through the first half of the Transformer (using billions of parameters). Our edit module uses substantially fewer parameters than that upstream computation to capture patterns in how the edits vary across response positions and examples. This gives us a compact model of the changes that matter for recovering counterfactual behavior.

We will add this discussion to the revised manuscript.


[W2-2] Fixed Response

We would like to provide some clarification about the generations and how the edit model is both trained and evaluated. All the generations are sampled from the model's distribution and as a result are a form of online behavior. Specifically, every response is first sampled autoregressively from the model. We then hold the sampled response fixed when comparing the control and intervention conditions, so both runs are evaluated on the same response prefix at each token position.

Furthermore, to help with generalization, for each prompt, the edit model is trained with both a response generated from the same prompt as well as a response generated from the counterfactual prompt. This way, the edits can be applied to both responses that are unchanged as well as responses that look more like the target behavior.

We also ran an additional experiment on a subset of the test set to see how the edits affect fresh generations. We applied the edits during autoregressive sampling and measured how often the model judged the author to be in the wrong. For both models, edits that simulate adding the intervention increased this rate, while edits that simulate removing it decreased the rate.

ModelDirectionOriginalEdited
GemmaControl to Intervention28%35%
GemmaIntervention to Control32%25%
DeepSeekControl to Intervention27%46%
DeepSeekIntervention to Control42%26%

[Q1] Citation

Thank you for pointing us towards this reference, this would be a great addition, and we have added it to the related works.

[1] Panickssery, Nina, et al. "Steering llama 2 via contrastive activation addition." 2023.

[2] Li, Nathaniel, et al. "The wmdp benchmark: Measuring and reducing malicious use with unlearning." 2024.

[3] Guan, Melody Y., et al. "Monitoring monitorability." 2025.

Dear Reviewer BSSP,

Thank you again for taking the time and effort to review our paper! A quick reminder: the discussion period closes today (November 18). If you have any remaining questions, we'd be happy to address them before the deadline!

Best regards, The Authors

Reviewer_5xzKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how prompt interventions affect model reasoning. The authors compare next-token distributions (neutral) and intervention prompts while keeping the response fixed. They then train an editing module to reproduce counterfactual output distributions through modifications of intermediate activations (by optimizing KL-divergence between the neutral and counterfactual distributions). They also use Integrated Gradients and ablations to identify influential token positions and validate the findings through native activation patching. Experiments on Gemma-4-E4B-it and DeepSeek-R1-0528-Qwen3-8B suggest that question and filler-token representations can mediate the influence of a user's suggestion on subsequent reasoning and final judgments.

Critical strengths
  • The paper tackles an important interpretability problem: understanding how prompt interventions affect model reasoning, including influences that are not explicitly acknowledged in the generated chain of thought. I find the proposed behavioral counterfactual framework particularly interesting, as it enables controlled comparisons of next-token distributions without relying on alignment between independently sampled reasoning trajectories. This provides a potentially useful approach to investigating prompt-induced influences on internal computations.

  • The authors compare different reconstruction objectives and editing architectures, examine the relationship between attribution scores and ablation effects, and validate their findings through native activation patching. I particularly appreciate that the analysis goes beyond attribution visualizations to provide causal evidence in the original models. These experiments strengthen the methodological contribution and demonstrate that the approach can identify causally relevant internal representations.

Critical weaknesses

While the paper has certain merits and includes an interesting methodological study, there are certain weaknesses that limit its current validity and robustness. Along with some clarifications about the paper's applicable contribution (what does one understand by observing the outputs of your method?), I think addressing these weaknesses could significantly improve the paper. Here I mention 2 of the major weaknesses I have identified:

  • The experiments rely on a single fixed suggestion across different ethical scenarios. Without comparisons against neutral prompt variations (e.g., paraphrases or neutral text additions), it is unclear whether the observed distributional differences and attribution patterns reflect the suggestion's semantic content or general sensitivity to prompt modifications. Furthermore, evaluating only one formulation and direction of the suggestion limits the generalizability of the findings. The experiments suggest the influence of this particular prompt modification, but do not provide a controlled experiment to show that the distributional differences are significantly different from distributional differences between similar prompts that lack this suggestion bias, and at the same time provide limited evidence that the identified patterns characterize suggestion-induced bias more broadly, beyond this specific setup.

  • The editing module recovers approximately 48–50% of the counterfactual KL divergence, but behavioral reconstruction does not necessarily reproduce the original model's computational mechanisms. Edits at different token positions may also encode overlapping information (e.g., similar directions in the residual stream), interact, or compensate for one another. Individual edit ablations measure contributions conditional on the remaining edits, without fully accounting for these dependencies. Group ablations or approximate Shapley-based attribution could help investigate such interactions without requiring exhaustive evaluation of all combinations. Although native activation patching provides valuable causal validation, it does not establish that the learned attributions uniquely or comprehensively characterize the underlying mechanisms. The reliability and completeness of the resulting token-level explanations therefore remain uncertain.

Questions for authors
  1. Could similar distributional differences and attribution patterns arise simply from changes in prompt wording, rather than the suggestion's semantic content? If the authors haven't checked this, they could consider comparing the observed KL divergence against that induced by neutral prompt variations, such as semantically equivalent paraphrases or neutral text additions.

  2. Have the authors examined the similarity between edit vectors at different token positions? Could highly aligned edits indicate that multiple positions carry overlapping information, rather than representing distinct contributions to the counterfactual behavior?

  3. How sensitive are the attribution rankings to interactions or redundancy between edits? Since individual ablations measure the effect of removing one edit while keeping the others fixed, would group ablations provide a different picture of which positions are important?

  4. To what extent does higher KL recovery correspond to more faithful reconstruction of meaningful changes in reasoning or final judgments, rather than lexical or stylistic differences? Have the authors considered evaluating this relationship more systematically?

3 comments

We're glad you find the framework interesting and believe the paper tackles an important problem. We also appreciate your appreciation of the analysis as well as your thorough feedback and questions. We address each below in detail.


[W1/Q1] Prompt Variations

Thank you for raising this important point. As suggested, we have added an experiment to demonstrate that the distributional differences due to the suggestion are larger than those of neutral prompt variations. For the paraphrased prompt, we used a neutral variation going from "Is the author in the wrong?" to "Do you think the author is in the wrong?". We found that the mean KL for responses when using the control prompt vs. the intervention prompt was 0.047 while the mean KL when comparing the control propmt vs. a paraphrased propmt was 0.018. Moreover, we demonstrate in Table 1 that the answer distribution has a noticable change when applying the intervention, suggesting that some of the change is due to the provided suggestion.


[W2/Q3] Group Ablations and Ranking Sensitivity

We appreciate the thoughtful concerns about interactions and redundancy. We would like to clarify that the attribution score itself is not based on individual token ablations but rather Integrated Gradients [1], which does satisfy completeness, and we used a path which involves gradually changing all edits simultaneously.

We have also computed group-based attribution values using KernelSHAP. We use 32 masks to keep the compute budget similar to Integrated Gradients. We find that the correlation between the attribution and individual ablations is weaker with a median of 0.013 for DeepSeek and 0.019 for Gemma. The cost of perform accurate group-based attribution and potentially weak correlation are likely constraints for group-based attribution at scale, especially for model reasoning, where there can be tens of thousands of tokens. Accounting for these effects is an important problem we hope can be addressed in the future.

Lastly, our activation patching experiment suggested that interactions/redundancy is relatively weak. We examined this for two groups highlighted by our attribution analysis: the question tokens and the filler tokens. In the original model, patching both groups together recovered approximately as much counterfactual behavior as the sum of patching each group separately. If the groups were strongly redundant, we would expect the second group to add little once the first was patched. The observed near-additivity suggests that these two groups make largely distinct contributions under the tested intervention.


[W2/Q2] Edit Vector Similarity

We measured the cosine similarity of each token-position edit to the average edit vector. The mean cosine similarity is 0.41 for Deepseek and 0.37 for Gemma. This does also tell us that the edit model is outputting relatively distinct contributions beyond just a fixed vector. However, whether edits are functionally redundant depends on how the model uses them at different positions. Even edits pointing in the same direction can affect different downstream predictions because later layers can attend to those positions differently.

We kindly refer the reviewer to our previous response W2/Q3 on the test for redundancy.


[Q4] Higher KL Recovery

This is a great point, and there are parts of the KL that can correspond to more stylistic differences. We can see in plots such as Figure 7 that the result of activation patching based on the attribution method does correspond to significant changes in the answer likelihoods ("A" versus "B"). We also agree that we can analyze this more systematically and have done so by analyzing the change in the final-answer KL divergence between the source and target prompt with and without the edit applied. We have added the results to the revised draft and provide the results below:

ModelMedian KL DifferenceMedian KL Recovery
DeepSeek0.01380%
Gemma0.0330%

We can see that 80% of the final-answer KL is recovered for DeepSeek by the edits, demonstrating that the edits do correspond to meaningful changes in reasoning or final judgements. We also note that for Gemma, the final answer KL between the source and target prompts was also near zero (on the scale of 1e-6), so the lack of recovery in the final answer is primarily due to the fact that the intervention primarily changes the wording in the reasoning. We provide multiple examples where we can see that ablating/patching a token with high attribution does result in a significant change in the wording distribution. For example, in Figure 6, we see the model shift the distribution for the author's feeling from "suspicious" towards "guilt".

[1] Sundararajan, Mukund, Ankur Taly, and Qiqi Yan. "Axiomatic attribution for deep networks." International conference on machine learning. PMLR, 2017.

Thank you for the detailed response and for taking the time to run certain additional experiments. The neutral-phrase comparison is useful, although I still think that testing more neutral variations would make me more confident in the findings.

I still have some reservations about the token-level attributions and whether reconstructed distributions reflect meaningful changes in reasoning, but since this is not the main focus of the contribution, I consider it a minor issue.

Overall, I find the additional results encouraging, and therefore I raise my score from 2 to 3, though I would encourage the authors to dive a bit deeper into the robustness analysis of their approach, particularly checking whether the findings remain consistent across different prompt formulations and other variations in the experimental setup.

We are glad you found our additional results are encouraging, and thank you for reading our rebuttal and providing the thoughtful feedback. We will continue to work on the robustness analysis, and we are grateful for your support!

Reviewer_cJZ1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new method for interpreting reasoning chains, by training a lightweight editor to edit the activations of a model so that its output distribution matches the output distribution it would've had given a minimally different input (e.g., with/without an opinion expressed in the prompt). Since estimating the full distribution of reasoning chains isn't possible, the authors decide to optimize the editor on token probabilities for fixed continuations, one from the original input and one from the minimally different input. The authors then present a case study showing that edits over whitespace tokens are effective in steering the model's generation.

Critical strengths
  • The paper is clearly written and well explained!
  • The paper focuses on interpretability in long-context reasoning chains, which is an important and relevant challenge. They present a sample-efficient approach that builds on existing works in causal mediation analysis.
  • The paper shows that methods for reconstructing activations can fail to reconstruct behavior (and that methods for reconstructing behavior might not accurately reconstruct reconstructions), which supports their choice of method as opposed to other popular methods in interpretability, such as SAEs
Critical weaknesses
  • The paper lacks baselines and alternative hypotheses for its claims: 1 First, the paper makes the claim that analyzing the model's behavior on fixed continuations can "examine shifts that may not be apparent from sampled text". However, they do not compare this analysis to a resampling based analysis. This is important, since it's unclear how well offline (fixed-completion) analysis translates to online (resampled) behavior on new prompts.
  1. Second, the paper presents a method based on trained activation edits for interpreting a model, and validates that it identifies causally relevant activations through activation patching. This is somewhat unclear. If activation patching is the ground truth, then why learn the trained editor (is it more efficient)? Sankaranarayanan et al. (2026), cited in this submission, could be a good baseline to consider.
  2. Baselines for the token attribution in the case study could also be helpful in interpreting the role of whitespace tokens during reasoning.
Questions for authors
  • The distinction between behavior and activation reconstruction is really interesting! This suggests that for methods that aim to reconstruct activations, it's important to also consider behavior. But also, as Grant et al. (2026) argue, it's also the case that behavior reconstruction could also benefit from activation reconstruction. What do you predict the token attribution in the case study would look like if we considered a method that optimized for both objectives?

Grant, Satchel, et al. "Addressing divergent representations from causal interventions on neural networks." International Conference on Learning Representations. Vol. 2026. 2026.

4 comments

We're glad you found the paper to be clear and find the problem to be important and challenging! We also appreciate the feedback and questions which we address here.


[W1] Baselines and alternative hypotheses

We would like to provide some clarification about the generations and how the edit model is both trained and evaluated. All the generations are sampled from the model's distribution and as a result are a form of online behavior. Specifically, every response is first sampled autoregressively from the model. We then hold the sampled response fixed when comparing the control and intervention conditions, so both runs are evaluated on the same response prefix at each token position.

Furthermore, to help with generalization, for each prompt, the edit model is trained with both a response generated from the same prompt as well as a response generated from the counterfactual prompt. This way, the edits can be applied to both responses that are unchanged as well as responses that look more like the target behavior.

We also ran an additional experiment on a subset of the test set to see how the edits affect fresh generations. We applied the edits during autoregressive sampling and measured how often the model judged the author to be in the wrong. For both models, edits that simulate adding the intervention increased this rate, while edits that simulate removing it decreased the rate.

ModelDirectionOriginalEdited
GemmaControl to Intervention28%35%
GemmaIntervention to Control32%25%
DeepSeekControl to Intervention27%46%
DeepSeekIntervention to Control42%26%

[W2] Activation Edit Validation

We appreciate the great question! We validate using activation patching for a few reasons.

  • The first is as you mentioned that the trained editor can provide a more efficient way for identifying key representations. We can think of learning a trained editor as a way of amortizing the cost of activation patching as it reduces the cost of localization to a few forward/backward passes used for IntGrad rather than possibly tens of thousands of forward passes for every token, allowing interventions to be studied systematically.

  • The second is that we also want to ensure that the edits have some correspondence to the underlying representations and are not simply finding shortcuts or learning a jailbreak-like edit solution, so that analysis based on the edits provides insight into the model computation relevant to the behavior. We chose activation patching at high attribution representations as a way to verify that this is the case.

We also choose to learn an editor using the KL divergence on a fixed response as it allows for studying interventions that do not result in clear verbalized changes as seen especially with Gemma. Contrastive responses like in Sankaranarayanan et al. (2026) are difficult to use for these interventions as the less visible distributional changes mean a significant portion of observed differences may be due to sampling. Furthermore, in such cases, it can be unclear what the effect of an intervention is and we hope that this provides a method for analysis in such settings and settings where the behavior of interest occurs with low likelihood.


[W3] Token Attribution Baseline

We agree baselines for token attribution would be helpful, and test out a simple baseline using Integrated Gradients as attribution, but following the path from zero to the current residual stream to test how attribution looks when the edit is not involved. We found that doing this results in identifying filler tokens as being high attribution as well although the overall set of high attribution tokens is much more generic. We provide the most frequent tokens in the top 5% of absolute attribution scores.

ModelMost frequent tokens among high-attribution events
Gemma\n, [4 spaces], ', [2 spaces], the, **, *, <bos>, ., (
DeepSeekthe, ., in, <|User|>, <|begin_of_sentence|>, and, this, ,, for, A

Based on this, it does seem that filler tokens do have a more general role of carrying information in the intermediate layers, but that they also do carry intervention-specific information given that when doing attribution along the edit, they maintain high-attribution scores along with the question and intervention specific tokens. We will include this in the revised manuscript.


[Q1] Joint Optimization

This is a great idea, and we think adding in an activation reconstruction term would likely help regularize the edits to be more in-distribution for the model, which would also directly address the concern earlier about the edit model possibly learning shortcuts. We think that there is a good chance that this helps with both behavioral recovery and activation recovery as shown in the referenced work Grant et al. (2026) as there is likely a lot of room for improvement in the training dynamics. We will include this discussion and the referenced work in our future work section.

I appreciate the inclusion of the resampling analysis and the additional integrated gradients baseline.

I would still like to see a quantitative validation of the claim that the "set of high attribution tokens (from IG) is more generic".

I also don't quite follow the argument around amortizing the cost of activation patching - are we taking the computation of training the interface edit module into account?

Finally, can you please elaborate on the claim that "Contrastive responses like in Sankaranarayanan et al. (2026) are difficult to use for these interventions"? My understanding is that the fixed response from the control prompt and the intervention prompt exactly constitute this contrastive response pair.

Taking the new baselines and my remaining concerns into consideration, I will keep my decision to accept this submission.

Thank you for the careful reading of our rebuttal and the continued support!

For the quantitative validation of high-attribution tokens based on the baseline IG being more generic, we compare ablating the edit at each of the top attribution tokens using the baseline IG and the edit IG. We find that the average effect of doing so is over 10x smaller for both Gemma and Deepseek suggesting that the representations selected are less intervention-specific.

ModelEdit IGZero-IGRandom
Gemma0.003870.0002640.0000527
DeepSeek0.014860.0006870.000348

Regarding the amortization argument, this is accounting for the computation involved in training as well. The idea is during training each example is involved in roughly a single forward and backward pass, so the average cost of performing the analysis for an example remains a small number of forward and backward passes. This is in contrast to the possibly many passes per example involved in directly localizing representations through token-wise ablation or patching. This average cost remains lower as long as the number of examples used for training remains comparable with the number of examples that are going to be analyzed.

Thank you for this clarification on the contrastive responses. Our discussion around contrastive responses was assuming this referred to two different responses. We have run initial experiments using the attribution patching + diff-in-means combination using fixed responses with hyperparameter tuning on a validation set. We found that the resulting recovery is significantly lower for both models, and will continue validation on the other combinations in Sankaranarayanan et al. (2026).

ModelGCM KL recoveryRandom heads
Gemma23.7%9.2%
DeepSeek19.5%5.5%

We hope these address your remaining concerns, and appreciate your effort in helping improve this paper.

Comments on the paper

1 comment

We would like to thank the reviewers and chairs for their time and effort in providing thoughtful feedback on our work. As reasoning models become more capable, understanding the internal computations that shape their chain of thought becomes increasingly important. Yet this remains challenging due to variance in sampled responses, making it difficult to isolate the effects of interventions and compare model reasoning systematically.

Our paper addresses the challenge by introducing a new method based on behavior counterfactuals: we identify where a prompt intervention exerts its influence within the model and trace that influence through intermediate representations of the chain-of-thought.

We appreciate the reviewers’ recognition of the following strengths.

  1. Studies an important problem in interpreting model reasoning.

"The paper focuses on interpretability in long-context reasoning chains, which is an important and relevant challenge." - Reviewer cJZ1

  1. A novel approach based on behavioral counterfactuals.

"I find the proposed behavioral counterfactual framework particularly interesting, as it enables controlled comparisons of next-token distributions without relying on alignment between independently sampled reasoning trajectories." - Reviewer 5xzK

"Holding response fixed is reasonably clean." - Reviewer BSSP

  1. Experiments that distinguish activation recovery from behavioral recovery and provide causal validation..

"I particularly appreciate that the analysis goes beyond attribution visualizations to provide causal evidence in the original models.” - Reviewer 5xzK

  1. Clear writing and presentation.

"The paper is clearly written and well explained!" - Reviewer cJZ1


We have added additional experiments and discussion to the revision, highlighted in blue.

Changes in the manuscript

  • Sandbagging dataset: We have added experiments based on a different dataset focused on sandbagging where we find a similar relationship between activation and behavioral recovery and attribution identifies key parts of the intervention.
  • Recovering judgement: We have run additional experiments demonstrating the KL recovered by the edits does in fact correspond to changes in the model's final judgement or reasoning.
  • Attribution baseline: We compare our method to performing token-level attribution directly using Integrated Gradients.
  • Layer sweep: We perform a layer sweep for edit model training and find that intermediate layers have the strongest recovery.
  • Additional discussion: We have added additional discussion and citations to the related works and future work sections.

These results strengthen our central contribution: behavioral counterfactuals can help trace how a prompt change shapes a model’s reasoning and judgments, even when its written reasoning leaves that influence unstated. For experimental details and answers to specific questions, please see our individual replies in the discussion threads under the reviews.

Thank you again for your feedback and for managing the review process.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.