acceptodds
Under review as a conference paper at ICLR 2027

Content-Dependent Recovery from Task-Relevant Activation Disruption in Large Language Models

Abstract

When prompts are modified or rewritten, large language models (LLMs) need to determine which information in the prompt remains valid in order to continue completing the specified task. Prior work has examined how models process such changed information, but what happens to the model when its internal computation is disrupted remains largely a black box. We gradually attenuate internal activations at task-related token positions, select intervention strengths from the resulting performance curves, and write activations recorded during normal runs back into the perturbed model. Experimental results across five open-weight models and four reasoning tasks show that reasoning performance drops more when task-related information is disrupted, while nearby layers can show similar effects. When activations carrying the correct information are written back, the models can recover their reasoning performance. We further find that recovery in reasoning does not simply correspond to internal activations returning toward their normal state: incorrect information can also move perturbed activations closer to the normal state while producing substantially weaker recovery of the receiver's correct answer. This pattern is observed across all five models, while component analyses show model-dependent effects, with the clearest effects occurring at different attention and feed-forward sites.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Mechanistic Interpretability, Internal Activations, Activation Intervention, Activation Patching, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zHTYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether a model’s answer after activation disruption depends on the information carried by the replacement activations. The authors identify layer and token positions where attenuation hurts performance, then patch in activations from clean runs that support either the receiver’s correct answer or a different answer. They test this across five models and several controlled information-update tasks. Correct-answer sources, including paraphrased ones, substantially restore candidate-ranking accuracy, and different-answer sources do not. The authors also claims that different-answer source can still move the measured activation closer to clean activations without recovering the correct answer. The authors validate the effects on separate evaluation prompts and compare against alternative positions, nearby layers, and random vectors.

Critical strengths
  1. The experiments are thorough. The authors test several models and tasks, evaluate on prompts separate from those used to select intervention sites, and include a useful range of controls. This gives me reasonable confidence that the reported effects are repeatable within this setup.

  2. The intervention and the measures are specified precisely. The math is well defined. I could follow what is being replaced, where it is replaced, and how recovery is scored.

Critical weaknesses
  1. I found the main recovery result quite predictable. The authors select sites because damaging them lowers answer scores, then often restore those same sites with activations from a run supporting the correct answer. It is also expected that patching activations supporting a different answer will do worse. The experiments establish that this pattern generalizes across the tested prompts, but I am not sure what new mechanism they reveal.

  2. I am not convinced that the activation-distance result supports a strong claim about recovery. The incorrect-state source is still a clean run of a very similar prompt. It uses the same task format and answer labels, but changes which answer is current. Patching its full activation vector into a disrupted run could therefore reduce the measured distance simply because it replaces damaged activity with a normal-looking vector from the same class of prompts(these prompts usually result in a latent space with similar geometry). The norm-matched random control does not rule this out( random vector cannot model this geometry). I would want to see a clean-source control that is similar in format but unrelated to the receiver’s task state. Without that comparison, the result shows that activation distance can improve while the answer remains wrong, but does not tell us much about what aspect of the activation is responsible for recovery.

Questions for authors
  1. The sites are selected because perturbing them hurts answer accuracy. What do these sites tell us about how the model handles the current task state, beyond identifying places where it is sensitive to perturbation?

  2. What is the main insight from the recovery experiments? It seems expected that activations from a prompt supporting the correct answer help recover that answer, while activations supporting a different answer do not. What does this comparison establish that was not already implied by the setup?

  3. Could the activation-distance result reflect the shared geometry of normal activations? Both correct and incorrect state sources come from clean runs of similar prompts, so either might move a disrupted activation closer to the clean references. What evidence distinguishes this explanation from the one you propose?

  4. What is the intended use of these findings? what do they add to our understanding of the model’s internal mechanism, and is there any path from this method to improving model performance when the correct answer is not known in advance?

Reviewer_DXiYOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The clear distinction between activation similarity and answer correctness is the strongest contribution and my main reason for supporting acceptance. Specifically, activations from incorrect sources can minimize the measured distance to unperturbed states while steering the model toward incorrect outputs. This provides a valuable cautionary insight for interpreting activation interventions, emphasizing that representational measurements must be evaluated alongside their behavioral outcomes. Demonstrating this phenomenon consistently across five different models further strengthens the empirical validity of the results.

Critical strengths

The clear distinction between activation similarity and answer correctness is the strongest contribution and my main reason for supporting acceptance. Specifically, activations from incorrect sources can minimize the measured distance to unperturbed states while steering the model toward incorrect outputs. This provides a valuable cautionary insight for interpreting activation interventions, emphasizing that representational measurements must be evaluated alongside their behavioral outcomes. Demonstrating this phenomenon consistently across five different models further strengthens the empirical validity of the results.

Critical weaknesses
  • Both primary distance metrics rely on a fixed 128-dimensional Gaussian random projection. While the standardized Euclidean nearest-neighbor metric nicely complements cosine distance, both ultimately depend on this lower-dimensional projection. It remains unclear whether the observed pattern holds in the unprojected activation space or across different projection dimensions and random seeds. Given that this finding is central to the paper's thesis, proving its robustness is crucial.
  • Although the paper outlines the selection procedure, it does not provide a comprehensive breakdown of the chosen attenuation sites and replacement layers for each model–task pair. Consequently, it is difficult to evaluate whether different tasks recruit consistent layers or if the selected locations vary substantially across settings. Clearer documentation of these intervention sites would help contextualize the scope and generalizability of the findings.
Questions for authors
  • Does the decoupling between representational distance and behavioral recovery hold under complementary metrics, such as paired Euclidean distance in the unprojected space or linear CKA across matching prompts? Additionally, would a regularized linear regression trained to predict clean activations from intervened activations (evaluated on held-out samples) yield similar trends? These checks would help clarify which latent properties are restored by incorrect-source replacements despite poor behavioral recovery.
  • Could you provide a detailed mapping of the selected attenuation layer–token-group pairs and replacement windows for each model–task combination? Specifically, I would like to understand the degree of overlap in selected layers across tasks within the same model and whether they consistently fall within similar depth ranges. This would clarify whether the observed phenomena reflect shared computational bottlenecks or task-specific intervention sites.
Reviewer_rLHgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks what information internal activations must carry for an LLM to recover after its task-relevant computation is disrupted. The tasks are synthetic state-tracking problems in which a single prompt gives an initial fact followed by revisions (e.g., custody transfers, signed corrections). The authors conclude that the content of the source, not just activation similarity, determines recovery.

Critical strengths

The problem this paper studies matters in practice. A model must judge which information in its context is still valid after revisions, and use only the current state. This is a core requirement for multi-turn assistants and LLM agents, which routinely face changed user requirements, overridden configurations, reverted operations, and superseded instructions within one context.

Studying this under controlled internal disruption complements purely behavioral evaluations of state tracking. The rule-generated task families have deterministic answers and controlled revision structure, so they are easy to scale. They could also be reused beyond this paper, for example as synthetic data for state-tracking evaluation or training.

I also appreciate that the paper treats answer recovery and activation similarity as separate questions rather than assuming one implies the other. It also covers five open-weight models, including relatively recent ones (Qwen3-14B, Gemma-4-12B-IT).

Critical weaknesses
  1. The authors cite Wang et al. (2025)—a study that simulates functional impairment in Mixture-of-Experts (MoE) models and restores performance by retraining the unaffected expert modules—yet the current paper evaluates only dense models. In an MoE architecture, applying attenuation (Equation 3) and subsequent restoration (Equation 4) to residual vectors alters how the router selects expert modules. Consequently, the impairment and restoration process conflates two distinct effects: changes at the content level and changes at the routing level. Given that routing decisions may shift, does the conclusion that "source content determines the effectiveness of restoration" still hold?

Although the introduction mentions multi-turn evaluations concerning information revision (Atri et al., 2026), the text later states: "We place the relevant records into a single prompt." The revision process described does not span multiple conversational turns, nor does it involve tool calls or environmental feedback. In real-world applications—particularly with code agents—tracking the current state of a task is precisely what makes such scenarios critical.

  1. There exists a more natural alternative that is equally relevant to the research question: state updates within code. Examples include variable reassignment, configuration overrides, the shadowing of earlier definitions by later ones, and the reversal of commits via git revert. Contemporary Large Language Models (LLMs) have been extensively trained on such data. Furthermore, these models provide deterministic outputs; the scenarios mentioned above represent precisely the types of contexts where "state tracking" is vital for practical deployment. It is therefore worth considering the domain of code.
Questions for authors

please refer to the weakness section

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.