- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether a model’s answer after activation disruption depends on the information carried by the replacement activations. The authors identify layer and token positions where attenuation hurts performance, then patch in activations from clean runs that support either the receiver’s correct answer or a different answer. They test this across five models and several controlled information-update tasks. Correct-answer sources, including paraphrased ones, substantially restore candidate-ranking accuracy, and different-answer sources do not. The authors also claims that different-answer source can still move the measured activation closer to clean activations without recovering the correct answer. The authors validate the effects on separate evaluation prompts and compare against alternative positions, nearby layers, and random vectors.
Critical strengths
-
The experiments are thorough. The authors test several models and tasks, evaluate on prompts separate from those used to select intervention sites, and include a useful range of controls. This gives me reasonable confidence that the reported effects are repeatable within this setup.
-
The intervention and the measures are specified precisely. The math is well defined. I could follow what is being replaced, where it is replaced, and how recovery is scored.
Critical weaknesses
-
I found the main recovery result quite predictable. The authors select sites because damaging them lowers answer scores, then often restore those same sites with activations from a run supporting the correct answer. It is also expected that patching activations supporting a different answer will do worse. The experiments establish that this pattern generalizes across the tested prompts, but I am not sure what new mechanism they reveal.
-
I am not convinced that the activation-distance result supports a strong claim about recovery. The incorrect-state source is still a clean run of a very similar prompt. It uses the same task format and answer labels, but changes which answer is current. Patching its full activation vector into a disrupted run could therefore reduce the measured distance simply because it replaces damaged activity with a normal-looking vector from the same class of prompts(these prompts usually result in a latent space with similar geometry). The norm-matched random control does not rule this out( random vector cannot model this geometry). I would want to see a clean-source control that is similar in format but unrelated to the receiver’s task state. Without that comparison, the result shows that activation distance can improve while the answer remains wrong, but does not tell us much about what aspect of the activation is responsible for recovery.
Questions for authors
-
The sites are selected because perturbing them hurts answer accuracy. What do these sites tell us about how the model handles the current task state, beyond identifying places where it is sensitive to perturbation?
-
What is the main insight from the recovery experiments? It seems expected that activations from a prompt supporting the correct answer help recover that answer, while activations supporting a different answer do not. What does this comparison establish that was not already implied by the setup?
-
Could the activation-distance result reflect the shared geometry of normal activations? Both correct and incorrect state sources come from clean runs of similar prompts, so either might move a disrupted activation closer to the clean references. What evidence distinguishes this explanation from the one you propose?
-
What is the intended use of these findings? what do they add to our understanding of the model’s internal mechanism, and is there any path from this method to improving model performance when the correct answer is not known in advance?