- Rating
- 2
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies overtopping in LLMs, where interventions on individual activation channels affect model behavior across many examples. It relates a channel’s causal influence to task competence, tests how channels interact and how effects change with intervention strength, and tracks whether influential channels shift during training. A controlled poisoning experiment further tests whether such channels can suppress unwanted behavior. The results show that joint interventions often weaken individual effects, behavioral changes frequently exhibit persistent thresholds, and effective intervention targets can change during training.
Critical strengths
-
The paper provides a detailed empirical analysis of activation interventions across several tasks and models. The experiments go beyond identifying influential channels and examine their interactions, intervention strength and timing.
-
The findings on channel interactions and changes during training are useful. In particular, the results show that combining individually effective interventions may reduce their effects, and that intervention targets identified at one checkpoint may become ineffective after further training.
-
The paper is organized around four clearly stated research questions, and the limitations section is candid about construct, internal, and external validity. That made it easier to see what the evidence does and does not support
Critical weaknesses
-
The evidence for a distinct spiking-like mechanism is not so convincing. Threshold-like changes in binary outputs can arise when smoothly changing internal representations cross a decision boundary. Although the authors examine logit margins and intervention timing, these results do not clearly distinguish the proposed interpretation from this simpler explanation. The finite-grid threshold correspondence also does not establish a distinct computational mechanism. The authors themselves acknowledge in Section 7 that a smooth latent trajectory can produce the same threshold pattern when it crosses a decision boundary. If so, it's unclear what the spiking analogy explains beyond that simpler account.
-
The practical implications of the intervention results is limited. The defense experiment focuses on a single controlled poisoning setting and uses a matched clean reference model. The current-checkpoint intervention results are encouraging, but they do not establish whether the approach would work under more realistic conditions or remain effective after further training.
-
The paper is dense and difficult to follow. There are so many terms, and they are introduced quickly. Maybe a notation table would help readers. I got lost while reading and had almost forgotten those terms by the time I reached Figure 4.
Questions for authors
-
What evidence separates the proposed spiking-like behavior from ordinary decision-boundary effects caused by continuous changes in model activations?
-
Do high-leverage channels exhibit threshold properties that differs from those of other channels with comparable causal strength?
-
Can the adaptive intervention approach work without a matched clean reference model? How well does it generalize to other poisoning settings or later training checkpoints?
4 comments
Thank you for the rebuttal. My two main concerns remain, so I keep my score.
The spiking-like threshold is the paper's central claim, and the evidence still does not separate it from an ordinary decision-boundary effect. The response describes the term as a descriptive analogy, and the paper itself notes that a smooth latent trajectory can cross a decision boundary in this way (Section 7). A single persistent crossing is also expected when one channel is interpolated linearly, and the peak margin change at the crossing is only 4.8 pp more frequent than at adjacent intervals. The answer to Q2 points the same way. Matched controls with comparable causal strength show the same threshold behavior, so the threshold seems to follow causal strength and is not specific to overtopping channels.
The new experiment drops the clean reference model but relies on known trigger information and attack prompts. This is conflicts the principle set earlier. Section 6 states that channel selection does not use trigger annotations or attack outcomes. It also does not yet appear in the revised PDF. On later checkpoints, the response confirms that previously selected targets have negative defense leverage, and the paper notes that same-checkpoint selection does not establish prospective transfer. The defense works only when targets are reselected and evaluated at the same checkpoint, and it seems not to hold up as training continues.
We want to clarify several inaccuracies in Reviewer Un4a's response and interpretation of our results, particularly since this discussion will remain public.
-
Spiking-like behaviour. The reviewer states that "the evidence still does not separate it from an ordinary decision-boundary effect." However, the paper explicitly goes beyond binary decision-boundary crossings, as shown by the continuous logit-margin and temporal analyses in Section 5 (Fig. 4a–c), further clarified in our rebuttal. The reviewer also argues that "threshold-like changes in binary outputs can arise when smoothly changing internal representations cross a decision boundary." This repeated framing of smooth nonlinear decision-boundary effects as an alternative to spiking-like behaviour is misleading: even spike initiation in the EIF model arises from smooth nonlinear dynamics. Our claim concerns spiking-like response geometry, not a distinct internal spiking mechanism. Moreover, dismissing the 4.8 pp result as "only 4.8 pp" overlooks that this measures excess event localization frequency, not the magnitude of the causal effect. The result is statistically significant (p < 0.0001) and is complemented by additional statistically significant evidence of margin-change concentration and temporal localization (Fig. 4b–c).
-
Matched controls. The reviewer concludes that threshold behaviour "is not specific to overtopping channels" because "matched controls with comparable causal strength show the same threshold behavior." However, our answer to Q2 explicitly states that controls with comparable high singleton causal strength are, by definition, also overtopping channels, regardless of how they were discovered. Therefore, the reviewer's conclusion does not follow logically from the comparison: the comparison does not establish threshold behaviour outside overtopping, only outside CHA-discovered candidates.
-
Poisoning experiments. These experiments are explicitly presented as a proof of concept. The reviewer states that the additional experiment using known trigger information "conflicts the principle set earlier." However, this experiment was specifically conducted to remove the clean-reference requirement in response to the reviewer's question, under different experimental assumptions. It does not invalidate the original experiments. Furthermore, the reviewer argues that "the defense works only when targets are reselected and evaluated at the same checkpoint" and "seems not to hold up as training continues." This is precisely one of the paper's main findings: static interventions become unreliable as training continues. We therefore do not see how restating this result contradicts the paper's claims.
Finally, we have not uploaded a revised PDF. All additional experiments and clarifications were provided in the rebuttal. The reference to a revised PDF is therefore incorrect, as no such version exists.
Thank you for the detailed clarifications.
They resolve several of my earlier points, in particular the meaning of the 4.8 pp result and the status of the matched controls. However, the paper still does not show what the term "spiking-like" adds beyond a sharp nonlinearity near the decision boundary, since Figure 4 uses equal-change and adjacent-interval references but no smooth nonlinear baseline.
The practical implications of the defense experiments also remain limited, since the new setting assumes known triggers and attack prompts, and the paper presents the defense as a proof of concept without prospective transfer.
The BackdoorLLM experiment in the response may remove the need for a matched clean model, but it is not in the current paper because no revised PDF has been uploaded, so I cannot credit it and take it into account in my assessment.
Q1
We use “spiking-like” to describe measured response properties, without claiming neuronal spiking or EIF dynamics in LLMs. Just as spike initiation in the EIF model arises from smooth nonlinear dynamics, continuous changes in model activations could produce abrupt output transitions consistent with a spiking-like analogy. Smoothness therefore does not invalidate this descriptive analogy. Smoothness alone therefore does not invalidate this descriptive analogy.
Overtopping channels are identified by behavioral effects, so output changes alone is not what motivates our use of “spiking-like”, as that would be tautological. Fig. 4a shows that the largest absolute logit-margin change between consecutive intervention strengths occurs 4.80 pp more often at the persistent behavioral transition (median difference; 95% CI [2.14, 6.40]). Fig. 4b shows that the same largest absolute logit-margin change significantly increases with causal strength: in the strongest channels, the largest of ten intervention intervals accounts for 15.99% of total absolute margin change (median across conditions), versus a 10% equal-change reference. Fig. 4c uses complementary temporal sweeps to localize behavioral effects during decoding.
These specific response properties motivate our use of “spiking-like”, and are not guaranteed by construction but empirically observed in the overtopping channels population. Our interpretation, formalized in Appendix D.2, concerns this spiking-like response geometry without implying that LLMs implement EIF dynamics or an internal spiking mechanism. Whether they reflect a shared computational principle or different mechanisms remains open for future research. We will better clarify that “spiking-like” denotes these measured response properties.
Q2
At comparable high causal strength, control channels are as well overtopping high-leverage channels, regardless of whether CHA discovered them. “Comparable causal strength” means comparable held-out singleton causal reach: the fraction of direction-eligible examples flipped by full replacement, within the same run/direction condition.
We matched candidates with same-locus controls on causal strength and compared their endogenous threshold association (Appendix D, Figure 11). Across 136 matched pairs in six conditions, we found no consistent candidate–control difference. Candidates do, however, show greater causal strength and a higher threshold-testable fraction than randomly sampled same-locus controls (Appendix D, Figure 10).
Q3
Yes. We do not need a matched clean model, provided we have access to a sample of benign/attack prompts and known trigger information.
During rebuttal, we conducted an additional poisoning-defense experiment using BackdoorLLM’s independently trained Llama-2-7B BadNets LoRA adapter [2]. We applied CHA without a matched clean model. This extends our Grammar evaluation to another (larger) model, poisoning attack, and safety-related, open-ended generation task. Using 50+ BackdoorLLM attack/benign prompts, we identified 20 candidate channels; 17 showed positive defense leverage (+10.0 to +32.1 pp; i.e., attack suppression minus benign damage). To test transfer to different prompts, we selected three highest-leverage channels and evaluated them on 100+ HarmBench harmful/benign prompts. Defense leverage was +21.0 (m1:3135), +19.0 (m1:2469), and +12.9 (m1:2350). Notably, each intervention replaces a single channel.
Regarding generalization across training checkpoints, our original submission investigates this question. Across three Grammar poisoning runs, current-checkpoint selection achieves median singleton defense leverage of +6.2, +44.9, +36.5, and +23.5 pp at 25%, 50%, 75%, and 100% of training, respectively. In contrast, targets selected at previous checkpoints exhibit negative median defense leverage (Fig. 6 and Appendix Table 9). Together with RQ1–RQ3, these results show that high causal reach alone does not make a fixed intervention reliable. Targets should therefore be reselected and validated at each checkpoint being defended (which is practical though, as explained in our responses to Reviewer honh).
Finally, we emphasize that the matched clean reference is an assumption of our controlled fine-tuning setting, not an intrinsic requirement of CHA. Similar reference-based assumptions appear also in prior work [3,4].
References