- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper presents a technique to evaluate whether an activation steering method will be reliable when applied out of the regime where it has been developed. It further proposes a more robust steering technique that can resist deployment shifts.
The evaluation technique is inspired by closed loop methods in neurostimulation, and based on modeling activations through the layers of a transformers as a discrete time dynamical system. Specifically, how much the dynamics deviates from a calibration dynamics, more precisely how this deviation is amplified through the layers, will dictate if the steering approach is reliable.
The authors then define robust steerability, and show that it can additionally identify vulnerable directions in activation space.
The experiments validate the methods and theory across four deployments shift for language tasks.
Critical strengths
This paper provides a refinement on a notion of steerability, moving away from a binary assessment of whether a network can be steerable, but rather to which extent (in terms of distributional deployment shift) it can be steered.
The experiments in four different settings are convincing.
Critical weaknesses
I was excited by the analogy with closed loop stimulation experiments in human brains at first, but ultimately didn’t find the analogy to hold since physical time and layer are conceptually different. The closed loop part is interesting, but I’m not sure the parallel between brains and machines brings something. To be precise, I know that depth-as-time is a classic way of viewing propagation through a transformer. However, depth already has an analogy in biological brains and time already has an analogy in neural networks (tokens). Hence, I feel that the narrative mixes the two.
Figure 3 is too dense and the caption not detailed enough.
Questions for authors
I'm confused about Figure 7, panel B: could the authors explain in more details what it shows, and in particular, why is time now tokens?
3 comments
Thank you for clarifying my concerns
I thank the authors for clarifying my misunderstanding/concerns on the link between neuroscience and AI, and on the figures. I will keep my score.
Thank you for your support and constructive feedback, which helped clarify our NeuroAI contribution. We are pleased that your concerns have been resolved and would be grateful if you would consider updating your rating to reflect your current assessment of the paper.
We thank the reviewer for recognizing the refined notion of steerability and the convincing experiments. We did not expect this level of neuro depth in a review: we optimized the manuscript for an ML-venue audience and the LLM subsections, and our legends' brevity reflects the neuro detail we judged suitable there. Encouraged by this solid NeuroAI review, we are expanding the brain-analogy sections.
We agree with the reviewer: depth has an analogue in the brain (space) and time has an analogue in neural networks (tokens). This is exactly the mapping we intend, and the narrative should not have mixed the two. Part of the confusion came from our notation: the index in the dynamical equation (line 133) and Fig. 3C was written as temporal but should be spatial. This is now corrected. We take layers as the variable of the dynamics and write a dynamical equation for the progression of activity across layers within a fixed time window (here in token). We apply the exact same layer-to-space analogy to the brain: we define a hierarchy over cortical regions (Fig3.D), index them by k, and study the propagation of activity across them for a fixed window of time (Δ). In both systems the dynamics propagate across space, and time is held fixed (physical time in the brain, tokens in LLMs).
We understand the reviewer drew a common analogy from the neurostimulation literature, which mostly models dynamics through time. However, state-dependent closed-loop stimulation is not limited to temporal dynamics. Neural states are inherently spatiotemporal, and their spatial organization can be functionally consequential for stimulation. Manipulating relative phase across cortical regions can enhance or impair cognition [1]. Direction-specific traveling-wave stimulation induces spatially organized responses and distinct cognitive outcomes [2], and TMS-EEG evidence shows that the response to stimulation depends on the ongoing global spatial configuration of activity [3,4]. Effective stimulation thus depends not only on when an intervention is delivered, but also on the spatial organization of the state it acts on. This is exactly analogous to state-based stimulation that uses a "spatial dynamical" equation to estimate the state; older procedures mostly estimate it from "temporal dynamics" at a single recording site.
We borrow only one concept from this literature: state-based closed-loop stimulation is more effective than open-loop. The parallel adds an empirical result: the same spatial-dynamics formalism shows local dynamics in both brains and LLMs (Fig. 7B, Q1). We made the space/time distinction explicit in the introduction and Fig. 3 and Fig.7 caption.
Background for brain–transformer analogy: Large-scale cortical recordings have established traveling waves as a prominent organizing principle of neural spatiotemporal dynamics. Building on this, recent works by Muller, Churchland, Sejnowski et al. has identified potential computational parallels between cortical traveling waves and Transformers, particularly in the spatial encoding of temporal context and attention [5,6]. Beyond superficial brain–LLM analogies, we argue spatial organization of neural dynamics provides a more mechanistically grounded framework than equating transformer depth with biological time. Our work contributes to this framework; a fuller treatment is underway in a separate manuscript.
We expanded the Figure 3 caption to clarify each panel, using the additional camera-ready page.
Once the dynamics are written across layers/electrodes (space), the remaining question is how time is frozen. We take time to be the context length: tokens in LLMs, and the temporal window fed to the spatial dynamical equation in the brain. Panel B shows that in both LLMs and brain regions, the residual in predicting the future spatial state increases with this horizon: the longer the context, the harder it is for the linearized model to explain the layer-wise (or region-wise) dynamics. This means the spatial dynamics is local in both biological brains and LLMs. The time/token horizon is the variable on which we test the spatial dynamics, not the variable along which they propagate. We extended the legend to convey this.
We hope these explanations and improvements address the reviewer’s concerns and are reflected in the final evaluation.
References
1. https://elifesciences.org/articles/22001
2. https://www.pnas.org/doi/10.1073/pnas.2527296123
3. https://www.nature.com/articles/s41467-019-10581-7
4. https://www.jneurosci.org/content/44/39/e0782242024
5. https://www.cell.com/trends/neurosciences/fulltext/S0166-2236(24)00149-8
6. https://arxiv.org/abs/2511.11668