- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work presents a method called contextual control, which learns a set of control signals that modulate the magnitude of the hidden activations at feed-forward blocks of a frozen transformer. They show that these control signals improve perplexity once learned on particular domains and reflects contextual information (superior to token-only modulations and noise baselines).
Critical strengths
- The experiments are comprehensive, providing valid evidence for getting the control signal modulation to work and testing on different models and benchmarks.
- The ablation experiments are also quite comprehensive, covering different baselines (token-only controls and multiple reassignment baselines) and scenarios (e.g. neutralization). Section 7's investigation of the small benefit on qwen on PG-19 is also a nice addition.
- The motivation/connection to computational neuroscience is interesting.
Critical weaknesses
- My biggest concern, and question after reading, is what insights we learn from these control signals. Beside a loose motivation from neuroscience and overall evidence that we can get these signals, I find the paper significantly lacking justification of why this method or these results are useful, especially beyond other interpretability methods that establish the contextual influence of token representations. If the control signals are more intended for interpretability, actual interpretable insights are missing from the write-up. The paper states "contextual control introduces an explicit, bounded signal whose neutral value exactly recovers the frozen model", however, it's unclear what we get from an explicit but obscure control signal.
- The writing is quite hard to follow, and I think a large part is due to references of concepts like "cohort", "donor", "gain", "endpoint" that are not well-defined nor well-established.
Questions for authors
- Could you clarify the practical value of this method?
- Can you share the motivation behind the contiguous grouping assumption?