- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
setup:
- inference-time activation steering setting
- heuristic technique based pre-computed contrastive vector is already available for a target task (the paper focuses on instruction following constraints and writing style guide)
- vector gets added to the hidden layers of an LM at inference time
problem:
naive addition behaves unevenly from token to token and its not trivial to calibrate it -- two extremes cases are it's hard to get rid of high confident base model tokens and on the other hand you risk running into repetitive loops and other such degenerate patterns.
proposed solution:
- the authors notice that when steering does not change the top token, it often boosts some other tokens by 10x+ relative to the base model, and they try to use this gain in their proposed decoding technique
- as no single "strength" for adding the steering vector works at every position, their method (MAC) runs the base model along with 5 different steering strengths in parallel at each step. it then scores tokens by combining the base log-probability with a weighted average of their gains across the strengths.
- to keep this log-gain score from picking noisy tokens or running into degeneracy, they use several heuristics on top.
- empirically, MAC gets noticeably stronger style transfer than prior baselines (while trading off some meaning preservation), whereas on instruction following its gains are only significant in the setting where the instruction is removed from the prompt.
Critical strengths
- practical set of tricks that make log-gain usable without collapsing into degeneracy. using raw log-prob gain relative to the base model is brittle (because of tail noise and flattened distributions easily blowing up) and the authors propose a nice set of guards that actually make it work in practice. the tricks intuitively make sense and the authors explain / justify with simple token-level observations:
- drop strengths whose distribution becomes too flat or low confidence
- if all remaining strengths still agree on the same top-1 token (from the base model's top 3), output it directly without running the gain score
- restrict scoring to only the top-5 tokens from each branch
- clip negative log-gains to zero and subtract a repetition penalty over the last few tokens
- solid empirical evaluation and ablations within their chosen setup: across 4 models and 2 style-transfer datasets MAC consistently gets stronger style transfer than previous relevant baselines, and the ablations are detailed enough to see what each of the bigger pieces contribute.
Critical weaknesses
-
weak motivation: the authors have a really nice observation (steering can boost runner-up tokens by 10x+ before having an effect on top tokens) but adding decoding heuristics on top of it in a contrived setting feels a bit shallow. the results with the IF setup somewhat hint at this where when the instruction is in the prompt, MAC has no gains. there could be better uses for this observation; maybe as an eval or scaling-law signal to see how models across sizes already surface the right tokens internally before they reach rank 1 or something like that. the paper makes no effort to justify otherwise.
-
i would have liked to see the failure modes (for which several guards were added) folded into the scoring formulation more elegantly. from the ablations it looks like the guards do quite a lot of heavy lifting.
Questions for authors
-
can you evaluate the main baselines with similar guards added as MAC (the usability filter, repetition penalty etc.)?
-
is there a more realistic task beyond style transfer where MAC shows a clear gain when the full prompt is actually given? the IF eval is not convincing.