- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper applies DCT across token activations to separate their average from slow and fast patterns of variation, showing that individual frequency components contain concept information. The author's method (SARA) compares these components from recent activations against positive/negative calibration examples and uses the resulting feedback to dynamically adjust a steering direction during generation.
Critical strengths
The DCT analysis shows that individual nonzero-frequency components contain decodable concept information, especially at low frequencies. They also test token shuffling and frequency-band ablations.
Critical weaknesses
- The paper is missing a comparison against simpler temporal summaries, like early-minus-late activations or a linear slope. The experiments don’t isolate the benefit of the DCT specifically.
- Concept prompting is absent from the comparison, despite SARA using explicit concept instructions to construct its online steering direction. AxBench is cited, their paper argue specifically argue about including this control.
- The fine-tuning comparison is limited to 72 positive examples per concept, with rank-4 LoRA applied only to attention output projections at four layers. This limits the result such technique could have on the final model.
- SARA gets much higher concept relevance (good), but lower fluency than other approaches. Without strength sweeps or comparisons at matched fluency, the results don’t establish a better conditioning - quality tradeoff. Sufficiently strong steering can produce those results which can be misleading on which approach is the best.
- Calibration uses the first activation window of each response, but generation applies those statistics to sliding windows throughout the answer. That mismatch isn’t directly tested.
Questions for authors
I would ask the authors to address the main weaknesses above.
2 comments
Response to Reviewer fK2o (Part 2)
(W4)
SARA gets much higher concept relevance (good), but lower fluency than other approaches. Without strength sweeps or comparisons at matched fluency, the results don’t establish a better conditioning - quality tradeoff. Sufficiently strong steering can produce those results which can be misleading on which approach is the best.
The intervention strength controls the balance between concept relevance and fluency. At , Prompting + SARA retains 98.8% of Prompting's fluency and improves by 7.9%, supporting stronger concept conditioning at comparable fluency. The results use Gemma-2-2B-it on all 192 concepts and 960 test responses. The configuration selected using validation results uses , layers 10 and 20, and throughout, with the same generation prompts and decoding as Prompting.
| Method | Concept | Instruction | Fluency | |
|---|---|---|---|---|
| Prompting | 1.1656 | 1.8615 | 1.1740 | 1.0895 |
| Prompting + SARA | 1.3438 | 1.8365 | 1.1594 | 1.1754 |
(W5)
Calibration uses the first activation window of each response, but generation applies those statistics to sliding windows throughout the answer. That mismatch isn’t directly tested.
Our current interpretation is that changes associated with the target concept maintain a relatively stable global structure throughout generation. The first window provides a representative observation of this structure, enabling DCT feedback to generalize across positions. A local window can capture the concept dynamics of the full response, allowing local calibration to support feedback throughout generation. We compare three settings on all 817 TruthfulQA questions with Llama-3.1-8B-Instruct. Single (Ours), used in our paper, calibrates on the first window of each training response. Separate fits calibration statistics separately at each window position. Combined combines windows from all positions to fit shared statistics. All three use the same inference parameters and decoding, with in fold 0 and in fold 1. At uncovered positions, Separate retains online RePE with zero SARA feedback.
| Calibration | MC1 | MC2 | MC3 | True | Info | |
|---|---|---|---|---|---|---|
| Single (Ours) | 56.55 | 74.16 | 53.84 | 81.55 | 98.02 | 79.93 |
| Separate | 55.81 | 74.15 | 53.51 | 82.50 | 97.76 | 80.65 |
| Combined | 56.67 | 74.11 | 53.41 | 81.98 | 98.68 | 80.90 |
All three schemes achieve comparable performance. This consistency supports our interpretation that the first window captures a globally shared structure in concept dynamics, enabling DCT feedback to transfer across generation positions.
Response to Reviewer fK2o (Part 1)
Thank you for the thoughtful comments on the role of the DCT and our experimental comparisons. We have replaced the main framework figure and shortened the appendix by removing overly detailed experimental descriptions and redundant data, while retaining the information needed to understand and reproduce our results.
(W1)
Additional comparisons. We added comparisons with early-minus-late and linear slope. The comparison below uses Llama-3.1-8B-Instruct on all 817 TruthfulQA questions. The temporal-summary baselines use their independently validation-selected settings.
SARA achieves the highest MC1, MC2, MC3 and True×Info in this comparison, supporting the use of feedback from separate frequencies.
Relationship to our SARA. DCT provides a unified mathematical view of linear temporal summaries: any such summary is a weighted sum of DCT coefficients. Let H∈RW×d contain the hidden states, U=[u0,…,uW−1] be the orthonormal DCT basis, and C=U⊤H. For any temporal weight vector a,
=\sum_{k=0}^{W-1}\beta_k C_k, \qquad \beta_k=u_k^\top a.$$ Early Minus Late and Linear Slope correspond to specific choices of $a$. In the standard symmetric setting, Early Minus Late uses opposite weights on equal halves of the window, and Linear Slope uses weights proportional to $t-(W-1)/2$. Both satisfy $a_{W-1-t}=-a_t$. Since the DCT basis satisfies $u_k(W-1-t)=(-1)^k u_k(t)$, $$\beta_k=(-1)^{k+1}\beta_k \quad\Longrightarrow\quad \beta_k=0\quad\text{for even }k.$$ Thus, early-minus-late and linear slope use only odd DCT frequencies under these symmetric definitions. This view makes their frequency content explicit: each method fixes a particular mixture of AC coefficients. SARA keeps the frequencies separate and uses concept feedback to adjust their contributions during generation. We have added these comparisons to Section 5.3 (Table 3) and the unified mathematical view to Appendix F.1. **(W2)** > Concept prompting is absent from the comparison, despite SARA using explicit concept instructions to construct its online steering direction. AxBench is cited, their paper argue specifically argue about including this control. We have added a Prompting baseline that places the target concept instruction directly in the generation prompt. The expanded comparison includes online RePE and all three RePS variants. Results use Gemma-2-2B-it on 35 math, 107 code and 50 text concepts. SARA achieves higher $H$ scores than Prompting across math, code and text, supporting the value of activation steering with explicit concept information. These results suggest that SARA generalizes beyond code and math to text concepts. We have added Prompting, online RePE and RePS baselines to Section 5.2 (Table 2; Appendix E.1). | Method | Math $H$ | Code $H$ | Text $H$ | | :--- | ---: | ---: | ---: | | Prompting | 0.9937 | 1.1361 | 1.0568 | | RePE (online) | 1.1749 | 1.2237 | 1.2316 | | RePS-SV | 0.3629 | 0.1764 | 0.5160 | | RePS-LoRA | 0.6697 | 0.3757 | 1.1324 | | RePS-LoReFT | 0.7194 | 0.4157 | 1.0864 | | SARA | 1.1914 | 1.2660 | 1.2540 | **(W3)** > The fine-tuning comparison is limited to 72 positive examples per concept, with rank-4 LoRA applied only to attention output projections at four layers. This limits the result such technique could have on the final model. Our training data and LoRA settings follow AxBench's published protocol and released configuration. AxBench allocates 144 examples per concept, with 72 positive and 72 negative examples, and trains its fine-tuning baselines on the positive examples ([AxBench, Sections 3.1 and 4](https://arxiv.org/html/2501.17148v3)). Our LoRA settings follow the official AxBench configuration ([AxBench LoRA configuration](https://github.com/stanfordnlp/axbench/blob/main/axbench/sweep/wuzhengx/2b/l20/lora.yaml#L19-L29)). For Gemma-2-2B-it, we use rank 4, attention output projections (`o_proj`), and layers 5, 10, 15 and 20. Our comparison also includes SFT, which updates all model parameters.