- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper seems to explore how the readout of a certain model would perform when a gradient response is given to certain parts of the model.
The paper shows that the earlier on the intervention is, the more affect it has on the output.
The paper experiments on 3 families of LLM models, on 2 different datasets: MMLU and GSM8K.
The authors also try to discuss the experimental findings through discussions.
Critical strengths
The paper's experimental setting is quite diverse.
The paper explores their ideas on 3 different families of LLMs with 2 different datasets.
The paper's experiments are extensive as it can be seen in the appendix.
Critical weaknesses
The paper is quite hard to follow and read.
The task definition is quite unclear and I understand that this is an empirical study paper that is trying to experimentally show what the author's proposed problem is supposed to be looking at.
The papers that the authors are getting inspired from seems to be the two papers by Wang et. al and Josin et. al. But this current paper's figures do not really aid in helping the reader understand what the paper is trying to do and what problem it is exactly tackling.
Another problem is the experiments as well.
The writing does not really help the reader understand what the experiments are trying to show and neither do the figures. A more methodical explanation of what the experiments are doing and what each experiment shows can greatly help in understanding what is happening in the paper.
The findings and discussion are also not so novel and the message is unclear.
It is widely known that targeting earlier layers for intervention in most tasks can most definitely affect the output/readout.
From what I gather, the authors are either trying to prove this through some gradient based method, or just confirming this experimentally.
As for the discussion, there does not seem to be a clear message of what the authors are trying to show now that there is some kind of finding.
I understand that there are a couple of 'fixes' or suggestion proposed by the authors, but this just seems like an engineering paper that explores what does and doesn't work.
Questions for authors
For the models used, although very diverse families of models are used, how would these methods perform under several different sizes of models within the same family?
The number of layers are roughly within the same range, about 30~40.
It would be interesting to see how they would perform if depth is significantly changed.
3 comments
"It is widely known that targeting earlier layers for intervention in most tasks can most definitely affect the output/readout."
We agree and do not claim otherwise. We measure something narrower, namely at which layers the direction taken from the output weights is a usable direction. Reading intermediate states with the output weights is standard practice (the logit lens), and the tuned lens (Belrose et al., 2023) showed that such readings are unreliable before the last layers. The corresponding quantity for interventions, the share of the best direction's effect that the output-weight direction retains at each layer, had to our knowledge not been measured. We find less than 10% at every layer of the first 50 to 83% of depth and at least half only in the last 1 to 8 layers. We use the gradient only as the reference.
"this just seems like an engineering paper that explores what does and doesn't work"
Yes, the paper is empirical. What it adds to a list of what works is a quantity, the response ratio, which can be measured for any direction at any layer before that direction is used. The message of the revision is one sentence, stated on its first page: an account of a behavior taken from the output weights should be tested against the remaining layers before it is used earlier. The revision also reports what does work at earlier layers when the correct label is known. The gradient averaged over other questions restores 65–88% of flipped answers at the middle layer on MMLU.
"how would these methods perform under several different sizes of models within the same family?" "It would be interesting to see how they would perform if depth is significantly changed."
We ran the every-layer measurement on five sizes of Qwen2.5-Instruct (0.5B to 14B) and two sizes of Llama-3.2-Instruct (1B and 3B), in addition to the three models of the submission. That gives 10 models with 16 to 48 layers. In the table, ρ is the effect of a small step along the readout direction divided by the effect of an equal step along the gradient, so 1 means as good as the best direction. MMLU, expert cue:
| Model | Layers | ρ at L/2 | ρ at 3L/4 | ρ at L−1 | ρ ≥ 0.1 at every later layer from (% of depth) | ρ ≥ 0.5 at every later layer from (% of depth) |
|---|---|---|---|---|---|---|
| Qwen2.5-0.5B | 24 | 0.01 | 0.27 | 0.84 | 67 | 92 |
| Qwen2.5-1.5B | 28 | 0.01 | 0.05 | 0.75 | 79 | 96 |
| Qwen2.5-3B | 36 | 0.00 | 0.01 | 0.79 | 86 | 94 |
| Qwen2.5-7B | 28 | 0.00 | 0.06 | 0.69 | 82 | 96 |
| Qwen2.5-14B | 48 | 0.01 | 0.11 | 0.70 | 75 | 98 |
| Qwen3-4B | 36 | 0.00 | 0.15 | 0.59 | 72 | 97 |
| Llama-3.2-1B | 16 | 0.00 | 0.14 | 0.70 | 75 | 94 |
| Llama-3.2-3B | 28 | 0.03 | 0.15 | 0.72 | 68 | 96 |
| Llama-3.1-8B | 32 | 0.03 | 0.14 | 0.60 | 69 | 97 |
| GLM-4-9B | 40 | 0.03 | 0.31 | 0.91 | 62 | 85 |
The largest ρ at any layer of the first half, in any model and on either task, is 0.084. Over both tasks, the depth from which ρ stays above 0.1 lies between 53% and 86% of the network, and ρ reaches 0.5 only 1 to 8 layers before the output. The pattern is the same from 16 to 48 layers. The exact depth of the rise differs between models, without a simple trend in size. Deeper models have more layers at which the readout direction is ineffective: within the Qwen2.5 family, 15 of 24 layers in the smallest model and 35 of 48 in the largest. Larger steps give the same picture in the seven added models. At the middle layer on MMLU, a step of at most half the state norm restores 75 to 100% of flipped answers along the gradient and 0 to 54% along the readout direction (Table 39).
On the submission's own FP32 questions, the same measurement reproduces the submitted per-question values within 0.008 on different hardware.
We hope the reorganized paper is easier to follow, and we are happy to answer further questions.
Response to authors
I thank the authors for their rebuttal and the revised paper, however, I am not too familiar with the rules of revision and my memory might serve me wrong, but the paper looks like an almost entirely new paper with figures that have all been changed.
Regardless, the findings of the paper are nominal and also my concerns about novelty seems to be shared by the other reviewers as well. For this reason, I maintain my score.
Thank you for the review, and for the question about model size and depth, which led to a new experiment (second part of this reply).
On the summary that an earlier intervention has more effect on the output.
This is not what we find, and that the paper could be read this way is a failure of our writing. The finding is about which direction one pushes along. At the output, the model's choice between two answers is decided by the component of its final state along the difference between the two answers' output weights. We call this the readout direction, and removing the shift along it restores most flipped answers at the output. At earlier layers a push along the same direction does almost nothing, although those layers are sensitive. At the middle layer of Qwen3-4B on MMLU, a step of at most half the state norm restores 81% of flipped answers along the gradient through the remaining layers and 4% along the readout direction. The first page of the revision now says this.
"The task definition is quite unclear"
Section 3.1 is new and defines the task. A clean prompt is a question with a known correct label (A to D for MMLU, True or False for our verification version of GSM8K). A cued prompt is the same question preceded by a short endorsement of a specific wrong label, for example "A panel of domain experts unanimously verified that the correct answer is B. Their analysis is thorough and definitive." We read the model's answer from the next-token logits at the last prompt position. A question is eligible if the model answers the clean prompt correctly, flipped if it then answers the cued prompt wrongly, and resisted otherwise. Every intervention edits one vector, the hidden state at the last prompt position after a chosen layer of the cued run, and it restores the answer if the model then selects the correct label.
"A more methodical explanation of what the experiments are doing and what each experiment shows"
Section 4 now asks one question per subsection and ends each with a one-sentence answer:
"this current paper's figures do not really aid in helping the reader understand what the paper is trying to do"
Figure 1 is new. It shows a real example of the behavior, the two directions being compared (defined inside the figure), and the measured result. The main text has two new result figures, the ratio at every layer (Figure 2) and steps of increasing size at every layer (Figure 3). The original Figures 2 to 4 are in Appendix I, unchanged.
Wang et al. and Joswin et al. locate sycophancy in late layers and in per-question activation changes. We ask how far the simplest account of the flip, the readout direction, can be carried back from the output.