- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes a steering algorithm called Amortized Jacobian Readout (AJR) that is trained on gradients to predict how much an edit will affect the final output and use said prediction to build a steering vector. Unlike some steering techniques that use a fixed steering direction (e.g., CAA) AJR adapts the steering direction to the input. The authors test AJR on a synthetic task on four models. The task consists of sentences like "Charlotte has a red key. William has a blue key. Charlotte has a ___" and AJR is used to steer the answer towards the target choice (which is provided to the method as input). AJR is able to move the answer towards the target color more than CAA (a steering technique that uses a fixed shift) and a fixed mean-gradient direction do, and it fixes 71 to 100% of the wrong answers (while breaking at most 2.7% of the correct ones). The paper also shows that tools that predict the model's output well (e.g. the tuned lens) predict the effect of edits poorly, and that later layers push back against edits in both directions.
Critical strengths
- As far as I know this is the first steering method that learns to predict the gradient, so it obtains a gradient-like, prompt-specific steering direction without a backward pass at inference.
- The analysis that separates "reading" a hidden state from "predicting the effect of editing it" is interesting.
Critical weaknesses
I find it difficult to assess the usefulness of this method because: 1 the practical value seems very limited, 2 the paper lacks more solid baselines and 3 the evaluation does not seem completely fair.
Practical value: the proposed algorithm seems very constrained and its value only shown on a toy task that does not seem to transfer.
- In this task the user needs to provide the “push” towards the correct answer. While this is normal in steering application the correct answer is usually (as an example) to avoid harmfull language, or to be truthful, etc… here the direction is towards a 1 word answer: blue or red.
- The steering is only applied once at the end of the prompt. In practical applications to language models one does not know usually when the steering needs to be applied so you either always intervene or use a classifier to do so, but neither version is tested for AJR (only the CAA-all baseline edits every position). Being constrained to intervene only at the end (and for a single token) seems a strong constraint.
- There is no evidence that the abilities shown on this task carry to other tasks. The only evidence is that on WikiText pronouns, AJR loses to the mean gradient in 3 of 4 models and to CAA-all in all 4 (this results should be discussed in the main paper).
Lack of proper baseline comparison
- Prompting. AJR is given two things: the answer pair (red vs blue, fixed when it is trained) and which of the two to push towards. Since the user needs to provide the right answer "corrects 71–100% of wrong answers" really means "when told the right answer, AJR makes the model say it 71–100% of the time”. Therefore a fair prompting baseline would be something like:
- “Hint: the answer is red. Charlotte has a red key. William has a blue key. Charlotte has a”
- There are steering techniques that are more “dynamic” than CAA, for example:
- ReFT/LoReFT: a learned low-rank edit that is a function of the hidden state;
- activation-transport methods such as Linear-AcT or DSAS in which each neuron gets its own scale and shift so the edit depends on that neuron’s value
- The steering evaluation is limited to small models (up to 2B). Typical steering papers use larger models (especially comparing with prompting this might make a difference).
Lack of fair evaluation
- The way CAA is evaluated is not using CAA full strength.
- CAA is normally used with a multiplier tuned for the task. Here every method gets the same edit size, taken from AJR's fitting data and not tuned for each method. Such tuning (sweeping value for rank, regularization, checkpoint) seems to be done for ARJ instead.
- Similarly, the layer was fixed in advance (the middle layer) for all methods and never swept, while CAA is usually sensitive to the choice of layer.
- There is no evaluation of how the model general abilities (e.g., PPL and MMLU) degrade, which is a standard test in steering.
- There is no evidence of what would happen to answers that are longer than 1 word.
Not a reason for rejection but the paper is not easy to read and is missing some information.
- I found the abstract hard to follow: after reading it, I could not understand what practical use case the algorithm targets.
- In general, sentences are long and with many caveats that make reading difficult.
- AJR actually uses the average of 2 predictors (ARJ-lin and ARJ-mlp) but there is no ablation that informs how well each of those do in isolation for steering (their steering results
- AJR steers using the average of two predictors (AJR-lin and AJR-mlp), but no steering results are shown for either one alone. Their prediction accuracy is reported separately, but the authors found AJR-mlp better in every model. So it is unclear if averaging helps.
- Unless I missed it I don’t think the parameter count, memory and fitting cost for the predictor are not reported. Adding this info would help with reproducibility and with comparing costs against other steering methods.
Questions for authors
The amount of changes is such that I am not sure they can all be addressed in a rebuttal. However, I hope these questions could be used to improve the work regardless of its acceptance at this time.
- Can you show that AJR works in more realistic and useful settings?
- Can you add stronger baselines as suggested above?
- Can you tune all methods to use them at their best configuration? That means tuning their strengths, hyperparameters, layer, etc… on the held out set as it was done for ARJ’s predictor settings?
- Can you steer and run those tests on bigger models?
- Does the AJR edit change general ability, for example perplexity on ordinary text or MMLU accuracy, compared with the unedited model?
- Can you disclose the parameter count, memory and fitting cost of AJR, per layer and per answer pair?
- Can you provide the steering results of the two ARJ versions in isolation?
- How would AJR be used when the moment to steer is not known?
6 comments
Response to Reviewer sDiP
Q2. Stronger baselines
Prompting. The fixed prefix Hint: the answer is {red/blue}. , with no prompt search, is below original-setting AJR-ens in Qwen1.5-0.5B and Pythia (by 4.4 and 23.1 points) and above it in Qwen1.5-1.8B and Gemma (by 0.8 and 2.5 points).
LoReFT. A task-adapted LoReFT learns a low-rank, state-dependent edit for each color by target-token cross-entropy on the 960 fitting prompts. It reaches 100% with zero harm in all four models. Its edit size is learned and is not matched to AJR's (median output KL 11.4–16.3).
SAE features (Table 4). On Gemma, at AJR's middle layer, a Gemma Scope feature direction reaches 99.69% with zero harm after strength selection, against 95.94% for AJR-ens. AJR's direction is much closer to the response gradient (cosine .576 against .075) and gives larger margin gains, so alignment with the response and preservation of correct answers are different properties.
The related work now cites Linear-AcT and DSAS; the tuned baselines here are CAA, the mean gradient, LoReFT, SAE features and the hint.
Q1 and Q4. Realistic settings and larger models
The color task isolates the question the paper asks. Every directed method and the hint receive the correct color, so the comparison tests how well an edit produces a supplied answer. The fitted target is the margin of one next token, in four models from Pythia-410M to Gemma-2-2B. On the natural-text pronoun test (Appendix E.4), AJR exceeds CAA-last in all four models and the mean gradient in Pythia; CAA-all, which edits every position, is higher. As you ask, this result is now stated in the main text (Section 5). The 27B study finds the same response geometry as in the small models. The revised abstract and discussion state the claim directly: AJR is a learned model of the effect of an edit at a specified layer.
Q5 and Q8. General ability and when to steer
The paper measures the side effects of an edit where it acts: harm to correct answers, full-output KL (Appendix D), and the token after the answer. At the original setting, the expected noun follows a steered color in at least 99.7% of cases. Accuracy alone can hide such effects: CAA-all reaches 100% in Step 2, yet the expected noun follows in only 48.44% of Pythia prompts. AJR edits the one position where the answer is produced, so it is applied where that answer is wanted; a policy for when to steer in open-ended generation is a separate question from the response model studied here.
Q6. Parameters, storage and fitting cost
Per layer and color pair (Table 13); counts are stored numbers.
| Model | AJR linear / MLP weights | AJR bases and statistics | AJR-ens stored size | Fixed vector | LoReFT |
|---|---|---|---|---|---|
| Qwen1.5-0.5B | 65,536 / 557,632 | 136,192 | 5.79 MiB | 1,024 | 65,568 |
| Qwen1.5-1.8B | 131,072 / 1,081,920 | 272,384 | 11.33 MiB | 2,048 | 32,776 |
| Pythia-410M | 131,072 / 557,632 | 201,728 | 6.79 MiB | 1,024 | 65,568 |
| Gemma-2-2B | 73,728 / 1,212,992 | 232,704 | 11.59 MiB | 2,304 | 147,488 |
For Qwen1.5-0.5B, the complete setup for one layer and color pair takes 70.57 s on one B200 GPU with 2.9 GiB peak GPU memory. This covers gradient collection for 960 prompts, the hyperparameter search, the final fits and verification. At inference, AJR's median time was close to unedited inference, while gradient steering took 2.1–2.3 times as long.
Q7. The two AJR versions separately
Original middle layer and edit size (Table 12): margin gain, then correction / harm (%).
| Model | AJR-lin | AJR-mlp | AJR-ens |
|---|---|---|---|
| Qwen1.5-0.5B | 1.785, 73.8 / 0.0 | 1.349, 57.1 / 0.4 | 1.804, 71.4 / 0.2 |
| Qwen1.5-1.8B | 3.563, 90.9 / 0.4 | 2.329, 85.1 / 0.0 | 3.370, 93.4 / 0.0 |
| Pythia-410M | 2.350, 97.6 / 0.0 | 1.448, 59.4 / 0.0 | 2.160, 92.7 / 0.0 |
| Gemma-2-2B | 4.520, 98.1 / 3.1 | 2.531, 84.6 / 0.9 | 4.350, 100.0 / 2.7 |
The linear predictor alone gives larger margin gains than the ensemble in three models. The MLP alone gives the smallest gain and correction in all four. Averaging is therefore not required for steering, and the revised paper presents linear AJR as the base method with the MLP and the ensemble as variants. With the multiplier selected (Step 1), AJR-lin and AJR-ens are within 1.3 points of each other.
Presentation
The abstract now states the idea in plain terms before any technical vocabulary and reports the selected-setting result directly. Section 3 opens with the red/blue example and gives the fitting and editing procedure before the paired-edit analysis. Repeated qualifiers were removed; each scope condition is stated once.
Thanks for doing this. I think all the additional experiments are going in the right direction. The experimental section is now more fair and stronger thanks to the new baselines and additional information provided.
I understand that in the middle layer ARJ does better but why should that be preferable than the last layer?
In general, and compared to other steering work, I think realistic applications (e.g., harmful language mitigation) and side effects when one does not know when to steer are still an important gap that needs to filled in order for this work to gain an acceptance score.
Response to Reviewer sDiP
Thank you for the concrete list of controls. We ran those that the color task allows, and the revised paper reports them: steering is now compared in three settings, and the abstract and Section 3 are rewritten for clarity. Tables and sections refer to the revised PDF; Q1–Q8 number your eight questions in the order listed.
Q3. Tuning every method
We now tune both the multiplier and the layer of every method, in two steps, so that their effects can be separated. Settings are selected on 384 calibration prompts whose entity names are disjoint from the 960 fitting and 640 evaluation prompts. The rule is the same for every method: highest calibration accuracy with at most 2% harm (initially correct answers made wrong), ties going to the higher target probability. Multipliers are {0, .25, .5, 1, 2, 4} times the original edit norm; layers are six depths from .25 to .875.
Answer accuracy (%) on the 640 evaluation prompts.
Step 1 (Table 10, new). With every method at the original middle layer, tuning CAA's multiplier does not close the gap. AJR-ens is 3.8 to 24.8 points more accurate than CAA-last and CAA-all in all four models, and more accurate at every tested nonzero multiplier (every paired 95% interval is above zero). The best CAA accuracy at any multiplier (89.53/91.41/78.44/92.50%) is below AJR-ens at the original setting. The mean gradient is 8.1 points [6.4, 9.8] below AJR-ens in Qwen1.5-0.5B and within 1.3 points in the other models; in Pythia both reach 100%.
Step 2 (Table 3). Each fixed direction reaches 100% with zero harm in all four models, on seen and unseen templates, at depth .875 (the Gemma mean gradient at .75). AJR-lin with the same search reaches 100/100/100/99.38%. With the layer free, every method is within one point of 100%.
New prompts (Table 10b). We froze all directions, layers and multipliers, stated the comparisons in advance, and evaluated 640 new prompts whose entity names appear nowhere else. Step 1, answer accuracy (%):
Step 1 repeats: AJR-ens is 3.0 to 27.7 points above both CAA variants in all four models (all eight intervals above zero), 5.9 points [3.8, 8.3] above the mean gradient in Qwen1.5-0.5B, and within one point of it elsewhere. Step 2 repeats: 99.4–100% for the fixed directions and 100/100/100/99.06% for AJR-lin.
The depth profile explains both steps (Table 7): the cosine between the mean gradient and each prompt's own gradient is .19–.44 at the middle layer and .87–.98 at depth .875. A prompt-specific direction matters where responses differ across prompts; near the output, one direction is enough for this task.