- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper asks whether a undesirable behavioral family (compliance and sycophancy prefix pairs) occupies a compact subspace of the residual stream, and whether ordinary training can suppress it without an inference-time edit. It proposes a three-gate protocol:
- Gate A (concentration): top-4 energy of the centered covariance of paired activation differences, compared against a matched-energy isotropic control.
- Gate B (exclusion by training): relative drop in held-out projected energy (or held-out response in the synthetic case) after training with . GO requires a drop of at least 30%.
- Gate C (capability): a benign "opener" cross-entropy must stay within 15% relative change.
Proposition 1 shows that for a linear readout the penalty equals . Evidence comes from three settings: (i) a synthetic contaminated-label residual stream (held-out response drop 0.989, clean accuracy non-decreasing); (ii) Gemma-4-12B with KL-regularized LoRA and a normalized projected-energy penalty (32.5% drop on a 6-prompt report slice, opener-CE +0.8%); (iii) Qwen2.5-7B-Instruct with the same prompts and splits (31.8%, opener-CE +0.1%). A sycophancy family is NO-GO at layer 16 and GO () at layer 48. Activation scrubbing is relegated to an appendix and is only marginal.
Critical strengths
-
Candid, falsifiable framing. The paper keeps its negative results: sycophancy fails at layer 16, null prefix families are rejected by the mid-layer rule, scrubbing is only marginal, and the CE ablation shows compliance CE is driven by the term and not by geometry. The observation that concentration (Gate A) licenses the question but does not imply trainable suppression (Gate B) is useful, and the limitations are stated plainly.
-
Clean synthetic sanity check. The planted-subspace setup with contaminated labels (Sec. 7.1) is well constructed, because it makes exclusion non-vacuous. The sweep (Appendix A) shows a smooth, monotone drop in held-out response (0.042 to 0.989) with clean accuracy essentially unchanged. This confirms that the penalty behaves as Proposition 1 predicts in the linear case.
Critical weaknesses
-
The headline metric is the quantity being optimized, and it is not tied to behavior. Eq. (3) directly minimizes projected energy on the compliance forward at layer 32. Observing that this energy is lower on six other prompts is largely expected and does not show that the targeted behavior or its causal pathway was reduced. The paper's behavior-adjacent measures do not move: compliance CE changes by (Gemma) and (Qwen), and the preference margin is slightly adverse (). The CE ablation shows CE responds only to the term, and inference-time scrubbing raises compliance CE by 44.7% whereas training does not. LoRA is applied at layers , including layers downstream of the penalized layer 32, so the model may simply recompute the same information elsewhere. The paper concedes that an energy drop is not a claim that behavior fell (Sec. 8), but that leaves the central claim ("training can suppress concentrated behavioral geometry") supported only by a proxy that the method itself optimizes.
-
Statistical and protocol weaknesses.
- The reported number rests on a 6-prompt report slice that is a post-hoc partition of a 13-prompt pool already used in an earlier pilot (). There are no seeds or confidence intervals, and the 30% GO threshold is cleared by about 2.5 points in a single run. The selected checkpoint is early-stop@80, which is also the maximum step count, so no early stopping actually occurred.
- The mid-layer Gate A requirement was added after seeing results (Sec. 6 addendum), and layer choice acts as a tuned hyperparameter (sycophancy fails at layer 16 yet passes at layer 48 with ).
- Gate A is weak evidence of concentration: top-4 energy is 0.527 at Gemma layer 32 against 0.284 for the isotropic control, the rank-2 penalty covers only 0.363 of train energy and 0.335 of held-out centered energy, and the null families also reach GO at layer 48.
- Gate C is a single opener-CE metric, and the paper itself states that general capability preservation is not shown. The Qwen "protocol replication" uses the same prompts and splits, so it says little about generality.
-
No baselines and limited novelty. There is no comparison with a random rank-2 subspace penalty, a mean-difference () direction penalty, training-time LEACE/INLP-style variants, representation noising or circuit breakers, or a plain KL/preference baseline. The theory is a one-line quadratic identity (Proposition 1), and Remarks 1 and 2 are imported from a self-cited preprint. The related training-time representation methods are acknowledged in Sec. 2 but not compared against, so the contribution over them is unclear.
Questions for authors
-
(Material.) Does the energy-penalized model show any change in an outcome that is not the training target, such as the rate of compliant or sycophantic generations judged independently, or the response to a steering vector along ? Without this, what does the energy drop show beyond successful optimization?
-
(Material.) Does the effect survive measurement away from the penalized site, for example energy at layers 36 or 40, a different token window, or a probe retrained after training? Is the bad subspace removed, or merely rotated or recomputed downstream?
-
(Material.) Please add controls: a random rank-2 subspace penalty and a mean-difference direction penalty at matched , with multiple seeds, confidence intervals, and a larger prompt set (). How stable is the GO/NO-GO call across alternative splits of the 13-prompt pool?
-
(Clarification.) The selected checkpoint is with early-stop@80, equal to the maximum number of steps. Was early stopping ever active? Would the selected or the report-slice drop change if the calib and report slices were swapped?
-
(Clarification.) How were the layer choices (32 for Gemma, 16 for Qwen) made, given that the mid-layer rule was added after the null-family results? For sycophancy, would layer 48 have been selected without the failure at layer 16?