- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes CGUS, a training-free steering method for SwiGLU feed-forward blocks that sits on top of the NPTI personality gate. It has two parts. The first is a gate operator (Eq. 3) that enhances or suppresses selected post-activation gate coordinates. The second is an up-branch steering vector (Eq. 5) added to the up projection before the elementwise product, extracted by replaying contrastive responses under an active gate state ("state-conditioned extraction"). Domain and facet controllers are meant to compose as P, P+A, P+B, and P+A+B (Eqs. 6–8). The paper also defines a post-parent actionability criterion (Eq. 4) that flags facet suppressions made redundant by a parent gate that has already zeroed the same coordinates. On the 459-question NPTI benchmark with Llama-3-8B-Instruct, the domain up-branch gives +0.062 over NPTI at ρ = 0.5 (the only dose whose corrected CI excludes zero) and collapses at ρ = 2. It gives no reliable gain on Gemma-2-9B-it. Adding one facet's gate and up-branch gives +0.040 to +0.051 personality with a 0.16 to 0.23 drop in fluency, relevance, and completion. About 44% of that gain comes from one facet-pole whose outputs are degenerate. The authors describe the results as development-stage findings.
Critical strengths
- The reporting is candid and partly preregistered. The paper separates preregistered doses from post-hoc extensions, uses energy-matched random directions and a second mask seed, applies Bonferroni correction, and reports negative results and degenerate cases (Table 14) instead of filtering them out. This is better practice than many steering papers.
- The actionability observation is a useful diagnostic. Showing that 51 of 52 suppression coordinates in one gate bank duplicate parent suppressions, and that the result is token-identical output (Case 2), is a concrete warning about stacking gate-style edits.
Critical weaknesses
-
The paper's own results do not support the proposed method, and the central composition claim is never tested.
- Domain up-branch. None of the three preregistered doses separates from NPTI or from random directions (Table 1). The only significant cell, ρ = 0.5, was added after seeing those results. Its gain is +0.062 on a 1–5 scale, and it lowers the quality pass rate from 61.65% (NPTI) to 56.05%. Higher doses fail, with 0% quality pass at ρ = 2. On Gemma no dose separates, and the one promising cell reverses sign on a second mask seed (+0.2889 to −0.1111, Table 11).
- Facet branch. The personality gain (+0.040 to +0.051, Table 2) is smaller than the quality cost (fluency and relevance −0.18 to −0.23). The up-branch adds nothing measurable over the facet gate alone: every "vs. facet-gate-only" interval in Table 13 contains zero. So the up-branch, which is half of the method's name and the target of state-conditioned extraction, has no demonstrated effect in the facet study.
- Composition. The title and Section 3.3 frame the method around composing facets (P+A, P+B, P+A+B), and Eq. 9 defines joint attainment J and an interaction score I. Neither J nor I is reported for any experiment. The only two-facet evidence is a short qualitative description of "earlier exploratory" runs in Section 7, which reports mutual attenuation. The parent-priority gate-merge study was generated but not scored (Appendix D gives only a cost estimate). Appendix F lists the confirmatory protocols as future work.
-
The experiments do not match the method described, and the evaluation has validity problems.
- Method mismatch. Sections 3.1–3.2 claim independent high/low endpoints and up-branch vectors read under the active gate state. But the domain-up study in Section 5.1 extracts its direction "with gate and up interventions off" and uses a single high-minus-low direction, with "its negation" for the low pole (Appendix D). That is the unconditioned, sign-reversed setup the paper argues against. The facet study uses raw, non-normalized vectors while the domain study uses unit vectors, so even the scaling convention differs between the two main experiments.
- Judge validity. In Table 14 the low Artistic-Interests output degenerates into strings of single letters and receives personality = 5. A metric that gives the top score to incoherent text cannot separate trait expression from model failure, and this slot supplies about 44% of the facet macro gain. The judge is named only as "openai/gpt-6-luna:batch", with no description or reference, so it cannot be identified or reproduced. There is no human evaluation.
- Baselines. The two main studies compare only against NPTI and random directions. PERSONA, DPN-LE, CAA, and prompting are not run on the official benchmark, even though they are discussed as the relevant prior work. DPN-LE appears only in the earlier IPIP study (Table 7).
- Effect sizes in the earlier studies. The component studies that "motivated these designs" report effects such as +0.00125 to +0.003 on a 1–5 scale (Table 8) and a pilot on six scenarios (Table 6). These are too small or too underpowered to justify design choices.
- Internal inconsistencies. The text in Section 5.1 says the quality pass rate "falls from 65.15% at ρ = 0.1", but Table 1 shows 65.15% at ρ = 0.01 and 63.27% at ρ = 0.1. The contributions claim "comprehensive" evaluations, while the abstract and conclusion state the results are not a validated recipe.
- Technical framing. The actionability criterion follows directly from Eq. 3: once max(g, 0) = 0, further suppression is zero by construction. It is a useful sanity check, but presenting it as a main contribution overstates it. The stochastic mask z in Eq. 3 is not defined.
Taken together, the paper reads as a progress report. The authors' own conclusion, that the results "do not yet establish a dose, facet selection, or cross-model recipe that is simultaneously effective, quality-preserving, and reproducible", is accurate and is the main reason for my recommendation.
Questions for authors
Items 1–3 would need to be resolved for the paper to be reconsidered. Items 4–6 are clarifications.
- Can you report J and I (Eq. 9) for the P / P+A / P+B / P+A+B configurations on the official benchmark, with the parent-priority gate-merge study scored?
- Why does the domain-up study use gate-off extraction and a sign-reversed low pole, when the method claims state-conditioned extraction and independent endpoints? Is there a version of Section 5.1 that follows Section 3?
- Please compare against PERSONA, DPN-LE, CAA, and prompting on the same 459-question benchmark, and add a human evaluation or a second judge. What exactly is "openai/gpt-6-luna:batch"?
- How many questions does each domain-pole/facet slot contain? The per-facet increments in Table 12 look like they are computed on roughly 90 questions per slot.
- The low Artistic-Interests gate has 27,124 coordinates. Across how many layers is this support spread, and was any coordinate cap or threshold ablation run?
- Please define the stochastic mask z in Eq. 3 and explain how λ, e, and s are set for each study.