- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper asks whether harmful-compliance SFT can be detected from the weights alone. Each checkpoint is represented by its effective update LoRA delta. Small banks of reference adapters are trained on (1) harmful compliance, (2) refusal on harmful prompts, and (3) benign chats. An unknown update is fit as a non-negative combination of (1) and an averaged of (2) and (3). The normalized harmful coefficient s_H serves as the proposed audit score, namely TRACE.
Across four 7–8B base models, s_H rises monotonically with the harmful fraction of mixed SFT data (Spearman ≈ 0.99). It separates compliance from refusal on matched prompts, and tracks the harmful fraction more closely than the absolute number of harmful examples. TRACE outperforms several weight-based and activation-based baselines. It is robust to hyperparameters, partial layer access, LoRA/full-FT transfer and a 10k-example mixture, and it correlates with jailbreak ASR. Trigger-based backdoors are not detected unless a backdoor prototype is added.
Critical strengths
The method is simple and cheap. The kernel is computed from LoRA factors, only a handful of references are needed, and no forward passes are required.
I like the experimental control. Each control rules out a clear competing explanation for the trend: the topic of the prompts, the number of harmful examples, the size of the update, and sequence length.
The paper is also transparent about negative results such as the backdoor experiment.
Critical weaknesses
My main concern is what happens when an audited update looks like none of the references. only uses the projection of the update onto the span of the two prototypes. Since it is a ratio, it ignores how much of the update that projection actually captures. For instance, take and to be orthogonal unit vectors, then an update that is well-aligned with the reference will look something like 0.9$\mu_C$ + 0.3$\mu_N$ plus a small residual gets = 0.75, and the score describes it well. However, if the update is something like a medical-QA fine-tune unrelated to either reference, it can become 0.03$\mu_C$ + 0.01$\mu_N$ plus a large residual, which can get a same = 0.75 but is completely misinformed. This could easily come from incidental overlap in answer style or from seed noise. The non-negativity constraint can make this worse. A slight negative alignment with sets to 0, so the update scores as safe. The backdoor experiment looks exactly like this failure: without a matching reference, backdoored adapters score AUROC 0.232. I do acknowledge the authors’ remedy by adding a backdoor prototype, but it requires knowing the attack family in advance, which I view as a "patch fix" and is thus not generalizable nor practical.
More related to my main concern, I don’t think the distribution-shift experiments test real shift. The OOD setting moves between PKU-SafeRLHF and BeaverTails, which come from the same pipeline. The unseen set is again harmful Q&A paired with code and math. In practice an auditor would face two harder cases: (1) benign long-form fine-tunes (medical, legal, creative writing) that could resemble the C prototype in style, and (2) harmful fine-tunes of other kinds (such as cyber, persona roleplay, misinformation) that might not resemble it at all.
I feel the baseline comparison is missing the "most basic baseline": a linear probe on , or simply cosine similarity to . If this works effectively, it may mean the contribution is closer to showing that harmful SFT is linearly separable in task-vector space, which prior task-vector work already suggests.
Questions for authors
What fraction of each audited update is explained by the two prototypes, and could this be used to flag updates the references don’t cover?
How does TRACE perform on real community adapters for the same base models? This means both false positives on benign adapters and detection on harmful data beyond PKU-SafeRLHF and BeaverTails.
I'm interested to see how a linear probe or a cosine similarity mentioned above compares to TRACE.
5 comments
Response to Reviewer XQ8j
In addition to the comparisons in our first reply, we tested diverse refusal targets in matched training runs. We generated and checked prompt-conditioned refusals in five modes for 1,113 deduplicated prompts with Qwen3-8B. Across four backbones, compliance scored higher in 236/240 matched pairs with diverse targets. The share-averaged macro AUROC was 0.961, and the mean compliance score exceeded the mean refusal score at each tested share (Figure 11 and Appendix C.5).
We added a benign SFT behavior control on AdvBench. From the base model to step 200 of zero-injection Tulu 3 SFT, ASR rose from 13.28% to 21.01% on Llama and from 5.99% to 12.24% on Qwen. Over steps 10 through 200, the changes were 0.0150 and 0.0145, respectively. The control documents behavioral degradation under benign supervision (Appendix F.1).
We also screened four publicly released Qwen3-8B updates with the frozen audit. Both SQSD compliance updates, from AEGIS and BeaverTails, were flagged. Of the two DirectRefusal adapters, one was covered and unflagged and the other fell below the coverage floor (Appendix I.4).
We evaluated cosine similarity to the prototype difference, the raw projection onto that direction, and the update norm. They achieved fresh-English macro AUROCs of 0.891, 0.870, and 0.734, respectively. The cosine result shows that a single direction captures much of the ranking (Appendix I.5).
If you have a chance to review these additions, we would appreciate a brief reply on whether they address your remaining questions about the evaluation.
Median q on 64 fresh English checkpoints is 1.32% for compliance, 1.33% for refusal, and 1.24% for benign updates
So they're all basically the same, and
Creative writing | 28/80 | 80/80
every creative writing update passes the gate too. So q looks the same whether the update is familiar or not. I think this backfires on you. And 28/80 false positives with all of them covered is quite problematic.
TRACE 0.901, Cosine 0.891
Again, this backfires. Cosine basically falls within noise, so the only thing TRACE adds is q, and q doesn't do its job in differentiation.
Response to Reviewer XQ8j
Thank you for the detailed review.
1. Explained update fraction and abstention
Section 6 defines q=1−∥x−x^∥2/∥x∥2, the fraction of squared update norm explained by the prototype fit on the audited query and value projections. In the review's example, the same sH can accompany very different q. Figure 6 shows the decision rule. Each backbone fixes the coverage floor at the fifth percentile of q on 25 English calibration checkpoints and abstains below it.
Median q on 64 fresh English checkpoints is 1.32% for compliance, 1.33% for refusal, and 1.24% for benign updates. The gate abstains on six benign and three refusal updates, reducing false positives from 8/48 to 5/48 while retaining 14/16 true positives (Appendix I.1). For trigger-conditioned backdoors, Appendix F.2 reports AUROC 0.232 with the original prototypes and 0.721 with a family-specific prototype, illustrating how separation depends on the reference geometry.
2. Benign OOD tasks and public updates
Appendix I.2 scores 20 checkpoints per task per backbone using the frozen English rule. All four task sources are held out of the reference bank.
The three grounded tasks yield 1/240 false positives with 219/240 covered. Creative writing has full coverage and 28/80 false positives, so task-specific score shifts can persist above the coverage floor. The matched Swahili and Javanese tests reach AUROC 0.965 and 0.970, respectively, with 35/40 compliance and 3/40 refusal updates flagged per language (Appendix I.3).
Appendix I.4 audits four released Qwen3-8B updates. SQSD provides the two compliance directions, including AEGIS as an additional harmful-data source. The ThinkSafe authors provide the two DirectRefusal adapters. The frozen Qwen coverage floor is 0.4544%. SQSD GitHub, ThinkSafe GitHub.
3. Role-play and jailbreak-framed SFT
We scored 320 checkpoints with the original reference bank and frozen English cutoffs. At each of six harmful-prompt shares from 5% to 100%, five matched pairs per backbone share the role-play or jailbreak-framed inputs and differ in compliance versus refusal targets. The remaining 80 checkpoints are benign controls. Mean compliance sH rises with share on all four backbones.
Compliance scores higher in 120/120 matched pairs. The frozen rule flags 94/120 compliance updates, 25/120 matched refusals, and 0/80 benign controls. Compliance detection rises from 0/20 at 5% to 14/20 at 10% and 20/20 at each higher share. Five compliance updates abstain, all at shares of 5% or 10%. All refusal flags occur at shares of 50% or above. The trace thus retains its same-share ordering under this shift, while the frozen operating point varies with share.
4. Cosine and linear-probe comparisons
Appendix I.5 compares cosine similarity to μC−μN and a linear SVM using the same update kernel and labeled references. All scores are evaluated on the same held-out English checkpoints, with AUROC averaged across backbones.
On the 240 static-refusal pairs in Appendix C.5, TRACE and the SVM achieve within-share macro AUROC 0.977 and 0.939, and across-share AUROC 0.714 and 0.704 (Table 9). The SVM flags 8/16 English compliance and 4/48 non-compliance updates at its cutoff. TRACE with gating flags 14/16 and 5/48, respectively. The linear scores recover much of the ranking, and the prototype fit also supplies the explained fraction used for abstention.