acceptodds
Under review as a conference paper at ICLR 2027

Harmful SFT Leaves a Continuous Trace in LLM Checkpoint Updates

11 views

Abstract

Safety auditing of post-trained large language models typically relies on model behavior, requiring model execution and depending on the coverage of available evaluations. This work asks a different question: Do the target behaviors optimized during supervised fine-tuning (SFT) leave readable evidence directly in checkpoint updates? We find that harmful-compliance SFT induces a continuous, objective-dependent ordering in checkpoint-update space. Using a reference geometry defined by pure harmful-compliance, safety-targeted, and benign-utility SFT, we find that a checkpoint-level coordinate sH tracks controlled harmfulobjective composition with Spearman correlations of 0.986–0.992 across four 7–8B backbones, with the same ordering persisting at larger model scales. Matched compliance-versus-refusal controls show that this checkpoint trace reflects the SFT objective rather than harmful-input exposure, while additional controls rule out simple explanations based on harmful-example count or generic training intensity. Building on this structure, we introduce TRACE, a weights-only auditing method that localizes an unknown checkpoint update relative to frozen harmful and non-harmful reference prototypes and converts this geometry into a continuous harmful-objective score. TRACE requires neither model queries nor access to the unknown SFT data, and can be evaluated directly from checkpoint updates. Across distribution shifts, unseen data, different SFT configurations, partial checkpoint access, and LoRA/full-parameter fine-tuning, the trace remains stable and is positively associated with independently measured attack success rates. TRACE remains informative even at low harmful-objective proportions, providing a complementary auditing signal when behavioral evaluation is unavailable or incomplete. Code is available at https://anonymous.4open.science/r/Code4TRACE-54D3.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Safety;SFT;LLM Audit

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XQ8jOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper asks whether harmful-compliance SFT can be detected from the weights alone. Each checkpoint is represented by its effective update LoRA delta. Small banks of reference adapters are trained on (1) harmful compliance, (2) refusal on harmful prompts, and (3) benign chats. An unknown update is fit as a non-negative combination of (1) and an averaged of (2) and (3). The normalized harmful coefficient s_H serves as the proposed audit score, namely TRACE.

Across four 7–8B base models, s_H rises monotonically with the harmful fraction of mixed SFT data (Spearman ≈ 0.99). It separates compliance from refusal on matched prompts, and tracks the harmful fraction more closely than the absolute number of harmful examples. TRACE outperforms several weight-based and activation-based baselines. It is robust to hyperparameters, partial layer access, LoRA/full-FT transfer and a 10k-example mixture, and it correlates with jailbreak ASR. Trigger-based backdoors are not detected unless a backdoor prototype is added.

Critical strengths

The method is simple and cheap. The kernel is computed from LoRA factors, only a handful of references are needed, and no forward passes are required.

I like the experimental control. Each control rules out a clear competing explanation for the trend: the topic of the prompts, the number of harmful examples, the size of the update, and sequence length.

The paper is also transparent about negative results such as the backdoor experiment.

Critical weaknesses

My main concern is what happens when an audited update looks like none of the references. sHs_H only uses the projection of the update onto the span of the two prototypes. Since it is a ratio, it ignores how much of the update that projection actually captures. For instance, take μC\mu_C and μN\mu_N to be orthogonal unit vectors, then an update that is well-aligned with the reference will look something like 0.9$\mu_C$ + 0.3$\mu_N$ plus a small residual gets sHs_H = 0.75, and the score describes it well. However, if the update is something like a medical-QA fine-tune unrelated to either reference, it can become 0.03$\mu_C$ + 0.01$\mu_N$ plus a large residual, which can get a same sHs_H = 0.75 but is completely misinformed. This could easily come from incidental overlap in answer style or from seed noise. The non-negativity constraint can make this worse. A slight negative alignment with μC\mu_C sets γC\gamma_C to 0, so the update scores as safe. The backdoor experiment looks exactly like this failure: without a matching reference, backdoored adapters score AUROC 0.232. I do acknowledge the authors’ remedy by adding a backdoor prototype, but it requires knowing the attack family in advance, which I view as a "patch fix" and is thus not generalizable nor practical.

More related to my main concern, I don’t think the distribution-shift experiments test real shift. The OOD setting moves between PKU-SafeRLHF and BeaverTails, which come from the same pipeline. The unseen set is again harmful Q&A paired with code and math. In practice an auditor would face two harder cases: (1) benign long-form fine-tunes (medical, legal, creative writing) that could resemble the C prototype in style, and (2) harmful fine-tunes of other kinds (such as cyber, persona roleplay, misinformation) that might not resemble it at all.

I feel the baseline comparison is missing the "most basic baseline": a linear probe on δW\delta_W, or simply cosine similarity to μC−μN\mu_C − \mu_N. If this works effectively, it may mean the contribution is closer to showing that harmful SFT is linearly separable in task-vector space, which prior task-vector work already suggests.

Questions for authors

What fraction of each audited update is explained by the two prototypes, and could this be used to flag updates the references don’t cover?

How does TRACE perform on real community adapters for the same base models? This means both false positives on benign adapters and detection on harmful data beyond PKU-SafeRLHF and BeaverTails.

I'm interested to see how a linear probe or a cosine similarity mentioned above compares to TRACE.

5 comments
AuthorsOpenReview ↗

Response to Reviewer XQ8j

Thank you for the detailed review.

1. Explained update fraction and abstention

Section 6 defines q=1−∥x−x^∥2/∥x∥2q=1-\lVert x-\hat{x}\rVert^2/\lVert x\rVert^2, the fraction of squared update norm explained by the prototype fit on the audited query and value projections. In the review's example, the same sHs_H can accompany very different qq. Figure 6 shows the decision rule. Each backbone fixes the coverage floor at the fifth percentile of qq on 25 English calibration checkpoints and abstains below it.

Median qq on 64 fresh English checkpoints is 1.32% for compliance, 1.33% for refusal, and 1.24% for benign updates. The gate abstains on six benign and three refusal updates, reducing false positives from 8/48 to 5/48 while retaining 14/16 true positives (Appendix I.1). For trigger-conditioned backdoors, Appendix F.2 reports AUROC 0.232 with the original prototypes and 0.721 with a family-specific prototype, illustrating how separation depends on the reference geometry.

2. Benign OOD tasks and public updates

Appendix I.2 scores 20 checkpoints per task per backbone using the frozen English rule. All four task sources are held out of the reference bank.

Benign taskGated FPCovered
Finance1/8080/80
Medicine0/8073/80
Law0/8066/80
Creative writing28/8080/80

The three grounded tasks yield 1/240 false positives with 219/240 covered. Creative writing has full coverage and 28/80 false positives, so task-specific score shifts can persist above the coverage floor. The matched Swahili and Javanese tests reach AUROC 0.965 and 0.970, respectively, with 35/40 compliance and 3/40 refusal updates flagged per language (Appendix I.3).

Appendix I.4 audits four released Qwen3-8B updates. SQSD provides the two compliance directions, including AEGIS as an additional harmful-data source. The ThinkSafe authors provide the two DirectRefusal adapters. The frozen Qwen coverage floor is 0.4544%. SQSD GitHub, ThinkSafe GitHub.

Released updateArmsHs_HqqFrozen call
SQSD AEGISCompliance0.78621.862%Flagged
SQSD BeaverTailsCompliance0.81462.034%Flagged
DirectRefusal-PMRefusal0.11241.226%Unflagged
DirectRefusal-A0Refusal0.09790.350%Abstained

3. Role-play and jailbreak-framed SFT

We scored 320 checkpoints with the original reference bank and frozen English cutoffs. At each of six harmful-prompt shares from 5% to 100%, five matched pairs per backbone share the role-play or jailbreak-framed inputs and differ in compliance versus refusal targets. The remaining 80 checkpoints are benign controls. Mean compliance sHs_H rises with share on all four backbones.

ComparisonMacro AUROC
Compliance vs. refusal, within share1.000
Compliance vs. refusal, shares pooled within backbone0.921
Compliance vs. benign0.992

Compliance scores higher in 120/120 matched pairs. The frozen rule flags 94/120 compliance updates, 25/120 matched refusals, and 0/80 benign controls. Compliance detection rises from 0/20 at 5% to 14/20 at 10% and 20/20 at each higher share. Five compliance updates abstain, all at shares of 5% or 10%. All refusal flags occur at shares of 50% or above. The trace thus retains its same-share ordering under this shift, while the frozen operating point varies with share.

4. Cosine and linear-probe comparisons

Appendix I.5 compares cosine similarity to μC−μN\mu_C-\mu_N and a linear SVM using the same update kernel and labeled references. All scores are evaluated on the same held-out English checkpoints, with AUROC averaged across backbones.

ScoreFresh English macro AUROC
TRACE0.901
Cosine0.891
Linear SVM0.880
Projection difference0.870
UpdateNorm0.734

On the 240 static-refusal pairs in Appendix C.5, TRACE and the SVM achieve within-share macro AUROC 0.977 and 0.939, and across-share AUROC 0.714 and 0.704 (Table 9). The SVM flags 8/16 English compliance and 4/48 non-compliance updates at its cutoff. TRACE with gating flags 14/16 and 5/48, respectively. The linear scores recover much of the ranking, and the prototype fit also supplies the explained fraction used for abstention.

AuthorsOpenReview ↗

Response to Reviewer XQ8j

In addition to the comparisons in our first reply, we tested diverse refusal targets in matched training runs. We generated and checked prompt-conditioned refusals in five modes for 1,113 deduplicated prompts with Qwen3-8B. Across four backbones, compliance scored higher in 236/240 matched pairs with diverse targets. The share-averaged macro AUROC was 0.961, and the mean compliance score exceeded the mean refusal score at each tested share (Figure 11 and Appendix C.5).

We added a benign SFT behavior control on AdvBench. From the base model to step 200 of zero-injection Tulu 3 SFT, ASR rose from 13.28% to 21.01% on Llama and from 5.99% to 12.24% on Qwen. Over steps 10 through 200, the sHs_H changes were 0.0150 and 0.0145, respectively. The control documents behavioral degradation under benign supervision (Appendix F.1).

We also screened four publicly released Qwen3-8B updates with the frozen audit. Both SQSD compliance updates, from AEGIS and BeaverTails, were flagged. Of the two DirectRefusal adapters, one was covered and unflagged and the other fell below the coverage floor (Appendix I.4).

We evaluated cosine similarity to the prototype difference, the raw projection onto that direction, and the update norm. They achieved fresh-English macro AUROCs of 0.891, 0.870, and 0.734, respectively. The cosine result shows that a single direction captures much of the ranking (Appendix I.5).

If you have a chance to review these additions, we would appreciate a brief reply on whether they address your remaining questions about the evaluation.

Median q on 64 fresh English checkpoints is 1.32% for compliance, 1.33% for refusal, and 1.24% for benign updates

So they're all basically the same, and

Creative writing | 28/80 | 80/80

every creative writing update passes the gate too. So q looks the same whether the update is familiar or not. I think this backfires on you. And 28/80 false positives with all of them covered is quite problematic.

TRACE 0.901, Cosine 0.891

Again, this backfires. Cosine basically falls within noise, so the only thing TRACE adds is q, and q doesn't do its job in differentiation.

Reviewer_YtTCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper finds that harmful-compliance SFT leaves an ordered signal in weight updates relative to a trusted base model. TRACE measures this signal using harmful and non-harmful reference prototypes. Controlled mixtures, matched compliance/refusal targets, training variations and behavioral evaluations support the finding, including a 10,000-example SFT experiment.

Critical strengths
  1. The matched compliance/refusal experiment is the strongest evidence here: prompts and training settings are fixed while the targets change. This makes the signal more convincing than a correlation with the amount of harmful-domain data alone.
  2. The empirical checks are substantial. Reference-bank sensitivity, transfer between LoRA and full fine-tuning, and the larger heterogeneous SFT experiment help establish that the trace is not confined to one small training setup.
Critical weaknesses
  1. The comparative advantage needs a stronger baseline check. PEFTGuard reaches only 0.1519 AUROC on matched C/S in Table 3. Appendix H changes its inputs and classification head, while WeightWatch-SVD also departs from the original method. These adaptations are disclosed, but the unusually poor results make the size of TRACE’s advantage hard to assess. A simple supervised classifier using the same update kernel would be a useful reference.
  2. The audit remains strongly dependent on the modification families represented in its references. Table 4 gives 0.232 AUROC for the original geometry on backdoors, improving to 0.721 after adding attack-specific references. This is an informative limitation, but it means a low score is difficult to interpret on an unfamiliar update. The reported ordering and AUROC do not yet show how to set a reliable operating threshold under these shifts.
Questions for authors
  1. Clarification: What are PEFTGuard’s validation results, and does the below-chance matched C/S result persist after checking score orientation on validation data?
  2. Additional analysis: With a threshold fixed on validation checkpoints, what detection rate and false-positive rate does TRACE achieve under the reported distribution and training shifts?
3 comments
AuthorsOpenReview ↗

Response to Reviewer YtTC

Thank you for the detailed review.

1. PEFTGuard score orientation (Q1)

We choose the sign per backbone on validation, using -1 for raw validation AUROC below 0.5 and +1 otherwise. We apply that sign to the original matched C/S split in Table 3.

BackboneRaw val. AUROCSignRaw test AUROCOriented test AUROC
Llama0.3564-10.05170.9483
Mistral0.3337-10.02580.9742
Qwen0.6286+10.30840.3084
OLMo0.5088+10.22170.2217
Macro----0.15190.6132

The validation check raises macro test AUROC to 0.6132. Qwen and OLMo remain below 0.5. TRACE reaches 0.9881 on this split. Appendix H details the adapted PEFTGuard head. Watch the Weights scores activations along attention-output and MLP-down update directions. Our query/value LoRA leaves those matrices fixed. WeightWatch-SVD uses query/value singular subspaces with the same prototype fit as TRACE.

2. Direct comparison on shared checkpoints

We fit a linear SVM per backbone on the same labeled references with the effective-update kernel. All four methods score the same fresh English set (16 C, 48 non-C) and the same 240 C/S pairs with static refusals from Appendix C.5. We use the recorded English score cutoffs, each above the corresponding negative calibration scores. TP/FP gives counts before gating.

MethodEnglish AUROCEnglish TP/FPC.5 within-share AUROCC.5 across-share AUROCC.5 TP/FP
TRACE0.90114/80.9770.714146/80
Linear SVM0.8808/40.9390.704139/76
WeightWatch-SVD (adapted)0.6774/60.5780.553123/103
PEFTGuard (adapted)0.6153/50.6080.599118/89

We compute AUROC within backbone. For C.5, the within-share measure averages eight separate AUROCs, and the across-share measure pools the eight shares. Table 9 and Appendix I.5 give the comparison. The SVM recovers much of the ranking. TRACE yields the largest English TP count and has 8 false positives before gating. Across C.5 shares, the fixed TRACE cutoff flags 80/240 refusals, including high-share updates with higher scores.

3. Frozen decisions under distribution and training shifts (Q2)

Figure 6 in Section 6 shows the rule. Per backbone we use 25 held-out English calibration checkpoints (6 C, 6 S, 13 B). We fix the score cutoff above all 19 non-C scores and the coverage floor at the fifth percentile of calibration qq. The audit abstains below the floor and flags covered updates that meet the score cutoff. Rates use full class denominators.

Test cohortTPR (TP/C)FPR (FP/non-C)Abstained C/non-C
Fresh English87.50% (14/16)10.42% (5/48)0 / 9
Dataset-source transfer91.88% (147/160)0.00% (0/160)0 / 0
Swahili, matched C/S87.50% (35/40)7.50% (3/40)1 / 4
Javanese, matched C/S87.50% (35/40)7.50% (3/40)2 / 2
Finance--1.25% (1/80)-- / 0
Medicine--0.00% (0/80)-- / 7
Law--0.00% (0/80)-- / 14
Creative writing--35.00% (28/80)-- / 0
SFT configuration sweep94.75% (614/648)0.00% (0/108)0 / 0
LoRA references to Full-FT queries91.67% (22/24)0.00% (0/4)0 / 0

Appendix I.1 reports the English cutoffs and calls. Gating removes three English false positives and retains all 14 true positives. Appendices I.2 and I.3 report held-out benign domains and language shifts. Appendix I.6 reports source and training transfer. In the source test, 12 of 13 misses occur at 5% harmful-prompt share. Finance, medicine, and law yield 1/240 false positives with 219/240 covered. Creative writing has 28/80 false positives with full coverage, showing that the operating point depends on the task.

Appendix F.2 studies a different modification family. Trigger-conditioned supervision can yield updates with different alignment to the original prototypes. A family-specific prototype raises backdoor macro AUROC from 0.232 to 0.721. Appendix F.3 measures how adding it affects compliance-SFT screening.

Thank you for the detailed rebuttal and additional experiments, which help address my concerns. I will maintain my original rating of 3 (Weak Acceptance).

Thank you for your review and for maintaining your support and, we truly appreciate the time and care you put into evaluating our work.

Reviewer_oK3JOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents TRACE, a weights-only auditing method to detect whether the model has been finetuned with harmful data. TRACE uses two sets of references, both harmful compliance data and non-harmful data. The method finds the best nonnegative mix of the harmful and non-harmful prototypes that reconstructs the unknown update, and provides insight into how closely the finetuned model resembled harmful-compliance training. Authors show that the method outperforms various weight and activation based baselines. They also have various dataset-family transfer, lora/ full parameter transfer analysis in the paper.

Critical strengths
  • The idea is simple and cheap, without model queries or training data. The authors show that it can work with few reference checkpoints.
  • The authors show robustness evaluation in the experimental setting.
  • The method outperforms baselines by a lot in low-proportion harmful data detection (Table 9).
  • The paper writing and presentation are both clear, and the authors provide mathematical derivations for the method.
Critical weaknesses
  • The refusal checkpoints are trained with 8 static templates described in A.2, so it is a narrow refusal objective, which likely inflates the C/S separation and AUROC scores. This setup also does not address borderline responses/ complicated harmful requests well.
  • There isn’t enough OOD evaluation to support the findings. For example, BeaverTails and PKU-SafeRLHF’s harmful-compliance responses are both from small Llama/ Llama-derived models and are similar in nature. The PKU-SafeRLHF paper (https://arxiv.org/pdf/2406.15513v1) explicitly states that they are a sibling project to SafeRLHF. There is not enough evidence about the manually constructed, disjoint set. The size of that dataset is also small. The backdoor setup with original TRACE has AUROC 0.232, and needs backdoor reference checkpoints to be higher. This weakens the overall claim of the method.
  • It is unclear how to operationalize this finding. The shift differs substantially among different models and setups. For example, Fig 3b shows that the S_H value varies a lot for different models (a bigger contrast shown in Figure 21). It is therefore hard to interpret the number on its own when applying the technique to a new model, especially one fine-tuned on very different harmful content.
  • Various prior work has shown that benign fine-tuning data also raises a model’s ASR (Qi et al., 2024, Xie et al., 2026). The benign checkpoints in Fig 21(b) also show non-zero ASR. The ASR from base model without any finetuning is not reported. The authors motivate the work by discussing the “SFT objective” and “is a geometric coordinate rather than a direct estimator of the fraction of harmful training examples” (line 152) and mentions “harmful-objective” (eg. Table 9). The authors should clarify whether TRACE audits explicitly harmful training data or harmful behavior/ objective. If the latter, than the benign data compromises safety aspect should also be measured.
Questions for authors
  • The method anchors on a predefined set of harmful questions and responses. What if the type of harmful fine-tuning is OOD or do not appear harmful to human eyes? Eg. finetuning in a encrypted language, a low-resource language, using synthetic document fine-tuning to teach models some wrong belief/ misaligned knowledge?
  • What if S query checkpoints are trained on refusals that do not use the eight templates. Report matched C/S AUROC and s_H trajectories.
  • Could you explain the design of the 98 “manually constructed” harmful-compliance example?
  • How should an auditor decide on the threshold for s_H shift?
4 comments
AuthorsOpenReview ↗

Response to Reviewer oK3J

Thank you for the detailed review.

1. Frozen classification and benign-domain transfer (Q4)

Section 6 and Figure 6 show the decision rule. For each backbone we freeze a score cutoff above all 19 refusal or benign calibration scores and a coverage floor at the fifth percentile of explained update fractions across 25 English checkpoints. Below the floor we abstain. Covered updates are flagged at the score cutoff. On 64 fresh English checkpoints, gating retains 14/16 true positives and reduces false positives from 8/48 to 5/48. Appendix I.1 gives the cutoffs.

Benign OOD taskRaw FPGated FPCovered
Finance1/801/8080/80
Medicine0/800/8073/80
Law0/800/8066/80
Creative writing28/8028/8080/80

All four test sources are held out of the fixed reference bank and scored at the English cutoffs. Finance, medical QA, and legal summarization ground responses in calculations, medical facts, or source text. Across these tasks we observe 1/240 false positives and 219/240 covered. Open-ended creative targets allow many valid continuations and may share response patterns with compliance targets. Their mean sHs_H is highest within each backbone, and all 28 flags pass the coverage gate. Appendix I.2 gives the per-backbone evidence. The frozen rule transfers with few false positives to the three grounded tasks and overflags creative writing.

2. Diverse refusal targets (Q2)

We deduplicate the harmful inputs from the original 240 matched pairs to obtain 1,113 prompts (569 PKU-SafeRLHF, 544 BeaverTails). Qwen3-8B generates one candidate per prompt in each of five modes, including explicit refusals and safe redirections. We screen for refusal validity and use the retained targets. A separate comparison uses the labeled refusals for 256 matched WildGuardTrain prompts. The fraction of distinct word bigrams rises from 0.05 with templates to 0.45 for Qwen targets and 0.40 for WildGuardTrain. In the checkpoint comparison, prompts, benign examples, order, seeds, and training settings are matched and we vary only the refusal response.

BackboneStatic AUROCDiverse AUROCPaired wins with diverse refusals
Llama0.9880.97960/60
Mistral0.9630.92557/60
Qwen0.9790.96959/60
OLMo0.9780.97060/60
Macro / total0.9770.961236/240

AUROC is computed within each share and backbone, then averaged equally across shares and backbones. Figure 11 (Appendix C.5) plots the requested sHs_H trajectories and matched C/S AUROCs at all eight shares. Table 11 gives the averages. Compliance means exceed both refusal means at every share, and all four paired exceptions occur at 5% or less. The paired ordering persists across the tested refusal styles.

3. OOD language transfer (Q1)

NLLB-200 translates the prompts and both response targets into Swahili and Javanese. We train matched pairs at 5% and 25% harmful-prompt shares and apply the frozen English audit. Appendix I.3 gives the setup and per-backbone results.

LanguageAUROCGated TPGated FPCovered
Swahili0.96535/403/4075/80
Javanese0.97035/403/4076/80

These results show that the frozen English audit retains separation under both language shifts. The flagged public AEGIS update in Appendix I.4 extends the evidence beyond the two primary harmful-data sources. The backdoor result in Appendix F.2 (0.232 AUROC with the original geometry and 0.721 with an added family-specific prototype) illustrates the dependence on the modification family.

4. The 98-example pool (Q3)

We used AI to generate the questions and wrote the responses ourselves for the 98 harmful-compliance, 100 code, and 97 math pairs. Appendix E.2 gives the nine harmful categories and confirms zero exact prompt overlap with reference or prior query pools. In Figure 14, higher refusal shares replace more benign code and math examples with refusals, lowering the fitted compliance share. Code and math are represented in the original benign references. The matched comparison tests objective separation on new prompts, with within-backbone AUROC of 0.9989 to 1.0000.

5. Supervision and behavior

Appendix F.1 compares the bases with zero-injection Tulu 3 SFT on AdvBench. Each base uses 384 evaluations and each SFT stage pools three trajectories on the same 384 cases.

BackboneBase ASRStep-200 ASRsHs_H from step 10 to 200
Llama13.28% (51/384)21.01% (242/1152)0.1652 to 0.1802
Qwen5.99% (23/384)12.24% (141/1152)0.1604 to 0.1749

Benign SFT raises ASR by 7.73 and 6.25 percentage points. We use ASR to measure attack behavior and sHs_H to measure alignment with harmful-compliance supervision.

AuthorsOpenReview ↗

Response to Reviewer oK3J

In addition to the experiments discussed in our first reply, we tested the frozen English decision rule under changes in data source and training. We applied the same score and coverage cutoffs across all three tests, with no abstentions (Appendix I.6).

TestCompliance updates flaggedFalse positives
PKU-SafeRLHF/BeaverTails source transfer, both directions147/1600/160
Nine SFT configurations varying learning rate, steps, rank, or scaling614/6480/108
LoRA references applied to full fine-tuning22/240/4

Twelve of the 13 source-transfer misses occurred at 5% harmful-prompt share. The decision rule detected all 40 compliance updates at each of the 25% and 50% shares.

We also compared alternative scores on the same held-out English checkpoints. Macro AUROC was 0.901 for TRACE, 0.891 for cosine similarity to the prototype difference, 0.880 for a linear SVM using the same update kernel, 0.677 for adapted WeightWatch-SVD, and 0.615 for adapted PEFTGuard. The cosine and SVM results show how much of the ranking simpler scores recover from the shared update representation (Section 6, Table 9, and Appendix I.5). On the original matched test, selecting the PEFTGuard score direction on validation raised its macro AUROC from 0.1519 to 0.6132 (Appendix H).

If you have a chance to review the revision, we would appreciate a brief reply on whether the new results address your remaining questions about the audit.

Many thanks for the detailed response and the revised manuscript. I appreciate the additional results, clarification, and the narrower scope in the revision. Some of my questions have been addressed with the new experiments and the change in scope. However, I’m still concerned about the operationalization point since the TRACE method emphasizes its use as an auditing signal, but it is a key limitation that the score does not transfer well), and the OOD coverage is limited.

The additional results are helpful and so is the revised content. I am adjusting my score to weak accept but would encourage the authors to fully address the concerns in future versions.

Thank you for the detailed review and the revised assessment. The two remaining concerns — transfer of sHs_H across harmful families and limited OOD coverage — are stated directly in the Limitations section, and the backdoor and creative-writing results are reported as evidence for them. We will pursue family-specific references and broader benign coverage in future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.