- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies context-dependent safety after persona-conditioned fine-tuning in a Llama-3.1-8B-Instruct LoRA case study. Compared with the standard lens, the two persona-conditioned arms show large drops in refusal and higher StrongREJECT harmful-usefulness scores under the compliant lens; the generic instruction-tuning control does not show the same pattern. The authors also show that an apostrophe-matching error in the refusal-pattern set and excluding apparently truncated responses change the reported contrasts. The paper argues for evaluating safety on checkpoint–context pairs and auditing the measurement instrument.
Critical strengths
-
The paper shows why a standard-lens safety score may not describe a fine-tuned checkpoint under other contexts. Both persona-conditioned arms retain high refusal under the standard lens but show large drops under the compliant lens, while the generic instruction-tuning control does not. The pattern persists under a held-out paraphrase, and StrongREJECT reports higher harmful usefulness under the compliant lens. This supports evaluating safety on checkpoint–context pairs.
-
The paper shows that measurement decisions change the reported comparisons. An apostrophe mismatch in the refusal-pattern set causes errors concentrated in the control arms; correcting it reverses the generic control’s contrast in an earlier cohort. A same-checkpoint generation-cap check also shows that excluding apparently truncated responses removes part of the persona-arm contrast. These audits identify specific scoring and exclusion errors that safety evaluations need to check.
Critical weaknesses
-
The paper provides a detailed case study, but its broader contribution remains limited. The main checkpoint–context safety pattern is established in one Llama-3.1-8B LoRA setting; the Qwen experiment yields a different relative pattern, and the 1B study supports only selected parts of the measurement analysis. Since prompt sensitivity after fine-tuning and limitations of automated safety scores have been studied before, the paper’s main addition is to quantify these issues in one pipeline. It remains unclear whether the proposed audit procedure yields new insights beyond this case.
-
The experiments show the weight-by-lens interaction but do not explain its cause. Response style was not matched across training groups, and the planned matched-data test produced no trained model. It remains unclear whether the persona itself matters or the models simply follow the compliant system prompt more strongly after tuning. The reported κ = 0.995 also needs clarification: if it uses the 283 labelled cases with two errors, agreement cannot exceed 0.993.
Questions for authors
-
What are the main methodological advances of the proposed reusable audit procedure beyond existing work on fine-tuning safety, prompt-template sensitivity, and automated safety evaluation? Which elements of the procedure are supported by evidence beyond the specific model family and LoRA pipeline studied in this paper?
-
Could the authors report descriptive comparisons of the training responses across arms (e.g., length, refusal wording, or tone) and discuss whether these differences track the weight-by-lens effects? Given that the prompt-strength results are also compatible with increased sensitivity to system prompts after tuning, what in the current evidence, if anything, favors a persona-specific account?
-
The same 313 StrongREJECT prompts were used for the pre-update screen and the main grid, so it is unclear how well the screening result generalizes beyond this set. Given the Qwen result, why is a stable baseline needed to interpret the difference-in-differences result?
-
How was the reported corrected regex–human agreement of κ = 0.995 calculated? The manuscript reports two false positives among 283 human-labelled records, implying an observed agreement of approximately 0.993, which Cohen's κ cannot exceed. Could the authors clarify the sample, confusion matrix, and calculation procedure used for this statistic?