- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a label-free signal for detecting fine-tuned backdoors, jailbreaks, and prompt injections in LLMs by tracking the movement of the last prefill token’s hidden state across layers. It requires a small set of unlabelled in-domain examples for calibration with low overhead. To demonstrate the proposed method’s effectiveness, the authors conduct empirical experiments on 4 instruction-tuned LLMs on backdoors, jailbreaks, and prompt injection attacks. The paper is also candid about several limitations of the method and provides further analysis of these limits.
Critical strengths
- The proposed method is lightweight and can deal with three security threats: fine-tuned backdoors, jailbreaks, and prompt injections.
- The evaluation scope is reasonably broad, covering all three security risks
Critical weaknesses
- Using signals across layers is not new. The major novelty of the proposed method lies in its processing of across-layer hidden states such as per-layer scoring, covariance-aware pooling, and calibration. However, the contribution of each component in these steps is not very clear and explicit. For example, the paper did not ablate on inter-layer differences (i.e., directly use the raw hidden states for the following processing)
- The paper claims that the proposed method is effective against three security risks: backdoors, jailbreaks, and prompt injections. However, the empirical experiments are not comprehensive enough to demonstrate the superiority of the method against existing methods in these three areas. The paper only compares with 1 baseline for jailbreaks. And the comparison results are mostly in the Appendix.
Questions for authors
- Could you test more ablations on the components of the processing of hidden states in the proposed methods?
- Could you move the comparison to existing methods to the main text for better readability and clarity? Also for backdoor detection, could you compare with DUP (AAAI25’), which also uses cross-layer information? For jailbreaks, could you add more baseline comparisons, such as monitor-based approaches such as linear probe etc.? These could help demonstrate the superiority of the proposed method on these security risks. Currently, the paper is a bit overclaiming.
4 comments
Revised manuscript: ablations and detector comparisons
Thank you again for your helpful suggestions. We have uploaded the revised manuscript and anonymous supplement, with the component ablation and detector comparisons now in the main text, including the DUP adaptation and supervised probe you requested. We also added an injection-detector comparison and clarified the supported pooling contribution and task-dependent results. We would appreciate your updated assessment and any remaining concerns, and would be grateful if you could reconsider your rating in light of these changes.
Thanks for the rebuttal. However, I think the results on transferred GCG are troublesome as LCF’s delta/LW combination appears to be the worst of all four variants. It's the same case for comparison with other baselines. Thus I remain unconvinced that the empirical evidence supports a sufficiently strong contribution in all three threats (backdoors, jailbreaks, and prompt injection). I maintain my current assessment.
LCF’s contribution and the transferred-GCG limitation
Thank you for the follow-up. We agree that delta/LW is the weakest of the four variants on transferred GCG, and the revised paper explicitly reports this result. This substantially limits the case for recommending that configuration for jailbreak detection.
LCF’s methodological contribution is its compact covariance-aware modeling of layer-update anomaly profiles: diagonal scoring produces one anomaly score per layer, and a small covariance model captures relationships among those scores. This supports prefill screening using clean calibration data, without attack labels, parameter updates, or a reference model.
The new experiments provide concrete support for this formulation. Holding delta features fixed, covariance pooling improves refusal-BadNet detection from 74.5% to 92.0% at retrospectively matched 1% FPR. Raw/LW reaches 93.0%, so this supports the pooling component while qualifying the necessity of deltas. Separately, with thresholds fixed using independent clean calibration data, negative-sentiment and refusal BadNet target-prefix successes fall from 200/200 to 0/200 and 198/200 to 4/200, at realized benign FPRs of 5.4% and 4.6%, respectively.
These results support a compact detector with demonstrated component-level and practical benefits in the evaluated backdoor settings. The three-threat evaluation also identifies where this design transfers poorly. We respectfully ask that the controlled pooling evidence and independently measured protection be weighed alongside the substantial GCG limitation when assessing the revised contribution.
Response to Reviewer URBP: component evidence and practical value of LCF
Thank you for recognizing LCF’s lightweight design and coverage of three threat families, and for requesting stronger component and detector comparisons. We have completed additional Qwen2.5-7B experiments, including two newly reconstructed BadNet adapters (one training seed each). The new evidence identifies a concrete covariance-pooling benefit and demonstrates strong backdoor protection at independently measured benign-rejection rates, with low measured prefill cost.
1. Isolating LCF’s components. LCF’s distinctive processing models the joint profile of layer-update anomalies: diagonal scoring within each layer, followed by a Ledoit–Wolf covariance score over standardized scalar layer scores. On Qwen7B, the cross-layer covariance is 28×28, alongside the diagonal coordinate statistics. This explicitly models clean co-variation across layers.
We evaluated raw states versus adjacent-layer differences, each with max and LW pooling, using identical activation caches, calibration data, and test IDs. Each representation fits its own clean statistics. Detection (%) at retrospectively matched 1% / 5% clean FPR is:
All four detect 100% of negative-sentiment BadNet attempts at both operating points. On refusal BadNet, LW raises delta-based detection from 74.5% to 92.0% at 1% FPR and from 93.0% to 97.0% at 5%, and improves raw-state detection as well. This isolates a useful pooling contribution. Raw/max performs best among the four variants on GCG; the benefit of deltas is task-dependent. We will include the complete 1%/5%/10% ablation.
2. DUP and monitor comparisons. Detection (%) at retrospectively matched 5% FPR is:
For DUP, we retain its distance and spectral signals and use all layers. Its class-based CH layer selection is not reproduced because the required clean task-class labels are unavailable for our heterogeneous generation prompts. We will explicitly label this generation-setting adaptation.
The fixed last-layer linear probe is trained separately for each setting using five-fold grouped out-of-fold evaluation, with 160 attack and 160 clean training labels per fold. Both GCG variants of each behavior, and each BadNet clean/trigger pair, stay in the same fold. This is a supervised reference with additional attack-label access. The table identifies strong alternatives for GCG and competitive clean-only backdoor results for LCF. The small 2.5-point refusal advantage over DUP is inconclusive: its descriptive paired 95% interval includes zero.
We also completed an AttentionTracker injection comparison. At retrospectively matched 5% FPR, code-injection detection is 100%/73% for LCF in email/code-QA versus 70%/59% for AttentionTracker. This compares systems with different backbones and wrappers: Qwen2.5-7B for LCF and AttentionTracker’s published Qwen2-1.5B configuration.
3. Practical contribution and presentation. Keeping the original 200-example feature fit fixed, we added 500 clean threshold-calibration examples and 500 separate clean tests. At a 5% target, LCF reduces admitted strict target-prefix successes from 200/200 to 0/200 for negative BadNet and from 198/200 to 4/200 for refusal, at 5.4%/4.6% realized FPR. These results connect the clean-calibrated score to useful tested backdoor protection. The final all-layer pipeline adds 0.98 ms mean prefill/capture/scoring cost in the tested Qwen2.5-7B batch-one A100 short-prompt benchmark.
We will move the component ablation and detector comparisons into the main text and explain their supervision, adaptation, and calibration requirements. The supported contribution is a compact clean-calibrated prefill detector with demonstrated backdoor operating points and low measured cost. We will scope superiority claims to the reported comparisons and retain the domain-shift and adaptive-evasion limitations; these new experiments do not establish broader model generality or adaptive robustness.