- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper compares behavioral safeguards and representation engineering for safety control and monitoring. For control, it evaluates DPO, CAA, probe-based steering, and flow-based steering using matched safe–unsafe training pairs across three models. For monitoring, it compares four representation probes with a fine-tuned text monitor and Qwen3Guard on native Qwen2.5-32B trajectories. It also evaluates post-generation blocking, corrective regeneration, and stronger steering after benign fine-tuning.
DPO provides the strongest control in the evaluated settings, while Flow is competitive with very small, high-quality datasets. Text monitors achieve stronger overall detection, and native probes require much less estimated marginal computation. Replay-based monitoring substantially reduces harmful outputs after benign fine-tuning. The low-cost native monitoring result and the safety-recovery result are established in different experimental settings.
Critical strengths
- The matched-data control comparison provides useful evidence for choosing between alignment and steering. The paper also examines data scaling, cross-domain transfer, and sensitivity to benign fine-tuning rather than relying on one aggregate safety score (Sec.3, Tables 1, 10, 14, and 16). The full-budget layer comparison is particularly useful because it shows that initial safety does not predict persistence after fine-tuning.
- The monitoring analysis distinguishes discrimination, realized false positives, and detection timing, and explicitly separates native activation reuse from replay (Sec.4, Appendix C.4, and Sec.5). The text-monitor integration baseline in Table 24 further shows that the recovery benefit is not unique to representation probes. These controls make the conclusions more credible.
Critical weaknesses
- Limited generalization across model architectures. The control experiments focus on Qwen2.5 and Llama-3.1, while native monitoring is evaluated only on Qwen2.5-32B (Sec.3–4). Although Appendix D includes Qwen3-4B Thinking, its trajectories are replayed through Qwen2.5-32B rather than evaluated using its native representations. Since the paper studies the utility of internal representations, it remains unclear whether the relative performance of behavioral and representation-based safeguards extends to newer MoE, hybrid, or reasoning models. The broader conclusions therefore require additional validation or a narrower scope.
- Incomplete coverage of existing intrinsic monitoring methods. The monitoring comparison evaluates four probe formulations on a fixed layer but omits recent intrinsic guardrail methods (e.g., ClaudeCLS++, PlugGuard, SingProbe). Prior work already provides evidence for efficient native monitoring and monitor-guided intervention. The paper should clarify what its monitoring findings add beyond this evidence and evaluate a representative recent method under the same protocol. Without such a comparison, it is unclear whether the reported advantage of text monitors reflects a broader difference between the paradigms or the particular probe configurations evaluated here.
Questions for authors
- Suggested additional validation: Could you evaluate representative newer MoE, hybrid, or reasoning models, using their native representations for monitoring, to test whether the relative performance reported in Sec.3–4 persists?
- Clarification: What do the monitoring findings establish beyond recent intrinsic guardrail work such as SingProbe? Suggested additional validation: Could you include a representative recent intrinsic monitoring method under the same training-data and evaluation protocol to distinguish method selection from a broader difference between internal and text-based monitoring?
- Clarification: How are the 1% and 5% thresholds selected with 51 safe full-response calibration trajectories and 28 safe streaming calibration trajectories? Please specify the quantile/tie convention and whether the results are stable across prompt-level splits (Appendix C.4).
5 comments
2. Regarding recent intrinsic monitoring methods (W2, Q2)
We appreciate the reviewer’s suggestion to compare with recent intrinsic monitoring methods and clarify our contribution relative to prior work. We have added CC++ and SingProbe as separate baselines under the same Qwen2.5-32B monitoring protocol.
Because not all implementation details needed for our single-model, single-task setting are specified in the respective papers, we adapted the reported architectures and available code as closely as possible. In particular, we use a single-layer CC++ adaptation with the reported linear probe and SWiM loss, and adapt SingProbe to score response tokens. Both are trained on the same SafeRLHF data splits as our sequence probes, using the same basic training budget. All methods are evaluated on the same set of model-generated trajectories with the same calibration procedure.
| Monitor | AUROC ↑ | AUPRC ↑ | TPR / actual FPR at target 5% FPR | Streaming recall ↑ | Streaming FPR ↓ | GFLOPs / trajectory ↓ |
|---|---|---|---|---|---|---|
| Rolling probe | 0.971 | 0.966 | 0.962 / 0.122 | 0.913 | 0.017 | 0.00452 |
| CC++ adaptation | 0.984 | 0.982 | 0.965 / 0.087 | 0.925 | 0.006 | 0.00452 |
| SingProbe adaptation | 0.972 | 0.976 | 0.928 / 0.060 | 0.928 | 0.052 | 1.91 |
| FT text monitor | 0.995 | 0.995 | 0.983 / 0.062 | 0.575 | 0.163 | 6,569.32 |
| Qwen3Guard | 0.996 | 0.995 | 0.968 / 0.056 | 0.948 | 0.058 | 3,149.44 |
The new probes are evaluated on Qwen2.5-32B’s own representations obtained through same-model replay. Actual test FPRs differ even though thresholds use the same calibration procedure. The estimated FLOPs reflect the probes’ marginal computation, excluding the replay forward pass.
The additional probes perform competitively at substantially lower estimated computational cost. CC++ improves over the rolling probe in full-response AUROC and streaming recall, while SingProbe also achieves high streaming recall. The text monitors retain higher full-response AUROC and AUPRC, but the streaming results depend on the particular monitor and operating point. We therefore do not interpret the full-response gap as evidence that text monitoring is universally superior to representation-based monitoring.
More broadly, prior work has already demonstrated efficient native monitoring and probe-guided intervention. Our contribution is to compare the effectiveness and computational cost of behavioral and representation-based safeguards within a common evaluation framework, including full-response and streaming detection, rather than to establish native monitoring or monitor-guided intervention as new capabilities. We have added these baseline adaptations to Section 4 and Appendix C of the revised manuscript and updated the related-work discussion in Section 2.
References:
Cunningham, H., Wei, J., Wang, Z., Persic, A., Peng, A., Abderrachid, J., Agarwal, R., Chen, B., Cohen, A., Dau, A., Dimitriev, A., Gilson, R., Howard, L., Hua, Y., Kaplan, J., Leike, J., Lin, M., Liu, C., Mikulik, V., … Sharma, M. (2026). Constitutional classifiers++: Efficient production-grade defenses against universal jailbreaks (arXiv:2601.04603). arXiv. https://doi.org/10.48550/arXiv.2601.04603
Sing Team. (2026). SingProbe technical report (arXiv:2608.30703). arXiv. https://doi.org/10.48550/arXiv.2608.30703
3. Regarding threshold calibration and split stability (Q3)
We appreciate the reviewer’s request for clarification. We split trajectories by prompt, using 10% of prompts for calibration and the remainder for testing. For full-response detection, the threshold is the -quantile of the safe calibration scores, computed with NumPy’s method="higher"; a trajectory is flagged when its score is greater than or equal to the threshold. For streaming detection, we first take the maximum score over each safe calibration trajectory and apply the same rule to these maxima. The original streaming protocol uses a 5% sequence-level FPR target.
With 51 safe full-response calibration trajectories, the 1% and 5% thresholds are respectively the largest and third-largest safe scores. With 28 safe streaming calibration trajectories, the 5% threshold is the second-largest trajectory maximum. Scores tied at the threshold are all flagged, so ties can further increase the realized FPR. Even without ties, the corresponding calibration FPRs are 1/51 = 1.96%, 3/51 = 5.88%, and 2/28 = 7.14%, respectively.
To examine stability, we repeated the prompt-level calibration–test split 1,000 times (seeds 0–999), keeping the trained monitors and trajectory scores fixed. The table reports the median and the 2.5th–97.5th percentile range across splits:
| Monitor | Full: TPR / FPR at 1% target | Full: TPR / FPR at 5% target | Streaming: recall / FPR at 5% target |
|---|---|---|---|
| Mean probe | 87.6 [32.5, 95.9] / 1.7 [0.0, 8.5] | 93.6 [81.1, 98.2] / 5.4 [0.4, 15.8] | 89.4 [4.8, 97.7] / 5.6 [0.0, 25.7] |
| Last probe | 68.0 [37.8, 91.0] / 1.4 [0.0, 7.1] | 87.4 [66.8, 97.1] / 5.2 [0.6, 14.4] | 50.1 [16.8, 70.5] / 5.4 [0.0, 23.7] |
| Rolling probe | 84.7 [16.1, 94.2] / 1.6 [0.0, 8.9] | 91.6 [74.2, 96.8] / 5.5 [0.4, 18.3] | 93.1 [16.8, 96.9] / 5.5 [0.0, 27.5] |
| Attention probe | 32.0 [6.5, 91.1] / 1.4 [0.0, 7.1] | 89.7 [16.9, 96.2] / 5.3 [0.4, 13.6] | 83.1 [26.2, 87.4] / 5.5 [0.0, 23.9] |
| CC++ | 85.3 [58.5, 96.3] / 1.3 [0.0, 7.8] | 94.1 [85.8, 98.0] / 5.5 [1.0, 14.5] | 97.4 [90.2, 99.2] / 5.4 [0.0, 21.9] |
| SingProbe | 84.3 [73.1, 94.2] / 1.5 [0.0, 8.1] | 91.9 [84.8, 96.5] / 5.5 [1.0, 14.6] | 93.9 [78.2, 97.3] / 5.6 [0.0, 21.9] |
| FT monitor | 95.1 [87.5, 98.8] / 1.5 [0.0, 8.5] | 98.3 [94.5, 99.4] / 5.4 [0.6, 15.3] | 56.6 [47.9, 62.1] / 6.9 [0.0, 24.9] |
| Qwen3Guard | 93.8 [88.6, 97.7] / 1.2 [0.0, 7.2] | 96.8 [94.0, 99.4] / 5.3 [1.0, 14.2] | 94.3 [80.2, 98.3] / 5.5 [0.0, 22.0] |
All values are percentages. FPRs are measured on the test split, not assumed to equal the calibration target.
The results show substantial split sensitivity for some probes, especially at low FPR targets. These ranges describe variation across overlapping splits of the existing trajectories. Nevertheless, the median results across the evaluated splits remain consistent with the paper’s main conclusions. We have added the threshold convention, split-sensitivity results, and the small-calibration-set limitation to Appendix C.5 of the revised manuscript.
We thank the reviewer for suggesting that we test generalization to additional model families and compare with more recent intrinsic guardrail methods.
1. Regarding generalization to new model families (W1, Q1)
We conducted additional control and monitoring experiments on Qwen3-30B-A3B, a mixture-of-experts model, and NVIDIA-Nemotron-Nano-9B-v2, a Mamba–Transformer hybrid.
For control, we report mean ASR over AIM and refusal suppression, ASR after a benign update, and XSTest over-refusal (OR):
All values are percentages. OR is measured before the benign update.
DPO provides the strongest attack resistance on both models in these settings, while Flow also substantially reduces ASR on Nemotron, with higher OR. The other two steering methods achieve ASRs close to the baseline. We note that the benign update reduces mean ASR in several settings. Investigating the cause is beyond the scope of this work and the time available during the rebuttal period.
We also evaluated general capabilities and control across safety domains. On Qwen3, MMLU accuracy changes from 76.8% for the base model to 76.4% with DPO and 74.0% with Flow; on Nemotron, it remains close to the base model’s 72.2% across methods. For granularity, general-domain DPO reduces Nemotron’s refusal-suppression ASR from 19.40% to 1.49% on cybercrime and from 14.67% to 6.67% on physical harm. However, domain-specific training does not consistently transfer: cybercrime-trained DPO reduces physical-harm ASR only from 14.67% to 13.33%. These results reinforce the importance of considering side effects and control scope alongside jailbreak robustness. We report additional control and monitoring results in Appendices B.4 and D.4 of the revised manuscript.
For monitoring, we trained and evaluated probes using each model’s native activations, following the shared protocol in Sec. 4. We compared two probe formulations with two text monitors:
CC++ is a single-layer adaptation. Thresholds were selected on calibration data. FLOPs are estimated for one full-response detection.
We retain the same evaluation sets to maintain consistency with the main experimental protocol and enable comparison with the main results. On Qwen3, the probes achieve strong ranking performance at substantially lower estimated computational cost than the text monitors. However, the full-response calibration set contains only 48 safe trajectories, which may contribute to conservative thresholds and low TPRs for some probes. On Nemotron, the full-response test set contains only 13 harmful trajectories out of 488, making detection metrics sensitive to individual examples and affecting AUPRC through the low positive prevalence. We therefore interpret the threshold-based results alongside AUROC and AUPRC with caution.
Together, these experiments extend our evaluation to MoE and hybrid architectures and show that the comparison between probes and text monitors remains relevant beyond the original Qwen2.5 and LLaMa setting.
We used both models in non-thinking mode. Reasoning can make successful jailbreaks rare under the evaluated attacks, leaving too few harmful responses for a meaningful monitoring comparison. We did not evaluate additional models in reasoning mode, and the revised manuscript also specifies the non-thinking setting in Appendices B.4 and D.4.