acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Temporal Risk Assessment and Concept Editing for Safer Language Generation

Abstract

Large language models (LLMs) can produce harmful content while generating a response. A useful safeguard should not only detect this risk during generation but also intervene while leaving safe responses largely unchanged and provide an auditable record of the internal patterns supporting its decisions. We propose TRACE (Temporal Risk Assessment and Concept Editing), a framework that tracks harmful-response risk during generation, selectively edits token activations, and leaves an auditable trace of its detection and steering decisions. TRACE learns a codebook of recurring activation patterns, with each codebook vector representing a latent concept. At inference time, it assigns each token activation to the nearest codebook vector to obtain the corresponding concept. A gated recurrent unit (GRU) processes the activations and their assigned concept features to predict completed-response harmfulness and a monotone cumulative streaming risk. During steering, activations assigned to harmful concepts are steered along directions defined by nearby benign targets. An external LLM labels the learned concepts post hoc, making detection sequences and harmful-to-benign edit records interpretable. Experiments across three generators and the WildGuard and S-Eval benchmarks show that TRACE improves streaming F1 over Kelp in five of six comparisons while remaining competitive on completed-response detection. TRACE's selective steering reduces Harmbench attack success with little change in refusal on originally safe responses. The post-hoc audit reveals recurring semantic structure in the learned concepts and distinct traces associated with harmful detection, safe generation, and activation edits.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, large language models, harmful content detection, activation steering, representation engineering, mechanistic interpretability, vector quantization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CwrxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TRACE combines a label-supervised cross-layer vector-quantized codebook, a causal GRU detector, and concept-conditioned activation steering. The detector jointly predicts completed-response harmfulness and monotone cumulative streaming risk from response-level labels; steering independently gates token-level, norm-preserving edits using harmful-to-benign concept-mean differences. Post-hoc LLM descriptions make the recorded concept sequences and edit pairs readable. Evaluation covers three 8B generators on WildGuard and S-Eval, with HarmBench-based judgments of regenerated outputs.

Critical strengths
  1. A coherent, inspectable safeguard design. Sharing a discrete representation across detection, intervention, and logging provides a concrete way to inspect what the system encountered and edited. Unlike prompt-conditioned CAST, steering is reassessed at each generated-token activation, allowing intervention decisions to change during a response. This integration is practically meaningful even without treating concept names as causal explanations.

  2. Substantial empirical support for detection. TRACE improves streaming F1 over the reproduced, layer-matched Kelp baseline in five of six settings, including 0.935 versus 0.896 on Qwen/S-Eval and 0.961 versus 0.919 on Llama/S-Eval (Table 1). Threshold-independent metrics and validation-calibrated detection-position analyses strengthen the evidence beyond a single operating point, although these comparisons do not establish statistical significance. The appendices disclose baseline-reproduction choices and report detector-seed variation, while correctly noting that the latter fixes the VQ representation.

Critical weaknesses
  1. The distinctive contribution is not sufficiently isolated. The cross-layer codebook builds on CLVQ-VAE; SCAR already combines concept detection and steering, and Breaking Bad Tokens includes token-conditioned detoxification (Section 5). Relative to these precedents, TRACE's distinctive contribution appears to lie in its shared discrete representation, dual-head temporal detector, and concept-pair steering rule. Table 9 shows slightly higher activation-only streaming F1 on Qwen/WildGuard (0.844 versus 0.837) and only a 0.001 advantage for combined inputs on Qwen/S-Eval; the authors appropriately acknowledge that concepts are not the main source of detection accuracy. The remaining case for their value relies on steering and auditing, but named concepts cover 46.4% of Qwen/WildGuard test-token assignments, while the unnamed share averages 75.75% across true-positive traces after consecutive concept repetitions are merged (Table 5). This limits semantic coverage, without invalidating the recorded edits; controlled evidence that the representation improves diagnosis or that nearest-benign targets outperform simpler alternatives would strengthen the contribution.

  2. The safety-utility claim is only partially validated. Three selected settings do not satisfy all quality criteria used to approve stronger steering: mean non-repetition is 0.39 for Llama/WildGuard and 0.48 for InternLM/S-Eval, while 26/200 originally safe validation cases newly fail the quality check for InternLM/WildGuard (Table 11). The stated selection rule permits retaining strength 0.5 in these circumstances, so this is not a contradiction in the procedure; rather, it leaves degradation as a possible contributor to reduced ASR, as the authors acknowledge. Stable aggregate, phrase-based refusal rates on originally safe responses do not establish preserved helpfulness, correctness, or individual refusal decisions, especially when many such responses already refuse. Finally, judging only the first 512 classifier tokens of outputs capped at 2,048 generator tokens leaves later harmful content unassessed (Appendix D.2); this limitation applies to all compared steering methods.

Questions for authors
  1. Contribution-isolation request: Beyond the Qwen input ablations, does the activation-only, dual-head GRU retain TRACE's detection gains on Llama and InternLM? With the harmful-concept gate and update rule fixed, do nearest-benign targets outperform a global refusal direction or randomly selected eligible benign targets at comparable output quality and validation-calibrated strength?

  2. Safety-utility validation request: Appendix F.2.2 explains that strength 0.25 is excluded from selection because it reduces ASR less, although Table 11 shows that it meets all four quality criteria in the three problematic settings. Do the safety-utility conclusions persist if strength selection requires every reported setting to satisfy those criteria? Full-output harmfulness assessment and helpfulness evaluation on benign, non-refusal tasks would materially strengthen this assessment.

  3. Methodological clarification: Equation (12) rotates the activation toward the normalized source-to-target difference vector, not toward the benign mean itself; norm preservation alone does not guarantee proximity to the target concept. This is a valid steering construction, not an algebraic error, but what evidence supports its intended benign effect beyond reduced ASR and the existing direct-addition and harmfulness-weighting ablations?

  4. Streaming-safety validation request: In a manually annotated subset, how often do threshold crossings precede the first harmful information, and how much harmful text is released before detection? The paper correctly disclaims onset localization; normalized detection position and response-level F1 nevertheless do not by themselves establish prevention of harmful disclosure.

7 comments

Q1 Contribution-isolation request: Beyond the Qwen input ablations, does the activation-only, dual-head GRU retain TRACE's detection gains on Llama and InternLM? With the harmful-concept gate and update rule fixed, do nearest-benign targets outperform a global refusal direction or randomly selected eligible benign targets at comparable output quality and validation-calibrated strength?

A1. Thank you for suggesting these comparisons. On both WildGuard and S-Eval, Llama and InternLM show similar detection performance with activation-only and activation-plus-concept inputs, consistent with the Qwen ablation. Their streaming F1 scores differ by at most 0.030 across the four settings. The activation-only detector also exceeds Kelp’s streaming F1 in all four. The concept representation therefore maintains competitive detection while providing the concepts used for selective steering and auditing. Appendix F.1.7 reports the detector variants.

For steering, we compared nearest-benign targets with a global refusal direction and randomly chosen eligible benign targets. We repeated the random-target experiment with three seeds, assigning each harmful concept a benign target at random. Both alternatives retained TRACE’s concept assignments, harmfulness scores, and update rule. We used the same validation procedure to select each method’s strength.

The below table reports averages across all six generator–dataset settings, with detailed results for each setting in Appendix F.2.6. Random-target results are averaged across three seeds. For these comparisons, we recomputed test ASR using the longer HarmBench judging window described in A2. Steering strengths were still selected using the original validation procedure.

MetricNearest-benign (TRACE)Random benign targetsGlobal refusal direction
ASR ↓0.2730.1310.206
Non-repetition ↑0.550.340.67
Safe quality failures (%) ↓8.921.57.3
Absolute change in refusal rate on safe responses (pp) ↓0.550.645.94

Compared with the average of the three random mappings, nearest-benign targets have higher non-repetition and fewer safe-response quality failures in all six settings. Random targets achieve lower ASR on average, but produce more repetition and cause more safe responses to fail the quality checks. Choosing a nearby target therefore helps preserve quality, even when a random target reduces ASR more.

The global refusal direction achieves lower average ASR, higher non-repetition, and fewer safe-response quality failures. However, it causes a larger change in the aggregate refusal rate on originally safe responses in every setting. Nearest-benign steering keeps this rate closer to its unsteered value. This global-direction ablation still uses TRACE’s concept assignments and harmfulness scores to decide when and how strongly to edit. It replaces only the direction with a shared refusal direction.

Response to Q2 (Part 1/2): Steering-strength selection

A2 Thank you for suggesting this test. When we require every selected strength to pass all four validation quality checks, three of the six settings switch to a lower strength of 0.25: Llama/WildGuard and both InternLM settings. At this strength, text quality improves, but the reduction in ASR is much smaller than at 0.5. The other three settings retain their reported strengths.

To test this, we included 0.25 in the strength selection and chose the strength with the lowest validation ASR among those passing all four quality checks. The table compares test ASR and text quality for the three affected settings. Safe quality failures count originally safe responses that pass the checks before steering but fail afterward.

SettingStrengthASR before → afterNon-repetitionSafe quality failures
Llama/WildGuard0.250.874 → 0.8200.77108/1,519
Llama/WildGuard0.500.879 → 0.5920.45133/1,519
InternLM/WildGuard0.250.862 → 0.8620.9090/1,486
InternLM/WildGuard0.500.870 → 0.4350.59150/1,486
InternLM/S-Eval0.250.782 → 0.6550.9277/615
InternLM/S-Eval0.500.766 → 0.1710.5186/615

For Llama and InternLM on WildGuard, 0.25 leaves harmful generation largely unchanged. This explains why we originally used 0.5 as the default: it reduces ASR much more, while 0.25 better preserves text quality. Even at 0.25, InternLM/S-Eval exceeds the 10% safe-response quality failure limit on the test set, at 12.5%.

We report the full-answer harmfulness and benign-task evaluations in Part 2 below.

Response to Q2 (Part 2/2): Full-answer and benign-task evaluation

A2. Full-answer assessment. For all test comparisons of TRACE, steering baselines, and steering ablations, we now use all the available space in HarmBench's 2,048-token context. We keep the judging instructions and original request in full, reserve one token for the verdict and eight as a buffer, and use the remaining space for the response. Answers that do not fit are truncated at the end. The test ASRs reported here use these judgments. We still choose the steering strength using HarmBench’s judgments of the first 512 tokens of each validation response. Appendix D.2 describes the HarmBench evaluation.

We also sent Gemini 3.1 Flash-Lite the complete unsteered and steered answers for all 284 Qwen/WildGuard prompts whose original dataset responses were labeled harmful, using the reported steering setting (lambda=0.75). Gemini did not see method names and judged content and quality separately. Partly coherent answers had quality problems but remained understandable, while substantially degraded answers had meaning that could not be understood.

Gemini judged fewer unsteered answers harmful than HarmBench: 125 versus 257 among the 280 answers judged by both. Our manual checks suggest that Gemini was sometimes misled by adversarial framing in the requests and classified harmful answers as safe. This raises concerns about relying on Gemini alone for harmfulness judgments in this setting. We therefore retain HarmBench for the reported ASR and use Gemini to examine whether the complete steered answers provide understandable refusals or safe alternatives, or become degraded.

HarmBench judged 261 of the 284 unsteered answers harmful. After steering, 169 of these were judged non-harmful and 92 remained harmful. The other 23 answers were judged non-harmful both before and after steering. To examine what accounts for the ASR reduction, the table below shows Gemini’s assessment of response quality for those 169 steered answers.

OutcomeNumber of answers
Coherent refusal or safe alternative47
Partly coherent refusal or safe alternative38
Substantially degraded78
Harmful answer, coherent or partly coherent1
Partly coherent, but content unclear2
Judging request blocked3

Of these 169 answers, Gemini classified 85 as understandable refusals or safe alternatives and 78 as substantially degraded. These results suggest that the reduction in ASR comes from both safer responses and degraded text.

Helpfulness on benign requests. We also evaluated the Qwen/WildGuard setting on 804 AlpacaEval requests. We gave Gemini each request and two answers, one generated without steering and one with TRACE. We randomized which answer appeared first and did not reveal which used steering. Gemini compared their helpfulness based on correctness, relevance, completeness, and instruction following. It also checked whether each answer provided what the user asked for.

MeasureResult
Similarly helpful after steering60.7%
More helpful after steering9.0%
Less helpful after steering30.3%
Adequately addresses the request, before → after87.9% → 75.9%

Most steered answers were judged equally or more helpful than the originals. However, steering reduced the proportion of answers judged to adequately address the user's request.

Correctness on math questions. We evaluated all six reported steering settings on the 1,319 questions in the GSM8K test set. We compared final numerical answers with the references, counting missing or invalid answers as incorrect. The table shows accuracy before and after steering.

GeneratorWildGuard-trained (%)S-Eval-trained (%)
Qwen94.2 → 93.693.7 → 21.5
Llama86.4 → 84.986.1 → 3.3
InternLM90.5 → 84.689.9 → 73.0

With WildGuard-trained steering, accuracy remains broadly stable for Qwen and Llama but decreases for InternLM. The S-Eval-trained settings cause larger drops across all three generators. These differences suggest that the data used to learn the concepts and select steering settings may affect how well correctness is preserved on tasks beyond the safety benchmarks.

Appendix F.2.2 reports the complete-answer, helpfulness, and correctness evaluations. Appendix F.2.3 gives the Gemini prompts.

Reviewer_KSzYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TRACE is a framework for detecting unsafe language generation, and selectively intervening on model activations. It also emphasizes the capability of recording the interventions in an interpretable form for auditing. It uses a learned VQVAE codebook of activation patterns to connect streaming risk detection with steering toward benign patterns, with retroactively applied labels helping interpret the records. Experiments across three language models and two datasets show competitive detection and reduced harmful generation.

Critical strengths
  • The paper is generally well written and easy enough to follow
  • The proposed pipeline for concept-based safety steering is sensible, connecting detection, selective intervention, and auditing through a shared representation.
  • Providing an auditable record of which activation patterns triggered interventions and which targets guided the edits is a meaningful motivation.
Critical weaknesses
  • Preservation of useful behavior is not sufficiently established. Similar refusal rates on originally safe responses do not show that those responses remain helpful. Basically I would want to see a helpfulness/harmfulness tradeoff that is standard in this kind of paper.
  • The methodological contribution appears primarily a system integration of existing methods, combining existing cross layer VQVAE representations, streaming detection, and conditional activation steering. The distinct benefits of the shared concept representation could be established more clearly, particularly because an activation-only detector achieves comparable accuracy
  • Some figures are difficult to read because of small text and could use the available space more effectively.
  • Figure 2’s takeaway is also unclear: TRACE and Kelp trade off detection timing and false-positive rate without consistent improement. This is exacerbated by the desaturated colors and small size.
Questions for authors
  • How much of the reduction in attack success comes from safer responses versus degraded generation? Can you evaluate helpfulness and correctness on benign tasks, beyond refusal rates and the reported text-quality checks?
  • Does the discrete concept representation provide benefits beyond auditing and interpretation, given that the activation only detector achieves comparable accuracy? In particular, how important is it for effective selective steering?
  • What conclusion should readers draw from Figure 2? Does TRACE detect harmful responses earlier than Kelp at matched false-positive rates and comparable recall?
4 comments

Q1. How much of the reduction in attack success comes from safer responses versus degraded generation? Can you evaluate helpfulness and correctness on benign tasks, beyond refusal rates and the reported text-quality checks?

A1. Thank you for raising this question. We examined the content and quality of responses before and after steering. We also tested helpfulness on benign requests and correctness on math questions. All evaluations used the reported steering settings.

Safety versus degraded generation. Of the 169 Qwen/WildGuard responses that HarmBench judged harmful before steering and non-harmful afterward, Gemini judged 85 as understandable refusals or safe alternatives and 78 as substantially degraded. This shows that the reduction in ASR comes from both safer responses and degraded generation.

We sent Gemini 3.1 Flash-Lite the complete unsteered and steered answers for all 284 Qwen/WildGuard prompts whose original dataset responses were labeled harmful, without revealing method names. Content and quality were judged separately. Partly coherent answers had quality problems but remained understandable, while substantially degraded answers had meaning that could not be understood.

Gemini judged fewer unsteered answers harmful than HarmBench: 125 versus 257 among the 280 answers judged by both. Our manual checks suggest that Gemini was sometimes misled by adversarial framing in the requests and classified harmful answers as safe. This raises concerns about relying on Gemini alone for harmfulness judgments in this setting. We therefore retain HarmBench for the reported ASR and use Gemini to examine whether the complete steered answers provide understandable refusals or safe alternatives, or become degraded.

HarmBench judged 261 of the 284 unsteered answers harmful. After steering, 169 of these were judged non-harmful and 92 remained harmful. The other 23 answers were judged non-harmful both before and after steering. To examine what accounts for the ASR reduction, the table below shows Gemini’s assessment of response quality for those 169 steered answers.

OutcomeNumber of answers
Coherent refusal or safe alternative47
Partly coherent refusal or safe alternative38
Substantially degraded78
Harmful answer, coherent or partly coherent1
Partly coherent, but content unclear2
Judging request blocked3

Helpfulness on benign requests. Among the 802 AlpacaEval comparisons judged successfully, 60.7% of steered answers were judged similarly helpful to the originals, 9.0% more helpful, and 30.3% less helpful. The proportion of answers that adequately addressed the request also decreased from 87.9% to 75.9%.

We evaluated the Qwen/WildGuard setting on 804 AlpacaEval requests. We gave Gemini each request along with its unsteered and TRACE-steered answers. We randomized the order of the answers and did not reveal which one used steering. Gemini compared their helpfulness based on correctness, relevance, completeness, and instruction following. It also checked whether each answer provided what the user asked for.

MeasureResult
Similarly helpful after steering60.7%
More helpful after steering9.0%
Less helpful after steering30.3%
Adequately addresses the request, before → after87.9% → 75.9%

Although most answers were judged equally or more helpful after steering, fewer answers adequately addressed the user's request.

Correctness on math questions. With WildGuard-trained steering, accuracy remains broadly stable for Qwen and Llama but slightly decreases for InternLM. S-Eval-trained steering causes larger accuracy drops across all three generators.

We evaluated all six settings on the 1,319 questions in the GSM8K test set. We compared the final numerical answers against the references, counting missing or invalid answers as incorrect. The table shows accuracy before and after steering.

GeneratorWildGuard-trained (%)S-Eval-trained (%)
Qwen94.2 → 93.693.7 → 21.5
Llama86.4 → 84.986.1 → 3.3
InternLM90.5 → 84.689.9 → 73.0

The differences between the WildGuard- and S-Eval-trained settings suggest that the data used to learn the concepts and select steering settings may affect how well correctness is preserved on tasks beyond the safety benchmarks.

Appendix F.2.2 reports the complete-answer, helpfulness, and correctness evaluations. Appendix F.2.3 gives the Gemini prompts.

Q2. Does the discrete concept representation provide benefits beyond auditing and interpretation, given that the activation only detector achieves comparable accuracy? In particular, how important is it for effective selective steering?

A2. Thank you for the question. The learned concepts are used to decide which activations to edit, how strongly to edit them, and which direction to use. Their role therefore goes beyond making the traces interpretable. Our comparison with K-means shows lower average ASR and fewer safe responses failing the quality checks after steering. This benefit is separate from detection accuracy, where the activation-only detector performs comparably.

During steering, TRACE checks the harmfulness score of the concept assigned to each token. It edits the activation when this score is positive, with higher scores giving stronger edits at the same lambda strength. It also selects a nearby benign concept. The difference between the harmful and benign concepts’ mean activations defines the steering direction. These decisions use the concept assignments and scores, independently of the GRU’s predictions.

We tested the learned representation by replacing CLVQ-VAE with 800 K-means clusters of the same read-layer activations across all three generators and both datasets. We kept the harmfulness-scoring formula, detector architecture and hyperparameters, and steering procedure fixed. Each method’s steering strength was selected on validation data using the same rule.

Detection performance remains close: TRACE has higher response F1 in all six settings and higher streaming F1 in five. The larger differences appear in steering. Average ASR is 0.273 for TRACE versus 0.499 for K-means. For each setting, we also measured the percentage of originally safe responses that pass the quality checks before steering but fail afterward. Averaged across the six settings, this is 8.9% for TRACE versus 29.0% for K-means.

For example, on Qwen/S-Eval, TRACE achieves ASR 0.043 versus 0.308. Among 467 originally safe responses, 30 fail previously passed quality checks after TRACE steering, compared with 149 for K-means. The advantage is not uniform: K-means achieves lower ASR on Llama/WildGuard and fewer safe-response quality failures on InternLM/WildGuard.

These results show that similar detection performance does not necessarily lead to similar steering outcomes. Compared with K-means, CLVQ-VAE achieves lower average ASR while causing fewer safe responses to fail the quality checks. Appendix F.3 reports the full comparison.

Q3. What conclusion should readers draw from Figure 2? Does TRACE detect harmful responses earlier than Kelp at matched false-positive rates and comparable recall?

A3. Thank you for the question. Figure 2 compares where each method first detects harmful responses and how often it incorrectly flags safe responses, using the thresholds specified in the experimental setup. At these thresholds, TRACE has fewer false positives in five of six settings, while Kelp detects harmful responses earlier in some settings.

The methods do not have equal FPR or recall in this figure. The median detection position is also calculated only from the harmful responses each method detects. A method can therefore have an earlier median while missing more harmful responses. We need to consider detection position, recall, and false-positive rate together.

Appendix F.1.2 compares the methods using a common 5% validation FPR target to select their thresholds. With these thresholds, TRACE detects more harmful test responses in all six settings and has an earlier median detection position in four. Kelp has an earlier median on Qwen/WildGuard and InternLM/WildGuard.

Additional concern

  1. Methodological contribution and the shared representation

Thank you for this comment. TRACE uses CLVQ-VAE for concept learning and designs its detector and selective steering procedure around the learned concepts. These concepts provide detector inputs, guide activation edits, and let us inspect the patterns preceding detection and the source and target of each edit. As discussed in our response to Q2, the new comparisons show that the learned representation affects steering performance even when detection accuracy is similar.

  1. Figure readability

Thank you for pointing this out. We enlarged the text and improved the spacing in Figure 2. We also use a clearer blue–orange color scheme consistently for TRACE and Kelp in Figures 2, 13, and 14, making the methods easier to distinguish.


We sincerely thank the reviewer for their thoughtful feedback and would be very happy to clarify any remaining questions. If our response has addressed the concerns, we would be truly grateful if the reviewer would consider revising the score.

Thank you again for your review. We have addressed your questions and updated the manuscript with additional evaluations of response quality, helpfulness, and correctness, along with further detection and steering analyses.

As the discussion period is nearing its end, we would appreciate any feedback on whether these results address your concerns. If our responses have addressed your concerns, we would be grateful if you could consider revising your score. We would be happy to clarify anything further.

Thank you for your response. Based on your rebuttal I would like to retain my score of 3.

Reviewer_jsbvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a framework for a streaming safeguard which utilizes a cross layer VQ-VAE to obtain 'concepts' for each token, which along with the corresponding activations, are then fed into a GRU to predict a full-response harmfulness risk and a cumulative streaming risk. This risk can be thresholded to stop generations. Additionally, their framework is used to edit concepts during generation by steering the activations of a harmful concept towards the nearest benign ones (via rotating the activations instead of adding a steering vector). The authors argue that the main advantage of their framework is interpretability and audibility; their approach allows for a complete trace of when and how unsafe responses were intervened on.

Overall, the framework is sensible, combining a few existing ideas (extracting concepts via codebooks and using them to trigger activation steering) which seems to improve the streaming F1 scores over other related methods, but there are a few missing evaluations and ablations which would validate the robustness and usefulness of their approach. I think the concept editing idea could be useful in theory, but I would need more evidence to convince me of its utility over simpler alternatives.

Critical strengths
  • Presentation is mostly good (though I got a bit lost with some of the concept cloud figures)

  • The framework is practical and sensible. It is a system that could realistically be implemented in production systems. The traceable records of concepts for why a generation may have been deemed harmful or not is a nice feature.

  • The work that was executed seems quite thorough and well implemented.

Critical weaknesses
  • No adaptive adversarial evaluations were done, only static jailbreaks were evaluated. I would expect that active/adaptive jailbreak algorithms would obfuscate the activations such that the concepts could be manipulated, bypassing your system entirely (e.g. [1]).

  • Concepts don't seem to be critical to the performance of the harmfulness detector. In Table 9, the activation input is comparable or better to the concept. I guess the point of this work is the interpretability angle (which is what you mentioned in the caption), but then you really need to focus on this angle rather than the explicit defense/ASR reduction angle you currently have. Furthermore, this ties to my next point...

  • It's unclear what the benefit of the concept editing is over just steering (e.g. refusal, other steering vectors) as I don't think this ablation was done, and most of the edited concepts are just refusals if I read the results correctly (please correct me if I missed this). If there's not much advantage to the concept editing portion of your system, then I'm not sure there's much point over existing streaming classifiers such as constitutional classifiers.

  • I don't think I like the use of a monotonic risk accumulation, as there are some very naive failure modes that can arise. For example, you could have a spike due to some potentially harmful keyword (e.g. "How do you kill a Linux process?" -> "To kill a linux process...") which means you're now forever stuck in this higher risk regime. Another failure mode is small scores accumulating over a long context. You could argue this is a calibration problem, but calibration is non-trivial to set correctly.

[1] Bailey, Luke, et al. "Obfuscated activations bypass LLM latent-space defenses." ICLR 2026

Questions for authors
  • Did I miss an ablation on the concept editing vs steering vector addition? I saw the comparison between rotation and addition, but how do the generated responses compare to just steering towards something simple like refusal? If this is comparable, then there's no real point to doing the whole VQ-VAE component because as per your results in Table 9, the concepts don't seem necessary for the classification. This would reduce your whole approach to a much simpler 'classify and refusal steer' framework.

    • Additionally, it would be nice to see actual traces of the generations pre/post concept edit. I'd like to have intuition for what this operation actually does rather than trust the aggregate metrics.
  • Is the risk accumulation a potential issue in longer contexts? What's the distribution of the scores scs_c, and how stable are they on a held out test set of benign and harmful tokens?

  • Does the learned codebook contribute beyond simpler alternatives? How do detection, steering, and audit results change if the CLVQ-VAE is replaced by (a) k-means clustering of the same read-layer activations, using the same label-derived harmfulness scores, or (b) SAE features, e.g. as in NExT-Guard?

7 comments

Q1. Did I miss an ablation on the concept editing vs steering vector addition? I saw the comparison between rotation and addition, but how do the generated responses compare to just steering towards something simple like refusal? If this is comparable, then there's no real point to doing the whole VQ-VAE component because as per your results in Table 9, the concepts don't seem necessary for the classification. This would reduce your whole approach to a much simpler 'classify and refusal steer' framework.

A1. Thank you for suggesting this comparison. We added a VQ-free "classify and refusal steer" baseline to test whether a simpler pipeline can achieve comparable results. An activation-only GRU monitors generation, and once its risk crosses a validation-selected threshold, the baseline applies a single refusal direction to subsequent token activations. We construct this refusal direction using the harmful-minus-harmless mean-activation method of Arditi et al. (2024) and apply the same norm-preserving update as TRACE. The threshold targets at most 5% validation FPR, and steering strength is selected using the same validation rule as TRACE.

The table reports averages across all six generator–dataset settings. Appendix F.4 in the paper gives the setup and results for each setting.

MetricTRACEClassify-and-refusal
ASR ↓0.2730.320
Non-repetition ↑0.550.50
Absolute change in safe-response refusal (pp) ↓0.553.36

TRACE achieves lower ASR in five of six settings and keeps refusal on originally safe responses closer to its unsteered value in all six. Average non-repetition is also slightly higher for TRACE, although each method performs better in three settings.

The activation-only detector performs similarly to the activation-plus-concept detector, consistent with Table 9. However, the results above show that this simpler pipeline has a worse ASR/refusal trade-off than TRACE. Additionally, beyond these metrics, the shared VQ-VAE representation allows us to track which concepts appear before a detection is triggered and which harmful concept and benign target are involved in each steering edit. We can then examine these records after generation, using semantic descriptions where agreed labels are available. This auditing capability is another important contribution of TRACE and is absent from the "classify and refusal steer" baseline.

Q1 follow-up. Additionally, it would be nice to see actual traces of the generations pre/post concept edit. I'd like to have intuition for what this operation actually does rather than trust the aggregate metrics.

A1 follow-up. We have added eight Qwen/WildGuard examples in Appendix E.4 (“Qualitative analysis of steering”) covering refusal, safe redirection, preserved answers to benign requests, and output-quality failures. We reviewed responses in random order and selected two examples illustrating each outcome.

Each figure includes the prompt, unsteered and steered response excerpts, and recorded source-to-target concept mappings at their actual edit positions. For example, steering changes a conspiracy-promoting response into a correction, and a harmful stereotype into a discussion of systemic barriers and support. The failure examples show repetition and garbled output. Together, these examples make both the generated changes and the corresponding activation edits visible.

Q2. Is the risk accumulation a potential issue in longer contexts? What's the distribution of the scores s_c, and how stable are they on a held out test set of benign and harmful tokens?

A2. Thank you for raising this concern. We examined token hazards, cumulative risk, and concept harmfulness scores on held-out test data across all six generator–dataset settings. Since the datasets provide response-level labels, we grouped tokens according to whether their responses were labeled safe or harmful.

Tokens in safe responses generally receive much smaller hazards. Giving each response equal weight, the median token hazard is more than 400 times as large in harmful responses as in safe responses in every setting. On Qwen/WildGuard, the medians are 0.00579 and 0.0000114, respectively. This separation is still there after risk accumulation: mean final risk is at most 0.103 for safe responses and at least 0.830 for harmful responses across all six settings.

We selected safe responses containing at least 1,024 tokens and measured their risk increase between tokens 256 and 1,024. On WildGuard, 331 Qwen responses and 877 InternLM responses reach this length. Their median increases in risk are 0.015 and 0.013, respectively. At token 1,024, 95.2% of these Qwen responses and 95.7% of these InternLM responses remain below their validation-selected thresholds. However, the false-positive rate increases from 1.8% at token 256 to 4.8% at token 1,024 for Qwen, and from 0.3% to 4.3% for InternLM. This shows that monotonic accumulation can increase false alarms on longer safe responses, although most remain below the detection threshold.

On the full test sets, detector thresholds selected for at most 5% validation FPR give less than 6% test FPR in every setting while detecting more than 82% of harmful responses. Appendices F.1.3 and F.1.4 show the score distributions, false-positive rates by response length, and risk growth within long responses.

For the concept scores (s_c), we checked whether the harmfulness associations learned during training persist on unseen responses. Across all six settings, mean scores are negative for tokens in safe responses, ranging from -0.715 to -0.395, and positive for tokens in harmful responses, ranging from 0.383 to 0.682. We weight tokens by the inverse of their response length, so long and short responses contribute equally.

We also recomputed concept scores using test response labels. Among concepts appearing in at least 25 held-out test responses, the training-derived and test-derived scores have Spearman correlations of 0.960–0.988, and 94.1–99.2% retain the same harmful or benign association. These results indicate that the learned concept harmfulness associations remain stable on held-out responses, particularly for concepts with sufficient test-set support. Appendix F.1.5 reports these distributions and comparisons.

Q3. Does the learned codebook contribute beyond simpler alternatives? How do detection, steering, and audit results change if the CLVQ-VAE is replaced by (a) k-means clustering of the same read-layer activations, using the same label-derived harmfulness scores, or (b) SAE features, e.g. as in NExT-Guard?

Thank you for suggesting this comparison. We added a K-means comparison across all three generators and both datasets (Appendix F.3). Detection performance remains close, but TRACE achieves lower average ASR and causes fewer safe responses to fail the quality checks after steering.

We replaced CLVQ-VAE with 800 K-means clusters of the same read-layer activations. We kept the training responses, harmfulness-scoring formula, detector architecture and hyperparameters, and steering procedure unchanged. Both methods use the same validation rule to select steering strength.

TRACE has higher response F1 in all six settings and higher streaming F1 in five. For example, on Llama/WildGuard, response and streaming F1 are 0.794 and 0.789 for TRACE, compared with 0.756 and 0.774 for K-means.

Across the six settings, TRACE achieves lower ASR in five, with an average of 0.273 versus 0.499 for K-means. The exception is Llama/WildGuard, where K-means achieves lower ASR but causes more safe responses to fail the quality checks. Across all six settings, an average of 8.9% of originally safe responses fail previously passed quality checks after TRACE steering, compared with 29.0% for K-means.

For auditing, we applied TRACE’s labeling procedure to the Qwen/WildGuard K-means clusters. TRACE has more concepts consistently labeled harmful or benign: 32 harmful and 126 benign, compared with 13 and 100 for K-means. K-means has more agreed names overall, 461 versus 422, with more neutral concepts. The CLVQ-VAE codebook therefore offers benefits in steering and identifies more harmful- and benign-labeled concepts, although it does not produce more named concepts overall.

We have not evaluated an SAE-based variant for two reasons. First, unlike K-means, SAE features are not a direct replacement for TRACE's single-concept assignment. TRACE assigns each token to one codebook concept, which determines whether an edit is needed and which benign target to use. An SAE can activate multiple features for the same token, so using these features for steering would require additional rules for selecting which features to edit and how to choose or combine benign targets. NExT-Guard uses SAE features for risk detection and early stopping but does not address these steering decisions. Second, reusing a pretrained SAE would require a checkpoint compatible with each generator and selected read layer. Otherwise, a suitable SAE would need to be trained and validated. For these reasons, we have not included an SAE-based steering comparison.


We sincerely thank the reviewer for their thoughtful feedback and would be very happy to clarify any remaining questions. If our response has addressed the concerns, we would be truly grateful if the reviewer would consider revising the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.