acceptodds
Under review as a conference paper at ICLR 2027

What Role Does Explicit Chain-of-Thought Play in Safety Moderation? A Circuit-Level Study

Abstract

In safety moderation, explicit chain-of-thought (CoT) is often taken as evidence that a model is making careful and trustworthy judgments. Yet it remains unclear whether generating CoT actually helps form the final moderation decision, or whether it mainly verbalizes a decision that is already taking shape internally. To answer this question, we compare direct-answer and full-CoT prompting in a reasoning-based safety moderation model and trace their internal computation at the circuit level. Across multiple benchmarks, we find that the two prompting modes often share a substantial attributed backbone, while disagreement is concentrated in a smaller set of boundary-related features. We also find that much of the model’s latent moderation prediction is already represented at intermediate layers, with later computation increasingly tied to expressing that preference at the output. These results suggest that the final label under full-CoT prompting does not typically emerge entirely from scratch during reasoning generation. Instead, even when direct-answer prompting produces a different final label, its internal state often already reflects support for the label produced under full-CoT prompting. Disagreement between the two modes arises mainly when the internal state under direct-answer prompting remains uncertain, which is also when explicit CoT appears to matter most. Based on this insight, we introduce an uncertainty-guided two-stage moderation pipeline that uses intermediate-layer sparse features for fast prediction and reserves full-CoT reasoning for uncertain cases only. The resulting system reduces reliance on expensive full-CoT inference while maintaining strong moderation performance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Safety Moderation, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZCVXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper reveals that explicit chain‑of‑thought mostly verbalizes pre‑existing latent moderation judgments and only alters outcomes for uncertain boundary cases via circuit‑level analysis of safety‑moderation models. Building on this mechanistic insight, it presents UTM, an uncertainty‑guided two‑stage moderation pipeline combining intermediate‑layer sparse‑feature readout and conditional full‑CoT fallback. UTM routes merely ~14% of samples to costly full‑CoT reasoning, obtains near‑full‑CoT F1 scores, and substantially reduces inference latency against reasoning‑based moderation baselines.

Critical strengths
  1. The problem is important and practically relevant. The paper investigates the real‑world role of explicit CoT in safety‑moderation models, which is both mechanistically puzzling and practically critical for trustworthy AI content safeguards.

  2. The method is practical. UTM works as a plug‑and‑play inference‑time pipeline and requires no retraining of the original safety model, bringing substantial inference‑cost reduction for reasoning‑based content moderation.

  3. The paper combines theoretical insights with practical application. It delivers circuit‑level mechanistic findings alongside the UTM moderation pipeline prototype.

Critical weaknesses
  1. The core mechanistic claim has restricted generalizability. Key circuit‑level causal evidence is derived primarily from RSafe, with only partial observational replication on GuardReasoner‑8B and no full validation across diverse safety‑moderation models.

  2. The interpretability analysis relies on linear approximations for nonlinear Transformer computations. Zero‑baseline EAP and feature‑negation interventions adopt first‑order linear assumptions to estimate feature contributions, which may introduce approximation error and bias the quantitative results of circuit attribution and causal flipping experiments.

  3. Practical deployment barriers remain substantial. UTM requires per‑model trained transcoders and feature extraction, which adds nontrivial engineering overhead and prevents simple off‑the‑box usage for arbitrary safety models.

  4. The methodological novelty is incremental. Transcoder‑based circuit tracing, edge attribution patching and intermediate‑layer readout are existing interpretability tools; this work’s main contribution lies in applying them to safety CoT rather than introducing fundamentally new interpretability techniques.

Questions for authors
  1. Beyond RSafe and the correlational observations on GuardReasoner‑8B, what indirect evidence or reasoning supports that the observed CoT mechanism exists across other safety‑moderation model frameworks, rather than being unique to RSafe’s training setup and architecture?

  2. What is the magnitude of approximation error introduced by zero‑baseline EAP and feature‑negation within nonlinear transformers? Can the authors provide empirical estimates or ablation studies to characterize these approximation effects?

  3. Can the authors report cross‑sample feature‑overlap baselines between circuits of fully unrelated safety inputs, to quantitatively disentangle input‑specific decision pathways from universal task‑background features?

  4. Can the authors systematically ablate different uncertainty thresholds tau_safe / tau_unsafe, reporting their joint impact on UTM F1 performance and stage‑2 routing ratio?

3 comments
AuthorsOpenReview ↗

Responses to Reviewer ZCVX (Part 1 / 3)

We sincerely appreciate the reviewer's recognition that the role of explicit CoT in safety moderation is an important and practically relevant problem. We have spent considerable time adding the empirical studies you suggested, which make our work more complete.

#[W1 & Q1 - Generalizability beyond RSafe.]: We explain why RSafe and GuardReasoner are representative of reasoning moderators, and summarize the new experiments on GuardReasoner-8B.

Thank you for this important question.

(a) Why these two models

Our study concerns a moderator's own reasoning before it decides, so it requires a moderator that reasons before answering, which makes the Direct and CoT modes comparable. The newer moderators we considered do not meet this requirement. Qwen3Guard hard-codes an empty thinking block and emits its label directly after "Safety:", and YuFeng-XGuard outputs its risk category as the first token, with an explanation afterwards.

RSafe and GuardReasoner represent the two main training paradigms of reasoning moderators, reinforcement learning (RSafe) and supervised fine-tuning on reasoning traces followed by preference optimization (GuardReasoner). They also differ in architecture, Qwen2.5-7B for RSafe and Llama-3.1-8B for GuardReasoner. Other reasoning moderators differ in implementation details but follow one of these two paradigms, so findings shared by both models are unlikely to be specific to RSafe's training setup or architecture.

(b) New experiments on GuardReasoner-8B

Feature interventions (our reply to Reviewer rK96, W3 (a)). Negating the label-associated features flips all 96 GuardReasoner labels in both modes, and setting them to zero removes 80–90% of the decision margin, so these features are causally necessary, as in RSafe.

Label-word replacement (Q3 below, and our replies to Reviewer rK96, W3 (b), and Reviewer Eo7F, W1 & Q1). GuardReasoner copies its label from its own CoT, whereas RSafe copies it from the system prompt. This explains the lower overlap and the persistent CoT attribution of GuardReasoner in App. D.8.

The main findings therefore hold in both models, now with causal evidence. Both models separate label selection from label copy, their label-associated features are causally necessary, and their Direct and CoT circuits diverge more on disagreements. In the revised version, we will add these experiments to App. D.8.

#[W2 & Q2 - Approximation error of linear attribution.]: Attribution is used only to select features, every causal effect is measured with an actual intervention, and we quantify the first-order error.

Thank you for raising this point. We agree that zero-baseline EAP is a first-order approximation. We therefore use attribution only to build the graphs and select candidate features, and we measure every causal effect with an actual intervention. In Table 1 and in all new intervention experiments, the selected features are modified and the model is run forward again, with all MLP and LayerNorm layers recomputed and attention patterns kept fixed, as in the standard circuit-tracer intervention. The reported label flips are therefore measured outcomes, not first-order estimates.

To quantify the approximation error (Q2), we compared the attribution-based prediction with the measured change of the label-logit margin in the full model when a fraction α of the selected feature contributions is removed (8 cases in Direct and CoT mode, three feature groups each).

α0.00010.0010.010.05
Mean absolute error of the attribution prediction (logits)0.00005–0.00130.0003–0.0130.003–0.150.014–2.1

When the attention patterns, normalization scales and MLP outputs are frozen, as the attribution assumes, prediction and measurement agree within 2×10⁻⁴ logits, so the graph edges are computed correctly. In the full model, the error is negligible for small perturbations but grows with their size, up to about 2 logits when 5% of the contributions are removed. Attribution scores should therefore be read as a ranking of candidate features rather than as effect sizes, which is why all our causal claims rest on interventions. The key mechanism results also hold in an experiment that uses neither transcoders nor attribution. In the revised version, we will report these error estimates and state this distinction explicitly.

AuthorsOpenReview ↗

Responses to Reviewer ZCVX (Part 2 / 3)

#[W3 - Deployment overhead.]: UTM only adds a one-time training cost and is efficient to deploy.

Thank you for raising this practical concern. As the reviewer notes, UTM does not retrain the moderator, and its extra cost is offline. Stage 1 needs the transcoder encoder of a single layer (L22) and a logistic readout, both trained once per model. At inference, Stage 1 stops the forward pass at layer 22 and adds one encoder multiplication, so it is faster than even Direct answering on one A100, and the full pipeline serves 5.6× more requests per second than Full CoT (our reply to Reviewer Eo7F, W2). When training a transcoder is not desired, a dense probe at the same layer can replace the sparse readout with a small loss in F1, at most 0.006 at the same routing budget (our reply to Reviewer rK96, W1 & Q1). The routing pipeline itself therefore does not depend on transcoders. The readout and the thresholds still need to be fitted for each new model, and we will state this cost in the revised version.

#[W4 - Incremental methodological novelty.]: Our contribution is the question, the mechanistic findings and their use in UTM, not a new interpretability tool.

We agree that transcoders, attribution graphs and intermediate-layer readouts are existing tools, and we do not claim a new interpretability technique. Our motivation is a question that matters for trustworthy moderation. When a safety moderator reasons before deciding, does the reasoning determine the decision, or does it mostly verbalize a decision that has already formed? Our contributions are the answers and their consequences.

  1. The moderation label is selected from the query and then copied into the answer, and CoT mostly carries an existing judgment, changing the label mainly on uncertain inputs.
  2. Where the label is copied from depends on the moderator, the system prompt in RSafe and its own CoT in GuardReasoner. We establish this with targeted experiments, namely instruction rewording, label-word replacement and unrelated-input baselines.
  3. These findings directly yield UTM, which invokes CoT only when the Direct-mode state is uncertain and keeps CoT's benefit at a fraction of its cost.

We will make this positioning explicit in the introduction of the revised version.

#[Q3 - Separating input-specific decision pathways from task-background features.]: We separate them with targeted experiments that change one part while holding the other fixed.

Thank you for this question. The input-specific decision pathway corresponds to label selection, which chooses the label from the evidence in the query, and the task-background features correspond to label copy, which writes the chosen label into the answer format for every input. A cross-sample overlap baseline shows how much two circuits share, but not which features play which role. We therefore separate the two directly, by changing one while holding the other fixed.

Experiment 1: reword the instruction, keep the query and the verdict (RSafe). On 48 held-out queries from OpenAI, Aegis, XSTest and Toxic-Chat, half Direct/CoT agreements and half disagreements, we reworded every instruction sentence, block marker and all 18 category descriptions, leaving 17% 4-gram overlap with the original, while keeping the query, category numbering and answer format. We then regenerated the CoT, attributed the graphs again, and compared each graph with its reworded version on the 29 queries whose Direct and CoT verdicts are both unchanged.

DirectCoT
Query-token features in layers 0–7 that survive rewording97.2%98.7%
Removed features recurring in ≥15/29 queries (share of all removals)1,037 (82.5%)1,049 (82.7%)
Added features recurring in ≥15/29 queries (share of all additions)836 (75.2%)838 (77.8%)

Label selection is query-driven and survives the rewording almost entirely, since 97.2% (Direct) and 98.7% (CoT) of the query-token features in layers 0–7 are retained. Label copy behaves like task background. When the instruction changes, the same features are removed and added for nearly every query, since 82.5% (Direct) and 82.7% (CoT) of the removed features and 75.2% and 77.8% of the added features recur in at least 15 of the 29 queries.

AuthorsOpenReview ↗

Responses to Reviewer ZCVX (Part 3 / 3)

Experiment 2: replace the label words (RSafe and GuardReasoner). In the label-word replacement test of W1 & Q1, the query and the recorded CoT are kept, and only the label words are replaced with A/B. Replacing them only in the system prompt changes RSafe's answer to A/B (P(A or B) = 0.970), so its output label is copied from the template rather than computed from the input, while GuardReasoner copies it from its own CoT.

ModelA/B placed inP(A or B)P(original label word)
GuardReasonersystem prompt only0.0090.991
GuardReasonerCoT only0.2550.744
GuardReasonerboth0.5270.473
RSafesystem prompt only0.9700.000
RSafeCoT only0.0001.000
RSafeboth0.9710.000

Together, the two experiments identify the input-specific pathway as the query-driven label selection and the shared background as the template-driven label copy. In the revised version, we will add both experiments and describe the shared computation in these terms.

#[Q4 - Sensitivity to τ_safe and τ_unsafe.]: We report UTM F1 and the routing ratio over a joint grid of thresholds.

For RSafe on the four Table 4 test sets, we evaluate UTM S1+2 over τ_safe ∈ {0, 0.01, 0.05, 0.1, 0.3} and τ_unsafe ∈ {0.7, 0.9, 0.95, 0.99, 1}. The thresholds are not tuned on these test sets, and the default (0.05, 0.95) row corresponds to UTM S1+2 in the revised Table 4. The symmetric bands are shown below (F1 / routed share).

τ_safe, τ_unsafeOpenAIAegisXSTestSafeRLHF
0.3, 0.70.783 / 1.6%0.822 / 1.2%0.851 / 1.6%0.895 / 3.4%
0.1, 0.90.788 / 4.5%0.821 / 4.1%0.863 / 4.1%0.907 / 9.0%
0.05, 0.95 (default)0.792 / 5.6%0.820 / 4.1%0.864 / 4.2%0.915 / 14.9%
0.01, 0.990.779 / 11.0%0.831 / 8.3%0.865 / 8.8%0.917 / 19.3%
0, 1 (Full CoT)0.771 / 100%0.831 / 100%0.870 / 100%0.921 / 100%

We use the default thresholds because they balance efficiency and accuracy. They send only a small share of inputs to CoT, while keeping F1 close to that of Full CoT. In the revised version, we will add the full grid to the appendix.

Reviewer_rK96OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the setting of LLM-as-a-judge for moderation tasks, analyzing the role of the CoT in the LLM judge. The authors ask whether and how explicit CoT prompting impacts the eventual decisions of the LLM judge, finding little difference between the moderation decision F1 scores from CoT prompting and direct prompting. Further, the paper investigates the mechanisms underlying this using sparse feature analysis (transcoders and attribution graphs), describing a “label selection -> label copy” mechanism, and finding that the attributed features largely overlap between the two prompting modes, with disagreement concentrated in a few boundary features, though the overlap holds for only one of the two studied models. Finally, they propose UTM, which trains a linear classifier on transcoder features from intermediate activations, and triages to full CoT judgements only when the classifier has low confidence.

Critical strengths
  • Mechanistic analysis of moderation models is timely and underexplored. Indeed, if we are using LLMs to make moderation decisions, and if we cannot understand or scrutinize the decision process, the judgements become black box, limiting their usefulness and safety.

  • The interventions on features provide some causal evidence for their importance to the model decisions, rather than solely correlational observations. This makes the claim about the observed label selection / label-copy mechanism more causal.

  • The two models the paper studies, RSafe and GuardReasoner-8B (fine-tuned with RL and SFT, respectively) are quite different from each other, implying that shared findings between them (such as greater feature divergence in disagreement cases) are not specific to one type of training recipe for the moderation model.

Critical weaknesses
  • Missing an important baseline: Stage 1 of UTM is a probe over the sparse-feature basis. The natural baseline to compare against is a probe over the dense activations of the model at the same location as the transcoder features. The authors do include a logit lens in table 9, which seems to nearly match the performance of the transcoder readout, and is not yet trained on the probe task. In general, dense probes tend to outperform sparse feature probes. I could understand the argument for using transcoder features being the monosemanticity (and thus interpretability) of these features, but the authors would need to make that argument, and show that indeed the transcoder features are interpretable, which is contested in the community (see Kantamneni et al., ICML 2025).

  • The mechanistic analysis is missing important controls: The observation of a largely overlapping feature backbone could largely be attributed to the overlapping prompts. Per appendix E2, the two prompting modes are identical until the CoT itself. Thus, activations and KV are identical up to the end of the prompt, so the high overlap is expected, by construction. Additionally, the interventions of flipping the feature signs important, but pushes the model very out of distribution. Ablating features (setting their activations to 0, rather than negative) is a more standard approach for establishing feature causality, which is missing. The random control (Table 1) may target dead/inactive features. It would help if authors either included feature ablation interventions, or argued why sign-flipping interventions are correct. It would also help if the authors include the actual text of the CoT vs direct prompts, which could explain why so much of the features overlap in RSafe, but not GuardReasoner.

  • Negative result for GuardReasoner undermines the mechanistic claims. The second model is only partially analyzed (missing the boundary-feature and intervention analyses). Where it does get the same analysis, the findings do not replicate; only 21-37% of features shared (Table 10), and the CoT segment attribution persists into later laters (Table 11). The only finding that does hold is that the divergence in features is greater in disagreement cases. I appreciate that the authors are candid about this in the appendix and provide a plausible explanation, but without testing it, the claim is substantially weaker. Finally, both RSafe and GuardReasoner are over 16 months old, and while the paper mentions more recent models like Qwen3Guard, they do not analyze them.

  • Smaller things:

    • The text on several figures is too small to read
    • Several references lack years (Zhou et al., Chen et al. a/b, Barez et al., Xiong et al., Liu et al., Zheng et al., Hanna et al. 2024, Marks et al)
    • Title and intro come across as CoT monitoring, but the paper studies the moderator’s own CoT.
    • Full prompts are never shown
    • Section 3 says layers after 22 are replaced by MLP’s, but Eq.11 shows a linear classifier?
    • Equation 5 adds an l1 penalty to the TopK transcoder, which is redundant?
    • Missing several closely related works: NoThinking (Ma et al., 2025), Reasoning Models Know When They’re Right (Zhang et al., 2025), Reasoning Theater from (Goodfire, Bopping et al., 2026)
Questions for authors
  • How does a linear probe on the dense residual stream compare to the transcoder based one? Or, why use transcoder features at all? The logit lens seems to be near the performance of the transcoder readout, so I would expect the dense probe to beat it, as also found by GDM in their Mech Interp Progress Update blog (Rajamanoharan, S., et al., 2025).

  • On the inputs UTM routes to stage 2, how accurate are the direct-answer vs CoT approaches? In other words, does the full CoT actually improve over direct answer on these cases? The motivation of UTM implies that CoT is substantially stronger and worth preserving via UTM, but F1 scores are quite close between the two (~1-3 point difference). It would be great to see the full-pipeline costs (stage 1 + full routed CoT).

  • How much of the observed feature overlap survives when the prompt templates are differently worded, but still semantically equivalent? Showing this would tease out if the overlap is simply because of the shared prompt prefix between the two settings.

5 comments
AuthorsOpenReview ↗

Responses to Reviewer rK96 (Part 1 / 5)

We sincerely appreciate the reviewer's constructive suggestions. We have spent considerable time adding the empirical studies you suggested, which make our work more complete.

#[W1 & Q1 - Dense probe baseline at the same location.]: Under identical training and the same routing budget, the sparse readout matches or slightly exceeds the dense probe.

Thank you for suggesting this natural baseline. For RSafe, we added the baseline the reviewer asks for: a dense probe at the same location, trained in exactly the same way as UTM's sparse readout. To compare the scores at the same cost, each of them routes the 10% of inputs closest to p = 0.5 to CoT, and routed inputs take the Full-CoT label. Because of this common routing budget, the S1+2 results differ from those in Table 4, where each input is routed by the uncertainty band (Eq. 12). The results (F1) are shown below.

ReadoutOpenAIAegisXSTestSafeRLHF
Dense, S10.7790.8130.8440.887
Sparse, S10.7840.8160.8560.893
Dense, 10% routing budget0.7740.8260.8480.908
Sparse, 10% routing budget0.7750.8320.8510.911

Under identical training and the same routing budget, the sparse readout is never worse than the dense probe. It is equal or slightly better on all four datasets. A plausible reason is that sparsity filters out task-irrelevant signal. Many weak directions that are unrelated to the moderation decision are therefore removed, and the readout is more focused on the task. With a narrow binary output (safe vs. unsafe), this focus helps. Prior work finds dense probes stronger in other settings, and we do not claim that sparse probes are better in general. In the revised version, we will add the dense baseline and state this finding, together with its task dependence.

#[W2 & Q3 - Feature ablation, and overlap due to the shared prompt prefix.]: Feature ablation also consistently weakens the decision, and the shared computation is query-driven label selection plus a template-driven label copy, whose source differs between the models.

Thank you for these insightful comments, which helped us understand the overlap better. We answer the two points in turn: (a) feature ablation, and (b) the overlap and the shared prompt prefix, together with Q3.

(a) Feature ablation instead of sign flipping

We added the standard ablation and compared it with the sign flipping used in Table 1, under the same protocol. For each held-out dataset, the label-associated features are taken from the graphs of the other three datasets, and the features of the input's own label are either set to 0 (ablation) or negated, across all layers. We tested new inputs from OpenAI, Aegis, XSTest and Toxic-Chat, 192 for RSafe (48 per dataset, half safe and half unsafe) and 96 for GuardReasoner (24 per dataset, in both Direct and CoT mode).

Model / modeInputsAblation (set to 0)Negation
RSafe, Direct1927191
GuardReasoner, Direct961596
GuardReasoner, CoT962896

Feature ablation removes these features' support for the current label. However, the model's decisions are usually confident, and the remaining evidence is enough to keep the label. Negation goes further, removing the support and adding an equal amount of opposite evidence, which is why it flips far more labels and also why it is the less natural intervention. In the revised version, Table 1 will report both interventions.

AuthorsOpenReview ↗

Responses to Reviewer rK96 (Part 2 / 5)

(b) Overlap and the shared prompt prefix (W2 and Q3)

The concern does not contradict our conclusion; it sharpens it. We agree that the identical prefix makes much of the overlap expected. The prefix matters especially for the label copy step, which writes the label into the answer and is closely tied to the system prompt. We therefore first show the system prompts of the two models (CoT mode, with the query as a placeholder; the full Direct and CoT prompts will be given in the appendix of the revised version).

RSafe:

You are a policy expert trying to help determine whether a user prompt is in violation of the defined safety policies.
Task: Check if there is unsafe content in 'User' messages in conversation according to out safety policy with the below categories.
    <BEGIN UNSAFE CONTENT CATEGORIES>
U1  - Violence & Physical Harm
...(U2–U17 omitted) 
U18 - National‑Security Espionage
    <END UNSAFE CONTENT CATEGORIES>
<BEGIN CONVERSATION>
User: <USER_QUERY>
<END CONVERSATION>

You FIRST go through a step-by-step reasoning process and then provide the final answer. The thinking process MUST BE enclosed within <think> </think> tags.
Your final answer safe or unsafe MUST BE put in \boxed{}

GuardReasoner:

You are a classifier for analyzing interactions between humans and AI. 

Given a request from a human user and a response from an AI assistant, finish three tasks.

Task 1: determine whether the human user's request is harmful or unharmful.
Task 2: determine whether the AI assistant's response is a refusal or compliance. 
Task 3: determine whether the AI assistant's response is harmful or unharmful. 

You must think step by step. Keep consistency between the reasoning and the Answers.

Human user:
<USER_QUERY>

AI assistant:
None

How the system prompt may shape the circuit.

  • RSafe's system prompt gives the model much more to work with: a detailed policy, 18 categories, and the exact answer format. GuardReasoner's only names the label words.
  • We therefore hypothesized that the label copy step reads from different sources in the two models:
    • RSafe copies its label from the system prompt, which both modes share. This would make their late computation largely shared.
    • GuardReasoner copies its label from its own CoT, which restates the label several times per trace and which Direct mode does not have. This would make its late computation mode-specific.
  • This is consistent with the lower overlap and the persistent CoT attribution of GuardReasoner (Tables 10 and 11). The two experiments below test the hypothesis.

Experiment 1: reword the instruction, keep the query and the verdict (RSafe).

On 48 held-out queries from OpenAI, Aegis, XSTest and Toxic-Chat, half Direct/CoT agreements and half disagreements, we reworded every instruction sentence, block marker and all 18 category descriptions, leaving 17% 4-gram overlap with the original, while keeping the query, category numbering and answer format. We then regenerated the CoT, attributed the graphs again, and compared each graph with its reworded version on the 29 queries whose Direct and CoT verdicts are both unchanged. The results are shown below.

DirectCoT
Query-token features in layers 0–7 that survive rewording97.2%98.7%
Removed features recurring in ≥15/29 queries (share of all removals)1,037 (82.5%)1,049 (82.7%)
Added features recurring in ≥15/29 queries (share of all additions)836 (75.2%)838 (77.8%)

The table shows that the shared computation has two parts. Label selection is query-driven and survives the rewording almost entirely, since 97.2% (Direct) and 98.7% (CoT) of the query-token features in layers 0–7 are retained. Label copy is a template-driven pathway that is rebuilt in the same way for nearly every query when the instruction changes. Of the features removed by the rewording, 82.5% (Direct) and 82.7% (CoT) recur in at least 15 of the 29 queries, and so do 75.2% and 77.8% of the added features. For Q3, this means that what survives a semantically equivalent rewording is the query-driven selection, and what changes is the template-bound copy.

AuthorsOpenReview ↗

Responses to Reviewer rK96 (Part 3 / 5)

Experiment 2: where is the label copied from? (RSafe and GuardReasoner). We tested where each model copies its label from on the 637 recorded CoT cases used for the GuardReasoner graphs, with RSafe run on the same queries. Keeping each model's own recorded CoT, we replace the label words with A/B only in the system prompt, only in the CoT, or in both, using two opposite mappings and leaving the query unchanged. We then measure the probability of A/B and of the original label word at the label position.

ModelA/B placed inP(A or B)P(original label word)
GuardReasonersystem prompt only0.0090.991
GuardReasonerCoT only0.2550.744
GuardReasonerboth0.5270.473
RSafesystem prompt only0.9700.000
RSafeCoT only0.0001.000
RSafeboth0.9710.000

When GuardReasoner outputs a letter, it is always the letter its CoT uses for its own conclusion. Together with the table, this shows that RSafe copies its label from the system prompt, whereas GuardReasoner copies it from its own CoT. The two experiments together explain the overlap. In RSafe, the high Direct/CoT overlap reflects two shared parts, label selection from the query and label copy from the shared system prompt.

The identical prefix therefore does contribute to the overlap, as the reviewer noted, because the copy step reads from it in both modes. GuardReasoner also separates selection from copy, but its copy step reads from its own CoT, which Direct mode does not have. Its late computation thus differs between the two modes, which explains its lower overlap and persistent CoT attribution. In the revised version, we will restate the "shared backbone" finding accordingly. The shared part is query-driven label selection plus a template-driven label copy, and how much is shared depends on where the moderator copies its label from.

#[W3 - Missing interventions on GuardReasoner, and the choice of models.]: GuardReasoner's labels depend causally on its label-associated features, the explanation in App. D.8 is now supported by causal evidence, and RSafe and GuardReasoner cover the two main training paradigms.

Thank you for this careful critique. We answer the three points in turn: (a) the missing interventions, (b) the test of the explanation, and (c) the choice of models.

(a) Feature interventions on GuardReasoner-8B

The second model lacked the causal intervention analysis that supports the RSafe findings. We therefore ran the Table 1 protocol on GuardReasoner-8B. We used 96 new inputs that were not used for the graphs or the catalogs, 24 from each of OpenAI, Aegis, XSTest and Toxic-Chat, balanced by GuardReasoner's own Direct label, and tested each in both Direct and CoT mode. For each held-out dataset, the label-associated features were taken from the graphs of the other three datasets. We set the features of the input's own label to zero or to their negative value, across all layers.

InterventionDirectCoT
set to zero1528
negated9696

Negating the label-associated features flips all 96 labels in both modes, and setting them to zero has a weaker but non-zero effect (15 and 28 flips). GuardReasoner's labels therefore depend causally on its label-associated features, as in RSafe. Where GuardReasoner copies its label from is tested in (b).

Reviewer_Eo7FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies how explicit CoT contribute to models safety moderation. Primarily in RSafe.

The main comparison is CoT vs direct-answer (without reasoning).

Firstly, transcoders + EAP used to find Circuits that correspond to the final moderation label. For that authors used constructed agreement and disagreement prompt pairs for each setting.

Authors report large shared feature backbone and divergence on disagreements. Some patterns are interpreted as label selection vs label copying. Additionally feature-negation interventions demonstrate that selected feature sets can change predictions.

The uncertainty-guided moderation method called UTM uses readout features from layer 22 and predicts probs, if the uncertainty is high then the full CoT is invoked. Thus UTM achieves efficiency–performance trade-off.

Critical strengths
  • Causal Experiments support analytical findings: large label flip rate reported in table 1 support the cusal claim thet the selected circuit components affect the output label.
  • Experiments on other GuardReasoner-8B model: App D.8 show results for GuardReasoner-8B model. Here the shared features are not the majority as it was in RSafe.
Critical weaknesses
  • Causal experiments is not enough to establish proposed roles of selected features: large flip rates in Table 1 do not distinguish label selection from label copy. It is also unclear if the direct answer state already supports the CoT label when the two predictions disagree.
  • Evaluation of the UTM is incomplete: While main F1 results show both S1, S1+2 latency (in App D.6) is reported only for S1.
Questions for authors
  • Can you show that intermediate states under direct-answer prompting predict the eventual CoT label specifically on disagreement examples? The current readout is trained on direct-answer labels, which does not directly test this claim.
  • Which Stage 1 architecture was actually evaluated? The two-layer MLPs (lines 384-385) or the logistic classifier (Eq 11)?
  • Can you also comparisons your method agains simpler alternatives: a direct-answer (probabilities not from features but from answer logits) confidence-based fallback and a dense intermediate-state (without selected features) classifier?
6 comments
AuthorsOpenReview ↗

Responses to Reviewer Eo7F (Part 1 / 3)

We are grateful for the constructive suggestions. We have spent considerable time adding the empirical studies you suggested, which make our work more complete.

#[W1 & Q1 - Table 1 cannot separate label selection from label copy, and whether the Direct state supports the CoT label on disagreements is untested.]: We add experiments to answer these questions.

Thank you for these insightful questions. We first separate label selection from label copy (a), and then test whether the Direct-answer state already supports the CoT label on disagreements (b, Q1).

(a) Separating label selection from label copy

A large flip rate shows that the selected features are necessary for the output, but not which role they play. Label selection chooses the label from the evidence in the query, whereas label copy writes an already chosen label into the answer by reading the label word from the context. The two can therefore be separated by changing one while holding the other fixed. Experiment 1 rewords the instruction but keeps the query and the verdict, so selection should survive while the template-bound copy should change. Experiment 2 leaves the query unchanged and only replaces the label words in the context, which should affect the copy step but not the selection.

Experiment 1: reword the instruction, keep the query and the verdict (RSafe).

On 48 held-out queries from OpenAI, Aegis, XSTest and Toxic-Chat, half Direct/CoT agreements and half disagreements, we reworded every instruction sentence, block marker and all 18 category descriptions, leaving 17% 4-gram overlap with the original, while keeping the query, category numbering and answer format. We then regenerated the CoT, attributed the graphs again, and compared each graph with its reworded version on the 29 queries whose Direct and CoT verdicts are both unchanged. The results are shown below.

DirectCoT
Query-token features in layers 0–7 that survive rewording97.2%98.7%
Removed features recurring in ≥15/29 queries (share of all removals)1,037 (82.5%)1,049 (82.7%)
Added features recurring in ≥15/29 queries (share of all additions)836 (75.2%)838 (77.8%)

The table shows that the shared computation has two parts. Label selection is query-driven and survives the rewording almost entirely, since 97.2% (Direct) and 98.7% (CoT) of the query-token features in layers 0–7 are retained. Label copy is a template-driven pathway that is rebuilt in the same way for nearly every query when the instruction changes. Of the features removed by the rewording, 82.5% (Direct) and 82.7% (CoT) recur in at least 15 of the 29 queries, and so do 75.2% and 77.8% of the added features. This means that what survives a semantically equivalent rewording is the query-driven selection, and what changes is the template-bound copy.

Experiment 2: where is the label copied from? (RSafe and GuardReasoner). We tested where each model copies its label from on the 637 recorded CoT cases used for the GuardReasoner graphs, with RSafe run on the same queries. Keeping each model's own recorded CoT, we replace the label words with A/B only in the system prompt, only in the CoT, or in both, using two opposite mappings and leaving the query unchanged. We then measure the probability of A/B and of the original label word at the label position.

ModelA/B placed inP(A or B)P(original label word)
GuardReasonersystem prompt only0.0090.991
GuardReasonerCoT only0.2550.744
GuardReasonerboth0.5270.473
RSafesystem prompt only0.9700.000
RSafeCoT only0.0001.000
RSafeboth0.9710.000

When GuardReasoner outputs a letter, it is always the letter its CoT uses for its own conclusion. Together with the table, this shows that RSafe copies its label from the system prompt, whereas GuardReasoner copies it from its own CoT. Label copy is therefore a separate step in both models. Table 1 establishes that the selected features are necessary, and the two experiments separate their roles. A query-specific judgment that survives rewording of the instruction (selection, Experiment 1) is followed by a separate step that copies the label word from the context into the answer (copy, Experiment 2). Experiment 1 is graph evidence, and Experiment 2 is the causal test. In the revised version, we will present Table 1 as necessity evidence and add both experiments.

AuthorsOpenReview ↗

Responses to Reviewer Eo7F (Part 2 / 3)

(b) Does the Direct-answer state support the CoT label on disagreements? (Q1)

We test whether the intermediate state under Direct prompting already reveals the label that CoT will produce, especially on the inputs where the two modes disagree. We apply the L22 readout to held-out OpenAI, Aegis and XSTest inputs and compare disagreement inputs with agreement inputs.

Probability of the non-Direct labelOpenAIAegisXSTest
disagreement0.1920.3140.177
agreement0.0310.0250.018

The readout is trained only on Direct labels, yet on disagreement inputs it gives the eventual CoT label 6–13× more probability than on agreement inputs, and this separates disagreements from agreements. The CoT prediction is therefore already present in the Direct-mode state, as uncertainty rather than as a decision, since the Direct label still dominates on average. This is the signal UTM uses. In the revised version, we will add this analysis and state the claim in this form.

#[W2 - Latency of S1+S2.]: We time the full pipeline (Stage 1 + routed CoT) end to end on an A100.

Thank you for pointing this out. Stage 2 runs the full CoT for every routed input, so the cost of the deployable system is S1+S2, not S1 alone. We therefore timed RSafe's S1+S2 end to end on one A100, with a fresh Full-CoT generation for every routed input. We timed each method on the four Table 4 datasets, over three rounds in random order. The results are shown below.

Methodp50 (ms)p95 (ms)Mean (ms)Req/sSpeedup (Req/s)
Full CoT1736.42790.02172.10.461.00×
Direct Answer52.277.258.317.1537.25×
UTM S142.063.247.321.1445.93×
UTM S1+243.02114.4386.52.595.62×

Most requests stop after Stage 1, so the p50 stays at the Stage 1 level, while the p95 is set by the routed requests. The gain is therefore in throughput and mean latency. These numbers differ from Table 8 because the setting differs. Table 8 covered Stage 1 only and used OpenAI-Mod requests. Here the requests come from all four datasets, each method is timed over three rounds, and the speedup is computed from Req/s so that it includes the routed CoT. We will replace Table 8 with these measurements in the revised version.

#[Q2 - Which Stage 1 architecture was evaluated.]: The logistic classifier of Eq. 11, on top of the frozen layer-22 transcoder encoder.

Thank you for this question. The two descriptions refer to the same Stage 1, which has two layers. The first is the frozen layer-22 transcoder encoder, at=TopK(Wencxt+benc)a_t = \mathrm{TopK}(W_{\mathrm{enc}} x_t + b_{\mathrm{enc}}) with k = 32, max-pooled over the tokens from the start of the query to the last Direct token, z=max⁡tatz = \max_t a_t. The second is the trained logistic classifier of Eq. 11, p(unsafe)=σ(w⊤(zS/s)+b)p(\text{unsafe}) = \sigma(w^\top (z_S/s) + b), over the features SS active in at least five training inputs, with a per-feature scaling ss fitted on the training split. Only the second layer is trained, and there is no other hidden layer. Stage 1 stops the forward pass at layer 22, which is where its latency advantage comes from (W2). In the revised version, we will rewrite lines 384–385 and Eq. 11 to define the whole Stage 1 in one equation and state that only ww and bb are trained.

AuthorsOpenReview ↗

Responses to Reviewer Eo7F (Part 3 / 3)

#[Q3 - Comparison with simpler alternatives.]: We replace UTM's Stage 1 score with a Direct-logit fallback and with a dense probe, keeping everything else fixed.

Thank you for suggesting these baselines. For RSafe, we added the baseline the reviewer asks for: a dense probe at the same location, trained in exactly the same way as UTM's sparse readout, and compared at the same routing budget. The Direct-logit score is the safe/unsafe softmax at the final Direct position, with no training. To compare the three scores at the same cost, each of them routes the 10% of inputs closest to p = 0.5 to CoT, and routed inputs take the Full-CoT label. Because of this common routing budget, the S1+2 results differ from those in Table 4, where each input is routed by the uncertainty band (Eq. 12). The results (F1) are shown below.

MethodOpenAIAegisXSTestSafeRLHF
Dense, 10% routing budget0.7740.8260.8480.908
Direct-logit, 10% routing budget0.7720.8340.8450.910
UTM, 10% routing budget0.7750.8320.8510.911

The dense state at the same layer and direct-logit fallback are therefore the weaker routing signal in comparison with the sparse probe. The logit score needs the full forward pass, while UTM's Stage 1 only runs up to layer 22 of 28, which makes it more efficient to deploy. A plausible reason is that sparsity filters out task-irrelevant signal. Many weak directions that are unrelated to the moderation decision are therefore removed, and the readout is more focused on the task. With a narrow binary output (safe vs. unsafe), this focus helps. In the revised version, we will add both baselines, including the matched budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.